Stopping Distributed IMAP/SMTP Brute-Force with Dovecot Tarpitting and Penalty Delays
Modern credential stuffing spreads a handful of attempts across thousands of residential-proxy IPs, and per-IP fail2ban barely touches it. Here's how to layer Dovecot's auth failure delay, per-identity penalty, and login tarpitting to make every attempt expensive — with the exact config, the penalty math, and an honest account of what tarpitting can't fix.
EvilMail TeamJuly 23, 202611 min read
The attack you're actually facing (not the one fail2ban was built for)
Pull a fresh Dovecot auth log and look for the pattern, not the volume:
Same mailbox. Five different source networks. One attempt each, spread across forty minutes. Multiply that by a botnet of a few thousand residential-proxy exits and you have a campaign that never trips a per-IP threshold, because from any single IP the attacker looks like a user who fat-fingered their password once and gave up.
This is the failure mode fail2ban was never designed for. Fail2ban counts events per source address inside a findtime window and bans when the count crosses maxretry. That model assumes the attacker is loud from one place — the classic dictionary attack: many passwords, one user, one IP. Against that it works beautifully. But the economics of abuse flipped years ago. Residential proxy pools are cheap, attempts are deliberately capped at one to three per IP to stay under every sane ban threshold, and usernames rotate against leaked-credential lists. The whole point is to be quiet everywhere.
So stop thinking about the source. The only two dimensions the attacker can't cheaply rotate are wall-clock time and identity (the mailbox being targeted). The principle behind everything below: make every failed attempt cost real time, and make that cost grow per identity, no matter where the packet came from. Dovecot already ships the primitives to do this. Layer them *underneath* fail2ban, not instead of it.
Dovecot's three levers, and where each one bites
There are three distinct mechanisms and people constantly confuse them:
`auth_failure_delay` — a flat delay the auth process adds before returning *every* failed authentication reply. Applies to all failures, everywhere, no state required.
The per-identity penalty (anvil-based) — Dovecot's anvil process tracks recent failures keyed on remote IP plus username, and multiplies the delay on each consecutive failure, doubling up to a ceiling. This is built in. No external service.
Login-process tarpitting and concurrency caps — mail_max_userip_connections plus service login limits hold or refuse connections so a flood can't exhaust your login workers.
Two clarifications that save hours of misconfiguration. First, the built-in anvil penalty is *not* the same thing as auth_policy_* HTTP forwarding (the weakforward mechanism that pushes decisions to an external policy server like weakforced). You want the built-in penalty for this job — no extra daemon. Second, version reality as of 2026: on Dovecot 2.3.x the anvil penalty is the well-trodden path; 2.4 reworked parts of the auth-policy plumbing, so verify with doveconf -n on your exact build rather than trusting a blog from 2019. Everything here targets 2.3.x semantics, which is still what most production mail stacks run.
Baseline: a flat failure delay that costs attackers, not users
Start with the one-liner. In /etc/dovecot/conf.d/10-auth.conf:
The math is the whole argument. A legitimate user who mistypes their password pays 2 seconds, once, then logs in. Nobody notices. But the attacker's connection is *blocked* for those 2 seconds before the failure is even returned — the delay is applied server-side, so the client can't short-circuit it by hanging up early. A spray hammering 10,000 accounts now serialises 2 seconds of dead air into every single bind. Across a botnet that's still throughput the attacker has to pay for: every proxy exit is sitting idle mid-handshake instead of moving to the next target. You've converted their cheap, parallel, fire-and-forget campaign into something that holds connections open and burns their proxy-time budget.
One caveat, and it's the reason the next section exists: a *too-large* delay is a liability. If you set auth_failure_delay = 30s, an attacker can deliberately open thousands of failing logins and make you hold them all open, tying up login processes. Two seconds is the sweet spot — long enough to hurt throughput, short enough that holding the connection isn't a resource win for the attacker. Pair it with the concurrency caps below.
Per-identity penalty: making repeat offenders pay exponentially
The flat delay is linear. The penalty is exponential, and that's what actually bends a determined campaign. Dovecot's anvil process remembers recent failures per (remote_ip, user) and doubles the delay on each consecutive failure: 0, 2, 4, 8, 16 seconds, up to a ceiling. A successful authentication resets that identity's counter to zero.
The key design decision is what you key on. Pure per-IP penalty is defeated by rotation — the botnet just moves to a fresh exit and starts the curve over at zero. The mitigation that matters is that the penalty accumulates on the username dimension too. A botnet rotating IPs but hammering [email protected] still watches that mailbox climb the curve, because the identity half of the key doesn't change no matter how many proxies they burn. Combine username-hash with the source /24 and you catch both the loud repeater and the account-focused spray.
Set the ceiling deliberately. You want compounding cost, but you never want to *soft-lock a real user out permanently*. Cap the added delay around 15–16 seconds. At that point an attacker guessing one identity is limited to roughly four attempts per minute against that mailbox regardless of how many IPs they own — a rate at which any realistic password is safe for geological timescales — while your genuinely-typo-prone user resets the moment they get it right.
Interaction to know about: auth_cache sits in front of the passdb, so a cached successful password can bypass a backend lookup — but failures still route through the penalty logic. Keep the cache for performance, it doesn't undermine the penalty:
auth_cache_size = 10M
auth_cache_ttl = 1h
Tarpitting the login process and capping concurrency
Delays only help if your login workers survive the flood long enough to apply them. In /etc/dovecot/conf.d/10-master.conf:
mail_max_userip_connections = 10 # imap; drop to 3 for pop3
service login {
process_limit = 128
client_limit = 64 # max concurrent logins = client_limit * process_limit
}
service auth {
client_limit = 12000 # must exceed sum of login client_limits + backends
}
mail_max_userip_connections caps how many simultaneous connections a single user from a single IP can hold — a bot that opens 200 sockets under one identity gets refused after ten. The service login product (client_limit * process_limit) is your absolute concurrent-login ceiling; size it to survive a burst without the OOM killer visiting Dovecot. Critically, service auth's client_limit must be larger than the sum of everything that talks to it, or auth itself becomes the bottleneck under load and legitimate users get "internal error."
The honest tradeoff: tarpitting holds sockets open, and those sockets are *your* file descriptors, not just the attacker's. Sizing process_limit and client_limit too generously lets an attacker exhaust your box; too tightly and a legitimate traffic spike self-DoSes. This is exactly why you still want a dumb per-IP connection cap at the firewall or HAProxy layer in front — let the cheap packet filter shed the obviously abusive concurrency before it ever reaches a login worker.
Wiring fail2ban as the blunt backstop, not the front line
Fail2ban is still worth running — for the loud single-IP dictionary attacks that the penalty curve would merely slow. Ban them outright and save the CPU. /etc/fail2ban/jail.local:
The dovecot filter matches the log signatures auth failed, 1 attempts in Xsecs, disconnected (auth failed, ... rip=<ip>), and Aborted login (auth failed). The recidive jail watches fail2ban's *own* log and escalates any IP that keeps earning short bans into a week-long one. Use nftables-multiport (or an ipset backend) rather than plain iptables — at thousands of banned addresses, a linear iptables chain adds latency to every packet, while nftables sets stay O(1).
The blunt truth: fail2ban is per-IP, so against the distributed case it is close to useless *on its own*. When each address makes one attempt and vanishes, it never reaches maxretry. That's not a reason to drop it — it's the reason the Dovecot layers live underneath it.
What tarpitting can't do — close the credential gap
Delays slow guessing. They do not stop a *correct* password from working on the first try. If a credential leaked, tarpitting buys you nothing on that account. So the auth layer is only half the job:
Hash with `default_pass_scheme = ARGON2ID` (Dovecot 2.3.15+). It's memory-hard and the right default in 2026 — preferred over BLF-CRYPT and SHA512-CRYPT. Existing hashes keep working; new and rotated passwords upgrade.
Issue per-application passwords so the real account secret never touches IMAP or SMTP AUTH. A leaked app password is scoped and revocable; the human password stays behind MFA on webmail only.
Monitor for the leaked-credential signature: a login that *succeeds* from an ASN or country the account has never used. That's your real breach signal, and no delay will surface it — alerting will.
SMTP AUTH has its own rules. Require STARTTLS+AUTH on submission (587), offer implicit TLS on 465, and make sure AUTH is off entirely on port 25 — 25 is for MX, never for authenticated relay. Finally, rate-limit *successful* outbound sends too. An attacker who does get valid credentials immediately pivots to spam, so cap it in Postfix:
Set auth_failure_delay = 2s and disable_plaintext_auth = yes in 10-auth.conf.
Enable the built-in per-identity penalty; confirm the doubling curve caps around 15–16s.
Cap concurrency: mail_max_userip_connections = 10 (3 for POP3).
Size service login and service auth limits so auth client_limit exceeds the sum of login workers.
Deploy the dovecot + recidive fail2ban jails with banaction = nftables-multiport.
Switch to default_pass_scheme = ARGON2ID and roll app passwords for IMAP/SMTP.
Require STARTTLS+AUTH on 587, disable AUTH on 25, cap smtpd_client_message_rate_limit.
Verify the config is live: doveconf -n | grep -iE 'auth_failure_delay|penalty|max_userip'.
Prove the delay end-to-end: time doveadm auth test [email protected] wrongpass — it should hang ~2s before failing, and climb on repeats.
That last command is the acceptance test. If it returns instantly, none of the above is actually loaded — reload Dovecot and check doveconf -n again before you trust the front door.
