High-Availability Dovecot: The Director Service and Shared Storage, Done Right
A single mailbox's index can only be safely written by one backend at a time. Put a round-robin load balancer in front of shared NFS and you'll corrupt dovecot.index.log the first time a user opens two connections. Here's how the Dovecot director enforces per-user backend affinity, and how to build a safe HA topology around that one invariant.
EvilMail TeamJuly 26, 202612 min read
A user has two devices open: a laptop on IMAP and a phone that just reconnected. Your load balancer is round-robin, so the laptop's connection lands on backend-A and the phone's on backend-B. Both open the same mailbox over NFS. Both read dovecot.index.log, both append transactions, both fsync. Now the log has two writers who never saw each other's appends. Dovecot notices the corruption on the next open and logs Broken file /var/mail/vhosts/example.com/alice/mdbox/mailboxes/INBOX/dbox-Mails/dovecot.index.log: ..., throws away the cache, and rebuilds. To the user, mail flickers out of the client and comes back minutes later after a reindex. Do it enough and you lose flags, \Seen state, and eventually message content.
The invariant that fixes this is one sentence: one mailbox is served by exactly one backend at a time. The director is the machinery that guarantees it across an entire frontend fleet. Everything else here — consistent hashing, the ring protocol, vhost weights, doveadm director move — exists to hold that single line true.
Why shared storage punishes concurrent access
High-Availability Dovecot with Director + Shared Storage | evilmail.pro — EvilMail Blog
NFS was designed for close-to-open consistency: a client is guaranteed to see another client's writes only after the writer closes the file and the reader opens it. Dovecot's index and transaction log are the opposite workload — long-lived, memory-mapped, appended to continuously by a process that assumes it is the only writer. Two backends touching the same mailbox violate every assumption at once:
Attribute cache lag. Backend-B caches the file size/mtime and never re-stats aggressively enough, so it appends over data backend-A already wrote.
Stale NFS file handles. A compaction or rename on one backend leaves the other holding a handle to a file that no longer exists: ESTALE, and the client sees an I/O error mid-session.
fcntl locks over NFS are advisory and fragile. They depend on rpc.lockd/NLM (or NFSv4's integrated locking) being healthy end to end. When lock state is lost on a server restart, two writers proceed as if they each hold the lock.
Dovecot once shipped mail_nfs_index and mail_nfs_storage to force cache flushes and make multi-writer *almost* work. In practice it never closed the race, and the maintainers effectively abandoned that path for the cleaner rule: never let two backends touch one mailbox. When director does its job you don't need those settings — and setting them anyway just adds fsync overhead that buys you nothing. The log lines you're trying to never see again: Broken file ... dovecot.index.log, Corrupted index cache file, and Maildir ... sync: UID inserted in the middle of mailbox.
The director in one sentence: sticky routing by user hash
The director is a specialized login proxy. When an authenticated session arrives, it hashes the username through a consistent-hash ring and maps it to exactly one backend. Every director node shares the same ring state over a ring protocol (TCP 9090), so all frontends agree on the same user-to-backend map. The session is then proxied to the chosen backend, which does the actual mailbox I/O.
The knobs you touch:
`director_servers` — the ring peers (the director nodes themselves).
`director_mail_servers` — the backends that hold mailboxes.
`director_username_hash` — what gets hashed; default %u (full user@domain). Use %d only if you deliberately want to pin a whole domain to one backend.
`director_user_expire` — how long an idle user's assignment is remembered; default 15 minutes. A user with no connections for this long can be reassigned.
vhost weight — default 100 per backend. A backend with double the RAM/IOPS gets 200 and receives roughly twice the users.
Frontends run imap-login/pop3-login/lmtp in director-aware mode. Backends do the real work and trust the proxied login via a master/auth token rather than re-authenticating.
The tiers, left to right: a keepalived VIP fronts haproxy (or LVS) doing plain TCP round-robin on 993/995/143/110 to N director nodes. The directors form a ring on 9090 and agree on the user map. Each director proxies the authenticated session to the assigned backend, and the backends mount the shared filer. Authentication lives on the frontend passdb; the backend accepts the proxied login through the master/auth mechanism.
One easy-to-miss requirement: LMTP must go through the director too. If your MTA delivers straight to a random backend while IMAP goes through director, delivery and read access can land on different backends and you're back to two writers. Point LMTP at the director-userdb socket so mail lands on the same backend the user's IMAP session uses.
Configuring the director ring
Drop this into /etc/dovecot/conf.d/10-director.conf and ship the identical file to every frontend. The director list must match byte-for-byte across nodes, or the ring splits.
service director {
unix_listener login/director {
mode = 0666
}
fifo_listener login/proxy-notify {
mode = 0666
}
unix_listener director-userdb {
mode = 0600
}
inet_listener {
port = 9090
}
}
# make the login processes director-aware
service imap-login {
executable = imap-login director
}
service pop3-login {
executable = pop3-login director
}
# deliver via director so mail lands on the assigned backend
protocol lmtp {
auth_socket_path = director-userdb
}
director_servers = 10.0.0.10 10.0.0.11 10.0.0.12
director_mail_servers = 10.0.0.20 10.0.0.21 10.0.0.22
director_user_expire = 15 min
director_username_hash = %u
Bring the backends into the ring with weighted vhost counts and verify state:
bash
# add backends (weight defaults to 100; give a beefier box more)
doveadm director add 10.0.0.20
doveadm director add 10.0.0.21 150
doveadm director add 10.0.0.22
# who is in the ring, and are all peers in sync?
doveadm director status
doveadm director ring status # every node should read "online", not stuck handshaking
# where does a specific user resolve right now?
doveadm director status [email protected]
# full live user -> backend map
doveadm director map
If doveadm director ring status shows any node stuck handshaking instead of online, the ring is split — usually a firewall dropping 9090 between two nodes or mismatched director_servers lists. Fix that before you route a single user, because a split ring means two subsets of frontends disagree on the map, which is exactly the corruption you're trying to avoid.
Making NFS actually safe for mail
Director guarantees one writer, but the backend still has to behave correctly on a network filesystem. On every backend:
mail_fsync = always
mmap_disable = yes
lock_method = fcntl
# and do NOT set mail_nfs_index / mail_nfs_storage — director makes them pointless
mail_location = mdbox:~/mdbox
Mount the filer with NFSv4.2, hard (never soft — a soft timeout mid-write is data loss), and a modest actimeo. Director already guarantees a single writer per mailbox, so attribute-cache staleness across backends can't hurt you, and the cache buys you real performance:
Use mdbox, not Maildir, on shared storage. Maildir stores one file per message and encodes flags in the filename, so every flag change (\Seen, \Flagged) is a rename(). A busy mailbox becomes a rename storm across NFS, and a 200k-message account becomes 200k inodes plus index files — millions of inodes on the filer, brutal for readdir, backup, and fsck. mdbox packs many messages into large m.N files, stores flags in the index, and does single-instance attachment storage, so you trade the inode explosion and rename storm for one operational chore: mdbox never reclaims space until you purge it. Run purge on cron:
bash
doveadm purge -A # nightly, off-peak
Object storage: what actually works in 2026
Be honest with yourself about the S3 dream. Dovecot CE has no native object-storage backend. None. If you see a design that FUSE-mounts an S3 bucket under a Maildir path, kill it: object stores have no atomic rename(), latency is tens of milliseconds per op where mail expects sub-millisecond, and lock semantics simply don't exist. That design corrupts mailboxes; it's not a tuning problem.
Real object-storage mailboxes on Dovecot mean Dovecot Pro's obox plug-in: fs = obox writes dbox objects into S3-compatible storage while a local-SSD metacache holds the hot indexes so IMAP stays fast. That's a commercial product, and if you're at the scale where it pays off you already know it.
For everyone on CE, the credible open-source scale-out path is CephFS — it gives you POSIX semantics (real rename, real locking) over a distributed object store, so Dovecot treats it like a very large, self-healing filer. The decision is simple: NFS or CephFS filer with mdbox on CE; obox only when you're on Pro and a single filer can no longer keep up.
Operating it: moves, flushes, and the health-check gap
The gotcha that bites every first deployment: CE director does not health-check its backends. A backend that dies stays in the ring, and director keeps hashing its share of users straight into the void — connection refused, over and over — until something removes it. There is no built-in probe. You wire it yourself: your monitoring (a systemd healthcheck, Consul, a Nagios/Prometheus alert hook) runs doveadm director remove on failure and doveadm director add on recovery.
That diagram is the whole reason director uses consistent hashing instead of hash(user) % N. With modulo, changing the backend count reshuffles almost everyone at once — a thundering herd of index reopens across the cluster. With a hash ring, removing one backend reassigns only that backend's slice; every other user stays pinned exactly where they were.
Day-to-day operations:
bash
# drain a backend for a rolling upgrade: move its users elsewhere first
doveadm director move [email protected] 10.0.0.22
# pull a dead backend (your monitor calls this automatically)
doveadm director remove 10.0.0.20
# force a full re-evaluation of the map (rarely needed)
doveadm director flush all
Version reality, and this one matters: director is a Dovecot 2.2/2.3-line feature. The local reference box here runs 2.3.16. Dovecot 2.4 removed the standalone director service. If your HA design depends on it, pin your packages to a 2.3.x line and hold there deliberately — do not let an unattended upgrade drag you to 2.4 and vanish the service out from under a live cluster. If you're planning greenfield on 2.4+, you're planning around the Pro architecture instead, and that's a different build.
Pre-flight checklist
[ ] All frontends ship an identical director_servers and director_mail_servers list (byte-for-byte).
[ ] LMTP delivery routed through the director-userdb socket, not straight to a backend.
[ ] mail_fsync = always and mmap_disable = yes set on every backend.
[ ] mail_nfs_index / mail_nfs_storage are not set anywhere.
[ ] mdbox chosen for mail_location; doveadm purge -A on nightly cron.
[ ] keepalived VIP + haproxy in TCP mode with health checks on 993/995.
[ ] External monitoring calls doveadm director remove/add on backend down/up (CE won't).
[ ] doveadm director ring status shows every node online (none stuck handshaking).
[ ] Dovecot packages pinned to 2.3.x (2.4 removed director).
[ ] Backup and restore tested against the shared filer as a whole — never per-backend, because no single backend owns the full picture.
