Self-Hosted Postfix + Dovecot: A Production Setup That Passes Its First Smoke Test
A copy-pasteable Postfix and Dovecot build on Debian 12, focused on the parts that actually break: the SASL handshake, TLS on 587 and 993, and the virtual-domain Maildir permission model — with the smoke tests that prove it works.
EvilMail TeamJuly 8, 20269 min read
The problem, stated plainly
Getting mail to move is easy. apt install postfix, accept the defaults, and packets will flow. The hard part — the part that fails the first time a real client connects or a real message arrives from the internet — is authentication that doesn't loop, a Maildir the delivery agent can actually write to, and TLS that encrypts instead of quietly falling back to plaintext.
Across thousands of domains, the same three failures recur on every fresh build: a submission client stuck in an infinite auth-retry loop because Postfix can't reach the Dovecot auth socket, inbound mail that bounces with permission denied because the Maildir tree is owned by root instead of the mail user, and a "secure" port that negotiates down to cleartext because nobody enforced encryption.
This guide is the build we actually run. Target: Debian 12 (bookworm), Postfix 3.7 as MTA and message submission agent, Dovecot 2.3 as the local delivery agent (over LMTP) and IMAP server, virtual users that are not Unix accounts, backed by flat map files, with mail stored in Maildir++
under
/var/mail/vhosts
.
Assumed before you start: a domain you control, a PTR record (reverse DNS) on your server IP that matches your mail hostname, and ports 25, 465, 587, and 993 open both ways at the firewall and the provider. Without the PTR, skip to the end — no config fixes deliverability to Gmail.
The architecture in one picture
Two decisions here matter. First, inbound mail hands off to Dovecot over LMTP, not the legacy dovecot-lda pipe. LMTP is a persistent, network-style protocol spoken over a Unix socket: you get proper per-recipient status codes, no fork-per-message cost, and a clean integration point for Sieve and quota. The old mailbox_command pipe still works, but it's slower and its error reporting is coarse.
Second, there is exactly one auth backend. Postfix's submission SASL and Dovecot's IMAP login both resolve credentials through the same Dovecot auth process, over the private/auth socket. One place to define users, one place a password bug can hide. Postfix has no user database of its own in this model — it asks Dovecot.
Users, domains, and the Maildir layout
Virtual users are not Unix accounts. Every mailbox is owned on disk by a single system user, vmail, uid/gid 5000, whose home is the mail root:
Postfix resolves three flat maps, all hash: files compiled with postmap:
text
# /etc/postfix/vmail_domains — which domains we accept
example.com OK
example.net OK
# /etc/postfix/vmail_mailbox — address -> maildir path (trailing slash = Maildir)
[email protected] example.com/alice/
[email protected] example.com/bob/
# /etc/postfix/vmail_alias — forwarders
[email protected][email protected]
The on-disk tree runs one directory per domain, then per user, then the Maildir triplet:
Maildir beats mbox because each message is its own file. Two deliveries never contend for a lock on one giant file — which matters the moment you have concurrent inbound mail plus an IMAP client rewriting flags. The single most common field failure on a fresh box: the tree gets created by root during testing, LMTP runs as vmail, and delivery bounces with permission denied. One chown -R vmail:vmail /var/mail/vhosts after any manual poking prevents it.
Postfix: main.cf and master.cf that pass a smoke test
The relevant main.cf. Note what is *not* here — no local Unix delivery for these domains, everything virtual:
reject_unauth_destination is the line that keeps you off blocklists. Drop it and you are an open relay within hours of spammers finding the box. On port 25, security_level = may is correct: opportunistic TLS, because a sending MTA that only speaks cleartext should still be able to deliver — refusing it just loses you legitimate mail. Note that SASL is off globally (smtpd_sasl_auth_enable = no); we switch it on per-port below, so port 25 never advertises AUTH.
Submission is different. In master.cf, enable 587 with mandatory STARTTLS and turn SASL on *only for these ports*:
security_level=encrypt on 587 means the client *must* run STARTTLS before it can authenticate — no plaintext password on the wire, ever. wrappermode=yes on 465 means the TLS handshake happens immediately on connect (implicit TLS), with no STARTTLS dance. The client_restrictions=permit_sasl_authenticated,reject on both closes the relay: authenticated senders get through, everyone else is rejected before they reach recipient checks.
Dovecot: auth, LMTP, and the shared socket
Three files do the work. 10-mail.conf sets where mail lives, using %d (domain) and %n (local part):
Each /etc/dovecot/users line follows the format user@domain:{SHA512-CRYPT}$6$...:5000:5000::/var/mail/vhosts/domain/user. Generate the hash with doveadm pw -s SHA512-CRYPT. Dovecot 2.3 also ships ARGON2ID — prefer it for new deployments. The PLAIN scheme belongs only on a closed test rig you will tear down; never ship it.
The piece that ties everything together is 10-master.conf, where Dovecot exposes two sockets *inside Postfix's chroot* under /var/spool/postfix/private/:
ini
service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0660
user = postfix
group = postfix
}
}
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
mode = 0600
user = postfix
}
}
Get the owner or mode wrong here and you meet the classic error: the client tries to authenticate, Postfix logs SASL authentication failure: no mechanism available, and the login loops forever. It isn't a credential problem — Postfix literally cannot open the socket to *ask*. mode = 0660 with user = postfix is what lets the Postfix smtpd process read it.
TLS that doesn't lie
One certificate, both daemons. Issue it with certbot in standalone mode (stop anything on port 80 first), or webroot if you already run a web server:
bash
certbot certonly --standalone -d mail.example.com
Point Postfix at fullchain.pem/privkey.pem (shown above) and Dovecot at the exact same files in 10-ssl.conf:
ssl_min_protocol = TLSv1.2 and the !aNULL:!eNULL exclusion are the difference between "encrypted" and "negotiated down to something a passive observer reads." The only distinction between 587 and 465 is *when* TLS starts — STARTTLS upgrades a plaintext connection on 587, implicit TLS wraps from byte zero on 465 — but because we set security_level=encrypt on 587, both are equally mandatory. There is no plaintext submission path in this config. Add a certbot deploy hook to run systemctl reload postfix dovecot on renewal, or a valid cert silently expires in 90 days.
MTA-STS and DANE (TLSA records) are the next hop for enforcing inbound TLS from other senders — worth doing, out of scope here.
Smoke tests: prove it before you trust it
Do not trust a mail server because it started. Run these in order; each isolates one layer.
1.Maps compiled. After any edit: postmap /etc/postfix/vmail_mailbox (and the other two), then postfix check. A stale .db is invisible until mail bounces.
2.Config sane.postfix reload && doveconf -n — the second prints Dovecot's effective config with no errors if the syntax is clean.
3.TLS and offered auth on 587.openssl s_client -starttls smtp -connect mail.example.com:587 -showcerts. Confirm the chain verifies and that the 250-AUTH line lists PLAIN LOGIN — and that it appears *only after* STARTTLS, never before.
4.Credentials in isolation.doveadm auth test [email protected] — prompts for the password and answers passdb: auth succeeded without touching Postfix. If this fails, the problem is Dovecot, not SMTP.
5.Authenticated submission end to end.swaks --to [email protected] --from [email protected] --server mail.example.com:587 --auth LOGIN --tls. Watch for 235 Authentication successful, then 250 Ok: queued.
6.Inbound landed with the correct owner.ls -l /var/mail/vhosts/example.com/bob/new/ — a file must appear, owned 5000:5000. Root ownership here means the chown was skipped.
7.Follow the delivery line.journalctl -u postfix -u dovecot -f — you want lmtp and status=sent (250 ... Saved), not deferred or bounced.
8.External deliverability sanity. Send to an authentication-results reflector and confirm SPF passes and rDNS/HELO align. This catches problems no local test can.
What breaks at scale
Across a large fleet, the recurring failures are boring and preventable:
Editing a map and forgetting `postmap`. The plaintext file is right, but the .db Postfix reads is stale. End every map edit with a postmap.
`%d/%n` not matching the tree on disk.mail_location says %d/%n but the directory was created as %n@%d, or vice versa. Deliveries land in a path IMAP never reads.
Missing `reject_unauth_destination`. One omitted line turns you into an open relay and a blocklist entry within a day.
PTR/HELO mismatch. The server is flawless and Gmail still 5xx-rejects you because reverse DNS doesn't match myhostname. No main.cf change fixes DNS — set the PTR at your provider.
The rule we hold teams to: if openssl s_client, swaks, and doveadm auth test all pass and the file shows up in new/ owned 5000:5000, you have a mail server. If any one fails, you don't have a mail server — you have a to-do item.
