Building a Webmail Backend on Dovecot IMAP: Connection Pooling and Session Caching
IMAP was never designed to sit behind a stateless HTTP API. A fast webmail backend is 90% connection lifecycle management and 10% rendering messages. Here is how to pool connections per user and sync mailboxes incrementally with QRESYNC against Dovecot.
EvilMail TeamJuly 30, 202612 min read
Why a naive IMAP-per-request backend melts down
Wire an HTTP handler straight to an IMAP library, open a connection, run the query, close it, and your webmail will work beautifully in the demo and fall over the moment real traffic arrives. The reason is that IMAP is stateful and expensive to establish, and an HTTP request is neither.
Trace the cost of one cold request. The TCP three-way handshake plus the TLS handshake is roughly two round trips before a single byte of IMAP is spoken. Then Dovecot attaches an imap backend process to your login, and LOGIN runs your password through the passdb — a real bcrypt or argon2 verification if you configured it that way, which is deliberately slow. Then CAPABILITY, then SELECT INBOX, which re-reads the mailbox index. On a LAN you are looking at 40ms of pure setup; across the internet with a busy box, 80–120ms is normal — and you have not yet fetched one header.
Now the failure mode that actually pages you. Dovecot's mail_max_userip_connections defaults to 10
. A webmail that opens a fresh connection per browser tab, per poll, per API call blows through ten sockets from a single user behind one NAT in seconds, and Dovecot starts rejecting logins with:
[LIMIT-EXCEEDED] Maximum number of connections from user+IP exceeded (mail_max_userip_connections=10)
The user sees a spinner. Support sees a ticket. You see imap-login errors and assume the box is undersized when it is fine — you are just using it wrong.
The design constraint that drives everything below: reuse connections, and never re-derive state you already have. Treat the IMAP connection as a scarce, long-lived, stateful resource you pin per user. Treat the mailbox as an append-mostly log you sync incrementally. Get those two right and one Node process comfortably serves thousands of sessions against a single Dovecot instance.
The connection pool: pin one long-lived connection per user
The pool is per user, not global. A shared anonymous pool would need to re-authenticate as a different user on every checkout, which throws away the entire point. Each authenticated session owns one — at most two — IMAP connections kept warm, keyed by user id in a Redis-backed registry.
Authenticating without hoarding plaintext. You cannot store the user's password on every request, and you should not want to. Use Dovecot master users: the backend logs in as user@domain*masteruser with a single master credential, and Dovecot authorizes it as that user. Your session store then only needs a short opaque token that maps to a user id — no mailbox password anywhere in your app tier. This is the safer path for a webmail backend even on deployments where the mailbox passdb scheme is plain, because the blast radius of a leaked session token is one inbox and a logout, not a permanent credential.
Idle management. Servers commonly drop connections after 30 minutes of silence. On a pooled-but-not-IDLE connection, send NOOP every 4–5 minutes to stay warm. Just as important, hard-cap idle sockets with a TTL sweep: an abandoned session must release its slot, or you slowly march every user toward the per-IP cap and starve your own login process.
Sticky routing. A socket lives inside one Node worker. If your load balancer sends the next request for the same user to a different worker, that worker does not hold the socket. Either pin sessions to a worker (session affinity keyed by user id) or run a dedicated IMAP-gateway process that owns all sockets and speaks to your stateless API workers over a local RPC. The registry in Redis records which worker or gateway holds each user's connection.
javascript
import { ImapFlow } from 'imapflow';
const pool = new Map(); // userId -> { client, lastUsed, keepAlive }
const MASTER = process.env.DOVECOT_MASTER_PASS;
async function acquire(userId, email) {
let entry = pool.get(userId);
if (entry?.client?.usable) {
entry.lastUsed = Date.now();
return entry.client;
}
const client = new ImapFlow({
host: '127.0.0.1', port: 993, secure: true,
auth: { user: `${email}*webmail`, pass: MASTER }, // master-user login
qresync: true, // enable CONDSTORE/QRESYNC
logger: false,
});
await client.connect();
const keepAlive = setInterval(() => {
client.noop().catch(() => {}); // stay under the 30-min idle drop
}, 4 * 60 * 1000);
entry = { client, lastUsed: Date.now(), keepAlive };
pool.set(userId, entry);
return client;
}
// TTL sweep: release sockets idle > 15 min so slots return to the pool
setInterval(() => {
const cutoff = Date.now() - 15 * 60 * 1000;
for (const [userId, e] of pool) {
if (e.lastUsed < cutoff) {
clearInterval(e.keepAlive);
e.client.logout().catch(() => {});
pool.delete(userId);
}
}
}, 60 * 1000);
SELECT once, then never full-scan again
This is where the caching pays off. The wrong way to refresh an inbox is FETCH 1:* (FLAGS) on every page load — it grows linearly with mailbox size and re-reads data that has not changed. The right way is to sync only the delta since your last look, using CONDSTORE and QRESYNC (RFC 7162), which Dovecot advertises by default. Confirm with a CAPABILITY and look for CONDSTORE and QRESYNC in the response.
The cache-invalidation key is UIDVALIDITY. It is the server's promise that UID→message mappings are stable. Store (UIDVALIDITY, UIDNEXT, HIGHESTMODSEQ) per mailbox in Redis. On the next SELECT, if UIDVALIDITY still matches, issue a QRESYNC select with your stored HIGHESTMODSEQ and Dovecot replies with only what changed: messages arrived since your UIDNEXT, flag changes via CHANGEDSINCE, and VANISHED UIDs for deletions. If UIDVALIDITY changed, the mapping is void — flush that mailbox's cache and do one full resync. That is the whole algorithm, and it turns "refresh inbox" from a full listing into a single round trip.
What to cache, where, and for how long
Split the cache by mutability, because bodies and flags have opposite lifecycles.
Envelope metadata — subject, from, date, RFC822.SIZE, BODYSTRUCTURE, and current flags — keyed by user:mailbox:uid in Redis. On a warm cache, rendering the message list touches Redis only, never IMAP.
Body and HTML parts cached separately under a long or effectively infinite TTL, keyed by UID under a valid UIDVALIDITY. A message body never changes once delivered; only its flags do. Caching it forever (until UIDVALIDITY flips) is correct, not a shortcut.
Flags are the single mutable field. Update them from QRESYNC CHANGEDSINCE deltas and from your own STORE calls, and nothing else needs touching.
Prefetch the visible window in one shot rather than N round trips:
a UID FETCH 4100:4130 (ENVELOPE FLAGS INTERNALDATE RFC822.SIZE BODYSTRUCTURE)
One command populates thirty list rows. And guard against cache stampede: when a popular mailbox's window expires and fifty requests miss simultaneously, only one should hit IMAP while the rest wait on a short lock. With imapflow, remember that msg.envelope and msg.flags can be undefined on partial responses — reach for them with optional chaining or you will crash on a malformed message.
Push and near-real-time: IDLE without exhausting connections
IDLE gives you push, but it holds a connection open per watched mailbox, and every held connection counts against mail_max_userip_connections. Watch five folders per user with IDLE and you are back to the connection-cap wall.
The practical stance: one IDLE connection per user, on INBOX only. Fan its notifications out to the browser over server-sent events, and poll other folders on demand when the user opens them. Re-arm IDLE before the 29-minute RFC 2177 limit, and actively detect dropped sockets — Dovecot can drop an idle connection silently, so a dead-man timer that expects periodic activity is worth the code. For high-churn temp-mail inboxes, where messages arrive and expire by the second, per-user IDLE does not scale; hook delivery at the MTA/LMTP layer or watch with doveadm and push events from there instead of parking an IMAP socket per inbox. That is the model behind evilmail.pro's temp-inbox delivery, where re-scanning simply is not an option.
Tuning Dovecot for a connection-holding client
Server-side, a few concrete changes make the whole design comfortable.
Raise mail_max_userip_connections well above your per-user pool size so legitimate multi-tab users never trip it, and size imap-login so front-end churn does not starve new logins. Load the imap_zlib plugin to advertise COMPRESS=DEFLATE — it materially cuts bytes on large ENVELOPE listings. Leave CONDSTORE and QRESYNC alone; Dovecot advertises them out of the box, and hand-overriding imap_capability only risks hiding capabilities real clients depend on. Configure the master passdb in /etc/dovecot/conf.d/auth-master.conf.ext for backend auth. Then watch live state with doveadm:
bash
doveadm who # active connections per user/IP
doveadm mailbox status -u user@domain '*' \
messages uidvalidity uidnext highestmodseq
Those uidvalidity/highestmodseq values are exactly what your cache stores — being able to diff them against Redis by hand is the fastest way to debug a stale inbox.
Operational checklist
Cap the per-user pool below mail_max_userip_connections; leave headroom for the user's own extra tabs.
TTL-sweep idle sockets so abandoned sessions return their slots.
Store (UIDVALIDITY, UIDNEXT, HIGHESTMODSEQ) per mailbox and validate UIDVALIDITY before trusting any cached UID.
Never store mailbox plaintext — authenticate via master user, hand the browser a short session token.
Cache immutable bodies separately (long TTL) from mutable flags (delta-updated).
One IDLE connection per user, INBOX only; poll the rest on demand.
Add stampede protection so a popular expired mailbox triggers one refill, not fifty.
Alert on [LIMIT-EXCEEDED] and imap-login saturation — they mean pooling is misbehaving, not that the box is too small.
Send a graceful LOGOUT on session end so slots free immediately instead of waiting for the sweep.
Build the connection lifecycle and the incremental sync correctly and the rest of a webmail backend is ordinary web engineering. Skip them and no amount of hardware saves you, because you will be paying the IMAP setup tax on every keystroke.
