Fast Full-Text Search in Dovecot: Flatcurve, Xapian, and Solr FTS for Large Mailboxes
Without an FTS index, an IMAP BODY search reads every message on disk — 90 seconds and a pinned server because one user hit the search box. Here's how to fix it with flatcurve, Xapian, or Solr, and how to pick the right one for your replication topology.
EvilMail TeamJuly 12, 202613 min read
A user types invoice into the webmail search box and the whole server falls over. Run the same query by hand and you can watch it happen:
bash
$ time doveadm search -u [email protected] mailbox INBOX body invoice
...
real 1m32.417s
Ninety seconds. Meanwhile iostat -x 2 shows the mail disk pinned at 100% util, Roundcube throws a 504 from behind nginx, and Thunderbird gives up at its 30-second socket timeout. Nothing is broken. This is Dovecot working exactly as designed: an IMAP SEARCH BODY with no full-text index is O(mailbox size). Dovecot opens and MIME-decodes every message file to satisfy one query. On a 40 GB mailbox with 400k messages that means reading 400k files off disk, and one user doing it can starve every other IMAP session on the box.
The fix is an inverted index. This article covers the two approaches that actually matter in 2026 — flatcurve/Xapian and Solr — with real config, real doveadm
output, and the operational traps that bite you six months later.
Why search is slow without an index
Not all searches are equal. A SEARCH HEADER FROM billing@ or SEARCH SUBJECT can be answered from dovecot.index.cache, the per-mailbox cache file that already holds parsed envelope fields. Those stay fast without any FTS. The moment a client issues BODY or TEXT, though, Dovecot has to look inside the message: open the file, decode the MIME structure, walk each text part, and substring-match. In maildir that's one open()/read() per message; in mdbox it's fewer files but the same total bytes decoded.
Cold cache makes it brutal. The 90-second run above drops to maybe 25 seconds warm, because the page cache is holding the message files — but you cannot keep a 40 GB mailbox resident in RAM, and every new search evicts the last one. The degradation is noticeable past roughly 10k messages or a few GB, and it's linear from there.
One dead end to name explicitly: fts_squat, Dovecot's old built-in indexer, is deprecated and gone from current releases. If a 2015-era blog post tells you to enable it, close the tab. In 2026 there are exactly three backends worth running.
The three backends that matter in 2026
fts_flatcurve — Xapian under the hood. Buildable as a plugin on Dovecot 2.3.x and the recommended FTS backend in 2.4. No external daemon. The index lives inside the mail store, next to the mailbox's control files.
fts_xapian — the community plugin (grosjo/fts-xapian). Same engine, but you build it yourself and it keeps its index in a separate directory. Useful if flatcurve isn't packaged for your distro, or for its explicit n-gram partial= tuning.
fts_solr — Apache Solr 9.x, a separate JVM service Dovecot talks to over HTTP. A shared, centralized index that many Dovecot backends can point at, with a real Tika pipeline for attachment extraction.
The decision comes down to one question: what is your replication topology?
For a single server or a dsync-replicated pair — which covers most self-hosted and small-platform Dovecot — reach for flatcurve first. Because its index sits inside the mail store, dsync replication and any mailbox-level backup carry the index along with the mail for free. Fail over to the replica and search just works.
Solr is the opposite. Its index is external and dsync knows nothing about it. When you fail over or restore a mailbox from backup, the Solr index for that user is empty until you rebuild it. That's the single most consequential operational fact in this whole article, and the trap most people discover during their first real failover. Choose Solr only when you genuinely need a shared/clustered index across many backends, or heavy Tika attachment search at scale — and then budget for reindexing as part of your failover runbook.
Setting up flatcurve (the default recommendation)
Enable the plugins and drop in a plugin block. This is Dovecot 2.3 syntax; the settings map cleanly onto 2.4's namespaced form.
fts_autoindex = yes indexes messages as they're delivered, so new mail is searchable within seconds. Without it, a mailbox only gets indexed on its first search — which dumps the expensive first-index cost onto an interactive user waiting on the search box.
commit_limit = 500 flushes to the Xapian DB every 500 docs. Lower is safer on crash, higher is faster on bulk index.
rotate_count / rotate_time control shard rotation. flatcurve writes multiple Xapian sub-databases and merges them; rotation keeps any single shard from growing unbounded during heavy indexing.
substring_search = no is the right default. Turning it on lets SEARCH BODY "voice" match "invoice", but it n-grams every term and can multiply index size 2–4x. Most users search whole words.
The index lands inside the mailbox's control directory — for mdbox, under the dbox-Mails control path; for maildir, alongside dovecot.index. That's the property that makes it replicate. Budget roughly 10–20% of mail size for the index on disk.
Bulk-index existing mail through the background indexer so you don't melt the server doing it inline:
-q queues the work to the indexer process instead of running it synchronously in the doveadm process, and '*' means all mailboxes. To cap the load, nice/ionice the doveadm call or limit the indexer worker count. Expect a few thousand messages per minute on spinning disks, much more on NVMe. Loop it across all users overnight for the initial rollout; autoindex handles everything after.
Setting up Solr FTS
If you've decided you need the shared index, install Solr 9.x and create a dovecot core, then load the fields from Dovecot's shipped solr-schema.xml and solr-config.xml (Solr 9 uses a managed schema, so you import the fields via the schema API rather than dropping the XML in blind). Then:
The 127.0.0.1 in that URL is not optional. Never expose 8983. Solr ships with no meaningful auth, and a reachable Solr admin is remote code execution waiting to happen. Bind it to localhost, and if backends are remote, tunnel over TLS or put it behind an authenticated reverse proxy on a private network.
Size the JVM heap deliberately — 2–4 GB is plenty for a few hundred thousand messages; oversizing just buys you longer GC pauses. And tune commits: Solr's autoSoftCommit makes new mail searchable, autoCommit (hard) flushes to disk. A too-aggressive hard-commit interval (say, every second) tanks indexing throughput because Solr constantly fsyncs segments. A soft commit around 10–30s and a hard commit around 1–2 minutes is a sane starting point. Reindex the same way as flatcurve, with doveadm index.
The real cost of Solr isn't the config, it's that you now run a second stateful service: monitored, heap-tuned, backed up, secured, and — per the diagram above — reindexed on every failover.
Attachments, stemming, and CJK
By default FTS indexes text parts, not the contents of a PDF or DOCX. To search inside attachments with flatcurve/Xapian, wire up the decoder script:
decode2text.sh shells out to catdoc, pdftotext, xls2csv and friends — install those packages or the script silently indexes nothing. Solr's richer path is Apache Tika via the ExtractingRequestHandler, which handles far more formats but is another moving part to run and secure.
Stemming is where recall quietly dies. A stemmer folds "invoices", "invoiced", and "invoicing" down to one root so any of them match. Solr uses Snowball plus language-specific analyzers; Xapian has built-in stemmers. But a stemmer is language-specific, and the wrong language stems wrong — point a German stemmer at Turkish mail and searches miss matches with zero errors in the log. On a mixed-locale platform running 12 locales, don't hard-code one stemmer and forget it; where per-mailbox language is unknown, a conservative default (English with no aggressive stemming) beats confidently wrong stemming.
CJK is the other special case. Chinese, Japanese, and Korean don't put spaces between words, so whole-word tokenization indexes nothing useful. You need n-grams: Xapian's partial= (community plugin), flatcurve's substring_search, or Solr's ICUTokenizer. The community Xapian plugin makes this explicit:
fts = xapian
fts_xapian = partial=3 full=20
partial=3 indexes 3-character grams for substring/CJK matching; higher values catch more but bloat the index. It's the classic recall-vs-disk tradeoff — turn it on where you actually have CJK users, not everywhere.
Operating the index in production
Your day-to-day toolbox:
bash
# Rebuild missing/stale index entries for one user
doveadm fts rescan -u [email protected]
# Compact and merge Xapian shards (cron this nightly)
doveadm fts optimize -u [email protected]
# Bulk (re)index one user via the background indexer
doveadm index -q -u [email protected] '*'
# Verify a known message is searchable, and time it
time doveadm search -u [email protected] mailbox INBOX body invoice
doveadm fts rescan is your recovery command: it marks the index stale and rebuilds what's missing without a full wipe. Run it after any restore or migration — even for flatcurve, a partial or interrupted sync can leave the index out of step with the mail. For Solr, rescan plus a full doveadm index is mandatory after failover because, again, the index came over empty.
Watch three signals: index size on disk (a Xapian index ballooning past ~25% of mail size usually means it needs an optimize), indexer-worker backlog (if indexer-worker processes are perpetually maxed, delivery is outpacing indexing — hunt for a stuck huge mailbox), and log lines mentioning Corrupted FTS or failed to index. Schedule doveadm fts optimize nightly for active users; it merges shards and reclaims space, and a fragmented Xapian DB searches slower.
Reindex a single user when their search is wrong or slow; reindex everyone only after a version upgrade that changes the index format or a schema change. And write down the restore runbook now, while it's calm, not during the outage.
Checklist before you flip it on
Pick the backend by replication topology — single server or dsync pair → flatcurve; multi-backend shared index or heavy Tika → Solr.
Size disk for the index — 10–20% of mail size for Xapian, 2–4x more if you enable substring/n-gram.
Enable `fts_autoindex = yes` so new mail is searchable and the first-index cost doesn't land on a live user.
Set the right stemmer language for your user base — the wrong language silently kills recall.
Bind Solr to 127.0.0.1 and never expose 8983; verify with ss -tlnp | grep 8983.
Verify with a known-message search plus `time` — confirm sub-second results before announcing it.
Schedule `doveadm fts optimize` nightly and add index size to monitoring.
Document the reindex-after-restore runbook — for Solr it's mandatory, for flatcurve it's cheap insurance.
If you run a single Dovecot box or a dsync pair, install flatcurve today and let the index replicate itself. Reach for Solr only when a shared, clustered index is a real requirement — and only after you've written down how you'll rebuild it when a backend fails.
