Why "just run imapsync" breaks at 500 mailboxes
The classic first attempt looks like this, and it dies around account 40:
# The loop that will page you at 3am
for user in $(cat users.txt); do
imapsync --host1 imap.gmail.com --user1 "$user" --password1 "$pw" \
--host2 mail.evilmail.pro --user2 "$user" --password2 "$pw2"
doneBy the fortieth mailbox, Gmail is answering [ALERT] Too many simultaneous connections on the source side, your destination Dovecot has thrown Quota exceeded (mailbox for user is full) on three accounts, and every Gmail user's mail history is duplicated because [Gmail]/All Mail copied every message a second time. None of this is random. imapsync at scale fails in exactly three structural ways, and all three are preventable:
- Folder mapping — source and destination disagree about folder names and hierarchy separators, and Gmail's labels-as-folders model double-copies everything.
- Quota — the destination fills mid-sync, and on-disk size on the target is bigger than the source reported.
- Rate limits — the source throttles you by connection count, bytes per day, or sustained IMAP load.
The lesson: the copy itself is trivial. The work is the mapping, the throttling, and the reconciliation around it. So you never run one monolithic unsupervised loop. You run two passes — a discovery pass to learn the source's quirks, then an incremental pass at cutover that moves only the delta. That second pass is the migration your users actually feel. This is the procedure we use to onboard bulk domains onto Dovecot at evilmail.pro without tripping Gmail, Exchange, or cPanel source throttles.
The reference invocation and what every flag is doing
Here is the canonical command — the one you tune once per source type, then fan out. Every flag earns its place at scale:
imapsync \
--host1 imap.gmail.com --ssl1 --user1 [email protected] --password1 "$SRC_PW" \
--host2 mail.evilmail.pro --ssl2 --user2 [email protected] --password2 "$DST_PW" \
--automap \
--useuid \
--syncinternaldates \
--addheader \
--maxsize 40_000_000 \
--exclude '\[Gmail\]/All Mail' \
--logfile /var/log/imapsync/alice.log \
--tmpdir /var/tmp/imapsyncWhat matters and why:
- `--automap` builds a best-guess mapping of the special folders (Sent, Drafts, Trash, Junk) across naming schemes. You still verify it — trust nothing here.
- `--useuid` keys message identity on the IMAP UID rather than a header hash. Together with imapsync's per-user state cache, this is what makes pass two cheap: it re-transfers only messages it hasn't already confirmed on the destination.
- `--syncinternaldates` preserves the server-side INTERNALDATE so migrated mail sorts by its real received time, not migration time. Skip it and every inbox looks like it all arrived today.
- `--addheader` stamps a tracking header on each copied message — invaluable when you need to prove which run touched what.
- `--maxsize` plus a
--tmpdiron fast local disk stops one monster attachment from stalling a run and keeps temp churn off network storage.
Before any of this moves a byte, run the discovery pass:
# Validate the folder tree, move nothing
imapsync ... --dry --justfolders
# Pre-flight sizes so you can set quota headroom
imapsync ... --justfoldersizes--dry --justfolders shows exactly which folders imapsync intends to create on the destination and how it mapped them, without copying a single message. It is the cheapest insurance in the whole process.
Folder mapping: --automap, --regextrans2, --sep1, --subfolder2
This is the part that eats a day if you skip the dry run. --automap handles the common special folders, but it does not know that your legacy Courier server used a dot separator with an INBOX. prefix, and it will happily copy Gmail's [Gmail]/All Mail — which holds a copy of every message carrying any label — leaving a destination roughly twice the size of the source, duplicates everywhere.
The fixes, flag by flag:
- Gmail All Mail / Important dedup. Exclude the label-as-folder pseudo-folders so each message copies exactly once from its real folder:
--exclude '\[Gmail\]/All Mail' --exclude '\[Gmail\]/Important'. - Renaming folders. Rewrite the transfer path with a regex:
--regextrans2 's,INBOX\.Sent,Sent,'turnsINBOX.Sentinto a cleanSent. Chain several--regextrans2rules as needed. - Separator and prefix mismatches. Old Dovecot and Courier use
.as the hierarchy separator with everything underINBOX.; Gmail and modern Dovecot use/. Describe the source layout with--sep1 . --prefix1 "INBOX."and imapsync re-roots the tree correctly on a/-separated destination. - Staged cutover. To land everything under a single parent so users can review before you promote it, add
Quota: the sync fails silently when the destination fills
The failure signature is Error: Store failed ... Quota exceeded (mailbox for user is full), buried mid-log, after which imapsync keeps trying every remaining message and failing each one. If you are not reading per-user logs, you find out when the user complains that half their mail is missing.
Two numbers drive this. First, on-disk size on the destination is larger than the source reported — IMAP re-encoding, per-message index overhead, and Dovecot's own metadata inflate it. The working rule: set destination quota to source size × 1.3 before you start. Pre-flight the source with --justfoldersizes, then provision the target quota with that headroom.
Check and reconcile Dovecot quota directly:
# Before: confirm the destination quota and current usage
doveadm quota get -u [email protected]
# After the migration: force a recount so reported usage is accurate
doveadm quota recalc -u [email protected]Bound what moves so a few giant mailboxes do not blow the budget:
- `--maxsize 40_000_000` skips individual messages over ~40 MB — usually a handful of huge attachments — which you chase in a separate later pass.
- `--maxage 365` (paired with
--minage) migrates the last year first, then backfills older mail once the account is live. - When the source itself is over quota, set
--maxsizeaggressively to move the bulk, cut the user over, then run a tail pass with the limit lifted.
Rate limits and throttling: the flags that keep you unbanned
Source-side throttling is the real ceiling on migration speed, and it is provider-specific. Gmail enforces roughly a 2500 MB/day/user IMAP download cap plus per-IP simultaneous-connection limits; blow past them and you get [ALERT] Too many simultaneous connections or a temporary lockout. Microsoft 365 throttles sustained IMAP load and starts answering Server Unavailable once you lean on it. cPanel/Dovecot sources are gentler but still refuse connections past their per-user limit.
Two layers of control:
- Per-process caps.
--maxbytespersecond 1_500_000and--maxmessagespersecond 10keep one imapsync process politely under the source's radar.--noabletosearchskips server-side SEARCH when the source's implementation is broken (some Exchange front-ends). - Parallelism at the orchestration layer. This is the one people get wrong. You cap the number of *concurrent imapsync processes*, not the throughput of one. Four to eight parallel processes against Gmail is sane; hundreds is a self-inflicted denial of service.
# CSV: host1;user1;password1;host2;user2;password2
cat accounts.csv | parallel -j4 --colsep ';' \
imapsync --host1 {1} --ssl1 --user1 {2} --password1 {3} \
--host2 {4} --ssl2 --user2 {5} --password2 {6} \
--automap --useuid --syncinternaldates \
--maxbytespersecond 1_500_000 \
--logfile /var/log/imapsync/{2}.logThe 2026 reality: basic auth is gone. Microsoft 365 removed basic auth for IMAP, and Gmail requires app-passwords or OAuth. For OAuth sources you pass a token file instead of a password: --oauthaccesstoken1 /path/to/token. Otherwise issue per-user app-passwords ahead of time — that provisioning step, not the copy, is often the migration's critical path.
Orchestrating hundreds of mailboxes
The pattern that scales is a CSV fed to parallel (above), one log file per account, and a retry/quarantine list built from exit codes. imapsync returns non-zero on failure, and --errorsmax 50 makes a single account give up after 50 errors instead of hammering a full mailbox against a wall:
# Retry/quarantine driver
while read -r line; do
user=$(echo "$line" | cut -d';' -f2)
imapsync ... --errorsmax 50 --logfile /var/log/imapsync/$user.log \
&& echo "$line" >> done.csv \
|| echo "$line" >> retry.csv
done < accounts.csvAfter the first full pass, retry.csv is your short list. Re-run only that — thanks to --useuid and the persistent state cache, accounts that partially succeeded resume as near-zero-delta.
The cutover is the two-pass timeline from the first diagram. Run the full pass days ahead. Lower your DNS TTL 24–48 hours before the flip so the MX change propagates fast. Flip the MX. Then run the final incremental pass — the same command, now moving only mail delivered during the propagation window. That last pass, not the big one, is the migration your users feel.
Pre-cutover checklist
- Verified
--dry --justfolderstree on at least three sample accounts per source type. - Destination quota set to source size × 1.3;
doveadm quota getconfirms headroom. - App-passwords or OAuth tokens issued for every account; no basic-auth assumptions.
- Parallelism capped (
-j4) and load-tested against the real source without triggering throttle alerts. - Excludes confirmed for
[Gmail]/All Mail,[Gmail]/Important, and oversize attachments. - Per-user logs under
/var/log/imapsync/and a working retry/quarantine loop keyed on exit code. - Incremental re-run tested end to end — the second pass reports near-zero moved messages.
- DNS TTL lowered ahead of the MX flip;
doveadm quota recalcscheduled post-migration.
Get these eight right and a 500-mailbox migration is boring — which is exactly what a migration should be.