Never Test a Mail Change on Production — Replay It
Editing a live Postfix box and running postfix reload is gambling with mail you can't get back. Here's how to build a staging relay that clones your production mail path byte-for-byte, feed it a corpus of real captured messages, and diff the outcomes before a single production byte moves.
EvilMail TeamJuly 29, 202611 min read
Someone tightens smtpd_recipient_restrictions to clamp down on a spam wave, or nudges a SpamAssassin required_score from 5.0 to 4.5 to catch a few more phish. They run postfix reload, watch the log for thirty seconds, see nothing on fire, and go to lunch. An hour later a customer calls: their supplier's invoices are bouncing with a 5xx. Those messages are gone — a 5xx is permanent, the sending server won't retry, and there is no undo. The quieter failure mode is worse: legitimate mail scores 4.7, crosses the new threshold, and gets silently binned into a quarantine nobody reads for a week.
This is the normal way mail changes ship, and it is indefensible. You would never merge an application change that had never run against a single realistic input. Yet mail infrastructure — where the "inputs" are irreplaceable and the failure mode is invisible — routinely gets shipped by editing the live server and praying. The professional alternative is a staging relay that is a byte-for-byte clone of the production mail path, fed by a corpus of real messages you captured beforehand, and replayed so you can diff the outcome before touching prod. If you can't answer "what would this change have done to last Tuesday's mail?", you are not testing. You are deploying.
Why "reload and watch the logs" is not testing
Staging Relay + Message Replay: Test Mail Config Changes Safely — EvilMail Blog
Production mail traffic is non-reproducible and adversarial in its timing. You cannot re-run yesterday's flow, you cannot A/B a live MX, and the interesting messages — the borderline ham, the malformed-but-legitimate invoice from a badly configured supplier — arrive exactly when you're not looking. Log-watching only tells you about what already broke, after the message is already lost. By the time
postfix/smtpd
logs a
550
, the sender has been told to give up.
A real test needs exactly two things that live-editing can never give you:
An identical mail path — same Postfix version, same restriction stack, same milters, same maps, same TLS policy. If staging isn't a clone, a "pass" there proves nothing about prod.
A fixed corpus of real inputs — the actual messages, with their actual envelopes, that flow through your MX. Synthetic test mail won't have the weird Received chains, the broken DKIM, the base64 attachments that trip content rules.
Get those two right and a mail change stops being a leap of faith. It becomes a diff.
Anatomy of a staging relay that matches prod
The mistake people make is standing up "another Postfix" and calling it staging. What has to match is the entire mail path, not just the software. Enumerate it and diff every layer:
Same Postfix version (postconf mail_version).
Same main.cf and master.cf — pull both and diff them, don't eyeball.
Same milters: OpenDKIM, OpenDMARC, and rspamd or SpamAssassin, at the same versions with the same rules and score sets.
Same lookup maps (access, transport, canonical, bcc).
Same smtpd_tls_security_level and cert chain — TLS state changes some reject decisions.
Config drift is the single biggest reason staging "passes" and prod fails. Automate the comparison and fail the run if it's dirty:
The two things that must differ are the ingress and the egress, and only those. Bind staging on 2525 instead of 25 so it never collides with the real listener, and swap the delivery transport for a sink so nothing ever leaves the box:
ini
# staging main.cf overrides — everything else is copied from prod
default_transport = discard
relay_transport = discard
soft_bounce = yes # every 5xx becomes 4xx: nothing is permanently rejected
smtpd_authorized_xclient_hosts = 127.0.0.1
ini
# staging master.cf — listen on 2525, not 25
2525 inet n - y - - smtpd
soft_bounce = yes is the load-bearing safety net. It rewrites every 5xx your restriction stack would emit into a 4xx for the duration of the run, so even a wildly misconfigured staging box can never permanently reject anything. default_transport = discard guarantees that a message which *is* accepted goes nowhere — no accidental re-delivery of two thousand replayed messages to real mailboxes.
Capturing a real message corpus
Your corpus is only as good as what you capture with it, and the trap everyone falls into is capturing the .eml and nothing else. Reject decisions hinge on the envelope and the client IP, not the message body. Three capture methods, ranked by fidelity:
1. BCC capture (`always_bcc` / bcc maps). Simplest to enable, gives you the full RFC 5322 message including the Received chain:
ini
# main.cf on prod — capture a copy of everything
always_bcc = [email protected]
# or, more surgically:
sender_bcc_maps = hash:/etc/postfix/bcc
recipient_bcc_maps = hash:/etc/postfix/bcc
The catch: BCC fires *after* the recipient decision, so it never records what the SMTP-time restriction stack decided about RCPT TO. The Received header preserves the original client IP, but the RBL/greylist verdict that actually happened at connect time cannot be reconstructed from headers alone.
2. Pre-queue tap (`smtpd_milter` or `smtpd_proxy_filter`). Highest fidelity. A small milter that records the SMTP envelope — MAIL FROM, RCPT TO, HELO, client IP, TLS on/off — alongside the message. This is the only method that captures everything the reject logic actually depends on.
3. Maildir / journal harvest. Reading already-delivered mail out of /var/mail/vhosts. Convenient, but you've lost the envelope and the original client IP entirely — you're testing against a fiction.
Whatever you use, record for each message: full RFC 5322 body, MAIL FROM, RCPT TO, HELO name, client IP, and TLS state. And treat the corpus as PII — it is real people's mail. Scrub it or give it a short TTL; a captured-message store is a breach waiting to happen if you leave it lying around for months.
The replay pipeline
Replay means re-injecting each captured message, with its original envelope, against the staging relay and recording what happened. swaks is the right tool:
The --xclient line is the subtle bit almost everyone misses. Without it, every replayed message appears to come from 127.0.0.1, and any restriction that keys on the client — reject_unknown_client_hostname, reject_rbl_client, greylisting, per-IP rate policy — reproduces the *wrong* decision. XCLIENT lets the replay client assert the original IP and hostname, so staging sees the connection exactly as prod did. It only works because staging trusts the injector via smtpd_authorized_xclient_hosts = 127.0.0.1.
Loop the corpus and emit one structured result row per message:
For each message you now have: filename, final SMTP result code, and the DKIM verdict. Pull the rspamd/SpamAssassin score out of the X-Spam-Score or Authentication-Results header the milter stamped on, and you have a complete outcome row.
Diffing outcomes — the part that catches regressions
A single run tells you nothing. The whole technique is two runs and a diff: run the corpus against staging with the *current* config to establish a baseline, then apply the proposed change and run it again. Compare row by row.
What to diff:
SMTP result class — did any 2xx become 4xx or 5xx?
Spam score delta — per message, how did the score move?
DKIM / DMARC verdict — still pass where it passed before?
Header rewrites — any new bounces or unexpected mangling?
Then define hard pass criteria and gate on them:
No message that was 2xx in baseline may become 5xx in the candidate.
No ham may land within 0.5 of the reject threshold (margin ≥ 0.5).
DKIM verdict unchanged for every message that verified before.
Wire it into CI. A config change to /etc/postfix/main.cf or /etc/rspamd/local.d/ becomes a pull request; the pipeline runs the replay and posts the diff as a comment. Mail infrastructure gets reviewed like code, because now it *is* code with a test suite.
Worked example: raising the spam threshold safely
Suppose you want to catch more spam by dropping required_score from 5.0 to 4.5 in SpamAssassin (or lowering the rspamd reject band). It feels harmless. Replay a 2,000-message corpus, baseline versus candidate, and read the diff:
text
msgid baseline candidate sa_score verdict
<[email protected]> 2xx 5xx 4.7 HAM -> REJECT
<[email protected]> 2xx 5xx 4.6 HAM -> REJECT
... 12 more ...
--- summary ---
2xx -> 5xx transitions: 14 (all ham)
ham within 0.5 of thresh: 31
dkim verdict changes: 0
Fourteen previously-accepted legitimate messages — invoices and billing notifications scoring 4.6 to 4.9 — would now be permanently rejected. On production, those fourteen senders would simply have been told "denied" and moved on, and you'd have found out from an angry customer, not a report. The diff tells you the right move before you ship: don't reject at 4.5. For the 4.5–5.0 band, switch to a soft action — add a header or route to quarantine (add header in rspamd) — and keep hard reject at 5.0 and above. Re-run, confirm zero 2xx → 5xx, and now you ship a change that catches more spam and loses zero ham.
Checklist: before you reload production
Config diff is clean — postconf -n and master.cf match prod except the documented staging overrides.
Corpus is fresh and envelope-complete — MAIL FROM, RCPT TO, HELO, client IP, TLS state all present.
smtpd_authorized_xclient_hosts set so replay can assert original client IPs.
Isolation confirmed — default_transport = discard and soft_bounce = yes verified; no message can leave the box or be permanently rejected.
Baseline run captured against current config.
Proposed-change run completed and diffed against baseline.
Pass criteria met — zero 2xx → 5xx, ham margin ≥ 0.5, DKIM verdicts unchanged.
DKIM keys and selectors mirrored on staging so signatures validate identically.
Rollback rehearsed — prior main.cf in git, postfix reload of the known-good config tested.
Every mail change becomes a reviewable diff against real traffic, and you ship with a number instead of a prayer — "0 ham→reject regressions across 2,000 messages" is an argument; "looked fine in the logs" is not. This is exactly the loop evilmail.pro runs before anything touches its MXs: capture, replay, diff, gate. Once the pipeline exists, editing the live server directly stops feeling clever and starts feeling like what it is — betting deliverability on luck.
