Rotating and Storing Mail Secrets: DKIM Keys, SMTP Relay Credentials, and API Tokens Without Breaking Deliverability
A leaked DKIM key lets an attacker sign spam as your domain and torch a reputation you spent years building — and reputation has no rollback. Here's how to store mail secrets in a real manager and rotate them with the dual-selector pattern, zero downtime, and a script that actually runs.
EvilMail TeamJuly 23, 202612 min read
The secrets that keep mail flowing are the ones you never rotate
Two ways this goes wrong, both of which I've watched happen to real teams. First: a DKIM .private file gets committed to a repo — usually because someone copied /etc/opendkim/keys/ into a config-management repo "just to back it up" — and the repo goes public, or a contractor forks it. Second: an SMTP relay password sits in a .env that leaks through a misconfigured backup, a Docker image layer, or a Slack paste.
Neither of these is a normal breach, and that's the point. A leaked database password gets you an incident, a forced reset, and an apology email. A leaked mail secret gets you something with no rollback: reputation damage. With your DKIM private key, an attacker signs their spam as evilmail.pro and it passes DKIM at every receiver on earth — the signature is cryptographically valid, so it *is* you. With your relay credential, they pump volume through your paid SES/Sendgrid/Mailgun account until the provider suspends you and Gmail quietly starts folder-filing everything you send, including the mail your customers actually asked for.
Rotating DKIM, SMTP & API Secrets Without Breaking Mail | EvilMail — EvilMail Blog
You can rotate a password in an afternoon. You cannot un-spam a domain. The half-life of a torched sending reputation is measured in weeks of careful warm-up, and some receivers never fully forgive you.
The frame for the rest of this piece: mail secrets are rotating material with a defined lifetime, stored in a real secrets manager, never sitting on disk in plaintext, and rotated on a schedule that survives an actual incident. There are three classes, and they have genuinely different rotation physics:
DKIM private keys — DNS-coupled. Rotation is a two-key dance across a DNS TTL, not a database UPDATE. This is the hard case and the centerpiece.
SMTP relay credentials — provider-side plus a local config file. Overlap two credentials, cut over, revoke. Easy.
Provider API tokens and webhook secrets — provider-side only, cheap to cut. Easiest.
Where these secrets actually live today
Before you can rotate anything, be honest about where it sits right now on a typical box.
DKIM private key. On an OpenDKIM signer it's /etc/opendkim/keys/evilmail.pro/s2026a.private — a PEM RSA or Ed25519 key. On rspamd it's under /var/lib/rspamd/dkim/ or wherever your dkim_signing.conf points. The public half lives in DNS as a TXT record at s2026a._domainkey.evilmail.pro. The problem: that .private file is a flat file on persistent disk, and if it's world-readable or backed up into a repo, it's gone. Correct perms are non-negotiable:
Compiled to a .db with postmap, applied with postfix reload. The plaintext line stays on disk. If your config repo tracks /etc/postfix/ — and most do — that credential is in git history forever unless you scrub it.
API tokens and webhook signing secrets. REST tokens for the provider API, plus the HMAC secret you use to verify inbound delivery webhooks. These usually end up in an app .env or a Kubernetes secret. The webhook secret is the sneaky one: teams set it once at integration time and never touch it again.
"Good" for all three looks the same: the secret is fetched at runtime from a manager, decrypted into memory or a tmpfs path the signer reads, never committed, and locked to 0600 owned by the service account. Nothing load-bearing lives in plaintext on the SSD.
Rotating DKIM without breaking a single message
This is where people get hurt, because the instinct is to overwrite the key and reload the signer. Do that and you break mail for hours.
Here's why. DNS records are cached. When you publish s2026a._domainkey, resolvers all over the world cache it for the record's TTL — commonly 3600 seconds. Receivers also validate mail that's been sitting in a retry queue, sometimes for hours after you sent it. The moment you swap the signer to a new key but the old public record has been yanked (or the new one hasn't propagated), any message signed with the key that DNS can't confirm fails DKIM. That failure feeds straight into DMARC, and now legitimate mail is getting quarantined or bounced.
The fix is the dual-selector pattern: two selectors live simultaneously, and you only remove the old one after everything has drained. A selector is just the label in the DNS record (s2026a, s2026b), so running two costs nothing.
The safe sequence, command by command. Generate the new key with a date-encoded selector so anyone reading DNS can see instantly which is current:
Only once opendkim-testkey reports key OK do you flip the signer to sign with s2026b. Monitor DMARC for a cycle (more on that below). Then — and not a minute earlier — remove the s2026a TXT record from DNS and shred the old private key: shred -u s2026a.private. Keep the old selector live until you're confident nothing signed with it is still sitting in a receiver's retry queue; a few extra days of a dead selector in DNS cost nothing.
On algorithms in 2026: sign with both Ed25519 and RSA-2048. Ed25519 keys are tiny and fast, but a stubborn minority of receivers still don't validate them, so RSA-2048 remains the compatibility floor. Dual signing means both signatures ride along and each receiver validates whichever it understands. Rotate Ed25519 selectors yearly; RSA every six months.
Rotating relay credentials and API secrets — the easy 90%
Everything else follows one rule: overlap-then-revoke, never revoke-then-create. Create the new credential while the old one is still valid, deploy, verify, and only then kill the old one. Zero downtime falls out of the overlap window for free.
SMTP relay. Mint the new credential in the provider console or API first — both are now valid. Push it to the secrets manager, re-render sasl_passwd, recompile, reload:
bash
postmap /etc/postfix/sasl_passwd
postfix reload
Verify an authenticated test send actually works with the new credential before you revoke anything:
Green light? Revoke the old credential provider-side. Done.
API tokens and webhook secrets. Same overlap shape. The webhook secret has one extra trick: during the cutover window, verify inbound signatures against both the old and new secret, accept if either matches, then drop the old after the window closes. That way you never hit a moment where in-flight webhooks fail HMAC verification because the sender is still signing with the previous key.
Putting it in a secrets manager
Stop keeping the source of truth on the mail server's disk. Three patterns that work for a self-hosted stack, in rough order of infrastructure weight.
Vault KV v2. Store the key, versioned so you keep the last N for rollback:
A Vault Agent template renders the key at boot to a tmpfs path the signer reads — not to persistent disk — and audit logging records every read, so you know exactly when and by whom a key was fetched. KV v2's version history means an accidental bad rotation is one vault kv rollback away.
SOPS + age is the lighter option and my default for small stacks. You encrypt the secret file and commit the *encrypted* blob to git safely; it's decrypted only at deploy time:
Cloud KMS — AWS Secrets Manager or GCP Secret Manager — gives you managed rotation Lambdas/functions for relay creds and API tokens with basically no code.
Whichever you pick, mount the live private key on tmpfs so it never touches the SSD, and inject it with systemd rather than a plaintext file:
Here's the honest part: rotation that isn't automated is rotation that never runs. Manual key rotation is where security theater goes to die — someone puts "rotate DKIM" on a quarterly checklist, it slips twice, and eighteen months later you're signing production mail with a key that's been in three ex-employees' laptop backups.
At minimum, run a cron job or systemd timer that *alerts* when any secret ages past policy. Full auto-rotation is safe for relay creds and API tokens — no human in the loop needed. DKIM stays gated on a human for the DNS flip, because publishing a bad TXT record silently breaks mail and you want eyes on that step.
Secret class
Cadence
Automation
DKIM RSA-2048
Every 6 months
Alert + human-gated DNS flip
DKIM Ed25519
Yearly
Alert + human-gated DNS flip
SMTP relay creds
90 days
Fully automatable
API tokens
90 days
Fully automatable
Webhook signing secret
On incident + yearly
Fully automatable
Detection after a flip: watch your DMARC aggregate reports (the rua address). Parse the XML for a rise in <result>fail</result> inside <dkim> blocks. A spike right after a rotation is the unmistakable signal that a signer-versus-DNS mismatch slipped through — the new selector isn't fully propagated, or the signer flipped early. Catch it in the first report cycle, not from a customer complaint.
Pre-flight checklist and the leaked-key fast path
Run this before every rotation, no exceptions:
Overlap window open? New credential/selector created and live *before* touching the old one.
TTL waited? For DKIM, dig confirms the new TXT resolves and you've waited past TTL plus margin.
Test verified?opendkim-testkey says key OK, or swaks authenticates against the new relay credential.
Old secret revoked? Only after the new one is confirmed serving.
Old key shredded?shred -u the private key, and only after retry queues have drained.
DMARC watched? First aggregate report post-flip checked for a DKIM-fail spike.
When a key is already leaked, throw out the patience. Skip the TTL wait — publish and sign with a new selector *immediately* and accept the short verification gap, because a few messages failing DKIM for an hour is vastly cheaper than an attacker continuing to sign as you. Revoke the compromised relay/API credential provider-side right now. Then do the forensics: pull DMARC aggregate reports and provider send logs for the entire exposure window and look for volume you didn't send, unfamiliar sending IPs, or DKIM passes from infrastructure that isn't yours.
The operational rule that keeps all of this honest: every mail secret has a next-rotation date, and if you can't name it off the top of your head, it's already overdue.
