A user clicked "This is spam" on one of your emails. The feedback loop fired, your webhook caught the complaint, and you suppressed the address. Correct behavior. Twelve minutes later a delivered event arrived for the *same* Message-ID — it had been sitting in a retry queue on the provider's side — and your handler, processing events in arrival order, flipped the recipient back to active. Your next campaign mailed them again. Second complaint. Now it's an FBL strike against your sending domain, and your shared IP's reputation takes the hit for a bug that lives in about four lines of state-transition code.
That failure is not exotic. It is the default outcome of trusting three things every provider quietly tells you not to trust. Webhook delivery events are unauthenticated (the endpoint is public HTTPS; anyone can POST to it), unordered (retries and internal queues mean delivered can land after complaint), and non-unique (at-least-once means the same event arrives two or three times). A correct handler doesn't paper over these with clever application logic. It stacks three independent guarantees, each enforced at a different layer: authenticate and reject replays at the edge, dedupe on a provider event id with a database constraint, and apply every event through a state machine that only ratchets one direction.
The delivery event isn't a fact, it's a claim
Treat the request body as hostile input, because it is. Your endpoint is reachable by the whole internet, the payload is trivially spoofable until you verify it, and the provider will happily resend it. Every major provider retries on any non-2xx response and on timeouts:
- AWS SNS (the transport for SES notifications) retries with backoff over roughly an hour, then hands off to a delayed retry policy that can span far longer.
- SendGrid retries the Event Webhook on failures.
- Mailgun retries failed webhook POSTs over several hours with increasing intervals.
There are two clocks you're racing. First, SNS expects a fast 2xx — seconds, not the full HTTP timeout budget — and a slow endpoint is treated the same as a failed one, which means your "successful but slow" handler generates duplicate deliveries. Second, any real work you do inline (DB writes, suppression-list calls, sending a downstream notification) is latency you're adding *before* the ACK. The rule that falls out of this is not negotiable:
Verify the signature, persist the raw payload, return 200. Then do the actual work off a queue.Everything downstream of that 200 is asynchronous. The edge handler's only job is to prove the request is authentic, durably record it, and get out of the way.
Layer 1 — Authenticate and reject replays at the edge
Every provider signs its webhooks differently, and the details matter because one wrong assumption turns your verification into security theater. The universal rule: verify against the raw request body, before you parse JSON. Frameworks that hand you a parsed and re-serialized body will silently change byte order, whitespace, and unicode escaping, and your signature check will fail — or worse, you'll "fix" it by verifying the re-serialized version, which verifies nothing.
AWS SNS (SES). The notification JSON carries a Signature, SignatureVersion, and a SigningCertURL. You download the certificate, verify the signature, and only then trust the message. The footgun: SigningCertURL is attacker-controllable in a spoofed payload, so you must validate that its host ends in .amazonaws.com *before* fetching it. Skip that check and you've built an SSRF that fetches attacker URLs on demand. Also handle Type: "SubscriptionConfirmation" — you have to GET the SubscribeURL exactly once to activate the topic.
SendGrid Event Webhook. ECDSA on the NIST P-256 curve. Two headers: X-Twilio-Email-Event-Webhook-Signature (base64) and X-Twilio-Email-Event-Webhook-Timestamp. The signed message is the timestamp string concatenated with the raw request body. Verify against the configured public key.
Mailgun. HMAC-SHA256 with your HTTP signing key over timestamp + token, hex-encoded. The payload hands you {timestamp, token, signature}. The token is single-use — remember it — and reject anything with a timestamp older than about five minutes.
Postmark. No HMAC. You secure it with HTTP Basic auth credentials embedded in the webhook URL plus a source-IP allowlist (Postmark publishes its outbound ranges). If your endpoint accepts unauthenticated POSTs, it accepts forged events.
import hmac, hashlib, time
def verify_mailgun(signing_key: str, timestamp: str, token: str, signature: str) -> bool:
# replay window first — cheap rejection before any crypto
if abs(time.time() - int(timestamp)) > 300:
return False
expected = hmac.new(
signing_key.encode(),
f"{timestamp}{token}".encode(),
hashlib.sha256,
).hexdigest()
# constant-time; never use ==
return hmac.compare_digest(expected, signature)Two non-obvious requirements bind all four: reject timestamps outside a ±5-minute window (this kills replayed captures), and use a constant-time comparison — hmac.compare_digest in Python, crypto.timingSafeEqual in Node. A plain == on the signature leaks bytes through timing and is a documented way to forge HMACs.
Layer 2 — Make idempotency a database property
Here is the mistake almost everyone ships first: check if the event exists, and if not, insert it.
# WRONG — races under concurrent redelivery
if not db.exists(event_id):
db.insert(event)
process(event)Under at-least-once delivery, the provider will send the same event twice within milliseconds, and two workers run that SELECT before either runs the INSERT. Both see "not exists." Both process. You've now double-suppressed, or worse, double-charged if the event triggers billing. Application-level uniqueness checks race by construction.
Idempotency has to be a property of the table, not the code path. One unique constraint, and you let the database reject the duplicate:
CREATE TABLE webhook_events (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
provider text NOT NULL,
event_id text NOT NULL,
payload jsonb NOT NULL,
received_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (provider, event_id)
);INSERT INTO webhook_events (provider, event_id, payload)
VALUES ($1, $2, $3)
ON CONFLICT (provider, event_id) DO NOTHING
RETURNING id;If the RETURNING clause gives you a row, this event is new — process it. If it comes back empty, it was a duplicate, and you skip everything downstream. Postgres enforces that unique constraint atomically at the index level, independent of transaction isolation, so concurrent redelivery collapses to exactly one winner. That single line does what a page of application logic can't do correctly.
The remaining question is *which field* to key on, because "event id" means something different everywhere:
- SendGrid —
sg_event_id. Unique per event and, critically, stable across retries. This is the correct dedup key, full stop. - Mailgun — the
event-idin the event payload, or the single-use signaturetoken. - SES — the SNS
MessageId(unique per SNS delivery) combined with the SESmail.messageId. - Postmark — there is no global cross-event id. You synthesize one:
RecordType + MessageID + timestamp, hashed into a composite key.
Store the full raw payload in that jsonb column. You will need it — for replaying a botched migration, for debugging a provider's schema change, for proving to a compliance auditor why an address is suppressed. Partition the table by received_at (monthly is fine) and drop old partitions, or the dedup table grows without bound. You only need enough history to cover the provider's maximum retry horizon plus a debugging buffer; 90 days is generous.
Layer 3 — Apply events through a monotonic state machine
Deduplication kills duplicates. It does nothing about *ordering*, and ordering is the harder enemy. The fix is a state machine where suppression is a one-way ratchet: certain states are terminal and dominant, and no later event can downgrade them.
The rules that make this correct:
- Resolve conflicts by event timestamp, not arrival order. Store the provider's own event timestamp. When an event arrives whose timestamp predates the last applied transition for that recipient, and the current state is terminal, drop it. The late
deliveredin our opening story has an *earlier* timestamp than the complaint — the state machine rejects it on both counts. - Permanent bounces and complaints are terminal and dominant. In SES the notification is
notificationType: "Bounce"withbounce.bounceTypeofPermanent,Transient, orUndetermined.Permanent→ suppress. Complaints (notificationType: "Complaint", delivered via FBLs in ARF format, RFC 5965) → suppress immediately, regardless of any prior state, no retry budget, no debate. - Transient bounces get a budget, not a suppression.
bounce.bounceSubTypeofMailboxFullorMessageTooLarge
Suppression only ever moves toward Suppressed. Nothing — not delivered, not open, not click, not a well-meaning support agent's manual re-activation triggered by a stale event replay — moves it back automatically. That single invariant is the difference between a handler that survives production and one that quietly re-mails people who reported you.
Wiring it together
The whole request path is small once the guarantees live in the right places. Edge handler:
@app.post("/webhooks/sendgrid")
async def sendgrid(req: Request):
raw = await req.body() # raw bytes, pre-parse
if not verify_sendgrid(raw, req.headers):
return Response(status_code=403)
db.execute(
"INSERT INTO webhook_events(provider,event_id,payload) "
"VALUES('sendgrid',$1,$2) ON CONFLICT DO NOTHING",
extract_event_id(raw), raw,
)
await queue.enqueue("apply_event", provider="sendgrid", body=raw)
return Response(status_code=200) # ACK now, work laterThe worker does the dedupe insert (or re-checks the row it inserted at the edge), loads the recipient's current state, computes the transition with timestamp-based conflict resolution, and upserts the suppression record. Suppression upserts are naturally idempotent — writing "suppressed" twice is a no-op. Any side effect that *isn't* idempotent (sending a downstream notification, issuing a refund) must itself be keyed on the event id, or you've just moved the double-processing bug one hop downstream.
Two operational gotchas: the SNS SubscriptionConfirmation handshake — you must GET the SubscribeURL once, or SES notifications never start flowing — and confirming that your queue's redelivery-on-failure semantics don't conflict with the provider's. Belt and suspenders is fine here; the dedupe key catches duplicates from both sources.
Operating it: the pre-flight checklist
- Verify signatures against raw bytes, before JSON parsing, with a constant-time compare.
- Reject any event whose timestamp is outside ±5 minutes; remember single-use tokens/nonces.
- For SNS: validate
SigningCertURLhost ends in.amazonaws.combefore fetching; handleSubscriptionConfirmation. - Enforce idempotency with a
UNIQUE(provider, event_id)constraint andON CONFLICT DO NOTHING RETURNING id— never aSELECT-then-INSERT. - Use the stable id per provider: SendGrid
sg_event_id, Mailgunevent-id, SES SNSMessageId+
Your alarms are what tell you whether the pipeline is healthy or under attack. A rising dedup hit-rate is normal and expected — it means your idempotency layer is earning its keep. A spike in signature failures is not a bug, it's someone probing your endpoint; alert on it. And watch queue lag against the provider's retry window: if your worker backlog exceeds the window in which the provider will retry, you're generating duplicate deliveries faster than you can dedupe them, and the whole pipeline starts to slip. Keep the edge fast, keep the constraint tight, and let suppression only ever move one direction.