Why fuzzy hashing catches what Bayes and RBLs miss
Picture a 40,000-message "invoice overdue" blast. Each copy swaps the recipient's first name, injects a unique tracking token in the footer, and rewrites exactly one word — "payment" becomes "settlement," "urgent" becomes "immediate." The sender walks a fresh IP every few hundred messages, all inside one freshly-warmed /20. None of this is exotic; it's the default shape of a modern campaign.
Watch your two workhorse filters fail in real time. RBLs blocklist IPs, but the IPs rotate faster than any DNSBL can list them, so the first several thousand messages land before the block catches up. Bayes trains on words, but every copy is 99% identical prose scoring the *same* value — and if the spammer tuned that value to sit at 4.9 against your 5.0 reject threshold, all 40,000 copies clear it. Per message, every signal is weak.
What the spammer *can't* cheaply mutate is the skeleton: the same three-paragraph structure, the same call-to-action button, the same attached PDF. Fuzzy hashing fingerprints that skeleton. It operates on normalized text *structure*, not exact bytes, so a name swap and a one-word synonym collapse to the same hash — or a near neighbor close enough to match. rspamd returns a match *probability*, not a boolean, and scores accordingly.
Set expectations correctly: fuzzy is a bulk / near-duplicate detector, not a content classifier. It is nearly blind on a single hand-written message and devastating on volume. It shines exactly where per-message signals are weak but the campaign is large — which is most of what actually reaches your MX.
How rspamd computes a fuzzy hash
The mechanics matter, because half the tuning mistakes come from not knowing them.
For each text/* MIME part, rspamd normalizes: strips HTML, lowercases, collapses whitespace, and tokenizes with language awareness. It then builds overlapping shingles — chains of three consecutive tokens — hashes each shingle, and keeps a fixed min-hash signature of 32 minima. That 32-number signature *is* the fuzzy hash. Two messages are "the same" when enough of those minima collide; the fraction that collide is the match probability.
There's a second regime. Text shorter than min_length and every binary attachment (image, PDF, docx) skip shingles entirely and get an exact blake2 hash instead. This is the key mental split:
- Shingle / near-dup for prose — catches mutated text.
- Exact hash for images, PDFs, and short parts — catches the identical attachment riding a thousand different cover letters.
Two thresholds you must set. min_length defaults to 0, which is a trap: with zero, a two-line "thanks, received" reply can shingle-match another short message and false-positive. Set it to around 64 symbols so only substantial text gets the near-dup treatment. min_bytes (~1024) does the same job for attachments, so a 200-byte tracking pixel never becomes a fingerprint.
Standing up your own fuzzy_storage
Here's what every shipped-config tutorial skips: out of the box, rspamd's fuzzy rule points at fuzzy.rspamd.com, the public feed. It's read-only, shared with the entire internet, and always a few minutes behind. You can query it, you can never teach it, and a campaign hitting your traps does nothing until rspamd.com's own machinery decides to publish. All the leverage is in running your own storage.
Put the server config in /etc/rspamd/local.d/fuzzy_storage.conf:
backend = "redis";
servers = "127.0.0.1:6379";
listen = "*:11335";
expire = 90d; # hash TTL
allow_update = "127.0.0.1,10.0.0.0/24"; # who may write hashes
keypair {
pubkey = "z8f..."; # from: rspamadm keypair
privkey = "y3a...";
}Two backends, and the choice is the whole point. Legacy SQLite stores everything in a single file at /var/lib/rspamd/fuzzy.db — fast, zero dependencies, but strictly single-node. Nothing you learn on mx1 exists on mx2. Redis (backend = "redis") is the shared option: every node reads and writes one hash space. Redis keys each hash by its signature with a TTL derived from expire, and the value carries the flag, the weight, and last-seen counters. That TTL is why old campaigns age out on their own instead of accumulating forever.
allow_update is an ACL, not decoration. Only listed sources may add or delete hashes; everyone else is read-only. Lock it to your own subnet.
Wiring fuzzy_check to your storage and encrypting the wire
The client side lives in /etc/rspamd/local.d/fuzzy_check.conf:
rule "local" {
servers = "10.0.0.5:11335";
encryption_key = "z8f..."; # storage's PUBLIC key
read_only = false; # this node may teach
min_length = 64; # ignore the default 0
min_bytes = 1024;
fuzzy_map {
FUZZY_DENIED { max_score = 20.0; flag = 1; }
FUZZY_PROB { max_score = 10.0; flag = 2; }
FUZZY_WHITE { max_score = 2.0; flag = 3; }
}
}Each flag is a bucket. Flag 1 is your hard-spam corpus, flag 2 is "probably bulk," flag 3 is deliberately negative — a whitelist bucket (more on that below). max_score is the ceiling, not the fixed hit.
Encrypt the wire. Fuzzy queries and teaches go over UDP; without encryption, anyone on-path can inject junk into your buckets or scrape what you've learned. Generate a curve25519 keypair once:
rspamadm keypairPut the public key in the client's encryption_key and the full keypair under the storage config. Now every packet between check and storage is authenticated and sealed with curve25519 — no injection, no scraping.
One behavior to internalize: the emitted symbol weight is `max_score` × match-probability. A full shingle collision fires the whole 20.0; a partial near-match might fire 6 or 8. That single scaling is why one symbol produces a soft nudge on a loose match and a hard reject on a dead-on duplicate.
Teaching it: manual, spam-trap, and autolearn
Teaching is where the system earns its keep. The flag picks the bucket, the weight sets your confidence:
# learn one spam into flag 1 at weight 10
rspamc -f 1 -w 10 fuzzy_add message.eml
# undo a false positive — pull the hash back out
rspamc -f 1 fuzzy_del message.eml
# controller commands are privileged; authenticate with -P
rspamc -P "$CONTROLLER_PASS" -f 1 -w 12 fuzzy_add spam.emlfuzzy_add and fuzzy_del are controller commands and need enable_password (passed as -P). That's a feature — you don't want an unauthenticated caller poisoning your buckets.
Now the part that makes this fleet-scale. Point a spam-trap mailbox — an address that only ever receives spam — at a cron job that feeds every new message straight into the fuzzy store:
#!/bin/bash
for f in /var/spool/trap/new/*; do
rspamc -P "$CONTROLLER_PASS" -f 1 -w 12 fuzzy_add "$f" && rm "$f"
doneRun it every minute. A campaign that lands in the trap becomes a fleet-wide block within one fetch interval, before most of the real recipients are even hit. Keep the trap ratelimited and audited, though: if an attacker learns the trap address and mails it *legitimate-looking* content, you don't want that auto-learned as spam. Learn conditions and ratelimits exist precisely to bound that blast radius.
Finally, seed FUZZY_WHITE (flag 3) with your *own* legitimate bulk — newsletters, transactional receipts, password-reset templates. Its negative score suppresses false positives when those high-volume-but-good messages inevitably start looking like near-duplicates of each other.
Sharing across the fleet: Redis vs master/mirror
This is the payoff.
Topology A — shared Redis. N rspamd frontends, one Redis (or Redis Sentinel / cluster for HA). Every node reads and writes the same hash space. The trap-hit-on-mx1 story just works: learned on mx1, written to Redis, instantly visible to mx2 and mx3. Simplest to reason about — one consistent view, one blast radius if Redis dies. For a single region, this is the answer.
Topology B — mirrored SQLite. When you genuinely can't centralize Redis — cross-datacenter, a hard latency budget, a requirement to survive a backend outage with local reads — run independent SQLite storages and sync them with a master plus mirror/slaves relationship. You trade instant consistency for local read latency and eventual convergence.
Tuning, and what fuzzy will and won't do
Weight FUZZY_DENIED so that on its own it lands *near* — not past — reject. A max_score around 12–15 that pushes a message over the line only when combined with a weak SPF softfail or a mild Bayes lean is far safer than a 20 that rejects on the fuzzy hit alone. Over-weight it and you *will* start bouncing quoted replies and forwarded newsletters, because a reply quoting the original spam shingles against the spam.
Tune expire to the threat, not to a round number. 90d is the sane default. Drop to 30d against fast-rotating campaigns so mutated *legitimate* mail — this quarter's newsletter reusing last quarter's template — doesn't keep matching a stale fingerprint. Too long and old hashes haunt good mail; too short and you re-learn the same campaign daily.
Verify what actually fired. Check the rspamd history in the web UI or grep /var/log/rspamd/rspamd.log, and confirm the exact symbol — FUZZY_DENIED vs FUZZY_PROB vs FUZZY_WHITE — and its scaled score. rspamc stat confirms your learn counts are climbing. If you think fuzzy blocked something but the log shows the reject came from DMARC, you're tuning the wrong knob.
Keep it in its lane: fuzzy is a bulk / near-duplicate detector. It complements — never replaces — SPF, DKIM, DMARC, and Bayes. It is the layer that catches the coordinated blast those four let slip through, and nothing more.
Deployment checklist
- Private storage, firewalled —
:11335reachable only from your MX nodes, never the public internet. - Encryption keypair deployed both sides — storage holds the keypair, clients hold the pubkey in
encryption_key. - `read_only = false` on at least the teaching node; leave leaf nodes read-only if you centralize learning.
- `min_length = 64` set — never ship the default
0. - Redis backend + TTL —
backend = "redis"withexpireat 90d (30d for aggressive rotation). - Spam-trap → `fuzzy_add` cron wired and running every minute, with
-Pauth. - FUZZY_WHITE seeded