Postfix Logs to Loki: Turn the Mail Firehose Into Queryable Metrics
Postfix logs are a firehose of unstructured text nobody can query at 3am. Here's how to keep parsing out of the mail path, scrape journald with Promtail, and stitch deliveries back together in LogQL — deferred rate, TLS anomalies, and per-relay latency without ELK's cost.
EvilMail TeamJuly 26, 202610 min read
You have done this. A customer says mail is slow, you SSH into the mail host, and you type journalctl -u [email protected] | grep 4WxY2k3Rz1z to reconstruct a single delivery from four lines scattered across four processes. Then someone asks "what's our deferred rate right now?" and you have nothing — no chart, no number, just a scrolling wall of text and a queue that spiked at 03:12 for reasons you'll never chart after the fact.
But journald already holds most of the structure you need. Loki and Promtail turn those records into queryable streams and alertable metrics, and the trick is doing queue-ID correlation at query time instead of bolting a regex parser onto the mail path. Keep parsing out of Postfix, keep label cardinality bounded, and extract the interesting fields only when you ask a question.
Why Postfix logs resist querying
Postfix is not one process — it's a supervised constellation, and every inbound message touches several of its daemons on the way out. One message produces one line per stage, and the only thing tying those lines together is a hex-ish queue ID like 4WxY2k3Rz1z.
Two things make this hard to query. First, correlation lives across lines — the client IP is on the smtpd line, the recipient and status verb are on the smtp line, and the sender and size are on qmgr. No single line tells the whole story. Second, the queue ID is reused once a message leaves the active queue. Postfix hands that slot to the next message, so 4WxY2k3Rz1z from 03:12 and 4WxY2k3Rz1z from 09:40 are unrelated. Correlation is only valid inside a bounded time window — treat the queue ID as a session key, not a primary key.
Let journald do the first 80%
On any modern systemd box, Postfix runs as [email protected] — the - is the default main instance — and it already logs to the journal with per-service identifiers. You don't need rsyslog, and you don't need to tail /var/log/mail.log. Confirm the structure is there:
You'll see fields Promtail can turn into labels for free: SYSLOG_IDENTIFIER (values like postfix/smtpd, postfix/qmgr, postfix/smtp), _SYSTEMD_UNIT, _PID, PRIORITY, _HOSTNAME, _BOOT_ID. That's the structure most people rebuild by hand with grok patterns.
Before you ship anything, fix journald's retention and the flood trap. Persistent storage is not the default on every distro, and the rate limiter will silently eat your busiest minutes — which is exactly when you need the logs. In /etc/systemd/journald.conf:
ini
[Journal]
Storage=persistent
SystemMaxUse=2G
MaxRetentionSec=30day
# The single most common cause of "missing" Postfix logs:
# under a mail burst journald drops lines and prints
# "Suppressed N messages". Disable rate limiting on the mail host.
RateLimitIntervalSec=0
RateLimitBurst=0
Then systemctl restart systemd-journald. If you'd rather raise the limit than disable it, RateLimitBurst=10000 over RateLimitIntervalSec=30s survives most sending patterns — but on a dedicated mail host I just turn it off. Dropped log lines during a delivery storm are worse than a bit of extra disk churn.
Promtail: scrape the journal, not the file
Point Promtail at the journal, not the file. Tailing /var/log/mail.log re-introduces logrotate races and throws away the structured fields you just confirmed exist. The journal scraper reads those fields as __journal_* relabel sources.
The relabels promote exactly four things to labels: service, unit, level, host. All four are bounded sets — a handful of Postfix services, one unit, a few priority levels, one host per agent. That is the whole cardinality discipline.
The queue ID, recipient, sender, and relay IP must never become labels. Those are unbounded — millions of distinct queue IDs a week — and each unique label combination is a new stream in Loki's index. Promote a queue ID to a label and you'll watch the ingester's memory climb until it OOMs. Keep those values in the log body and extract them at query time, which is what the next section is about. Set Promtail's positions file on a persistent disk (/var/lib/promtail/positions.yaml) so a restart doesn't replay half a day of journal.
Correlating a delivery in LogQL
With bounded labels in place, the interesting fields get pulled on read. LogQL's | regexp (and | pattern or | logfmt where the format is clean) extracts named captures from the line body without ever touching the index:
Now status, relay, and delay are available as extracted labels for the rest of the pipeline — filter on them, aggregate on them, chart them. This is the payoff: the fields that would have blown up your index as ingest-time labels are free at query time, scoped to whatever window you're looking at.
Be honest about what Loki can't do. It is not a relational store, and there is no cheap line-to-line JOIN. You cannot ask it to glue the recipient from cleanup onto the status from smtp across two streams the way SQL would. You have two workable options. For an incident, query by queue ID interactively and let the human read the story:
That one line returns every service's contribution to that single message in timestamp order — the full delivery narrative, no JOIN required. For dashboards and alerts, accept per-service aggregate metrics computed from the smtp and lmtp lines, which carry the status verb and the relay and are all you actually page on.
From lines to metrics you alert on
Turn extracted fields into rates. Deferred ratio is the workhorse — the fraction of delivery attempts that got a temporary failure:
Deferred ratio > 0.05 over 15m — the query above. A sustained 5% temp-fail rate means a relay, DNS, or greylisting problem you want to catch before the queue backs up.
Bounce (5xx) spike — a sustained rise in status=bounced or dsn=5.x.x. This is reputation or a bad recipient list, and it's the metric that predicts a blocklisting.
TLS anomaly count — occurrences of Untrusted TLS connection established or deferrals citing TLS. A jump here means a peer's cert changed or an on-path device is interfering with STARTTLS.
Per-relay p95 `delay` — use the delays= breakdown (before-queue / queue-manager / connection / transmission) to see whether the latency is your queue or their server.
Running these raw over 30 days on every dashboard refresh is slow and expensive, because unbounded LogQL re-scans log bodies each time. Pre-compute the ones you alert on with a Loki recording rule in the ruler, evaluated every minute:
Now postfix:deferred_ratio:5m is a cheap numeric series. Dashboards and alerts read the recorded metric instead of re-scanning raw logs, so you can keep 30 days of logs while retaining 90 days of the recorded metric at a fraction of the cost.
Grafana panel and alert wiring
Build a stat panel bound to postfix:deferred_ratio:5m for the current number. To name the worst-behaving relays, add a table panel:
logql
topk(5, sum by (relay) (
rate({service="postfix/smtp"}
| regexp `status=(?P<status>\w+)`
| regexp `relay=(?P<relay>[^,]+)`
| status="deferred" [15m])))
Attach an alert rule to the recording-rule metric with for: 15m so a single flaky minute doesn't page anyone. In the alert annotation, link back to an Explore query pre-filtered by the offending relay — the on-call clicks the alert and lands on the raw lines in one hop instead of reconstructing the query at 3am.
Operational checklist
Persistent journald storage on (Storage=persistent), retention sized to your window.
RateLimitBurst=0 (or raised well past your peak) on the mail unit so bursts aren't silently dropped.
Promtail journal scraper with cardinality-safe relabels — service, unit, level, host only.
Positions file on a persistent disk so restarts don't replay the journal.
Queue ID, recipient, sender, and relay IP are NEVER labels.
Status, relay, and delay extracted at query time with | regexp.
Recording rules for the four alert metrics; dashboards read the recorded series, not raw logs.
Retention sized deliberately — e.g. 30d logs, 90d recorded metrics.
Test the pipeline for real: inject deferrals with a null-routed relay, or hold and re-queue messages with postsuper, and confirm the deferred alert actually fires end to end.
Once the queue ID is your correlation key and cardinality stays bounded, the mail host stops being a black box. Grep-at-3am becomes a saved LogQL query, and the deferred-rate spike you used to reconstruct after the fact becomes an alert that fired fifteen minutes before the customer noticed.
