Grafana for Postfix: Queue Depth, Delivery Rate, Deferrals, and Rejection Reasons
postqueue -p tells you how many messages are stuck right now and nothing about why or for how long. Build a split-brain dashboard: Prometheus for queue depth as a time series, Loki for the high-cardinality reasons mail actually left the queue — with alerts that know the difference between "page me" and "log it."
EvilMail TeamJuly 26, 202612 min read
The queue is "empty" and you're still bleeding mail
You get paged because outbound is slow. You SSH in, run mailq, and see twelve messages. Twelve. Everything looks fine, so you close the ticket and go back to bed. What you didn't see: the deferred queue had been oscillating 0 → 4,000 → 0 on every retry cycle for the last forty minutes — an upstream RBL listing your relay, backing off, retrying, clearing, backing off again. You happened to catch a trough. The spike is invisible to postqueue -p because a snapshot has no memory.
That is the whole problem with running a mail plant on point-in-time commands. postqueue -p | wc -l answers "how many right now" and refuses to answer anything that actually matters:
Is the queue growing or draining this minute?
Are we in a retry storm — the same messages cycling active → deferred → active?
Is delivery delay creeping, so mail still leaves but takes 90 seconds per message instead of 400ms?
Did the rejection mix drift after last night's smtpd_recipient_restrictions change?
A dashboard has to answer two continuous questions, and they are not the same shape of data. One is "what is sitting in the spool right now" — a cheap, low-cardinality number you sample every fifteen seconds. The other is "why did the last N thousand messages leave, or fail to leave, the queue" — free-form text with hundreds of distinct reject strings and remote-MTA responses. Put both in the same backend and you either lose the reasons or melt your time-series database. So we build with two.
Two data sources, one dashboard
Depth is a gauge. It has a handful of possible label values — incoming, active, deferred, hold, maildrop — and that is the entire cardinality. Prometheus was built for exactly this.
Reject reasons are the opposite. "Recipient address rejected: User unknown in virtual mailbox table", "blocked using zen.spamhaus.org", "450 4.2.0 mailbox busy", "Connection timed out" — thousands of distinct strings, often carrying an embedded recipient or remote hostname. The cardinal sin, the one that pages *you* at 3am when your TSDB OOMs, is promoting reason or recipient_domain to a Prometheus label. Each unique value mints a new time series. A busy relay manufactures hundreds of thousands of series in an afternoon and takes Prometheus down with it.
So reasons go to Loki, where high-cardinality text lives *in the log line*, not in a stream label, and you query it with LogQL at read time. The split looks like this:
Assumptions for everything below, current as of 2026: Postfix 3.8+, Prometheus 2.x/3.x, Grafana 11/12, Loki 3.x, Debian-style paths (/var/log/mail.log; on RHEL it's /var/log/maillog).
Wiring postfix_exporter to the spool and the log
The exporter of record is kumina/postfix_exporter — a small Go binary that does two jobs at once: it reads the showq socket for live queue depth, and it tails the mail log to build typed counters (deliveries, rejects, delays). Grab a release or build it, then drop in a systemd unit.
Two access problems trip up everyone. First, the exporter must read the showq unix socket at /var/spool/postfix/public/showq; if the service user isn't in the postfix group the scrape returns permission denied and your depth panels stay flat at zero. Second, if you point it at the logfile it needs read on that too.
Confirm it before wiring Prometheus: curl -s localhost:9154/metrics | grep postfix_showq. If that's empty, it's the group membership — check id postfix_exporter, make sure postfix is listed, then systemctl daemon-reload && systemctl restart postfix_exporter.
Latency (from log):postfix_smtp_delivery_delay_seconds — histogram with a stage label (before_queue_manager|queue_manager|connection_setup|transmission).
Start with the picture of the spool itself. Stack the queues so the shape of an incident is obvious at a glance:
promql
sum by (queue) (postfix_showq_message_size_bytes_count)
A healthy plant shows incoming and active hugging zero and deferred as a low, slowly-varying baseline. The active queue is the tell: normal active is near-empty — single digits. A persistent active backlog means qmgr is paused, a transport is wedged, or a concurrency limit is throttling you. That's why the second panel is a simple threshold visual on active.
For throughput, the honest caveat: qmgr_messages_removed_total counts everything that left — delivered, bounced, and expired lumped together. On its own it is a lying delivery metric. A relay bouncing 100% of mail shows a beautiful "removal rate."
Graph it, but always pair it with the log-derived status=sent count from Loki (next section) so "removed" and "actually delivered" sit side by side. When they diverge, you're bouncing.
Delivery delay is where slow-motion failure shows up before the queue does. Use the histogram:
promql
histogram_quantile(0.95,
sum by (le, stage) (rate(postfix_smtp_delivery_delay_seconds_bucket[5m])))
Break it out by stage. The queue_manager stage delay is your early-warning signal: when qmgr can't hand messages to delivery agents fast enough, that number climbs *before* the active queue visibly backs up.
And the one panel everyone forgets — the deferred queue's derivative, not its absolute value:
A deferred queue of 4,000 that is draining is a non-event; the retry cycle is working. A deferred queue of 400 with a sustained positive slope is an incident in progress. You alert on the slope.
Deferrals and rejection reasons from the logs
Now the other half of the brain. Promtail tails the maillog and ships it to Loki. The pipeline extracts only the coarse status as a stream label; the reason text stays in the raw line where Loki can search it without exploding cardinality.
Four label values total. That's the whole point — you can now split delivery status over time cheaply:
logql
sum by (status) (
count_over_time({job="postfix"} | regexp `status=(?P<status>\w+)` [$__interval])
)
Stack that as a timeseries and the sent vs deferred vs bounced ratio becomes your real deliverability pulse. Then the two tables that make the board worth having. Top reject reasons over the last hour:
logql
topk(10, sum by (reason) (
count_over_time({job="postfix"} |= "reject:"
| regexp `reject: RCPT from \S+: \d+ \d\.\d\.\d+ (?P<reason>[^;]+)` [1h])
))
And deferral reasons — filter to status=deferred and pull the remote MTA's own words out of the said: / delivery temporarily suspended: text. This is where you learn whose problem it is. The strings you'll actually bucket in production:
Greylisting — "Greylisted, see http://..." → transient, self-heals on retry, ignore unless the *rate* jumps.
RBL — "blocked using zen.spamhaus.org" → your reputation. This is your problem, right now.
Recipient — "User unknown in virtual mailbox table", "Sender address rejected: Domain not found" → usually your senders' bad lists, sometimes a broken alias.
Remote throttle — "421 4.7.0 too many connections", "450 4.2.0 mailbox busy", "Connection timed out" → their capacity, back off your concurrency.
Reading the remote-MTA text is the single skill that separates "the queue is deep, panic" from "Gmail is deferring us for reputation, fix SPF alignment" or "the far end is just slow, leave it."
Alerts: page vs. inform
Split every alert by one question — must a human wake up? Absolute-threshold alerts on the deferred queue fail this test badly: the queue *breathes* by design, so a "deferred > 1000" rule fires every retry cycle and gets muted within a week. Alert on behavior, not level.
Page (a person is needed):
yaml
- alert: DeferredQueueGrowing
expr: deriv(postfix_showq_message_size_bytes_count{queue="deferred"}[15m]) > 0.5
for: 15m
labels:
severity: page
annotations:
summary: "Deferred queue growing for 15m — sustained backlog, not a retry blip"
The for: 15m is load-bearing. It's longer than one retry cycle, so ordinary breathing never trips it; only a genuine sustained climb survives the window. Also page on a stuck active queue — because normal active is near-zero, a persistent backlog is unambiguous:
Third pager: delivery-rate collapse, comparing rate(qmgr_messages_removed_total[5m]) against its trailing-hour avg_over_time baseline.
Inform (dashboard or Slack, no pager) — a SASL auth-failure spike (postfix_smtpd_sasl_authentication_failures_total, usually a credential-stuffing probe, not an outage), a single reject reason surging, or delivery-delay p95 creeping up. As a Grafana-managed alert you can drive "inform" straight off LogQL:
Exporter perms verified: curl localhost:9154/metrics | grep showq returns non-zero, service user is in the postfix group.
Prometheus scrape up{job="postfix"} == 1, and the delivery-delay histogram actually has le buckets (no buckets → histogram_quantile returns NaN).
Promtail shipping: status is the only high-value label; confirm in Loki with topk(20, count by (reason)(...)) that you never promoted reason or a recipient domain to a stream label.
Loki retention set deliberately — reason tables are worthless if you can't look back past a shift change.
Alert for: durations tuned: deferred derivative ≥ 15m, active ≥ 5m. No absolute-count alert on deferred.
Dashboard has a transport template variable so you can filter per-transport when one relay wedges while the rest are fine.
Keep the ops commands one keystroke away for when the board says "look here": postqueue -j (JSON, 3.1+), qshape deferred, postconf maximal_queue_lifetime (default 5d — the clock behind expired).
Two backends, one board, and alerts that know the difference between the queue breathing and the queue drowning. That's the difference between finding out from Grafana and finding out from your senders.
