Install and operate
Monitoring
Mail fails quietly. A queue that stops draining, a certificate that expired, a disk that filled — none of them announce themselves, and all of them look like "my email is a bit slow today" until someone notices a week of missing messages.
These are the things worth watching, in the order they matter.
Alert on these#
| Signal | Threshold | Why |
|---|---|---|
| Process not running | Any | Nothing else matters |
| Queue depth climbing | Growing for 30 min | Delivery is stuck |
| Oldest queued message | Older than 1 hour | Something is failing repeatedly |
| Certificate expiry | Under 14 days | Renewal or the reload hook broke |
| Disk free | Under 20% | Postgres stops writing when it fills |
| Bounce rate | Over 5% of sends | Reputation damage in progress |
| Webhook endpoints disabled | Any | Twenty consecutive failures disabled one |
| Backup heartbeat | Missing for 36 hours | The backup silently stopped |
Is it running#
systemctl is-active corsair
curl -sf localhost:3000/.well-known/mta-sts.txt >/dev/null && echo "http ok"/.well-known/mta-sts.txt is served without authentication, so it is a clean HTTP liveness probe. It does not touch the database — every /api route either requires a session or is a POST, so there is no single endpoint that proves both. Check Postgres separately:
psql "$DATABASE_URL" -Atc "SELECT 1" >/dev/null && echo "db ok"Probing an authenticated endpoint and treating 401 as healthy works too, but it proves only that the HTTP tier is answering — a 401 is returned before anything queries the database.
For the mail listeners, check the sockets:
for p in 25 587 465 143 993 110 995; do
nc -z localhost $p && echo "$p ok" || echo "$p DOWN"
doneThe delivery queue#
The single most useful thing to watch. If it drains, mail is flowing.
SELECT status, count(*), min(created_at) AS oldest
FROM deliveries
GROUP BY status;Depth alone is not the alarm — a spike after a burst of sending is normal. What matters is depth that keeps growing and an oldest entry that keeps getting older.
Failures back off from one minute out to about five days, so a genuinely undeliverable message sits in the queue for days by design. A pile of them appearing at once usually means one destination is refusing you.
-- What is failing, and where to. last_error is the verbatim final SMTP reply,
-- which is what distinguishes a greylist from a block.
SELECT split_part(rcpt_to, '@', 2) AS domain,
count(*),
max(last_code) AS code,
max(last_error) AS reply
FROM deliveries
WHERE attempts > 2 AND status <> 'sent'
GROUP BY 1
ORDER BY 2 DESC
LIMIT 10;One domain dominating that list is a reputation problem with that provider. Every domain appearing means your outbound path is broken.
Inbound flow#
SELECT date_trunc('hour', created_at) AS hour, count(*)
FROM mail_log
WHERE direction = 'inbound' AND created_at > now() - interval '24 hours'
GROUP BY 1 ORDER BY 1;An hour with zero inbound on a domain that normally receives is worth investigating. It is usually DNS or a firewall, not the application.
Storage#
-- Per account. Tombstoned rows still occupy the folder until the retention
-- sweep clears them, so exclude them or the numbers read high.
SELECT u.email, pg_size_pretty(sum(m.size)::bigint) AS used
FROM messages m
JOIN addresses a ON a.id = m.address_id
JOIN domains d ON d.id = a.domain_id
JOIN users u ON u.id = d.user_id
WHERE m.expunged_at IS NULL
GROUP BY 1 ORDER BY sum(m.size) DESC
LIMIT 20;Corsair emits quota.warning and quota.exceeded webhooks; subscribing to those is easier than polling.
Host disk matters separately:
df -h /var/lib/postgresqlPostgres stops accepting writes when the filesystem fills, and a mail server that cannot write is a mail server that rejects. Alert well before it happens.
Certificate expiry#
openssl s_client -connect mail.example.com:993 -servername mail.example.com </dev/null 2>/dev/null \
| openssl x509 -noout -enddateCheck the port, not the file. The file on disk being fresh proves nothing if the process is still holding the old one — which is exactly the failure mode of renewal without a reload hook.
# Days remaining, for a monitoring check
expiry=$(openssl s_client -connect mail.example.com:993 </dev/null 2>/dev/null \
| openssl x509 -noout -enddate | cut -d= -f2)
echo $(( ($(date -d "$expiry" +%s) - $(date +%s)) / 86400 ))Webhook health#
SELECT url, status, consecutive_failures, last_success_at
FROM webhooks
WHERE status <> 'active' OR consecutive_failures > 0;An endpoint that fails twenty times in a row is disabled automatically. Nothing tells you except the panel, so query for it.
Bounce rate#
-- mail_log.status is one of accepted, rejected, delivered, deferred, bounced, spam
SELECT
count(*) FILTER (WHERE status = 'bounced')::float
/ NULLIF(count(*), 0) AS bounce_rate
FROM mail_log
WHERE direction = 'outbound' AND created_at > now() - interval '24 hours';Above a few percent and receivers start treating you as a source of junk. Corsair suppresses nothing automatically, but it records every bounce — repeatedly delivering to an address that hard-bounces is one of the fastest ways to damage a sending reputation.
Logs#
journalctl -u corsair -f
journalctl -u corsair --since "1 hour ago" | grep -i error
docker compose logs -f corsair # if containerisedWorth grepping for:
| Pattern | Meaning |
|---|---|
unhandled route error |
An API bug. Should be rare; investigate each one |
job .* failed |
A worker job threw. It retries |
rejected |
Inbound refusals — no such user, over quota, filtered |
From the panel#
The instance owner can read the journal over the API without a shell. Both routes need the owner's panel session.
| Method | Path | Does |
|---|---|---|
| GET | /api/logs/sources |
The units that can be read (key, label, unit) and the accepted since windows |
| GET | /api/logs |
One read of one unit's journal |
/api/logs takes unit (a key from the sources list, default corsair; an unknown one is a 400), lines (default 300, at most 2000), since (15m, 1h, 6h, 24h, 7d or all; default 1h), q (a literal, case-insensitive search of at most 200 characters) and priority (0 to 7, entries at that syslog severity and worse). The answer is a list of entries (id, at, unit, priority, message) plus truncated and available. If the service account cannot read the journal, available is false and reason says to add it to the systemd-journal group. Output is redacted for connection strings and similar before it leaves the server.
The worker#
Four periodic jobs, guarded by a Postgres advisory lock so several workers do not all start the same sweep:
| Job | Does |
|---|---|
domain.verify |
Re-checks pending domains, every half hour |
transfer.run |
Runs mailbox migrations |
quota.recompute |
Recalculates account storage |
retention.sweep |
Expires tombstones, logs, bans, and sessions |
SELECT kind, status, count(*), max(updated_at)
FROM jobs GROUP BY 1, 2;Jobs stuck in running with an old updated_at mean a worker died mid-job.
Prometheus metrics#
Corsair serves its own counters at GET /metrics in the Prometheus text format. It is off until you set a token:
# .env
METRICS_TOKEN=$(openssl rand -hex 32)With no METRICS_TOKEN the path answers 404, as if it did not exist. With one, every request needs Authorization: Bearer <token>; a wrong or missing token is a
- There is no "trust loopback" shortcut, because behind a reverse proxy every request arrives from
127.0.0.1. Metrics live in the process that serves them and reset on restart, which is whatrate()expects.
scrape_configs:
- job_name: corsair
scheme: https
metrics_path: /metrics
authorization:
credentials_file: /etc/prometheus/corsair.token
static_configs:
- targets: ["mail.example.com"]None of the labels carry an address, a domain or an IP, so the number of series stays small however much mail you handle.
| Metric | Labels | Meaning |
|---|---|---|
corsair_smtp_sessions_total |
listener (mx, submission, submissions) |
Connections accepted |
corsair_smtp_messages_total |
listener, result, reason |
Messages finished at DATA; reason is spam, auth, policy, size, other, or none when accepted |
corsair_smtp_recipients_rejected_total |
listener, reason |
RCPT refusals |
corsair_smtp_message_size_bytes |
listener |
Histogram of message sizes |
corsair_delivery_attempts_total |
outcome (delivered, deferred, bounced) |
Outbound attempts |
corsair_delivery_deferrals_total |
reason (blocklist, tls, connect, 4xx, other) |
Why attempts were deferred |
corsair_queue_depth |
status (queued, deferred, sending) |
Deliveries waiting, read at scrape time |
corsair_queue_oldest_age_seconds |
Age of the oldest queued or deferred delivery | |
corsair_queue_overdue_age_seconds |
How long the longest-waiting due delivery has been due | |
corsair_auth_attempts_total |
protocol, result (ok, fail, banned) |
Logins |
corsair_bans_active |
IPs currently banned | |
corsair_connections_active |
protocol (imap, pop3) |
Open client connections |
corsair_commands_total |
protocol, command |
Protocol commands; unknown names are other |
corsair_webhook_deliveries_total |
outcome (delivered, failed, exhausted) |
Webhook attempts |
corsair_storage_errors_total |
op (read, write, delete) |
Object store failures |
corsair_process_*, corsair_event_loop_lag_seconds, corsair_build_info |
Uptime, memory, loop lag, version |
The queue gauges come from two small queries run when you scrape, cached for five seconds with a two-second deadline. If the database is slow they keep their last values instead of failing the scrape.
Counters are per process. On a split deployment, scrape the web tier and the process that runs the worker and listeners separately, and sum across them.
Alerts worth starting with:
groups:
- name: corsair
rules:
- alert: CorsairQueueGrowing
expr: deriv(sum(corsair_queue_depth{status!="sending"})[30m:]) > 0 and sum(corsair_queue_depth{status!="sending"}) > 50
for: 30m
- alert: CorsairOldestQueuedTooOld
expr: corsair_queue_oldest_age_seconds > 3600
for: 10m
- alert: CorsairWorkerBehind
expr: corsair_queue_overdue_age_seconds > 300
for: 5m
- alert: CorsairDeferralsBlocklist
expr: sum(increase(corsair_delivery_deferrals_total{reason="blocklist"}[15m])) > 0
- alert: CorsairDeferralRateHigh
expr: |
sum(rate(corsair_delivery_attempts_total{outcome="deferred"}[15m]))
/ sum(rate(corsair_delivery_attempts_total[15m])) > 0.3
for: 15m
- alert: CorsairAuthFailureSpike
expr: sum by (protocol) (rate(corsair_auth_attempts_total{result="fail"}[5m])) > 1
for: 10m
- alert: CorsairStorageErrors
expr: sum(increase(corsair_storage_errors_total[10m])) > 0
- alert: CorsairDown
expr: up{job="corsair"} == 0
for: 2mExternal checks#
Things a check on the host cannot see:
- Deliverability. Send to a seed address at a large provider on a schedule and assert
dkim=pass. This catches an expired DKIM record, which nothing local will. - Blocklists. Query your IP against Spamhaus and SpamCop daily. Being listed is silent until mail starts bouncing.
- Inbound reachability. Connect to port 25 from outside. A firewall rule that changed is invisible from inside.
- DNS. Assert the MX, SPF, DKIM, and DMARC records still resolve. Someone editing DNS for an unrelated reason is a common cause of sudden failure.
A minimal setup#
If you do only four things:
- Alert when the process is down.
- Alert when the oldest queued message is over an hour old.
- Alert when the certificate has under fourteen days.
- Alert when the backup heartbeat goes quiet.
Those four cover the failures that lose mail. Everything else on this page is refinement.