Measuring Delivery Latency: Synthetic End-to-End Mail Tests
Here is a scene every mail administrator eventually lives through. The dashboard is green. The relay answers on port 25, the gateway answers on port 25, and the mailbox server's services are all running. And a user is standing at your desk explaining that nothing they have sent since breakfast has arrived anywhere. Every check you have is telling the truth, and every check you have is missing the point. The doors of the post office are open. The mail is not moving.
The only check that cannot be fooled this way is the one that behaves like a user. It writes a message, sends it through the same path real mail takes, and waits to see whether — and how quickly — it arrives. That is a synthetic end-to-end mail test, and it is the single most honest thing you can add to your mail monitoring. It sees stuck queues, broken filters, dead hosted services, and slow delivery long before any of them turns a light red.
This guide shows you how to build one. It covers why port checks are not enough and how the round-trip test works in both directions. It shows how to set timing budgets that reflect your own normal, and how a rising delay warns you before total failure. It explains how to script the test with nothing but standard protocols, and the hygiene that keeps it from becoming spam or noise. And it shows how to alert both on delay and on the scarier case — the test message that never shows up at all.
In this guide
- Why "port 25 answers" is not "mail delivers"
- The round-trip test, in both directions
- Setting timing budgets per direction
- Catching slow delivery before total failure
- Scripting the test with standard protocols
- Test hygiene: staying out of the spam folder
- Alerting on delay and on silence
- Where to go next
Why "port 25 answers" is not "mail delivers"
A port check opens a TCP connection to a mail server, waits for the greeting banner, and calls it a day. It is a fine test of one narrow thing: the process that answers the door is alive. It says nothing about what happens after the door opens. Consider everything that can be wrong while port 25 answers perfectly:
- The relay accepts messages and stores them in a queue that its delivery process has stopped draining.
- The relay cannot resolve MX records, so every outbound message defers.
- The gateway accepts mail, scans it, and silently discards or quarantines it because a filter rule went wrong.
- Your sending address is on a blocklist, so the far side accepts the connection and then refuses every message.
- The hosted mail service is having an incident: connections succeed, delivery takes hours.
- The mailbox store is offline, so the gateway defers everything inward while looking healthy from outside.
This is the mail-specific version of a general truth about monitoring: a host being up is not the same as the service working. Port checks still belong in your setup, because they are cheap and they localize a failure fast. But the check that answers "is mail flowing?" has to send mail.
The round-trip test, in both directions
The idea is simple. You control two mailboxes. One is an internal test mailbox on your own mail system. The other is an external test mailbox at some other provider on a different domain, out on the internet. On a schedule, a script sends a message from one to the other and stamps the moment it was sent. It then watches the destination mailbox until the message appears and records how long it took. Run it in both directions and you have exercised the entire chain described in the guide to how mail flows.
The two directions are genuinely different tests, and it pays to keep them separate in your results:
| Leg | Path exercised | A failure or delay implicates |
|---|---|---|
| Outbound: internal mailbox → external mailbox | Submission to your relay, the relay's queue and DNS lookups, your internet link, the external provider's inbound gateway | Your relay (queue, resolver, disk), your sending reputation, or the external provider |
| Inbound: external mailbox → internal mailbox | The external provider's outbound relay, your MX records, your edge gateway or filter, your mailbox server or hosted service, final delivery | Your MX records, gateway, filter rules, mailbox store, or hosted service |
The outbound leg is easy to run from inside: your script submits through the relay exactly as a user's client would. The inbound leg needs something on the outside to originate a message. There are three common ways to arrange that. The simplest is an auto-reply: configure the external mailbox to answer every message it receives. That way, your outbound test automatically triggers an inbound one. And the script measures both halves of the round trip. The second is to have your script log in to the external mailbox's submission service and send from there. That gives you independent timing for each leg. The third, if you run a second site or a cloud host, is a tiny external sender of your own. Any of them works; the auto-reply is the quickest to set up and the least flexible.
Whichever way you originate it, the measurement is the same at both ends. The script generates a unique identifier, puts it and the send timestamp in the subject line, and sends. Then it polls the destination mailbox for a message whose subject contains that identifier. When it finds one, delivery latency is the arrival time minus the send time. Arrival time can come from the poll (coarse, limited by how often you poll). Better, it can come from the timestamp in the topmost Received: header. That records the moment final delivery happened regardless of when your script noticed.
Setting timing budgets per direction
A round-trip test produces a number every few minutes; the question is what number is bad. As with every mail metric, the honest answer starts with your own baseline. Run the test for a week or two before alerting on it, and look at the results for each leg separately. Healthy mail between two well-configured systems typically arrives within seconds to a minute or two. But your normal is shaped by things you can see in the data. These include the filter's scan time on the inbound leg, a hosted provider's own processing, and first-contact greylisting. That greylisting makes the very first messages from a new sender slow until the far side has learned to trust it.
From that baseline, set two thresholds per direction. The slow threshold sits comfortably above your normal worst case — far enough that a busy afternoon does not trip it, close enough that a real slowdown does. The missing threshold is the point at which you stop waiting and declare the message lost. It should be long enough to outlast a routine deferral and retry on the far side. It should be short enough that a real outage does not hide behind "still waiting." Only you know what your business tolerates: a company whose orders arrive by email has a much tighter budget than one whose email is mostly internal chatter. Write both numbers down, per leg, and revisit them after the first month.
Tip: keep the budgets separate for the two legs even if they start out identical. Inbound and outbound fail for different reasons and at different rates. The moment you tune one threshold for a hosted provider's slow-but-normal inbound scanning, you will be glad the outbound one did not move with it.
Catching slow delivery before total failure
The latency number earns its keep before anything actually breaks. Mail systems rarely fail all at once; they slow down first. Suppose the far side starts deferring your relay — greylisting a new address, rate-limiting a sudden burst, throttling a sender whose reputation has dipped. Your relay retries on its back-off schedule and the message still arrives, just late. Your users notice nothing yet. Your synthetic test notices immediately: a leg that normally takes twenty seconds starts taking eight minutes, then fifteen. That trend is the same story the queue age metric tells from the relay's side, seen from the receiving end. That trend often appears on the test first because the test is sent every few minutes while a queue alert waits for its threshold.
A rising inbound latency is just as telling. The external provider is delivering fine. But your gateway is taking longer and longer to hand mail inward. That usually means the gateway is under load, its queue toward the mailbox server is growing, or the mailbox store is struggling. Again: nothing is down yet, and the graph is already sloping upward. Alert on the slope — two or three consecutive tests above the slow threshold. You get a warning with time to act, instead of a critical alert with none.
One more thing the test can catch that nothing else will: mail that arrives but lands in the wrong place. If the destination mailbox has a junk or quarantine folder, poll it too. A test message that consistently shows up in junk means the receiving filter has started distrusting your sender. That is an early symptom of the reputation problems in the blocklist guide. It is a symptom your customers will experience as "your emails go to spam" long before any bounce appears.
Scripting the test with standard protocols
You do not need any special tooling to build this. Sending is SMTP; reading is IMAP (or POP, or a hosted provider's mailbox API); both are available in the standard library of most scripting languages. The sequence every implementation follows:
- Generate a unique identifier and note the current time.
- Send a small plain-text message with the identifier in the subject, through the same relay and submission port your users use, with a dedicated test account.
- Wait and poll the destination mailbox for a subject containing the identifier, at a sensible interval, until it appears or the missing threshold passes.
- Compute the latency from the send time and the arrival time.
- Report a value and a status to the monitoring system, then clean up the test message.
Here is the shape of it in a scripting language, trimmed to the essentials. It sends through the internal relay and polls the external mailbox over IMAP:
import smtplib, imaplib, time, uuid
from email.message import EmailMessage
RELAY, RELAY_PORT = "relay.example.com", 587
IMAP_HOST = "imap.provider.example"
TEST_FROM, TEST_TO = "mailtest@example.com", "mailtest-ext@provider.example"
SLOW_SEC, MISSING_SEC = 120, 900 # your budgets, from your baseline
token = uuid.uuid4().hex[:12]
sent_at = time.time()
msg = EmailMessage()
msg["From"], msg["To"] = TEST_FROM, TEST_TO
msg["Subject"] = "mailtest %s %d" % (token, int(sent_at))
msg.set_content("Synthetic delivery test. Safe to delete.")
with smtplib.SMTP(RELAY, RELAY_PORT) as s:
s.starttls()
s.login(TEST_FROM, RELAY_PASSWORD)
s.send_message(msg)
latency = None
while time.time() - sent_at < MISSING_SEC and latency is None:
time.sleep(15)
with imaplib.IMAP4_SSL(IMAP_HOST) as m:
m.login(TEST_TO, IMAP_PASSWORD)
m.select("INBOX")
ok, hits = m.search(None, 'SUBJECT', '"%s"' % token)
if hits[0]:
latency = time.time() - sent_at
m.store(hits[0].split()[0], "+FLAGS", "\\Deleted")
m.expunge()
status = "MISSING" if latency is None else ("SLOW" if latency > SLOW_SEC else "OK")
print("status=%s latency_sec=%s" % (status, "" if latency is None else int(latency)))
A few notes on that skeleton. The subject carries both the token (to find the message) and the send time (so a human reading the mailbox can see when it was sent). The poll loop deletes the message once found, which keeps the mailbox small. Latency here comes from the poll time, so its resolution is the sleep interval; parsing the top Received header would be more precise. The passwords come from somewhere safe, not the script. And the last line prints exactly the kind of labeled output a monitoring system can collect over SSH or from a scheduled task. The system can turn that into a graph and a status.
Run it from the monitoring server or a small dedicated host, on a schedule — every few minutes is plenty. If your users are spread across sites, run the outbound leg from more than one of them. A relay that works from headquarters and not from the branch is a finding in itself.
Test hygiene: staying out of the spam folder
A synthetic mail test is a small robot that sends email all day, every day, forever. Treat it with the respect that deserves, or it will become the problem it was meant to detect.
- Dedicated accounts, on both ends. Never test through a real person's mailbox. Create test mailboxes whose only purpose is this, with strong credentials that grant nothing beyond sending and reading their own mail. Store those credentials outside the script and rotate them like any other service credential.
- Unique identifiers, always. A subject like "test" will eventually match an old message, a human's message, or a reply, and your latency figures become fiction. A random token per message makes every match unambiguous.
- Clean up. Delete each test message after measuring it, and add a periodic sweep that removes anything older than a day in case a run crashed mid-way. A test mailbox holding tens of thousands of messages makes every IMAP search slow, and eventually the slow search is what trips your alert.
- Do not look like spam. Plain text, no links, no attachments, a consistent sender address, and a modest volume. Filters learn; a sender that emits the same tidy message every five minutes for months builds a good reputation on its own. If a filter still quarantines the test, you can allowlist the test sender. But notice the trade-off: an allowlisted test proves the path works and stops proving the filter treats ordinary mail sensibly. Some teams run both an allowlisted and a non-allowlisted test for exactly that reason.
- Keep the volume polite. One message every few minutes is unremarkable. One every ten seconds looks like an attack to the external provider and can get your relay throttled. That would be a memorable way to cause the incident you were monitoring for.
- Watch for loops. If you use an auto-reply on the external mailbox, make sure the internal test mailbox does not auto-reply back. Two auto-responders facing each other will happily exchange mail until one of them fills up.
Alerting on delay and on silence
The test produces three distinct outcomes, and each deserves its own alert rule rather than one lump "mail test failed" condition. Separating them turns the alert into a diagnosis.
| Outcome | What it means | Suggested handling |
|---|---|---|
| Send failed | The script could not even submit the message: relay down, authentication broken, submission port blocked | Alert on the relay, not on delivery; this is a hop-one problem and usually urgent |
| Delivered slow | The message arrived, but later than the slow budget | Warning after two or three consecutive slow results; someone looks within the hour |
| Missing | The message was accepted for delivery and never arrived within the missing budget | Critical after two consecutive misses; this is the stuck-mail incident in progress |
The missing case is the important one, and it needs a specific mental model. A synthetic message that never arrives is not an error the system reports; it is an absence. If your monitoring only reacts to results it receives, a test whose messages vanish — or whose script silently stopped running — produces no result and therefore no alert. The fix is the dead-man's-switch pattern. The monitoring system expects a successful test result within a certain window and raises the alarm when the window passes with nothing. That is the same design the heartbeats and dead-man switches guide uses to catch a silent monitoring server. Silence becomes the signal. Whether the message got lost or the script died, you hear about it.
Two refinements keep the alerts trustworthy. First, confirmation: require two consecutive misses or slow results before paging. A single slow delivery is normal life on the internet, and greylisting alone can produce one. Second, dependencies: make the mail test alerts depend on your internet link and on the relay's basic health. That way, when the link goes down you receive one alert about the link rather than a cascade of mail-test failures on top of it. Route the result to whoever owns mail. And — the same rule as every mail alert — deliver it by a channel that does not depend on the mail system under test.
Where to go next
A port check proves the door is open; a synthetic round-trip test proves the mail moves. Build one in each direction with dedicated mailboxes, unique tokens, and standard SMTP and IMAP. Baseline it for a couple of weeks and set separate slow and missing budgets per leg. Watch the trend to catch slowdowns while they are still only slowdowns. Keep the test tidy so it never becomes spam or noise. And alert on silence with a dead-man's-switch rule so a vanished message is as loud as a failed one. Add that to the queue metrics from the previous guide. Then you cover both what you can see from inside your relays and what you can only see from the far end.
The remaining guides under Email Flow Monitoring cover the infrastructure the test rides on — MX records, relays, gateways, TLS, and connectors. They also cover the reputation problems that make a perfectly healthy path deliver nothing at all.
Put this into practice with SysPeep Monitor
A script like the one in this guide needs a home that runs it on a schedule and keeps every result. That home needs to know what to do when the results stop coming. SysPeep Monitor can collect the script's output as a custom metric and graph delivery latency next to the relay's queue and disk. It can apply thresholds and dependencies so a down internet link does not bury you in mail-test alerts. It can notify by email and other channels. The free trial installs on any Windows machine in minutes.
Download Free TrialLearn About SysPeep