Relays, MX Records, and Gateways: Monitoring the Mail Path
Between a user's Send button and a recipient's inbox sits a short chain of infrastructure. It includes a couple of DNS records, one or two relays, an edge gateway or filter, and — increasingly — a connector into a hosted mail service. None of it is exotic. All of it can break in quiet ways. There might be a record that points at a server you retired, or a certificate that expired on a port nobody browses to. There might be a subscription that lapsed on an appliance that kept passing traffic, or a connector password that rotated on the other side. Each one produces the same ticket: "email is weird."
The queue guide and the synthetic test guide tell you that mail has stopped moving. This guide is about the checks that tell you why — and, more often, that catch the underlying rot weeks before mail stops at all. It walks the path component by component: MX records, SMTP ports and banners, and TLS on the mail path. It covers the signs that your relay is being abused, gateway and filter appliance health, and hosted-mail connectors. It ends with a checklist table you can turn into monitoring configuration and a short section on getting the resulting alerts to the right person.
In this guide
- MX records: resolve, point where expected, sane priorities
- SMTP banner and port checks on every relay and gateway
- TLS on the mail path
- Signs your relay is being abused
- Gateway and filter appliance health
- Hosted-mail connectors and their credentials
- The mail-path checklist
- Routing the alerts
- Where to go next
MX records: resolve, point where expected, sane priorities
An MX record (mail exchanger) is the DNS entry that tells the rest of the world which servers accept mail for your domain. Each record names a host and carries a preference number; sending relays try the lowest number first and fall back to the others. Your MX records are the front door of your mail system, and they have a nasty property. They can be wrong for a long time without anyone inside noticing, because internal mail never uses them. Only outsiders do. Ask DNS what the world sees, with any lookup tool:
$ dig +short MX example.net 10 gw1.example.net. 20 gw2.example.net. $ dig +short A gw1.example.net 203.0.113.10
A monitoring check for MX records asks three questions on a schedule, ideally from a resolver outside your network. That way, you see what senders see rather than an internal split-horizon view:
- Do the records resolve at all? A domain whose MX lookup fails or times out gets no mail; a relay that cannot find your MX will defer and eventually bounce. This is the check that catches an expired domain registration, a broken DNS provider, or a zone someone saved with a typo.
- Do they point where you expect? Compare the returned hostnames against a list you maintain: your gateway pair, or the hosted service's published names. A stale record may still point at the gateway you decommissioned last quarter. That means some fraction of inbound mail is being tried against a dead host first, arriving late every single time. A record you did not add at all is a security incident.
- Are the priorities sane and do the hosts exist? The lowest number should be your real primary; every listed host should resolve to an address and answer on port 25. A backup MX that nobody has tested in years is a common way to discover that "backup" meant "accepts mail and quietly loses it."
Alongside the MX records, check the reverse DNS (PTR record) for each of your sending IP addresses. It should resolve to a name that itself resolves back to that address. That name should be what your relay announces in its greeting. Receiving gateways treat a mismatch as a spam signal. All of this is DNS, which means all of it fails the way DNS fails. It fails silently and with symptoms that look like something else, as the why-is-it-always-DNS guide explains at length.
The expected-host list also makes migrations safer. When you move to a hosted service or replace a gateway, update the list at the same moment you change the records. Then let the external check confirm that the new answer is what the world actually receives once the old record's time-to-live has run out. Many "the new mail system is not getting mail" tickets are simply the old MX record still being served from caches for a few more hours. And a check that shows the transition happening is far more reassuring than a resolver on your desk.
SMTP banner and port checks on every relay and gateway
Every relay and gateway on your path listens on one or more of three ports, and each deserves a check from the vantage point that matters for it:
| Port | Role | Check it from |
|---|---|---|
| 25 | Server-to-server SMTP; the port MX records point at | Outside your network for the gateway; inside for internal relays that receive from other servers |
| 587 | Submission: clients and devices send with authentication, usually upgrading to TLS | Wherever your users and devices are — the office network, the VPN, each branch |
| 465 | Submission over implicit TLS (encrypted from the first byte) | Same as 587, if you offer it |
The minimum check is a TCP connect followed by reading the banner. That is the first line the server sends, which should begin with the code 220 and name the server. A better check goes one step further and sends EHLO, the greeting that makes the server list its capabilities. Then it verifies the ones you depend on are present. Doing it by hand looks like this:
$ openssl s_client -starttls smtp -connect gw1.example.net:25 -quiet 250 gw1.example.net EHLO monitor.example.com 250-gw1.example.net 250-SIZE 52428800 250-STARTTLS 250-AUTH PLAIN LOGIN 250 8BITMIME QUIT 221 2.0.0 Bye
Three things in that exchange are worth turning into monitored conditions. The banner arrived at all, and arrived quickly. A banner that takes many seconds usually means the server is overloaded or is stalling on a reverse-DNS lookup of your address. Slow banners are an early sign of a gateway in trouble. The STARTTLS capability is advertised, so encryption is on offer. And the AUTH capability appears only where it should: on the submission port for your users. It should not appear on the public port 25 of a gateway that has no business accepting logins from the internet. Any decent monitoring system can run a scripted SMTP conversation like this on a schedule and alert when the response changes.
Vantage point matters more for mail than for most services. A check of the gateway's port 25 run from inside your network tests your firewall's ability to loop traffic back to itself. It does not test the path the internet uses. Run that one from a remote probe, a branch office, or a small external host. Conversely, the submission port is only interesting from where clients sit. So a check from the office VLAN, the VPN address pool, and each branch tells you what users will experience there.
TLS on the mail path
Encryption between mail servers is mostly opportunistic. When one server connects to another on port 25, it asks whether the other side supports STARTTLS. If yes, they encrypt the session; if no, they carry on in plain text. Historically, most sending servers do not even validate the certificate they are offered — any certificate, expired or self-signed, is accepted rather than fail delivery. That leniency is why certificate problems on the mail path are so easy to miss. Nothing bounces, no user complains, and your mail simply travels less protected than you think.
Two things change that picture. First, the submission ports are different: mail clients and devices connecting on 587 or 465 generally do validate the certificate. So an expired certificate on your relay's submission port stops every client and every scanner from sending at once, on the morning it expires. Second, a growing number of receiving domains publish policies that require verified TLS from senders. Senders that honor those policies will defer or refuse delivery to a gateway whose certificate is expired or whose name does not match the MX hostname. Either way, the certificate on your mail ports needs the same watching as the one on your website. The certificate expiry monitoring guide covers how to do that for any port, mail included.
The practical checks apply to each relay and gateway, on each mail port it serves. Verify that STARTTLS (or implicit TLS on 465) negotiates successfully. Verify that the certificate's expiry is comfortably far off and that its chain is complete. Verify that its subject name matches the hostname in your MX records and in your clients' settings. Alert weeks ahead on expiry, and alert immediately if STARTTLS stops being offered, because a configuration change that drops encryption will not announce itself any other way.
Signs your relay is being abused
A relay exists to forward mail on behalf of others, and that makes it a target. Spammers hunt for relays that will forward mail for anyone, and attackers hunt for accounts on relays that will forward mail for authenticated users. Both end the same way — your IP address on a blocklist and your legitimate mail refused everywhere, the subject of the reputation guide. The monitoring goal is to notice abuse in minutes rather than when the bounces begin.
The open relay test
An open relay accepts mail from an outside address to an outside address without authentication. Test your own gateway from outside your network by pretending to be a stranger sending to a stranger; a correctly configured server refuses at the recipient step:
$ openssl s_client -starttls smtp -connect gw1.example.net:25 -quiet EHLO test.example.org MAIL FROM:<stranger@example.org> 250 2.1.0 Ok RCPT TO:<victim@example.com> 554 5.7.1 <victim@example.com>: Relay access denied QUIT
That 554 Relay access denied is the answer you want. A 250 at the recipient step means you are an open relay. And the sensible next step is to fix it before finishing your coffee. Run this test on a schedule from an external vantage point, because open relays are usually created by accident. That might be a "temporary" allow rule for an application server, or an appliance restored from an old backup. The accident goes unnoticed until the blocklists notice it for you.
Authentication failures and volume
The submission port is a login prompt exposed to the network, and attackers treat it as one. A spike in failed authentication attempts on port 587 — many usernames from one address, or one username from many addresses — is password guessing. And a log-based alert rule on the relay's authentication failures is the cheapest early warning you can build. When an attacker succeeds, the second signal is volume. That might be a single account that sends more mail in an hour than it usually sends in a month. It might be outbound totals far above your baseline, a queue that climbs vertically, or thousands of recipients at domains your organization has never written to. Track outbound messages per hour, both in total and for the top few senders, and alert when a sender's rate leaves its normal range.
A subtler sign is a flood of bounces you did not earn. When spammers forge your domain as the sender of their mail, the rejections come back to you — a phenomenon called backscatter. A sudden pile of non-delivery reports for messages nobody here sent means your domain is being spoofed somewhere. That is a reputation problem rather than a relay problem, but the relay's inbound counters are where you will see it first.
Gateway and filter appliance health
Edge gateways and spam filters — whether a physical appliance, a virtual machine, or a cluster of them — are mail servers with extra opinions. They need the mail-server checks plus a few of their own. Most publish these over SNMP through a vendor MIB, through a management API, or as forwarded syslog. A monitoring system can collect them like any other device metric.
- Queues, inbound and outbound. A gateway holds mail in both directions: inbound toward your mailbox server, outbound toward the internet. Both queues get the depth-and-age treatment from the queue guide, because a filter whose inbound queue is growing is telling you the mailbox server behind it has stopped accepting.
- CPU, memory, and disk. Scanning is expensive. An appliance at sustained high CPU scans slowly, and slow scanning shows up as rising inbound latency in your synthetic tests. Disk holds the queue, the quarantine, and the logs; when it fills, all three suffer.
- Subscription and update state. Filtering depends on subscriptions for signatures, reputation data, and rules. A lapsed subscription can mean anything from "no new signatures" to "the appliance stops filtering," depending on the product. Monitor whatever the appliance exposes about license or subscription validity and the age of its last update, and alert well ahead of the renewal date.
- Cluster and failover state. If you run a pair, verify that both members are healthy and that mail actually flows through the secondary. The second MX record that "should" work is the one to test on purpose.
Hosted-mail connectors and their credentials
If your mailboxes live at a hosted mail service, the mail path still has pieces you own, and they are the pieces most likely to be forgotten. A connector is a configured route between your environment and the hosted service. An outbound one lets your local relay hand mail to the service, so scanners, applications, and alarm panels can send through it. Sometimes there is an inbound one that routes the service's mail through your own gateway or to on-premises systems. Each connector authenticates somehow — a username and password, a certificate, or a list of permitted source addresses. Each of those can expire, rotate, or change on the other side without anyone telling you.
Monitor connectors the way you monitor any hop. First, the local relay that feeds the connector gets queue depth and age. A growing deferred queue whose reasons mention authentication or policy is the classic signature of a connector whose credentials just broke. Second, a synthetic message from a device-style sender through the connector to a real mailbox, on a schedule, proves the whole route including authentication. Third, the certificate a connector presents, if it uses one, joins the certificate inventory with an expiry alert. And fourth, when the service publishes health information for your tenant, collect it, but do not rely on it alone. Your own synthetic test will usually notice a problem before a status page admits to one.
Tip: keep a plain list of every device and application that sends mail. Record which relay or connector it uses, what credential it authenticates with, and when that credential expires. The scanner that stops delivering scans because its mail password expired is a help-desk classic. And the list turns it from a mystery into a calendar reminder.
The mail-path checklist
Pulling the sections together, here is the path as a monitoring checklist. Frequencies are suggestions; adjust to how quickly your business needs to know.
| Check | Run from | How often | Alert when |
|---|---|---|---|
| MX records resolve and match the expected host list | An external resolver | Every few minutes | Lookup fails, hosts differ from the list, or a listed host does not answer on 25 |
| Reverse DNS for sending IPs matches the relay's announced name | External | Hourly | Mismatch or missing PTR |
| Banner and EHLO on port 25 of each gateway | External | Every minute or two | No 220 banner, slow banner, STARTTLS missing, AUTH offered on 25 |
| Banner and EHLO on 587/465 of each relay | Each user network and branch | Every minute or two | No banner, slow banner, AUTH or STARTTLS missing |
| Certificate on every mail port | Where clients or senders connect from | Daily | Expiry within your warning window, chain broken, name mismatch |
| Open relay test | External | Daily | Recipient accepted without authentication |
| Authentication failures on submission | Relay logs | Continuous | Failure rate leaves its normal range |
| Outbound volume, total and per sender | Relay logs or counters | Hourly | Rate far above baseline, or a new top sender |
| Gateway queues, CPU, memory, disk, subscription state | SNMP or API | Every few minutes | Queue age above drain time, resources sustained high, subscription near lapse |
| Connector synthetic message and credential expiry | Internal sender through the connector | Every few minutes; expiry daily | Message missing or slow; credential within its warning window |
Routing the alerts
Ten checks on a handful of hosts can generate a lot of noise if every one pages the same person at the same severity. So spend a few minutes on routing. Severity should follow user impact. A submission port down, an open relay, or a connector authentication failure interrupts someone now. A certificate three weeks from expiry, a stale backup MX, or a subscription nearing renewal becomes a ticket for the week. Use dependencies so a gateway that is entirely down produces one alert rather than one per check. Also use them so an internet outage does not fire every external check at once. Send mail-path alerts to whoever owns mail, but never by mail alone. The alert that says "the relay is down" has to travel by a channel the relay is not part of.
Where to go next
The mail path is a short list of ordinary components. And each one has an ordinary check. The checks cover MX records that resolve to the hosts you expect and banners that answer promptly on the right ports with the right capabilities. They cover TLS that negotiates with a valid certificate, and a relay that refuses strangers and notices guessed passwords and runaway senders. They cover an appliance whose queues, resources, and subscription are watched, and connectors whose credentials are on a calendar. Run the checks from the vantage point that matters — outside for the gateway, inside for the relay — and route the results by impact.
With the path healthy, the remaining way for mail to go nowhere is reputation. A blocklisted address or a broken authentication record makes a perfect path deliver nothing. That is the last guide in Email Flow Monitoring, and it is the one to read before the bounces start rather than after.
Put this into practice with SysPeep Monitor
Relays, gateways, and filter appliances are exactly the kind of devices SysPeep Monitor was built to watch. It discovers them and collects their queues, CPU, memory, and disk over SNMP, WMI, and SSH. It runs port and service checks on the mail ports and uses thresholds and dependencies to turn ten checks into one meaningful alert. The free trial installs on any Windows machine, and remote probes let you run checks from the branch or the outside.
Download Free TrialLearn About SysPeep