Early-Warning Metrics Worth a Weekly Look
Some metrics tell you a thing has broken. Others tell you a thing is going to break, and they tell you weeks in advance, quietly, in a counter nobody has put on a dashboard. The second kind is worth more. A warning with six weeks of notice can be fixed on a Tuesday afternoon with a coffee. A warning with six seconds of notice is called an outage.
This guide is a catalogue of the second kind. Each entry says what the metric is, why it creeps rather than jumps, what it is trying to tell you, and where to find it. None of these numbers needs an alert in the ordinary sense. What each needs is a look once a week, on one page, by someone who remembers what it looked like last week. That is a smaller commitment than it sounds. And the last sections show you how to build the page and what to check on it.
If you have read the guide on what degradation creep looks like, you already know why a threshold cannot see a slope. This is the list of slopes worth watching.
In this guide
- What makes a metric an early warning
- On the wire: errors, retransmits, and drops
- Response times: the 95th percentile and DNS
- Inside the boxes: memory, temperature, and power
- Counts that run out: leases and certificates
- The monitor watching itself
- A level to alert on, a slope to review
- Building the weekly-look page
- The weekly look checklist
- Six weeks of retransmits
- Where to go next
What makes a metric an early warning
An early-warning metric is one that starts moving before the thing it measures fails, and keeps moving in the same direction until it does. A disk that is 60 percent full is not a warning; a disk that was 50 percent full last month is. The information is in the change, which means the metric only becomes useful when you keep its history and compare across weeks. A number you look at once tells you a level. A number you look at every week tells you a story.
Three things make a metric a good candidate. It should move slowly, so that a weekly look is often enough. It should move for a reason you can act on, so that the look leads somewhere. And it should be cheap to collect, because you are going to collect it for a year. Most of the entries below come out of SNMP or the operating system on a server. SNMP is the standard question-and-answer protocol for asking devices about themselves. And most of these metrics are already in your monitoring database. The work is in putting them on a page, not in collecting them.
On the wire: errors, retransmits, and drops
Interface errors and CRCs
An interface error is a frame the switch received but could not use, and a CRC error is the commonest kind. The frame's checksum did not match its contents, so something changed the bits on the way in. Errors creep because their causes are physical and physical things degrade: a cable ages, a connector oxidises, a transceiver's laser weakens. They are trying to tell you that a link is going to fail. Until it does, every corrupted frame is being resent, which users feel as slowness they cannot describe. See them on the interface graph of every uplink and server port, as a rate per day rather than a running total. The guide on interface metrics that matter covers the counters in depth. For the weekly look, the question is only whether the daily count is higher than last week's.
Retransmits and duplicate ACKs
A TCP retransmit is a chunk of data sent again because the sender never heard back that it arrived. A duplicate ACK is the receiver saying "I am still waiting for the piece before this one." Three of them in a row are what usually trigger the retransmit. Both mean loss or reordering somewhere on the path between two machines. Both creep when a path is getting worse. Think of a link filling, a flaky cable, a firewall struggling, or a storage network with a failing port. Those counters are trying to tell you that the network between two specific hosts is degrading, which interface errors on a single switch cannot. See them in the operating system's TCP statistics on your busiest servers, especially anything that talks to storage over the network.
# Windows: retransmitted segments since boot (compare week to week) netstat -s | findstr /i "retransmitted" # Linux: the same counter nstat -az TcpRetransSegs
Queue drops and discards
A discard is a frame that was perfectly good and the device threw away anyway, usually because the queue it needed to sit in was full. Discards creep as a link approaches saturation in short bursts that the five-minute average never shows. A port can read 40 percent utilised on the graph and still drop frames every few seconds. The 40 percent is an average over five minutes, and the drops happen in five-millisecond bursts. Discards are trying to tell you that a link is full in the moments that matter, well before the utilisation graph agrees. See them on the interface counters next to the errors, and read them as a daily rate.
Response times: the 95th percentile and DNS
95th-percentile response time
A percentile is a value that a given share of samples fall below. The 95th percentile of response time is the number that 95 percent of checks came in under. That makes it the experience of your unluckiest one-in-twenty user. Averages hide creep in response time because most requests stay fast while a growing minority get slow. The 95th percentile shows that growing minority directly. It is trying to tell you that the bad moments are getting worse before the typical moment does. See it on any check that records a response time, if your tool can graph a percentile; the guide on percentile thresholds explains the mechanics.
DNS response time
A DNS lookup turns a name into an address, and every connection your users make starts with one. DNS response time creeps as a resolver gets busier, a forwarder path degrades, or a server accumulates zones and clients. It is trying to tell you that everything is about to feel slightly slow, everywhere, for no reason anyone can find. DNS delay is added to the front of every request. See it on a check that resolves a real name against each internal DNS server and records how long the answer took. The guide on expected latency baselines gives healthy ranges for round trips generally. For a quick manual reading, time a lookup by hand.
# PowerShell: how long one lookup takes against a specific server
Measure-Command { Resolve-DnsName www.example.com -Server 10.0.0.5 } | Select-Object TotalMilliseconds
Inside the boxes: memory, temperature, and power
Memory and swap
Swap is disk space the operating system uses as pretend memory when the real memory runs out. It is slow enough that a server using it in earnest feels broken. On servers, watch swap used; on network devices, which mostly have no swap, watch free memory. Both creep for the same reason: a process that takes memory and does not give it back. That could be a leak or a legitimate cache that grows with the data. They are trying to tell you that the device will run out, and the slope tells you when. Watch the monitoring server itself here too, because it has the largest database and the most connections of anything on the network. See them on the device health graph, as the daily minimum of free memory rather than the average.
Temperature and fan speed
Temperature creeps for two reasons and fan speed for one. Room temperature climbs as equipment is added to a closet that was cooled for half as much. Board temperature climbs as the fans that cool it slow down with age. Fan speed falls as bearings wear. Together they are trying to tell you that a device will overheat, and the fan is usually the first to speak. See both in the sensor section of any managed switch, firewall, or server, and on the environmental probes the temperature and humidity guide describes. Compare the same sensor against itself a month ago, not against another device; every model runs at its own normal.
PoE budget
Power over Ethernet, or PoE, is a switch supplying electricity to phones, access points, and cameras through the network cable. Every switch has a fixed power budget to share among them. The budget creeps towards full because devices get added one at a time and nobody adds up the watts. It is trying to tell you that the next phone plugged in will either not power up or will knock a camera off. See it on the switch's PoE status, as watts allocated against watts available, and the guide on PoE budgets covers the details.
Counts that run out: leases and certificates
DHCP lease count
A DHCP lease is an address handed to a device for a fixed time, and a scope is the pool those addresses come from. Lease counts creep because devices multiply: phones, laptops, printers, sensors, the visitor network. A scope is trying to tell you that at some rate of growth it will be empty. On that day new devices will simply not join the network while everything else looks fine. See the count per scope on your DHCP server, as a daily maximum against the scope size. The guide on scope exhaustion covers what happens when the pool runs dry.
Certificates approaching expiry
A TLS certificate is the file behind the padlock in a browser, and each one has a fixed expiry date. The creeping number is not any single certificate's days remaining; your monitoring should already alert on that. It is the count of certificates that will expire within the next sixty days. That count creeps because certificates get issued for internal tools, test systems, and appliances, and each one is forgotten the moment it works. It is trying to tell you how much renewal work is coming and whether it is arriving in a lump. Keep a list, check each certificate's end date, and count. One line reads the date off any service.
# expiry date of the certificate a service presents openssl s_client -connect mail.example.com:443 -servername mail.example.com </dev/null 2>/dev/null | openssl x509 -noout -enddate
The monitor watching itself
Database size and poll cycle duration
The monitoring database grows with every device, every counter, and every day of retention. The poll cycle grows with it. That is the time it takes the collector to ask every device every question once. Both creep because you keep adding devices and nobody removes counters. They are trying to tell you that the poll cycle is approaching the polling interval. When it gets there the collector starts skipping samples and the graphs grow gaps. See the database size on the monitoring server's disk graph and the cycle duration in the tool's own statistics, if it exposes them. The monitoring the monitor pillar covers keeping the collector healthy.
Alert count per week
The number of alerts your system sent last week is a metric, and it creeps upwards as thresholds go stale, devices are added, and nobody tunes anything. It is trying to tell you that the people receiving the alerts are learning to ignore them. The alert fatigue pillar covers that as a risk in its own right. See it in the alert history, counted per week, and split by which rule fired. A rising count with the same three rules at the top is a tuning job, not a network problem. (The rules will not tune themselves. I have waited.)
A level to alert on, a slope to review
Every metric above has two numbers in it, and it helps to keep them apart. The level is where the metric is today. Some levels deserve an alert: a scope at 95 percent, a certificate inside fourteen days, a temperature above the vendor's limit. The slope is how the level has moved over the past month, and a slope deserves a look, not a page. Nobody needs to be woken because CRC errors rose from twelve a day to thirty. Somebody needs to notice by Friday.
The practical split is this. Put a static threshold on the level where a hard limit exists and crossing it is an emergency. Put the slope on the weekly-look page and review it with a human. The guide on alerting on trends adds a middle option: a rule that fires when the slope itself gets steep. That is for the handful of metrics where a week is too long to wait. Most of the catalogue does not need it. A weekly look with a person attached catches nearly everything on this page.
Building the weekly-look page
The weekly-look page is one dashboard, or one scheduled report, that shows every metric in this catalogue over the last ninety days, with nothing else on it. One page, because the habit dies the moment it needs four clicks. Ninety days, because that is long enough to see a slope and short enough to see this month's change against last month's. The dashboard design pillar covers layout properly; the rules that matter here are simpler.
Give every panel the same time range and a vertical axis that starts at zero. That way, a rising line looks like a rising line and not like an auto-scaled wobble. Put the daily maximum on the panel, not the average, for anything measured in errors or milliseconds, and the daily minimum for free memory. Label each panel with the metric, the device, and the direction that is bad. That way, a colleague covering for you can read it without asking. If your tool can email the page as a report on Monday morning, do that too. A page that arrives in an inbox gets opened on weeks when nobody would have gone looking for it.
Tip: keep last week's version of the page. A saved PDF, a screenshot, a report in a folder. The weekly look is a comparison, and memory is a poor second copy. Two pages side by side make a rise obvious that one page and a vague feeling would not.
The weekly look checklist
This is the checklist to run against the page. Copy it, put it beside the dashboard, and tick it every week. Each line takes under a minute if the graphs are on one page.
- Interface errors and CRCs: is any uplink's daily error count higher than last week's? Note the port.
- Retransmits: on each storage and database server, is the weekly count of retransmitted segments rising?
- Discards: does any port show discards on a day when its utilisation looked low?
- 95th-percentile response time: is any service's weekly 95th percentile above last week's by more than its usual wobble?
- DNS response time: is any resolver's daily maximum higher than a month ago?
- Memory and swap: is any server's swap use, or any device's daily minimum free memory, trending the wrong way?
- Temperature and fans: is any sensor higher, or any fan slower, than the same sensor a month ago?
- PoE budget: is any switch's allocated power within ten percent of its budget?
- DHCP leases: is any scope's daily maximum lease count closer to the scope size than last week?
- Certificates: how many expire within sixty days, and is that count higher than last week?
- Monitoring database size and poll cycle: is the cycle duration closer to the polling interval than a month ago?
- Alert count: how many alerts fired last week, which three rules fired most, and is the total rising?
- For every "yes" above, write one line in the findings log: metric, what changed, owner, review date.
The last line is the one that turns a look into a result. The guide on the weekly trend review gives you the findings log template and a thirty-minute agenda built around exactly this checklist.
Six weeks of retransmits
Bellweather Logistics ran its database servers against a storage array over a dedicated storage network. The TCP retransmit count on those servers began rising in the second week of a quiet quarter. It rose every week for six weeks, from a few hundred a day to several tens of thousands. The graph that showed it lived on a dashboard that had been built during the storage project and not opened since. In the seventh week the failing port on one of the array's controllers gave up entirely. The array failed over to its other controller. The pause was long enough to send three databases into recovery during the morning dispatch run. The post-incident review found the retransmit graph, set it to ninety days, and passed a laptop around the table in silence. The dashboard was later confirmed to have been working perfectly the whole time.
Nothing in that story needed a new tool or a new metric. The counter was collected, graphed, and stored. It needed one person to look at it once a week and notice that this week's number was bigger than last week's, six times in a row. That is the entire job, and it fits in a Monday morning.
Where to go next
Twelve metrics, one page, one look a week. Start with the four you can put on a page this afternoon, usually interface errors, free memory, disk, and alert count. And add the rest as you find where each one lives in your tool. Once the page exists, the guide on reading growth curves tells you what the shapes on it mean. The guide to alerting on trends covers the few metrics that deserve a rule as well as a look. The full Trend Analysis and Degradation Creep pillar is here, and the capacity planning from history pillar picks up the metrics that end in a purchase.
Put this into practice with SysPeep Monitor
Every metric in this catalogue comes out of SNMP, WMI or SSH. The weekly-look page is only useful if the history is kept and the graphs sit on one screen. A central Windows service collects all of them and keeps the long-term graphs. It gives you thresholds and alert rules, dashboards, and scheduled reports from one web console. That makes the Monday look a habit rather than a project. The free trial installs in minutes.
Download Free TrialLearn About SysPeep