Long-Term Trending: Letting the Graphs Catch What You Can't
You cannot watch a network for three weeks. You have tickets, meetings, and a need for sleep. A monitoring system has none of those weaknesses. It measures the same things, the same way, every minute of every day, and it never gets bored. Trending means collecting measurements on a schedule and keeping them for a long time. It turns that tirelessness into a searchlight for intermittent problems.
No single measurement matters. What matters is what weeks of them show when you step back. You see the spike that lands every day at the same minute and the slowdown that only visits on certain weekdays. You also see the mess that follows every long weekend. I have watched a fault that looked random for a month turn out to keep better time than the office clock. As the first guide in this topic argues, intermittent almost never means random, and trend graphs are where the hidden schedule first shows its face.
This guide covers the three decisions that make trending work for ghost-hunting. Those are what resolution to collect at, how long to keep the data, and which patterns to hunt. Then it walks through a pattern hunt, graph by graph, so you can see the method in action.
In this guide
- What trending actually is
- Resolution: matching the shutter speed to the event
- Retention: how long is long enough
- The three calendars: patterns worth hunting
- Reading the graph like a hunter
- An annotated pattern hunt
- Trend data you collect yourself
- Your trend-hunt setup checklist
- Where to go next
What trending actually is
A trend is just a series of measurements of one thing, taken on a fixed schedule and stored with timestamps. Engineers call that a time series. Interface traffic sampled every minute. Disk latency every five minutes. Temperature every ten. Each sample is boring on its own. The value comes from the accumulation, the same way one frame of film is a photo but thousands are a movie.
Trending differs from live troubleshooting in one crucial way. It is set up before you need it, and it runs whether or not anything seems wrong. That is exactly what an intermittent fault cannot dodge. The observer effect works on humans — the fault "hides" when you show up because your showing up changes conditions. It does not work on a poller that asks the same question every sixty seconds forever. The fault cannot tell the difference between being watched and not being watched, because it is always being watched, identically.
Which things deserve a trend? For ghost-hunting, think in three rings. The inner ring is the symptom itself, measured as directly as you can — response time of the slow application, packet loss on the flaky path. The middle ring is the suspects: every interface, link, CPU, disk, and service that the symptom's traffic touches, plus their error counters. The outer ring is the environment: temperature, power events, and the overall load of the segment. Faults love to hide one ring further out than anyone thought to look, so be generous — storage is cheap compared to another month of mystery.
There is a hard catch, though: trend data cannot be collected retroactively. The time to have started trending the suspect switch was a month before the trouble started. Failing that, today. That is why the setup checklist at the end of this guide is worth running now, before your next ghost. It is also why building baselines of normal behavior is a habit that pays for itself. Every week of history you have when trouble starts is a week you will not have to wait for.
Resolution: matching the shutter speed to the event
Resolution (or granularity) is how often you sample — the polling interval. It is the shutter speed of your camera, and it decides which events can appear in your data at all. The rule is blunt: an event much shorter than your polling interval will usually leave no trace. A thirty-second outage is one-tenth of a five-minute polling cycle. Most cycles, both polls land on healthy moments and the outage falls through the gap as if it never happened.
Averaging makes it worse. Most trend metrics are stored as the average over each interval. So a ten-second burst that pins a link at full capacity becomes a barely visible ripple in a five-minute average. (The users felt the burst. The graph felt the ripple.) The spike is not hidden by bad luck; it is hidden by arithmetic. Short, violent events need either a short interval or a metric that preserves peaks — ideally both. The mechanics of these blind spots get a full treatment in what your polling interval cannot see. Here we only need the practical consequences:
- Estimate your event's duration from user reports ("it froze for about half a minute") and pick an interval several times shorter. For a thirty-second symptom, poll every five to ten seconds on the few metrics that matter most.
- You do not need heroic resolution everywhere. Run fast polling on the suspect scope — the handful of interfaces, servers, and services near the symptom. Leave the rest of the network at its normal rhythm.
- Prefer counters to gauges where you can. A device's error counter only goes up, so even a slow poll catches that errors happened between polls. You lose the exact moment, not the fact. A gauge like CPU percentage forgets its own past instantly.
- Fast polling has a cost: more load on the device, more data to store. It is a hunting configuration, not a permanent one. Note what you changed so you can relax it after the catch.
For pattern-hunting over weeks, ordinary one-to-five-minute polling is usually enough. That is because you are looking for when trouble clusters, not for a perfect recording of each event. Crank the resolution on the suspects; hunt the calendar with everything else.
Retention: how long is long enough
Retention is how long the data is kept, and it decides which patterns you are even allowed to discover. You cannot see a weekly rhythm in five days of data, and you cannot see a month-end collision in three weeks of it. A useful rule of thumb: to trust a cycle, you need to see at least three repeats of it. One occurrence is an event, two is a coincidence, three on schedule is a pattern worth betting on.
| Pattern you suspect | History you need | Resolution that will show it |
|---|---|---|
| Time-of-day (daily rhythm) | Two to three weeks | One to five minutes |
| Day-of-week rhythm | Four to six weeks | Five minutes is fine |
| Month-end / calendar jobs | Three to four months | Hourly rollups — if peaks are preserved |
| Seasonal or thermal | A full year | Daily rollups with max values kept |
Long retention collides with disk space, and monitoring systems resolve the collision with rollups (also called downsampling). After some age, many raw samples are merged into one summary value per hour or per day. Rollups are sensible — and they contain a trap that matters enormously for this hunt. If the rollup keeps only the average, it quietly erases exactly the spikes you are hunting. A savage two-minute latency spike, averaged into an hour, vanishes. Months later, the graph will swear the quarter was calm.
The fix is to keep the maximum (and ideally the minimum) alongside the average in every rollup tier. The max line is the ghost-hunter's line. It answers "what is the worst thing that happened inside this hour?" — which is usually the exact question. Check your monitoring system's storage settings before you need them. This single setting decides whether your history can testify about bad moments or only about bland averages.
Tip: when trouble starts, protect the evidence. If your system rolls raw data up after a couple of weeks, export or archive the raw-resolution history for the affected devices now, before the rollup mulches it. Future-you, three weeks into the hunt, will want to re-examine the early occurrences at full detail.
The three calendars: patterns worth hunting
Almost every punctual ghost is driven by one of three calendars, and it pays to know the usual suspects on each.
The clock: time-of-day patterns
Daily rhythms come from two worlds. The human one: the morning login storm, the lunchtime lull, the end-of-day rush. And the scheduled one: backups, antivirus scans, database maintenance, report generation, replication, update checks — the nocturnal machinery of IT. A spike at a suspiciously round time (01:00, 02:30, 03:00 on the dot) is practically a signed confession from a scheduled task. Remember that the physical building keeps a clock too. That includes heating and air-conditioning setbacks in the evening, and timer-controlled lights and outlets. It also includes the cleaning crew arriving at nine with a vacuum cleaner that shares a circuit with your edge switch. Facilities equipment keeps your clock, and quite often your circuit.
The week: day-of-week patterns
Weekly rhythms are the classic blind spot, because no human troubleshooting session lasts long enough to see one. The usual drivers include weekly full backups (often weekend nights) and weekly scans and maintenance jobs. There are Monday-morning surges as a week's worth of updates and sync backlogs land at once. And there are weekend building setbacks. Think of the server closet that runs warm from Saturday to Monday because the air conditioning relaxes when the office empties. If your ghost log says "mostly Mondays," believe it and go looking for what makes Monday special.
The long cycle: monthly, seasonal, and arithmetic calendars
Month-end brings payroll runs, invoicing, closing reports — heavy batch work that only exists a few days per month. Seasons bring heat, and heat finds marginal fans, dusty vents, and optics that were only ever barely healthy. And then there are the arithmetic calendars: cycles built into protocols and leases. Address lease renewals, tunnel rekeys, and counter wraps repeat on periods no human schedule uses. When two of them line up only occasionally, the result looks random until you plot months of history and the beat pattern emerges. If the trail points at anything with a timer, write down every timer's period. The least common multiple of two timer periods has explained a great many "random" mysteries.
There is a fourth calendar, and nobody writes it down: the building's. Meridian Parts had a warehouse terminal that dropped off the network for a few seconds at about twenty to eleven, "most Thursdays". Four weeks of one-minute trending on the access switch showed the port going down at 10:41 every Thursday and on no other day. The admin spent a further week looking for a Thursday job in every scheduler in the building. There was none. The cleaner's floor polisher was on a Thursday rota. It shared a socket circuit with the switch in the warehouse office. And the cleaner started at 10:40. The fix was a different socket, and the investigation record lists the floor polisher as "environmental", which it was.
Reading the graph like a hunter
Graph-reading is a skill, and a few habits do most of the work:
- Zoom out first. Start at a month, not an hour. Shapes and rhythms appear at a distance; detail comes later. The single most common graph-reading mistake is starting zoomed in on one bad afternoon.
- Compare like with like. Lay this Monday over the last four Mondays. A weekday-versus-weekend comparison will "discover" a pattern that is really just the business being open.
- Mind the y-axis. Auto-scaling makes molehills into mountains: a graph that looks dramatic may span a trivial range. Read the numbers, not the shape, before getting excited.
- Overlay pairs of suspects. Latency on top of utilization, errors on top of traffic, application response on top of disk queue. Two lines that always move together are trying to tell you something.
- Find the birthday. Scroll back and ask: when did this pattern start? The week it first appears is gold — something changed that week, and change records or memory can often name it.
- Annotate. Mark known events on the graphs — sightings from the ghost log, maintenance, config changes. Most monitoring tools support annotations; even a side list of dates works. A graph with sightings marked on it is halfway to a timeline.
An annotated pattern hunt
Here is how a hunt actually plays out, compressed from weeks into six steps. The complaint: "the project app is slow opening files, some mornings, for maybe an hour." No pattern anyone can name; the app server looks healthy whenever anyone checks. Classic ghost. You have several weeks of trend history on the file server, the WAN link, and the core interfaces. The WAN is the wide-area connection that joins this site to the others.
Step 1 — zoom out. You open the file server's disk latency graph at a one-month view, max values. There it is immediately: tall latency spikes on some mornings, roughly 08:30 to 10:00, then weeks of calm baseline. The complaints were real, and they were punctual. Chalk one up for the max rollup line — the average line at this zoom shows almost nothing.
Step 2 — hunt the weekday. Which mornings? You list the spike dates and check the weekdays: mostly Mondays. But not every Monday, and one spike lands on a Wednesday, which breaks the tidy theory. Instead of discarding the Wednesday as noise, you look closer — and notice it followed two days when the office was closed. The real pattern is not "Monday." It is "the first working morning after any break." The longer the break, the taller the spike.
Step 3 — overlay a second metric. Same date range, WAN utilization on top of disk latency. On every spike morning the WAN is pinned flat at its ceiling for the same ninety minutes. Now there are two suspects moving together. A mechanism starts to suggest itself: something floods the network on first mornings, and the file server's slowness may be a bystander symptom.
Step 4 — find the source. You check the interface trends for the usual nocturnal machinery. You find the update distribution server's port moving a mountain of data during exactly those windows. The mechanism writes itself. Over a break, workstations accumulate pending updates and sync backlogs. On the first morning back, dozens of machines fetch everything at once, saturating the shared path. Everything crossing that path — including the project app's file traffic — wades through mud.
Step 5 — check the birthday. Scrolling further back: the pattern is absent before the week a new site came online across the same WAN link. The graph even dates the change. A change record — or a colleague's memory — confirms what was rolled out that week.
Step 6 — act like a scientist. The theory predicts the next first-morning-after-a-break will spike unless the update traffic is tamed. You throttle the update server's schedule, wait for the next break, and watch: flat. The prediction is confirmed. The same graphs that caught the ghost now prove the fix. That is a loop that reproducing rare failures turns into a formal habit.
Notice what never happened: nobody caught the problem live, and nobody needed to. Also notice the twist — the complaint pointed at the app, the app pointed at the file server, and the actual fault was neither. Trend graphs are good at exactly that kind of exoneration: the symptom tells you where it hurts, the trends tell you where the disease is. The file server received no apology. They rarely do.
Trend data you collect yourself
Sometimes the metric you need is not one your monitoring tool collects — say, response time of one specific operation from one specific desk. Do not let that stop the hunt: a few lines of script make a perfectly good special-purpose trender. On a Windows machine, this loop measures reachability and response time of a server every ten seconds and appends it to a file with a timestamp:
while ($true) {
$stamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
$r = Test-Connection -ComputerName 10.0.0.5 -Count 1 -ErrorAction SilentlyContinue
if ($r) { $val = $r.ResponseTime } else { $val = "FAIL" }
Add-Content -Path C:\trend\fileserver-rtt.csv -Value "$stamp,$val"
Start-Sleep -Seconds 10
}
Swap the middle for whatever operation matters — a timed file copy, a web request, a database query. You have a symptom-shaped trend at ten-second resolution, ready to be graphed in a spreadsheet. Log every sample, success and failure alike, because "it was fine at 03:10" is evidence too. This is also the seed of a tripwire, a theme the watchdog guide develops properly.
Your trend-hunt setup checklist
When a ghost appears, run through this list the same day. Not tomorrow. The same day, because every item on it is cheaper before the fault has settled in than after:
- Confirm every device and interface near the symptom is actually being trended — the unmonitored closet switch is a classic hiding spot.
- Shorten the polling interval on the suspect scope; leave the rest of the network alone.
- Check that rollups keep max values, and archive the raw history for the suspects before it ages out.
- Add the symptom itself as a trend if the tool cannot see it — script it if needed.
- Annotate every known sighting on the graphs, and keep annotating as reports arrive.
- Book a recurring fifteen minutes each week to zoom out and compare like with like — the pattern reveals itself to whoever actually looks.
Where to go next
Trend graphs tell you when the ghost walks and hint at what else moves when it does. That is a great deal to get from three weeks of letting a poller run. It costs you nothing but the discipline to set it up before you need it and to zoom out once a week. The fault keeps better time than you do, and the graphs are where you catch it checking its watch. The next step is precision. That means merging graphs, device logs, alerts, and human sightings into a single ordered story. The story shows what always happens just before the fault. That is the craft of building timelines, the next guide in Catching Intermittent Problems.
Put this into practice with SysPeep Monitor
Pattern-hunting needs weeks of clean, boring measurements — collected before you knew you would need them. SysPeep Monitor discovers your devices and interfaces automatically. It trends health and performance over SNMP, WMI and SSH. It turns the history into dashboards, graphs and reports you can zoom out on when a ghost appears. The free trial runs on any Windows machine.
Download Free TrialLearn About SysPeep