Home › Topics › Alert Correlation & Root-Cause Thinking › Manual Correlation

Poor-Man's Correlation: Time, Topology, and Common Sense


Somewhere out there is a brochure promising that an "event correlation engine" will read your alerts, deduce the root cause, and hand you the answer. Some of that technology is genuinely useful. But here is the secret experienced admins know. A human running three honest heuristics — time, place, and common sense — correlates most real-world events faster and more accurately than most software. That is because the human knows things no engine was ever told. And the three heuristics work with any tooling at all, including "a pile of alert emails and a text editor."


Correlation, if the word is new, just means recognizing that several alerts describe one underlying event. It turns "eleven things are broken" into "one thing broke, and here is what it took with it." It is the skill that stands between you and treating every storm as fifty separate mysteries.


This guide turns the instincts good admins develop by accident into a deliberate method. It covers the three heuristics in detail and a scratchpad workflow for running them under pressure. It also covers the cheap tooling tricks that support them and honestly examines where the manual method hits its limits. Then come two practice scenarios to test yourself on. No purchases required.


In this guide

  • Heuristic one: time
  • Heuristic two: place
  • Heuristic three: common sense
  • The workflow: a correlation scratchpad
  • Cheap tooling for the method
  • Where manual correlation struggles
  • The reference card
  • Two practice scenarios

Heuristic one: time


Time is the cheapest evidence you have, and it carries three separate reads.


The burst test

Alerts that arrive within the same short window — the same minute or few — are probably one event. Failures caused by a common trigger are discovered together, smeared only by polling intervals. So a tight cluster after quiet is the fingerprint of a single cause. The reverse read matters just as much. Alerts spread evenly across an afternoon are probably separate problems. Gluing them into one story will send you hunting a phantom. Your first move with any noisy screen: sort by time and look for the cliff edge where the burst begins.


The sequence read

Within a burst, order is a clue: cause tends to precede effect, so the earliest alerts often sit closest to the root. But respect the caveat: order within seconds is noise; order across minutes means something. Your checks run on polling schedules. So which victim gets "discovered" first inside the same polling cycle is essentially a coin flip. A child polled at :00 beats its parent polled at :04 even though the parent failed first. A gap of several minutes, though, is real information: the disk-latency warning that fired six minutes before everything else deserves your attention. And all of this assumes your clocks agree. If devices and the monitoring server disagree about what time it is, the sequence actively lies to you. Timestamp sanity covers making time trustworthy. Until then, sort by your monitoring server's receive times, which at least come from one clock.


The recovery read

Time has one more gift: recoveries carry information too. When the parent of a storm is fixed, its children recover together in another tight burst. A "mass recovery" a minute after you swap the switch is confirmation you found the real cause. The inverse is diagnostic as well. If half the victims recover and half stay down, you were probably looking at two overlapping problems wearing one storm's coat. And a rapid alternation — down, up, down, up on the same check — is not a storm at all but a flap. A flap points at an intermittent cause (a marginal link, a device rebooting in a loop) and deserves a different investigation entirely. Before declaring any event over, glance at the recovery pattern and ask whether it matches your story.


The change read

The third read compares alert times against a different timeline: what did we change? An enormous share of incidents begin minutes after somebody's "quick change." So the question "what happened at 14:02?" should always be followed by "and what did anyone touch around 14:00?" Check the change log, the ticket queue, and — honestly — the chat channel. If your monitoring knows about planned work windows, alerts during them arrive pre-explained. The maintenance windows & suppression guides cover building that change-awareness in. Manually, the habit is simply: burst time, minus zero to fifteen minutes, equals suspect window for changes.


Heuristic two: place


Time tells you the alerts belong together; place tells you what they have in common. Sort the alerting devices by where they live — site, building, closet, rack, subnet, VLAN. Then ask the shared-container question: is there one box, room, or network segment that holds all the victims?


When the answer is yes, the culprit is almost always the container's shared infrastructure. Everything in one rack points at that rack's switch or power strip. Everything in one closet points at the closet switch or its uplink. Everything on one VLAN points at that VLAN's gateway or configuration. Everything at one site points at the site's WAN link or router. You are not diagnosing fifteen devices — you are diagnosing one container, and containers have short parts lists. Notice that the container question also works at more than one zoom level. "All of building B" and "all of rack 4 within building B" are different answers pointing at different suspects. So zoom in until the container gets as small as the evidence allows.


When the answer is no — victims scattered across sites and subnets with no geographic logic — the heuristic flips. In that case, stop asking where they live and start asking what they all need. Scattered, weird, simultaneous failures are the signature of a shared service, and the suspect list is mercifully short. Run the common-dependency checklist:


  • DNS — failures mention names, timeouts, "cannot resolve"; things answer pings but nothing works by name.
  • Authentication — logins hang or fail everywhere; services that were already logged in keep working, new sessions do not.
  • Shared storage — VMs and apps freeze or crawl together; hosts fine, guests miserable.
  • NTP / time — the sneaky one: authentication errors, certificate complaints, and log weirdness caused by clock drift.
  • A common upstream path — "scattered" victims that secretly share one firewall, proxy, or ISP.

Knowing your network's real dependency roots is what makes this read fast. The guide to dependency thinking teaches you to see them before the outage does. The guide to dependency mapping shows how to teach your monitoring the same trick permanently.


Heuristic three: common sense


The third heuristic is the one no engine replicates: asking whether the story actually hangs together.


The plausibility filter

For each alert you are about to fold into the event, ask: would my suspected cause actually produce this? A dead switch explains unreachable printers behind it. It does not explain a full disk on a server in another building — no cable in the world makes a disk fill up. If you cannot draw a physical or logical line from cause to symptom, the alert does not belong in the story, however convenient it would be.


The coincidence discipline

Storms can and do overlap unrelated failures. Networks are big enough that two things sometimes break in the same half hour by pure chance. The alert that fails the plausibility filter is your odd one out. The discipline is to give it its own thread. That means its own ticket, its own eventual owner, and its own five minutes after the main event is stable. One classic mistake is forcing it into the storm's narrative (wrong diagnosis). The other is mentally filing it under the storm (real problem forgotten until it grows up into its own outage).


The Occam ordering

When several explanations fit, prefer the one that explains the most with the least. One cause accounting for ten symptoms beats ten independent causes every time. That is because ten simultaneous independent failures is a lottery ticket. This is why the shared-container and shared-service reads work at all. You are searching, always, for the smallest story that covers the facts.


The known-behavior library

Finally, pattern memory is legitimate correlation. Every environment has signature failure patterns. One is "when the backup job overruns, the file server's latency checks whine until nine." Another is "when that ISP has a bad day, exactly these three site checks flap together." Recognizing a known pattern in two seconds is not cheating; it is expertise. The only failure mode is keeping the library in one person's head. Write the patterns down — "these alerts together usually mean X; verify by checking Y" — where the whole team can find them. The runbooks guides cover the format. Future-you at 3 a.m. is effectively a different, dumber admin who deserves notes.


The workflow: a correlation scratchpad


Under pressure, heuristics need a container, or you will just re-read the same alerts in a loop while the phone keeps buzzing. The container is a three-column scratchpad — paper, notepad file, whiteboard, anything:


Column 1: the time cluster. Everything inside the burst window. Column 2: the place/service cluster. The subset of column 1 sharing a container or a suspected common service — annotated with what they share. Column 3: doesn't fit. Anything outside the burst or failing the plausibility filter. Running it takes five steps:


  • 1. Freeze the window. Note the burst's start and end times. New alerts after your cutoff get triaged separately — do not let the columns become a moving target.
  • 2. Sort by time and copy everything in the burst into column 1, earliest first.
  • 3. Ask place. Mark what each victim lives behind or depends on; move the ones sharing an answer into column 2 with the shared thing named.
  • 4. Apply sense. Anything that fails the plausibility filter — wrong time, wrong place, impossible mechanism — goes to column 3 with one word of reasoning.
  • 5. Name the suspect. Column 2's shared thing is your hypothesis; the earliest line in column 1 is your supporting witness. Go test it.

Here is a filled scratchpad from a real-shaped event, eleven alerts around 14:02:


BURST (14:02-14:05)          SHARED                    DOESN'T FIT
--------------------         -----------------------   ------------------
SRV-FILE unreachable    ->   all behind SW-CLOSET-3    PRN-LOBBY offline
SRV-APP1 unreachable    ->   all behind SW-CLOSET-3      (down since 11:40,
AP-3F-E unreachable     ->   all behind SW-CLOSET-3       it's a printer)
AP-3F-W unreachable     ->   all behind SW-CLOSET-3    SRV-MAIL disk 91%
CAM-3F unreachable      ->   all behind SW-CLOSET-3      (creeping for days,
SW-CLOSET-3 unreachable ->   the container itself!        not this event)
SVC-INTRANET check fail ->   intranet runs on SRV-APP1
UPS-CL3 on battery      ->   closet 3 power (14:01!)

Read the columns and the verdict writes itself. Closet 3 lost power at 14:01 (earliest, and it explains the switch). The switch took six devices and one service check with it, and the two misfits get their own tickets. Total sorting time: maybe three minutes. The scratchpad is not bureaucracy. It is a thinking tool that stops the 3 a.m. brain from chasing the loudest line. When the incident is over, it pastes straight into the ticket as a timeline.


Cheap tooling for the method


You do not need a correlation engine to make the three heuristics fast. You need your alerts in sortable, searchable form, which almost every tool can manage:


  • A sortable alert view or export. Sort by time for the burst test; sort or filter by group/site for the place test. If the console will not sort, export to CSV and let a spreadsheet do it.
  • A grep-able alert log. If alerts land in a text log (or a mail folder you can search), one line pulls a burst: grep "14:0" alerts.log | sort. Crude, fast, effective.
  • Group-by-site dashboards. A dashboard arranged by location turns the place heuristic into a glance — one red closet is a diagnosis. Network maps do the same job pictorially.
  • Naming conventions. The quiet hero: if devices are named SW-B2-CLOSET instead of device047, every alert already announces its own place. In that case, half the scratchpad fills itself.

Tip: set your alert view's timestamps to one timezone — ideally the monitoring server's — and keep it that way. Half of all manual correlation errors are actually timezone errors wearing a disguise, and they are much easier to prevent than to notice at 3 a.m.


Where manual correlation struggles


Honesty section. The manual method has three real limits, and knowing them is part of the method.


The volume ceiling. A human can sort a few dozen alerts; nobody sorts four hundred. If your normal, non-storm days already produce more alerts than you can triage, no heuristic will save you. In that case, the fix is upstream: deduplication, sane thresholds, and deleting alerts nobody acts on. That is the territory of the alert fatigue guides. The upstream fix is a prerequisite for everything in this article: correlation assumes a signal worth correlating.


Subtle multi-cause events. Two overlapping failures with interleaved symptoms can defeat tidy column-sorting. Say, a flaky uplink degrades during an unrelated DNS problem. No single suspect passes the plausibility filter for everything. The tell is a cluster that refuses to resolve into one plausible story: every candidate explains six alerts and is contradicted by three. The countermeasure is to split deliberately. Divide the alerts into two piles along the cleanest seam you can find (usually place, sometimes symptom type). See whether each pile suddenly makes sense on its own. If yes, you have two events, two suspects, and ideally two people working them in parallel.


The 3 a.m. brain. Judgment degrades exactly when storms prefer to arrive. You will anchor on the first alert, force misfits into the story, and skip verification — unless the method is externalized. That is why this guide ends with a card: checklists are how tired people borrow their rested selves' judgment.


The reference card


  • TIME: Sort by time. Find the burst edge. Inside the burst = one suspect event; outside = separate. Order across minutes is a clue; order within seconds is noise. Ask "what changed just before this?"
  • PLACE: Sort victims by location/subnet. One container → suspect its switch, power, or gateway. Scattered → suspect a shared service: DNS, auth, storage, NTP, common path.
  • SENSE: Would the suspect actually produce each symptom? Misfits get their own thread. Prefer the smallest story that explains the most. Check the pattern library first.
  • Then: test the suspect directly, and write the scratchpad into the ticket.

Two practice scenarios


Scenario 1: the scattered six

Between 08:59 and 09:02 you get six alerts. The public web check times out, the mail-flow check fails, and the VPN portal check fails. Two internal servers report "name resolution failed" from their application logs, and a cloud-app reachability check fails. The devices span three subnets and two buildings. Pings to all six targets succeed. What is your first suspect, and your first verification?


Answer: there is a tight burst (time says one event). The places are scattered (so a shared service, not a container). Everything failing involves names, and pings still work. The common-dependency checklist points straight at DNS. Two of the alerts even say so. Verify in ten seconds with a manual lookup against each DNS server, and check what changed around 08:55. A firewall or DNS change just before nine o'clock is the classic opening scene of this exact story.


Scenario 2: rack four, plus a printer

Between 14:10 and 14:13: four servers unreachable, the rack-4 top-of-rack switch unreachable, and the storage array's management interface unreachable — all six live in rack 4. Also in the pile: a printer in the other building went offline at 14:11. What is the story, and what do you do with the printer?


Answer: one burst, one container — rack 4 — so suspect the rack's shared parts: its switch or its power. The switch being among the victims tilts toward power. It is a victim rather than the explanation for them, since the management interfaces of everything else in the rack died too. Verify by checking the rack PDU or the UPS feeding it. The printer passes the time test but fails place and plausibility — nothing in rack 4 kills a printer across the street. It is a coincidence rider: its own (low, unhurried, it is a printer) ticket. Correct answers accepted in any order at 3 a.m.


These are exactly the reads that anatomy of an alert storm applies to full-scale meltdowns. When you are ready to pick the single alert that explains the rest, root cause vs. symptom is the next skill. The whole toolkit lives at Alert Correlation & Root-Cause Thinking.


Put this into practice with SysPeep Monitor

Every heuristic in this guide gets easier when your alerts arrive organized. SysPeep Monitor groups devices the way your network is really built and supports dependencies that quiet downstream noise. It gives you live dashboards and network maps that make the place heuristic a single glance. All of this comes from one web console on a Windows machine. The free trial sets up in minutes.

Download Free TrialLearn About SysPeep