Home › Topics › Alert Correlation & Root-Cause Thinking › Cause vs Symptom

Root Cause vs. Symptom: Which Alert Is the Real One?


When something real breaks, your monitoring rarely sends you one alert. It sends you a chorus: the thing that broke, the things that broke because of it, and the checks that merely noticed the commotion. Every alert is "real" in the narrow sense — each describes a true condition — but only one or two of them describe the problem. The rest are echoes. Triage, at its heart, is telling the difference.


The trap is that alert screens are sorted by loudness. The all-caps CRITICAL about the customer-facing service is a symptom. The modest little warning about an interface three rows down is the cause. And every instinct you have pulls you toward the wrong one. A symptom is a condition produced by something else failing. A root cause is the condition that, if you fixed only it, would make the rest of the screen clean itself up. The skill of this guide is finding the alert with that property — the one that explains the others.


You will learn a simple test for ordering any two alerts by causality. You will also learn a chain-walking method for following symptoms down to their source, and the reasons loudness and even timing can mislead you. The guide explains how deep to chase a cause mid-incident (not as deep as you think). It shows how to confirm a suspect cheaply before committing to a fix. It also shows how to turn every solved storm into a pattern you recognize instantly next time.


In this guide

  • The explanation test
  • The chain walk: five whys, live-fire edition
  • Why the loudest alert misleads
  • Proximate vs. deeper causes: how far to chase
  • Confirm before you fix
  • Building a signature library
  • A worked triage: 12 alerts, 2 causes, 1 coincidence
  • The cause-hunt checklist

The explanation test


Here is the whole idea in one question. Take any two alerts, A and B, and ask: would A's condition produce B's condition? Then ask it the other way around. "Would a dead switch make pings to the printer behind it fail?" Yes, obviously. "Would failed pings to a printer kill a switch?" No (nothing kills switches except time and lightning). The arrow only points one way — and that direction is your causality ordering: A sits closer to the root than B.


Run that pairwise question across an alert screen and the pile organizes itself into a rough hierarchy. A small number of alerts explain others, and a large number are explained. The alert that explains the most and is explained by nothing on the screen is your prime suspect. Notice what you are not using: severity labels, alphabetical order, or which alert is scariest to read. Just the directional question, over and over. It feels almost too simple, and it is the single most reliable tool in triage.


Sometimes the test ends in a tie — neither condition would produce the other. That is information too: you are looking at two independent problems (or a problem plus a coincidence). The screen then needs to be split into two piles before you go further. The time-and-place sorting from manual correlation is how you draw that seam.


Arrows you can pre-compute

You do not have to reason every pair from scratch, because most arrows follow the shape of the stack. Causality flows upward. Power failures produce network failures, and network failures produce unreachable hosts. Host failures produce dead services, and dead services produce miserable user-experience checks. Causality flows from shared to specific: one database explains three applications; three applications never explain one database. And it flows from physical to logical. A cable can break a VPN tunnel. A VPN tunnel cannot fray a cable. So when two alerts sit at different layers, the lower and more shared one almost always holds the arrow. That is why seasoned admins scan a storm bottom-up, reading the infrastructure warnings before the application criticals. That is exactly the opposite order the alert screen presents them.


The chain walk: five whys, live-fire edition


The explanation test compares alerts you already have. The chain walk extends it to causes that have not alerted at all. Start from any symptom and iterate three questions:


  • 1. What conditions would produce this? (List the two or three realistic candidates, not twenty.)
  • 2. Is any of those candidates alerting, or checkable right now?
  • 3. If yes, move to it and repeat. If no, you have walked off the edge of your monitoring — check the candidates by hand.

"Web app check failing — what would cause that? App service down, or its database unreachable. Is the database alerting? Yes: disk full. What fills a disk? Logs, dumps, backups gone rogue — nothing else is alerting, so look by hand." Three hops from symptom to a directory listing showing a log file the size of a small moon. Each hop is small and testable; you never leap from "web down" to a grand theory.


Two disciplines keep the walk honest. First, list candidates before checking any. If you dive down the first idea, you will follow it for twenty minutes on momentum alone. That is how "it must be the firewall" outlives all evidence. Second, prefer the candidate that would also explain your other symptoms. The walk is happening inside a wider screen. A candidate that accounts for three alerts outranks one that accounts for only the alert you started from.


If this smells like the classic "five whys" exercise, it is — but note the difference in stakes. The version run in a calm meeting room days later, with coffee and hindsight, belongs to post-incident reviews. This is the live-fire version. You walk the chain only far enough to find something you can act on now, and you write down where you stopped. The chain walk also has a humbling side effect. When the walk dead-ends because the true parent has no check on it, you have just discovered next week's monitoring improvement.


Why the loudest alert misleads


Two forces conspire to point tired admins at the wrong alert, and both are built into your tools.


Severity is about impact, not causality

Severity levels answer "how much does this hurt?" Pain concentrates at the top of the stack, where users live. Causes concentrate at the bottom, where cables live. So the service-down alert screams CRITICAL while the interface-down alert that caused it murmurs WARNING. Both labels are correct for what they measure. The mistake is triaging by them: severity tells you what to communicate and how fast to move; only explanatory power tells you where to dig. When you catch yourself opening the scariest alert first, ask the explanation test's question instead: does this explain anything, or is it explained?


The earliest alert can lose the race

"Whatever fired first is the cause" is a good heuristic with a known failure mode: detection lag. An alert's timestamp is when the problem was noticed, not when it began. Notice-time depends on the check's polling interval, confirmation retries, and notification delays. A symptom polled every thirty seconds will reach your pager before its cause polled every five minutes — the child outruns the parent. So treat "earliest" as one witness among several. It is reliable when the gap is minutes and the intervals are similar. It is unreliable when the gap is seconds or the intervals differ wildly. (Interval choices and their side effects are a whole subject of their own — see the polling and granularity guides.)


Proximate vs. deeper causes: how far to chase


Here is a truth that surprises people new to root-cause thinking: "the" root cause is a matter of how far you keep asking why. The disk is full (proximate cause — the nearest one). Why? A runaway debug log (deeper). Why? Log rotation silently failed after an update (deeper still). Why? Nobody monitors whether rotation runs (deepest, and now it is a lesson about your monitoring). Every layer is honestly called a cause; they are just useful at different times.


The mid-incident rule is: stabilize at the proximate cause, and write the deeper chain down for later. During the outage, clearing disk space and restoring the service is the right scope — that is the fix-now half. The failed rotation and the missing check belong to the learn-later half: the review. There, the calm version of the chain walk finishes the job. The incident response guides cover keeping those two halves from blurring. Chasing ultimate causes at 3 a.m. burns the hours users are down; never chasing them guarantees a rerun of the same incident. The split is what lets you do both well.


One distinction keeps this honest. Treating a symptom is legitimate triage when you know it is symptom treatment and schedule the real fix. Restarting the crashed service to end the outage, with the memory-leak hunt booked for tomorrow, is professional work. The same restart becomes whack-a-mole when it quietly replaces the real fix. In that case, the service now crashes weekly, the restart becomes a ritual, and the underlying cause compounds interest. The test is simple: is there a follow-up item with an owner, or just a habit forming?


Confirm before you fix


Triage produces a suspect, not a verdict. Acting on an unconfirmed suspect has a specific, painful cost: you fix the innocent component while the guilty one keeps burning. Rebooting a healthy switch because "it must be the switch" adds a second outage on top of the first and resets the evidence. It costs you the twenty minutes in which the actual cause (the upstream fiber, say) went from degraded to dead.


So insert one small step between hypothesis and action: the cheap test. Cheap means fast, low-risk, and decisive — usually under a minute:


  • Suspect the switch? Try to reach it from the right side of the network, and check its uplink port status from the device above it.
  • Suspect DNS? Run one manual lookup against each server.
  • Suspect the disk? A directory listing settles it faster than any dashboard.
  • Suspect the WAN link? One traceroute shows where packets stop.

If the cheap test confirms, act with confidence. If it refutes, you just saved yourself the wrong big fix at the cost of sixty seconds. And when triage produces two plausible candidates, resist the urge to pick a favorite. In that case, run both cheap tests, in parallel if there are two of you. Candidates come from the heuristics; verdicts come from tests. Keeping those roles straight is most of what separates calm troubleshooting from thrashing.


Tip: beware the "usual suspect" shortcut cutting the wrong way. Experience whispering "it's always the VPN concentrator" is a fine source of hypotheses and a terrible source of conclusions. Let the pattern skip you to the front of the queue — then run the cheap test anyway. The day the usual suspect is innocent is exactly the day you cannot afford to skip it.


Building a signature library


After a few storms you will notice something valuable: causes repeat, and each repeating cause announces itself with the same combination of alerts — a signature. "These five alerts together" stops being a puzzle and becomes a name: oh, that is the storage array's path flap again. Recognition is the fastest form of diagnosis, and unlike intuition, it can be written down and shared.


The format that works is a small signature table, kept with your runbooks. Each row should end by pointing at one. The guide to runbooks covers writing the procedures themselves. Here is the table:


Signature (alerts seen together)Usual causeVerify withThen
Core port down + one closet's devices all unreachableUplink transceiver or cablePort status and light levels from the core sideOptics runbook
ERP + webshop + reporting checks fail togetherThe shared database hostDB service state and disk spaceDB recovery runbook
All guests on one host slow + storage latency warningsStorage path flapPath status on the hostStorage path runbook
Every remote site's checks fail at onceHQ edge or ISP, not twelve sitesTraceroute outward from HQWAN outage runbook

Add a row whenever a diagnosis took you longer than five minutes — that is the sign the pattern was not yet in the library. A dozen rows in, most of your incidents open with recognition instead of investigation. The library doubles as training. It is your environment's tribal knowledge, finally written where the newest team member can use it on day one.


A worked triage: 12 alerts, 2 causes, 1 coincidence


Time to run the whole method. Here is the screen, sorted by receive time:


10:31:04  WARN  SW-CORE       port Gi0/24 down (uplink to SW-4)
10:32:10  CRIT  SW-4          host unreachable
10:32:15  CRIT  PRN-4F        host unreachable
10:32:21  CRIT  AP-4F-N       host unreachable
10:32:26  CRIT  AP-4F-S       host unreachable
10:32:40  CRIT  TIMECLOCK-4F  host unreachable
10:33:05  CRIT  SRV-DB        disk D: 100% full
10:33:30  CRIT  SVC-ERP       application check failed (error 500)
10:33:41  CRIT  SVC-SHOP      checkout check failed (timeout)
10:33:55  CRIT  SVC-REPORTS   report generation check failed
10:34:02  WARN  SRV-APP1      CPU sustained > 90%
10:35:56  WARN  CERT-WILDCARD expires in 14 days

Split first. The explanation test ties between, say, SW-4 and SRV-DB — neither produces the other — so this is not one story. Place draws the seam: five alerts are floor 4 hardware; four are application services; two need a look.


Pile one: the humble 10:31 warning explains everything unreachable after it — port Gi0/24 is SW-4's uplink, and a dead uplink produces exactly this pattern. Would any of the reverse arrows hold? No. Cause one: the uplink (cheap test: check the port's light levels; likely a failed transceiver). Note the loudness lesson: the causal alert is the screen's quietest line, and the earliest.


Pile two: chain-walk from SVC-ERP: what causes a 500 error? App service down or its database sick. Is the database alerting? Yes — disk full at 10:33:05, before the service checks. ERP, shop, and reports all share SRV-DB, so one condition explains three service failures. SRV-APP1's high CPU? Plausibly explained too — applications retrying a dead database in a tight loop do exactly this — so it joins pile two as a symptom, pending confirmation. Cause two: the full disk (proximate; the review can hunt what filled it).


The leftover: the certificate warning fails every test. It has the wrong mechanism (nothing here affects expiry dates). Its timing is explained by a daily scheduled check that happened to run mid-crisis. Coincidence rider: its own unhurried ticket. Final tally: twelve alerts, two causes, one rider, and the two loudest CRITICALs on the screen were both symptoms. That ratio is typical, which is the entire argument for triaging by explanation instead of volume.


The aftermath confirms the reads. A replacement transceiver in Gi0/24 brings floor 4 back. The five unreachable alerts recover in one tight burst. That mass recovery certifies pile one had a single cause. Clearing the database disk revives all three service checks within minutes. SRV-APP1's CPU drifts back to normal without anyone touching it, confirming the retry-loop theory. Two notes go into the review pile. Find what filled drive D (the deeper cause). Give SW-CORE's uplink ports louder alerting, since the causal warning nearly drowned. The certificate gets renewed on Thursday, unhurried, by whoever loses the coin toss.


The cause-hunt checklist


  • Split before you dig: if the explanation test ties, you have more than one event.
  • Run the arrows: for each pair, would A produce B? Build the little hierarchy.
  • Ignore severity while hunting: it ranks pain, not causes. Use it afterwards to rank communication.
  • Trust "earliest" only across minutes, and remember slow-polled parents lose races.
  • Chain-walk past the screen's edge: the cause may not be alerting at all — note the gap for later.
  • Stop at the proximate cause; write down the deeper chain for the review.
  • Cheap-test before the big fix — both candidates, when there are two.
  • Afterwards: add the signature to the library so next time is recognition, not investigation.

This skill compounds with the rest of the pillar. The guide to anatomy of an alert storm covers the mass-alert version of the hunt. The guide to dependency mapping teaches your tools to pre-label parents and children so fewer hunts are needed. The guide to event enrichment makes every alert arrive carrying the context the hunt feeds on. The full set is at Alert Correlation & Root-Cause Thinking.


Put this into practice with SysPeep Monitor

Cause-hunting is faster when the echoes never page you in the first place. SysPeep Monitor's dependency and alert rules are designed to keep symptom noise down. Its live dashboards and maps put the evidence for your cheap tests one click away in a single web console. That evidence includes port states, disk space, and service health. The free trial installs on a Windows machine in minutes.

Download Free TrialLearn About SysPeep