One incident per real cause
The classic failure of monitoring is not silence. It is the storm: a core switch dies and forty devices page you at once, or the monitoring server loses its own cable and declares the entire estate down.
Key points
- A failed upstream raises one incident; the devices behind it are listed under it as unreachable.
- If the monitoring host itself loses its link, judgement freezes and exactly one incident is raised.
- Down means unreachable. A full disk or a stopped service is degraded, not down.
- The alert is the incident: acknowledge, assign, comment and archive in one place.
- Thresholds carry a direction and a debounce, or learn what is unusual from the sensor's own readings; events are handled by rules with a built-in tester.
- Every alert names the sensor and, when a device is watched from several places, the place that raised it.
On this page: Down vs. degraded Dependency suppression When the monitor goes blind The alert is the incident Thresholds and rules Quiet and loud Sensor and place On call
After a few storms, the alerting gets tuned down, then ignored, then switched off — and the tool that was bought to end reactive firefighting quietly becomes decorative. SysPeep's alerting was designed around one rule: tell the operator about the real problem, once. Everything on this page is a consequence of that rule.
Down means unreachable
For each device you can name which sensor proves it is reachable — the ping, the SNMP system check, the TCP port that matters — or let SysPeep choose automatically when you have not. Only that sensor decides whether the device is down. Everything else decides whether it is degraded.
A server with a full disk is degraded, not down, so the word keeps its meaning and the on-call response matches the reality: nobody drives to the data centre for a disk that needs a cleanup script.
Two words, two meanings
- Down — the chosen reachability sensor failed
- Degraded — reachable, but a disk, service, query or threshold is in a bad state
- Degraded raises its own alert and leaves availability history alone
One page for the switch, not forty for the hosts behind it
Dependency relationships appear on their own from passive LLDP/CDP capture, so the map reflects the cabling rather than someone's memory of it, and the device workspace shows parents and children stepped and connected.
When an upstream device fails, everything behind it is marked unreachable under the parent's incident rather than down under its own. The on-call phone buzzes once, the incident names the switch, and the forty hosts behind it are listed as consequences rather than causes.
When the monitor itself goes blind
If the SysPeep host loses its link at 3 a.m., judgement freezes: no device is marked down, no availability history is corrupted, no SLA figure is dented, and exactly one incident is raised — about the monitor. When connectivity returns, a single all-clear closes it.
Failures that happen during a service restart are caught and batched the same way, so restarting the monitor never produces a notification storm and never strands an open incident. Alert status survives the restart too: each sensor's stored status is taken as confirmed before the first poll cycle, so a restart never raises an open alert a second time and never reports a recovery that did not happen.
| What happened | What SysPeep raises | What happens to history |
|---|---|---|
| Core switch fails; 40 devices behind it stop answering | One incident, for the switch; dependents listed as unreachable under it | Dependents are recorded as unreachable, not as down |
| Monitoring host loses its own network link | One incident, about the monitor | Untouched — no device is marked down, no SLA is affected |
| Monitoring service is restarted during a patch window | Failures during the restart are batched; nothing pages | Open incidents survive the restart |
| Server disk fills, but the server still answers | A degraded-state alert on the disk sensor | Device availability stays "up" |
| Interface utilisation spikes for one poll, then recovers | Nothing, if the threshold's duration or repeat count was not met | The spike is on the graph, where it belongs |
| A router stops exporting NetFlow | A flow-silence warning event on that device within minutes, naming the silent addresses; an event rule makes it an incident | The traffic totals say how much of the window was observed |
| Probe B cannot reach a server that this server and probe A still can | An alert that reads "from probe B"; the device goes down only when every place fails, if each place is an availability source | Each place keeps its own history |
The alert is the incident
There is no separate alert that becomes a ticket somewhere else. The incident is one object, and everything that happens to it happens in one place, with the full history kept.
- Acknowledge it from whatever browser is nearest, including one with scripting disabled
- Assign it to a person
- Comment on what you found
- Clear and archive it, with who did what, and when, on the incident forever
- Forward it by webhook into chat, ticketing or paging, with no per-integration licensing
Thresholds that know which way is bad
Thresholds, for metrics
Every sensor threshold carries a warning level, a critical level, a direction and a debounce. "Bad when high" fits utilisation and latency; "bad when low" fits free disk, battery runtime and days until a certificate expires, so countdown metrics alert in the right direction. Duration and repeat-count damping mean a momentary spike pages nobody and a sustained problem does. For the sensor whose normal you do not know yet, an Unusual values threshold sets the warning or critical level in sigma against the sensor's own recent readings, learned over eight polls.
Rules, for events
Syslog lines, SNMP traps and informs, Windows events, availability transitions and flow notices become alerts by kind, application and severity. A built-in rule tester shows which rule wins for a given event and which rules are shadowed by earlier ones, so the rule set can be reasoned about instead of guessed at. The thousand events that do not matter stay in the log.
Quiet when it should be, loud when it must be
- Maintenance windows keep planned work silent without anyone remembering to pause and un-pause devices.
- Repeat reminders re-notify unacknowledged incidents on an interval, so a page that arrived during a meeting is not the last you hear of it.
- Escalation sends an incident nobody acknowledged to a second channel.
- Digests bundle a channel's notifications into one message per window, up to 24 hours, so the team gets one summary instead of a hundred emails.
- Batched device-down notifications group simultaneous failures into one message.
- Recovery always clears. A recovery event closes its incident every time; nothing is left open by accident.
- Test-send from the console proves the email, webhook or script channel works before you depend on it at 3 a.m.
- Pause alerts on one device for one or four hours from its Manage sensors page while you work on it.
The sensor and the place, on every alert
Incidents, alerts, events, notifications, reminders, digests and report rows all name the sensor, and for a sensor monitored from more than one place they name the place too: "CPU load from probe B". When a device is watched from several sites, the responder knows at once whether the fault is the device or the path from one probe. Removing a probe never makes an incident disappear, either: an open incident on a place whose probe was deleted gets a note, stops reminding, and stays open until a person closes it.
What this feels like on call
- The phone goes once. The incident names the device that caused the problem, the sensor, the place it was seen from, and the devices that are merely behind it.
- The graph shows the spike, the gap or the slow decline that led up to it, and the events timeline shows what happened right before.
- You acknowledge it from whatever browser is nearest, assign it, and go back to sleep or to work. Nothing else pages you about the same cause.
That is the entire design. It is less exciting than a dashboard full of red, and considerably easier to live with.
See it on your own network
Install SysPeep on a Windows server you already have, let discovery draw the dependency map, and watch how the next real failure arrives: as one incident, not a storm.
Download Free TrialNext: Reports you can stand behind