Alerting on Experience, Not Just Infrastructure
There are two ways for a monitoring system to embarrass you. The first is the quiet one: every infrastructure check is green, and the users cannot log in. The second is the loud one: forty alerts fire in three minutes because a switch rebooted, and not a single user noticed. Both come from the same root cause. Infrastructure alerts describe the health of things — servers, links, services. Things can be perfectly healthy while the experience built on top of them is broken. Or they can be perfectly broken while the experience carries on regardless.
Experience alerts close the gap from the other side. They come from scripted user journeys and say something an infrastructure alert never can: "a user trying to do this, right now, from here, would fail." This guide covers how to design them. It explains the three kinds of journey alert and the thresholds behind them. It covers the confirmation logic that keeps them from paging you for a single hiccup. It shows how to set severity by what the broken step actually costs the business. It explains where to route the alert and what evidence to attach. And it shows how to stop one broken dependency from turning every journey into a siren at once. It assumes you already have journeys producing per-step results; if not, start with scripted user journeys.
In this guide
- Two kinds of alert, two kinds of truth
- The three journey alert types
- Confirmation before paging
- Severity by business impact of the step
- Routing: both teams, evidence attached
- Dashboards that show user-visible health
- Avoiding storms when one dependency breaks every journey
- An alert-design table per journey type
- Where to go next
Two kinds of alert, two kinds of truth
An infrastructure alert is cause-side. It fires when a measured component crosses a line: a link is down, a disk is full, a service stopped, CPU has been pegged for ten minutes. It is precise about what is wrong and vague about whether anyone cares. A redundant link failing over is a real event that no user will ever feel. An experience alert is symptom-side. It fires when a journey step fails or slows. It is precise about what users feel and vague about why. "Login is failing from every site" is unmistakable and names no component. You need both. The mistake most teams make is trying to get by with only the first kind. Then they wonder why the help desk keeps hearing about outages before monitoring does.
The two kinds also fail differently, which is why they complement each other. Infrastructure alerts miss anything that is not explicitly measured: the locked database table, the expired application certificate, the identity provider answering slowly, the search index rebuilding. Experience alerts miss nothing that a user would notice, because they do what a user does, but they cannot tell you which component to fix. The best incident timeline reads like this. The experience alert says login is broken from every site. The infrastructure dashboard shows the directory server's CPU spiked at the same minute. Fix the directory server, and the experience alert clears. Each half of that story is useless alone.
The three journey alert types
Every experience alert is one of three types. It is worth keeping them separate rather than collapsing them into a single "journey unhealthy" signal. They mean different things and deserve different responses.
1. Step failed
A step did not produce its expected result. The login returned an error, the search returned no rows, the form submission was refused, or the step timed out entirely. This is the clearest signal a monitoring system can produce, because it is binary and it names the step. It is the type most likely to deserve a page. It is also the type most in need of confirmation, since a single failed run can be a blip.
2. Step slow
A step passed, but took longer than its threshold. The threshold must be per step and must come from that step's own baseline. That baseline comes from the timings you collected during the first weeks of running the journey from that vantage point. A login that normally takes one second and now takes four is a strong signal. A report step that normally takes eight seconds and now takes nine is not. A single journey-wide threshold cannot tell those apart. A workable rule is to alert when a step exceeds a multiple of its baseline for several consecutive runs. Revisit the baseline whenever the application changes.
3. Journey total over budget
Every step passed and none tripped its own threshold. But the whole journey took longer than the experience budget you agreed with the application owner. This catches the death-by-a-thousand-cuts case where every step is a little slower than usual. That usually points at something shared: the path, the database, the hypervisor host. It is rarely urgent and almost always interesting.
# Alert rules for journey "records-core" (illustrative configuration) journey: records-core budget_total: 12s steps: open_app: slow_after: 2.0s fail_after: 15s severity_on_fail: high login: slow_after: 3.0s fail_after: 20s severity_on_fail: high search: slow_after: 2.5s fail_after: 30s severity_on_fail: high open_record: slow_after: 2.0s fail_after: 30s severity_on_fail: medium submit_note: slow_after: 3.0s fail_after: 30s severity_on_fail: medium logout: slow_after: 2.0s fail_after: 15s severity_on_fail: low confirm: consecutive_runs: 2 vantage_points: 2 # of 5 configured slow_rule: 3 consecutive runs above slow_after notify: app-records-owners, network-ops
The exact syntax depends on whatever runs your journeys; the structure is the point. Thresholds per step, a total budget, an explicit confirmation policy, and a severity that belongs to the step, not to the journey.
Confirmation before paging
Journeys touch many layers, so they occasionally fail for reasons that are real but momentary. Think of a load balancer draining one member, a database failing over, a runner's own hiccup. Paging a human for every one of those is the fastest route to an ignored alert. Confirmation is the rule that says how much evidence must accumulate before an alert leaves the monitoring system. There are two kinds worth using together.
A second run. When a step fails, run the journey again immediately rather than waiting for the next scheduled slot, and treat two consecutive failures as confirmed. This roughly halves the detection delay compared with simply waiting for the next interval, and it eliminates most single-run blips.
A second vantage point. If the same step is failing from a second runner in a different place, the problem is not the runner. Requiring two of your configured vantage points to agree is the single most effective filter for false pages. It comes free once you have the runners described in the guide on where to run transaction tests from. A failure seen from only one vantage point is still worth recording and worth a low-priority notification, because a single branch really can be broken. It is just not worth waking anyone until something else agrees.
| What happened | Interpretation | Action |
|---|---|---|
| One run failed, one vantage point | Blip, or the first sign of something | Log it; re-run immediately; no notification |
| Two consecutive failures, one vantage point | Real from that place; may be that place's path | Low-priority notification to the network team naming the site |
| Two consecutive failures, two or more vantage points, same step | Confirmed; the application or a shared dependency | Alert at the step's severity, both teams |
| Step slow for three consecutive runs, one vantage point | Probably that path; possibly load | Ticket to the network team with timings versus control |
| Step slow for three consecutive runs, all vantage points including control | The application or its dependency | Ticket to the application owner with per-step timings |
| Runner stopped reporting | Missing witness, not a failed journey | Separate "runner down" alert to whoever owns runners |
Confirmation costs time. Two consecutive runs at a five-minute interval means a confirmed alert arrives up to ten minutes after the first failure. That is the honest trade-off: the run frequency sets a floor on detection time, and confirmation sits on top of it. For the journeys that matter most, run them more often rather than confirming less.
Severity by business impact of the step
The severity of an experience alert should not come from the journey; it should come from the step, because different steps carry different consequences. If login is broken, nobody can do anything, and that is a high-severity, page-someone-now event during business hours. If the report export step is slow, a handful of people wait a little longer for a file that they will read tomorrow. That is a ticket. Treating both as "journey failed, severity critical" is how experience alerts lose their credibility in the first month.
Setting severity per step forces a useful conversation with the application owner: which steps, if broken, stop the business? Which are inconvenient? Which would nobody notice until Monday? Write the answers into the alert configuration, and add two modifiers. Time of day: a login failure at 03:00 for an application nobody uses overnight can wait for the morning. A rule that knows the application's business hours turns a page into an email. Breadth: a step failing from every vantage point is worse than the same step failing from one branch, and the severity should say so.
| Step (records journey) | If it fails | Business hours | Out of hours |
|---|---|---|---|
| Log in | Nobody can work | High: page on-call | Medium: notify, page if still failing at start of day |
| Search | Core workflow blocked | High: page on-call | Medium: notify |
| Open record | Core workflow blocked | High: page on-call | Medium: notify |
| Submit note | Writes lost; people re-key later | Medium: ticket, notify owner | Low: ticket |
| Export report | Inconvenience | Low: ticket | Low: ticket |
| Log out | Cosmetic, possible session build-up | Low: ticket | Low: ticket |
Routing: both teams, evidence attached
An experience alert describes a symptom, and symptoms have two plausible owners: the application team and the network team. Send it to both. Sending it only to the application team produces "the network must be slow" replies. Sending it only to the network team produces "the app must be broken" replies. Sending it to both with the evidence attached produces a fix. Each team can see in the first minute whether the evidence points at them. The guide on the anatomy of a good alert covers its wording and structure: what happened, where, since when, what to do first. What follows is the experience-specific evidence that belongs inside it.
- Journey and step. By name, with the step number and what it does in plain words.
- What was expected and what was received. "Expected one result containing SYNTH-TEST-CUSTOMER-01; received page titled 'Search unavailable'." For browser journeys, attach the screenshot.
- Vantage points. Which runners see the failure and which do not, including the control. This one line usually settles which team owns the next hour.
- Per-step timings versus baseline. The steps before the failure and their timings, so the reader can see that login was normal and only search broke.
- Since when, and the last good run. A timestamp the reader can line up against change records and infrastructure graphs.
[HIGH] records-core: step 3 "search" FAILED (confirmed) Seen from: control-dc, hq-lan-01, branch-north-01, remote-vpn-01 (4 of 5) Not seen from: outside-01 (last run 09:12, pass) Expected: 1 result containing SYNTH-TEST-CUSTOMER-01 Received: HTTP 200, page title "Search temporarily unavailable" Steps before failure: open-app 0.4s (baseline 0.4) | login 1.1s (baseline 1.0) Failing since: 09:15 Last good run: 09:10 Consecutive failures: 2 Because the control sees it too, the path is exonerated; start with the search service.
The last line is the most valuable one and the easiest to automate: a sentence that states what the vantage-point pattern implies. Alerts that interpret their own evidence get acted on; alerts that dump evidence get forwarded.
Dashboards that show user-visible health
Alerts are for the moment something breaks. Between those moments, the question people actually ask is "is everything okay for users right now?" The honest answer lives on a dashboard that shows journeys, not devices. The most useful layout is a grid: applications down the side, vantage points across the top. It has one cell per journey per place, colored by its latest confirmed result. A glance shows whether a problem is one application everywhere (a row goes red) or one site for everything (a column goes red). Or only one cell shows a problem: a single application from a single place.
Put that grid beside the infrastructure view, not instead of it. The infrastructure dashboard — device status, link utilization, server health — is what engineers use to find causes. The journey grid is what the help desk, the application owners, and the manager who wants a one-word answer use to understand impact. When both are visible together, a red journey cell next to a red WAN link tells the whole story without anyone opening a ticket. Add a small history strip to each cell, showing the last day of step timings. And the grid also becomes the easiest way to spot a step that is slowly getting worse.
Tip: Display the control vantage point in its own column, first. When the control cell is red, everyone reading the board knows the network is not the suspect, and the room gets quieter.
Avoiding storms when one dependency breaks every journey
Journeys share dependencies. Nearly every internal application logs in against the same directory or identity provider, resolves names through the same DNS, and reaches its servers through the same core. When one of those breaks, every journey fails at once. A naive alert design sends one high-severity alert per journey per vantage point. Eight journeys times five runners is forty pages for one broken directory server. That is an alert storm, and it is exactly the loud embarrassment from the introduction, now wearing experience-monitoring clothes.
The defense is dependency thinking: recognizing that a set of alerts sharing a cause should be delivered as one. Two techniques handle most of it. First, group by step signature. If every journey is failing at its login step within the same minute, the shared identity provider is the story. In that case, the alert should say "all journeys failing at login; identity provider suspected" once, rather than eight times. Include the list of affected journeys inside it. Second, declare the shared dependencies explicitly and suppress the children when the parent is known to be down. If the infrastructure check on the identity provider has already fired, the journey alerts become supporting evidence attached to it, not separate pages. Modeling those relationships so that alerts fire for causes rather than symptoms is covered in the guide on dependency mapping for alerts.
| Signature across journeys | Likely shared dependency | Deliver as |
|---|---|---|
| All journeys fail at login | Directory or identity provider | One alert naming the provider, journeys listed |
| All journeys fail at the first step, all vantage points except control | DNS, core network, data-center firewall | One alert to the network team |
| All journeys fail from one vantage point | That site's link, or its runner | One alert naming the site; check runner health first |
| Journeys for applications on one database server fail at their data steps | That database server | One alert naming the server |
| One journey fails, others fine | That application | The normal per-step alert |
An alert-design table per journey type
Different kinds of journey call for different defaults. The table below is a starting point; adjust the numbers to your baselines and the severities to your business.
| Journey type | Fail severity | Slow rule | Confirmation | Route to |
|---|---|---|---|---|
| Internal web app (login, search, write) | High for login and core steps; low for cosmetic steps | 3 runs above 3x step baseline | 2 runs, 2 vantage points | App owner + network team |
| API sequence (token, call, verify) | High; APIs feed other systems | 3 runs above 2x baseline (APIs should be stable) | 2 runs, 2 vantage points | API owner + network team |
| Mail round trip (send, receive) | Medium; delays tolerated for minutes | Round trip over an agreed minutes budget, 2 runs | 2 runs | Messaging team + network team |
| File share (map, write, read, delete) | High during business hours, low overnight | 3 runs above 3x baseline | 2 runs, 2 vantage points | Storage or server team + network team |
| Directory bind and lookup | High; everything depends on it | 3 runs above 2x baseline | 2 runs, 2 vantage points | Identity team + network team; suppresses child journeys |
| Public customer portal from outside | High at all hours | 3 runs above 3x baseline from outside | 2 runs, 2 outside points | App owner + network team + whoever owns the edge |
Where to go next
Experience alerts say what users feel. Infrastructure alerts say what broke. A monitoring system needs both, side by side. Build the three journey alert types on per-step thresholds from real baselines. Confirm with a second run and a second vantage point before anyone is paged. Set severity by what the failing step costs the business. Route to both plausible owners with the evidence and an interpretation attached. Group by shared dependency so that one broken directory server produces one alert instead of forty.
Alert rules only stay accurate while the journeys and thresholds behind them are maintained, which is the subject of keeping synthetic tests honest. For the network-side evidence that experience alerts point at, throughput testing done right covers measuring a suspect path. Everything in this series starts from the Synthetic Transactions & End-to-End Testing pillar.
Put this into practice with SysPeep Monitor
Experience alerts tell you a step is broken; the infrastructure side has to answer why, and quietly. SysPeep Monitor's thresholds, dependency rules, and alert rules are built to keep noise down. When every journey fails at login, the alert about the directory server arrives once. The affected devices appear beside it on the same dashboard. The free trial installs on any Windows machine.
Download Free TrialLearn About SysPeep