Alerts With Context: Making a Chat Message Actually Useful
A monitoring alert lands in the team channel. It says DOWN: sw-03. Within a minute four people have asked four questions. Which sw-03 (there are two sites)? How long has it been down? Is anything behind it? Is anyone on it? The alert took one line to send and ten minutes of the team's time to decode. Then it clears, silently, and someone asks a fifth question: is that fixed, or did it just stop alerting?
A message that generates questions has failed at its only job. I have sent that exact message myself, for years. The fix is not a bigger message. It is a message with the right fields, laid out the same way every time, that answers the questions before they are asked. This guide gives you that message. It covers the eight things it should contain and how to lay it out so eyes learn where to look. It covers how to keep the follow-ups in a thread and how to update it in place instead of posting five more. And it covers how to close the loop when the problem clears. You will finish with a checklist you can hold every alert against. You will also have two practical ways to get the fields into the message with the tools you already have.
In this guide
- Why a bare alert fails in chat
- The message anatomy: eight fields
- Formatting for chat
- Threads, in-place updates, and the recovery message
- Before and after: the same alert, rewritten
- The field checklist
- Getting context into the message
- Where to go next
Why a bare alert fails in chat
A bare alert is one that names the condition and nothing else. It is the natural default of most monitoring tools, because the tool knows what crossed the threshold and assumes you will look up the rest. In a console, that assumption is fine: the alert sits next to the device page, the graph, and the history. In chat it is stranded. The reader has no context except the message itself, and the message was written for a screen it is not on.
The cost is not just the questions. A bare alert makes people hesitate, because acting on it means first opening the console to find out whether it matters. Hesitation in a channel is contagious: everyone waits to see if someone else opens the console. The alert that could have been claimed in ten seconds sits unowned for ten minutes. A good chat alert is one that a person can read, judge, and claim without leaving the chat window. That is the design target for everything below.
Meridian Parts learned this from an alert that read, in full, "CRITICAL: srv-12." There were three servers called srv-12, one at each depot, because each depot had been set up by a different contractor. Three admins opened three consoles, found three healthy servers, and each posted "all fine here" within four minutes of one another. The fourth srv-12, a retired box at head office that a backup job still pointed at, stayed down for two days. The review described the naming scheme as "historically layered." The alert now carries a site and a role, and the backup job has been told the news.
If the alert itself is badly designed — a threshold that fires on noise, a condition nobody would act on — no amount of formatting will save it. The guide on the anatomy of a good alert covers that underlying question. This guide assumes the alert deserves to exist and asks only how it should look when it arrives.
The message anatomy: eight fields
Every useful chat alert answers the same eight questions. Not all of them will be available for every alert — "what changed" is often unknown. But the layout should have a place for each, so that a missing field is visibly missing rather than silently absent.
1. What: the condition in plain words
Say what is wrong the way you would say it to a colleague across the desk. "Uplink down" beats "ifOperStatus changed to down(2)." "Disk almost full" beats "threshold breach on volume C: used percent." Keep the technical detail; put it lower down. The first words of the message are read by everyone, including the manager who joined the channel to stay informed, so they must be plain.
2. Where: device, site, service
The device's name, the site or building it is in, and the service it belongs to if that is more meaningful than the name. "sw-03" is ambiguous; "sw-03 (warehouse, second floor closet) — wireless uplink" is not. If your naming convention already encodes the site, good, but do not assume every reader has memorized it. The person reading at 8 AM on their phone may be new. The switch knows exactly which closet it is in and has never told anyone.
3. How bad: severity and blast radius
Severity is the label the monitoring system assigns — critical, warning, informational. It belongs at the very front of the message, ideally as a word and a colour or symbol that can be spotted while scrolling. Blast radius is the part most alerts miss: who or what is affected by this. "Uplink down, roughly sixty users and the warehouse scanners are behind it" turns a technical event into a business fact. Blast radius is the field that decides how fast people move. If your monitoring system tracks dependencies (which devices sit behind which), it already knows the blast radius; the message should say it.
4. Since when
The time the condition started, and how long ago that is. Both, because "since 10:12" is precise but "4 minutes" is what the brain wants. Duration is also the field that distinguishes a fresh problem from one that has been quietly alerting for an hour. In the latter case, everyone assumed someone else was on it.
5. What changed, if known
Most outages follow a change. Suppose the monitoring system, a change calendar, or a configuration-backup tool knows that something was modified on this device recently. That one line is then the most valuable thing in the message. For example, "firmware updated on this switch last night" sends the responder straight to the likely cause. When nothing is known, say so explicitly, as "no recorded changes," so that the absence is information, not an omission. "No recorded changes" is also, quite often, the first clue that the change was not recorded.
6. The graph link
A link that opens the relevant graph in the monitoring console, already scoped to this device and this metric and a sensible time window. One click from the chat message to the picture. The difference between a link and no link is the difference between "let me look" and "let me find it." The second one is where responders get lost.
7. The runbook link
A runbook is the written procedure for handling a particular kind of alert. It covers what to check first, what usually causes it, what to do and what not to do. Linking it from the message means the junior person on the rota can act with the same first steps as the senior one. The Runbooks pillar covers writing them; here, the rule is simply that every alert type with a runbook links to it. And every alert type without one is a runbook waiting to be written.
8. The owner or on-call name
Who is expected to pick this up. For a critical alert that is the on-call person by name, pulled from the rota. For a warning it may be the team that owns the device. Naming a person in the message does two things. It tells everyone else they can stand down, and it makes the named person's silence visible if they do not respond. "Everyone," by contrast, has never answered an alert. Once someone acknowledges, this field changes to their name; more on that below.
Formatting for chat
Chat messages are read on a phone in a queue, on a laptop between meetings, and in a rushed scroll after lunch. The format has to survive all three. Three rules.
One line that says it all. The first line should be readable on its own: severity, condition, device, site. If the reader sees nothing else — and a notification preview often shows nothing else — they know whether to open it. Everything below the first line is detail for the person who decides to act.
Consistent layout so eyes learn where to look. Same fields, same order, same labels, every time, for every alert type. After a week, people stop reading alerts and start scanning them, which is faster and more reliable. The moment you vary the layout per alert type, that skill breaks. Boring consistency is the goal.
No walls of text. Eight short labelled lines is the maximum. Raw log excerpts, full device output, and the monitoring system's boilerplate paragraph belong behind a link or in a thread reply, never in the message. If the message needs a scroll bar, it has failed the phone test.
Here is a layout that satisfies all three, written as a template. The ${...} markers stand for values the monitoring system or your formatter fills in.
${SEVERITY} ${CONDITION} — ${DEVICE} (${SITE})
where: ${DEVICE} · ${SERVICE_OR_ROLE}
impact: ${BLAST_RADIUS}
since: ${START_TIME} (${DURATION})
changed: ${RECENT_CHANGE_OR_"no recorded changes"}
links: graph ${GRAPH_URL} · runbook ${RUNBOOK_URL}
owner: ${ON_CALL_NAME}
status: ${STATE} ← edited in place as the alert changes
Eight lines, labels aligned, links on one line, status last. The first line is the notification preview; the status line is the one that changes over the alert's life.
Threads for follow-ups
A thread is the chat tool's way of attaching replies to one message so they stay together. The rule is absolute: replies about an alert go in its thread, not in the channel. "Looking now," "port shows no light," "swapped the transceiver, waiting," "resolved" — all of it in the thread. The channel stays a clean list of alerts; each alert carries its own story underneath.
This matters more than it sounds. When replies go in the channel, two alerts on the same afternoon produce one tangled conversation. The third alert scrolls the first out of view. Anyone arriving late cannot tell which remark belongs to which problem. Threads fix all three at once. They also make the record. Later, the thread on an alert message is the complete account of what happened, in order, without anyone having to write it up.
Updating in place
An alert has a life: it fires, maybe it escalates from warning to critical, someone acknowledges it, and eventually it clears. The naive integration posts a new message for each step. So one problem becomes five messages and the channel fills with the same alert wearing different hats. The better pattern is editing in place: the bot, or the monitoring tool's chat integration, changes the original message as the state changes. The severity word updates, the status line reads "ACKED by Sam 10:14," and when it clears the whole card turns to a recovered state. One message, one problem, always current.
Editing in place needs two things. The integration must be able to edit messages it posted, which most chat tools allow for bots but not for plain email-to-channel delivery. And the monitoring system must send an update, not just the initial alert — a notification on acknowledgement, on escalation, and on recovery. If either is missing, the fallback is to post updates as replies in the alert's thread rather than as new channel messages. The channel still shows one message per problem; the thread shows its history. That is nearly as good and needs no special abilities.
The recovery message
When the condition clears, say so, briefly. The recovery message has one job: close the loop so nobody wonders. It should name the alert it closes, the time, the duration of the whole episode, and — if a human resolved it — who and what. It should not be a second full card. If you edit in place, the recovery becomes the card's final state. If you cannot, it is a one-line thread reply plus a reaction on the original message. Either way, someone glancing at the channel should be able to tell "open" from "closed" without reading anything.
RECOVERED Uplink down — sw-core-1 port 24 (main office) duration: 26 min (10:12–10:38) resolved: Priya · replaced transceiver status: CLOSED
Tip: resist the urge to send recovery messages for alerts that clear themselves in under a couple of minutes. If the monitoring system's threshold has a "for at least N minutes" condition, a flapping interface never becomes a chat message at all. In that case, you have solved the noise where it starts instead of decorating it. The quiet channel begins in the alert rule, not the formatter.
Before and after: the same alert, rewritten
Here is the alert from the opening scene as it usually arrives, followed by the same event with context. Read both as if you were on your phone, in a queue, deciding whether to act.
DOWN: sw-03 Host sw-03 is in state DOWN. Check duration: 2 checks. Please investigate.
Four questions, no answers. "Please investigate" is the tool's way of saying it has no idea either. Now the rewrite.
CRITICAL Switch unreachable — sw-03 (warehouse, 2nd floor closet) where: sw-03 · access switch for warehouse scanners + 2 APs impact: ~25 users and all handheld scanners; 2 APs suppressed as dependents since: 14:02 (3 min) changed: no recorded changes on sw-03 in the last 7 days links: graph [reachability, last 2h] · runbook [access switch unreachable] owner: on-call: Sam status: OPEN — unacknowledged
The first line now carries severity, plain condition, device, and site, so the notification preview alone is enough to know this matters. "Where" identifies the switch's role, which is what a human actually cares about. "Impact" comes from the monitoring system's dependency map and turns the event into a business fact. The scanners are down, and the two access points behind the switch have been suppressed rather than alerting separately. "Since" gives both the clock time and the duration. "Changed" says explicitly that nothing is recorded, which points the responder away from a config cause and toward power or cabling. Two links put the graph and the procedure one click away. "Owner" names a person, and "status" is the line that will be edited as the alert is acknowledged and cleared.
Notice, too, what the rewrite removed: "Please investigate" and "Check duration: 2 checks" are tool boilerplate that tell the reader nothing. Every line in the rewrite earns its place. That is the standard.
The field checklist
Hold every alert type you route into chat against this table. "Source" says where the value typically comes from, because the practical question is never "should we include it" but "where do we get it."
| Field | Answers | Typical source | If unavailable |
|---|---|---|---|
| Severity | How urgent? | Alert rule | Never omit; fix the rule |
| Condition (plain) | What is wrong? | Alert rule name, rewritten | Rename the rule in plain words |
| Device and site | Where? | Inventory, device properties | Add a location field to the inventory |
| Service or role | What is it for? | Device group, tags, description | Tag devices by role; takes an afternoon |
| Blast radius | Who is affected? | Dependency map, group membership | Say "unknown" explicitly; build dependencies later |
| Since / duration | How long? | Alert start timestamp | Always available; include both forms |
| What changed | Likely cause? | Config backups, change calendar | Say "no recorded changes" |
| Graph link | Show me | Console URL with device and metric | Link the device page at minimum |
| Runbook link | What do I do? | Documentation store | Link a stub; write the runbook this week |
| Owner / on-call | Whose is it? | On-call rota, device group owner | Name the team, not "everyone" |
| Status line | Open, acked, closed? | Alert state, edited in place | Use thread replies for state changes |
You will not have every source on day one. The right order starts with severity, condition, device and site, and since — the four fields the monitoring system already has. Then come links, then owner, then blast radius and changes as the inventory and dependency map mature. Each added field is a measurable reduction in questions asked in the channel, which is the honest way to know the work is paying off.
Getting context into the message
All of this is only useful if you can actually produce the message. There are two generic ways, and most teams start with the first and move to the second only when they need to.
A template in the monitoring tool's notification
Nearly every monitoring system lets you customize the text of a notification with placeholders. These include the device name, the alert name, the severity, the time, and sometimes a link back to the console. Suppose the tool can send its notification somewhere the chat tool can receive it (email into a channel is the most common route). Then the whole message format lives in one template inside the monitoring system. In that case, you can build most of the eight fields with no code. The limits are that placeholders vary by tool, the blast-radius and what-changed fields may not be exposed, and email delivery usually cannot edit a message in place. For a small team this is still the right first step. It gets the first line, the device, the site, the duration, and the links into chat this afternoon.
A small formatter script
When the template cannot reach far enough, put a small program in between. The monitoring system sends its notification to the script by email, by a callback, or by whatever mechanism it offers. A callback is the monitoring system calling a web address you give it. The script looks up the extra fields: the site from an inventory file and the dependents from the monitoring system's own data. It looks up the last change from the config-backup store and the on-call name from the rota. It assembles the message in the fixed layout and posts it to the channel through the chat tool's bot interface. It remembers the message identifier so it can edit the same message later when the acknowledgement or recovery arrives. In pseudocode, the whole thing is short:
on notification(alert):
site = inventory.lookup(alert.device).site
role = inventory.lookup(alert.device).role
impact = monitor.dependents_of(alert.device)
change = configstore.last_change(alert.device, days=7)
oncall = rota.current(alert.team)
text = render(TEMPLATE, alert, site, role, impact, change, oncall)
if alert.state == "new":
msg_id = chat.post(channel_for(alert.severity), text)
store.save(alert.id, msg_id)
else:
msg_id = store.load(alert.id)
chat.edit(msg_id, text) # acked, escalated, or recovered
This is not a product. It is a few dozen lines that any admin comfortable with a scripting language can write and, more importantly, read a year later. The Monitoring APIs and Webhooks pillar explains the general mechanisms a monitoring system uses to hand data to a script like this. Whichever mechanism yours offers, the design above stays the same: one template, one lookup per missing field, one message per alert, edited rather than repeated.
Where to go next
A chat alert that carries its own context is the single upgrade that turns "alerts in a channel" into something the team relies on. You now have the eight fields, a layout that survives a phone screen, and the thread rule. You have in-place updates, a closing message, a checklist, and two ways to build it. Build the first line this afternoon and add the rest as the inventory catches up. The next question is what people do once the message arrives: acknowledging, snoozing, escalating, and asking the bot questions. The rest of Monitoring in the Team Chat (ChatOps) takes that up, along with keeping the channel quiet enough that these good messages are read. Get the card right and nobody has to ask the fifth question.
Put this into practice with SysPeep Monitor
Most of the eight fields in this guide are things the monitoring system already knows. SysPeep Monitor auto-discovers devices and interfaces into an inventory. It tracks the dependencies that become an alert's blast radius and keeps the graphs a message should link to in a single web console. Its thresholds, dependency rules, and alert rules are designed to keep noise down before anything is sent, by email and other notifications. Check the product documentation for its notification options, and try the free trial on any Windows machine.
Download Free TrialLearn About SysPeep