Home › Topics › Monitoring in the Team Chat (ChatOps) › Chat During Incidents

Chat During Incidents: Threads, Timelines, and Handoffs


An incident is the moment ChatOps either pays for itself or falls apart. On a good day three people work a real outage in one thread. The alerts appear in order, and each person says what they are trying. The lead makes decisions out loud, and someone takes over at lunch with a clean summary. When it is over the thread is the timeline. On a bad day the same outage is scattered across the alerts channel, two direct messages, and a voice call nobody wrote down. It is also in a general channel where the boss is still waiting for an update. The write-up afterward takes longer than the outage did, because nobody can say what happened when.


The difference is a handful of habits that cost nothing during the incident and save hours after it. I learned them by writing the bad-day version up, more than once, from memory and screenshots. This guide covers them. It explains one thread per incident and what goes in it, the timeline the bot posts, and the notes humans add. It covers who plays which role and how they say so, and the handoff message that lets someone leave without losing the thread. It also covers a pinned status line, the discipline of not letting the conversation fork, and how the finished thread becomes the post-incident review's raw material. It ends with two templates you can paste today and the honest small-team version.


In this guide

  • The incident thread
  • The bot-posted timeline, and the human notes
  • Roles in chat
  • Handoffs
  • Pinning current status
  • Avoiding side conversations
  • Exporting the thread into the review
  • Two templates
  • The small-team version
  • Where to go next

The incident thread


A thread is a reply chain attached to one message. And the rule for incidents is absolute: one thread per incident, and everything about that incident goes in it. Not the alerts channel, not a direct message, not a new top-level post every time something changes. One thread. The top-level message is the incident's headline — what is broken, since when, who leads — and the thread underneath is the working room.


Where does the thread live? Two options, and the choice depends on team size. The simplest is to use the thread on the alert message itself. The alert that started the incident already has a thread, so work there. This suits small teams and short incidents. The more scalable option is a dedicated incidents channel. There, a human (or the bot, on command) posts a new top-level message for each incident, and the thread hangs off that. The channel's main view then becomes a list of incidents, newest first, with each headline showing current status. That is exactly what someone joining late or a manager checking in needs to see. When an alert grows into an incident, the lead posts the headline in the incidents channel. The lead then drops a link to it in the alert's thread, so both paths lead to the same room.


Give the incident a short name in the headline — "warehouse switch outage," "mail delivery delays" — and use it consistently. A name is what lets people refer to it in the handoff, the status line, and the review without ambiguity. "That switch thing" is not a name; it is three incidents.


When does an alert become an incident?

Not every alert deserves a thread of its own in the incidents channel. Most are handled in a few replies under the alert message and closed. The promotion happens when any of these becomes true. Users are affected and are noticing, or more than one alert is part of the same story. More than one person is needed. Or the fix is going to take longer than the time it takes to type the opener. Any one of them is enough. Promote early rather than late. An incident thread that turns out to be a ten-minute non-event costs one message. A real outage worked in the wrong place costs the whole record.


The bot-posted timeline


A bot — a small program with an account in the chat tool — can post into the incident thread automatically whenever something it knows about changes. Each new related alert, each recovery, each acknowledgement, each escalation appears in the thread the moment it happens, timestamped, with no human effort. That stream is the timeline's skeleton: the objective record of what the monitoring system saw and when. Nobody has to remember to write "the second switch went down at 10:19"; it is already there.


Humans add the flesh. Observations, hypotheses, decisions, and actions are typed into the same thread as they happen, and the chat tool timestamps them automatically. The important habit is to type it when you do it, not afterward. "Reseating the transceiver now" at 10:25 is a timeline entry. "I think I reseated it around half ten" at 4 PM is a guess. A few short labelled conventions make the human entries scannable next to the bot's:


  • OBSERVED: a fact you saw. "OBSERVED: no link light on port 24."
  • THEORY: what you think is going on. "THEORY: transceiver failed after last night's firmware update."
  • DECISION: a choice the lead made and why. "DECISION: swap the transceiver before rolling back firmware, because it is faster to test."
  • ACTION: something you are doing to a system, right now. "ACTION: replacing SFP in port 24."
  • STATUS: a summary for people arriving late. "STATUS: warehouse still down, cause suspected, fix in progress, next update 10:45."

Here is how the two streams look interleaved, bot lines and human lines in one thread. The SFP in the entries is the small plug-in transceiver module that sits in a switch port:


[bot]    10:12  CRITICAL  Uplink down — sw-core-1 port 24 (main office). 60 users, 3 dependents suppressed.
[bot]    10:14  Acknowledged by Sam. Owner: Sam.
[Sam]    10:16  OBSERVED: no link light on port 24. Remote end shows light.
[bot]    10:19  WARN  Latency high — building B link (traffic on backup path).
[Sam]    10:21  THEORY: SFP on our side failed. Firmware was updated on this switch last night.
[Sam]    10:25  DECISION: reseat the SFP first (2 min), then swap if no link. Rollback firmware only if swap fails.
[Sam]    10:26  ACTION: reseating SFP port 24.
[Sam]    10:29  OBSERVED: no change after reseat.
[Priya]  10:31  HANDOFF: taking over from Sam (lunch). See summary below.
[Priya]  10:33  ACTION: swapping SFP with spare from cabinet 2.
[bot]    10:38  RECOVERED  Uplink up — sw-core-1 port 24. Duration 26 min.
[bot]    10:39  RECOVERED  Latency high — building B link.
[Priya]  10:40  STATUS: resolved. Cause: failed SFP. Review scheduled Friday. Old SFP labelled and set aside.

Read that back and notice what it already is: a complete, timestamped account with the monitoring facts and the human reasoning side by side. That is not a by-product of the incident; it is the most valuable thing the incident leaves behind.


Roles in chat


Even a three-person incident has three jobs, and they go badly when everyone does all of them at once. (Everyone doing all three at once is called "a conference call.") The jobs are the same ones any incident process names. What chat adds is the need to say out loud who holds each one, in the thread, so nobody has to guess.


The lead

One person owns the incident: makes the decisions, sets the next-update time, and is the person the thread addresses questions to. The lead does not have to be the most senior person or the one doing the fixing. The lead is the one keeping the picture whole. The lead's name goes in the headline and is updated when it changes.


The communicator

One person talks outward — to the users, the service desk, the management channel, whoever is waiting for news. That keeps the lead and the investigators from being interrupted for updates. The communicator reads the thread and translates; they do not need to understand the fix, only the state. The guide on communicating during an outage covers what those outward updates should say and how often. In chat, the communicator's job is to take the STATUS lines from the thread and carry them to the people who cannot see it.


The investigators

Everyone else who is actually looking at systems. Their job in the thread is to post OBSERVED, THEORY, and ACTION lines as they go. They must also read the thread before acting so they do not repeat what someone tried five minutes ago. Two investigators pulling the same cable is the classic failure that a thread prevents — but only if both of them typed "ACTION" before they walked to the closet.


On a small team, one person may hold two or all three roles. That is fine; the point is not the headcount but the sentence. "I'm leading and communicating; Sam, you investigate" takes five seconds to type and removes an hour of confusion.


Handoffs


People go to lunch, go off shift, or are simply better at something else. A handoff is the moment ownership moves from one person to another, and it is where incidents lose the most information. The outgoing person's mental picture walks out of the room with them. It includes what has been tried, what has been ruled out, and what they were about to do next. The fix is a handoff message in the thread, posted by the person leaving and acknowledged by the person taking over. It contains the whole picture in a fixed shape:


  • State: what is still broken, what has recovered, who is affected right now.
  • Tried: every action taken and its result, including the ones that did nothing. "Reseated SFP, no change" saves the next person from doing it again.
  • Ruled out: theories that have been tested and dismissed, and how.
  • Next: what the outgoing person was about to do, and why.
  • Who else is involved: the vendor case number, the user who first reported it, the manager waiting for the update.

The incoming person replies "taking over" in the thread, edits the headline to show the new lead, and the handoff is done. My own worst handoff was the words "back in twenty"; the next person retried everything. This is the same discipline that a shift change on an on-call rota needs. And the guide on on-call handoffs covers that broader case; inside an incident, the message above is the whole of it.


Pinning current status


A long thread is a good record and a bad summary. Someone joining at 10:35 does not want to read forty entries; they want one line that says where things stand. Most chat tools let you pin a message so it sits at the top of a channel or thread regardless of how much scrolls past. The lead pins one STATUS message at the start and edits it in place as things change, rather than pinning a new one each time. The pinned line always answers four questions: what is affected, what is being done, who leads, and when the next update is due.


STATUS (updated 10:33)  —  warehouse switch outage
affected:  warehouse users and scanners, ~60 people; building B on backup link
doing:     swapping SFP in sw-core-1 port 24
lead:      Priya  (took over from Sam 10:31)
next:      update by 10:45 or on change

This is also what the communicator copies outward, so the team and the users are reading the same sentence. The moment the incident resolves, the pinned status becomes the closing line: resolved at, cause, follow-up scheduled.


Avoiding side conversations


The single most common way an incident thread fails is that it stops being the only conversation. Two investigators start a direct message to "not clutter the thread." The lead takes a phone call from the vendor. Someone answers the boss in the general channel. Each of these feels helpful, and each removes facts from the record and from the people who need them. One thread, not three.


Ashgrove Housing ran its first big outage, the tenant portal down on rent day, in the general channel, because that was where everyone already was. Ninety messages went past in forty minutes. They included questions from the service desk, three theories, and two people each asking whether anyone had restarted the web server. The manager was asking for an update every six minutes, because nobody had given one. The person who found the fix posted it at 11:02, a one-line note about an expired certificate on the portal's web server. By 11:04 it had scrolled off everyone's screen under a fresh round of "any update?" The lead found it at 11:31 by searching for the word "certificate," and the portal was back at 11:40. The review counted ninety-one messages and no decisions. They now run incidents in one thread with a pinned status line, and the manager gets that line, on time, every fifteen minutes.


Three habits hold the line. First comes the "bring it back" rule. Anything discussed outside the thread — a call, a hallway conversation, a direct message — is summarised into the thread. Whoever had it does that immediately, in one line. "Vendor call: confirmed known SFP batch issue, replacement shipping" is enough. Second, a voice call during an incident gets a scribe. This is one person on the call whose only job is typing what is decided into the thread as it is said. Third, the communicator answers questions from outside outside, so that the thread never fills with "any update?" from people who are not working the problem. If someone who is not on the incident posts in the thread, the polite move is to answer in the general channel. Leave the thread for the work.


Tip: the test for whether a conversation belongs in the thread is whether the person writing the post-incident review would want to know about it. A decision, an observation, a vendor's answer: yes. Two colleagues agreeing which of them walks to the closet: also yes, because it is an ACTION. "Are you okay, you sound stressed": no, and kindness is allowed in a direct message.


Exporting the thread into the review


A post-incident review is the meeting, and the document, where the team works out what happened and what to change. Its foundation is the timeline. The timeline's usual problem is that it has to be reconstructed after the fact from alert history, graphs, logs, and people's memories. A well-kept incident thread removes most of that work: the thread is the raw timeline. The bot's entries supply the monitoring facts with exact times. The human entries supply the reasoning and actions with times the chat tool recorded, not times anyone remembered.


Export it the same day, while it is fresh. Most chat tools can copy a thread or export a channel as text. Paste the result into the review document as an appendix, then build the review's clean timeline table from it. The guide on reconstructing the timeline from monitoring data covers reconciling that thread with logs and graphs and marking what is known against what is inferred. The thread's job is to make that reconciliation short. Check two things as you export. Check that the thread's times and the monitoring system's times agree. Clock drift between the chat tool and the monitor is a real thing. Also check that any facts brought in from outside the thread — the vendor call, the user's original report — were actually summarised in. If they were not, add them now, marked as reconstructed. "Reconstructed" is the review's polite word for "remembered."


Two templates


Both are meant to be pasted into the thread and filled in. Keep them pinned in the incidents channel so nobody has to remember the shape at 2 AM. The ${...} markers are the blanks.


The incident-thread opener

INCIDENT: ${SHORT_NAME}
started:   ${START_TIME}   (first alert or first report)
affected:  ${WHO_AND_WHAT}
severity:  ${SEVERITY}
lead:      ${LEAD_NAME}      communicator: ${COMMS_NAME}
links:     alert thread ${ALERT_LINK}  ·  graph ${GRAPH_LINK}  ·  runbook ${RUNBOOK_LINK}
status:    ${ONE_LINE_STATUS}         next update: ${TIME}

Rules for this thread: everything goes here. Label lines OBSERVED / THEORY /
DECISION / ACTION / STATUS. Summarise any call or DM into the thread at once.

The handoff message

HANDOFF: ${OUTGOING} → ${INCOMING}   at ${TIME}
state:      ${WHAT_IS_STILL_BROKEN};  ${WHAT_HAS_RECOVERED}
tried:      ${ACTION_1} → ${RESULT_1}
            ${ACTION_2} → ${RESULT_2}
ruled out:  ${THEORY} (${HOW_TESTED})
next:       ${PLANNED_NEXT_STEP}  because  ${REASON}
others:     vendor case ${CASE_ID};  reporter ${USER};  waiting on ${PERSON}
incoming:   reply "taking over" and update the pinned STATUS lead line.

Neither template needs a bot. They are text, and the discipline is in using them every time, especially on the small incidents where it feels unnecessary. That is where the habit is built for the large ones.


The small-team version


If you are two or three people, all of the above collapses into something you can start using on the next problem without any setup. A thread on the alert message, or a fresh post if there is no alert. A pinned status line that one of you edits. The five labels, typed by hand. A handoff message if anyone leaves. A copy of the thread pasted into a document afterward. There is no bot posting the timeline. You type "10:19 second switch alerting" yourself, which takes four seconds. The chat tool stamps the time on it anyway.


What you give up without a bot is the automatic skeleton. On a small network with a handful of alerts per incident that is a small loss. What you keep is everything that matters. You keep one place, one story, a status anyone can read, and a record that is finished the moment the incident is. A small team that runs incidents this way for a few months tends to find something when the bot eventually arrives. It has very little to add except saving them the typing.


Where to go next


You now have the whole incident practice for chat. You have one thread, a timeline the bot starts and humans finish, and roles said out loud. You have a handoff that carries the picture and a pinned status that answers the four questions. You have one conversation rather than three, and an export that turns the thread into the review's raw material. The practice rests on the earlier layers of this pillar: alert messages that carry context and actions taken from chat. It also needs a channel quiet enough that an incident thread is not competing with wallpaper. The rest of Monitoring in the Team Chat (ChatOps) covers each of those. If you take one thing from this guide, make it the labelled lines: OBSERVED, THEORY, DECISION, ACTION, STATUS. They cost nothing and they are the difference between a thread and a timeline. Keep them and the write-up takes less time than the outage did, which is the right way round.


Put this into practice with SysPeep Monitor

The bot's half of the timeline is only as good as the monitoring behind it. That includes exact alert times, recoveries, and the dependency picture that says who was affected. SysPeep Monitor keeps that alert history alongside live dashboards, network maps, and reports in a single web console. Its thresholds, dependency rules, and alert rules are designed so the alerts that reach your team are the ones worth a thread. Those alerts reach you by email and other notifications. SysPeep Monitor does not post into chat itself. Check the product documentation for its notification options, and try the free trial on any Windows machine.

Download Free TrialLearn About SysPeep