From Reactive to Proactive: The Moves That Actually Change the Level
Reactive is the level where the alerts exist and you still hear about most outages from a person. It is the level most small teams live at, and it is a perfectly respectable place to live. The monitoring is real, someone responds, and the bad days are shorter than they were. What it is not is calm. Every week has one morning where the phone rings before the console does. And every one of those mornings feels like it should have been avoidable.
It usually was, and not by buying anything. The distance between reactive and proactive is a handful of practices, none of them exotic. Most teams have heard of them. Few have done them in the right order. Done in the wrong order they fight each other. That is why so many teams have tried "getting proactive" twice and are still reactive. I have done it in the wrong order myself, and the second time in the right one. The second time took a quarter.
By the end of this guide you will know the seven moves that change the level. You will know why each one makes the next one possible. You will know how to lay them across ninety days with evidence of "done" for each. You will also know roughly what they cost in hours per week. That is stated as an estimate you can check against your own team.
In this guide
- Reactive is not a failure
- The seven moves, in order
- Why the order matters
- The ninety-day plan
- What it costs in hours
- The reactive-to-proactive checklist
- Where to go next
Reactive is not a failure
A team at level 1 has already done the hard part. In the model this guide belongs to — four levels and six dimensions, laid out in the four levels, plainly — reactive means alerts exist, reach a person, and get acted on. Getting there from nothing is the work nobody thanks you for. Its result is that things stop being invisible. And the reward for making things visible is a longer list.
What keeps a team reactive is not effort. It is that the monitoring is watching the wrong layer. Ping says the box is up; the users say the service is down; both are correct. The alert count is high because one switch reboot produces thirty notifications, so the one that matters is read last. Nobody looks at graphs except during an incident, so the slow problems arrive as surprises. And the fix for every alert lives in one person's head, which is fine until that head is on holiday.
Every one of those is a practice, not a purchase. If you have not yet scored yourself, the self-assessment will tell you which of the seven moves below you have already made without calling them that. Most reactive teams have made one or two. (Nobody has made zero. Nobody has made all seven, either, or they would not be reading this.)
The seven moves, in order
These are the practices that move a team from level 1 to level 2, in the order that works. Each has a plain description, a reason, and the evidence that it is done. The evidence matters, because "we are doing dependencies" and "dependencies are configured on every downstream device" are different sentences. Only the second one changes the level.
1. Define "down" and monitor the service, not the ping
Write one sentence per service that says what "down" means to the people who use it. For example: "the mail service is down when a message cannot be sent from the test mailbox within sixty seconds". Then build a check that tests exactly that. A service check does what a user does. It sends the message, fetches the page, or opens the share instead of asking whether the box answers a ping. Defining down has the method. Evidence of done: a written list of services, each with its definition of down and the name of the check that tests it.
2. Dependencies, so one fault is one alert
Tell the monitoring which devices sit behind which. A dependency is the recorded fact that the annex switch is only reachable through the core switch. So when the core fails, the annex is not alerted separately. This is the single largest cut in alert volume most teams ever make, and it costs an afternoon. Dependency thinking explains how to work out the tree. Evidence of done: the last upstream failure in the alert history produced one notification, not a page of them.
3. Baselines, and thresholds reviewed on a calendar
A baseline is a record of what normal looks like for a metric. For example, the link runs at forty per cent on a weekday afternoon, and the database server's memory sits at seventy. A threshold is the line an alert fires at, and a threshold set without a baseline is a guess with a bell attached. Collect baselines for the key metrics, set thresholds from them, and put a recurring entry on the calendar to revisit both. The baselining pillar covers collecting them and the threshold review ritual covers the calendar entry. Evidence of done: a baseline figure per key metric, and a past review with notes.
4. A runbook for every alert that fires more than once a month
A runbook is a short written procedure for what to do when a particular alert fires. It says what to check first, what usually fixes it, and who to tell. You do not need one for every alert. You need one for every alert that fires often enough that a second person will meet it. First runbooks shows the shape; keep each to a page. Evidence of done: the list of alerts that fired more than once last month, beside a folder that has a file for each.
That list is a query, if your monitoring database can be read. Names vary by product, so treat this as the shape and check the product documentation for yours:
SELECT rule_name, COUNT(*) AS fired
FROM alert_history
WHERE fired_at >= DATETIME('now', '-30 days')
GROUP BY rule_name
HAVING COUNT(*) > 1
ORDER BY fired DESC;
Every row is either a runbook to write or an alert to fix so that it stops firing. Both are progress. The top three rows are usually the same three alerts everyone already complains about, and writing their runbooks takes an hour.
5. A weekly fifteen-minute look at the graphs
Put fifteen minutes on the calendar, the same time every week, to open the graphs for the key metrics and look for lines going somewhere. Not the dashboard; the graphs, with a thirty-day range. "Going somewhere" means a line rising towards its threshold, a Monday that looks like a Friday, or a flat line that used to move. This is the practice that catches the disk filling and the link creeping up before either becomes an incident. The weekly trend review lays out what to look at. Evidence of done: a one-line note per week saying what was looked at and what, if anything, was found.
6. Mini post-incident reviews
After each incident, spend fifteen minutes writing down what happened, when it was noticed and by whom, what fixed it, and one thing to change. A mini review is that paragraph and nothing more; the point is that it happens every time, not that it is thorough. Mini reviews gives the template. Evidence of done: a review paragraph attached to each of the last three incident tickets.
7. The monthly page for management
Once a month, send one page: what the monitoring caught first, what it did not, what changed, and what is coming. This is the move that makes the other six visible to the people who decide what the team gets. It is the one most admins leave until last because it feels like paperwork. It is not paperwork; it is the receipt. Making invisible work visible is the pillar on the subject. Evidence of done: last month's page, sent.
Why the order matters
Each move makes the next one possible, and doing them out of order makes each harder than it needs to be. Dependencies built on ping checks suppress the wrong alerts. That is because the dependency tree describes the network but the checks describe boxes. Define "down" first and the tree has something real to protect. Thresholds set before baselines are guesses, and the review ritual then spends its first three sessions correcting guesses instead of reading history.
Runbooks come fourth because the first three moves change which alerts exist. Write them earlier and you document alerts the threshold review is about to delete. The weekly graph look comes fifth because it needs baselines to compare against. Without them it is fifteen minutes of "the graphs look like graphs". Mini reviews come sixth because by then incidents are rarer and each one is worth a paragraph. The monthly page comes last because by then there is something to put on it.
Northgate Retail tried all seven in a month, with a spreadsheet and considerable enthusiasm. The dependency map was finished in week one, on top of ping checks. The map suppressed the file service alert the first time the core switch hiccupped, because the file server's ping was fine. Nine runbooks were written in week two for alerts that the threshold review deleted in week three. The weekly graph review was held twice, found nothing, and was cancelled. At the end of the month the alert count was higher than at the start. The spreadsheet recorded four of seven moves as "mostly". They started again, one move at a time, and finished the seventh in week twelve.
The ninety-day plan
This is the seven moves laid across thirteen weeks, with the evidence that each is done and a column for a name. It is the short version of the twelve-month roadmap, which takes a team from firefighting to foresight. This plan covers only the first leg, from level 1 to level 2, and stops there on purpose. Copy it, fill in the owner column with real names, and put the week numbers against real dates before you show it to anyone.
| Week | Move | Evidence of done | Owner |
|---|---|---|---|
| 1–2 | Write the service list with a definition of "down" for each | Service list document, one line per service, agreed with the service's main user | |
| 2–3 | Build a service check for each service on the list | Check list in the console matches the service list one to one | |
| 4 | Map dependencies for every monitored device | Dependency set on every downstream device; a test failure of one upstream produces one alert | |
| 5–6 | Collect baselines for the key metrics | Baseline figure recorded per metric, from at least two weeks of history | |
| 7 | Set thresholds from baselines; book the review ritual | Threshold change notes; recurring calendar entry with the first date in the past | |
| 8–9 | Write a runbook for every alert that fired more than once last month | Alert-frequency query output beside a runbook folder with a file per row | |
| 9 | Start the weekly fifteen-minute graph look | Calendar entry, and a one-line note from each week since | |
| 10–11 | Write a mini review for every incident from here on | Review paragraph on each incident ticket since week ten | |
| 12 | Send the first monthly page to management | The sent email, one page, with what was caught first and what was not | |
| 13 | Re-run the self-assessment | Worksheet with evidence; coverage, alerting, and response at level 2 |
Two things about the plan. First, weeks eight and nine overlap on purpose. The runbooks and the graph look start together because both are short and neither needs the other. Second, week thirteen is the assessment, not a celebration. If a dimension is still at level 1, the plan tells you which statement failed, and the next ninety days start there.
The plan moves five of the six dimensions directly. Moves one and two are coverage. Moves two and three are alerting. Moves four and six are response. Moves five and seven are data and reporting. And moves three, five, and six put the first three entries on the calendar. That is the process row. The people row moves sideways. Give the runbooks and the weekly look to the second person on the team. Then by week thirteen there are two people who can operate the monitoring. That is the level 2 statement for that row. If there is no second person, that row stays at level 1 whatever else you do. That is worth saying in the monthly page.
Tip: the plan will slip. The honest response is to move the dates, not to mark a move "done" because its week has passed. A move is done when its evidence exists. A plan with three moves done and real evidence is worth more to your manager than a plan with seven ticks and nothing to show.
What it costs in hours
The plan costs time, and it is better to say how much before someone asks. What follows is an estimate for an example team, built from stated assumptions so that you can swap in your own numbers. Suppose forty monitored devices, twelve services, and six alerts that fire more than once a month.
- Service definitions and checks: twelve services at roughly half an hour each to agree the definition and build the check. About six hours, once.
- Dependencies: forty devices at a few minutes each, plus an hour to draw the tree first. About four hours, once.
- Baselines and thresholds: two hours to record the figures once two weeks of history exist. Allow an hour to set thresholds, and an hour per quarter for the review. About four hours, then one hour a quarter.
- Runbooks: six alerts at half an hour each. About three hours, then a quarter of an hour for each new alert that earns one.
- Weekly graph look: fifteen minutes a week. One hour a month.
- Mini reviews: fifteen minutes per incident; at three incidents a month, under an hour a month.
- Monthly page: about an hour a month, less once the first one exists to copy.
Added up, that example is around seventeen hours of one-off work across the ninety days, and then roughly three hours a month to keep. Your numbers will differ. A team with a hundred devices will spend longer on dependencies. And a team with clean service definitions already will skip most of week one. The shape is what to take from it: a modest one-off cost, followed by an ongoing cost small enough to fit between other things. Put your own figures in the same list and it becomes the time line of the budget case.
The hours are not the real cost. The real cost is that the one-off work has to happen in the same weeks as the incidents it is meant to prevent. That is why the plan spreads it thin rather than front-loading it. Fifteen hours in a quarter is an afternoon a fortnight. Fifteen hours in a week is a fortnight of not answering the phone, and the phone will not wait.
The reactive-to-proactive checklist
Use this at week thirteen, and at any point you want to know how far along the plan is. Each line is a yes or a no, and each yes should have evidence you could open.
- Every service on the service list has a written definition of "down".
- Every service on the list has a check that tests that definition, not a ping.
- Every downstream device has a dependency configured, and the last upstream failure produced one alert.
- Every key metric has a recorded baseline figure.
- Thresholds were set from those baselines, and the review ritual has a past date with notes.
- Every alert that fired more than once last month has a runbook of one page or less.
- The weekly graph look is on the calendar and has a note for each week since it started.
- Every incident since week ten has a mini review paragraph on its ticket.
- Last month's one-page report was sent to management.
- The self-assessment has been re-run, and the result is recorded beside the previous one.
Where to go next
Seven moves, in order: define down, dependencies, baselines and thresholds, runbooks, the weekly look, mini reviews, the monthly page. Each makes the next one possible, which is why the order is the advice and not just the list. Ninety days, an afternoon a fortnight, and the evidence for each move written down as it is done. At the end, the phone still rings; it just rings after the console does, which is the whole difference between level 1 and level 2.
A team that has made the seven moves and kept them for a quarter is ready to ask whether the next level is worth it. And from proactive to predictive answers that honestly, including the parts that are marketing. When the plan needs funding, turning maturity into budget and credibility shows how to present the same moves as a budget case. All of them are part of A Monitoring Maturity Model for Small Teams.
Put this into practice with SysPeep Monitor
Moves one to four are done inside the monitoring tool. They are service checks, dependencies, thresholds set from history, and alert rules that keep one fault to one notification. SysPeep Monitor auto-discovers devices and interfaces. It collects over SNMP, WMI, and SSH from a central Windows service with optional remote probes. It puts dependencies, thresholds, and alert rules beside the long-term graphs and reports in one web console. The free trial is a reasonable place to run the first ninety days against your own network.
Download Free TrialLearn About SysPeep