Home › Topics › A Monitoring Maturity Model for Small Teams › Scoring Your Team Honestly

Crawl, Walk, Run: Scoring Your Team Honestly


The first time a team scores itself on a maturity model, the scores come out about one level too high. Not because anyone is lying. Because "are we proactive?" is answered by the part of the brain that remembers the good week, and the good week was real. The disk alert did fire early, once. The runbook did exist, for one alert. The weekly look at the graphs did happen, in the month after the outage.


The cure is evidence. A self-assessment that asks for a file, a screenshot, or a calendar entry beside every claim produces a different answer from one that asks for an opinion. The different answer is the useful one. It takes an hour, two people, and a willingness to write "no" where you expected to write "yes". I have run it on my own team and written "no" more often than I would like to admit here.


By the end of this guide you will be able to run the assessment in an hour with a worksheet of written-out statements. You will be able to apply the one scoring rule that stops the drift upward. You will be able to recognise the three ways teams overscore and the one way they underscore. You will be able to read a jagged result and explain how this differs from a health check.


In this guide

  • An hour, two people, and evidence
  • How the worksheet works
  • The scoring rule
  • Overscoring: the three usual ways
  • Underscoring: the solo admin
  • Reading a jagged result
  • Levels versus healthy signs
  • The self-assessment worksheet
  • The assessment checklist
  • Where to go next

An hour, two people, and evidence


The assessment is a short meeting with a worksheet, not a survey. A survey asks people what they think; this asks the monitoring system what is true. Book an hour and invite exactly two people. Invite the person who runs the monitoring day to day and one who does not. Ideally, the second is someone who has been on the receiving end of an outage. The first knows where the evidence is. The second asks to see it.


Bring the monitoring console, the shared drive, the calendar, and the ticket system, all open. Every statement is answered by looking at one of those, not by remembering. If the evidence is a runbook, open the folder. If it is a review, open the calendar to the day it happened. If it is "alerts name an action", open ten alerts and read them.


The model you are scoring against is the one in the four levels, plainly. It has four levels from ad hoc to predictive and six dimensions from coverage to people and ownership. And it repeats the warning that a level is a map, not a grade. The warning matters most here, because this is where a grade would be handed out if there were one. There is not. The output is six numbers and a folder of evidence, and the numbers exist only so the plan knows where to start.


How the worksheet works


The worksheet has one block per dimension, and inside each block three statements per level. A statement is a plain sentence that is either true of your team this week or not. For example: "every alert that fired more than once last month has a runbook". Beside each statement is a space for the evidence, which is a thing you can point at. No thing to point at means "no", whatever anyone remembers.


Evidence is narrower than proof, on purpose. A file with a recent date. A screenshot of an alert rule. A calendar entry that says "threshold review" and is in the past. A ticket with three timestamps. The bar is low because the purpose is not to defend the score in court; it is to stop the score being a feeling. (A feeling can be level 3. A folder cannot.)


Work down the sheet dimension by dimension, and within each dimension level by level, starting at level 1. Do not skip to the level you expect to be at. If a level 1 statement is "no", write it down and carry on through level 2 anyway. A "no" at level 1 with three "yes" answers at level 2 is a specific shape, which the scoring rule handles next.


Some evidence can be fetched rather than found. A folder of runbooks with last-modified dates tells you in one line whether "we have runbooks" means that, or "we had runbooks":


Get-ChildItem \\fs01.example.com\it\runbooks -Filter *.md |
    Sort-Object LastWriteTime -Descending |
    Select-Object Name, LastWriteTime

Paste the output into the evidence column as it is. Nine files, two of them touched in the last quarter, settles an argument without anybody needing to win it.


The scoring rule


You are at the highest level where every statement has evidence. Not the level where most statements do. Not the level where the important ones do. Every statement, with something in the evidence column, or you are at the level below. This rule separates the assessment from a survey, and it is the one people argue with first.


The argument is always the same: "we do two of the three, surely that is level 2". It is not, and the reason is not pedantry. A team that does all three behaves differently from one that does two. Runbooks without timed incidents is a team that fixes things well and cannot say how long it took. Timed incidents without reviews is a team that knows how long it took and changes nothing. The level names a behaviour, and the behaviour needs all three.


Apply the rule in a spreadsheet so nobody applies it by hand. With columns for dimension, level, statement, evidence, and met (yes or no), a dimension's level is the level below the first one containing a "no":


=IF(COUNTIFS(A:A,"Coverage",B:B,1,E:E,"no")>0, 0,
   IF(COUNTIFS(A:A,"Coverage",B:B,2,E:E,"no")>0, 1,
      IF(COUNTIFS(A:A,"Coverage",B:B,3,E:E,"no")>0, 2, 3)))

Copy the formula for each of the six dimension names and you have the row of six numbers. Leave nothing blank in the "met" column: a statement nobody looked for evidence on is a "no" that has not been written down yet.


Overscoring: the three usual ways


Teams overscore in three recognisable ways, and the evidence column catches all of them if you let it.


Counting intentions as practices

"We review thresholds quarterly" is a practice if there are calendar entries in the past with notes attached. It is an intention if the entry was created last week for next month. Intentions are fine; they belong in the plan, not the score. A ritual that has not yet had its first occurrence is not a ritual. It is a hope with a reminder.


Counting the tool's features as your practices

The product can do dependencies, so the team scores level 2 on coverage. It has a baseline feature, so the team scores level 2 on alerting. Neither is evidence. The evidence is the dependency configured on the annex switch and the alert history from the last time it rebooted. That history shows one notification instead of twelve. A feature that is not configured is a menu item, and menu items are not a level. (Vendors will disagree. Vendors are not in the room.)


Counting the one time you did it

The team did a post-incident review once, after the bad outage, and everyone remembers it. That is one review. The level 2 statement says a review followed each of the last three incidents, and the ticket system knows whether it did. This is the most sympathetic kind of overscoring, because the one time was real. Score it as what it is: proof that the practice is possible here, which is the best starting point for making it regular.


Underscoring: the solo admin


There is one common way to score too low, and it belongs almost entirely to the administrator who works alone. The solo admin does level 2 things. They check the graphs every Monday and keep a text file of what to do when the backup job alerts. They add every new server the day it arrives. They call none of it a practice, because it is "just what I do". Asked whether there is a runbook, they say no. Asked what they do when the backup alert fires, they open the text file.


The text file is a runbook. The Monday habit is a weekly review. The evidence exists; it is simply not labelled. The second person in the room is there for exactly this. They hear "I just check it on Mondays" and write "weekly trend review: yes, evidence: Monday calendar block and the note in the ticket from last week". The monitoring bus factor pillar is about what happens when that person is the only one who knows. That is the real level 1 in the people row, however good the other five rows are.


So the correction for underscoring is the same as for overscoring: look at the thing. If it exists, it counts, whatever it is called. If it does not, it does not.


Reading a jagged result


The result will be jagged, and the first thing to do with it is nothing. Do not average the six numbers. Do not pick the highest one for the slide. Six numbers side by side are the result, because the lowest of them usually limits the others.


The limiting dimension is the one to fix first, even when nobody wants to work on it. Level 2 alerting on top of level 0 people means the alerts are excellent and nobody but one person can change them. Level 2 data on top of level 1 coverage means beautiful graphs of the wrong things. Fixing the lowest row is rarely glamorous, usually cheap, and makes the next row's work stick.


Write the plan from the lowest row up: the dimension, the statement that failed, and the evidence that would make it pass. "Coverage: the three line-of-business services get a service check each, evidence: the check list, by the end of the month" is a plan. "Improve coverage" is a heading. The reactive-to-proactive guide has the moves in the order that works, and most failed statements map to one of them.


Harbourview Clinic scored itself before the evidence column was added, and the row read 2, 2, 2, 1, 1, 2. With the evidence column it read 1, 1, 1, 1, 0, 1. Alerting fell because ten alerts opened at random included seven that said only "threshold exceeded". Response fell because the runbook folder held three files, the newest of which predated the current administrator. Process fell to 0 because the quarterly review existed only as a recurring calendar entry that everyone declined. The second row was less comfortable and was the one that produced a plan; the first had produced a feeling.


Levels versus healthy signs


This assessment is not a health check, and it is worth being clear which one you are running. The healthy-signs guide lists symptoms of a working practice. These include alerts that get acknowledged, graphs that get opened, and a quiet on-call phone. The guide asks whether you see them. It is a health check by symptom: quick, reassuring when it passes, pointed when it does not.


This is a level by dimension. It does not ask whether the practice is healthy. It asks what the practice does, in six areas, and places each on a scale with the next step written beside it. They answer different questions. Run the signs when someone asks "is monitoring okay?". Run the levels when someone asks "what should monitoring do next?" or "what will this budget buy?".


Tip: keep the evidence folder after the assessment, with the worksheet in it, and date the folder. The next assessment is the same hour, six months later, with the old folder open in a second window. The difference between the two folders is the progress report, and it writes itself.


The self-assessment worksheet


This is the worksheet for levels 1 and 2 in all six dimensions, three statements per level, with the evidence to look for. Copy it into a spreadsheet with a "met" column, work down it with the evidence open, and let the formula compute the row. Level 0 in any dimension means none of its level 1 statements is met. Level 3 statements follow the table.


DimensionLevelStatementEvidence to look forMet?
Coverage1Every server and switch we know about has at least a ping check.Inventory list beside the console device list; no missing rows.
Coverage1The key services (mail, files, the line-of-business system) have at least one check beyond ping.The check list for those hosts.
Coverage1A device going down produces an alert, not only a colour on a page.An alert from the last outage.
Coverage2Every service on the written service list has a check that does what the user does.Service list beside check list, one to one.
Coverage2Dependencies are configured so a parent failure suppresses child alerts.Dependency setting on a downstream device; alert history from a switch outage.
Coverage2New systems are added to monitoring at intake, not after an outage.Intake checklist item; add dates of the last three systems.
Alerting1Alerts go to a person or a rota, not only to a mailbox.The notification rule.
Alerting1Someone acted on an alert in the last month.A ticket or note citing the alert.
Alerting1Alerts exist for the outages that matter most, not only for ping loss.The alert rule list.
Alerting2Every alert message names the service and the first action.Ten alerts opened at random; all ten pass.
Alerting2One fault raises one alert; the last switch outage produced one notification.Alert history from that outage.
Alerting2Thresholds were reviewed against baselines within the last quarter.The calendar entry and the change notes.
Response1A named person is expected to respond to alerts in working hours.A rota, a role description, or the notification target.
Response1The responder can reach every monitored device from where they respond.A remote session from the responder's machine.
Response1The last three incidents were resolved before a user escalated to management.Ticket timestamps.
Response2Every alert that fired more than once last month has a runbook.Alert frequency list beside the runbook folder.
Response2Every incident has a recorded start, detection, and resolution time.The last three tickets.
Response2A short review followed each of the last three incidents.The review notes.
Data and reporting1Metric history is kept for at least ninety days.A ninety-day graph with data across it.
Data and reporting1Graphs were used during the last incident to see what changed.A screenshot in the ticket.
Data and reporting1"How busy was the link last month" can be answered within an hour.Answer it now.
Data and reporting2Baselines exist for the key metrics.A baseline document or the baseline configured per metric.
Data and reporting2Someone looks at the graphs weekly, on purpose, and notes it.The weekly note or ticket.
Data and reporting2A one-page monthly report reaches management and went out last month.The sent email.
Process and rituals1Monitoring was changed after the last serious outage.The change and the outage, linked.
Process and rituals1A list of monitoring changes from the last six months exists.The list.
Process and rituals1Something checks that the monitoring itself is running.A heartbeat check or a documented routine.
Process and rituals2Threshold review has a calendar entry and the last one happened.The calendar entry and its notes.
Process and rituals2Coverage review has a calendar entry and the last one happened.The calendar entry and its notes.
Process and rituals2Post-incident review is a standard step, not an occasional one.Reviews for the last three incidents.
People and ownership1One named person owns the monitoring.The name in a document or role description.
People and ownership1The owner can explain every alert rule.Pick three; ask.
People and ownership1Monitoring credentials are in the shared password safe.The safe entry.
People and ownership2A second person can add a device, change a threshold, and acknowledge an alert.Watch them do it.
People and ownership2The checks and alert rules are documented outside the tool.The document.
People and ownership2The monitoring ran without a gap during the owner's last week off.Alert history for that week.

There are level 3 statements, two per dimension, for a dimension already at level 2. They cover coverage reviewed on a calendar against the service list, with synthetic checks for the services users feel first. They cover trend alerts that warn weeks ahead, with the alert list pruned quarterly. They cover failovers and restores rehearsed on a schedule, with notes. They cover a capacity register with a projected date per resource, reviewed quarterly. They cover a quarterly plan with items that came from graphs, and changes rehearsed in a lab first. They cover ownership written down, with a practised handover. The proactive-to-predictive guide describes each of these as a practice with a cadence.


The assessment checklist


Run through this before the hour ends. Each line is a yes or a no, and the assessment is not finished until every line is a yes.


  • Two people were present: the person who runs the monitoring and one who does not.
  • Every statement has either an evidence entry or the word "no"; nothing is blank.
  • No evidence entry is a recollection; each is a file, a screenshot, a calendar entry, or a ticket.
  • No statement was marked met because the tool has a feature for it.
  • The scoring rule was applied by formula, and the six results are recorded side by side, unaveraged.
  • The lowest dimension is named, with the failed statement and the evidence that would make it pass.
  • The next assessment is on the calendar, six months out, with the same two people invited.

Where to go next


An hour, two people, and the evidence open. Three statements per level, each answered by pointing at a thing. The highest level where every statement has evidence, computed by formula so nobody has to be the one who says "no". Six numbers, never averaged, with the lowest one circled. That is the whole assessment. The honest version produces a plan; the flattering one produces a slide.


The lowest row usually points at the moves from reactive to proactive, which most failed statements map to. When the row is ready to be shown to someone with a budget, turning maturity into budget and credibility explains how the evidence column does the talking. All of them are part of A Monitoring Maturity Model for Small Teams.


Put this into practice with SysPeep Monitor

Half the evidence on the worksheet is a screenshot of a check, a dependency, or an alert rule. The other half is a graph with enough history behind it. SysPeep Monitor keeps checks, dependencies, thresholds, and alert rules in one web console alongside long-term graphs and reports. It uses SNMP, WMI, and SSH collection from a central Windows service. So the evidence column can be filled from one place. The free trial is enough to see how many of the level 2 statements your network could meet.

Download Free TrialLearn About SysPeep