Involving the Team in the Verdict
There is a particular kind of monitoring tool that gets chosen by one person, installed with great care, and quietly resented by everyone else for years. The help desk finds the daily view confusing. The on-call engineer mutes the alerts. The network admin keeps a spreadsheet on the side because discovery never quite got the switches right. Nobody was asked, so nobody bought in, and a good product ends up used badly. The rollout is recorded as a success by the one person who was asked.
The fix is not a bigger meeting. It is a structured way of letting each person who will live with the tool test the part of it they will live with. They score it on criteria agreed in advance, and that score counts. Done well, this takes a few hours of each person's time across the proof of concept (POC). It produces two things: a better decision, and a team that already knows the tool on the day it goes live. Both come from the same few hours, which is the best bargain in the whole evaluation.
By the end of this guide you will know who should evaluate what, and how to run hands-on sessions built around real tasks. You will know how to score by role using one shared sheet, and how to capture feedback so it can be compared. You will know how to stop the loudest voice from deciding, and why the future maintainer must be in the room. And you will know how to agree veto conditions before anyone has a favourite. There is a role table and a feedback form template you can copy.
In this guide
- Why the verdict needs more than one pair of hands
- Who evaluates what
- Hands-on sessions with real tasks
- Scoring by role with the agreed sheet
- Capturing feedback in a structured way
- Avoiding the loudest-voice decision
- The person who will maintain it
- Veto conditions agreed in advance
- The role, task, and criteria table
- The feedback form template
- Where to go next
Why the verdict needs more than one pair of hands
A monitoring tool is several products wearing one console. To the on-call person it is an alerting system. To the help desk it is a status page. To the network admin it is a discovery engine and an SNMP poller. SNMP is the Simple Network Management Protocol, the standard way a tool asks a device for its counters. To the manager it is a report. Each of those people is an expert in their own slice and a tourist in the others. And the person running the POC is usually an expert in exactly one of them. When one person scores everything, three quarters of the tool is being judged by a tourist.
There is a second, quieter benefit. People support what they helped choose. Consider a help-desk technician who spent an hour finding a device's status in each candidate, and whose score was visible in the decision. They will use the winning tool differently from one who was handed a login on launch day. The POC is the cheapest training and the cheapest buy-in you will ever get, and both come from the same hour of hands-on work.
At Ashgrove Housing the team was involved in the POC by being sent the slide deck afterwards. It had forty slides, a green tick on every one, and a final slide headed "Team Consensus" that nobody on the team could remember reaching. The help desk got its login on launch day and found the status page in the second week. The on-call engineer discovered the alert-suppression setting eight months later, from a colleague at another organisation. The tool was fine. The network admin's spreadsheet is still there, and so, on a shared drive, is the deck.
Who evaluates what
Match each part of the tool to the person who will depend on it. The mapping below fits most small and mid-sized IT teams. In a very small team, several roles will belong to one person, but keep the roles separate on paper so the scores stay separate too.
The on-call person scores alerting
Whoever carries the phone at night judges the alerts. Are they timely? Are they readable at three in the morning? Do they name the actual problem rather than every symptom below it? How hard is it to quiet one that fires too often? This person also judges acknowledgement and escalation. Can they say "I have this" and stop the tool nagging? Does it tell someone else if they do not?
The help desk scores the daily views
Front-line support sees the tool more often than anyone and asks it the simplest questions: is this thing up, since when, and what else is affected? They score the dashboard and the search. How many clicks does it take to go from "user in the east wing says the internet is slow" to a screen that helps? They also judge whether the views make sense to someone who did not configure them.
The network admin scores discovery and SNMP depth
The person who knows every switch by name judges whether discovery found them all and identified them correctly. They judge whether the tool reads interface counters, errors, and hardware sensors properly, and how it handles the awkward devices. They also judge how much work a custom check for an unrecognised device takes. They are also the best judge of polling behaviour and of what the tool does to the network in return.
The manager scores reports
Whoever presents uptime and capacity to the rest of the business judges whether the tool can produce the reports they actually send. They judge whether it can do so on a schedule, without a request to IT each month. They also judge whether the numbers are explainable, because a report that cannot be defended in a meeting is worse than none.
The maintainer scores everything else
Someone will patch this tool, back it up, add users to it, and fix it when it breaks. That person has their own section below; for now, note that they get their own row in the table and their own score.
Anyone else who consumes monitoring
Suppose someone outside the IT team depends on the tool. That might be a facilities manager who watches the building systems, or a security lead who wants to know when an unknown device appears. Give them a short session of their own with two or three task cards. Their scores carry a small weight, but their feedback often catches a usability problem the technical evaluators have stopped noticing. A week in a console is enough to stop seeing the extra click that a newcomer trips on every time.
Hands-on sessions with real tasks
A demo is something you watch. A session is something you do. The difference is the whole point. Watching a vendor's engineer find a device tells you the tool can do it. Finding it yourself tells you whether you can, in how long, and with how much swearing. Every evaluator gets at least one session per candidate, on the POC environment, driving the mouse themselves, with a short list of tasks drawn from their role. Nobody from the vendor is on the call.
A session that works looks like this:
- Book an hour. Shorter and people rush; longer and the day job reclaims them.
- Hand the evaluator their task cards — three to five real tasks, each phrased as a request they would actually receive. "Find out whether the second-floor access point has been up all week" rather than "explore the device view."
- Give them a login and nothing else. If they get stuck, they may use the product's own documentation and search, but not the POC owner. The POC owner sits in the corner taking notes on where they got stuck and for how long.
- Time each task loosely. A task that takes twelve minutes in one candidate and two in another is a finding, whatever the evaluator's overall impression.
- Finish with the feedback form, filled in before they leave the room and before they talk to anyone else about it.
Use the same task cards for every candidate, and rotate the order in which evaluators meet the candidates. That way, the first tool tested is not always the one everyone learned the concepts on. If your team is small enough that one person plays several roles, run their sessions on different days for different roles. Scoring alerting and reports in the same hour blurs both. Sessions can be remote if they must be, with the evaluator sharing their screen and the observer watching quietly. But in-person sessions catch the small hesitations that a shared screen hides.
Tip: the most revealing task card is a request the evaluator finds slightly annoying in real life. For example: "the finance manager wants to know how often the file server was slow last month." If a tool makes an annoying request easy, people will use it. If it makes it harder, they will go back to the spreadsheet.
Scoring by role with the agreed sheet
Each evaluator scores only their own criteria, on the sheet agreed before the POC began. The sheet is not built here. The guide to the weighted scoring sheet explains how criteria, weights, and the definitions of a one, three, and five are established. The POC test plan says which rows belong to which role. What matters for the team is the discipline around it.
Scores go on the sheet immediately after the session, while the experience is fresh and before any conversation. Each score points to evidence — a task card completed, a time recorded, a screenshot, an alert message pasted in — not to a feeling. Where an evaluator cannot give a score because the task could not be completed, that is the lowest score with a note, not a blank. And a score is the evaluator's own. The on-call person's alerting score is not adjusted because the network admin, watching over their shoulder, would have given it a four. (The network admin may give their own four, on their own row.) Separate people, separate rows, separate opinions; that is how the sheet stays honest and how each role's experience survives to the decision meeting.
Capturing feedback in a structured way
Numbers compare well; words explain. Alongside each score, every evaluator answers the same short set of questions in writing, so that feedback from different people and different candidates can be laid side by side. Three questions do most of the work:
- What worked? The tasks that went smoothly and why. Be specific — "the search found the device by half its name" is useful; "nice interface" is not.
- What did not? Where they got stuck, how long for, and what finally unstuck them. A stuck moment that ended with a documentation search is a different finding from one that ended with giving up.
- What surprised you? Good or bad. Surprises are the things the test plan did not predict, and they are often where the real differences between candidates live.
Add one closing question that cuts through everything: would you want this tool on a bad day? Not on a demo day, not on a quiet Tuesday — on the day the core switch dies and the phone will not stop. The answer, and the reason for it, is frequently the most honest sentence in the whole POC. Collect the forms centrally, one per evaluator per candidate, and keep them with the decision record.
Avoiding the loudest-voice decision
Every team has someone whose opinion arrives first and largest. Sometimes it is the most senior person, sometimes the most confident, sometimes the one who has used a similar product before. None of that makes them wrong. But if their view is heard before anyone else has committed to a score, every other score drifts towards it. Then the team's verdict becomes one person's verdict with extra signatures.
Two rules stop this. The first is scores first, discussion second. Every evaluator's scores and feedback forms are submitted before the review meeting, and the meeting opens by showing the sheet with all of them already on it. Discussion then explains the numbers rather than producing them. The second rule is that in the meeting itself, the most senior person speaks last. That is uncomfortable for exactly one meeting, and after that it is simply how the team decides things.
The third safeguard is a role rather than a rule: the skeptic, named in the test plan. Their job during the review is to challenge every pass. "Did discovery actually find everything, or did we stop checking after the first forty?" "Would that alert rule still hold at twice our size?" "Did that task take two minutes because the tool is good or because you had already done it in the other candidate?" A good skeptic is not against any candidate. They are against rounding up, and they should be someone the team likes, so their questions land as questions.
The person who will maintain it
It is surprisingly common for a monitoring tool to be chosen by the people who will consume it and then handed to someone else to keep alive. That someone has different concerns entirely. How is it backed up? How is it upgraded? How are users and permissions managed? What does it do to the server it lives on? How does it tell you when it is unhealthy? How much of its configuration is in a form that can be version-controlled (kept in a system that records every change) or at least exported? None of those appear on the on-call person's task cards or in the manager's report. I skipped the maintainer's session once, because I was the maintainer and I was busy. I scored it a five from memory. The first upgrade revised it for me.
So the future maintainer gets their own session and their own criteria. Their task cards are the boring ones. Take a backup and restore it. Apply a settings change and find it in the audit log (the record of who changed what, and when). Add a user with limited rights. Check disk usage after a week of polling. Find the log file when something is wrong. Their score carries weight equal to anyone's, because a tool nobody can maintain is a tool that decays. Their feedback form is the one to read first if you are short of time.
Veto conditions agreed in advance
A veto is the right of one role to rule out a candidate regardless of its total score. Vetoes are dangerous when they are improvised in the review meeting — that is the loudest-voice decision wearing a costume. They are valuable when they are written into the test plan before the POC starts. The difference is timing and specificity.
An agreed veto condition is a sentence that names a role, a test, and an outcome. For example: "the on-call engineer may veto any candidate that cannot suppress downstream alerts when a parent device fails, because that requirement is a must-have and they are the person who receives the flood." "The maintainer may veto any candidate whose backup cannot be restored during the POC." "The network admin may veto any candidate that cannot monitor the core switch model." Each one is really a must-have from the requirements list, assigned to the person best placed to judge it. It is stated so plainly that invoking it needs no argument. A parent device is the switch or router that other devices sit behind. Downstream alerts are the ones from everything behind it, which is why the on-call engineer is the person to hold that veto.
Keep the list short, three or four conditions at most, and require that a veto be accompanied by the evidence from the session that triggered it. A veto exercised this way is not a power play. It is the test plan's own pass/fail gate, with a human attached.
The role, task, and criteria table
| Role | Session tasks (same cards for every candidate) | Criteria scored | Veto condition |
|---|---|---|---|
| On-call engineer | Receive the staged outage alert; acknowledge it; quiet the flapping port; trace an alert to its cause | Alert timeliness, clarity, noise, suppression, acknowledgement and escalation | Cannot suppress downstream alerts on parent failure |
| Help desk | Find a named device's status and history; answer "what else is affected?"; locate a user's switch port | Daily views, search, clicks-to-answer, readability for non-configurers | None (scored only) |
| Network admin | Verify discovery against the lab inventory; read interface errors on the core switch; build a check for the awkward device | Discovery accuracy, SNMP depth, custom checks, polling behaviour | Cannot monitor a must-have device type |
| Manager | Build and schedule the monthly uptime report; explain one number in it | Report fit, scheduling, explainability | None (scored only) |
| Maintainer | Back up and restore; add a limited user; find the audit log and the log files; check disk growth | Backup, upgrades, administration, self-monitoring, footprint | Backup cannot be restored during the POC |
| Skeptic | Attend sessions; challenge passes in the review | None — questions only | None |
The feedback form template
One form per evaluator per candidate, filled in at the end of the session, before any discussion. Keep it to a page; the aim is comparison, not an essay.
POC FEEDBACK FORM Candidate: ______________________ Evaluator / role: ______________________ Session date: ______________________ Task cards (from the plan): 1. ________________________ done / not done time: ____ 2. ________________________ done / not done time: ____ 3. ________________________ done / not done time: ____ 4. ________________________ done / not done time: ____ Scores (my criteria only, on the agreed 1-5 scale, with evidence): criterion ____________ score __ evidence ______________ criterion ____________ score __ evidence ______________ criterion ____________ score __ evidence ______________ What worked? __________________________________________________________ What did not? (where stuck, how long, what unstuck it) __________________________________________________________ What surprised you? __________________________________________________________ Would you want this tool on a bad day? yes / no because: __________________________________________________________ Veto invoked? no / yes: condition ______ evidence ______
Collect the forms centrally, keep them unedited, and attach them to the decision record. Six months after the choice, when someone asks why the team went this way, these pages are the answer.
Where to go next
A verdict built from separate sessions, separate scores, and a structured form is one the whole team owns. It produces something a solo evaluation never can: a group of people who already know the tool before it is installed for real. It also means that when the rollout is recorded as a success, more than one person was asked. The sessions run inside the environment and against the plan described elsewhere in Running a Fair Proof-of-Concept. The scores feed directly into the final step, moving from POC to decision, where gates are applied, surprises are handled, and the choice is written down.
Put this into practice with SysPeep Monitor
If the SysPeep Monitor free trial is one of your candidates, give it the same session. Hand the on-call engineer the staged outage and the help desk the device search. Hand the network admin the discovery results and the maintainer the backup and restore. A tool with a single web console for dashboards, maps, and reports should make every one of those task cards straightforward. The scores will say whether it did.
Download Free TrialLearn About SysPeep