From POC to Decision: Scoring, Documenting, and Committing
The trials are over. The lab is full of alerts, and the feedback forms are in a folder. The vendors are asking politely how it went, and somebody senior wants an answer by Friday. This is the moment a proof of concept (POC) either turns into a decision or dissolves into a feeling. The difference is entirely in what you do with the results in the next few days. I have watched the decision made from the scoring sheet. And I have watched it made from the sales engineer's laptop on the last afternoon of the trial. And I know which one I could still explain a year later.
Done badly, this stage is one meeting where people remember the candidate they liked, argue about it, and pick one. Done well, it is almost boring. The gates are applied, and the scores are added up. The surprises are handled by rules you wrote weeks ago. The choice is written on a single page that explains itself to anyone who reads it later. Boring is the goal. Boring is what a fair process looks like from the end.
By the end of this guide you will be able to score against the plan. You will be able to deal with a candidate that failed a gate but shone elsewhere, and write the one-page decision record. You will be able to negotiate with the decision already made, and plan the rollout as the migration it is. You will be able to tell the candidates who did not win, and recognise the case where the honest answer is "none of them." There is a decision-record template at the end.
In this guide
- Scoring against the plan: gates first, then weights
- Handling surprises: the gate wins
- The one-page decision record
- Negotiating with the decision in hand
- Planning the rollout as a migration
- Telling the candidates who did not win
- The valid "no" outcome
- The decision-record template
- Where to go next
Scoring against the plan: gates first, then weights
Everything here depends on the test plan having been written before the trials. Each must-have requirement must have been expressed as a pass/fail test, and each should-have as a scored test. The scoring method must have been agreed in advance. If that plan exists, the results stage is mechanical, in two passes.
Pass one: the gates
Go down the must-have tests for each candidate and mark each pass or fail from the recorded results, not from memory. A candidate that failed any gate is out. This is decided before anyone looks at a weighted score. That is because the moment a total appears next to a candidate's name, the temptation to reinterpret its gate failures becomes strong. Gates first. Not totals first "just to get a feel". Gates. Two cautions. A blank result is a fail for scoring purposes. Perhaps the test was never run, the vendor asked you to skip it, or the evaluator ran out of time. And the plan should have said that a blank result counts as a fail. And a gate result that two people disagree on goes back to the evidence: the screenshot, the alert message, the timed task. If the evidence does not settle it, the test was not written tightly enough. In that case, the fair remedy is to re-run it for every candidate, not to argue it out.
Pass two: the weighted scores
For the candidates still standing, take each should-have test's score from the role that owned it. Multiply by the weight agreed in the plan, and add up. Do this on the sheet, in front of everyone, from the submitted forms. The mechanics of the sheet — criteria families, weights, what a one and a five mean — are laid out in the weighted scoring sheet. They do not change at this stage. The whole point of agreeing them earlier was so that nobody could adjust them now. Show the totals, and then show the breakdown by role. A candidate that wins narrowly overall while losing badly with the on-call engineer is a different kind of winner from one that led in every row.
Only after both passes does the discussion begin, and it begins from the numbers. "Candidate B leads by a small margin; the difference is almost entirely in reporting; alerting is level. Is that where we expected the difference to be?" That is a conversation about evidence. "I just felt B was more polished" is not, and the sheet makes it visible when someone tries to have the second conversation instead of the first.
The review meeting, in order
The meeting that does all this is short if it is run in a fixed order, and long if it is not. A sequence that works for a small team:
- The POC owner confirms every score and form was submitted before the meeting. Anything missing is scored as the lowest value, as the plan said.
- Gates are read out per candidate, with the evidence for each fail. Eliminated candidates are named and set aside.
- Weighted totals and the by-role breakdown are shown for the survivors.
- Each evaluator explains their own scores, briefly, in role order; the most senior person speaks last.
- The skeptic challenges any pass or score that looks generous, and the evidence answers.
- The choice, or the "no," is stated, and the decision record is drafted while everyone is still in the room.
Handling surprises: the gate wins
The classic surprise is the candidate that failed a must-have and dazzled everywhere else. Its dashboards were the best, the help desk loved it, and the vendor was a pleasure. But it could not suppress downstream alerts when the core switch went down, which was gate number three. Downstream alerts are the flood from every device behind a failed switch. Every instinct in the room wants to find a way to keep it.
The gate wins. A must-have was a must-have because its absence causes real harm on a real day. In this example, that means forty alerts instead of one, at three in the morning, for as long as you own the product. The dazzle does not cancel that; it just makes it easier to forget. If you keep the candidate anyway, you have not made a decision. You have discovered that your requirements were not really requirements, and every future POC will be judged on charm.
At Tollgate Foods the candidate with the best dashboards failed the dependency gate. The review meeting kept it anyway, on the grounds that the core switch "hardly ever" went down. It went down the following spring, on a night shift. The on-call engineer received a hundred and thirty alerts before the one that mattered, which arrived somewhere in the middle. The minutes of the review meeting recorded the gate as "noted". The next POC at Tollgate had a skeptic — one person whose job was to challenge every pass — and the skeptic had a copy of the minutes.
There is exactly one honest escape, and it is not a small one. Suppose the room genuinely believes the requirement was wrong: it was tagged a must-have carelessly and is really a should-have. Then the requirement is changed openly, in writing, for every candidate. The scoring is redone from the start. That means a candidate you had already eliminated on the same gate comes back in. If the team is not willing to do that, the requirement stands, and so does the gate. This rule feels harsh once and saves you every time after.
Two smaller surprises deserve rules too. A candidate that passed a gate only with heavy vendor help gets the pass. But the help is recorded in the decision record, because it means the team has not yet shown it can do that thing alone. And suppose a candidate discovered something during the POC that no test covered. That might be a device nobody knew about, or a misconfiguration the old system never noticed. That discovery earns a note in "what surprised you," not a score. It can be a tiebreaker. It cannot outrank the plan.
Tip: if the totals are within a whisker of each other, do not manufacture a winner from decimal places. Look at the by-role breakdown and the "would you want this on a bad day?" answers. If it is still level, the candidate with the more specific, more bounded vendor answers is usually the safer choice. Write down that this was a close call; it matters for the negotiation.
The one-page decision record
A decision record is a single page that states what was decided, by whom, from what evidence, and at what cost. It exists because memory is unreliable and staff change. A year from now someone will ask why this tool and not the other one. And the honest answer will otherwise be "we liked it." The record turns that into an answer with tests, scores, and names attached.
One page is the discipline. If it needs more, the extra goes in appendices — the full sheet, the feedback forms, the vendor answers — and the page summarises them. The sections are fixed:
- The candidates: which tools were evaluated, over what dates, in what environment.
- The tests: a reference to the plan, and the count of gates and scored tests.
- The scores: gate results per candidate, weighted totals, and the by-role breakdown.
- The choice: one line.
- The reasons: three to five sentences drawn from the evidence, not from adjectives.
- What we gave up: the things the losing candidates did better, and the winner's known limits, stated plainly. This is the section future-you will thank you for, and it is the one most records omit.
- Who decided: the names of the evaluators, the skeptic, and whoever signed off. Decisions with names on them are made more carefully.
Write the record the same day as the review meeting. Circulate it to everyone who scored. And treat their corrections as part of the record. Then file it where the next person to hold your job will find it. Nobody reads it for a year. Then a stranger reads every word.
Negotiating with the decision in hand
Notice the order: decide, then negotiate. Negotiating before deciding means the price shapes the choice. A vendor who senses a decision is still open has every reason to sweeten the demo rather than the deal. Negotiating afterwards means you are talking about the terms of something you have already, on evidence, decided you want. That is a much stronger position than it sounds. You know precisely what you value and what you are willing to lose.
The POC gives you specific material. The vendor's own written answers to the completeness questions tell you what is not in the quote. The by-role breakdown tells you which modules matter and which were nice-to-haves. "What we gave up" tells you where you have room. And the runner-up's existence, documented and close, is a legitimate thing to mention, especially if the call was close and you said so in the record. The subject of Licensing Models and Hidden Costs is what the licence lines mean and how growth affects them. It also covers where costs hide across the life of the contract. Bring that reading to the table with the decision record.
The things worth asking for are often not discounts at all. They include a licence for a second, non-production instance on which to rehearse upgrades. They include the roadmap commitment the vendor made in writing, attached to the contract. They include training seats for the evaluators who will become the first users. They include a support tier that matches how the on-call team actually works. Each of these came straight out of the POC's findings, which is why they are easy to justify and hard to refuse.
One boundary keeps the negotiation honest: nothing agreed here changes the decision record. If the winner's price turns out to be beyond reach, that is a "no" outcome, handled below, not a reason to re-score the runner-up upwards. (The runner-up did not get better because the winner got expensive.)
Planning the rollout as a migration
A POC proves a tool can work on a slice of your network for a few weeks. It does not install the tool on the whole network, tune every alert, or retire the old system. And the trial instance in the lab is not the production deployment. It was built with lab credentials, on a lab VM (virtual machine), and it gets torn down with the rest of the POC. What follows is a migration, whether or not you had a monitoring tool before. It deserves its own plan. Inventory the devices to be monitored, build production properly, and bring alerts across in stages. Run the old and new side by side long enough to trust the new one, and cut over deliberately. That is a different project from the POC, with different risks. Migrating Monitoring Platforms covers it from the first inventory check to the day the old system is switched off.
Three things from the POC carry straight into it. The test plan's scenarios become acceptance tests for the production build. The evaluators become the first trained users, and their feedback forms are a list of the rough edges to address in configuration. And "what we gave up" becomes the list of known gaps to work around or to raise with the vendor, before anyone reports them as surprises.
Telling the candidates who did not win
Every vendor who took part gave you time, engineers, and trial licences. They deserve to hear the outcome from you, promptly, with reasons. Give them the shape of the record, not the full record. Explain which requirement they did not meet, or where the scores separated, in specific and bounded terms. That is the same standard you asked of their answers. "Your product did not suppress downstream alerts when a parent device failed, which was a must-have for our on-call team" is useful to them. "We went another way" is not.
This is not only courtesy. The tool that came second is, quite often, the choice next time. Contracts end, products change, and the requirement they missed may be fixed a release or two later. A vendor who was told exactly why they lost, politely and in writing, will remember you as a customer worth returning to. A vendor who never heard back will not.
The valid "no" outcome
Sometimes the gates eliminate everyone. Sometimes the only survivor scores so poorly with the team that nobody wants it on a bad day. Sometimes the winner's price puts it out of reach. In each case the honest result of the POC is "none of these, not now." It is important to say clearly that this is a success of the process, not a failure of it. The purpose of a POC is to find out. You found out. The alternative was to buy the wrong tool and find out later, at considerably greater expense.
A "no" outcome has three constructive paths, and the decision record should say which one you are taking:
- Revisit the requirements. If every candidate failed the same gate, the gate may be describing something the market does not do in the way you asked. Re-examine the requirement with the people who wrote it. If it changes, it changes for everyone, and a new short POC on the survivors may be enough.
- Extend the trial. Suppose one candidate failed a gate by a whisker. Suppose also that the vendor has committed in writing to a fix on a date you can live with. In that case, a bounded extension — new timebox, same tests, same environment — is fair. An open-ended one is not.
- Stay put. If the current tool, whatever its faults, passes the gates the candidates failed, keeping it is a decision. That decision goes in the record like any other. Note what would have to change for the question to be reopened.
Whatever the path, the POC's outputs keep their value. The test plan, the environment checklist, the vendor answers, and the feedback forms are the starting point for the next evaluation. Next time it will take half as long.
The decision-record template
The table below is the one-page record with example content in a two-candidate evaluation. Replace the examples and keep the rows.
| Section | Content | Example |
|---|---|---|
| Candidates | Tools evaluated, dates, environment | A and B; three weeks each in the lab subnet; identical VMs and credentials |
| Tests | Plan reference; count of gates and scored tests | Test plan as agreed before the trials; 4 gates, 9 scored tests, 3 veto conditions |
| Gate results | Pass/fail per candidate per gate | A: 4 of 4 passed. B: 3 of 4 — failed dependency suppression |
| Weighted scores | Totals and by-role breakdown for survivors | A: total 3.8 of 5; alerting 4.2, daily views 3.5, discovery 4.0, reports 3.2, maintenance 4.1 |
| Choice | One line | Candidate A |
| Reasons | Three to five evidence-based sentences | Only candidate to pass all gates; discovery matched the lab inventory; backup restored in under an hour; on-call rated alerts readable |
| What we gave up | Losers' strengths; winner's known limits | B's reports were stronger and its console faster; A's custom checks require the documentation; A's mobile view is basic |
| Vendor answers | Notable written commitments and limits | A: adds a probe at roughly double our size, licensed separately; report scheduling improvement committed in writing |
| Who decided | Evaluators, skeptic, sign-off | Five evaluators by role; skeptic; IT manager signed |
| Outcome path | Proceed, revisit, extend, or stay put | Proceed to negotiation, then migration plan |
| Close call? | Yes/no and why | No — decided at the gate |
| Appendices | Where the evidence lives | Scoring sheet, 10 feedback forms, 2 written vendor replies, lab inventory, alert channel export |
Where to go next
A decision made this way puts gates first and weights second. Surprises are handled by rules written in advance, and everything is on one signed page. You can defend that decision to your manager, to your team, and to yourself on the first bad day with the new tool. It will also be boring to read, which is the highest compliment a decision record can be paid. It closes the loop that began with a test plan written before anything was installed. The rest of Running a Fair Proof-of-Concept covers the earlier steps: the plan, the environment, the vendor questions, and the team's sessions. Suppose the question underneath your decision is a broader one: whether to buy at all, or build on something open. In that case, Open Source vs Commercial is the neighbouring topic to read next.
Put this into practice with SysPeep Monitor
Whatever you decide, decide it from the sheet. If the SysPeep Monitor free trial was one of the lanes in your POC, it belongs in the decision record on exactly the same terms as every other candidate. In that case, record the gates it passed or failed, the scores each role gave it, and what you would be giving up. The download is at /download/ if you want it in the next round.
Download Free TrialLearn About SysPeep