Vendor Questions That Reveal Weaknesses
A proof of concept shows you what a tool does on your network for a few weeks. It cannot show you what happens at twice your size, or what the third upgrade feels like. It cannot show how the support desk behaves at eleven on a Sunday night. For those, you have to ask. The quality of what you learn depends almost entirely on the quality of the questions and on how well you listen to the answers. "Does it scale?" gets you "yes". "What is the first thing that gets slow?" gets you an engineer thinking, and the thinking is the answer you came for.
"Questions that reveal weaknesses" does not mean trick questions. Every product has limits, and a good vendor knows theirs better than you do. The questions below are honest ones that anyone selling infrastructure software should be able to answer clearly. That is because their existing customers have already lived through every one of them. A vendor who answers them well is showing you strength. A vendor who cannot is showing you something too, and both are useful.
You will come away with eight groups of questions, and a way to tell a good answer from an evasive one. You will have a method for putting the same questions to every candidate fairly, and a table you can paste into your POC notes. Think of the vendor as a partner in the POC. Your job is to make it easy for them to be honest.
In this guide
- Why honest questions belong in a POC
- The eight groups of questions
- How to read the answers
- The same questions, every candidate, in writing
- The question and good-answer table
- The fairness note
- Where to go next
Why honest questions belong in a POC
A POC tests behaviour you can observe. Questions cover behaviour you cannot: the future, the failure modes you are not willing to stage, and the parts of the relationship that are not software at all. Without them, the decision is made on the two or three weeks you happened to watch, which is a small window onto a tool you may run for many years.
Questions also do something the hands-on testing cannot: they reveal how the vendor thinks about their own product. Someone who can say "that breaks at around this point, and here is what you do about it" has been through it with customers. That person cares enough to have an answer ready. That kind of candour is one of the most valuable things you can buy. And you only find it by asking questions that invite it. (It is also the only one that never appears on the price list.)
The eight groups of questions
The questions below are grouped by what they are trying to find out. You will not ask every one of them; pick the ones that match the risks you care about. But keep at least one from each group, because each group covers a part of the relationship the others cannot.
Scale: what happens at twice our size?
You are evaluating for the network you have. You will run the tool on the network you will have. The scale questions ask the vendor to describe the road between the two.
- If we doubled the number of devices and interfaces, what changes? Does the same server cope, or do we add hardware, probes, or a different database backend?
- What do we have to buy at that point — licences, modules, additional collectors — and what is included in what we are already quoted?
- What is the first thing that gets slow as a deployment grows, and how do you tell it is happening?
- What is the largest deployment of this product you support that looks like ours, and what did it need?
A good answer names a specific bottleneck: the database, the polling engine, or the number of sensors per probe. A probe is a satellite collector that polls one segment on the central server's behalf. The answer also names a specific remedy. It also mentions what you would see first, which tells you the vendor has watched customers hit the limit. A weaker answer is "it scales", with nothing after it. Everything scales until it does not; you are asking where.
Failure: what happens when things break?
Monitoring software has a special problem. It is the thing that tells you about failures, so its own failures tend to be silent. These questions ask the vendor to describe the product on its worst day.
- What happens when the database fills its disk? Does polling stop, does history get dropped, does the console still open, and how do we find out?
- What happens when the central server dies? What is lost, what recovers on its own, and what has to be restored by hand?
- What happens when a remote probe loses connectivity to the server? Does it keep polling and catch up later, or is that window gone?
- How does the product tell us that it, rather than the network, is unhealthy?
- If we want the monitoring itself to survive a server failure, what does that look like, and what does it cost in complexity?
Good answers are concrete and slightly uncomfortable: "polling continues but history stops being written. The console shows a warning banner. You will get an email if you configured self-monitoring, which we recommend." Answers of the form "that never happens" or "customers do not report that" should prompt a polite second attempt. Disks fill. Servers die. A vendor who has never seen it is either new or not looking.
Operations: how do upgrades work?
Installation happens once. Upgrades happen for as long as you own the product, and they are where the day-to-day cost of a tool hides.
- How are upgrades delivered and applied? Is it an installer we run, a package, or something more involved?
- How long does an upgrade take on a system our size, and is monitoring down during it?
- What commonly breaks during an upgrade — custom checks, integrations, database schema, remote probes on older versions?
- Can we test an upgrade on a copy before applying it to the real system? Is a second instance for that purpose licensed separately?
- How far behind the current release can we fall before we lose support or the upgrade path?
You are listening for a routine: a documented procedure, a realistic duration, and a known list of things to check afterwards. You also want an honest note about what has bitten customers before. "Upgrades are seamless" is not a routine. It is a hope, and you will be the one holding it during a maintenance window.
Completeness: what is not included in this quote?
This is the single most useful question in the guide, and vendors who are confident in their pricing answer it happily. A quote describes what you are buying; you need to know what you are not.
- What is not included in this quote that a customer like us usually ends up needing?
- Which features we saw in the POC belong to a module or tier that is not in this price?
- What do remote probes, additional users, high availability, and a commercial database backend add, if anything?
- What happens to the price at renewal, and what triggers a change in it?
Line the answers up against the hidden-costs checklist. It walks through the add-ons, seats, maintenance, and infrastructure lines that surface after signing. The vendor question is how you find out which of those apply to this product before you sign. A good answer is a list. A great answer is a list that includes something you had not thought of.
Support: who answers, how fast, and what do they know?
Support is the part of the product you will meet at the worst moments. It is the part least visible during a POC, when the vendor's most helpful people are on your call.
- When we open a ticket, who answers first — a triage desk, or an engineer who knows the product?
- What are the response and resolution expectations for a "monitoring is down" ticket versus a "how do I" question, and in which time zones?
- Is support included in the licence, a separate contract, or tiered?
- Can we open a real ticket during the POC, on something genuinely puzzling, and see how it goes?
- Where does the community live — forums, documentation, knowledge base — and how current is it?
That fourth question is worth acting on. Nothing describes a support desk as well as one real interaction with it. So keep one of your POC problems for a ticket rather than for the friendly engineer's mobile number. Score the experience. Use the front door, even for the small thing.
Exit: how do we get our data out?
You are asking a vendor to describe leaving them, which feels awkward and is completely normal. A vendor who has a clear answer is one who expects to keep you on merit. One term in the list needs a definition. An API (application programming interface) is the documented way another program reads or writes the product's data without a person clicking through the console.
- If we leave, how do we export our configuration — devices, checks, thresholds, alert rules — and in what format?
- How do we export historical performance data, and is there a practical limit on how much?
- What happens to the product when the licence lapses: does it stop, go read-only, or keep running unsupported?
- Is there a documented API or export tool, or does an exit mean a services engagement?
You may never use the answer. But the shape of it tells you how the vendor thinks about your data. And a product with a clean exit is also a product with a clean integration surface, which you will use.
Roadmap: what is committed, and what is an idea?
Every vendor has things coming. The useful distinction is between what is built and scheduled and what someone once drew on a whiteboard.
- Of the features we asked about that are not in the product today, which are committed with a rough timeframe, and which are ideas?
- Are you willing to put a committed item in writing as part of the agreement?
- What has the product stopped doing or deprecated recently, and why?
A good answer sorts cleanly into two piles and does not pretend the second pile is the first. Score the product you can install today; a roadmap item is, at most, a tiebreaker. A roadmap is a direction, not a delivery date. The deprecation question is quietly revealing. Here, deprecation is the formal notice that a feature is on its way out. The question shows whether the vendor makes hard choices or lets the product sprawl.
References: someone our size, please
A reference customer is an existing user the vendor connects you with. The request that makes references useful is specificity.
- Can we speak to a customer of roughly our size, in a similar industry, who has been through at least one upgrade and one support ticket?
- Can we speak to one who migrated to you from another tool, and hear what the migration was like?
When you get the call, ask the reference the same failure, upgrade, and support questions you asked the vendor, and compare. References are chosen by the vendor and will be favourable; the value is in the details they volunteer when you ask something concrete.
How to read the answers
You now have more questions than any one meeting can hold, and you will get answers ranging from crisp to cloudy. Sort them with one rule. Specific and bounded is good. Take this answer: "The polling engine is comfortable to around this many sensors per probe. Past that you add a probe, which is a licensed item." It is specific (it names a number and a mechanism) and bounded (it admits a limit). That answer may reveal a weakness, and it is still the answer you want, because it is one you can plan around.
Vague is a signal to ask again, not a verdict. "It scales very well" and "our customers love the upgrade process" are not wrong; they are simply not yet answers. Ask a narrower version: "What was the largest deployment you personally helped through an upgrade, and what took longest?" Most vendors get specific once you show that specific is what you want. If a third attempt still produces fog, note it; the fog itself is your finding.
"That never happens" is always a prompt for a second question. Databases fill, servers die, probes lose connectivity, upgrades break custom checks. A vendor who has genuinely never seen one of those has not been watching their customers, which is unlikely, or is being polite, which is likelier. Either way, ask gently: "When it does, what would we see?"
And "we don't do that" is honest. It is the most useful answer in the whole exercise, because it saves you from a scenario you would otherwise discover on your own time. Record it as a fact, score the requirement accordingly, and thank them. Do not punish candour more than you would punish the same gap discovered later. If anything, reward it. A vendor who is straight with you about limits will be straight with you about everything else.
Tip: take notes in the vendor's own words rather than your summary of them. "Grows fine" and "adds a probe at roughly this scale, licensed separately" are very different notes from the same conversation, and only one of them helps you later.
The same questions, every candidate, in writing
Fairness applies to questions exactly as it applies to tests. Every candidate receives the same written list, at the same stage of the POC, with the same deadline for a written reply. Written matters for three reasons. It gives vendors time to consult the people who actually know. It produces answers you can quote in the decision record. And it removes the difference between the vendor who happened to send their most fluent presenter and the one who did not.
At Harbourview Clinic the four-week POC went smoothly until week three. Then someone remembered to ask whether the tool could read the patient-monitoring gateway, the one device the whole exercise had been about. The answer was "yes, on the roadmap". Two candidates had already been scored and one had already been sent home. The team put the question to every vendor again, in writing, and the tally changed. The decision record now opens with the gateway question, and the written list goes out on day one, before the first installer.
A workable sequence:
- Send the written list at the start of the POC, so the vendor can answer while you test.
- Hold one follow-up call per candidate, of the same length, to go through anything vague and ask the "when it does, what would we see?" questions.
- Score the answers on the agreed sheet at the same time as the hands-on results, with the written replies attached as evidence.
- If a candidate raises a question you had not thought to ask, add it — and send it to every other candidate too.
The message itself can be short. A plain-text version you can adapt:
Subject: POC questions — written answers by (date) As part of our proof of concept we are asking every candidate the same questions, in writing. Specific, bounded answers are what we are after; "we don't do that" is a perfectly good answer and will be treated as one. 1. Scale: at twice our current device count, what changes and what would we need to buy? 2. Failure: database disk full / central server lost / probe disconnected — what happens, what is lost, how do we know? 3. Operations: upgrade procedure, typical duration, downtime, what commonly needs attention afterwards. 4. Completeness: what is not included in the quote that a customer like us usually needs? 5. Support: who answers first, expected response for outage vs how-to tickets, what is included. 6. Exit: how we export configuration and history; what happens at licence lapse. 7. Roadmap: of the features we asked about, which are committed and which are ideas? 8. References: one customer of our size who has been through an upgrade and a support ticket. Please attach documentation where it exists rather than summarising it. Thank you — we know these take time.
The question and good-answer table
| Group | Core question | What a good answer looks like | Ask again if |
|---|---|---|---|
| Scale | What changes at twice our size? | Names the bottleneck, the remedy, and what you would notice first | "It scales" with nothing after it |
| Failure | Database full / server dead / probe cut off? | Describes what stops, what is lost, and how the product tells you | "That never happens" |
| Operations | How do upgrades work? | A procedure, a duration, a list of things to check afterwards | "Seamless" |
| Completeness | What is not in this quote? | A list, including items you had not thought of | "Everything you need is included" |
| Support | Who answers, how fast, what do they know? | Named tiers, response expectations by severity, what is included | No distinction between outage and how-to |
| Exit | How do we get our data out? | Formats, tools, and what happens at licence lapse | "Nobody leaves" |
| Roadmap | Committed versus idea? | Two clearly separated lists; willingness to put commitments in writing | Everything is "coming soon" |
| References | Someone our size? | A comparable customer who has been through an upgrade and a ticket | Only much larger or much smaller customers offered |
The fairness note
Vendors are not the opposition in a POC. The good ones want you to choose correctly almost as much as you do. That is because a customer who bought the wrong tool is a customer who leaves unhappy. The best vendors will tell you when their product is not the right fit. The questions in this guide are designed to make that honesty easy and to make it count.
So when a vendor says "we don't do that," write it down as a point in their favour on candour even as you score the requirement as unmet. When a vendor's written answers are specific, bounded, and slightly unflattering, treat them as the most reliable data in your whole evaluation, because they are. The tool you eventually choose will have limits either way. The difference is whether you learned them from the vendor in week one or from an outage in year one.
Where to go next
Written answers from every candidate now sit beside your hands-on results. Together, they cover both what the tools do today and what the vendors will be like to live with. The notes worth the most will be the ones where an engineer stopped to think before answering; "yes" never needed a pause. The remaining steps are to bring the whole team into the scoring and to turn everything into a recorded decision. Both are covered in Running a Fair Proof-of-Concept, alongside the test plan and the environment these questions belong to. The neighbouring topic, Licensing Models and Hidden Costs, gives the wider context. It covers how a vendor's licensing affects the answers to the completeness and scale questions.
Put this into practice with SysPeep Monitor
These questions apply to every candidate, and the SysPeep Monitor free trial is no exception. If it takes a lane in your POC, send the same written list to its makers and hold the answers to the same standard. A Windows-hosted service with optional remote probes, a choice of database backends, and high-availability clustering should be able to describe its own limits plainly.
Download Free TrialLearn About SysPeep