Home › Topics › Running a Fair Proof-of-Concept › A Fair POC Environment

Building a Fair POC Environment


The most common proof-of-concept environment is a laptop, a spare switch, and good intentions. The tool installs, the switch appears on the map, and every light is green. After two weeks you know exactly one thing: the tool can monitor a switch that nothing is wrong with. Meanwhile production has four hundred devices, six of them strange and two of them broken in ways nobody has written down. The lab has three, all of them polite. That is not nothing, but it is not what you are about to spend money and years of your working life on.


A monitoring tool earns its keep on the parts of your network that are strange, busy, and occasionally broken. So a fair proof of concept (POC) environment has to contain strangeness, traffic, and breakage, in a controlled dose. It needs credentials you can revoke and a cleanup plan you wrote before the install. It also has to be the same environment for every candidate you test. If it is not, you are comparing labs rather than tools. In that case, the lab that happened to be bigger on the day will decide which tool you buy.


By the end of this guide you will know which devices to include, how much scale is enough, and why alerts must actually flow somewhere. You will know how to hand out credentials safely and where to place the trial on the network. You will know why a vendor-hosted demo is not a substitute, and how to tear it all down afterwards. There is a checklist at the end you can use as the "environment ready" gate in your test plan.


In this guide

  • What "fair" means for an environment
  • Representative devices, awkward ones included
  • A realistic slice of scale
  • Real alerts flowing to a test channel
  • Read-only, separate credentials
  • Placement that mirrors production
  • Why a vendor-hosted demo is not a POC
  • Auto-discovery as the first honest test
  • The same environment for every candidate
  • Cleaning up afterwards
  • The environment checklist
  • Where to go next

What "fair" means for an environment


Fair has two meanings here, and both matter. Fair to you means the environment is realistic enough that a tool which passes in the lab will also work in production. A POC that only tests the easy parts of your network is unfair to your future self. Fair to the candidates means each one meets the same devices, the same traffic, the same credentials, and the same obstacles. A candidate tested on a bigger or stranger lab than its rivals has been handicapped, and one tested on a smaller one has been flattered.


The environment therefore gets built once, before the first candidate is installed, and then frozen. It is described in the POC test plan in enough detail that anyone could rebuild it. And "environment ready" is a gate the plan does not pass until the checklist at the bottom of this page is complete.


Representative devices, awkward ones included


The guiding rule is one of each kind you actually own. Not one of each kind you wish you owned, and not only the modern ones with good documentation. Walk through your inventory and pick a representative from every category that monitoring will have to cover:

  • A switch from each vendor family you run, including the oldest model still in service.
  • Your firewall, or a spare of the same model if the production one cannot be touched.
  • Include a Windows server for WMI, the Windows management interface monitoring tools use to read disk, memory, and services. Also include a Linux server (for SNMP or SSH).
  • A hypervisor host, if you virtualise.
  • A wireless access point, and its controller if there is one.
  • A UPS, a printer, and whatever building or security kit lives on your network: cameras, door controllers, HVAC gateways.
  • The device you would least like to explain to a vendor's engineer.

That last item is the important one. Every network has an awkward device. It could be an appliance that speaks SNMP badly. (SNMP is the Simple Network Management Protocol, the standard way a monitoring tool asks a device for its counters.) Or it could be a management card that stopped getting firmware long ago, or a box whose vendor documentation is a single forum post. If the tool cannot cope with it, you need to know now, while "cannot cope" costs nothing. If the tool can cope, perhaps through a custom check you build by hand, you learn how much effort the edge cases will take. That is a better predictor of your future workload than anything on the happy path.


Where you cannot spare a production device, a spare of the same model is fine. Where you cannot spare either, a carefully scoped read-only credential on the production device is acceptable for polling. This is acceptable as long as the test plan forbids any scenario that changes its state. Do not, however, substitute a different model and assume it is close enough; the awkwardness is usually in the specific firmware.


At Northgate Retail the lab held a core switch, two access switches, and a Windows server, all recent and all well documented. The winning tool discovered them in four minutes, and the scoring meeting took twenty. Production also had a hundred and sixty tills behind a point-of-sale gateway that answered SNMP only on a non-standard port. Nobody had mentioned this because nobody thought of the tills as network devices. The tool could be taught about them, eventually, one custom check per till. The review recorded the tills as "out of scope for the POC". They had not been out of scope for the shop.


A realistic slice of scale


One switch with twenty-four ports tells you nothing about polling behaviour. Polling is the act of the monitoring server asking every device every few minutes for its counters. Its problems only appear with volume. These include timeouts when a device is slow to answer, polling cycles that overrun their interval, and databases that grow faster than expected. Dashboards also turn sluggish once there are a few thousand sensors instead of forty. You want enough scale in the POC to see those behaviours start, not enough to recreate production.


A practical way to choose the slice is by location rather than by count. Choose one whole floor, one small branch office, or one subnet (one block of network addresses) with everything on it. That gives you hundreds of interfaces and a mix of device types without any selection bias toward the well-behaved. During the POC, watch three numbers on each candidate. Watch how long a full polling cycle takes, and whether that time grows as you add devices. Also watch how much the database grows per day. Write them down; they are the raw material for the "what happens at twice our size?" conversation in the vendor questions.


Real alerts flowing to a test channel


A tool that is never allowed to send an alert cannot be judged on alerting, and alerting is the thing you will live with every day. So the environment needs a destination: a dedicated mailbox, a chat room, or both. It exists only for the POC and receives every notification each candidate generates. Not your personal inbox, and not the production on-call channel, because you want to count the noise without suffering it.


Once the destination exists, resist the urge to tune. For the first several days, leave every candidate on its default thresholds and count what arrives. The count is data. A tool that sends two hundred messages in its first weekend on a healthy network is telling you something about its defaults. It also says something about how much tuning work your team is signing up for. Only after that baseline is recorded do you start the "quiet the noisy alert" scenario from the test plan. Record how many messages arrive per day after tuning as well.


Real alerts also mean real events. Stage them safely and on schedule: pull a cable on the lab switch at a known time, stop a service on the lab server, fill a disk. Every candidate gets the same staged events at comparable times, so that "noticed it in two minutes" can be compared against "noticed it in twelve."


Read-only, separate credentials


Everything the trial uses to talk to your devices should be created for the trial and destroyed after it. Not kept in case. Not reused for the winner. Destroyed. This is partly security hygiene and partly evaluation hygiene. A separate identity lets you see, in device logs and firewall logs, exactly what each candidate did. It lets you revoke access with one change when the POC ends. The list is short:

  • A separate SNMP community or SNMPv3 user, read-only, that exists only on the lab devices. SNMP (Simple Network Management Protocol) is how most network devices report their health. A community string is effectively a password, and a read-only one cannot change device configuration. If some lab devices do not have SNMP turned on yet, enabling SNMP on devices covers the general procedure.
  • A dedicated, low-privilege Windows account for WMI, granted only the rights to read performance data on the lab servers. Not a domain administrator, however convenient that would be at two in the afternoon.
  • A dedicated SSH user or key on Linux hosts with read-only commands, if a candidate collects over SSH.
  • Read-only API accounts for any hypervisor or controller the candidates will query.

Before installing the first candidate, prove the credentials work from the monitoring server using a plain command-line tool. That way, a discovery failure later is a finding about the tool and not about your lab. On the server that will host the trial:


snmpwalk -v2c -c poc-readonly 10.20.0.1 system
SNMPv2-MIB::sysDescr.0 = STRING: (device description)
SNMPv2-MIB::sysUpTime.0 = Timeticks: (2141712) 5:56:57.12
SNMPv2-MIB::sysName.0 = STRING: lab-core-sw

snmpwalk -v2c -c poc-readonly 10.20.0.40 system
Timeout: No Response from 10.20.0.40

The first device answers with its description, uptime, and name; the second one times out. That second result is exactly what you want to find now. It could be a firewall rule, a missing community, or a device that needs SNMP switched on. You want to find it now rather than discovering it as a mysterious gap in a candidate's discovery results. Keep a simple register of every credential you create, where it is configured, and who knows it. That register is also your cleanup list.


Tip: never hand a vendor's engineer production credentials, and never let a trial reuse the community string your production monitoring already uses. If you later need to prove which system polled a device at a given moment, separate identities are the only way to tell them apart in the logs.


Placement that mirrors production


Where the trial server sits on the network is part of the test. Your production monitoring may live in a server VLAN (a virtual LAN, a separately addressed slice of the network) and poll devices across a firewall. It may reach a branch office through a VPN tunnel (a virtual private network, an encrypted link between two sites). If so, the POC should do those things too. A trial server plugged into the same switch as the devices it polls proves that the tool works when nothing is in the way. That is not the condition you will run it in. I have done this. The production server spent its first week waiting on a firewall change request that the laptop never needed.


In practice this means three placement decisions. Put the central server where the real one would go, with the firewall rules a real deployment would need. Opening SNMP, WMI, and the console port through the firewall for the trial is a small, reversible job. And it tells you what the production change request will look like. If your network has segments the central server cannot reach directly, include one. It could be a branch, a DMZ (the fenced-off segment where public-facing servers live), or a plant floor. Require each candidate to place a remote probe or collector there and report back through the firewall. The probe or collector is a small satellite of the tool that polls its own segment and sends the results home. And if you have more than one site, make one remote site part of the lab. That way, the "onboard a new site" scenario runs against real latency rather than a pretend one.


The server itself should match the class of machine you would really use: a virtual machine (VM) with a sensible amount of memory and disk. It should not be the biggest host you own or a laptop. Give each candidate an identical VM from the same template. Snapshot it clean before installation. A snapshot is a saved copy of the VM's whole state that you can roll back to. Taking one also makes the backup-and-restore scenario and the cleanup straightforward.


Why a vendor-hosted demo is not a POC


Most vendors can offer a hosted demonstration instance. This is a running copy of the product, full of tidy sample data, that you can log into from a browser. It is useful, and you should accept it. It is the quickest way to learn the interface and to work out which of your scenarios are even worth running. But it is not a proof of concept, for one simple reason: it tests their network. The devices in it were chosen by the vendor. The data was generated under ideal conditions. The alerts were tuned by people who know the product. And nothing in it is awkward. Nothing in it is yours.


The same is true of a demo where the vendor's engineer shares a screen and drives. Everything you see is real, and everything you see was chosen. Treat these sessions as step zero: a chance to ask questions and understand the shape of the product. Treat the moment the installer lands on your own VM as the moment the POC actually begins. The scoring rule in your test plan should reflect this. Nothing observed only in a hosted demo earns a top score, because it has not yet been observed on your network.


Auto-discovery as the first honest test


The first thing you do with each installed candidate is point its discovery at the lab subnet and walk away for an hour. Auto-discovery is the feature by which a monitoring tool scans an address range, finds the devices on it, identifies what they are, and lists their interfaces and sensors. The guide how auto-discovery works explains the mechanics if you want them. As a POC test it is nearly perfect, because you already know the right answer: you built the lab, and you have the list.


Compare the candidate's results against your list line by line. Which devices were found? Which were found but misidentified, shown as a generic device with a ping check and nothing else? Which interfaces were missed? Did it find things you forgot were there? Note how long the scan took and how much manual correction it needed afterwards. Do the same for every candidate and you have your first side-by-side result, obtained from an identical starting point and free of anyone's opinion.


The same environment for every candidate


Once the lab is built, it changes only in ways that apply to every candidate. Do not add the interesting new device that arrived halfway through the second trial. Do not remove the awkward UPS because it embarrassed the first candidate. Do not widen the subnet for the third one because its discovery was fast. A lab that changes between candidates is two labs, and two labs compare nothing. If a change is genuinely necessary, make it, record it, and re-run the affected scenarios on every candidate that has already completed them.


Whether you run candidates in parallel or one after another affects the environment in one specific way: polling load. Two or three tools polling the same lab devices at once put a real, if modest, extra load on those devices and on the network. Each tool's results can be affected by the others' timing. For a lab-sized slice this is usually tolerable. Parallel running lets you compare how each candidate reacted to the same staged event at the same moment. If you choose sequential runs, reset each device's counters and the alert channel between candidates so the second one starts as clean as the first. Either way, give each candidate its own VM from the same template, its own credential set from the register, and the same staged events on the same schedule.


Cleaning up afterwards


A POC that leaves things behind is a small security incident waiting to be discovered by whoever audits you next. The cleanup should be in the plan before the install, and it should run for every candidate, including the winner. The winner's trial instance is not your production deployment; it is a lab build with lab credentials, and production gets built properly later. The list below mentions ACLs, access control lists: the rules on a device or firewall that say which addresses are allowed to talk to it. They are the item most often left behind, because they are invisible until someone reads the config.


  • Remove the trial SNMP community or SNMPv3 user from every lab device, and the ACLs that allowed the trial server to poll.
  • Disable and delete the dedicated Windows, SSH, and API accounts. Check the register so none is forgotten.
  • Uninstall any agents or remote probes from servers and remote segments. Confirm with a process list, not with the uninstaller's cheerful message.
  • Remove the firewall rules opened for the trial, and any DNS entries or static addresses assigned to the trial server.
  • Delete or archive the trial VMs. If you keep a snapshot for the decision record, store it somewhere it will not be mistaken for a live system.
  • Close the test alert mailbox or chat room after exporting its contents; the message counts are evidence.
  • Tell each vendor the trial has ended, and ask them to deactivate any trial licences on their side.

The environment checklist


This table is the "environment ready" gate. Every row should be true before the first installer runs, and the same table, with the cleanup column, closes the POC.


AreaReady whenCleanup
DevicesOne of each device type you own is in the lab, including the awkward ones; the list is written downTrial credentials and ACLs removed from each
ScaleA whole floor, branch, or subnet — hundreds of interfaces, not one switchCounters and polling ACLs reset
Alert channelDedicated mailbox and/or chat room exists and receives a test message from the serverContents exported, channel closed
SNMP credentialsSeparate read-only community or v3 user on every lab device; snmpwalk succeeds from the trial serverCommunity or user deleted
Windows / SSH / APIDedicated low-privilege accounts created and listed in the registerAccounts disabled, then deleted
ServerIdentical VM per candidate from one template, sized like production, snapshot taken cleanVMs deleted or archived
PlacementServer in the production VLAN; firewall rules opened and documented; one unreachable segment for a probeFirewall rules and DNS entries removed
Staged eventsCable pull, service stop, disk fill scheduled and known to testersDevices returned to normal
Inventory answer keyThe list of every lab device and interface, for comparing discovery resultsKept with the decision record
Vendor accessAgreed in writing: what each vendor's engineer may do, with which credentialsAccess revoked, vendors informed

Where to go next


An environment built this way is representative, realistically sized, alerting for real, credentialed separately, placed like production, and frozen for every candidate. It turns a trial into an experiment whose results you can defend. It is more work than a laptop and a spare switch. And the extra work is the whole point: the three polite devices would have passed anything. With it in place, the test plan's scenarios have somewhere honest to run. The vendors have something concrete to answer questions about. And the team has a system they can put their hands on. The other guides in Running a Fair Proof-of-Concept cover those next steps: the questions to ask each vendor, involving the team in the verdict, and turning the results into a decision.


Put this into practice with SysPeep Monitor

If a Windows-hosted tool is among your candidates, the SysPeep Monitor free trial installs on a VM like the one described here. It discovers the lab subnet and can place a lightweight remote probe in the segment your firewall keeps apart. Give it the same lab, the same credentials, and the same staged events as every other candidate, and score what you see.

Download Free TrialLearn About SysPeep