Home › Topics › Monitoring Across Multiple Sites and Clients › Probe Fleet Management

Managing a Fleet of Remote Probes


Deploying a probe is a project. You plan it, you do it, it works, you move on. Running thirty of them is not thirty projects. It is a program. That is a standing set of routines that keep every probe healthy, current, trusted, and replaceable. The probes are at sites you rarely visit, behind firewalls you do not control, on hardware you may not own.


The probe is the one piece of your monitoring that lives in somebody else's building. And a fleet of probes is only as good as the least-loved one. I know which of mine that was. It sat on an old desktop under a desk in a branch office. I found out it had been unplugged for a fortnight when the branch rang about something else.


This guide is about the program, not the project. Getting a single probe placed, enrolled, and talking to home is covered step by step in Remote Probe Deployment. That covers what a probe needs from the central platform, where it sits in the branch network, and the trust bootstrap. The bootstrap is how a new probe first proves to the platform that it is yours. Here we pick up on the day after the last deployment and ask what keeps the fleet running for years. That means the inventory, the health checks on the probes themselves, and updates in waves with a way back. It means credential rotation, fast replacement, and the firewall dependencies you must write down. And it means the one page per site that makes all of it possible.


In this guide

  • From deployments to a fleet
  • The probe inventory
  • Watching the probes themselves
  • "Probe silent" is not "site down"
  • Updates in waves, with a way back
  • Rotating credentials across the fleet
  • Replacing a failed probe quickly
  • The probe lives behind someone else's firewall
  • One page per site
  • Where to go next

From deployments to a fleet


Somewhere between the fifth and the tenth probe, individual memory stops working. You no longer recall which site has the probe on the old desktop under a desk. Nor do you recall which one is a VM on the client's hypervisor (the server that runs their virtual machines). You no longer recall which one was updated last month and which one was skipped because the site was closed that day. The fleet needs a lifecycle. That is a small set of stages every probe passes through, each with a trigger, a routine, and a record that gets updated. The table below is the whole program on one screen.


StageTriggerRoutineRecord updated
EnrolledDeployment completesAdd to inventory; add to health dashboard; file the site sheetInventory row created; site sheet
HealthyOngoingProbe-health checks run; monthly fleet reviewHealth dashboard
BehindA new probe release is adoptedScheduled into an update wave; soak; verifyVersion state; update log
RotatedScheduled rotation, staff change, or exposureNew credentials on devices, then probe, then old removedCredential register; rotation date
DegradedHealth check failsTriage: probe, host, site, or link; fix or replaceIncident note
ReplacedHost failure or planned refreshSpare image; site specifics; re-enroll; revoke old identityInventory row updated; old identity retired
RetiredSite or client leavesUninstall; revoke credentials; remove firewall rule; archive recordInventory row archived

Nothing in the table is difficult. What makes it a program is that every stage names a record, and the records are kept. A fleet where the routines happen but nothing is written down is a fleet that works until the person who remembers it goes on holiday.


The probe inventory


The inventory is the register of every probe you run, and it is the first thing to build if you do not have one. A spreadsheet is fine to start; a page per probe in whatever documentation system you already use is better, because you will attach things to it. The fields matter more than the tool.


Probe record (one per probe)

Probe name .......... NWD-PROBE01            (client code + role, matches the platform)
Client / site ....... Northwood, Riverside branch
Physical location ... comms cabinet, first floor, rack position 12
Form ................ VM on client hypervisor | dedicated mini-PC | service on existing server
Host OS ............. (family, and who patches it: us / client)
Addressing .......... static 10.20.0.50/24, gateway 10.20.0.1, DNS 10.20.0.10
Reaches home via .... outbound agent | site-to-site VPN
Version state ....... current | one behind | more than one behind | unknown
Last updated ........ date, by whom, wave name
Credentials held .... SNMP (read-only), Windows poll account, SSH key   (names, not secrets)
Last rotation ....... date
Enrollment expiry ... date the probe's own certificate or token needs renewing
Buffer capacity ..... hours of local storage at current poll rate
Site contact ........ name, phone, hours; who can power-cycle it
Remote hands ........ yes / no; how to reach them
Firewall record ..... link to the connectivity dependencies for this site
Spare ............... image name and date; hardware spare on shelf: yes / no

Two fields deserve a comment. Version state is written as a state, not a number, because the question you ask at fleet scale is "who is behind?" A column that says "one behind" sorts itself into a to-do list. Credentials held lists the names of the credentials a probe carries, never the secrets. That means when a credential is rotated you can find every probe that needs the new one without reading configuration files at thirty sites. The secrets themselves live in your credential vault, labeled with the same client code.


Watching the probes themselves


A probe is part of the monitoring machinery, and a silent probe means a blind site. The probes therefore belong on the top tier of things you monitor, with checks that are about the probe and not about the site behind it. (A probe cannot report that it has stopped reporting. Somebody else has to notice.) The general case — making sure the monitoring platform and everything feeding it is itself watched — is the subject of Monitoring the Monitor. The fleet version is this checklist.


  • Heartbeat. The probe checks in on a fixed interval. Missing two intervals in a row is the anchor alert for the whole checklist.
  • Last data received. Different from the heartbeat, and the one people forget. A probe can say "I am alive" while producing no results, because its polling engine is wedged or its credentials all expired at once. Alert when the age of the newest result from a probe exceeds a few poll intervals even though the heartbeat is fine.
  • Poll success rate. The fraction of the probe's checks that complete. A sag with a healthy probe means something at the site changed — a firewall rule, a credential, a VLAN.
  • Local resources. CPU, memory, and especially disk on the probe host. Warn early on disk, because the buffer lives there.
  • Buffer state. How much unsent data the probe holds. A growing buffer while the site is healthy is a WAN problem announcing itself; a buffer near its limit means history is about to be lost.
  • Clock drift. A probe whose clock is wrong stamps results with the wrong time, and its history will not line up with anyone else's. Check it against the platform's clock.
  • Enrollment expiry. The probe's own certificate or token has a lifetime. Warn well before it ends; a fleet where twenty probes enrolled in the same month will all expire in the same week.
  • Version state. Alert on "more than one behind" as a low-priority, standing item, so the straggler is never invisible.

Put all eight on one fleet health dashboard: one row per probe, columns for each check, sorted so that anything not green rises to the top. It is the screen the fleet owner opens first every morning and the screen that turns the monthly fleet review into a ten-minute scan.


"Probe silent" is not "site down"


When a probe stops checking in, two very different things may be true. The probe (or its host) may have died while the site is perfectly fine. Or the site may be unreachable — circuit down, power gone — and the probe is silent because everything is. These deserve two separate alerts, because they go to different people and start different work.


The distinguishing check is simple: can the central platform still reach something at the site by another path? The branch router's WAN address, an external check against the site's public services, a second probe if there is one. Wire the alerts like this:


  • Probe silent, site reachable. Alert named for the probe, routed to the fleet owner, severity high in business hours and moderate overnight. The site's users are not affected; your visibility is. Devices behind the probe are marked unknown, not down, and their own alerts are suppressed. That is the same "unknown because unreachable" state described in Site Health Rollups.
  • Probe silent, site unreachable. Alert named for the site, routed to whoever handles client-facing incidents, severity high. The probe alert is suppressed as a child of the site alert, so one event produces one page.

Getting this wrong in either direction is expensive. Treat every silent probe as a site outage and the on-call engineer is woken for a hung VM. Treat every silent probe as a probe problem and a real outage waits until morning. The two-alert design costs one extra check and buys years of calm.


Updates in waves, with a way back


Probes are software, and at fleet scale you never update them all at once. The habit of trying one probe first is where every fleet starts; a wave plan is the grown-up version. It answers, before you begin, which probes go first, how long you wait, what "good" looks like, and how you go back if it is not.


Wave plan for one probe release

Wave 0  your own office probe, or a lab probe
        soak: one full business day
        pass: heartbeat, last-data age, poll success, buffer all normal

Wave 1  three to five friendly sites
        chosen for: mixed forms (VM + mini-PC), both transports (agent + VPN),
                    remote hands available, nothing contractual in flight
        soak: two business days
        pass: as Wave 0, plus no new site alerts attributable to the probe

Wave 2  the rest, in batches sized to what you can watch in one morning
        order: sites with remote hands first, sites without last
        soak: overnight between batches
        stop rule: any batch with a failure pauses the plan until understood

Rollback (rehearsed before Wave 0)
        VM probes: revert to the snapshot taken before the update
        others:    reinstall the previous installer kept in the release folder,
                   restore the saved configuration, re-enroll if required
        then:      mark version state, note the reason, hold the plan

The two rules that make waves work are the stop rule and the rehearsed rollback. A plan that continues through a failure is not a plan; a rollback you have never tried is a hope. Snapshot every VM probe before it is updated (a snapshot is a saved copy of the VM you can revert to in minutes). Keep the previous installer alongside the current one. Actually roll one probe back in Wave 0 once. That way, on the night it matters the procedure is a memory and not a discovery. Update the inventory's version state as each wave completes, and keep the platform and probes inside whatever compatibility window your tool documents, in whatever order it prescribes.


Bellweather Logistics ran a probe at each of its forty depots and updated all forty on one Saturday, by hand. They used a remote-desktop session to each host in turn. The new release had a polling fault that only showed on the mini-PC probes, which were thirty-one of the forty. There was no snapshot and no previous installer in any known folder. So the fix was to find the old installer on a technician's laptop and repeat the forty sessions on the Sunday. The review put the weekend at twenty-two hours of remote-desktop work, which it described as "eleven hours, twice". The fleet now updates in waves, and the previous installer lives in the release folder next to the current one, where it has so far never been needed.


Rotating credentials across the fleet


Every probe carries secrets. They include SNMP community strings (the password-like strings that let it read the network gear) and a Windows account for WMI (Windows Management Instrumentation) polling. They include SSH keys for anything it logs into, API tokens for appliances, and its own enrollment credential for talking to the platform. Secrets age, and rotation is a routine, not an emergency. Rotation is triggered on a schedule, when a technician who knew the secrets leaves, when a client asks, or when you suspect exposure. That could be a lost laptop or a probe host that was compromised.


The procedure across a fleet has an order, and the order is the whole trick. Add the new credential on the devices first. Switch the probe to the new credential. Confirm the poll success rate is back to normal, and only then remove the old credential from the devices. The overlap window is what keeps the site from going blind halfway through. Done client by client, it looks like this:


  • Generate the new secret and store it in the vault under the client code, with a rotation date.
  • Add it to every device the probe polls (a configuration push, a script, or a patient afternoon), keeping the old one active.
  • Update the probe — centrally, if your platform pushes credentials to probes, which is the great advantage of central credential storage over per-probe files.
  • Watch poll success for a full interval. Anything that fails is a device that missed the new secret.
  • Remove the old secret from the devices, and from the vault's active list. Record the date on the probe's inventory row.

Rotate the probe's own enrollment credential the same way when your tool allows. Treat the "credentials held" field in the inventory as the map. It tells you which probes are touched by any given rotation, so nothing is missed and nothing is rotated twice.


Replacing a failed probe quickly


Hardware dies, hypervisors are rebuilt, a client's cleaner unplugs the small box under the desk. The measure of a fleet is how long a site stays blind after its probe dies, and the answer should be "hours," never "when someone can get there." Three things make that possible.


A spare image. Keep a pre-built VM image, or a tested install procedure with the exact installer you run. Refresh it with every fleet update so that a restore does not begin with an unplanned upgrade. For sites on physical hardware, a spare mini-PC on your shelf, already imaged, is worth its small cost the first time you courier it overnight instead of driving.


The site specifics. The image is generic; the site is not. The specifics include the static address, gateway, DNS, and the VLAN (the network segment) it sits in. They include the switch port it plugs into, the firewall rule that lets it out, and the proxy if there is one. They include where in the building it lives — everything on the inventory record and the site sheet. Replacement is fast exactly when nothing has to be rediscovered.


A replacement runbook the site can follow. For sites without remote hands (someone on site who can do physical work on your behalf), write the instructions for a non-technical person. The instructions are: plug in the box, connect the blue cable to the port labeled in the photo, press power, call us. Keep a photo of the cabinet on the site sheet. Then, from your desk, re-enroll and confirm the probe's health checks are green. Confirm the site's devices are polling, and retire the old probe's identity. Revoke its enrollment credential and mark it replaced so a recovered or stolen host can never reconnect as if nothing happened.


Tip: rehearse a replacement once a year at a friendly site, on purpose. Swap the probe as if it had failed and time it from "probe silent" to "all checks green." The rehearsal finds the missing photo, the outdated firewall rule, and the spare image that no longer matches the platform, on a quiet afternoon instead of during a real outage.


The probe lives behind someone else's firewall


A probe's connectivity is a set of dependencies you do not control. Outbound to the platform on one port; DNS to resolve the platform's name; time from somewhere; and reachability across the client's internal network to every device it polls. Any of these can be taken away by a client's routine change. That could be a firewall rule cleanup, a new proxy, a VLAN reshuffle, or a DNS server move. The first you hear of it is a silent probe on a Monday.


Write the dependencies down as a short record per site, and make it part of what the client agrees to when the probe is installed. A dependency the client never agreed to is a dependency they are entitled to remove. The shapes of the connection — VPN versus outbound agent, and what each needs — are explained in MSP-Style Monitoring Architectures; the fleet job is keeping the record true.


  • The outbound rule. Source, destination name or address, protocol, port, and the words "please tell us before changing this."
  • DNS and time. Which servers the probe uses, and what happens if the client changes them.
  • Internal reachability. Which VLANs and subnets the probe must reach, and the management protocols it uses — SNMP, WMI, SSH, ping — so a new internal firewall does not strand it.
  • A named client contact. The person who will hear about firewall changes first, and who should tell you.
  • A yearly reconfirmation. Ask the client, once a year, whether anything on the record has changed. Half the time it has.

The best defense is the probe-health checklist. A dropping poll success rate with a healthy heartbeat is what an internal firewall change looks like from your side. And a silent probe with a reachable site router is what an outbound rule cleanup looks like. When either fires, the connectivity record is the first thing you open.


One page per site


Everything above converges on one document: the site sheet. One page per site, kept next to the probe's inventory record, containing the things a technician who has never been there needs at two in the morning. The probe record and connectivity record from this guide. A photo of the cabinet. The address, the site contact and their hours, how to get in, where the power is. The client code and the escalation path. The date it was last confirmed accurate.


The sheet stays true only because the lifecycle table at the top of this guide names it as the record to update at every stage. An update wave touches the version state; a rotation touches the credential dates; a replacement touches the form and location; a reconfirmation touches the firewall record. Documentation that is updated by routine survives. Documentation that is updated by good intentions does not, and a fleet is thirty sites' worth of good intentions.


Where to go next


A probe fleet is a program of small routines. The program includes an inventory with the right fields, eight health checks on the probes themselves, and two alerts that tell a dead probe from a dead site. There are updates in waves with a rehearsed way back, credential rotation in the right order, and replacement measured in hours. There is a connectivity record the client has agreed to, and one page per site that all of it keeps current. Each routine is modest. Together they are the difference between a fleet you run and a fleet that runs you. They are the only reason the least-loved probe under the least-visited desk gets any love at all. The rest of Monitoring Across Multiple Sites and Clients covers the architecture the probes report into, and the site rollups their data feeds. It covers the separation that keeps one client's probe from ever serving another, and the reports that turn the data into proof of service. For the broader questions of a growing platform — performance, high availability, and the deployment steps this guide assumes — see Scaling Monitoring.


Put this into practice with SysPeep Monitor

The routines in this guide assume a platform built around a central service and lightweight remote probes. That is how SysPeep Monitor is designed. It has one Windows service at the center, optional probes at other sites and segments, and a single web console for the dashboards, maps, and reports. The free trial installs on any Windows machine and lets you stand up a first probe in an afternoon.

Download Free TrialLearn About SysPeep