Home › Topics › Writing Custom Checks and Sensors › Collectors for SNMP-less Devices

Small Collectors for Devices Without SNMP


Every network has a few of them. The air-handler controller whose only interface is a web page with a green "Normal" on it. The access-control panel that writes a log file to a share and nothing else. The temperature probe on a serial cable. The storage box that is managed exclusively through a vendor utility. The alarm system whose status lives in a cloud portal you log into with a browser. None of them speak SNMP or have an agent. All of them matter to someone who will call you when they stop working.


The answer is a collector. This small program reads whatever the device does offer and turns it into a proper monitoring result. It hands that result to the monitoring system. This guide walks through the five shapes those devices come in, the five approaches that work, and the ways results get into the monitor. It also covers — the part people skip — what it costs to own a collector once it exists. It ends with an annotated status-page scraper, fragility notes included, because that is the one nearly everyone ends up writing.


The contract a collector's output must meet is the same as for any check: a state, a value, a message, timeouts, UNKNOWN when it could not measure. That is covered in The Anatomy of a Custom Check; this guide is about the messy input side.


In this guide

  • The devices that speak nothing
  • Five approaches that work
  • Publishing results into the monitor
  • The maintenance warning
  • The synthetic-check crossover
  • Device type, approach, fragility
  • An annotated status-page scraper
  • Where to go next

The devices that speak nothing


"Unmanaged" covers a lot of ground, from a plastic desktop switch to a building-automation controller worth more than your car. The general picture of what such devices can and cannot tell you is drawn in Devices You Can't Manage. For the purpose of building a collector, five shapes cover nearly everything you will meet:


  • A controller with only a web status page. Building systems, lab instruments, older UPS and PDU units, industrial gateways. The page is generated for a human, sometimes by a script that runs in the browser, and often behind a login.
  • A device that only writes a log. An access-control panel, an old phone system, a badge printer, a backup appliance. Somewhere there is a text file, on the device or on a share it writes to, and the lines in it are the only signal.
  • A serial or USB gadget. Environmental probes, some UPS units, modems, door sensors. It is plugged into a nearby computer, and that computer is the only thing that can read it.
  • A vendor command-line tool. Storage arrays, tape libraries, some backup and antivirus agents: the health status exists, but only the vendor's utility on a management host can fetch and print it.
  • A cloud-only thing with a portal. Alarm systems, managed printers, smart thermostats, some network gear. The device reports to the maker's cloud; you see it through their web portal, and perhaps through their API.

What these have in common is that the monitor cannot ask them anything directly. Something has to stand between the device and the monitor and translate. That something is small, and it is yours. The rest of this guide is about building it well enough that it does not become the thing that fails.


Five approaches that work


Scrape the status page

Fetch the page with a timeout, find the value you want, and turn it into a state. Before you parse HTML, open the page in a browser with the developer tools' network panel showing. Many "status pages" are a thin shell that calls a JSON endpoint behind the scenes. That endpoint is a far better thing to read — structured, small, and less likely to change when someone restyles the page. If it really is HTML, anchor on the label next to the value ("Supply Temperature") rather than the position of the value on the page. In that case, cap how much you read and treat "label not found" as UNKNOWN. Then version the scraper. Put a version string in the code and in its output. Keep a saved copy of the page as a test fixture. Bump the version whenever the layout forces a change. That way, "which scraper wrote this number" is never a mystery. The full sketch at the end of this guide does all of this.


Tail the log and turn lines into states

Read the newest lines of the file, match the ones that carry status, and map the most recent match to a state. Two rules make this reliable. First, silence is a state. A device that normally writes a status line every five minutes and has written nothing for thirty is not "fine, no errors." It is missing. So the collector must compare the newest line's timestamp to the clock and report CRITICAL or UNKNOWN when it is stale. Second, remember where you were. Track the file position or the last timestamp seen. Handle the file being rotated or truncated (position past end of file means start again). And never reread a large log from the top every minute.


line = newest_line_matching(path, r"^(\S+ \S+) .* status=(\w+)")     # timestamp, status word
if line is None or age_seconds(line.timestamp) > max_age:
    state, message = "CRITICAL", "no status line in the last " + str(max_age) + " s"
else:
    state = "OK" if line.status == "NORMAL" else "CRITICAL"

Wrap the vendor tool and parse its output

Run the utility from a script with a hard timeout, capture its output and exit code, and parse the one field you need. Everything from the SSH-and-CLI discipline applies. Anchor on stable fields and keep a saved sample as a fixture. Return UNKNOWN when the format does not match. Two extra hazards are specific to vendor tools. Some prompt interactively (license acceptance, "press any key") and will hang a script forever. Find the non-interactive or quiet flag, and put a timeout on the subprocess regardless. And some tools have side effects — a "status" command that also triggers a rescan, say. So read the documentation and pick the command that only reads. Run it as an unprivileged account with just enough rights to query.


Read a sensor via a small script on a nearby host

For a serial or USB device, the collector runs on whatever machine the cable is plugged into. It opens the port with a serial library (described by role: it gives you open, read with a timeout, close). It asks the device for a reading in whatever tiny protocol it uses, and converts the answer. Three things bite. Port names change (a different USB socket, a reboot). So identify the device by its serial number or a stable path where the operating system offers one. Reads must have timeouts, because a disconnected cable produces silence, not an error. And that nearby host has just become part of your monitoring, so it needs to be watched too. And it should be a machine that stays on, not the receptionist's desktop.


Poll the vendor portal's API, if it has one

Many cloud-managed devices report to their maker's service. Many of those services offer an API that returns the device's state and, crucially, when it was last heard from. Poll it with a token held by reference and respect its rate limits. Treat "last seen" as a first-class value. A device the portal has not heard from in an hour is down as far as you are concerned, whatever its cached status says. Two cautions. The portal is a dependency you do not control, so a portal outage must show up as UNKNOWN rather than as every device going CRITICAL at once. And if there is no API, only a login page, think hard before scripting a browser against it. That is the most fragile collector of all, and the maker may forbid it.


Publishing results into the monitor


A collector that has worked out the state now has to hand it over. Monitoring products differ here more than anywhere else, so the honest instruction is: check your product's documentation for how it accepts custom results. The routes fall into three families.


  1. Push, or passive, results. The collector sends the result to the monitor. The collector uses an HTTP endpoint, a message the monitor's agent accepts, or a command-line submission tool. The monitor records it as if it had run the check itself. Cleanest when supported; the collector owns the schedule, the monitor owns alerting.
  2. A file the monitor reads. The collector writes a small result file somewhere the monitor can read it. The file is one line of text or a JSON object, always with a timestamp. The monitor has a check that reads the file. This route is simple and universal. The danger is a stale file that keeps saying OK after the collector has died. That is why the timestamp is not optional.
  3. An SNMP-style proxy. The collector host runs an SNMP agent that can be extended with scripts. The collector's values are exposed under an OID of your own. So the monitor's ordinary SNMP collector reads them like any other device. More setup, but it turns a strange device into a normal-looking one from the monitor's point of view.

# A result file: one object, always timestamped. Stale = UNKNOWN, never OK.
{"target": "ahu-3", "check": "supply_temp", "state": "OK", "value": 21.5,
 "unit": "C", "message": "supply temperature 21.5 C", "checked": 1700000000}

Whichever route you take, build in staleness. Every result carries the time it was measured. The monitor (or a tiny check in front of it) treats a result older than two or three intervals as UNKNOWN. Without that, a collector that crashes leaves its last "OK" on the dashboard indefinitely. That is the exact failure the collector was built to prevent.


The maintenance warning


Here is the part that is easy to nod along to and hard to do. Every collector is code you own. It has a dependency on a page layout, a log format, a tool's output, a cable, or a vendor's API. Every one of those will change without asking you. When a dependency changes, the collector will either fail loudly (good) or keep reporting a plausible number that is no longer true. The latter happens if it was written carelessly (very bad). The discipline that keeps this manageable is short:


  • Keep it small. One collector, one device type, one or two values. A five-hundred-line script that handles eight devices is a liability nobody will touch.
  • Document it in the file. What device, what it reads, what the fixture looks like, who owns it, what to do when it fails. The header comment is the documentation most likely to be read.
  • Put it in version control with its fixtures and its wrapper. That way, a change is a diff someone can review and a broken change can be reverted. The Monitoring as Code guides make the wider case for treating monitoring configuration this way.
  • Fail loudly. Layout not found, tool exited nonzero, port did not answer: UNKNOWN with a message that says what happened, never a default value.
  • Monitor the collector itself. A collector that stops running is silent, and silence looks like health. Have it emit a heartbeat, or have the monitor alert when its results go stale. The pattern is in Heartbeats and Dead-Man Switches. Silence must be an alert.
  • Plan to retire it. When the device gets a firmware update with SNMP, or is replaced, delete the collector. Collectors for devices that no longer exist are a surprisingly common source of confusing alerts.

Tip: Before writing a collector, spend fifteen minutes checking whether the device really has no other door. Walk it with SNMP in case a default community is on. Look for an API in the documentation. Ask the vendor whether a management card or a newer firmware adds one. A collector you never write is the cheapest collector to maintain.


The synthetic-check crossover


A status-page scraper is, when you squint, a synthetic user. It visits the page on a schedule, the way a person would, and reports what it saw. That means the discipline of designing synthetic probes applies to it. That includes what to assert, how often to run, and how to avoid false alarms from the probe's own environment. Designing Synthetic Probes covers that ground. The practical crossover is worth stating plainly: run a plain "is the status page answering" HTTP check alongside the scraper. When the scraper reports UNKNOWN and the HTTP check reports CRITICAL, the controller's web server is down. When the scraper reports UNKNOWN and the HTTP check is OK, the layout changed. Two cheap checks turn one ambiguous alert into a diagnosis.


Device type, approach, fragility


Device typeApproachFragilityWhat breaks it
Controller with a web status pageRead its JSON endpoint if one exists; otherwise scrape the HTML, anchored on labelsHigh (HTML), medium (JSON)Restyled page, renamed label, login change, script-rendered content
Device that only writes a logTail, pattern-match, and treat a stale log as an alertMediumFormat drift, log rotation, wrong clock, share permissions
Serial or USB gadgetA small script on the host it is plugged into, with read timeoutsMediumPort renamed, cable moved, host rebooted or switched off
Vendor command-line toolWrap with a timeout, parse one field, keep a sample as a fixtureHighTool update changes output, interactive prompt, hidden side effects
Cloud-only device with a portalPoll the vendor API, alert on "last seen" ageMediumToken expiry, rate limits, portal outages, API changes
Truly dumb deviceWatch it from outside: ping, port, power draw from a metered PDULowTells you it is on, not that it is working

An annotated status-page scraper


This reads one temperature from a controller's status page and reports it as a check. It is shown in Python because the standard library covers everything it needs, but the structure is what matters. Read the comments; they are the fragility notes.


"""scrape_supply_temp.py - read ONE value from a controller's status page.
FRAGILE BY DESIGN: the page exists for humans. This file is versioned; a saved copy of the
page lives beside it as a fixture; when the layout changes, update PATTERN and bump VERSION."""
import argparse, json, re, sys, time, urllib.request, urllib.error

VERSION = "ahu-status-page/layout-A"          # appears in every result: "which scraper said this?"
STATES  = {"OK": 0, "WARNING": 1, "CRITICAL": 2, "UNKNOWN": 3}

# Anchor on the LABEL, not the position: labels survive restyling better than table cells do.
PATTERN = re.compile(r"Supply\s+Temperature\D{0,40}?(-?\d+(?:\.\d+)?)", re.IGNORECASE)

def fetch(url, timeout):
    req = urllib.request.Request(url, headers={"User-Agent": "scraper " + VERSION})
    with urllib.request.urlopen(req, timeout=timeout) as r:        # timeout: the page may never answer
        return r.read(200000).decode("utf-8", errors="replace")   # size cap: never read a runaway page

def main():
    p = argparse.ArgumentParser()
    p.add_argument("--url", required=True)
    p.add_argument("--warn", type=float, default=24.0)
    p.add_argument("--crit", type=float, default=28.0)
    p.add_argument("--timeout", type=float, default=8.0)
    p.add_argument("--fixture", help="parse this saved page instead of fetching (tests)")
    a = p.parse_args()
    state, message, value = "UNKNOWN", "check did not complete", None     # fail closed
    try:
        html = open(a.fixture, encoding="utf-8").read() if a.fixture else fetch(a.url, a.timeout)
        m = PATTERN.search(html)
        if not m:
            message = "label not found: layout changed? (" + VERSION + ")"   # loud, never a default
        else:
            value = float(m.group(1))
            state = "CRITICAL" if value >= a.crit else "WARNING" if value >= a.warn else "OK"
            message = "supply temperature " + str(value) + " C"
    except urllib.error.HTTPError as e:
        message = "status page returned " + str(e.code)                   # UNKNOWN: we asked about temperature
    except (urllib.error.URLError, TimeoutError, OSError) as e:
        message = "could not fetch status page: " + type(e).__name__      # UNKNOWN: question never asked
    print(json.dumps({"state": state, "value": value, "unit": "C", "message": message,
                      "scraper": VERSION, "checked": int(time.time())}))
    sys.exit(STATES[state])

if __name__ == "__main__":
    main()

The fragility notes, spelled out. The pattern anchors on the label and allows a bounded amount of markup between label and number. If the vendor renames the label, the scraper says so and returns UNKNOWN rather than guessing. The version string is in the code and in every result, so when two collectors disagree you know which layout each was written for. The fixture switch lets a test run the parser over a saved page without touching the device. Save a new page whenever the firmware changes and run the test before trusting the numbers again. The size cap and timeout mean a broken page cannot hang or exhaust the collector. And an HTTP error is UNKNOWN here, because the question was about temperature. The companion HTTP check described above is the one that turns a dead web server into CRITICAL.


Three things the sketch deliberately leaves out, because they depend on the device. A login: if the page needs one, use a session cookie or basic authentication with the password held by reference. In that case, log in once per run, not per value. Script-rendered pages: if the number is not in the HTML the server sends, the scraper will never find it. In that case, go back to the developer tools and use the endpoint the page calls. And politeness: fetch the page once per interval and extract everything you need from that one fetch. That is because the web server on a small controller was never designed to be polled by a computer.


Where to go next


A device without SNMP is not a device you cannot monitor. It is a device that needs a small translator. Read what it offers. That could be a status page (preferably its JSON endpoint), or a log with staleness treated as an alert. It could be a vendor tool wrapped with a timeout, or a serial port on a nearby host. Or it could be a portal API with "last seen" watched. Publish a timestamped result into the monitor by whichever route your product documents. Keep the collector small, versioned, documented, and monitored. That way, when its input changes it fails loudly and when it dies its silence is an alert.


The language mechanics for building these collectors are in Writing Checks in Python and Writing Checks in PowerShell. The guide to choosing SNMP, SSH, or an API for devices that do have a proper door is SSH, SNMP, and API Collection Patterns. All of it lives under Writing Custom Checks and Sensors.


Put this into practice with SysPeep Monitor

For everything on the network that does speak a management protocol, SysPeep Monitor already does the collecting. It discovers devices and interfaces and gathers health and performance data over SNMP, WMI, SSH, and similar collectors. It applies thresholds and alert rules from a single web console. The stubborn devices in this guide need a translator of your own. Check the product documentation for how it accepts custom results, and let the built-in collectors handle the rest. The free trial installs on one Windows machine.

Download Free TrialLearn About SysPeep