Home › Topics › Writing Custom Checks and Sensors › SSH, SNMP, and API Patterns

SSH, SNMP, and API Collection Patterns


A check is a question, and before you can ask it you have to pick a door. Most devices offer more than one. A managed switch will answer over SNMP, over its command line through SSH, and increasingly through an HTTP API. All three can tell you whether the uplink port is up. They differ enormously in how much work the answer takes to extract, how likely that work is to break next year, and how much the device minds being asked.


This guide is about choosing the door and walking through it well. It covers when SNMP is the right answer (more often than people think). It covers when only the command line has the value. You will learn how to parse its output without building something that shatters at the next firmware update. The guide also covers when an API is worth the extra ceremony. It also covers manners: caching, rate limits, and the small devices that fall over if three checks open SSH sessions at once.


It assumes you know what a check looks like on the outside. That means one question, a state and a value, timeouts, and UNKNOWN when the question could not be asked. If you do not, start with The Anatomy of a Custom Check. Everything here is about the inside: getting the number out of the device.


In this guide

  • Three doors to the same number
  • SNMP: when the value has an OID
  • SSH and the CLI: when only the command line has it
  • An API: when the device or service offers one
  • Politeness: caching, rate limits, and small devices
  • Timeouts everywhere
  • Method, when, parsing risk, effort
  • Worked example: the same value three ways
  • Where to go next

Three doors to the same number


Think of the device as a building holding a fact you want — say, the operational status of port 25. SNMP (Simple Network Management Protocol) is the delivery hatch built for machines. The device keeps its facts in a giant numbered filing cabinet, each at a numeric address called an OID. The device hands one over on request with its type attached. SSH to the command line is the front door built for humans. You log in, type the command an engineer would, and read the screen of text an engineer would read. An API (application programming interface, here meaning HTTP requests that return structured data, usually JSON) is the newer service entrance. It is designed for programs, documented, and typed, but only present where the makers bothered to build one.


The rule of thumb that falls out of that picture: prefer the door built for machines. SNMP when the value has an OID; an API when there is a good one; the command line when there is no other way. The rest of this guide is the detail behind each.


SNMP: when the value has an OID


If the device exposes the number you want under an OID, use SNMP and stop looking. The value arrives already typed, and the protocol is tiny and cheap for the device. Monitoring systems understand it natively, and nothing changes when the vendor redesigns the command-line output. The only real work is finding the OID.


Two operations cover nearly everything. A get fetches one value at a known address. A walk starts at an address and returns everything beneath it, which is how you read a table such as the status of every interface. The hands-on route to both is snmpwalk From Scratch; this guide only needs the shape.


# Version and community string come from the tools' own configuration file
# (defVersion / defCommunity in snmp.conf), so they never appear on a command line.

# One value: device uptime, quiet output (value only), 2-second timeout, one retry
snmpget -t 2 -r 1 -Oqv 10.0.0.2 1.3.6.1.2.1.1.3.0
812449600

# A table: the operational status of every interface, in bulk
snmpbulkwalk -t 2 -r 1 10.0.0.2 1.3.6.1.2.1.2.2.1.8
IF-MIB::ifOperStatus.1 = INTEGER: up(1)
IF-MIB::ifOperStatus.2 = INTEGER: down(2)
IF-MIB::ifOperStatus.25 = INTEGER: up(1)

Use the bulk form of a walk whenever the device supports it. It fetches many values per request instead of one, which is kinder to a small device and faster for you. And read the whole table only when you need the whole table. A check that wants one port's status should get exactly that OID, index included, not walk the interface table every minute and pick one line out of it.


Two things go wrong with SNMP. The number after the dot (.25 above) identifies the row. On some devices interface indexes shift after a reboot or a module change. So a check that hard-codes ifOperStatus.25 may one day be reporting on a different port. The fix is to anchor on the name. Walk the interface name column, find the index whose name matches "uplink-core," and cache it. Re-resolve if the name at that index ever changes. The second is walk cost: a full walk of a large table on a tiny device is not free. Doing it from several checks in parallel is how you discover which switch has the smallest CPU.


SSH and the CLI: when only the command line has it


Sometimes the number you want exists only in the output of a show command. Vendor-specific health details, optical signal levels, a stacking status, an on-box diagnostic — the SNMP tables never had it. And the API, if there is one, does not expose it yet. Then you log in the way a human would, run the command, and parse what comes back. It works. It is also the most fragile collection method there is, and every rule in this section exists because of that.


The fragility is structural. Command output is formatted for a person at a terminal. Columns are aligned with spaces, headers are words, values carry units, long lines wrap, and none of it is a promise. The next firmware release may add a column, rename a header, or change "connected" to "up." Your parser will not get an error. It will get a different string and, if written carelessly, turn that into a wrong number or a silent zero.


# One session, one command, batch mode (never prompts), bounded connect time,
# a dedicated read-only account with a key held on the collector.
ssh -o BatchMode=yes -o ConnectTimeout=5 -i /etc/checks/monitor_key \
    monitor@10.0.0.2 "show interfaces status" > /var/cache/checks/sw01-int-status.txt

Port       Name               Status       Vlan   Duplex  Speed  Type
Gi1/0/1    desk-101           connected    10     a-full  a-1000 10/100/1000BaseTX
Gi1/0/2                       notconnect   10     auto    auto   10/100/1000BaseTX
Gi1/0/25   uplink-core        connected    trunk  a-full  a-1000 10/100/1000BaseTX

Five habits turn that from a liability into something you can live with:


  1. Anchor on the stable fields. Match the row by the port name in the first column, which is an identifier, not by the column position of "Status," which is layout. Extract with a pattern that names what it expects and take only the one field you need.
  2. Expect the format to change. If the pattern does not match, the check returns UNKNOWN with "unexpected output format," never a default value. A silent zero on a graph is far worse than a loud UNKNOWN.
  3. Test with saved samples. Capture the real output once and keep it next to the check as a fixture. A tiny test that runs the parser over the sample tells you in seconds whether a change broke it. The test also documents what the device looked like when it worked.
  4. Defeat paging. Many device CLIs pause long output with a "more" prompt, and a script that does not disable paging hangs there until the timeout kills it. Find the device's "no paging" setting and send it first, or run commands that produce short output.
  5. One session, then leave. Log in, run one command, log out. Do not keep a session open between runs, do not open one per value, and do not let two checks log in at once. Small devices have small session limits and small CPUs.

On the security side, the account the check uses should be able to run show commands and nothing else. Most device operating systems can restrict a user to a read-only privilege level or a command list. Authenticate with a key, keep the key file readable only by the collector's service account, and pin the device's host key rather than accepting whatever is offered.


An API: when the device or service offers one


When the device or service has an HTTP API, the answer usually arrives as JSON. That means no column-counting and no guessing about units. The field is named, its type is a type, and a missing field is a clear error rather than a misaligned row. That makes an API the most durable door for anything without an OID — and for software services, which rarely speak SNMP, the only sensible one. The wider landscape of APIs and webhooks in monitoring is the subject of the Monitoring APIs and Webhooks guides.


# Token by reference (an environment variable the launcher sets), hard time limit,
# TLS verification left ON. Ask for exactly one object.
curl -sS --max-time 5 -H "Authorization: Bearer $API_TOKEN" \
     https://sw01.example.internal/api/interfaces/25
{"id": 25, "name": "Gi1/0/25", "status": "up", "speed": 1000}

Three things need care. Tokens are credentials: hold them in a protected store or an environment variable the launcher populates, never in the script or a URL. Give the check a read-only token. Pagination is how APIs return lists: one page plus a pointer to the next. So a check that reads a list must loop over pages — with a hard cap. That way, a broken "next" pointer can never make it loop forever. Rate limits are the service telling you to slow down, usually with a status of 429. Sometimes a header says how long to wait. Treat a rate-limited response as UNKNOWN for this run and reduce the frequency. Do that rather than retrying in a tight loop and making it worse.


items, url, pages = [], base + "/api/interfaces?limit=100", 0
while url and pages < 50:                       # hard cap: a bad "next" link can never loop forever
    page = get_json(url, headers=auth, timeout=remaining())
    items.extend(page["data"])
    url = page.get("next")                      # cursor to the next page, or nothing
    pages += 1

Two smaller points. Do not disable certificate verification to make an internal API "just work." Add the internal certificate authority to the collector's trust store. Otherwise, you have taught your monitoring to accept anyone claiming to be the switch. And look for a health or status endpoint before building your own; many services already expose exactly the summary a check needs.


Politeness: caching, rate limits, and small devices


A monitoring system asks every device its questions every minute, forever, and the device does not get to opt out. Custom checks add to that load in ways the monitor cannot see or schedule around, so the manners are up to you. Three habits cover most of it.


Cache what will not change. Serial numbers, firmware descriptions, module inventories, the mapping from interface name to index, the size of a disk — these change on a scale of weeks. Asking for them every minute is waste. Write them to a small cache file with a timestamp and reread the device only when the cache is older than an hour or a day. The interface-index lookup is the classic case: resolve the name once, cache the index, and re-resolve only if the cached index stops matching the name.


cache = "/var/cache/checks/sw01-ifindex.json"
if is_fresh(cache, max_age_seconds=6 * 3600):
    index_by_name = json.load(open(cache))
else:
    index_by_name = walk_interface_names(host, timeout=remaining())   # the expensive part
    json.dump(index_by_name, open(cache, "w"))
target_index = index_by_name["uplink-core"]                           # KeyError means UNKNOWN

Respect stated limits, and infer unstated ones. An API that documents a request budget means it; stay well inside it even when several copies of the check run for several targets. Devices rarely document limits, but they have them. Those limits might mean only a handful of concurrent SSH sessions. Or a CPU might serve SNMP at low priority behind forwarding packets. Or a web interface might never have been meant to be polled. When a device gets slow to answer while a check is running, that is the limit.


Never hammer a small device with parallel sessions. Twelve values over SSH from one switch means one session with a few commands, not twelve simultaneous logins. If four checks each walk the same table, walk it once, cache it for the interval, and have the checks read the cache. Stagger start times so every collector does not hit the same device at the top of the minute. The device is smaller than your collector; act like it.


Tip: Watch the device while your new check runs for the first time. Its CPU graph, session count, and log will tell you within a minute whether the check is a polite guest or an unwelcome one. And that is cheaper to learn in a test window than from a helpdesk ticket about a slow switch.


Timeouts everywhere


Every method above has more than one place to hang, and each needs its own deadline. SNMP over UDP has no connection to fail; silence is its only error. So the timeout and retry count you pass are the entire failure mechanism. A generous retry count multiplies your worst case. SSH can stall at the TCP connect, the handshake, authentication, the "more" prompt, and while the command runs. Bound each one and set an overall budget for the session. An HTTP call can stall at name resolution, the connection, the TLS handshake, and the body. One total time limit on the request covers all of them, and a paginated loop needs a budget across all its pages. Whatever the method, the total must be comfortably shorter than the check's interval, and it must report UNKNOWN — not silence, not OK — when it runs out.


Method, when, parsing risk, effort


MethodUse it whenParsing riskEffort
SNMP getThe value has a known OID and you want one rowLow: typed values. Watch for index shifts; anchor on namesLow once the OID is found
SNMP walk / bulkYou need a table: all interfaces, all sensors, all disksLow to medium: indexes, large tables, counters that need two readsLow to medium; cache what is static
SSH command outputThe value exists only in a show command or on-box diagnosticHigh: text for humans, paging, wrapping, format changes without noticeMedium to build, ongoing to maintain
HTTP APIThe device or service documents endpoints; anything without an OIDMedium: schema changes, pagination, tokens, rate limitsMedium; the most durable long-term
Status page or vendor toolNothing above existsVery high: see the small-collectors guideHigh, and it is code you own

Worked example: the same value three ways


The question: is the core uplink on switch sw01 — port Gi1/0/25, labelled "uplink-core" — operationally up? The state mapping is the same in every version: up is OK, anything else the device reports is CRITICAL, a failure to ask is UNKNOWN. Only the door changes.


Over SNMP

# 1. Resolve the name to an index (cached for hours): walk ifDescr, find "Gi1/0/25" = 25
# 2. Ask for exactly that row of ifOperStatus, value only
snmpget -t 2 -r 1 -Oqv sw01 1.3.6.1.2.1.2.2.1.8.25
1

# state: 1 (up) is OK; 2 (down) or 7 (lowerLayerDown) is CRITICAL; timeout is UNKNOWN
# value: 1 ; message: "Gi1/0/25 (uplink-core) operStatus=up"

Cost: one UDP packet each way. Parsing: an integer. Fragility: the index, which the cached name lookup handles. Build this version if the device speaks SNMP at all.


Over SSH

# One session, one command, output saved for the parser and as a future test fixture
ssh -o BatchMode=yes -o ConnectTimeout=5 -i /etc/checks/monitor_key \
    monitor@sw01 "show interfaces status | include Gi1/0/25" > /tmp/sw01-uplink.txt
Gi1/0/25   uplink-core        connected    trunk  a-full  a-1000 10/100/1000BaseTX

# parser (Python): anchor on the port name, take the one field that follows the description
m = re.match(r"^Gi1/0/25\s+\S*\s+(\S+)", line)
status = m.group(1) if m else None            # None means UNKNOWN "unexpected output format"
# state: "connected" is OK; "notconnect" / "disabled" / "err-disabled" is CRITICAL

Cost: a TCP connection, a key exchange, a login, a command, a logout — seconds of device CPU rather than microseconds. Parsing: a regular expression over a line whose layout is a courtesy, not a contract. A port with an empty description shifts the columns, which is why the pattern allows an empty name. Fragility: high; the saved output file is your fixture for the day the format changes. Build this only when the value is not in SNMP or the API.


Over the API

curl -sS --max-time 5 -H "Authorization: Bearer $API_TOKEN" https://sw01/api/interfaces/25
{"id": 25, "name": "Gi1/0/25", "description": "uplink-core", "status": "up", "speed": 1000}

# parser: a named field with a documented set of values
status = body["status"]                       # KeyError means UNKNOWN; "up" is OK; else CRITICAL

Cost: an HTTPS request, heavier than SNMP but light. Parsing: a named field. Fragility: low, unless the schema changes, which is usually documented. The token needs storing and rotating, and TLS needs a trust store; that is the ceremony you pay for a durable answer. If the device has both this API and SNMP, either is fine for a single value; the API pulls ahead when you want details SNMP never had.


Three doors, one number, one state. The choice is rarely about which can answer — all three can. It is almost always about which will still be answering correctly, cheaply, and without your help a year from now.


Where to go next


Prefer the door built for machines. Use SNMP when the value has an OID (get for one row, bulk walk for a table, anchor on names not indexes). Use an API when there is a documented one (tokens by reference, pagination with a cap, rate limits respected). Use the command line only when nothing else has the value. In that case, anchor on stable fields, test with saved samples, defeat paging, and use one session then leave. Cache what does not change, never open parallel sessions to a small device, put a deadline on every step, and report UNKNOWN when time runs out.


The language-specific mechanics of each door are in Writing Checks in Python and Writing Checks in PowerShell. When a device offers none of the three doors, Small Collectors for Devices Without SNMP covers the workarounds and their costs. All of it lives under Writing Custom Checks and Sensors.


Put this into practice with SysPeep Monitor

For the machine-friendly doors, you may not need a custom check at all. SysPeep Monitor collects health and performance data over SNMP, SSH, and similar collectors. It discovers devices and interfaces automatically, and applies thresholds and alert rules with dependencies to keep the noise down. All this runs from one Windows service with lightweight remote probes for other sites. For values only a script can reach, check the product documentation for how it accepts custom results.

Download Free TrialLearn About SysPeep