Home › Topics › Writing Custom Checks and Sensors › Checks in Python

Writing Checks in Python


Not everything you need to watch is a Windows box that answers CIM queries. There is the Linux appliance in the rack, the storage controller that only talks SSH, and the switch that only speaks SNMP. There is the web service whose health endpoint returns JSON, and the small collector host in a remote office that has to ask all of them. For that world, Python is the tool most administrators reach for. It runs on anything. Its standard library covers sockets, HTTP, JSON, and argument parsing. For the two protocols it lacks — SNMP and SSH — mature libraries exist.


The catch is the same as with any language. A check can hang or crash with a traceback on standard output. It can print "healthy" because an exception skipped past the line that would have said otherwise. Python will let you do all of it. This guide covers the patterns that prevent it. These patterns include a skeleton that fails closed and a timeout on every network call. They include a clean mapping from what happened to a state, and packaging that survives the next system update.


The rules behind all of it — one question, one target, state plus value plus message, UNKNOWN is not OK — are laid out in The Anatomy of a Custom Check. This guide shows what they look like in Python.


In this guide

  • Why Python for checks and collectors
  • The check skeleton
  • The patterns: sockets, HTTP, SNMP, SSH, parsing
  • Timeouts, errors, and state mapping
  • Packaging and running on the collector
  • A complete small check: TCP port plus banner
  • Patterns, library roles, and gotchas
  • Where to go next

Why Python for checks and collectors


PowerShell wins on Windows because it is already there and speaks the operating system's management interfaces. Python wins everywhere else for a related reason. One script runs unchanged on a Linux collector, a Windows probe host, or your laptop. The standard library alone is enough for a surprising number of checks. A TCP connect, an HTTP request with a timeout, a JSON parse, a regular expression over command output, a log file — none of that needs a third-party package.


For SNMP and SSH you will add a library. This guide describes those by role rather than by name because there are several good ones and the pattern is the same with each. An SNMP library gives you get and walk operations with timeouts and retries. An SSH library gives you a connection, a way to run one command with a time limit, and the output. The price is that Python is not preinstalled the way PowerShell is, and libraries have to be managed. The packaging section deals with that; it is smaller than it sounds.


The check skeleton


Every check in this guide is the same shape. Parse arguments, start at UNKNOWN, and measure inside a try block. Map the value to a state, emit one result, and exit with the matching code.


import argparse, json, sys

STATES = {"OK": 0, "WARNING": 1, "CRITICAL": 2, "UNKNOWN": 3}

def state_for(value, warn, crit, higher_is_worse=True):
    """Map a number to a state. Thresholds are parameters, never constants."""
    if higher_is_worse:
        return "CRITICAL" if value >= crit else "WARNING" if value >= warn else "OK"
    return "CRITICAL" if value <= crit else "WARNING" if value <= warn else "OK"

def emit(state, message, value=None, unit="", as_json=False):
    """The ONLY function that writes to stdout, and the only way out."""
    if as_json:
        print(json.dumps({"state": state, "value": value, "unit": unit, "message": message}))
    else:
        extra = f" | value={value}{unit}" if value is not None else ""
        print(f"{state} - {message}{extra}")
    sys.exit(STATES[state])

def main():
    p = argparse.ArgumentParser(description="one question about one target")
    p.add_argument("--host", required=True)
    p.add_argument("--warn", type=float, default=80)
    p.add_argument("--crit", type=float, default=95)
    p.add_argument("--timeout", type=float, default=5.0)
    p.add_argument("--json", action="store_true")
    a = p.parse_args()

    state, message, value = "UNKNOWN", "check did not complete", None   # fail closed
    try:
        value = measure(a.host, a.timeout)              # one of the patterns below
        state = state_for(value, a.warn, a.crit)
        message = f"{value} (warn {a.warn}, crit {a.crit})"
    except TimeoutError:
        message = f"timed out after {a.timeout}s asking {a.host}"
    except Exception as e:                              # anything unexpected stays UNKNOWN
        message = f"unexpected: {type(e).__name__}"
    emit(state, message, value, as_json=a.json)

if __name__ == "__main__":
    main()

Three details carry most of the weight. argparse gives you typed parameters with defaults and a free --help, so a colleague can run the check by hand without reading the source. The state is UNKNOWN before anything can go wrong, and only a successful measurement replaces it. And the catch-all prints the exception's type rather than its text. That is because error text has a habit of echoing back a URL with a token in it.


The patterns: sockets, HTTP, SNMP, SSH, parsing


Each pattern is a measure function for the skeleton. All take a timeout and all raise rather than guess.


A TCP port with a socket timeout

import socket, time

def measure(host, timeout, port=443):
    t0 = time.monotonic()
    with socket.create_connection((host, port), timeout=timeout):
        return int((time.monotonic() - t0) * 1000)      # connect time in ms

One line does the work, and the timeout argument is the whole point. Without it a filtered port can hold the socket for minutes. The exceptions tell you which state to pick. ConnectionRefusedError and a timeout are the target answering badly (CRITICAL). socket.gaierror means the name did not resolve, so the question was never asked (UNKNOWN). The complete check at the end shows that mapping in full.


HTTP with the standard library

import urllib.request, urllib.error, time

def measure(url, timeout):
    req = urllib.request.Request(url, headers={"User-Agent": "custom-check"})
    t0 = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            status, body = resp.status, resp.read(4096)
    except urllib.error.HTTPError as e:
        status, body = e.code, b""                  # 4xx / 5xx: a real answer, just a bad one
    ms = int((time.monotonic() - t0) * 1000)
    return status, ms, body

The standard library is enough for "what status, how fast, does the body contain the word I expect." A non-success status arrives as an exception, which the fragment turns back into a number so the caller can compare it to the expected one. A 503 is CRITICAL, not UNKNOWN. A URLError whose reason is a timeout is also CRITICAL; one whose reason is a resolution failure is UNKNOWN. If you need sessions, retries, or authentication schemes, a request library (by role: a friendlier HTTP client) makes the code shorter. That comes at the cost of becoming a dependency to manage.


SNMP get via an SNMP library

import os

def measure(host, timeout, oid="1.3.6.1.2.1.1.3.0"):        # sysUpTime
    community = os.environ["SNMP_COMMUNITY"]                 # by reference: the launcher sets it
    result = snmp.get(host, community, oid, timeout=timeout, retries=1)   # library API varies
    return int(result.value) / 100                           # timeticks to seconds

Whatever library you choose, the call has the same properties. It takes an OID, a timeout, and a retry count, and it raises when the device does not answer. Keep retries low; SNMP runs over UDP, so "no answer" is the only failure signal it has, and each retry multiplies your worst-case time. For a table of values (every interface's status, say) use the library's walk or bulk operation rather than a loop of gets. If you would rather not add a library, the standard command-line tools can be run from Python with subprocess.run and its timeout argument. Those tools can read the community string from their own configuration file, which keeps it off the command line.


An SSH command via an SSH library

def measure(host, timeout, user="monitor", key_path="/etc/checks/monitor_key"):
    client = ssh.Client()                                    # an SSH library, by role
    client.load_known_hosts()                                # explicit host-key policy: no silent trust
    try:
        client.connect(host, username=user, key_filename=key_path,
                       timeout=timeout, banner_timeout=timeout, auth_timeout=timeout)
        _, out, _ = client.exec_command("cat /proc/loadavg", timeout=timeout)
        return out.read().decode(errors="replace")
    finally:
        client.close()

Three separate things can hang in an SSH session — the TCP connect, the protocol handshake, and the command itself. A good library lets you bound each one. Use a dedicated, read-only account with a key rather than a password. Keep the key file readable only by the account that runs the check. Decide host-key policy on purpose. Silently accepting any host key defeats the point of SSH. The system's own ssh client via subprocess with -o BatchMode=yes -o ConnectTimeout=5 is a good library-free alternative. Batch mode guarantees it never stops to ask a question.


Parsing what came back

import re

def parse_load(text):
    m = re.match(r"\s*([\d.]+)\s+([\d.]+)\s+([\d.]+)", text)   # anchor on the stable part
    if not m:
        raise ValueError("unexpected output format")            # UNKNOWN, never a guess
    return float(m.group(1))

Command output is written for humans and changes without notice. Anchor your pattern on the fields least likely to move, and take only what you need. Treat a non-match as UNKNOWN rather than defaulting to zero, which would look like "no load" on a graph. Save a real sample of the output as a test fixture. That way, when the format changes you find out from a failing test rather than a quiet week of wrong numbers. JSON endpoints are kinder: json.loads and a key lookup, with KeyError mapped to UNKNOWN.


Timeouts, errors, and state mapping


Python has no global "this script may run for ten seconds" switch, so the budget is enforced call by call. Every function above takes a timeout and passes it to the network layer. Two habits close the gaps. First, set socket.setdefaulttimeout(timeout) near the top of the script as a backstop. It applies to any socket a library opens without setting its own. Second, when a check makes several calls, pass each one what is left of the budget. Otherwise, three calls with a five-second limit can take fifteen.


deadline = time.monotonic() + args.timeout
def remaining():
    return max(0.1, deadline - time.monotonic())

first  = fetch(url_a, timeout=remaining())
second = fetch(url_b, timeout=remaining())      # only what is left

Error mapping is where the CRITICAL-versus-UNKNOWN rule lives, and it deserves explicit except clauses rather than one catch-all. Refused, reset, timed out, or a bad status: the target answered, badly — CRITICAL. Name resolution failed, authentication failed, a library is missing, the output could not be parsed: the question was never properly asked — UNKNOWN. The catch-all at the bottom is for the cases you did not think of, and it leaves the state at UNKNOWN too.


For the state itself, resist writing threshold comparisons inline in every check. The state_for helper in the skeleton takes the value, both thresholds, and a direction. So "free disk below 5 GB" and "CPU above 95 percent" use the same function. Validate the pair before measuring: if the thresholds are in the wrong order, exit UNKNOWN with a message that says so. That is because a check that quietly produces nonsense is worse than one that refuses.


Tip: Never print anything except the result on standard output — not a progress line, not a library warning, not a traceback. Send diagnostics to a log file with the standard logging module. The monitor will parse whatever appears on that stream, and one stray line turns a clean result into a mystery.


Packaging and running on the collector


A Python check is code plus its dependencies plus the interpreter it was tested with. And all three need to stay put on the host that runs it. That is usually a small collector or probe machine near the devices rather than the central monitoring server. The system's own Python belongs to the operating system; updates can change it under you. So give the checks their own:


  1. A virtual environment. python -m venv /opt/checks/venv creates an isolated interpreter and package directory for the checks alone. Nothing installed there affects the system, and nothing the system updates affects it.
  2. Pinned dependencies. List each library the checks need in a requirements file, with the exact release you tested, and install from that file into the venv. The same file installed later produces the same environment, and an upgrade becomes a deliberate act you can test rather than a surprise.
  3. A wrapper script. The monitor, cron, or Task Scheduler calls a tiny wrapper that runs the venv's interpreter. That way, nobody has to remember the path and the check cannot accidentally run under the wrong Python.

# /opt/checks/bin/check_tcp_banner  (Linux wrapper; make it executable)
#!/bin/sh
exec /opt/checks/venv/bin/python /opt/checks/check_tcp_banner.py "$@"

# C:\checks\check_tcp_banner.cmd  (Windows wrapper)
@echo off
"C:\checks\venv\Scripts\python.exe" "C:\checks\check_tcp_banner.py" %*
exit /b %ERRORLEVEL%

The wrapper passes arguments straight through and, crucially, passes the exit code back out, which is what the monitor reads. Run the checks as a dedicated, unprivileged service account. Keep key and credential files readable by that account only. And point logging at a file that account can write, with rotation so a chatty check cannot fill the disk it is watching. Put the whole /opt/checks directory — scripts, requirements file, wrappers, a short README — in version control. It is code you own now.


A complete small check: TCP port plus banner


Here is the skeleton and the socket pattern assembled into one file. It answers one question — is this port open, and does the service greet me the way it should? — with a connect-time value and proper states. Many services (mail, SSH) send a greeting line on connect. Checking that it starts with the expected text catches the case where the port is open but the wrong thing is listening.


"""check_tcp_banner.py - Is ONE port on ONE host open, and does its greeting look right?
Exit: 0 OK, 1 WARNING, 2 CRITICAL, 3 UNKNOWN.  Read-only: connect, read a line, disconnect."""
import argparse, json, logging, socket, sys, time

STATES = {"OK": 0, "WARNING": 1, "CRITICAL": 2, "UNKNOWN": 3}

def parse_args():
    p = argparse.ArgumentParser(description="TCP port and banner check")
    p.add_argument("--host", required=True)
    p.add_argument("--port", type=int, required=True)
    p.add_argument("--expect", default="", help="text the banner must start with")
    p.add_argument("--warn-ms", type=int, default=500)
    p.add_argument("--crit-ms", type=int, default=1500)
    p.add_argument("--timeout", type=float, default=5.0)
    p.add_argument("--json", action="store_true")
    p.add_argument("--log", default="/var/log/checks/tcp_banner.log")
    return p.parse_args()

def main():
    a = parse_args()
    logging.basicConfig(filename=a.log, level=logging.INFO, format="%(asctime)s %(message)s")
    state, message, value = "UNKNOWN", "check did not complete", None      # fail closed
    sock = None
    started = time.monotonic()
    if a.crit_ms <= a.warn_ms:
        message = "thresholds inconsistent: --crit-ms must exceed --warn-ms"
    else:
        try:
            sock = socket.create_connection((a.host, a.port), timeout=a.timeout)
            value = int((time.monotonic() - started) * 1000)
            banner = ""
            if a.expect:
                sock.settimeout(max(0.1, a.timeout - (time.monotonic() - started)))   # spend only what is left
                banner = sock.recv(256).decode("ascii", errors="replace").strip()
            if a.expect and not banner.startswith(a.expect):
                state, message = "CRITICAL", f"port {a.port} open but banner '{banner[:40]}' is not '{a.expect}'"
            elif value > a.crit_ms:
                state, message = "CRITICAL", f"port {a.port} open but connect took {value} ms"
            elif value > a.warn_ms:
                state, message = "WARNING", f"port {a.port} open, slow connect at {value} ms"
            else:
                state, message = "OK", f"port {a.port} open in {value} ms" + (", banner ok" if a.expect else "")
        except ConnectionRefusedError:
            state, message = "CRITICAL", f"port {a.port} refused on {a.host}"
        except socket.timeout:
            state, message = "CRITICAL", f"no response from {a.host}:{a.port} within {a.timeout}s"
        except socket.gaierror:
            message = f"cannot resolve {a.host}"                # UNKNOWN: the question was never asked
        except OSError as e:
            message = f"socket error: {type(e).__name__}"       # UNKNOWN
        finally:
            if sock:
                sock.close()
            logging.info("%s:%s %s %sms", a.host, a.port, state, value)
    if a.json:
        print(json.dumps({"state": state, "value": value, "unit": "ms", "message": message}))
    else:
        print(f"{state} - {message} | connect_ms={value}")
    sys.exit(STATES[state])

if __name__ == "__main__":
    main()

Read it against the contract. The target is mandatory; thresholds, timeout, and output format have defaults. The state is UNKNOWN before the first network call and only measured outcomes can change it. The connect has a timeout; the banner read gets only what remains of the budget. Each exception maps to a deliberate state, and a wrong banner is CRITICAL because the port answered and the answer was wrong. The socket is always closed, a log line always written, and standard output receives exactly one line. Try it against a mail server with --port 25 --expect 220, then against a nonexistent name and a filtered port with a stopwatch. Watch all four states appear.


Patterns, library roles, and gotchas


PatternLibrary (by role)Gotcha
TCP portStandard socket moduleConnect timeout and read timeout are separate; set both. Resolution failure is UNKNOWN, refusal is CRITICAL
HTTP status and timingStandard urllib, or a request library for sessions and authNon-success statuses raise; catch them and read the code. Read a little body, not all of it
SNMP get or walkAn SNMP library, or the command-line tools via subprocessUDP: silence is the only failure signal, so keep retries low. Community string by reference
SSH commandAn SSH library, or the system ssh client in batch modeConnect, handshake, and command each need a timeout. Decide host-key policy explicitly
Parsing outputStandard re, str.split, jsonAnchor on stable fields; a non-match is UNKNOWN, never zero. Keep sample output as a test fixture
Result outputStandard json and sys.exitOne line on stdout, nothing else; diagnostics go to logging
Budgeting timePer-call timeout=, socket.setdefaulttimeout backstopPass each call what is left, not the full budget; subprocess.run needs timeout= too
Packagingvenv, pinned requirements file, wrapper scriptThe system Python is not yours; an unpinned upgrade breaks checks silently

Where to go next


A Python check is argparse for typed inputs and a state that starts at UNKNOWN. It is one measuring function with a timeout on every call, and exception mapping that separates "bad answer" from "no answer." It is one line or one JSON object on standard output, an exit code, and a log file. Package it in its own virtual environment with pinned dependencies behind a wrapper. Run it as an unprivileged account on the collector, and keep the directory in version control.


Choosing between SNMP, SSH, and an API for a given device is the subject of SSH, SNMP, and API Collection Patterns. For devices that offer none of those, Small Collectors for Devices Without SNMP shows how to build the bridge. The Windows-native equivalent of this guide is Writing Checks in PowerShell, and the full set lives under Writing Custom Checks and Sensors.


Put this into practice with SysPeep Monitor

SysPeep Monitor already collects most of what a Python check reaches for: an SNMP value, an SSH command's output, a port that should answer. It uses built-in SNMP, SSH, and similar collectors. Collection runs from a central Windows service and lightweight remote probes for other sites, with thresholds and alert rules on top. Where a script of your own is still the right answer, check the product documentation for how it accepts custom results. The free trial installs in minutes.

Download Free TrialLearn About SysPeep