The Nagios Family: Check-Based Monitoring and Its Descendants
The first monitoring system I ran had one configuration file. When I handed it over it had ninety, and one existed only to list the other eighty-nine. That system was Nagios, ancestor of an entire family of monitoring tools. And its central idea is so simple and durable that you will find it hiding inside systems that never mention its name.
That idea is the check: a small program asks one question about one thing, and answers with a state and a line of text. Everything else in the family — hosts, services, notifications, escalations, dependencies — is scaffolding built around that one move. Understand the check and the rest becomes readable.
This tour walks through the check-and-plugin model and explains hosts, services, and states. It looks at the text-configuration heritage and how descendants such as Icinga modernized it. It ends with an honest account of strengths, learning curve, and fit. By the end you should be able to open any project in this family and know what you are looking at.
In this guide
- The check and the plugin: a small program answers one question
- Hosts and services: the two nouns
- Soft states, hard states, and why the first failure stays quiet
- The configuration heritage: text files, objects, and templates
- Dependencies and parents: telling the system how the network is shaped
- How the descendants modernized the model
- Strengths, learning curve, and where it fits
- The concepts in one table
- What to check in the current docs before committing
- Where to go next
The check and the plugin: a small program answers one question
Picture a monitoring system that knows nothing about networks, disks, or web servers. It cannot ping or read free space. All it can do is run a program at a scheduled time, wait for it to finish, and look at two things. The first is the program's exit code (the small number every program hands back to the operating system when it ends). The second is the first line the program printed. That is the core of the check-based family, and the programs it runs are called plugins.
The contract between the scheduler and a plugin is tiny, which is the whole point. A plugin must exit with a code that means one of four states, and it should print one human-readable line explaining itself:
| Exit code | State | What it means |
|---|---|---|
| 0 | OK | The thing being checked is fine |
| 1 | WARNING | Not broken, but worth a look |
| 2 | CRITICAL | Broken, or about to be |
| 3 | UNKNOWN | The plugin could not decide — wrong arguments, no answer, a bug |
Running a plugin by hand looks like this. This one checks free space on a filesystem, warning at twenty percent free and going critical at ten:
$ ./check_disk -w 20% -c 10% -p /var DISK OK - free space: /var 41200 MB (68%);| /var=19000MB;48000;54000;0;60000 $ echo $? 0
Read the output in two halves. Before the vertical bar is the line a human sees in the status board: the state word, then a short explanation. After the bar is optional performance data: the raw numbers, in a compact format, so that a graphing component can draw the trend over time. The echo $? prints the exit code, and 0 means OK. That is the entire interface: any program in any language that follows this contract is a valid plugin. (Yes, that language too.)
This is why the family's plugin ecosystem is enormous. Writing a plugin is a lunch-break job, so somebody has usually written one for almost every device, service, and protocol. You can write your own just as easily. A plugin that checks whether last night's backup is fresh is a few lines of shell:
#!/bin/sh # check_backup_age: CRITICAL if the newest backup is older than a day FILE=$(ls -t /backups/*.tar.gz 2>/dev/null | head -n 1) if [ -z "$FILE" ]; then echo "BACKUP UNKNOWN - no backup files found"; exit 3 fi AGE_HOURS=$(( ( $(date +%s) - $(stat -c %Y "$FILE") ) / 3600 )) if [ "$AGE_HOURS" -gt 24 ]; then echo "BACKUP CRITICAL - newest backup is $AGE_HOURS hours old"; exit 2 fi echo "BACKUP OK - newest backup is $AGE_HOURS hours old"; exit 0
Drop that in the plugins directory, schedule it hourly, and you have monitoring for something nobody ever wrote an integration for. That is the family's superpower: any question you can script becomes a check.
Hosts and services: the two nouns
The family organizes checks under two nouns. A host is a thing with an address — a server, a switch, a printer, a firewall. A service is a check attached to a host: "HTTP on web-01", "disk space on file-02", "port 24 status on sw-core-01". A host can have many services, each running its own plugin on its own schedule.
Hosts have their own check, usually a ping, and their own states: UP, DOWN, and UNREACHABLE. That third state gets its own section below. Services use the four plugin states: OK, WARNING, CRITICAL, UNKNOWN. The status board is simply every host and service with its current state and last output line, colored green, yellow, and red. That is the clipboard from the family overview.
The family thinks in states, not numbers. A CPU at eighty-five percent is not a number to this family; it is a service that is currently WARNING. The number exists in the performance data and graphing components can draw it, but the system's own reasoning is about state. That feels odd for a week and then natural, because "what is broken right now?" is the question an operator asks most.
Soft states, hard states, and why the first failure stays quiet
A naive monitoring system pages you the instant a check fails. A network hiccup, a slow response, a plugin timing out during a backup — every one becomes an alert. Within a month everybody has muted them. The check-based family solved this early with soft and hard states.
When a check that was OK comes back CRITICAL, the system does not believe it yet. It marks the service as being in a soft CRITICAL state and re-runs the check sooner than usual, on a shorter retry interval. Only when the check has failed a configured number of times in a row — the maximum check attempts — does the state become hard. Notifications are sent on hard state changes, not soft ones. So a single blip produces a soft failure that quietly recovers, and a real outage produces a hard failure that wakes someone up.
The sequence, in plain steps:
- 1. Normal polling. The service is hard OK and is checked on its regular interval.
- 2. First failure. The plugin returns CRITICAL. The service becomes soft CRITICAL, attempt one. No notification.
- 3. Retries. The check is rerun on the retry interval. If it returns OK, the service goes back to hard OK and nobody was bothered. If it keeps failing, the attempt count climbs.
- 4. Hard state. On reaching the maximum attempts, the service becomes hard CRITICAL. This is a state change, so notifications go out to the configured contacts.
- 5. Recovery. When the plugin returns OK again, the service returns to hard OK — another state change, so a recovery notification is sent.
Around this core sit a few more ideas the family shares. Acknowledgement lets an operator say "I know, I am on it," silencing repeat notifications until the state changes again. Scheduled downtime tells the system that a host will be off for maintenance, so nothing fires while you patch it. Flap detection notices a service bouncing between states and suppresses the resulting alert storm. And escalations send a notification to a wider group if a problem stays unacknowledged for a while. These concepts reappear under various names in the alert design guides, because the family got them right early and everyone else borrowed them.
The configuration heritage: text files, objects, and templates
The family's other defining trait is how it is configured: in plain text files describing objects. A host is an object. So is a service, a command (how to run a plugin), or a contact (who to tell). A time period (when checks and notifications are allowed) is an object too, as are groups of all of these. The scheduler reads the files at startup, builds its picture of the world, and starts running checks.
Object definitions support inheritance: an object can say "use this template," pick up all of the template's settings, and override only what differs. In practice you define a handful of templates — a generic host, a generic service, and a "critical production" service with tighter retries and louder notifications. Every real object is a short list of what makes it special. A classic-style definition looks like this:
define service {
use generic-service ; inherit intervals, contacts, notification options
host_name web-01
service_description HTTP
check_command check_http!-H www.example.internal
max_check_attempts 3
check_interval 5
retry_interval 1
}
Every line is one decision. The lines say which template to inherit from, which host this belongs to, and what to call it. They say which command runs and with what arguments (the exclamation mark separates them). They also say how many failures before a hard state, and how often to check normally and while retrying. Nothing is hidden, which is both the beauty and the burden.
The beauty: the configuration is text, so it lives in source control, can be generated by scripts, reviewed, and diffed when something changes. The burden: somebody has to write it. A hundred hosts with ten services each is a thousand service definitions. While templates and groups collapse that considerably, the family assumes you will describe your environment explicitly. This is what "configuration discipline" means: naming conventions, template hierarchies, and a habit of tidiness. That discipline pays off for years or, if neglected, produces the novel-sized files people sigh about.
Tip: if you inherit an existing installation, resist the urge to rewrite it. Instead, read the templates first, then the groups, then a few real objects. Most old configurations are more orderly than they look; the disorder is usually in recent additions, not the foundation.
Dependencies and parents: telling the system how the network is shaped
Remember the UNREACHABLE host state? It exists because the family lets you describe network topology. Each host can name its parents: the devices between it and the monitoring server. A branch server's parent is the branch router; the router's parent is the head-office firewall. When the branch router goes DOWN, the scheduler does not report the twenty servers behind it as DOWN too. Instead, it marks them UNREACHABLE, notifies about the router, and stays quiet about the rest. One alert instead of twenty-one.
Service dependencies do the same at a finer grain: "do not notify about the web application if the database service on the same host is already CRITICAL." Modeling these relationships is real work, and it is the second half of the learning curve. It is also where the family earns its reputation for well-behaved alerting, and the ideas transfer directly to the alert correlation guides.
How the descendants modernized the model
The original design has one process on one server reading text files and running plugins. That is elegant, but it strains in three places. Hand-edited files do not suit every team. One server cannot comfortably poll a very large or very distributed estate. And modern tooling expects an API (application programming interface: a way for programs, rather than people, to read and change the system). The descendants — Icinga among others — kept the check-and-plugin model intact and rebuilt the scaffolding around it.
The modernizations follow common themes, recognizable across projects even though each implements them differently:
- An API. Hosts, services, downtimes, and acknowledgements can be created and queried programmatically. So provisioning tools can add monitoring when they add a server. Other systems can read state without scraping a web page.
- Web-based configuration. Some descendants offer a graphical way to define objects, so that colleagues who do not live in a text editor can still add a host.
- Distributed and clustered designs. Instead of one scheduler, several nodes share the work. Satellite nodes at remote sites run the checks locally and report back. Pairs of nodes can take over for each other. The general idea of remote collectors is covered in the probes and collectors guide.
- A newer configuration language. Some descendants replaced the classic object format with a language that supports rules such as "every host in this group gets these services automatically." That cuts the repetition that made old configurations grow.
- Built-in paths for performance data. Descendants tend to ship ways to write performance data into a time-series store and use a dashboard tool on top.
What did not change matters more. Plugins still return four states and a line of text. Hosts still have services. Soft and hard states still govern notifications. A plugin written for the ancestor long ago still works in its descendants, which is why the plugin ecosystem is shared across the family. How far any descendant has gone in each direction is a moving target; check its current documentation rather than trusting a summary, including this one.
At Ashgrove Housing the move from classic object files to a newer configuration language was booked for a weekend. It took three months, because nobody had written down what the four hundred checks were for. The checks were the documentation. Sixty watched hosts retired long ago, CRITICAL for so long that the red band on the status board had become furniture. The new configuration has a comment on every service, headed "why".
Strengths, learning curve, and where it fits
Strengths
Three things stand out. First, reach: any check you can script, in any language, is a first-class citizen. So this family can monitor the strange, the old, and the home-grown alongside the ordinary. Second, the plugin ecosystem: for common devices and services, someone has usually already shared the check. Third, predictable behavior: the state machine is simple and stable. So "why did this alert fire?" is traceable through intervals, attempts, and notification options rather than opaque magic.
Learning curve
The effort goes into configuration discipline and dependency modeling. You will spend your first weeks learning the object types, deciding on templates and naming conventions, and typing or generating definitions. You will then spend time describing parents and dependencies so that alerts are quiet and accurate. Neither is intellectually hard, but both reward care and punish shortcuts, and the family will not do them for you. Graphing is a separate concern: expect to attach a time-series and dashboard component rather than find one built in.
Where it fits
The family is at home in heterogeneous environments. That means a mix of operating systems, network gear, appliances, and applications where no single integration covers everything. It is also at home with teams who script. If your colleagues reach for a shell when they meet a problem, they will be productive quickly. It fits a small network well too: ping the hosts, check the services, query the switches with SNMP plugins, and you have solid coverage. Where it is a weaker fit is for teams who want rich graphs and auto-discovered ports on day one without writing anything. The network-first tools and the all-in-one platform assume less scripting.
The concepts in one table
| Concept | Plain-language meaning | Why it matters |
|---|---|---|
| Plugin | A small program that answers one question with an exit code and a line of text | The unit of everything; write your own for anything unusual |
| Check | One scheduled run of a plugin against one target | Its result sets the state of a host or service |
| Host | A thing with an address; states UP, DOWN, UNREACHABLE | The anchor that services hang from |
| Service | A check attached to a host; states OK, WARNING, CRITICAL, UNKNOWN | The row you see on the status board |
| Soft and hard state | Unconfirmed versus confirmed after repeated failures | Keeps single blips from paging anyone |
| Max check attempts and retry interval | How many consecutive failures, how quickly rechecked | Your main tuning knobs for noise versus speed |
| Notification and contact | A message about a hard state change, sent to a person or group | How a state becomes a human's problem |
| Acknowledgement and downtime | "I know" and "expected to be off" | Silences repeats without hiding real changes |
| Escalation | Tell more people if a problem stays unresolved | Matches the ideas in the on-call guides |
| Parents and dependencies | Which devices sit between the monitor and a host; which services rely on others | Turns twenty alerts into one |
| Template | An object others inherit settings from | Keeps configuration short and consistent |
| Performance data | Numbers after the pipe in plugin output | Feeds graphs; not used for state decisions |
What to check in the current docs before committing
The descendants differ and change over time, so do not choose from memory or a comparison article. Open the current documentation of the project you are considering and answer these yourself:
- How is configuration written today? Classic object files, a newer language, a web interface, an API, or several of these together? Pick the one your team will actually maintain.
- How does it scale out? Read the distributed or clustered setup chapter. Check what runs at a remote site, what happens when the link drops, and what high availability means in this project's vocabulary.
- Where do graphs come from? Find the recommended path from performance data to a dashboard, and how many separate components it involves.
- How are plugins run on remote hosts? Checking a disk means running the plugin on that server. Learn the project's recommended agent or remote-execution method and its security model.
- How are SNMP devices handled? The family monitors network gear through SNMP plugins; look at what is included and whether interfaces must be defined by hand or can be discovered.
- What does upgrading look like? Read recent upgrade notes to see how much changes between releases and how compatible old configurations remain.
- How active is the project? Look at the release notes and community channels and form your own view rather than taking anyone's word for it, including ours.
Where to go next
One durable idea: a small program answers one question with a state and a line of text. And a scheduler turns thousands of those answers into a quiet, accurate picture of what is broken right now. Everything else is scaffolding, and the scaffolding is yours to keep tidy; mine grew to ninety files, and yours does not have to.
If the state-centric mental model resonates, this family rewards learning. If you found yourself wishing for numbers, discovery, and graphs by default, continue the Guided Tour of the Open-Source Monitoring Stacks with the all-in-one platform, the network-first tools, and the metrics-first stack. Each answers that wish in its own way. Whichever you pick, the threshold tuning guides will help you decide what WARNING and CRITICAL should mean.
Put this into practice with SysPeep Monitor
Every good monitoring system borrows the soft-and-hard-state idea, whichever family it belongs to. It also borrows dependencies that turn twenty alerts into one, and acknowledgements and downtime. SysPeep Monitor uses the same ideas: thresholds, dependencies, and alert rules built to keep noise down. It is a Windows service that auto-discovers your network and collects over SNMP, WMI, and SSH. If you would rather evaluate a finished product alongside a check-based stack, the free trial is one more option to put on the shortlist.
Download Free TrialLearn About SysPeep