Home › Topics › The Fundamentals That Outlive Every Tool › Ten Principles

Ten Monitoring Principles That Outlive Every Tool


Every monitoring tool you will ever use is temporary. The one you are learning now will be acquired, repriced, rewritten, or outgrown, and you will learn another, and then another. I have configured five in my working life and I expect to configure a sixth. None of that has changed what I actually know about monitoring, because what I know is not in the tool.


What I know is a short list of principles that were true before the first tool and will be true after the last one. They are not clever. Most of them sound obvious written down. Every one of them is being broken somewhere on a network near you right now, usually by someone who would agree with it if asked. The principles are easy to state and hard to keep, which is exactly why they are worth writing down.


By the end of this guide you will have ten of them. Each comes with the reasoning behind it and the failure it prevents. Each also has one line on how to apply it in whatever console you happen to be sitting in front of. You will also have a card to print and pin above the desk. That way, the next time someone asks "can we just add an alert for that?", you have something to point at.


In this guide

  • Why principles matter more than tools
  • The ten principles
  • The principles card
  • The migration that kept the principles
  • The checklist
  • Where to go next

Why principles matter more than tools


A tool is a way of doing something; a principle is a reason for doing it. Tools change every few years because vendors merge, licences change, and the platform you run on moves under you. Principles do not change, because they come from how networks and people behave rather than from how a particular console is laid out. When you swap the tool, you keep the principle, and the principle tells you what to rebuild first.


The danger of learning monitoring from a tool is that you learn the tool's opinions as if they were laws. Every product has defaults: a default threshold, a default polling interval, a default list of severities. Those defaults are somebody's guess about an average network, and yours is not average. A principle lets you ask whether the default is right for you. Without one, you can only ask whether it is switched on.


This is also why the common monitoring myths survive tool changes so well. A myth is a principle with the reasoning removed, copied from console to console by people who were never told why. The ten below come with their reasons attached, so you can check them rather than just believe them. If you want the picture they all sit inside, the network visibility mental model is where this library starts.


I will admit that ten is a slightly theatrical number. I had twelve and cut two, and I had eight and found I had left out the one that costs the most sleep. Ten is where the list stopped arguing with me.


The ten principles


Each principle has four parts: the statement, the reasoning, the failure it prevents, and one line on applying it in any tool. Read the statement first. If you disagree with it, read the reasoning before you decide.


1. Monitor what users feel, not just what devices report

A device can be healthy while the service it delivers is broken. The switch reports every port up. The server reports low CPU, and the login page has been returning an error for an hour. Devices report their own state. Users experience a service that runs across many devices, and the two are only loosely connected. If your monitoring is a list of devices, you are measuring the parts and guessing at the whole.


The failure it prevents is the green dashboard during an outage, which is the most expensive kind of green there is. To apply it in any tool: for every service people rely on, have at least one check that does what a user does. Examples include fetching the page or resolving the name. Treat that check as the definition of "working". What to monitor first starts from the same place.


2. Every alert must demand an action from a named person

An alert is a message that interrupts someone. If the message has no action attached, the interruption was the whole event, and interruptions with no purpose teach people to ignore the next one. "Disk at 80 per cent" is information. "Disk on the file server will be full in nine days; extend the volume or clear the archive share" is an alert. It goes to the person who can do one of those two things.


The failure it prevents is alert fatigue, the state where everyone has learned that the alert channel is noise and the one real alert drowns in it. To apply it in any tool: before saving an alert, write down the action and the name. If you cannot, downgrade it to a dashboard panel or a report. The anatomy of a good alert shows what the finished message looks like.


3. Know normal before you judge abnormal

A number is not high or low on its own; it is high or low compared with what that thing usually does. Forty per cent CPU is idle on one server and a crisis on another. A baseline is a record of what a metric normally does, by hour of the day and day of the week. It is what turns a number into a judgment. Without one, every threshold you set is a guess dressed as a decision.


The failure it prevents is the threshold that fires every Monday morning because Monday morning is busy. The principle also prevents its twin, the threshold that never fires because it was set during a quiet week. To apply it in any tool: collect for a few weeks before you alert, and look at the graph before you pick the number. Baselining covers how.


If your monitoring database can be queried, "normal" is one query away. Table and column names differ between products, so treat this as the shape of the query and check the product documentation for yours:


SELECT strftime('%H', sample_time) AS hour_of_day,
       AVG(value) AS typical,
       MAX(value) AS worst
FROM samples
WHERE sensor_name = 'fs-01 disk C: used pct'
  AND sample_time > DATETIME('now', '-28 days')
GROUP BY hour_of_day
ORDER BY hour_of_day;

Twenty-four rows come back, one per hour. The threshold you set afterwards is a decision about those rows rather than a number you remembered from a previous job.


4. One fault should raise one alert

When a core switch goes down, everything behind it goes unreachable. In that case, a monitoring system that does not know the layout will tell you so forty times. A dependency is a recorded relationship that says "these devices are reached through that one". It lets the tool report the cause and hold back the symptoms. Forty alerts for one fault is not thoroughness. It is the same fact shouted forty times while the useful one waits its turn.


The failure it prevents is the alert storm, in which the on-call person spends the first twenty minutes of an outage reading rather than fixing. To apply it in any tool: record the dependency between every device and the device it is reached through. Start at the core and work outward. Dependency thinking explains the idea in depth.


5. The monitoring system is production and gets monitored too

Monitoring answers the question "is anything broken?", and a broken monitoring system answers it with a confident "no". A collector that stopped polling looks exactly like a network with nothing wrong, and it will keep looking that way until someone notices by other means. That makes the monitoring server the one machine whose silence you can never trust.


The failure it prevents is the outage nobody was told about, because the thing that would have told them had been down since Tuesday. To apply it in any tool: have something outside the monitoring system confirm that the monitoring system is alive. One example is a heartbeat message that a second, simpler system expects and complains about when it stops. Monitoring the monitor is the pillar for this.


6. Keep the evidence

Arguments about outages are won with graphs, timestamps, and log lines, not with memory. The internet provider says the link was fine; your latency graph says otherwise, with a time on it. The application team says the network was slow; the interface graph says the link was at ten per cent. Evidence that exists only in the tool's live view is gone once the retention window passes, and the argument usually arrives after that.


The failure it prevents is paying for an outage that was somebody else's, and the slower, quieter cost of being disbelieved. To apply it in any tool: when something matters, export the graph and the log lines to a folder with the date in the file name, before you fix anything. (Nobody has ever regretted saving a graph.)


7. Define "down" in writing

"Down" means different things to different people, and until it is written down, each of them is right. To the network team a service is down when the ping fails. To the users it is down when the page does not load. To the manager it is down when the report says so. A written definition specifies this check, failing for this long, from this location. It is what lets two people look at the same event and agree on what happened.


The failure it prevents is the availability number nobody trusts, because every incident becomes a negotiation about whether it counted. To apply it in any tool: for each service, one sentence that names the check, the duration, and the vantage point, kept where the alert rule can quote it. Uptime and availability monitoring goes deeper on the definition.


8. Review on a calendar or it does not happen

Thresholds, alerts, dependencies, and inventory all drift, because the network they describe keeps changing while they sit still. Nothing tells you they have drifted; drift is silent by nature. The only mechanism that catches it is a fixed appointment — a threshold review, an alert review, a coverage review. It is held on a date rather than when someone feels like it, because nobody ever feels like it.


The failure it prevents is the monitoring system that describes the network as it was two years ago. To apply it in any tool: put a quarterly review on the calendar with a name attached, and treat cancelling it as the exception that needs a reason. The threshold review ritual is one worked example of the appointment.


9. Write down why, not just what

A configuration tells the next person what was set; it never tells them why. "Threshold: 85 per cent" is a fact. "85 per cent because the archive job needs ten per cent free to run" is a reason. The reason is what lets the next person change it safely. Without the reason, every setting becomes untouchable, because nobody knows what breaks if it moves.


The failure it prevents is the setting nobody dares change, and its twin, the setting somebody changed without knowing what it protected. To apply it in any tool: every threshold, alert, and exception gets one sentence of "because". That goes in the tool's comment field if it has one and in a document beside it if it does not. Documentation that survives is about making that document last.


10. Prefer boring, understood, and documented over clever

A clever monitoring setup is one that only its author understands, and its author will leave. A boring setup uses the standard collectors, the standard checks, and the plain threshold. It can be handed to a new starter with a document and an afternoon. Clever is fun to build. Boring is what still works at three in the morning when the person who built it is asleep, or gone.


The failure it prevents is the monitoring system nobody can maintain. That becomes the one nobody trusts, which becomes the one replaced by a migration that loses everything. To apply it in any tool: when two designs both work, choose the one a junior colleague could explain back to you. Monitoring anti-patterns is a catalogue of what clever looks like a year later.


Tip: the ten are not equally urgent. If you can keep only three this month, keep 2, 5, and 7. Those are alerts that demand an action, a monitor that is itself monitored, and a written definition of "down". Those three are the ones whose absence is discovered during an outage rather than during a review.


The principles card


The card is the ten principles compressed onto one page. It lists the principle, the one question to ask when you are about to make a change, and the sign that you have drifted away from it. Print it. Pin it where you can see it from the chair. Its job is to be visible at the exact moment somebody says "can we just add an alert for that?"


PrincipleThe question to askThe sign you have drifted
1. Monitor what users feelWould a user agree this service is up?Dashboard green during an outage
2. Every alert demands an action from a named personWho does what when this fires?Alerts nobody opens
3. Know normal before judging abnormalWhat does this metric usually do at this hour?Thresholds that fire every Monday
4. One fault, one alertWhat does this device depend on?Forty alerts for one switch
5. Monitoring is productionWhat tells us if the monitor stops?A quiet week, then a surprise
6. Keep the evidenceWhere is the graph with the time on it?Arguments settled by memory
7. Define "down" in writingWhich check, for how long, from where?Availability numbers nobody trusts
8. Review on a calendarWhen is the next review, and who owns it?Alerts for devices retired last year
9. Write down whyWhat breaks if this changes?Settings nobody dares touch
10. Boring over cleverCould a new starter explain this back?One person who "knows how it works"

Ten rows, one page. If the card ever needs a second page, something has been added to it that is not a principle.


The third column is the useful one in practice. You will rarely sit down and ask whether you are honouring principle four. You will notice, one Tuesday, that a switch reboot produced forty alerts. The card tells you which principle that was and which question to ask next. Drift is easier to spot as a symptom than as a rule broken.


The migration that kept the principles


Meridian Parts replaced its monitoring platform after the vendor was acquired and the renewal arrived with a new logo and a new number. The team lost every dashboard, every custom check name, and several years of graphs, which the export tool had turned into a spreadsheet with one column. What survived was a single document. It contained the list of services and who felt each one, and the written definition of "down" for each. It contained every threshold with its reason, and every alert with its action and the name of the person who owned it. The new tool was polling the same devices within a week and raising the same alerts within two. The first review after the migration found three alerts that had been wrong in the old tool as well, which the team recorded as "a bonus".


Notice what was lost and what was kept. Everything lost lived inside the tool: screens, names, history in a proprietary shape. Everything kept lived in a document that described the network and the decisions about it, and the document did not care which tool read it. Migrating monitoring platforms covers the mechanics of the move. The point here is that the mechanics are survivable only when the principles were written down beforehand.


The graphs were a real loss, and I will not pretend otherwise. Principle six says keep the evidence, and evidence in a proprietary format is only half kept. The lesson Meridian took from the one-column spreadsheet was to export the important graphs on a schedule. That meant exporting them as images and as plain data, into a folder the next tool would never need to read. That is a small chore. It is a much smaller chore than explaining to an auditor why the history begins on the day of the migration.


The checklist


Work through this once now and again at every quarterly review. Each line is a yes or a no. A "no" is not a failure; it is the next piece of work.


  • Every service people rely on has at least one check that does what a user does.
  • Every alert names the action to take and the person or role who takes it.
  • Every threshold was set after looking at a baseline graph of that metric.
  • Every device has a recorded dependency on the device it is reached through.
  • Something outside the monitoring system will notice if the monitoring system stops.
  • There is a folder of dated graphs and logs for every incident that mattered.
  • Every service has a one-sentence written definition of "down".
  • The next threshold, alert, and coverage reviews are on the calendar with an owner.
  • Every threshold and exception has a "because" written beside it.
  • A new starter could explain the monitoring design back to you after a week.
  • The principles card is printed and visible from the chair.

Where to go next


The ten principles are not a methodology and they are not a product feature list. They are the things that stay true when the console changes, the vendor changes, or you change jobs. They are the things you will be judged on when the outage review asks why the dashboard was green. Keep them in a document, print the card, and read the third column when something feels wrong. The tool will change. The reasons will not.


If the principles are what you keep, Surviving vendor churn is about where to keep them so that the next tool can read them. When everything changes turns the same ideas into a checklist for a new tool, a new job, or a merger. For any term in this guide that was new, the monitoring glossary defines it in plain words. All of these belong to The Fundamentals That Outlive Every Tool.


Put this into practice with SysPeep Monitor

Principles survive a tool change, but they still need a tool to run on. SysPeep Monitor polls devices over SNMP, WMI, and SSH from a central Windows service. It records dependencies so that one fault raises one alert instead of forty. It keeps the graphs, maps, and dashboards in one web console. The free trial installs on a Windows machine if you want to try the ten principles against your own network.

Download Free TrialLearn About SysPeep