Version-Controlling Your Checks and Thresholds
"Who changed the CPU threshold on the file servers, and why?" is a question every monitoring admin eventually has to answer, usually while someone is annoyed. In a console-only setup the honest answer is a shrug. The setting is whatever it is now. The audit log (if there is one) shows a username and a timestamp at best. And the reason left the building with whoever made the change. Version control turns that shrug into a one-line command with a name, a date, a diff, and a sentence of explanation.
If you have never used version control, the good news is that you need about six commands and one habit. Version control was built for programmers, but nothing about it cares whether the files contain a program or a list of thresholds. This guide explains the idea in plain language and shows how to get your monitoring configuration into files. It then covers the parts that matter for monitoring specifically. Those include committing on a schedule so that drift becomes visible and writing commit messages that work as a change log. They also include getting a second pair of eyes on a threshold before it pages anyone, and tagging the states you know were good.
In this guide
- Version control in plain language
- Getting the configuration into files
- A repository layout that works
- Committing on a schedule and after changes
- Commit messages as the change log
- Reviewing threshold changes
- Tags for known-good states
- Answering "who changed it and why"
- Secrets never go in the repository
- Where to go next
Version control in plain language
A repository is a folder with memory. It looks like an ordinary folder full of files. But alongside the files it keeps a complete history of every version they have ever been in. You work with the files normally — open them, edit them, save them. When you reach a point worth remembering, you tell the repository to record it.
That recorded point is a commit: a snapshot of every file as it was at that moment. It is stamped with who made it, when, and a short note saying why. Commits pile up in order, so the history reads like a diary of the folder. You can look at any old commit, compare any two, or put the whole folder back the way it was at any of them.
A diff is the comparison between two versions of a file — the lines that were removed, the lines that were added, and nothing else. Diffs are what make review practical: a threshold change is two lines, and a reviewer reads two lines rather than a whole file.
A branch is a scratch copy of the folder inside the same repository. You make your changes on the branch, show someone the diff, and when it is approved you merge the branch back into the main line. For a small team, branches are optional. For changes that page people, they are a cheap way to make "review before apply" a real step rather than a good intention.
Git is the common tool for all of this. It runs on Windows, Linux, and everything else. It works perfectly well on a single machine with no server, and everything in this guide uses it. Many teams also keep a second copy of the repository on another machine or a hosting service so the history survives the loss of the admin's laptop. That is a good idea, but it is a backup of the repository, not a requirement for using one.
Getting the configuration into files
Before you can version anything, the configuration has to exist as text files. Most monitoring tools provide some way to get it out. That might be a configuration export, a backup feature that produces a file, or a directory of configuration files the tool reads on startup. It might also be an API (an interface scripts can call) that returns settings in JSON, a structured text format. Which of these your product offers, and what the files look like, is something to check in the product documentation. This guide stays generic about the mechanism and specific about what to do with the result.
What you want in the files is everything that counts as a decision. That includes the device list, check templates, thresholds, alert rules, and notification routing. It also includes dashboards if the tool exports them, and recurring maintenance windows. What you do not want is data (metric history, alert history) or secrets (community strings, passwords, tokens). Data makes the repository huge and its diffs meaningless. Secrets are covered in their own section below, and the short version is: never.
One practical wrinkle: exports are often not designed for diffing. A single enormous file, keys in random order, a timestamp on the first line, or numeric IDs that change on every export all produce diffs full of noise. A small wrapper script that runs the export and then tidies it is worth an hour of effort. It tidies the export by splitting one file into one-per-device or one-per-role, sorting keys, and stripping the timestamp. That is the difference between a diff that says "disk warn changed from 85 to 80" and a diff that says "everything changed."
The first commit looks like this. The commands are the same on any operating system; only the path differs.
$ cd monitoring-config $ git init Initialized empty Git repository in monitoring-config/.git/ $ git add . $ git commit -m "Initial import of the monitoring configuration export" [main (root-commit) a91f0c2] Initial import of the monitoring configuration export 214 files changed, 6120 insertions(+)
That is it. The folder now has memory. Every file in it is recorded, and from here on the repository will tell you exactly what changed and when.
A repository layout that works
How you arrange the files matters more than it seems, because the layout decides what a diff looks like and who can find what. A layout that has served many teams well separates the things that change for different reasons.
| Path | What lives there | Changed when | Changed by |
|---|---|---|---|
inventory/ | Device list with variables (a CSV spreadsheet-style text file, or one file per site) | A device is added, moved, or retired | Whoever onboards devices; often generated from the inventory system |
roles/ | One template per device role: checks, thresholds, dependencies, routing | A standard changes for every device of a kind | Monitoring admin, after review |
overrides/ | Per-device exceptions with a reason and review date | One device needs to differ | Anyone, with review |
alerting/ | Alert rules, notification groups, escalation, schedules | Who is told, and when, changes | Monitoring admin, after review |
dashboards/ | Dashboard and map definitions, if exportable | Rarely; cosmetic changes | Anyone |
export/ | The raw nightly export from the running system | Every night, by the scheduled job | The scheduler, never a person |
scripts/ | The export wrapper, the generator, validation checks | Occasionally | Monitoring admin |
README | How to run the scripts, where secrets come from, who reviews what | When the process changes | Everyone |
If your tool only offers a raw export and you are not yet generating from templates, the layout is simply export/ plus scripts/ and a README. That is a perfectly respectable place to start. The other folders arrive when you adopt templates, described in Templated Device Onboarding.
Committing on a schedule and after changes
There are two kinds of commit in a monitoring repository, and they do different jobs.
The deliberate commit happens when a person changes something on purpose: a threshold, a template, a routing rule. It is made by the person, with a message that explains the reason. This is the change log.
The scheduled commit happens every night, made by a script that runs the export and commits whatever came out. Its message is boring and automatic ("Nightly export"). Its job is to be a drift detector. If nobody made a deliberate change today, the nightly commit should contain no changes at all. In that case, if it does contain changes, someone edited the running system in the console without going through the files. The next morning's diff is the evidence.
$ git log --oneline -5
c40e2b1 Nightly export
9d7a3f8 Nightly export
3e9c1a0 Lower disk thresholds on file servers after the FS02 full-disk incident (ticket 4471)
5f21e77 Nightly export
a91f0c2 Initial import of the monitoring configuration export
$ git show --stat c40e2b1
commit c40e2b1
Author: monitoring-export <noreply@example.internal>
Nightly export
export/devices/hq-fw-01.yaml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Nobody committed a change to the headquarters firewall, yet the nightly export found one. Whatever it is — a threshold, a disabled check, a changed notification group — it is now visible and attributed to a date. It is up for a decision: keep it, in which case it gets written into the files properly. Or reverse it, by applying the files again. A scheduled export commit that runs every night is the single cheapest control you can add to a monitoring system.
Commit messages as the change log
A commit message is the only documentation most changes will ever get, so it is worth spending twenty seconds on it. The rule of thumb: the first line says what and where, and the rest says why, with a pointer to wherever the discussion happened. Compare:
Bad: "updated thresholds"
Bad: "fix"
Good: "Raise CPU warning on file servers to 90% after the threshold review (ticket 4488)"
Good: "Route UPS alerts at the Brussels site to facilities as well as the site team
Facilities asked to be paged directly for on-battery events after the
outage where they were only told the next morning. Agreed at the ops
meeting; see ticket 4502."
The good messages answer, months later, the questions that will actually be asked: what changed, which devices it affected, and what problem it was solving. Mentioning the scope ("on file servers," "at the Brussels site") is especially valuable. That is because a reviewer or a future reader can immediately check whether the diff matches the intent. A message that says "on file servers" attached to a diff that touches every role is a bug caught for free.
A small team can go further and make the commit message the change record, full stop. Suppose the message references a ticket, the ticket references the incident, and the repository history is the chronological list of every monitoring change ever made. Then you have a change log that nobody had to maintain separately — which is the only kind that stays accurate.
Reviewing threshold changes
Some changes in a monitoring system are low-stakes: a dashboard title, a description field. Others decide whether a human is woken up. Thresholds, alert rules, and notification routing are in the second group, and they deserve a second pair of eyes before they take effect. Choosing the value itself — the reasoning behind a good static threshold — is covered in static thresholds done right. This section is about the workflow that gets the chosen value safely into production.
With a branch, the workflow is short. Make the change on a scratch copy, commit it, and show the diff to a colleague. When they are happy, merge.
$ git switch -c raise-file-server-cpu
Switched to a new branch 'raise-file-server-cpu'
$ (edit roles/file-server.yaml)
$ git commit -am "Raise CPU warning on file servers to 90% after the threshold review (ticket 4488)"
$ git diff main..raise-file-server-cpu
--- a/roles/file-server.yaml
+++ b/roles/file-server.yaml
cpu_percent:
- warn: 85
+ warn: 90
critical: 95
sustained_minutes: 15
$ git switch main
$ git merge raise-file-server-cpu
Updating 3e9c1a0..b8e0d12
Fast-forward
roles/file-server.yaml | 2 +-
What the reviewer is checking is not mainly the number. It is scope and side effects. Does the diff touch only the role the message names? Is the critical threshold still above the warning? Does the sustained period still make sense? Is there an override on some device that now makes no sense? Does this change interact with a dependency or a routing rule? These are questions that take a colleague two minutes to ask and that a tired admin will miss at the end of the day. The wider practice of periodically reviewing alerts as a team is described in the alert review process. Commit-by-commit review is the small, continuous version of it.
If you are a team of one, you still get most of the value by reading your own diff out loud before merging. It sounds silly. It catches things.
Tip: agree in writing which paths need review and which do not. "Anything under roles/ or alerting/ gets a second reader; dashboards/ and inventory/ do not" is a rule people can actually follow. It keeps review from becoming a bottleneck for harmless changes.
Tags for known-good states
A tag is a name attached to a specific commit so you can find it later without remembering a hash. Use tags to mark states you know were good. Examples are the configuration that ran quietly for a month, the state just before a big threshold review, and the state that matched the last audit. Then, when something goes wrong three changes later, you can compare against the tag instead of guessing which commit was the last calm one.
$ git tag -a quiet-baseline-spring -m "Ran for four weeks with normal alert volume; before the threshold review" $ git tag quiet-baseline-spring pre-branch-rollout $ git diff quiet-baseline-spring -- roles/ alerting/
That last command answers "what has changed in the things that page people since the last time everything was fine?" in one line. Tag before any large change — a threshold review, a template rollout, a reorganisation of notification groups — so that the way back has a name.
Answering "who changed it and why"
Back to the question from the introduction. With the configuration in a repository, the history of any file is one command away:
$ git log --oneline -- roles/file-server.yaml b8e0d12 Raise CPU warning on file servers to 90% after the threshold review (ticket 4488) 7b2d44f Revert "Lower disk thresholds on file servers after the FS02 full-disk incident (ticket 4471)" 3e9c1a0 Lower disk thresholds on file servers after the FS02 full-disk incident (ticket 4471) a91f0c2 Initial import of the monitoring configuration export
Add -p to see the actual diff of each commit. Or use git log -S "warn: 90" to find every commit that added or removed that exact text anywhere in the repository. For a per-line view, git blame roles/file-server.yaml prints each line of the file next to the commit and author that last touched it. So the CPU threshold line shows, right beside it, who set it and in which commit. The commit message says why.
$ git blame -L 12,14 roles/file-server.yaml a91f0c2 (Priya ...) cpu_percent: b8e0d12 (Tomasz ...) warn: 90 a91f0c2 (Priya ...) critical: 95
Who: Tomasz. When: the date on commit b8e0d12. Why: the threshold review, ticket 4488. What it was before: 85, visible in the diff. Whether anything else changed at the same time: the diff says no. Every one of those answers is available to anyone with access to the repository, without asking Tomasz, and without Tomasz needing to still work there.
Secrets never go in the repository
A repository is designed to be copied. Clones end up on laptops, in backups, attached to tickets, and on hosting services, and every clone carries the entire history. A community string committed once is in every copy forever, and deleting it in a later commit does not remove it from the earlier ones. So the rule is absolute. It bans SNMP community strings (the shared passwords network devices accept for monitoring queries). It also bans passwords, API tokens, private keys, and notification-service credentials. These bans apply in any file, in any commit, in any branch.
The pattern that replaces them is reference by name. The configuration says credential: snmp-readonly-branch. The value lives in the tool's own credential store, in a password manager with an API, or in an operating-system secret store. Or it lives in a protected file outside the repository that only the apply step can read. When the configuration is applied, the script looks the name up and supplies the value. The files stay shareable and the secret stays in one place, which is also where you rotate it.
Three defensive habits back this up. First, check what the export contains before the first commit. Some tools include credentials in their configuration export, and your wrapper script should strip or mask them. Second, keep a .gitignore file listing any local secrets file so git refuses to stage it by accident. Third, add a small check to the validation script. The check scans for things that look like secrets (a line containing community: with a value, a long random token) and fails the commit. If a secret does slip in, treat it as exposed: rotate it immediately, then clean the history. Rotating is the part that matters; cleaning the history is tidying up.
Where to go next
A repository, a nightly export commit, honest commit messages, and a second reader for anything that pages people: that is the whole practice. It fits on an index card. From here, the files can generate the configuration rather than merely record it, which Templated Device Onboarding covers. The files can also become the source for rebuilding the whole system, which Reproducible Setups covers. Rolling changes out without surprises is the subject of Testing and Rolling Out Monitoring Changes Safely. The pillar page Monitoring as Code puts them in order.
Put this into practice with SysPeep Monitor
Version control is a habit you bring to whatever monitoring tool you run. How you get the configuration out and back in depends on the product. So check the product documentation for its import and backup options. What you will be versioning from SysPeep Monitor is the part that pages people: the thresholds, dependencies, and alert rules it uses to keep noise down. You will also version the devices and interfaces its auto-discovery found. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep