Backing Up the Monitoring System Itself
Everyone backs up the file server. Most people back up the database server, the mail system, and the accounting application. And then there is the monitoring box — the machine that would raise the alarm if any of those backups' hosts died. In a remarkable number of organizations, it has no backup at all. Ask around. It happens because the monitor grows up informally. It starts as an experiment on a spare VM, quietly becomes load-bearing, and never gets promoted onto the "systems we protect" list. Nobody decided it was disposable. Everybody just forgot to decide it wasn't.
Here is what is actually sitting on that unprotected disk: every check you ever tuned, every threshold you adjusted after a false alarm, every alert route and dependency you configured. Plus years of performance history sit there: the baselines, the capacity trends, and the recorded evidence of every outage and every slow decline. It is the operational memory of your network. Lose it, and you do not just lose a tool; you lose everything the tool learned, and everything you encoded into it.
This guide makes the monitor a first-class citizen of your backup regime. It covers what there is to lose and which parts are truly irreplaceable. It also covers a backup design you can set up in an afternoon. It covers the restore drill that turns backups from rumor into fact, the rebuild runbook, and the two partial-loss scenarios worth rehearsing. Plus one important clarification: why high availability, useful as it is, is not a backup.
In this guide
- What there is to lose: the three layers
- Two kinds of irreplaceable
- The backup design: exports, database, off-box
- The export-then-diff habit
- The restore drill — the real product
- The rebuild runbook
- Partial-loss playbooks
- Why HA is not backup
- The monitor-backup checklist
What there is to lose: the three layers
A monitoring system's disk holds three distinct kinds of value, and they fail differently, so name them separately. The first layer is configuration. That includes the list of devices and checks, the thresholds, the alert rules and notification routes, and the dependency tree. It also includes the maps and dashboards and the references to credentials. This is not just settings — it is tuned judgment. Every threshold that no longer cries wolf represents a false alarm you investigated once and encoded forever. Rebuilding configuration from scratch is not an afternoon of typing; it is re-learning months of lessons, at the pace you originally learned them.
The second layer is historical data: the time-series of every metric the system ever collected. This is your baseline library ("what does normal look like?") and your capacity-planning evidence ("the file server's disk grows 4% a month"). It is also your paper trail for disputes with providers and budget conversations with management. That history is the raw material behind the availability reporting that the uptime and availability guides lean on. The third layer is platform state: the installed software itself, its certificates, integration hookups, report schedules, and local customizations. Individually small, collectively the difference between "restored" and "restored, but subtly broken for weeks."
If the inventory feels abstract, run the thought experiment admins only ever run once: the monitor is gone, and you are rebuilding it from memory. Which devices had the non-standard SNMP settings? What was the threshold on the WAN link that finally stopped false-alarming — 80%? 85%? Sustained for how long? Which alerts went to whom, and which were deliberately routed nowhere after that noisy month? You knew all of it once, in the moment you configured it. You do not know it now. The gap between what the system knew and what you remember is exactly what a backup preserves.
Two kinds of irreplaceable
Now the insight that shapes the whole design: these layers are irreplaceable in two different ways. Configuration is irreplaceable by effort — it can, in principle, be rebuilt, at a painful but finite cost in weeks of retuning. History is irreplaceable, period. The past cannot be re-measured. No amount of budget, overtime, or heroics will recreate last spring's traffic baseline or the graph from the week of the big outage. Data not captured then is gone forever. Platform state sits at the cheap end: reinstallable from scratch, annoying at worst.
This tiering lets you make honest decisions about acceptable loss. For configuration, losing a day or two of recent changes is usually tolerable — you remember what you changed this week. For history, decide consciously how much you can shrug off: a day of lost metrics is a small scar; a year is amputation. Write your answers down — "config: lose at most 24 hours of changes; history: lose at most 24 hours of data". Those two sentences are the specification your backup schedule has to meet, and everything below follows from them.
The backup design: exports, database, off-box
The design has three moving parts, none exotic. First, configuration exports. Nearly every monitoring platform can export its configuration somehow — a built-in export function, a settings backup file, config files on disk, or an API. Find yours and schedule it: nightly, plus manually after any significant change. Keep versioned copies — a dated file per export — not one file endlessly overwritten. That is because "the config as of last Tuesday, before the bulk edit went wrong" is precisely the thing you will one day want.
Second, database backups, and here the method depends on your backend class — the same fork described in the database backend guide. Embedded file databases (the SQLite class) cannot be safely copied while the system is writing to them. A live copy risks a torn, corrupt file. Use the database's own backup command, or briefly stop the service and copy during a quiet minute. Server databases (the MySQL/MariaDB and SQL Server class) come with proper dump and backup tooling designed to run hot. So schedule those the way your database documentation recommends. Either way, keep the database from becoming monstrous in the first place with a deliberate retention policy. A smaller database backs up faster, restores faster, and fails less.
Third, the off-box rule: backups must leave the monitor's own disk. A backup stored on the patient dies with the patient — the full-disk failure that kills the monitor happily eats the local backup folder on its way down. Apply the spirit of the classic 3-2-1 rule (three copies, two different media, one somewhere else) at small scale. Keep the live system and a copy pulled nightly to another machine or your regular backup platform. Add whatever off-site protection that platform already provides. The pull matters more than the polish; even a scheduled copy job to a file share on a different host clears the bar that matters most. And none of this requires new software. If you already run a backup platform for the other servers, fold the monitor in. In that case, protect the export folder and the dump location like any other file set, and let the platform handle scheduling, retention, and the off-site leg. The hand-rolled copy job is the fallback, not the ideal; the goal is that the monitor rides the same protected train as everything else.
How many copies to keep is a small decision worth making once. Dated config exports are tiny — keeping a year of them costs almost nothing and preserves the audit trail. Database backups are bulkier, so a common shape is a week of nightlies plus a monthly copy kept much longer. The exact numbers matter less than one property. You must be able to reach back not just to yesterday, but to before the problem started. Corruption discovered on Thursday sometimes began the previous Friday, and a backup rotation only one day deep faithfully preserves the damage.
Frequency, then, falls out of your acceptable-loss statement: config exports nightly and after changes; database backups nightly. Nightly backups mean accepting up to 24 hours of lost history in the worst case. Say that sentence out loud, and if it stings too much, back up the database more often. For most small networks it does not sting; the point is that it be a decision, not an accident.
The export-then-diff habit
Versioned config exports hide a free bonus feature. If every night's export sits next to the previous one, then comparing them tells you exactly what changed in the last day. That is a diff, the line-by-line comparison every developer lives on. Schedule the comparison and mail yourself the result:
nightly job, after the export completes:
compare today's export with yesterday's
if differences found:
email the diff to the admin mailbox
keep dated exports; prune ones older than a year
Most nights the mail says "no changes," and you delete it in a second. But the morning after a misfired bulk edit, that email is a searchlight. Four hundred checks disabled shows up as four hundred deleted lines. You catch the config self-wound before breakfast instead of after the next missed outage. The same trail doubles as an audit log — "who changed the threshold, and when?" becomes a question with an answer. One backup habit, three products: recovery, change detection, and history of intent.
The restore drill — the real product
An untested backup is a rumor of a backup. Files exist, jobs report success, and none of it is knowledge until you have restored once. Backup jobs fail quietly and in style: exports that omit half the settings, database dumps that are truncated, archives nobody can open. The fix is the annual rebuild exercise. Restore the whole system onto a scratch VM (a throwaway virtual machine, deleted afterward) as if the real one had died. Annually is the honest minimum — often enough that the knowledge stays warm and the runbook stays true, rare enough that it actually happens. Book it like a dentist appointment: on the calendar, mildly dreaded, always worth it.
- Step 1. Create the scratch VM, disconnected from production alerting so your test does not page anyone.
- Step 2. Install the monitoring platform from scratch, noting where you found the installer — that detail goes in the runbook.
- Step 3. Restore the newest config export and the newest database backup, in the order your platform expects.
- Step 4. Verify configuration: the check count matches production, dashboards render, alert rules exist.
- Step 5. Verify history: open a few graphs and confirm the past is actually there.
- Step 6. List every credential you had to re-enter by hand — this list is gold; keep it.
- Step 7. Time the whole exercise, then delete the VM.
That timing number is your real recovery time — not the optimistic guess, the measured fact. It goes in the recovery plan, and it quietly answers a design question too. If the rebuild takes four hours, that is four blind hours on the worst day. That tells you how much to care about the spare-hardware and high-availability conversations.
Tip: many platforms deliberately exclude stored passwords and secrets from config exports, for good security reasons. That means a restore prompts you for credentials you may not have written down anywhere else. Discover this during the drill, not the disaster: the drill's step 6 list of "secrets I had to re-enter" belongs in your password vault's emergency kit.
The rebuild runbook
The drill produces one more artifact. And it is the one that turns a catastrophe into an afternoon: the rebuild runbook, the written procedure for resurrecting the monitor. Write it during the drill, while every stumble is fresh — documentation written from memory a month later is fiction with headings. It should contain where the installer lives and which settings the installation needs and the restore steps in exact order. It should also contain the credential re-entry list and where each secret is kept. Include the device re-authentication quirks you hit (SNMPv3 users that needed re-syncing, Windows permissions that needed re-granting). And include the verification list from the drill, so future-you knows what "done" looks like. Add the measured time at the top, like a recipe's cook time. And keep the format ruthless: numbered steps, exact paths and commands, no essays. The reader is future-you at your worst, tired and rattled, executing rather than studying. If a step seems to need a paragraph of explanation, the explanation belongs somewhere like this guide; the runbook gets the imperative sentence.
Then store it where the disaster cannot reach it. A rebuild runbook stored only on the monitoring server is a joke the universe will eventually get around to telling. Keep it with your other outside-the-blast-radius material — the same printed-or-offline break-glass pack discipline covered in the out-of-band access guide. The test is simple: could you open the runbook while every server you own is off? If not, it is decoration.
Partial-loss playbooks
Total death is actually the tidy case — restore everything, follow the runbook. The messier, more common cases are partial, and they are asymmetric in an instructive way. Case one: configuration damaged, history fine. A botched change or a corrupted settings store, but the database of metrics is healthy. Recovery: restore last night's config export over the damaged configuration (or reinstall the platform and import the export). Reconnect it to the existing database and re-enter the secrets from your list. You lose at most a day of config changes and no history. This is the recovery the export-then-diff habit makes almost boring — which is the highest compliment a recovery can earn.
Case two: database lost, configuration fine. The disk corruption or failed migration that eats the metrics store. Recovery: stand up a fresh, empty database. Restore the newest database backup into it if the backup is usable. If it is not, accept the loss, reconnect the config, and let monitoring resume. Checks come back within the hour; the judgment survives; but whatever history the backup did not cover is gone for good. That is the irreplaceable-period tier living up to its name. Start rebuilding baselines immediately, and expect a few weeks of tuning alerts without their usual context of "normal." Rehearse both cases mentally during the annual drill; they take five minutes each to talk through and remove most of the panic from the real event.
Why HA is not backup
A tempting thought arrives about here: "we run a high-availability pair — surely that covers us." It does not, and the reason is that replication is faithful. A high-availability setup copies changes to the standby node quickly and accurately. That includes the corrupt database page, the botched bulk edit, and the accidental deletion. Each is replicated to the standby within moments of happening. HA answers one question: "what if the hardware or host dies mid-day?" — it keeps the service available. Backups answer a different question: "what if the data itself is bad, and we need to go back to before it was bad?" — they make the past recoverable. You may well want both; the busier your monitoring becomes, the more likely, and the high-availability guide covers that side properly. Just never let the existence of a standby node talk you out of the humble nightly export. They are different insurance policies against different disasters.
The monitor-backup checklist
| What | How | Cadence | Verified when |
|---|---|---|---|
| Configuration export | Platform's export function, dated versioned files | Nightly + after significant changes | Diff mailed daily; restored at the annual drill |
| Database backup | Backend-appropriate: dump tooling, or stop-copy-start for embedded files | Nightly | Restored to scratch VM at the annual drill |
| Off-box copy | Pull exports + dumps to another machine or the backup platform | Nightly, automatic | Spot-check the target monthly for fresh files |
| Secrets re-entry list | Password vault emergency kit + printed pack | Update on change | Exercised during the drill's credential step |
| Rebuild runbook | Written during the drill; stored off-system | Revised at each drill | Someone follows it verbatim at the drill |
| Full restore drill | Scratch VM rebuild, timed | Annually | The drill is the verification |
| Backup-age self-check | The monitor alerts if its newest export is stale | Continuous | Part of the self-monitoring starter kit |
That last row closes a pleasing loop. The monitoring system watches the age of its own backups, so a silently failing export job becomes an alert instead of a surprise. That is one more check for the self-monitoring kit from the failure-modes guide.
Where to go next
With exports running, dumps leaving the box, and one timed drill behind you, the monitor finally enjoys the protection it provides everyone else. Its death becomes a bounded, boring event with a runbook. The natural next reads include the day you deliberately take the monitor down, covered in Upgrading the Monitor Without Going Blind (which begins, not coincidentally, with "take a backup"). They also include the rest of the resilience story in Who Watches the Watcher.
Put this into practice with SysPeep Monitor
Backup friendliness starts with the data layer, and SysPeep Monitor keeps that layer standard. Its database runs on SQLite, MySQL/MariaDB, or SQL Server. So the dump-and-restore tooling your backups already use applies directly. And configuration lives centrally on one Windows machine. If you outgrow single-box recovery, high-availability clustering is built in. Download the free trial and see the whole layout.
Download Free TrialLearn About SysPeep