Reproducible Setups: Rebuilding Your Monitoring from Files
Here is an uncomfortable question. Suppose the monitoring server died this morning — disk gone, no warning. Could you stand up a new one from documentation and files by the end of the afternoon? Not "restore the virtual machine (VM) snapshot," which is a fine answer until the snapshot is on the same failed storage; not "reinstall and rebuild from memory," which is a week of work and a lot of forgotten thresholds. A new machine, the files in your repository, a runbook, and a few hours.
For most teams the honest answer is no, and the reasons are always the same. A handful of settings only exist in the console, and a secret only exists in the old database. A glue script only exists on someone's desktop. And the install procedure was done once by a person who has since moved on. None of those are hard to fix. They just have to be found, and the only reliable way to find them is to try the rebuild before you need it.
This guide breaks a monitoring setup into its layers and names the source of truth for each one. It shows how the layers come together into a rebuild you can rehearse. Along the way it covers the two practical payoffs that make the effort worthwhile even if the server never dies. Those are disaster recovery that actually works, and the ability to upgrade by building a fresh system rather than operating on a live one.
In this guide
- The rebuild test
- The six layers of a monitoring setup
- Layer, source of truth, rebuild step: the table
- Why reproducibility is disaster recovery
- Upgrade by rebuild
- Writing the rebuild runbook
- Rehearsing on a lab VM
- The rebuild checklist
- Where to go next
The rebuild test
The test is simple to state. It starts with a fresh machine, the repository, access to the secret store, and the runbook. With those, a competent colleague who did not build the original system should be able to produce a working monitoring server in an afternoon. "Working" means the devices are being polled, a deliberately triggered alert reaches the right inbox, remote probes are reporting in, and the main dashboard renders. It does not necessarily mean the metric history is back; that is a separate decision covered below.
The test is deliberately not "can the original admin do it," because the original admin carries the missing pieces in their head. The whole point is to get those pieces out of their head and into files. When a colleague hits a step that says "configure the notification settings" with no further detail, the runbook has found a gap. When they need a password that is not in the store, the secret handling has found a gap. Every gap the test finds is one you would otherwise have found during an outage.
If you already keep configuration in a repository, most of the work is done and this guide is about the remaining layers. If you do not, start with Version-Controlling Your Checks and Thresholds; a rebuild from files needs files.
The six layers of a monitoring setup
A monitoring system feels like one thing, but a rebuild reveals six separate layers. Each has its own source of truth and its own way of being restored. Treat them separately and the rebuild becomes a sequence of small, checkable steps instead of one large, mysterious one.
1. The server
This layer covers the operating system, its prerequisites, the service account the monitoring software runs as, and local firewall rules. It also covers the installer for the monitoring software itself and the settings the installer asks for. The source of truth is an install script where the product allows unattended installation. Where it does not, the source is a numbered list of documented steps. Keep a copy of the installer alongside the runbook; the download page you used originally may have moved on. Record the settings you chose at install time — ports, paths, the account — because "whatever the default was" is not a reproducible answer.
2. The database
The monitoring tool stores its configuration and its collected data in a database. Which backend you chose is a decision recorded in the runbook. The options are an embedded file database for a small install, or a separate database server for a larger or clustered one. The tradeoffs are covered in the guide on database backend choices. For a rebuild you have two options. Restore the last database backup, which brings back both configuration and history. Or create an empty database and import the configuration from the repository, which brings back a clean configuration and no history. The backup side of this — what to back up, how often, and how to verify it — is covered in backing up the monitoring system. The rebuild side is knowing which of the two options you will take and having rehearsed it.
3. The configuration import
Device list, roles, thresholds, dependencies, alert rules, routing, dashboards: everything that lives in the repository. The rebuild step is to run the generator, if you have one, and load the result into the tool through whatever import path the product offers. That might be a configuration directory, a CSV (spreadsheet-style text) import, or a bulk-add feature. It might also be an API (an interface a script can call) or a file uploaded through the console. Order matters more than people expect. Credentials must be registered by name before devices that reference them. Groups must come before the devices that belong to them. Devices must come before the dependencies that link them. Alert rules must come after the devices they apply to. Write the order down. Check the product documentation for its import options and for which objects must exist before which.
4. Collectors and probes
Anything that gathers data on the central server's behalf: remote probes at other sites, agents on servers, collectors on isolated network segments. Each has its own small configuration — the address of the central server, a registration key or certificate, the local port — and its own install. The source of truth is the inventory (which probes exist and where) plus an install script or documented steps for a probe. During a rebuild, probes usually need to be pointed at the new server or re-registered with it. The runbook should say how, and the smoke test should confirm that every probe in the inventory is reporting.
5. Integrations
These pieces connect monitoring to the rest of the world. They include the mail relay it sends through, the webhook or paging endpoints it posts to, and the ticketing hook. They also include the scheduled export job, the glue scripts that reformat something for something else, and the certificate on the web console. These are the layer most often lost, because each one was set up once, in a hurry, in the console or on a desktop. Every integration setting belongs in the alerting/ or scripts/ part of the repository, with its secret referenced by name. The guides under Monitoring APIs and Webhooks cover how these connections work. For the rebuild, what matters is that each one is listed and restorable.
6. Secrets
Secrets include SNMP community strings (the shared passwords network devices accept for monitoring queries). They include the read-only account used for WMI and SSH collection (the Windows management interface and the remote command channel used for other systems). They also include the mail relay password, notification tokens, the database password, and probe registration keys. None of these are in any file in the repository. They live in a secret store. That might be a password manager with a command-line interface or an operating-system credential store. Or it might be a protected file outside the repository readable only by the account that runs the build. The repository contains the list of secret names the build needs. The build script fetches each by name and injects it at the moment it is required. It injects the secret into the tool's credential store, into the probe installer, or into the notification settings. Because the values are never written to disk in the repository, a rebuild on a new machine simply points at the same store and gets the same secrets.
Layer, source of truth, rebuild step: the table
| Layer | Source of truth | Rebuild step | Proof it worked |
|---|---|---|---|
| Server | Install script or numbered steps in the runbook; a saved copy of the installer | Provision the machine, run the install, apply recorded settings | Service running; console reachable |
| Database | Backend choice in the runbook; the latest verified backup | Restore the backup, or create an empty database | Tool starts cleanly against it |
| Configuration | The repository: inventory, roles, overrides, alerting, dashboards | Run the generator; import in the documented order | Device count matches the inventory |
| Collectors and probes | Inventory of probes; probe install script | Install or re-point each probe; register it | Every probe in the inventory reports in |
| Integrations | alerting/ and scripts/ in the repository | Import notification settings; reinstall scheduled jobs and glue scripts | Test alert arrives; nightly export runs |
| Secrets | The secret store; the list of names in the runbook | Fetch by name and inject during the steps above | Collection succeeds; nothing was typed by hand |
The last column is the one to take seriously. Each layer has a cheap, observable check, and the rebuild is not finished until every check passes. "The console loads" proves the server layer and nothing else.
Why reproducibility is disaster recovery
A backup answers "what did the system look like?" A reproducible setup answers "how do I make one?" You want both, because they fail differently. A backup can be corrupt, incompatible with a newer version of the tool, on the storage that just failed, or simply out of date. A rebuild from files sidesteps all of those. The files are text, and they are in a repository with its own copies elsewhere. They do not care what version of the tool reads them (beyond the import format). And they are as current as the last commit.
The realistic recovery plan combines the two. Rebuild the server and configuration from files, then restore the most recent good database backup for history if it is available and healthy. If it is not, accept the loss of history and carry on. Being able to say "we lose the graphs but the alerting is back in three hours" is an enormously better position than "we have a backup somewhere and nobody knows how to install the software."
There is a subtler benefit. A monitoring system that can be rebuilt is a monitoring system whose configuration is fully known. The rebuild test is the only audit that cannot be fooled. It does not check that the documentation looks complete. It checks that the documentation is complete, because anything missing produces a system that does not work.
Upgrade by rebuild
Once a rebuild is routine, a new option opens up for upgrades. Instead of upgrading the software on the live server, you build a second server on the new release from the same repository. That avoids hoping the installer, the database migration, and every integration survive. You point a few probes at the new server. You compare its behaviour with the old one for a few days, and then switch over. If the new one misbehaves, the old one is still running. Nothing was risked. The broader practice of upgrading without losing visibility, including how to keep the old system available during the switch, is covered in upgrading monitoring safely.
The same move handles hardware replacement, moving the server to a different virtual host, and changing the database backend. Each becomes "build a new one, compare, switch" rather than "modify the only one we have." And the build you make to test an upgrade is, not coincidentally, a staging instance. That is exactly what Testing and Rolling Out Monitoring Changes Safely asks for.
Writing the rebuild runbook
The runbook is a document, kept in the repository, that a colleague follows top to bottom. It is written as commands and checks, not prose. The phrase "configure the notification settings" is a heading, and under it is the exact command or the exact fields and values. Where a step is a script, the runbook names the script and what it expects. Where a step is manual because the product offers no other way, the runbook says exactly which console page and which values. In that case, it marks the step as a known gap to automate later.
A skeleton that works, with each step pointing at a script in the repository:
# RUNBOOK: rebuild the monitoring server from scratch
# Expected time: one afternoon. Needs: repository access, secret store access,
# a fresh Windows machine meeting the prerequisites in docs/prerequisites.md.
1. Install prerequisites and the monitoring service scripts/install-server.ps1
check: service is running; console answers on the documented port
2. Create or restore the database scripts/create-db.ps1 | scripts/restore-db.ps1
check: tool starts cleanly; note which option you took
3. Load secrets by name from the store scripts/load-secrets.ps1
check: every name in docs/secret-names.txt resolved; nothing typed by hand
4. Register credentials in the tool, by name scripts/register-credentials.ps1
5. Generate device definitions scripts/generate.py
check: build/ contains one file per inventory row
6. Import groups, devices, dependencies, alert rules scripts/import.ps1
check: device count in console equals inventory row count
7. Install or re-point remote probes, one per site scripts/install-probe.ps1 -Site <code>
check: every probe in inventory/probes.csv reports in
8. Import notification settings and schedules scripts/import-alerting.ps1
check: send a test alert; confirm it lands in the right inbox
9. Import dashboards and maps scripts/import-dashboards.ps1
10. Reinstall scheduled jobs (nightly export, generator) scripts/install-jobs.ps1
11. Run the smoke test scripts/smoke-test.ps1
check: all green, then record the rebuild date and who ran it in docs/rebuilds.md
The script names are illustrative, and the mix of scripting languages does not matter. A Windows monitoring server will lean on PowerShell. The generator may well be Python, and some steps may be a page of documented clicks for now. What matters is that every step has a check, and that nothing in the runbook depends on the reader already knowing how the system was built.
Keep one copy of the runbook and the secret-name list somewhere that does not depend on the monitoring server, the repository host, or the network being up. That could be a printed copy in the server room, or an offline copy on an encrypted drive. The rebuild you most need will happen on a bad day, and bad days have a way of taking down the wiki along with everything else.
Tip: the rebuild runbook is also the best onboarding document you will ever write. A new admin who follows it once on a lab VM understands the whole system better than a week of shadowing would teach them. That covers every layer, every integration, every secret.
Rehearsing on a lab VM
A runbook that has never been run is a guess. The rehearsal is cheap: a small virtual machine on a lab network, the repository, and a colleague with the afternoon free. Time it. Note every place the colleague stops to ask a question, because each one is a missing line in the runbook. Note every value they had to look up in the old console, because each one is configuration that never made it into the files. Fix the gaps, commit the fixes, and the next rehearsal is shorter.
Do the first rehearsal against an empty database and a subset of the inventory — one site's devices. That way the lab system is not polling the whole network alongside the real one. Use read-only credentials. If the lab server can reach the notification channels, point them at a test inbox so a rehearsal does not page anyone. Once the rehearsal works, repeat it after any change to the install procedure, after a version upgrade, and on a calendar reminder a few times a year. A rehearsal that takes an hour and finds nothing is not wasted; it is the evidence that the plan still works.
The lab VM you build this way does not have to be thrown away. Kept up to date from the same repository, it becomes the staging instance where changes are tried before production. Kept up to date that way, it also becomes the machine on which the next upgrade is tested. One small VM, three jobs.
The rebuild checklist
Use this to judge where you stand today. Every "no" is a task, and none of them is large.
- Installer and prerequisites: a saved copy of the installer and a written list of prerequisites, ports, paths, and the service account.
- Database decision: the backend is named in the runbook, and both "restore the backup" and "start empty" have been tried.
- Configuration in the repository: inventory, roles, overrides, alert rules, routing, and dashboards, with a nightly export commit catching anything changed in the console.
- Import order documented: credentials, groups, devices, dependencies, alert rules, dashboards — in the order the product requires.
- Probe inventory and install script: every remote probe and collector is listed, and a new one can be installed and registered from the runbook.
- Integrations listed: mail relay, webhooks, ticketing hooks, scheduled jobs, glue scripts, console certificate — each restorable from the repository.
- Secret names listed, values in a store: the runbook names every secret the build needs; the repository contains none of their values.
- Smoke test defined: device count, probe check-in, a test alert delivered, dashboards rendering.
- Offline copy: runbook and secret-name list available without the network, the server, or the repository host.
- Rehearsed: by someone other than the original builder, on a lab VM, within recent memory, with the time recorded.
Where to go next
A reproducible setup is what all the other monitoring-as-code habits add up to. It combines the repository from version-controlling your checks, the generator from templated onboarding, and a runbook that turns them into a working server on demand. The lab VM you rehearse on doubles as the staging instance for testing changes, which is where the next guide picks up. The pillar page Monitoring as Code collects the full set and the maturity ladder they climb.
Put this into practice with SysPeep Monitor
Rebuilding from files is a habit that applies to any monitoring tool. The import and backup mechanics are product-specific. Check the product documentation for its import and backup options. Two of the layers in this guide map directly onto SysPeep Monitor. It runs as a central Windows service with optional lightweight remote probes for other sites. It stores its configuration and data in SQLite, MySQL/MariaDB, or SQL Server. So the database decision in your runbook is a real choice with a documented answer. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep