Home › Topics › Who Watches the Watcher › Upgrading the Monitor

Upgrading the Monitor Without Going Blind


Most of this pillar is about failures that sneak up on you. This guide is about the one time you walk up to the monitoring system and switch it off on purpose. An upgrade — new platform version, migrated database, patched components — means deliberately blinding your network's only witness for however long the work takes. Every check pauses. Every alert pipeline stops. If something breaks elsewhere on the network during that window, nobody will be told.


That sounds like an argument for never upgrading, and plenty of admins quietly treat it as one. That is how you end up with a monitoring server running a version so old the vendor's documentation refers to it in the past tense. Wrong lesson. The blindness is plannable, bounded, and survivable. The real risk is not the ninety dark minutes but doing them casually — no backup, no rollback plan, no verification. Or it is doing them so rarely that each upgrade becomes a heroic multi-version leap.


This guide covers the whole discipline. It covers framing the upgrade as planned blindness with a stated budget and the pre-flight ritual. It covers your options for coverage while the lights are out and the execution rhythm. It covers the verification pass that deserves star billing, the soak period afterward, and how often to do this at all. It borrows deliberately from every other guide in this pillar — upgrades are the day the backup, the dead-man switch, and the failure catalog all report for duty at once.


In this guide

  • Planned blindness: the framing and the budget
  • Pre-flight: backup, release notes, rollback deadline
  • Covering the dark window
  • The execution rhythm
  • Verification: the star of the show
  • The soak period
  • How often to upgrade
  • The full upgrade checklist

Planned blindness: the framing and the budget


First, a clarification that calms new admins considerably: upgrading the monitor does not take the network down. Switches keep switching, users keep working, nothing user-facing so much as flickers. What goes down is your awareness — the network keeps running, but its witness is asleep, and any failure during the window goes unannounced. That is a real risk, but a different and smaller one than a production outage, and it deserves to be planned as exactly what it is.


Now promote the event anyway. A monitoring upgrade is not "quick maintenance on a utility box". It is a maintenance window on a tier-one service. Give it the same planning weight you would give the file server or the phone system. That is because while the monitor is down, your visibility into everything else is down. All the discipline of a well-run planned-downtime window applies: a scheduled time, a written plan, a communication step, and a definition of done.


Two framing decisions come first. The blindness budget: state, before touching anything, how long you are willing to be dark — "up to two hours, absolute ceiling three". A stated budget is what turns "hmm, this is taking a while" into a decision point instead of a slow slide. And the schedule: put the window in genuinely quiet hours, which your own monitoring can tell you. Months of collected data know exactly when your network is at its sleepiest. That is the same way the baselining guides use that data to define normal. The traditional wrong answer is Friday at 5 PM, chosen by optimists who believe nothing interesting happens on weekends. Pick a time when a surprise has the smallest audience and you are still fresh enough to handle one.


Pre-flight: backup, release notes, rollback deadline


The pre-flight ritual is five items, none skippable. First, you need the full backup, taken and spot-verified immediately before. That means not last night's scheduled run, but a fresh export and database backup made after the final config change. Do a thirty-second sanity check: the files exist, are not zero bytes, and open. Everything about what and how is in the monitor backup guide; upgrade day is that guide's whole reason for existing. The backup taken five minutes before the upgrade is the one that matters most in your life.


Second, read the release notes for real — not the skim, the read. You are hunting three things: breaking changes (features removed or renamed, sensor types deprecated, integrations changed), migration steps (does the database schema convert on first start? how long might that take?), and version-path requirements. That last one surprises people. Many platforms upgrade cleanly only from the immediately preceding versions. So a very old installation may need to step through intermediate releases rather than jumping straight to current. Each hop has its own notes. Better to discover this at your desk than halfway up the ladder.


Third, write the rollback plan down, in three lines. Write the restore path ("reinstall previous version, restore config export and database backup"). Write the time it takes (you know this number from your restore drill — another gift from the backup guide). Write the decision deadline: "if the system is not healthy by 22:30, we roll back." The deadline is the crucial line. It is easy to be sensible about sunk cost at the kitchen table and nearly impossible at 1 AM, four fixes deep into a broken migration. The deadline is a letter from calm-you to tired-you, and tired-you should obey it.


Fourth, snapshot the VM if the monitor is virtual — a snapshot (the hypervisor's freeze-frame of the machine) makes rollback nearly instant. Take it with the service cleanly stopped, and remember the caveat: a snapshot is not a backup. It lives on the same storage as the VM, ages badly, and can slow the machine while it exists. It is a short-lived safety net for this window, deleted after the soak period, never a substitute for the real backup you already took. Fifth, notify the humans. Whoever receives alerts should know the dark window's start, expected end, and that silence during it means nothing. And anyone who was about to do risky work elsewhere on the network might sensibly wait a day, since they would be working without a net too.


There is a sixth item for the cautious: rehearse the upgrade on a scratch copy first. It is nearly free if you already run the backup guide's restore drill. Restore your backups onto a throwaway VM — a skill you have practiced — run the upgrade there, watch the migration complete, and time it. Every surprise you meet on the rehearsal stage costs nothing; the same surprise at midnight costs your rollback deadline. The rehearsal also converts the release notes' vague "migration may take some time" into "took forty minutes on our data." That upgrades your blindness budget from a guess to an estimate. For a small point release this is overkill. For a major version jump sitting on years of history, it is the difference between an upgrade and an adventure.


Covering the dark window


What watches the network while the watcher is on the operating table? Scale the answer to the window's length and your tolerance. For a short window in quiet hours, the honest answer is often: nothing, and that is acceptable — a bounded, scheduled gap you decided on in daylight. But one piece of housekeeping is mandatory either way. Your dead-man switch will correctly notice the monitor going silent and will correctly raise the alarm. So pause it deliberately, with the suppression discipline that guide teaches. That means a written end time, and un-pausing as an explicit checklist step at the end. A dead-man left paused is the upgrade's most dangerous leftover — you would exit the window believing you are covered, permanently wrong.


For longer windows — a big version jump, a slow database migration — run a minimal side-watcher. That is a temporary script on a different machine that pings the crown jewels and emails you on failure. A stopgap sketch:


every 60 seconds, on another machine:
    for each of: firewall, core switch, main host, key server
        ping it
        if no reply 3 times in a row:
            send email/text: "DARK-WINDOW WATCH: host X down"
(delete the script when the monitor is verified healthy)

It is crude on purpose — no thresholds, no history, a handful of pings. But it converts "totally blind" into "blind except for the six devices that matter most," which is a different risk category. And if you run a high-availability pair, you have the luxurious third option: the staged path. In that case, upgrade the standby node first, confirm it is healthy, fail over, then upgrade the other. That shrinks the blind window to the failover itself. The high-availability guide covers that rolling pattern properly.


The execution rhythm


The window itself follows a rhythm worth keeping boring: stop, backup, upgrade, migrate, start, verify. Six beats, same order every time:


  • 1. Stop cleanly. Shut the service down properly and let it finish writing — give queues a moment to drain rather than yanking the plug mid-poll. A clean stop is also the moment to take the VM snapshot, so the frozen image contains a resting system, not one caught mid-sentence.
  • 2. Confirm the net. Thirty seconds re-checking that the fresh backup and the snapshot actually exist. Yes, again. This is the last exit before the bridge.
  • 3. Upgrade. Run the installer or upgrade package, exactly as the release notes prescribe, resisting all creativity.
  • 4. Migrate, patiently. Schema migrations over years of monitoring history can take genuinely long. The classic self-inflicted disaster is killing a migration that looked hung but was quietly working. A corrupted half-migrated database is strictly worse than a slow one. If the notes said "may take a while" and the disk light is blinking, make tea.
  • 5. Start. Bring the service up and watch the log scroll for the first minute; startup complaints are cheapest to read now, while the cause is unambiguous.
  • 6. Verify. The part most people get wrong — wrong enough that it gets its own section.

Verification: the star of the show


The most common upgrade failure is not the dramatic crash — it is the upgrade that seems fine. The service starts, the console loads, the first dashboard is green, and everyone goes to bed. And a collector was deprecated, or a device class needs re-authentication. And a slice of your checks never came back — the silent partial, self-inflicted, wearing an upgrade's clothes. "It starts" is not the test. "It is doing everything it did before" is the test, and that has to be verified item by item:


  • 1. Service up and stable — running, and still running fifteen minutes later, logs free of repeating errors.
  • 2. All checks resumed — the check-count comparison: before the upgrade you recorded the number of active checks and monitored devices. The same numbers must appear now. A count that dropped from 412 to 361 is fifty-one silent holes with names. Find the list, and you have found what the upgrade quietly dropped.
  • 3. Fresh data flowing — the newest-sample-age sweep: spot-check busy devices and confirm their latest samples are newer than the window's end, graphs growing on the right edge.
  • 4. A test alert, end to end — fire one deliberately and watch it arrive on every channel. Upgrades are notorious for resetting mail settings and dropping integration tokens. And this is the notification drill's moment.
  • 5. Dashboards, maps, and reports render — open the ones people actually use.
  • 6. Integrations reconnected — webhooks, directory logins, anything holding a token; upgrades love logging you out of things quietly.
  • 7. Un-pause the dead-man switch — and confirm its pulse resumed.

The before-snapshot makes step 2 honest work instead of vibes. Five minutes before the window, save the check inventory — counts at minimum, an exported list at best — and diff it against the after state. The diff catches exactly the category of loss that eyeballing a green dashboard never will, for the same reason frozen graphs fool everyone. Green and absent look identical at a glance.


Tip: declare victory out loud, against your written definition of done: "all seven verification items passed at 21:40, inside the budget." If you cannot say that sentence by the rollback deadline, you do not negotiate with the deadline. In that case, you roll back, restore, and try again another night with what you learned. A rolled-back upgrade is a successful evening; a limping monitor you "will fix tomorrow" is next month's mystery outage.


The soak period


Passing verification does not end the upgrade; it begins the soak period — several days of elevated attention while the subtle breakage surfaces. Some things only show themselves on their own schedule. There is the weekly report that renders wrong, or the sensor type that was deprecated and stopped populating one graph. There is the device class whose collector now wants re-authentication and fails only on its daily cycle. There are thresholds behaving differently around a changed default. During the soak, glance at the self-monitoring checks daily — poll success rate, staleness alarms, check counts. Do this precisely because they are the instruments that catch quiet erosion. Keep the rollback window open until the soak passes. Hold the VM snapshot and the pre-upgrade backups until a full weekly cycle — including the scheduled reports — has run clean. Then delete the snapshot (it is not a keepsake) and return to normal watchfulness. Before you close the file on the whole event, write three lines of notes while the window is fresh. Record what the migration actually took, what broke, and what you would do differently. The next upgrade inherits those notes, and after two or three cycles the ritual compresses from a nervous evening into a practiced hour.


How often to upgrade


Finally, cadence philosophy. Staying reasonably current beats heroic leaps, every time. Small, regular upgrades mean small release-note reading, small migrations, small diffs, and a short list of suspects when something misbehaves. A five-version leap means compound breaking changes, stepped migration paths, and an evening where everything changed at once. The monitoring system also rewards currency for its own sake — new device support and fixed collectors are the difference between watching your newest equipment and guessing at it. A sensible rhythm for a small shop is minor updates within a month or so of release. Make bigger version jumps once the release has visibly settled. Put each update through this same checklist. The checklist gets faster every time you run it. If "read release notes on a schedule" sounds unlikely to survive contact with your calendar, tie it to a rhythm you already keep — patch night, the monthly maintenance review. That makes staying current a habit rather than a project.


The counterweight is timing hygiene: never upgrade during storms. Not while the network has an active incident, not the week of a major organizational event, not the day before you leave for vacation, and not stacked on top of other infrastructure changes. If the monitor and the hypervisor both change on the same night, every mystery for the next week has two suspects and you will interrogate both. The monitor's job during everyone else's chaos is to watch; upgrade it when the house is calm.


The full upgrade checklist


PhaseStepDone when
Pre-flightSchedule quiet window; state the blindness budgetWindow and ceiling written down
Pre-flightFresh config export + database backup, spot-verifiedFiles exist, open, and are current
Pre-flightRelease notes read; version path confirmedBreaking changes and migration steps listed
Pre-flightRollback plan with decision deadlineRestore path, duration, and deadline on paper
Pre-flightVM snapshot (service stopped); humans notified; dead-man paused with end timeAll three confirmed
DuringSide-watcher running (longer windows)Test email from the script received
DuringStop cleanly, upgrade, let migrations finish, startService stable; logs quiet
VerifyThe seven-item verification pass, against the before-snapshotCounts match; data fresh; test alert arrived; dead-man un-paused
SoakDaily glance at self-checks; weekly cycle completesReports and collectors clean for a full week
SoakDelete snapshot; retire side-watcher; file notesSystem back to normal watchfulness

Where to go next


An upgrade done this way is the pillar's ideas rehearsed under controlled conditions. The backup is proven fresh, the dead-man is paused and revived, and the silent partial is hunted with a diff instead of discovered by accident. Do a few, and the worst day — the unplanned one — gets easier too, because every tool got exercised. The rest of the toolkit lives in Who Watches the Watcher — start with the failure catalog if you have not read it, since it is the map this checklist defends against.


Put this into practice with SysPeep Monitor

Upgrade discipline is universal, but the mechanics are gentler when the platform is tidy. SysPeep Monitor runs as one central service on a Windows machine. It keeps its data in SQLite, MySQL/MariaDB, or SQL Server where your backup tooling can reach it. It offers high-availability clustering for the rolling-upgrade path. The free trial is a fine way to rehearse the whole checklist on a system nobody depends on yet.

Download Free TrialLearn About SysPeep