Home › Topics › Power & Environment Monitoring › Graceful Shutdowns

Graceful Shutdowns: Automating the Last Five Minutes of Power


The power has just failed. Your UPS units have caught the load. The batteries are draining. Somewhere a countdown is running that ends with every server in the room losing power mid-thought. You have, say, ten minutes. What happens in those ten minutes is the difference between a rough evening and a bad week. The only useful form of the question is: who decided it, and did they decide it in advance?


If the answer is "someone runs to the server room and starts clicking", you do not have a plan. You have a stress test for whoever is on call. Real outages arrive at night, during holidays, and while you are on the highway with no signal. I have taken the call in all three places. The last five minutes of power have to run themselves: triggered by the UPS, sequenced in the right order, and finished before the battery quits. That is what a graceful shutdown is. Every system is asked politely to stop, given time to finish writing what it was writing, and powered off clean instead of cut off mid-write. I have seen a great many shutdown scripts. I have seen far fewer that had ever been run on purpose.


This guide walks through the whole design. It covers what order things should shut down in and how to choose the trigger that starts the sequence. It explains why waiting a minute beats reacting instantly, and how to keep humans informed while the robots work. It covers how to test all of it without pulling any plugs. And it explains how to bring everything back in the right order when the power returns. None of it requires exotic tools — UPS vendors ship shutdown agents, operating systems take instructions, and your monitoring system supplies the honesty.


In this guide

  • The goal: nothing dies mid-write
  • Shutdown order: who goes first
  • Designing the trigger
  • The timing arithmetic, worked through
  • Tell the humans while the robots work
  • Testing without a real outage
  • When power returns: restarting in reverse
  • Write it down

The goal: nothing dies mid-write


Why does an abrupt power cut hurt? Because computers constantly hold work in flight. A database is halfway through updating an index. A file server has changes sitting in memory that it has promised to write "soon." A virtual machine host is juggling a dozen guests, each with its own in-flight writes. Cut the power at a random instant and you freeze all of that mid-motion. Sometimes you get lucky and everything replays cleanly at boot. Sometimes you get a database that needs repair, a virtual disk that will not mount, or a file system that comes back with damage. You find out which kind of day it is only after the power returns. That is the worst possible time for a surprise.


A graceful shutdown removes the randomness. Services are stopped so they finish their current work and close their files. Caches are flushed — everything held in memory gets written to disk. Virtual machines are shut down as guests, not switched off like lamps. Then the operating system itself stops, and only then does the power go away. The outage still costs you uptime; it just stops costing you data and a weekend of repair work. That is the entire mission, and every design choice below serves it: when the battery finally runs out, it must be interrupting nothing.


Shutdown order: who goes first


Order matters because systems depend on each other. Shut down storage while servers are still writing to it and you have created, on purpose, exactly the mid-write disaster you were trying to avoid. The safe order follows a simple rule: things that others depend on shut down last. In practice, that produces a sequence like this:


  • 1. Expendable compute first. Test machines, build servers, development boxes, the lab — anything whose sudden absence hurts nobody. Shutting these early buys the important systems extra minutes of battery, because every watt you shed slows the drain for everyone else.
  • 2. Storage-dependent services next. Application servers, databases, and virtual machines — the systems that hold files open on shared storage. Databases get a clean service stop; virtual machines get guest shutdowns, host by host. This is usually the longest, slowest step, which is why it must start early.
  • 3. Then storage itself. Only after its clients are gone does the file server or storage appliance shut down. It has nothing left to serve, all its caches flush cleanly, and nobody is mid-write. Storage shutting down last among servers is the step people get wrong most often, and the most expensive one to get wrong.
  • 4. The monitoring server near-last. Your monitoring system is your eyes during the event. It is telling you what is down, what is still up, and whether the sequence is working. So it should outlive the systems it watches. What happens when it goes dark is a story of its own, told in the guide on monitoring system failure modes.
  • 5. Network gear last, where feasible. Shutdown commands travel over the network: the UPS agent's "go" message, the guest-shutdown instructions, your own remote session. Kill the switches early and the rest of the sequence goes deaf. Most switches and routers do not need a graceful shutdown at all. They hold no data in flight and tolerate losing power. So the usual choice is to let them simply ride the battery to the end. The UPS management card itself stays powered by the UPS until the very last breath, which is exactly what you want.

Write your own version of this list with real hostnames, and you have the skeleton of the whole automation. Every environment has a wrinkle — a database that takes ages to stop, an appliance with no remote shutdown. The sequence is where those wrinkles get handled deliberately instead of discovered live.


Harbourview Clinic had a written sequence, and the monitoring server was first on it. The person who built it had a reason: if monitoring went down first, nobody's phone would fill with alerts about servers being switched off on purpose. It worked exactly as designed on the night a transformer failed, which meant the rest of the evening happened in the dark. The storage appliance shut down with three virtual machines still writing to it. That was because the one screen that would have shown those machines were still up had gone first. The sequence had never been run on purpose before that night. The review recorded that the order had been "reconsidered".


The mechanics are less mysterious than they sound. Each protected server runs a small piece of software — the UPS vendor's shutdown agent, or the operating system's own UPS support. This software hears about the power event from the management card over the network and runs its assigned shutdown when told. Ordering comes from configured delays or from a coordinator: "web servers act at minute two, storage at minute six." Many UPS models also offer switchable outlet groups — banks of outlets the UPS can turn off in stages. These give you a blunt but reliable ordering tool for devices too simple to run an agent. Virtual machine platforms add their own layer. The host's tools can shut guests down in a configured order before the host itself goes. Every one of these mechanisms is ordinary vendor equipment; the design work is deciding who acts when.


Designing the trigger


Something has to decide when the sequence starts. UPS shutdown agents generally offer two kinds of trigger, and the difference is worth understanding before you pick:


Trigger styleFires whenStrengthWeakness
On-battery durationThe UPS has been on battery for a set time, e.g. three minutesSimple, predictable, easy to reason aboutIgnores how much battery actually remains
Remaining-runtime thresholdEstimated runtime falls below a set floor, e.g. eight minutesAdapts to load: heavier load fires soonerTrusts the runtime estimate, which drifts as batteries age
Both combinedWhichever condition arrives firstCovers each trigger's blind spotSlightly more to configure and test

The duration trigger's flaw is that it treats a fresh battery and a dying one identically. The runtime trigger's flaw is that its input is the estimate. As the UPS monitoring guide explains at length, that estimate assumes a healthy battery and drifts toward fiction as the battery ages. It lies politely, but it lies. Using both, so that whichever fires first wins, is the defensive choice. The duration trigger acts as a backstop on the day the estimate turns out to be optimistic.


One more design decision hides in that table: the delay before doing anything at all. It is tempting to start shutting down the instant the UPS reports on-battery. Resist. Most power events are blips — a few seconds of flicker while the utility switches something, gone before you finish reading the alert. An instant-reaction sequence turns every two-second flicker into a self-inflicted outage of your entire server room. That is a hard thing to explain in the morning. Waiting even one to three minutes filters out nearly all blips while spending only a small slice of battery. The UPS exists precisely to buy you that waiting time; use a little of it to make sure the outage is real. Panic is not a strategy, and in this case it is not even fast — it is just early.


Finally, decide where the decision is made, and make it one place. If every server watches the UPS independently with its own thresholds, you eventually get the split-brain evening. Half the room has shut itself down over a blip while the other half never heard about a real outage. The cleaner pattern is one brain — the management card or one coordinator host — deciding "this is real" once and broadcasting the order to everyone else. And check the path that broadcast travels. If the "shut down now" message crosses a switch, that switch must be on UPS power too. Otherwise, your trigger design dies of irony at the first real test.


The timing arithmetic, worked through


Every trigger design must satisfy one budget, and it is easiest to see worked as a concrete example. Suppose your UPS reports eleven minutes of runtime at today's load. Suppose also that a full drill (below) has shown the shutdown sequence takes six minutes end to end. The first number must be measured, not hoped; the second must be timed, not guessed. Build the budget from the end backwards:


runtime at current load (measured):              11 min
safety margin for an aged battery (keep ~1/3):   -4 min
                                                 ------
battery you may actually plan against:            7 min
full shutdown sequence (timed in a drill):       -6 min
                                                 ------
latest possible trigger after power fails:        1 min

The uncomfortable conclusion falls straight out: this system can afford a blip-tolerance delay of about one minute, no more. There is no room for a leisurely "wait five minutes and see." If you want a longer decision window, the arithmetic tells you exactly what to buy it with. Shed load so runtime rises, replace a tired battery so the margin can shrink, or speed up the sequence by trimming what it must shut down. This little calculation is the single most useful thing to redo whenever anything changes — new battery, new servers, new services. Every change moves one of the numbers, and the numbers do not care that nobody told them.


Tip: if the arithmetic ever produces a negative number — the sequence needs more minutes than the battery honestly holds — do not fix it by trimming the safety margin. Fix it by shedding expendable load earlier, shortening the sequence, or adding battery. A budget balanced by optimism fails exactly once.


Tell the humans while the robots work


Automation should never be silent. The moment the UPS reports on-battery, a human should be told — immediately, before any shutdown begins, at whatever hour it is. This is not because the human must do anything, but because the human can do things the automation cannot. They can notice that the power company already announced a planned interruption. They can decide tonight is the night to cancel the sequence and ride it out, or start driving toward the building. The automation knows none of this. It knows the battery is draining, and it is right.


The notification set worth building is short. First, the on-battery alert, sent to the on-call person the instant the event starts. Second, a sequence-started message: "shutdown automation has begun, step 1 of 5," so nobody wastes the emergency wondering whether the plan is running. Third, send a completion or failure message. The failure case matters more. A sequence stuck halfway is the scenario where a human with a phone and a doorkey still changes the outcome. Finally, when utility power returns, say that too. Make every message say which UPS at which site. "On battery" means something very different from the server room than from a wiring closet three buildings away. And an alert that forces the reader to go look it up has wasted its own urgency. During all of this your monitoring dashboard is the shared source of truth for what is up and what is already down. That is one more reason the monitoring server was near the end of the shutdown list.


Testing without a real outage


An untested shutdown sequence is a rumor. The good news: testing does not require cutting power to anything. Work up through four levels, each catching what the previous one cannot.


  • Level 1 — the tabletop. Read the sequence out loud with a colleague and ask rude questions at every step. Does the storage really wait for the virtual machines? What talks to what, and over which switch? Tabletops are free and catch the embarrassing ordering mistakes before any machine has to.
  • Level 2 — single-system tests. Trigger the shutdown agent's action on one expendable machine and confirm the mechanics: the command arrives, services stop cleanly, the box powers off. Repeat for one of each kind of system — a virtual machine host behaves nothing like a storage appliance.
  • Level 3 — the simulated trigger. In a scheduled maintenance window, fire the whole sequence for real — with utility power still present. Most UPS agents let you signal the event manually or temporarily lower a threshold so it fires while nothing is actually wrong. Run everything end to end, and time each step with a stopwatch. This drill is where the six-minutes figure in the arithmetic comes from. And the first drill almost always finds a surprise — a box that hangs on a stuck service, a dependency nobody documented.
  • Level 4 — the real-power drill, rarely. Some teams eventually cut power to a UPS deliberately, in a planned window with backups verified, to prove the entire chain including the battery. It is genuinely valuable and genuinely riskier. Earn your way to it through the first three levels, and never let it be the way you discover the battery is weak. That is what self-tests were for.

Then repeat level 3 on a calendar rhythm. Environments drift: new servers join, sequences slow down, agents get unconfigured by accident. A drill that passed long ago describes an environment you no longer have.


When power returns: restarting in reverse


Half the plan is the way back up. The restart order is the shutdown order reversed — network first, then storage, then the systems that depend on storage, then everything expendable. But three cautions keep the reversal from becoming its own incident.


Do not trust the first minutes of restored power. Utility power often returns unsteadily — back for two minutes, gone again. If everything powers on the instant the lights blink on, a second failure catches your systems mid-boot with a drained battery. Configure the UPS or its agent to wait before it restores output to the load. Power must have been stable for a while and the battery must have recharged past a floor. Patience here is cheap insurance.


Respect dependencies with real checks, not hope. Storage must be fully up — volumes mounted, shares served — before the application servers boot. The applications need their databases answering before they start. Where you can, encode waits: staggered power-on delays in the UPS outlet groups, delayed service starts, or scripts that poll for "is the database answering?" before starting the next tier. A boot race you win by luck on Tuesday is one you lose on Sunday.


Stagger the power draw. Everything switching on at once produces a surge of startup demand — spinning disks, charging capacitors, fans at full. That surge comes at the exact moment your battery is at its emptiest. Staggering the restart by tier, which the dependency logic gives you anyway, spreads the surge out. And bring the monitoring server up early in the parade. It went down near-last so it could watch the descent, and it should be up near-first so it can watch the climb. When the dust settles, walk its dashboard and confirm every service came back. The restart that silently misses one box is the classic sequel to a well-handled outage.


Write it down


Everything above belongs in one written runbook. Include the order with real hostnames, the trigger settings and where they live, and the arithmetic with today's numbers. Include the drill results, the restart checks, and what a human should do when the automation reports failure. Store it where it can be read during a power outage. That means printed or on battery-powered devices, not only on the file server that just shut itself down. If your team has never written one, the guide on writing your first runbooks shows the shape. The power-failure runbook is one of the best first runbooks to write, because the procedure is real, bounded, and guaranteed to be needed someday.


Where to go next


Decide the order and pick a trigger with a blip-tolerant delay. Do the arithmetic against a battery you distrust by the right amount. Alert the humans, drill it in a window, and script the way back up. Power failures then stop being emergencies and become events: things that happen, run their choreography, and get talked about calmly the next morning. You will also own a shutdown script that has been run on purpose, which puts it ahead of most of the scripts I have met. The battery you were budgeting against deserves the same honesty. And the rest of Power & Environment Monitoring covers that side: what the UPS reports, why its runtime estimate flatters itself, and the environment around all of it.


Put this into practice with SysPeep Monitor

The shutdown itself belongs to your UPS vendor's agent and your operating systems — but knowing the moment the countdown starts is a monitoring job. SysPeep Monitor watches UPS state over SNMP and alerts you the instant a unit reports on-battery. Its dashboards show what is still up while the sequence runs. The free trial installs on a Windows machine in minutes.

Download Free TrialLearn About SysPeep