Twelve Homelab Exercises That Build Monitoring Muscle
A lab that is never broken is a small, quiet network with nothing to teach. The value is in the breaking: pulling the cable, filling the disk, stopping the database. It is also in watching what the monitoring server does about it while you hold a stopwatch. Each break is a failure you will one day meet in production. The point of meeting it here first is that you will recognise it. Recognition is most of diagnosis.
These twelve exercises assume the lab described in Building a Monitoring Homelab for Almost Nothing. That lab has a managed switch and a hypervisor host with a server, a database, and a monitoring server on it. It also has a single-board computer running a web service, and an access point. Every one of these exercises is safe in that lab and none of them is safe anywhere else. I have run all twelve on my own lab more than once, and two of them still surprise me.
By the end of this guide you will have a method for every exercise. You will have the twelve exercises themselves with what to watch and what you should have alerted on. You will have a table to record each one in, and a handful of lab-only commands to cause the trouble. And you will have a checklist for the ritual around each break.
In this guide
- The method: break, watch, write, fix, repeat
- Exercises one to four: links, disks, ports, and names
- Exercises five to eight: memory, addresses, bandwidth, and the monitor
- Exercises nine to twelve: clocks, dependencies, certificates, and recovery
- The exercise table
- Lab-only commands for causing trouble
- Fifty-three alerts on a Saturday
- The exercise checklist
- Where to go next
The method: break, watch, write, fix, repeat
Every exercise follows the same five steps, and the order matters. Break it: cause one fault, deliberately, and note the time on a clock you trust. Watch: do nothing for at least fifteen minutes except look at the graphs, the check states, and your inbox. Note the time each one changes. Write down what you would alert on: not what did alert, but what should have, and how soon, and who should have been told. Fix it: undo the break and watch the recovery as carefully as the failure. Repeat: run the same exercise again another day until nothing in it surprises you.
The third step is the one that builds the muscle. Anyone can watch a graph go red. Deciding what a good alert for that graph would say, and when it should fire, and when it should not, is the actual skill. The guide to the anatomy of a good alert describes what you are aiming for. Write the alert you wish you had received, in words, before you touch any threshold.
Snapshot before every exercise. A snapshot is a saved copy of a virtual machine's exact state that you can return to in a minute. Safe Experimentation covers the ritual. Several of these exercises leave a virtual machine in a state that is easier to roll back than to clean up. (Exercise five in particular does not clean up after itself. That is rather the point of it.)
Exercises one to four: links, disks, ports, and names
1. Pull a cable
- Set up: unplug the single-board computer from the switch. Note the time.
- Watch: the web check, the ping check, and the switch port's operational status. Note which fails first and how many seconds apart they are. Watch the port's link counter on the switch.
- Alert on: one alert, for the device, arriving within a couple of polling intervals. Not three alerts, one per check.
- Lesson: a single physical fault shows up in several checks at different speeds, and the alert should describe the device, not each symptom.
2. Fill a disk
- Set up: on the server virtual machine, create one large file that takes free space from plenty to almost none. The command is in the section below.
- Watch: the disk-space graph, and what the operating system does in the last five per cent: logs stop, services complain, the login gets slow.
- Alert on: a warning with enough time left to act, and a critical alert before the disk is actually full. Decide what "enough time" means for a disk that fills at this rate.
- Lesson: a percentage threshold is meaningless without the rate. Ninety per cent of a large, slow disk is a week; ninety per cent of a small, busy one is ten minutes.
3. Flap a port
- Set up: work from the server virtual machine's console (not over the network). Run a loop that brings its network interface down for twenty seconds and up for forty, six times.
- Watch: the availability check turning red and green, the alert count climbing, and whether the monitoring server sends a recovery notice for every recovery.
- Alert on: one alert that says "this device is flapping", after the second or third cycle. Not twelve emails.
- Lesson: a check that fires on a single failed poll is a check that emails you every time a cable is wobbled. Hysteresis and confirmation is the cure, and this is where you see why it exists.
4. Break DNS
- Set up: point the database virtual machine at a resolver address that does not exist. The resolver is the server a machine asks to turn names into addresses; without it, a machine can reach everything and find nothing.
- Watch: the ping check (still fine), the port check (still fine), and anything on the machine that uses a name. Examples are package updates, log shipping, and a script that calls another host. Watch how long each one takes to fail.
- Alert on: a check that resolves a name from that machine's point of view, not just from the monitoring server's.
- Lesson: "up" is not "working". A machine with broken DNS passes every basic check and does nothing useful. The DNS troubleshooting guides cover what a name-resolution check should look like.
Exercises five to eight: memory, addresses, bandwidth, and the monitor
5. Leak memory
- Set up: on the server virtual machine, run a small script that allocates a fixed amount of memory every few seconds and never frees it. Snapshot first.
- Watch: free memory, swap or page-file use, and response time of the web page or share on the same machine. Note the point where the machine becomes unpleasant to use, and the point where it stops responding to the monitor.
- Alert on: the trend, not the level. A machine at seventy per cent memory that has been at seventy for a year is fine. A machine that went from thirty to seventy in an hour is not.
- Lesson: the interesting figure is the slope, and the machine stops answering the monitor before the memory is actually gone.
6. Exhaust a DHCP scope
- Set up: on the lab firewall or whichever device hands out addresses, shrink the DHCP pool to three addresses. Then connect a fourth device, such as a phone on the access point.
- Watch: the pool's lease count, the fourth device's failure to get online, and whether anything on the monitoring server notices at all.
- Alert on: free addresses in the pool falling below a small number, well before zero.
- Lesson: a full scope produces no error on any server and a queue of users at the desk. DHCP monitoring covers watching the pool rather than the complaints.
7. Saturate a link
- Set up: copy several gigabytes from the server virtual machine to the single-board computer, so the single-board computer's switch port carries as much as it can.
- Watch: the port's utilisation graph, its discard counter, and the response time of the web check during the copy. Note when the copy started and stopped and find both on the graph afterwards.
- Alert on: sustained utilisation near the port's speed for longer than a few polling intervals, and any discards at all.
- Lesson: a saturated link is not down and rarely errors; it discards, and everything behind it gets slow. Knowing who is filling it is a job for flow monitoring, which the utilisation graph cannot tell you.
8. Stop the monitoring database
- Set up: on the monitoring server itself, stop its database service, or stop the database virtual machine if the monitor's data lives there. Leave it stopped for half an hour.
- Watch: the dashboard. Does it show stale data as if it were fresh? Does the last-updated time move? Does anything tell you that the monitor is no longer monitoring?
- Alert on: the monitor's own silence, from something outside it. Use a heartbeat check on a second machine that expects to hear from the monitor and complains when it does not.
- Lesson: a monitoring server that cannot write is a monitoring server that shows the last thing it saw, indefinitely, in green. Somebody else must be watching the watcher.
Exercises nine to twelve: clocks, dependencies, certificates, and recovery
9. Change the switch's clock
- Set up: set the managed switch's clock to a time three hours wrong, then do something it will log, such as bouncing a port. Restore the clock afterwards.
- Watch: the log collector's view of the event. Where does the entry appear in the timeline? Can you tell it happened just now?
- Alert on: a device whose clock differs from the monitor's by more than a minute, checked as its own thing.
- Lesson: an incident timeline built from logs with one wrong clock has the cause after the effect. The syslog and log collection guides cover time-stamping and why the collector's clock is the one that counts.
10. Take down the "core" switch and count the alerts
- Set up: power off the managed switch, with everything behind it. Note the time. Count every alert that arrives in the next ten minutes.
- Watch: the order the alerts arrive in, and whether any of them says "switch". Or do they all say "single-board computer", "access point", "server", and so on, in whatever order the poller reached them?
- Alert on: one alert, for the switch, and the rest suppressed as consequences.
- Lesson: without a dependency model, one fault produces one alert per device behind it, and the real one is buried in the pile. Dependency thinking is the model; this exercise is where you feel the need for it. Configure the dependencies, run it again, and count again.
11. Expire a certificate
- Set up: give the single-board computer's web service a self-signed certificate with a one-day lifetime. Add a check that fetches the page over HTTPS. Wait a day.
- Watch: what the web check reports the morning after. A browser warning is not an alert; find out whether your check fails, warns, or cheerfully ignores it.
- Alert on: days remaining on the certificate, with a warning at thirty days and a louder one at seven, checked daily, not the expiry itself.
- Lesson: certificates fail on a known date and still surprise everyone, because the only check most people have is a user with a browser.
12. Reboot everything at once
- Set up: power off the switch, the host, the single-board computer, and the access point together. Wait a minute, and power them all on together. This is what a power cut and its recovery look like.
- Watch: the order things come back, and which ones come back wrong. Look for a virtual machine that boots before the switch has a link, or a web service that starts before the database. Look for a monitor that starts and immediately alerts on everything still booting.
- Alert on: things that have not recovered after a reasonable settling time, not things that are still starting.
- Lesson: recovery order is a dependency graph too. A monitor without a settling period announces a power cut twice, once when it happens and once when it ends.
The exercise table
Record every run in one table, one row per exercise per run, in the same notebook as the exercise log. The columns are deliberately plain, and the last two are the ones an interviewer or a colleague will care about. Three example rows follow to show the level of detail; yours should be about this long.
| Exercise | What I broke | What the graph showed | What I would alert on | Lesson |
|---|---|---|---|---|
| 1. Pull a cable | Unplugged lab-sbc1 from port 5 at 19:02 | Web check red at 19:03, ping red at 19:04, port down on the switch at the same poll; recovery in one poll after replug | Device down after two failed polls, one alert for the device, not three | Checks fail at different speeds; the alert names the device |
| 3. Flap a port | Interface down 20 s, up 40 s, six cycles on lab-srv1 | Six red-green pairs; eleven emails, including five recoveries | "Flapping" after the second cycle; suppress until stable for ten minutes | Confirmation counts exist for this; set them |
| 10. Core switch down | Powered off lab-sw1 at 20:15 | Seven alerts in three minutes, in polling order; none said "switch" | One alert for lab-sw1; the rest suppressed as dependent | Build the dependency model before the next run |
Notice that "what I would alert on" is written as a sentence a monitoring server could act on: a condition, a count, a duration. Vague entries such as "when it breaks" are the sign that the exercise needs another run.
Lab-only commands for causing trouble
A few of the exercises need a command, and here are the ones I use. Every line here is for the lab and nothing else. Two of them will wreck a machine that has a job, which is the reason they are useful on one that does not. Run them on the virtual machine you are breaking, from its console.
# Lab only. Exercise 2: fill a disk with one large file.
fallocate -l 20G /var/tmp/filler # shell; delete /var/tmp/filler to recover
fsutil file createnew C:\filler.bin 21474836480 # Windows; twenty gigabytes
# Lab only. Exercise 5: leak memory slowly. Stop with Ctrl-C or roll back the snapshot.
$hog = @(); while ($true) { $hog += [byte[]]::new(50MB); Start-Sleep 5 } # PowerShell
a=""; while true; do a="$a$(head -c 10M /dev/zero | tr '\0' x)"; sleep 5; done # shell
The port flap and the link saturation are one line each. The flap must be run from the hypervisor's console window. The first "down" would otherwise cut off the session running it, which is a lesson of its own.
# Lab only. Exercise 3: flap the interface six times. Run from the VM console, not over SSH. for n in 1 2 3 4 5 6; do ip link set eth0 down; sleep 20; ip link set eth0 up; sleep 40; done # Lab only. Exercise 7: push four gigabytes of nothing at the single-board computer. dd if=/dev/zero bs=1M count=4096 | ssh lab-sbc1 'cat > /dev/null'
For exercise eleven, most platforms have a built-in way to make a short-lived self-signed certificate. On Windows the certificate cmdlet takes an expiry date, and one day from now is the value you want. For exercise nine, the switch's own clock command is the tool. Check the documentation for yours; the important part is that the time is wrong by an amount you will recognise in the logs.
Tip: keep the commands in a file named for what they do, such as lab-break-disk.sh, and keep the undo beside each one. Half the value of an exercise is running it a second time without having to remember how.
Fifty-three alerts on a Saturday
At Northgate Retail the core switch in the main store lost power at ten past nine on a Saturday morning. Fifty-three alerts arrived in four minutes: every till, every access point, the back-office server, two printers, and the door counter. The on-call administrator had never seen more than three alerts at once. He acknowledged them one at a time in the order they arrived, which happened to be alphabetical. The switch was found forty minutes later by a store manager who noticed its lights were off. The review recorded that the monitoring had been "comprehensive".
He built a lab the following month and ran exercise ten until it bored him. The next time a switch went dark, at a different store, he read the first three alerts. He recognised the shape and went straight to the switch. Nobody had taught him that. He had counted twenty-six alerts from a lab switch on an old desktop, twice, and the pile looked the same at fifty-three. The dependency model he proposed afterwards turned the next one into a single alert with the switch's name in it.
The exercise checklist
Run through this for every exercise, every time. Each line is a yes or a no, and the exercise counts only when every line is a yes.
- The virtual machines involved have a fresh snapshot, taken in the last hour.
- I have written down the time on a clock I trust before causing the fault.
- I know how to undo the fault, and the undo is written beside the command.
- I watched for at least fifteen minutes without fixing anything.
- I noted the time each check, graph, and alert changed.
- I wrote the alert I wish I had received, as a condition with a count and a duration.
- I undid the fault and watched the recovery as closely as the failure.
- The exercise table has a new row with all five columns filled in.
- I have changed one threshold, dependency, or check as a result, and noted what.
- I have a date for running the same exercise again.
Where to go next
These are twelve exercises, each a fault you will one day meet with a manager in the doorway. Each one is better met first with a cup of tea and a stopwatch. Break it, watch it, write the alert you wanted, fix it, and do it again. When the twelfth run of an exercise teaches you nothing, you have learned it, and that is the only graduation a lab offers.
Before you run the destructive ones, read Safe Experimentation for the snapshot ritual and the isolation rules. When the exercise table has a few dozen rows, Carrying Homelab Lessons to Production covers turning "what I would alert on" into something that survives contact with a real network. The guide Why a Homelab Teaches What Production Cannot is honest about the limits. All of these are part of The Homelab as a Monitoring Practice Ground.
Put this into practice with SysPeep Monitor
Most of these exercises are about what the monitor says when something breaks. The free trial at /download/ gives you a monitor to say it. It has a central Windows service with SQLite for a lab, auto-discovery of the lab subnet, and SNMP, WMI, and SSH collectors. It also has thresholds, dependencies, and alert rules you can tune between runs of the same exercise.
Download Free TrialLearn About SysPeep