Home › Topics › Monitoring Skills and Certifications for Career Growth › Learning by Lab

Learning Monitoring by Lab: Exercises That Build Real Skill


You can read every guide in this library and still freeze the first time a real alert fires. That is because reading tells you what monitoring looks like and not what it feels like at the moment it matters. That feeling is learned by doing. The fastest way to do it safely is in a lab, where you can break things on purpose and nobody emails you about it.


A lab is any small network you are allowed to break. It can be a virtual machine on your laptop, an old desktop under the desk, or a handful of devices that were on their way to the skip. What makes it a lab is not the hardware. It is that you are allowed to pull the cable.


By the end of this guide you will have a method. Break something, watch what the graphs and alerts do, and write down what you would have alerted on and why. You will have seven exercises in order from first ping to first full incident and a place to do them. You will also have a write-up habit that turns them into a portfolio and an exercise log you can copy.


In this guide

  • Reading about monitoring versus doing it
  • The method: break, watch, write
  • The exercise progression
  • Where to do it
  • The write-up habit
  • The exercise log
  • Forty alerts at Harbourview Clinic
  • The lab checklist
  • Where to go next

Reading about monitoring versus doing it


Reading teaches vocabulary and doing teaches judgement. After reading, you know what a threshold is. After setting one too low and being woken by it three nights running, you know what a threshold is for. The second kind of knowledge is the kind interviews probe and incidents test, and no amount of reading produces it.


The gap shows at the moment of the alert. The reader knows, in principle, that a core switch failure will make everything behind it look dead. The person who has pulled a lab switch's uplink and watched forty alerts arrive recognises the shape of it when it happens for real. And recognition is most of speed. Their hands know which graph to open. The reader's hands are still deciding.


There is no exam for this kind of knowledge. (The exam is the outage, and it is not multiple choice.) A lab lets you sit it early, in private, as many times as you like.


The method: break, watch, write


The method has three steps, and the third is the one people skip. Break something deliberately and note the time. Watch what the monitoring shows: which graph moved, which alert fired, how long it took, what the alert said. Write down what you would have wanted it to tell you and why. The writing is where the skill forms, because it forces you to say what a good alert for this fault would have been. That is the whole job.


"Break" means a controlled, reversible fault: unplug a cable, stop a service, fill a disk, block a port. "Watch" means the monitoring system's view, not yours. You know the cable is out. Does the system know? How many minutes passed before it said so? What did the graph do in the seconds before the alert? "Write" means one page, in a fixed shape, which the write-up section below gives you.


Break before you watch, so that you know the truth and can judge whether the monitoring told it. This is hypothesis-driven diagnosis run backwards. Instead of guessing the cause from the symptoms, you know the cause and study the symptoms. Do enough of that and the forward direction gets much faster, because you have seen the symptoms before and know what they are attached to.


The exercise progression


Seven exercises, in order, each building one skill and revealing one trap. Each fits in an evening. Do not skip the early ones because they look easy. The trap in each is the thing that catches experienced people, and it is cheaper to meet it in a lab.


First ping

Add one device to the monitoring system with a ping check and watch it turn green. Then unplug it and time how long the system takes to say so. A ping is a small packet that asks a device "are you there?" and expects a reply. The protocol it uses is called ICMP. The skill built is the most basic one: knowing what "up" means to a monitoring system, which is only that something answered. The trap is the polling interval, the gap between one check and the next. If the system checks every five minutes and needs three failures before it alerts, a device can be dead for a quarter of an hour before anyone hears. You will feel that quarter of an hour, standing there holding the cable.


A useful companion to the first ping is your own ping, running with timestamps in a terminal. That lets you compare when the device actually stopped answering with when the monitoring noticed:


ping -t 192.0.2.10 | ForEach-Object { "$(Get-Date -Format HH:mm:ss) $_" }

That is the Windows PowerShell form; on a Linux host, ping 192.0.2.10 | while read line; do echo "$(date +%T) $line"; done does the same. The difference between the two timestamps is the first number you write down.


First SNMP poll

SNMP (Simple Network Management Protocol) is the question-and-answer language most network devices speak. An OID is the numbered address of one fact inside the device, like a drawer in a filing cabinet. The exercise is to poll one interface's byte counter on a switch or on the monitored virtual machine and watch the graph build. Start with a single manual query to prove the device answers at all:


snmpget -v2c -c public 192.0.2.1 1.3.6.1.2.1.1.3.0

That asks for the device's uptime. Once it answers, add the device to the monitoring system and let it poll an interface counter for an hour while you copy a large file across. The skill built is reading a rate, bytes per second, from a counter that only ever counts up. The trap is that the first sample shows nothing, because a rate needs two samples to exist. Another trap is that a wrong community string (the SNMP password) produces not an error but silence. Silence is the most common SNMP failure and the hardest to notice, because a graph with no data looks calm.


First threshold

Set a disk-space warning on the monitored machine at a value you choose. Then fill the disk past it with one large file and watch the alert fire. Delete the file, and watch it clear. A threshold is the value at which a metric becomes an alert. On Windows, fsutil file createnew C:\lab\fill.bin 5000000000 makes a five-gigabyte file in a second; on Linux, fallocate -l 5G /tmp/fill.bin does the same. The skill built is choosing the number, and the trap is that most people choose a round one. The right threshold is the value at which you would actually do something. And the way to find it is to ask "at what point would I start deleting files?" and set the alert a little before that. Eighty is not an answer to that question. It is a habit.


First dependency

Put two monitored machines behind a switch, real or virtual, monitor the switch too, and unplug the switch's uplink. Count the alerts. Then tell the monitoring system that the two machines depend on the switch. That means "if the switch is down, do not alert on the things behind it". Unplug the uplink again. Count again. The skill built is dependency modelling. And the moment the count drops from three to one is the moment most people understand why it matters. The trap is the opposite mistake: a dependency set wrong hides a real fault. If a machine dies while its parent switch is also down, you will not hear about the machine until the switch comes back. In that case, a badly drawn dependency tree can hide it forever.


First false alarm

Create an alert that fires for something harmless, such as CPU above a low value, on a machine that runs a scheduled job every night. Then leave it alone for a week. The skill built is recognising noise and its cost. The trap is you: by day four you will have stopped reading the alert. And by day six you will have deleted one without looking. And the lesson is that this happened to a careful person who knew the alert was fake. Alert fatigue is a mechanism, not a character flaw. Once you have felt it in a lab you will fight it harder in production. You will also be gentler with the colleague who missed the real alert among the fake ones.


First runbook

Write, for the disk alert from the third exercise, what to do when it fires. Then have someone else follow it, or follow it yourself a fortnight later, without remembering. A runbook is that short document. The skill built is writing for the person at three in the morning, who is tired and does not know what you know. The trap is the runbook that says "investigate the cause", which is not an instruction but a wish. Runbooks as training shows what a runbook can teach beyond its steps; for the lab, the test is simpler. If the reader has to ask you a question, the runbook failed.


First full incident with a timeline

Break two things ten minutes apart, say stopping a web service and then filling a disk on a different machine. Play both roles: the person who broke it and the person on call. Handle both from the monitoring alone. Afterwards, reconstruct a timeline, a list of what happened and when, using only the monitoring system's timestamps. Then compare it with what you remember. The skill built is the whole incident loop: detect, diagnose, fix, record. The trap is your memory, which will disagree with the monitoring's clock about the order of events, and the monitoring will be right. Finish by writing a short review of it in the style of a blameless review. That feels strange when you are the only person to blame and is excellent practice for when you are not.


Where to do it


A virtual machine on a laptop is enough to start. One virtual machine runs the monitoring system, one is monitored, and the virtual network between them is the switch. Exercises one, three, five, and six need nothing more. The SNMP exercise wants a device that speaks SNMP, which can be the monitored virtual machine with its SNMP service turned on. The dependency exercise wants a second monitored machine, which is a second virtual machine. The full incident wants both. Nothing here requires buying anything.


If you want real cables to pull, homelab monitoring practice is the group of guides about building a small physical lab. The lab uses an old desktop, a managed switch with a handful of ports, and whatever else was heading for the skip. The building details live there. Everything in this guide runs on either kind of lab; the physical one only adds the satisfaction of a cable in your hand.


Use the monitoring system you will use at work, if you can. A free trial of a commercial product is a legitimate lab install. Learning your work tool in a lab doubles the value of every exercise, because the muscle memory transfers on Monday. If you do not yet know what work will use, any system with checks, thresholds, and dependencies will teach the same lessons.


Do not run the exercises on the production network. Not "just the ping one". Not "only for a minute". A lab is a place where breaking things is the point, and the production network has a different point.


The write-up habit


Write one page per exercise, in the same shape every time. The shape is: what I broke, what the monitoring showed, what I would alert on and why, what surprised me. Include one screenshot of the graph with the moment of the break marked on it. One page is a rule, not a guideline, because it forces you to decide what mattered, and deciding what mattered is the skill.


These pages become your portfolio. Seven exercises produce seven one-page write-ups with seven annotated graphs, and that is more evidence of monitoring skill than most working administrators can show. Building a monitoring portfolio covers how to keep and present them. The only rule that matters now is to write the page the same evening, while the surprise is still fresh. Surprise fades by morning, and it is the most valuable line on the page.


Tip: put the clock in the screenshot. A graph with the break marked at 19:42 and the alert at 19:51 tells the whole story of the polling interval trap in one image. It is the image an interviewer will remember. A graph with no times on it is a shape.


The exercise log


Keep one row per exercise in a table, and add to it every time you repeat one. The rows below are filled in as an example of what a first pass looks like; the fifth column is where the learning lives.


ExerciseWhat I brokeWhat the monitoring showedWhat I would alert onWhat surprised me
First pingUnplugged lab-host-1 at 19:42Host down alert at 19:51; graph flat then gapThree missed polls, not one, to avoid flappingNine minutes felt very long
First SNMP pollNothing yet; copied a file to make trafficEmpty graph for five minutes, then a rateInterface rate above 80 per cent of link speed for ten minutesWrong community string produced no error at all
First thresholdFilled C: with a five-gigabyte fileWarning at 85 per cent; cleared when deletedWarning at the point I would start deleting; critical at the point services failI chose 80 because it was round, then changed it
First dependencyUnplugged the lab switch uplinkThree alerts; after setting the parent, oneThe switch only, with children suppressedA dependency set wrong hid a real host fault
First false alarmSet CPU warning at 30 per cent on a machine with a nightly jobAlert every night at 02:00 for a weekNothing; deleted the alertI stopped reading it by day four
First runbookNothing; wrote the disk runbookNot applicableNot applicableMy reader asked what "investigate" meant
First full incidentStopped the web service, then filled a disk ten minutes laterTwo alerts; timeline reconstructed from timestampsService check on the site, not just the hostI remembered the order wrong; the monitoring did not

Repeat an exercise when its "what surprised me" cell is empty. An empty surprise column can mean the exercise was too easy for you now. In that case, move on. Or it can mean you did not look closely enough. In that case, do it again with the graph open. It is usually the second.


Forty alerts at Harbourview Clinic


An administrator at Harbourview Clinic built a lab at home from an old managed switch and six virtual machines. He added every one of them to a monitoring system with ping and SNMP checks. To see what would happen, he pulled the switch's uplink. Forty-one alerts arrived in ninety seconds, one per host and one per interface. The switch itself arrived last because its check had the longest polling interval. He spent the evening drawing the dependency tree on paper. He set the switch as the parent of everything behind it and pulled the cable again, which produced one alert. Six months later the clinic's real core switch lost power on a Saturday. The monitoring system sent exactly one alert, with the right device name on it. His manager asked how he had known to set that up. He said he had done it at home, which was true, and did not mention the forty-one.


The forty-one alerts were not a failure of the lab. They were the lab working. Every one of those alerts, met for the first time on a real Saturday, would have cost a minute of confusion. There is no reading that substitutes for having once been the person receiving them. Layered troubleshooting teaches the same lesson from the other side. A fault low in the stack looks like many faults higher up, and the person who has watched that happen on purpose is not fooled by it.


The lab checklist


Run through this before you call the progression finished.


  • I have a lab that is separate from any production network, and I can break it without asking anyone.
  • I have timed the gap between a device going down and the monitoring saying so, and written the number down.
  • I have polled one counter over SNMP and watched a rate appear on a graph.
  • I have set one threshold from "what would I do at this value" rather than from a round number.
  • I have set one dependency and counted the alerts before and after.
  • I have lived with one false alarm for a week and noticed when I stopped reading it.
  • I have written one runbook and had someone follow it without asking me a question.
  • I have handled one two-fault incident from the monitoring alone and reconstructed its timeline from timestamps.
  • Every exercise has a one-page write-up with an annotated graph, written the same evening.
  • The exercise log has a surprise in every row, or a note that the row was repeated.

Where to go next


The lab teaches what reading cannot. It teaches how long nine minutes is and what silence from SNMP looks like. It teaches why forty alerts arrive when one thing fails and how quickly a careful person stops reading a fake alert. Each exercise is small. Together they are most of rungs one to three on the monitoring skill ladder, done in evenings, with evidence left behind.


When the write-ups are stacking up, building a monitoring portfolio turns them into something you can hand across a table. Both guides are part of Monitoring Skills and Certifications for Career Growth.


Put this into practice with SysPeep Monitor

A lab needs a monitoring system that behaves like the one you will use at work. SysPeep Monitor installs as a Windows service. It auto-discovers whatever is on the lab network and polls it over SNMP, WMI, and SSH. It lets you set thresholds and dependencies from one web console. So the exercises above work on it as written. The free trial is a legitimate lab install on a spare machine or a virtual machine.

Download Free TrialLearn About SysPeep