Home › Topics › The Homelab as a Monitoring Practice Ground › Why a Homelab

Why a Homelab Teaches What Production Cannot


You have read about thresholds and dependencies and SNMP, and you have nowhere to try any of it. The network at work is the real network. Every switch on it has a user behind it. Every server has a job to do. And the monitoring server is the thing that tells the on-call person whether to get out of bed. You cannot pull a cable to see what the graph does. You cannot fill a disk to see which alert fires first. So you read, and you nod, and the knowledge stays on the page.


A homelab is a small network you own entirely, at home or on a corner of a desk, whose only user is you. It exists so that you can break things. I built my first one because I had changed a threshold at work without understanding it and spent a weekend explaining why. I have kept one ever since. That is because there are things a lab teaches that thirty years of production never will.


By the end of this guide you will know why production is a poor classroom. You will know what a lab gives you that nothing else does, and the honest list of what it cannot give you. You will know whether you are someone who needs one at all. You will know how small a useful lab can be. You will also know how to write down what you want it to teach you before you plug anything in.


In this guide

  • Production is a terrible classroom
  • What a lab gives you
  • What a lab will never teach you
  • Who a lab is for, and who can skip it
  • The minimum lab
  • The lab goals template
  • A threshold change with nowhere to test it
  • The starting checklist
  • Where to go next

Production is a terrible classroom


Production is a terrible place to learn because every mistake has a victim. Production is the network and the servers that people rely on to do their jobs. That includes the file server with the accounts on it and the switch the reception desk hangs off. It also includes the monitoring server that pages someone at three in the morning. Learning means trying things, and trying things means some of them fail. In production, a failure is not a lesson. It is an outage with your name on it.


The second problem is that production does not repeat itself on request. You will see a resolver fail perhaps once in two years. You will see a switch flap a port for an afternoon, once. Then the cable will be replaced and you will never see it again. One exposure is not enough to learn what a fault looks like on a graph, any more than one drive around the block teaches you to park. You need to see the same failure enough times that you stop being surprised by it.


The third problem is that you only own a slice. In most organisations one person runs the switches, another runs the servers, and a third gets the complaints. The switch counters that would explain the slow file server belong to a team you have to email. By the time the reply arrives, the fault has cleared and the counters have rolled over. You learn a lot about email that way.


So people learn monitoring in one of two ways: slowly, by watching other people's incidents from the side, or dangerously, by making changes to the real thing and hoping. Both work eventually. Neither is a method.


What a lab gives you


A lab gives you four things production cannot, and they matter in this order.


Safe failure

In a lab you break things on purpose, and nobody is harmed. You pull the cable. You watch the interface graph flatten and the availability check go red and the alert arrive in your inbox. You time the whole sequence with a clock. Then you plug it back in and watch the recovery. You have now seen a link failure from beginning to end. Almost nobody in production ever does. That is because in production the first thing you do is fix it and the last thing you do is look at the graph.


Repetition

A lab lets you do the same thing ten times until it is boring, and boring is the goal. The tenth time you fill a disk you already know which counter moves first and how long the warning takes to arrive. You also know what the graph looks like in the last five minutes before the disk is full. That knowledge is what lets you glance at a production graph and say "that disk is going to fill tonight" before anyone has asked. It cannot be read into you. It has to be watched into you.


Ownership of the whole stack

In a lab you are the network team, the server team, and the user, all at once. When the web service on your single-board computer stops responding, you can look at the switch port and the server's log. You can look at the browser in the same minute. You see that the port counter, the log line, and the error page are one event seen from three places. In production those three views belong to three people, and the meeting where they compare notes is scheduled for Thursday.


Time

In a lab nobody is waiting. You can leave a broken thing broken overnight and see what the monitoring system does about it at hour six. Does the alert repeat? Does it escalate? Does the dashboard still look alarmed, or has it quietly decided the fault is normal? Production never grants you that time, because in production the fix comes first and the understanding comes never.


What a lab will never teach you


A lab is a classroom, not a simulator, and it is worth being honest about the walls. Here is the list I give everyone who asks whether a lab will make them a senior administrator. It will not. It will make them a junior administrator who has seen things.


  • Scale. Three virtual machines do not behave like three thousand devices. Polling load, database growth, and an alert storm with four hundred alerts in it are things you can read about. You cannot reproduce them on an old desktop. The shape is the same; the weight is not.
  • Politics. In a lab nobody says no. There is no change board, no owner of the switch who wants a ticket, no manager who has a different idea about what "critical" means. A great deal of monitoring work is persuading people, and there is nobody in the lab to persuade.
  • A real ISP. Your home connection is not a business circuit. It has no service agreement, no carrier ticket queue, and no account manager who calls back on Tuesday. The lab teaches you what a saturated link looks like. It does not teach you what a carrier says when you tell them about it.
  • The weight of a real outage. Your pulse does not change when the lab goes down. In production, it does. Troubleshooting with a manager in the doorway is a different skill from troubleshooting at a desk with a cup of tea. Only production teaches it.
  • Other people's changes. The lab only changes when you change it. Production changes when somebody else does, at four on a Friday, without telling you, and that is most of what troubleshooting turns out to be.
  • Real traffic. Users do strange things to networks, and a lab has no users. The traffic patterns you generate on purpose are tidy. Real ones are not.

None of this is a reason not to build one. It is a reason to know what you are building. A lab gives you the mechanics: what the graphs look like, what the alerts say, which order things fail in. Production supplies the rest, and you will be much better at the rest if the mechanics are already boring to you.


Who a lab is for, and who can skip it


A lab is for three kinds of people, and I have been all three.


The junior administrator needs repetitions that nobody at work will give them. Nobody hands the new starter the core switch to experiment on, and rightly so. A lab is the only place they can be trusted with the whole network, because the whole network is theirs.


The solo administrator cannot experiment at work because every change is a production change. When you are the only one, there is no test network, no colleague to cover, and no afternoon when a mistake would be harmless. A lab is where the solo admin does the experiments that a team would do in a staging environment.


The career changer needs the vocabulary and the graphs before the first interview. It is one thing to say you understand SNMP. It is another to say "I set up a switch, walked its interface table, and watched the error counter climb when I plugged in a damaged cable." The second sentence is what an interviewer remembers.


You can skip a lab if your employer gives you a proper test environment. That means a spare switch or two, a test monitoring instance, and permission to break them. Where the monitoring server lives covers running a test instance beside the production one. If you have that, use it. It is closer to your real network than anything you will build at home. A home lab is for people who do not have one, which in my experience is most people.


Whichever you are, treat the lab as one rung on a longer ladder. Monitoring skills and careers lays out the skills in order and suggests keeping an exercise log. The lab is where most of the entries in that log come from.


The minimum lab


The minimum useful lab is one old computer running a free hypervisor with three virtual machines on it. And one of those is the monitoring server. That is enough to learn most of what this library teaches. A hypervisor is software that lets one physical computer run several pretend computers at once, each with its own operating system. Each pretend computer is a virtual machine, or VM. Free hypervisors exist for every desktop operating system; pick whichever installs on the machine you have.


The three virtual machines are a "server", a "database", and the monitor. The server is any operating system with a disk and a service on it, such as a web server or a file share. The database is anything that listens on a port and writes files to a disk; it does not have to hold real data. The monitoring server is either a classic checks-based monitoring server or a time-series stack, whichever you want to learn. That is the whole lab, and it will keep you busy for months.


This is enough because most monitoring lessons are about what a graph looks like when something goes wrong. A virtual machine going wrong looks the same on a graph as a physical server going wrong. A disk fills the same way. Memory leaks the same way. A stopped service stops the same way. What the minimum lab lacks is a switch, and you will want one later for the per-port counters. But you do not need it on day one.


Before you install anything, check whether the old computer can do the job. On a machine running a common desktop operating system, three commands answer these questions. Can the processor run a hypervisor? How much memory is there? And how many cores are there? (Lab only, as everything in this pillar is.)


# Lab only. Can this old machine host a few virtual machines?
grep -c -E 'vmx|svm' /proc/cpuinfo   # more than 0: the CPU supports virtualisation
free -g                              # total memory, in gigabytes
nproc                                # number of CPU cores

# Windows equivalent: run systeminfo and read the Total Physical Memory
# line and the virtualisation lines near the bottom of the output.

Then do the memory sum in front of yourself. Suppose the hypervisor itself needs two gigabytes to run comfortably, the monitoring server VM wants four, and the two small VMs want two each. That is ten gigabytes. A machine with sixteen is comfortable. A machine with eight is possible if the small VMs are trimmed to one gigabyte and the monitoring server is given three. In that case, it will be slow, and slow is fine in a lab. Nobody is waiting.


Tip: the monitoring server does not have to be new software to be a good teacher. Any monitoring system whose alerts you can watch arrive and whose graphs you can watch move will do. What matters is that you are allowed to break the things it is watching.


The lab goals template


Write down what you want the lab to teach you before you build it, because a lab without goals becomes a hobby. Hobbies are fine. But a hobby lab gets a rack and a second switch and a name. Eighteen months later its owner can explain how it is cabled and still cannot say what a saturated uplink looks like on a graph. The goals table keeps the lab pointed at learning.


The lab goals template has three columns. They are what I want to learn, what I need to build to learn it, and how I will know I have learned it. The third column is the one people skip and the one that matters. That is because "understand SNMP" is not a goal; "find the interface table on the switch from memory" is. Fill in five rows. When a row is done, replace it.


What I want to learnWhat I need to buildHow I will know I learned it
Recognise a saturated link on a graphOne VM, one large file copy, the switch port polled every minuteI can point at the graph and say when the copy started and stopped without looking at the clock
Understand why a threshold flapsOne check with a tight threshold on a noisy metricI can explain confirmation counts to a colleague and show a before-and-after graph
See what a dependency outage looks likeTwo VMs behind a "core" switch, dependencies configured on the monitorTaking the switch down produces one alert, not five
Know what the monitor does when its database stopsThe monitoring server VM, and permission to stop its database serviceI can describe what the dashboard showed and what it should have shown
Add a device by hand over SNMPA managed switch with SNMP enabled and a walk toolI can name the columns of the interface table from memory
Rehearse a monitoring server upgradeA snapshot, the current version, the next versionI have done it twice and written the steps down

Notice that every "how I will know" row is something you could show another person. That is deliberate. A goal you can demonstrate is a goal you can put in the exercise log, and later in an interview. And a goal you cannot demonstrate is a feeling.


A threshold change with nowhere to test it


At Harbourview Clinic the only administrator wanted to tighten the disk-space warning on the file server from ninety per cent to eighty-five. There was no test system, so she changed it on the production monitoring server at lunchtime. The threshold turned out to be inherited by every disk on every server, not just the one she meant. Forty-one warnings arrived in six minutes. Two of the forty-one were real, and they were the two nobody read. The file server filled that night. The review recorded that the change had been "tested in production", which is a phrase that only sounds like testing.


The change itself was right. The place was wrong. In a lab with three virtual machines she would have discovered the inheritance in ten minutes. There would have been three warnings instead of forty-one and nobody's file server involved. That is the whole argument for a lab in one afternoon: the same lesson, at a fraction of the cost, with no victims. The threshold tuning guides cover how to set the number; the lab is where you find out what the number does.


The starting checklist


Work through this before you buy or build anything. Every line is a yes or a no, and the lab is ready to start when every line is a yes.


  • I can name three things I want the lab to teach me in the next month.
  • Each of the three has a "how I will know" that I could show to another person.
  • I have checked whether my employer already provides a test environment I could use instead.
  • I have one computer that supports virtualisation and has enough memory for three virtual machines.
  • I have chosen a free hypervisor that installs on that computer.
  • I have chosen a monitoring server to learn: a classic checks-based server or a time-series stack.
  • I know which of the three roles I am: junior, solo, or career changer, and what that means I need most.
  • I have accepted the list of things the lab will not teach me, and I am building it anyway.
  • I have somewhere to write down what I did and what I saw, and I will use it from the first day.

Where to go next


Production cannot be a classroom because every failure there has a victim, and learning requires failing. A lab gives you the four things production withholds: safe failure, repetition, the whole stack, and time. It will not give you scale, politics, a real carrier, or the weight of a real outage. And you should build it anyway, because the mechanics it teaches are the part of the job that can be learned in advance. Start with one old computer and three virtual machines. Write five goals with a "how I will know" for each. Break something before the end of the first weekend.


The next guide, Building a Monitoring Homelab for Almost Nothing, turns the minimum lab into a small network with a switch and a few branch devices. Twelve Homelab Exercises gives you things to break once it is built. The guide Safe Experimentation keeps the breaking from reaching your home network or your job. And Carrying Homelab Lessons to Production covers what to bring to work and what to leave at home. If you want the ideas the lab is there to make concrete, Monitoring Fundamentals That Last is the list. All of these are part of The Homelab as a Monitoring Practice Ground.


Put this into practice with SysPeep Monitor

If the lab needs a monitoring server, the free trial at /download/ installs as a central Windows service on one virtual machine. It runs on SQLite for a small install and auto-discovers whatever you put on the lab subnet. It polls those devices over SNMP, WMI, and SSH. Break things in front of it and watch what it says.

Download Free TrialLearn About SysPeep