Safe Experimentation: What Belongs in the Lab and What Never Touches Production
A lab is safe because nothing in it matters, and that stays true only as long as you keep it that way. The moment the lab can reach the household's network, an experiment that exhausts a DHCP scope exhausts the family's. The moment a lab password is reused at work, a lab that anyone can log into is a production network that anyone can log into. Suppose you are afraid to break the lab because rebuilding it would take a week. At that moment, you have stopped experimenting and started running a second, smaller production network with worse hardware.
Safety in a lab has two directions. The lab must be safe for the home: nothing you do inside it may leak out. And the lab must be safe for you: nothing you do inside it may cost more than an hour to undo. Both come down to a handful of rules and one ritual. I keep them on a card by the switch because I have broken every one of them at least once.
By the end of this guide you will have the three rules that keep the lab away from the house. You will have the two habits that keep it cheap to break, and a way to rehearse real production changes in it. You will have a clear line between what belongs in the lab and what must never leave it. You will have a numbered snapshot-and-rollback procedure, and a safety checklist to print and fill in before every experiment.
In this guide
- Safe for the home: three isolation rules
- Safe for you: snapshots and a rebuild plan
- Rehearsing production changes in the lab
- What belongs in the lab
- What never touches production from the lab
- The snapshot-and-rollback procedure
- The lab safety checklist
- The lab config that reached a production switch
- The standing safety checklist
- Where to go next
Safe for the home: three isolation rules
The lab must not be able to affect anything in the house, and three rules make that true. They are cheap, they are boring, and every story of a lab breaking a home involves skipping one of them.
Rule one: the lab is its own subnet or VLAN
The lab lives on a range of addresses of its own, separated from the house by either a firewall or a VLAN on the managed switch. This is described in Building a Monitoring Homelab. The reason is not tidiness. The single most common way a lab breaks a home is a lab device handing out addresses. In that case, a firewall or a virtual machine runs a DHCP server that answers the television before the home router does. On a separate network it cannot hear the television ask.
Rule two: no default credentials facing the house
Every device in the lab gets a password and an SNMP community string that are not the ones it shipped with, before it is connected to anything. A community string is the password-like word an SNMP device expects before it answers, and the default ones are known to everybody. Reused equipment arrives with its old owner's settings, and new equipment arrives with the factory's. A lab full of defaults that can be reached from the home wireless network is an open door with a welcome mat.
Rule three: only the lab can reach the lab
The firewall between the lab and the house allows exactly one thing inbound. That is your own laptop's address reaching the monitoring server's web console and the hypervisor's management page. Nothing else from the house may reach the lab, and nothing in the lab may reach the house at all. Outbound to the internet for updates is your choice; I allow it, and turn it off during experiments that generate traffic. Written as rules, in the order a firewall evaluates them:
| Order | From | To | Action | Why |
|---|---|---|---|---|
| 1 | Admin laptop (one home address) | Monitoring console and hypervisor management, specific ports | Allow | The one door in |
| 2 | Home network (any) | Lab subnet (any) | Deny and log | Nothing else from the house |
| 3 | Lab subnet (any) | Home network (any) | Deny and log | Nothing from the lab reaches the house |
| 4 | Lab subnet (any) | Internet | Allow, or deny during traffic experiments | Updates only |
| 5 | Anything | Anything | Deny | Because a firewall ends with deny |
Verify the rules from both sides, once when you build the lab and again whenever the firewall or the switch is touched. From a lab virtual machine, nothing in the house should answer a ping; from a household device, the lab's monitoring console should not load. (Lab only, and only against addresses you own.)
# Lab only. Run from a lab VM: nothing in the house should answer. for ip in 192.168.1.1 192.168.1.20 192.168.1.50; do ping -c 1 -W 1 $ip > /dev/null 2>&1 && echo "$ip REACHABLE - isolation is broken" || echo "$ip blocked (good)" done
If any line says reachable, stop and fix the isolation before running anything else. Every other rule in this guide assumes this one holds.
Safe for you: snapshots and a rebuild plan
The lab is only useful if you are willing to break it, and you are only willing to break it if breaking it is cheap. Two habits keep it cheap.
The first is a snapshot before every experiment, without exception. A snapshot is the hypervisor's saved copy of a virtual machine's exact state at one moment, and reverting to it takes about a minute. It is not a backup: it lives on the same disk as the virtual machine, and a long chain of snapshots slows the machine down. So keep two or three per machine at most and delete the old ones. The procedure is later in this guide.
The second is a rebuild plan. It is a script or a checklist that takes the lab from bare hardware to "monitoring six things" in an evening. Reproducible monitoring setups covers doing this properly, with the monitoring configuration kept as files you can apply again. For a lab, even a plain numbered list in the notebook is enough, as long as you have followed it at least once from the top. A lab you can rebuild in an evening is a lab you will break without hesitation. A lab you cannot is one you will start protecting, and a protected lab teaches nothing.
Tip: rebuild the lab from the plan once a year on purpose, even if nothing is wrong with it. The rebuild is itself an exercise, and it is the only way to find out whether the plan still matches the lab.
Rehearsing production changes in the lab
A lab is not only for breaking things. It is also the place to do a real production change once, badly, before doing it once, well, at work. Three kinds of change rehearse particularly well.
A monitoring server upgrade is the first, and the most valuable. Install the version you run at work on the lab monitor, and add checks that resemble the ones at work. Then upgrade it exactly as you would in production and see what breaks. That might be a check type that changed, a report that vanished, or a database that needed longer than the maintenance window. Upgrading monitoring safely gives the production procedure; the lab is where you find out which of its steps you would have skipped.
A threshold philosophy change is the second. Suppose you want to move every disk warning from a fixed percentage to a rate, or add a confirmation count to every availability check. Apply the new rule to the lab, rerun the disk and port-flap exercises from Twelve Homelab Exercises, and compare the alerts before and after. A change that reads well in a policy document behaves differently on a graph, and the lab is where the difference is harmless.
A new dependency model is the third. Build the tree in the lab, take the core switch down, count the alerts, adjust, and count again. When the lab produces one alert for one fault, you have a model you can describe at work. You can use a before-and-after number instead of a diagram and a hope.
The rehearsal is only as honest as the resemblance. Match the version, match the check names where you can, and match the order of steps. A rehearsal on a different version with different checks proves that the lab can be upgraded, which was never in doubt.
What belongs in the lab
Three kinds of work belong in the lab, and the test is simple. If the honest answer to "what happens if this goes wrong?" is "I don't know", it belongs in the lab.
- Anything destructive. Filling disks, exhausting scopes, stopping databases, powering off switches. Every exercise in the previous guide. Anything whose purpose is to make something fail.
- Anything new. A collector you have never configured, a check type you have never used, a monitoring product you are evaluating, a version you have not run. New things fail in ways the documentation did not mention, and the lab is where the documentation gets corrected in your own handwriting.
- Anything you do not understand yet. Inherited thresholds, a dependency tree someone else built, an SNMP table with columns you cannot name. Reproduce it in the lab, poke it, and watch. Understanding is cheaper to buy here than anywhere else.
A fourth belongs here by elimination: anything you would rather not explain in a post-incident review. If the thought "I hope this works" crosses your mind with a production console open, close the console and open the lab's.
What never touches production from the lab
The lab is safe because it is separate, and the separation runs both ways. Four things must never make the trip from the lab to work, and each of them has, somewhere, at some time, made it.
- Lab credentials. The lab's admin password, its SNMP community, its service accounts. They are chosen for convenience and written on a card by the switch. Make them recognisably lab-shaped, such as a community string that begins with
lab-. That way, seeing one in a production config is an obvious mistake rather than a subtle one. - Lab configs pasted without review. A switch configuration, a check definition, an alert rule, copied from the lab into production because it "worked". It worked in a network with six devices, one user, and no consequences. Every line of it needs reading with production eyes before it goes anywhere, and the war story below is about what happens otherwise.
- "It worked at home" as an argument. A lab result is evidence that something can work. It is not evidence that it will work under real load, with real users, on the real version, next to the other things production runs. Present lab results as a hypothesis, and Carrying Homelab Lessons to Production covers how.
- Lab addresses and names. A check that points at 10.99.0.40, a dependency on a device called lab-sw1, a report filtered on a lab group. They look like configuration and behave like a fault that nobody can find, because the address does not exist at work and the check quietly reports nothing forever.
The common thread is that things leave the lab by copy and paste, and copy and paste does not review. Nothing leaves the lab except through a person reading it. That is the entire rule.
The snapshot-and-rollback procedure
This is the ritual around every experiment. It takes five minutes at the start and one at the end, and it is the reason the lab stays cheap to break. Do the steps in order; the numbering is the point.
- 1. Name the experiment. One short phrase, such as "exercise 8, stop the monitoring database" or "rehearse upgrade". Everything else is named after it.
- 2. List the virtual machines the experiment touches. Include the monitoring server whenever the experiment changes its configuration, even slightly.
- 3. Snapshot each one. Name every snapshot
pre-followed by the experiment name and the date. A snapshot called "snapshot 3" is a mystery within a week. - 4. Export the monitoring configuration if the software can. If it cannot, the snapshot of the monitoring server is the export; make sure that snapshot exists.
- 5. Log the start. Time, experiment name, snapshot names, what you expect to happen.
- 6. Run the experiment. Watch, note times, write the alert you wished you had received.
- 7. Decide: keep or roll back. Keep the result if the change is one you want in the lab. Roll back if the machine is damaged or the change was only to see what would happen.
- 8. To roll back: power off the virtual machine, revert to the
pre-snapshot, and power it on. Confirm the monitoring server is polling it again before you do anything else. - 9. To keep: run the lab normally for a day, then delete the
pre-snapshot so the chain stays short. - 10. Log the end. Time, what happened, keep or roll back, and the one thing you learned.
Steps five and ten are the ones people drop, and a script makes them cost nothing. Keep a one-line logger on the hypervisor host and use it at the start and end of every experiment. (Lab only; the log is for you.)
# Lab only. Usage: lab-log START "exercise 8" "pre-ex8-mon" or lab-log END "exercise 8" "rolled back" #!/bin/sh echo "$(date '+%F %T') $1 $2 $3" >> "$HOME/lab/experiments.log" tail -n 1 "$HOME/lab/experiments.log"
The log reads, after a month, like a very short diary of failures you caused on purpose. And it is worth more than any certificate on the wall, because every line is something you have actually seen.
The lab safety checklist
Print this, keep a stack by the switch, and fill one in before every experiment. It is deliberately short enough that you will actually do it. The top half is the standing state of the lab, the bottom half is this experiment.
LAB SAFETY CHECKLIST Date: ________ Experiment: ____________________ Standing state (verified when the lab was last changed) [ ] Lab subnet or VLAN: __________________ isolated from the home network [ ] Isolation verified from a lab VM (nothing in the house answers) [ ] Isolation verified from the house (lab console does not load) [ ] No device in the lab has a factory password or community string [ ] Only the admin laptop can reach the lab console and hypervisor [ ] Rebuild plan exists and was last followed from the top on: ______ This experiment [ ] Virtual machines involved: __________________________________ [ ] Snapshot taken for each, named pre-- [ ] Monitoring configuration exported, or its snapshot confirmed [ ] Undo for the fault is written beside the command [ ] Start logged at: ________ [ ] Traffic experiment? Outbound internet rule set to deny: yes / n.a. After [ ] Keep / roll back decided and logged at: ________ [ ] If rolled back: monitor polling the reverted VM again [ ] If kept: pre- snapshot deleted after one day of normal running [ ] One lesson written in the exercise table
The lab config that reached a production switch
An administrator at Bellweather Logistics kept his lab switch configuration in the same folder as the depot switch configurations, with the same naming scheme. During a rushed replacement of a failed depot switch he pasted what he believed was the depot's config. In fact it was the lab's. It had the same VLAN numbers, a read-only community string of lab-ro, and a management address of 10.99.0.2. It had a default route pointing at his living room. The switch came up, forwarded the depot's traffic perfectly, and vanished from monitoring. That was because the monitoring server was polling the old address with the old community. The switch was found eleven days later in an audit of devices that had stopped reporting. The review recorded that the configuration had been "reused".
Three of the rules above would each have caught it alone. Lab configurations in a folder called lab, and nowhere else. Credentials that are visibly lab-shaped, so that lab-ro in a production paste is a shout rather than a whisper. And nothing leaving the lab without a person reading every line. He now keeps the lab folder on a different machine, which is one rule more than necessary and, he says, about right.
The standing safety checklist
Run through this whenever the lab changes shape: a new device, a firewall change, a new VLAN, a rebuild. Every line is a yes or a no, and the lab is safe to break when every line is a yes.
- The lab has its own subnet or VLAN, and the address range is written in the parts and layout table.
- A ping from a lab virtual machine to three household addresses gets no reply.
- The monitoring console does not load from a household device other than the admin laptop.
- No lab device is running with a factory password or community string.
- Every lab credential is recognisably lab-shaped.
- The firewall's last rule is deny, and the deny rules log.
- No lab device can hand out addresses to the household.
- Every virtual machine has a clean snapshot, and no machine has more than three.
- The rebuild plan exists, and I have followed it from the top at least once.
- Lab configurations live in a folder named for the lab and nowhere else.
- The experiment log has an entry for the last thing I broke.
Where to go next
A lab is safe because nothing in it matters, and keeping it that way is three isolation rules, two habits, and a five-minute ritual. Isolate it, strip the defaults, let only yourself in. Snapshot before every experiment and keep a plan that rebuilds it in an evening. Rehearse real changes in it. Keep the destructive and the new and the not-yet-understood inside it. Let nothing out except through a person reading every line. Do that, and the lab stays the one network where "I wonder what happens if" is a sentence you can finish.
With the safety rules in place, Twelve Homelab Exercises gives you things to break. And Carrying Homelab Lessons to Production is the honest way to take what you learned to work. If you are still deciding whether to build one, Why a Homelab Teaches What Production Cannot is the argument. All of these are part of The Homelab as a Monitoring Practice Ground.
Put this into practice with SysPeep Monitor
A lab monitoring server that lives in a snapshot is one you can upgrade twice before you upgrade production once. The free trial at /download/ installs as a central Windows service on a single virtual machine. It runs on SQLite for a lab this size. It gives you thresholds, dependencies, and alert rules to rehearse changes against before they go anywhere near work.
Download Free TrialLearn About SysPeep