Home › Topics › Config Backup and Drift Detection Alongside Monitoring › Golden Configs

Golden Configs: Defining Normal and Catching Deviations


A daily diff tells you when a device's configuration changed. It cannot tell you whether the configuration was right to begin with. The access switch on the third floor has not changed in two years, and the diff is perfectly happy about that. The switch has also never sent a log to your log server, still answers to the default SNMP community, and allows management logins from the guest network. Nothing drifted. It was simply never correct, and no comparison against its own past will ever say so.


A golden configuration fixes that by giving the comparison a different reference point. Instead of "does today match yesterday?", the question becomes "does this device match what a device of its role is supposed to look like?" You write down, once, the lines every access switch must have and the lines none of them may have. You also write down the parts that legitimately differ from one device to the next. Then a script checks every backup against that standard and reports the deviations, by device and by rule.


This guide shows how to define a golden configuration per role and how to express required lines, forbidden lines, and per-device variables. It shows how to check backups against it, what the report looks like, and how to handle what it finds. That includes the case where the standard itself is what needs to change. This guide assumes a repository of configuration backups already exists; the sibling guide on automating config backups gets you there.


In this guide

  • What a golden config is (and is not)
  • Required lines, forbidden lines, and variables
  • Goldens by role
  • Writing your first golden
  • Checking backups against the golden
  • Reading the compliance report
  • Reporting by device and by rule
  • The remediation workflow
  • Keeping goldens with the backups
  • Where to go next

What a golden config is (and is not)


A golden config is a per-role standard: a description of what a correctly configured device of a particular kind looks like. "Access switch" is a role. So are "distribution switch", "branch router", "edge firewall", and "wireless controller". Every device in your inventory belongs to exactly one role, and every role has one golden. The golden is not a full configuration you could paste onto a device. It deliberately leaves out the hostname, the addresses, and the port assignments that make each device unique. It is the invariant part. That means the settings that should be identical on every member of the role. It also includes the settings that must never appear on any of them.


It helps to be clear about what it is not. It is not a backup; backups describe what is, goldens describe what should be. It is not a template for provisioning new devices, although once you have one, building a provisioning template from it is a short step. And it is not a security policy document. It is the small, mechanical, checkable subset of policy that can be expressed as "this line must be present" or "this line must be absent". That narrowness is the point. A rule a script can check is a rule that gets checked every night, which is more than can be said for most policy documents.


Required lines, forbidden lines, and variables


A golden has three parts. Required lines must be present on every device in the role. They include the logging destination, the time servers, the SNMP settings the monitoring system depends on, and the login banner. They include the management access list and the fact that it is applied to the management sessions. They also include the rule that only encrypted management protocols are accepted. Miss one and something downstream silently degrades — no logs from that device, clock drift that scrambles timestamps, a device the monitoring system cannot see.


Forbidden lines must be absent from every device in the role. These include the plaintext management protocol enabled, the default SNMP communities, and an unencrypted web interface. They also include unused services left on from the factory and a "permit anything" management access list. These are the lines an audit would flag and an attacker would enjoy. They are the ones most likely to survive on a device that was set up in a hurry and never revisited.


Variables are the parts that legitimately differ per device and must not trigger a finding. These include the hostname, the management address, the interface descriptions, and the VLAN assignments on each port. The golden lists them so the checker knows to ignore them, and so that a reader knows which differences between two access switches are expected. You do not need to enumerate every variable line; you need to be sure that no required or forbidden rule accidentally matches one.


A rule is best written as a pattern rather than an exact string. That way, ^ntp server 10\.0\.5\.21$ catches the line whether or not there is trailing whitespace. And ^snmp-server community (public|private) catches both default communities in one rule. Keep patterns tight: a rule that matches too broadly hides problems, and a rule that matches too narrowly cries wolf. The exact command syntax is whatever your switch's CLI uses; the shape of the rules is the same on every platform.


Goldens by role


The table below is a starting set. Your roles and rules will differ — adopt the shape, not the contents.


RoleRequired (examples)Forbidden (examples)Variables
Access switchLogging host; NTP servers; SNMP read-only community or SNMPv3 user for monitoring; login banner; management ACL applied to management sessions; encrypted management only; password encryption onDefault SNMP communities; plaintext management protocol; unencrypted web interface; unused discovery or file-transfer servicesHostname; management address; port descriptions and access VLANs
Distribution / core switchEverything in access, plus: spanning-tree root or priority settings for the role; the standard uplink template; a second logging hostAs access, plus: any port left in the default VLAN with no descriptionHostname; addresses; routed interface list
Branch routerLogging, NTP, SNMP, banner as above; the site-to-site VPN template; the standard outbound access list; the WAN interface monitoring settingsPlaintext management; any inbound access list that permits everything; default routing protocol passwordsHostname; WAN and LAN addresses; site identifier in descriptions
Edge firewallLogging to the central collector; NTP; management restricted to the admin network; the baseline deny rule at the end of every policy; the standard object naming schemeManagement exposed on the outside interface; a permit-any rule anywhere but the explicit exceptions list; default admin account enabledHostname; public addresses; site-specific policy blocks
Wireless controllerLogging, NTP, SNMP; the corporate and guest network definitions; the guest network isolated from internal networksOpen or legacy encryption on any corporate network; default management credentialsHostname; addresses; access point list

Writing your first golden


Do not start from a blank page. Start from the best-configured device you currently have in the role — the one a careful colleague set up most recently. Pull its backup from the repository. Read it top to bottom and sort each block into one of three piles. The piles are "every device in this role should have this", "this is specific to this device", and "this should not be here". The first pile becomes the required list, the third the forbidden list, and the second tells you what the variables are. Add the forbidden items you know about from experience even if this device does not have them. The point of that list is the devices you have not looked at.


Then run the checker against every device in the role and look at the findings before you agree on anything. You will learn two things immediately. One is which rules are widely broken (the standard was aspirational and nobody knew). The other is which rules are wrong (a required line that half the fleet lacks for a good reason you had forgotten). Adjust, agree the result with whoever else changes those devices, and store it. The first golden takes an afternoon; each subsequent role is faster because most of the required list is shared.


Keep the file format plain enough to edit without tooling. Three sections, one pattern per line, comments allowed:


# golden/access-switch.golden
# One pattern per line. Required must match at least one line of the
# device config; forbidden must match none. Variables are documentation.

[required]
^logging host 10\.0\.5\.20$
^ntp server 10\.0\.5\.21$
^snmp-server community \S+ RO$
^banner motd .*Authorized use only
^ip access-list standard MGMT-ACCESS$
^ access-class MGMT-ACCESS in$
^ transport input ssh$
^service password-encryption$

[forbidden]
^snmp-server community (public|private)
^ transport input (telnet|all)
^ip http server$
^no service password-encryption

[variables]
hostname
interface .* description
ip address
switchport access vlan

Checking backups against the golden


The checker is a short read-only script. For every device in the inventory it loads the golden for that device's role and reads the device's latest backup from the repository. It tests each rule. A required pattern that matches no line is a MISSING finding. A forbidden pattern that matches any line is a FORBIDDEN finding, reported with the offending line. The checker never touches a device. Run it nightly after the backup and diff steps, and on demand after a change.


#!/usr/bin/env python3
"""Compliance check: compare each device's latest backup with the golden
for its role. Read-only; reads files in the repository, prints findings."""

import csv, pathlib, re

REPO = pathlib.Path("/srv/config-backups")

def load_golden(role):
    rules = dict(required=[], forbidden=[])
    section = None
    for raw in (REPO / "golden" / (role + ".golden")).read_text().splitlines():
        line = raw.strip()
        if not line or line.startswith("#"):
            continue
        if line in ("[required]", "[forbidden]", "[variables]"):
            section = line[1:-1]
        elif section in rules:
            rules[section].append(re.compile(line))
    return rules

def check(config_text, rules):
    lines = config_text.splitlines()
    findings = []
    for rule in rules["required"]:
        if not any(rule.search(l) for l in lines):
            findings.append(("MISSING", rule.pattern))
    for rule in rules["forbidden"]:
        for l in lines:
            if rule.search(l):
                findings.append(("FORBIDDEN", l.strip()))
    return findings

def main():
    goldens = dict()
    with open(REPO / "inventory.csv") as f:
        for dev in csv.DictReader(f):
            role = dev["role"]
            if role not in goldens:
                goldens[role] = load_golden(role)
            cfg = (REPO / role / (dev["hostname"] + ".cfg")).read_text()
            findings = check(cfg, goldens[role])
            if not findings:
                print(dev["hostname"], role, "OK", "", sep="\t")
            for kind, detail in findings:
                print(dev["hostname"], role, kind, detail, sep="\t")

if __name__ == "__main__":
    main()

Two refinements are worth adding once the basic version works. First, exceptions: a small file that says "device X is allowed to break rule Y because of Z, until this date". A branch router that genuinely needs a non-standard access list should be an exception with a reason and an expiry. It should not be a permanent red line in the report. Second, severity: tag each rule as high or low, so that a missing banner and an enabled plaintext protocol do not look equally urgent. Both are a few extra lines in the golden format and the script.


Reading the compliance report


The raw output is tab-separated so that it can be sorted, filtered, and pasted into a spreadsheet. A readable version of a morning's run looks like this:


device          role             result      rule / offending line
sw-floor1-01    access-switch    OK
sw-floor2-01    access-switch    MISSING     ^ntp server 10\.0\.5\.21$
sw-floor2-01    access-switch    FORBIDDEN   snmp-server community public RO
sw-floor3-01    access-switch    OK
sw-core-01      core-switch      MISSING     ^ access-class MGMT-ACCESS in$
rtr-branch-04   branch-router    FORBIDDEN    transport input telnet
fw-edge-01      edge-firewall    OK

Devices checked: 7   Compliant: 4   With findings: 3   Findings: 4
By rule:  ntp server (1)  default community (1)  mgmt ACL applied (1)  telnet (1)

Read it the way you would read a diff: one finding at a time, asking what the line does and who would have wanted it. sw-floor2-01 has no time server and still uses the default community. It is almost certainly a switch that was replaced from a spare and configured from memory. The missing NTP line explains why its log timestamps have been odd. sw-core-01 has the management access list defined but not applied to the management sessions, which means it protects nothing. That is a one-line fix with a large security payoff. rtr-branch-04 still accepts the plaintext management protocol, probably since the day it was installed.


Reporting by device and by rule


The same findings support two views, and you need both. By device answers "what do I fix on this box?" It is the view for the person doing remediation: log into sw-floor2-01, add the time server, replace the community, done. Sort the raw output by hostname and you have it.


By rule answers "how widely is this rule broken?" and it is the more interesting view. A rule broken on one device is a mistake on that device. A rule broken on half the fleet is not a fleet of mistakes. It is a standard that was never rolled out, a golden that is wrong, or a rollout that stopped halfway. When the by-rule count for "second logging host" is fourteen out of sixteen distribution switches, the right response is not fourteen tickets but one conversation. Was this ever really the standard? If yes, plan the rollout; if no, remove the rule. The by-rule view is how the golden stays honest, because it makes the cost of an aspirational rule visible every morning.


Track both counts over time in the simplest way you can — a line per run in a text file is plenty. The numbers you want to watch are the count of non-compliant devices and the count of distinct rules with findings. Both should trend toward zero and stay there, with occasional spikes when a new rule is added. A spike that does not come back down is a rule nobody has agreed to.


Tip: the first full run against an inherited network will produce a long report, and that is fine. Do not try to fix it all. Fix the high-severity forbidden lines first (plaintext management, default credentials, exposed management interfaces). Then fix the required lines the monitoring and logging depend on. Let the rest drain over a few weeks. A report that shrinks every Monday is a report people keep reading.


The remediation workflow


A finding has exactly three possible resolutions, and the workflow is choosing between them deliberately rather than by default.


  • Fix the device, then re-check. The common case. Make the change by hand (or with whatever tooling you use for changes) and record it like any other change. Trigger a post-change backup and re-run the checker for that device. The finding disappears from the next report; the backup diff shows exactly the lines that changed; the change record explains why. Fix by hand deliberately at first. Automated remediation is a separate discipline with its own risks. The guide on Monitoring as Code is where the versioned, reviewable approach to that lives.
  • Grant an exception, with a reason and an expiry. When the device is right and the rule does not fit this case. The exception goes in the repository beside the golden, names the device, the rule, the reason, and a review date. Expired exceptions reappear as findings, which is how they get reconsidered instead of forgotten.
  • Update the golden. When the standard itself has moved: a new logging host, a retired time server, a rule that turned out to be wrong. Changing the golden is a change like any other. It gets a change record and a commit with a clear message. Then look at the by-rule view to see what the new rule flags. A golden that changes without a record is drift in the standard, which is a more confusing kind of drift than the usual sort.

The workflow closes the loop with the rest of this topic. A fix produces a change record and a diff; the diff matches the record; the next compliance run confirms the fix; the report shrinks. Suppose a change is made without that sequence — someone quietly fixes a finding and does not say so. The diff catches it as drift. The drift alert asks for an explanation, and the explanation is "compliance fix". That is a tolerable outcome, and the sibling guide on the change workflow makes it a rarer one.


Keeping goldens with the backups


Store the goldens, the exceptions, and the checker in the same repository as the backups. Put them in a golden/ folder beside the per-role backup folders. Three reasons. The goldens get the same history and the same access control as the backups. So "when did we make the second logging host mandatory?" is a version-control question with a precise answer. The checker can find everything it needs by relative path. So it runs identically on the backup host, on a laptop, and on whatever replaces the backup host. And a change to a golden shows up in the same nightly commit stream as changes to devices. That keeps the standard and the fleet in one timeline.


The one thing to keep out of the goldens is secrets. A required rule for the SNMP community should match the shape of the line (^snmp-server community \S+ RO$), not the community itself. Suppose a golden contains the real community, a real key, or a real password. That turns the repository's least sensitive folder into one of its most sensitive. That brings no benefit to the check.


Where to go next


A golden configuration turns "did this change?" into "is this right?". You define one per role — required lines, forbidden lines, and the variables that may differ. Start from your best existing device and adjust until the fleet and the standard agree on what normal is. A short read-only script checks every backup against its role's golden each night. It reports MISSING and FORBIDDEN findings by device (what to fix) and by rule (whether the standard is real). Findings are fixed, excepted with an expiry, or resolved by updating the golden. All of it lives in the same repository as the backups so that the standard has a history too.


The golden is most useful when the rest of the loop is in place. In that loop, diffing configs catches the changes between runs. And the change workflow makes sure fixes and golden updates arrive with their records. If you are wondering whether the whole effort is worth it, why configuration backups belong next to monitoring makes the case. The full set of guides is under Config Backup and Drift Detection Alongside Monitoring.


Put this into practice with SysPeep Monitor

Several of the required lines in a golden config exist for the monitoring system's sake. They include the SNMP settings it polls with, plus the logging and time servers that keep its timeline straight. SysPeep Monitor does not check configurations against a standard — that is the companion practice this guide describes. But it will show you the consequence of a missing line the same day. That could be a device that stopped answering SNMP or a metric that went quiet. SysPeep Monitor's alert rules can tell you before anyone notices. The free trial installs on a Windows machine in minutes.

Download Free TrialLearn About SysPeep