Home › Topics › Polling Intervals & Data Granularity › Retention Policy

A Metric Retention Policy for Small Shops


"How long do we keep monitoring data, and at what detail?" is a question with two possible kinds of answer. The first kind is discovered. Mid-investigation, you zoom into the week that matters and find it has already been averaged into mush. That is because the answer was whatever your tool's defaults decided. The second kind is written. One page says what you keep, at what resolution, for how long, and why. The answer is decided once, on a calm afternoon, and defended in one sentence forever after.


This guide is about writing that page. It is the capstone of the granularity story. You already know (or can skim from the sibling guides) how intervals are chosen and how data ages down a rollup ladder. The policy is where those mechanics become commitments. And it is deliberately sized for small shops. It includes four metric classes, one ladder table, a short exceptions list, and a storage projection you can compute on a napkin. No committee required; the whole thing fits in an afternoon.


By the end you will have three things. You will have the questions that generate the right retention numbers for your shop. You will have a filled-in example for a 100-device network you can adapt, and a copy-ready template. The next time someone asks "can we see the graphs from that outage in March?", the answer will be whatever your policy chose it to be — on purpose.


In this guide

  • Why write it down at all
  • The three questions that size retention
  • Think in metric classes, not metrics
  • The per-class ladder: small-shop defaults
  • The exceptions list
  • What goes in the policy document
  • Enforcement: from paper to tool settings
  • When to revisit the policy
  • A worked example: the 100-device shop
  • The policy-in-a-page template

Why write it down at all


Unwritten retention is not "no policy" — it is the vendor's policy, adopted sight unseen. Every monitoring tool ships with default retention, sized to make the product look tidy on an average install. Ad-hoc shops run those defaults until the day they hurt. That might be the intermittent fault whose early occurrences have aged into hourly blur. Or it might be the annual report that needs last February at better resolution than last February still has. Retention mistakes are uniquely unfixable. You can lengthen a polling interval tomorrow, but you cannot retroactively un-delete the samples a purge job removed last month. A written policy is simply the act of making those irreversible decisions before the deadline, instead of at it.


There are two quieter payoffs. First, pre-made decisions are cheap to defend. When disk pressure or a curious manager asks "why are we storing all this?", the answer is a page, not a scramble. Second, the audit bonus: if you report availability to anyone — management, customers, an auditor — your historical data is evidence. Evidence has retention requirements of its own. A policy that says "availability history: three years, with outage events preserved" is the difference between substantiating last year's uptime claim and shrugging at it. The Uptime & Availability Monitoring pillar covers what those reports promise; the retention policy is what keeps the receipts.


The three questions that size retention


For each kind of data you keep, the right numbers fall out of three questions — each mapping to a different rung of the rollup ladder.


  • 1. The operational question: how far back do investigations reach? When trouble is reported late, correlated slowly, or recurs before it is understood, how old is the oldest data you realistically zoom into at full detail? For most small shops honest answers cluster around two to six weeks. This sets your raw window — the span kept at native polling resolution.
  • 2. The analytical question: how far back do trends need to reach? Capacity planning, seasonal comparisons ("is this September like last September?"), and baselines want at least a full year, ideally two or three. But they want it at summary resolution, since these are questions about typical behavior. This sets your summary rungs. The Baselining pillar is the biggest consumer of this rung, and hourly summaries serve it beautifully.
  • 3. The evidence question: what must you be able to prove, and for how long? SLA substantiation, uptime reporting, dispute ammunition with providers. This sets your final horizon — often longer than either need above, but satisfied by coarse data plus preserved outage events.

Then reality gets a vote: the disk budget. Ask the three questions first and the budget second, because at small-shop scale the answers almost always fit. As the worked example below shows, a generous policy for a hundred devices lands in single-digit gigabytes. The budget question is rarely "can we afford it?" and usually "did we decide it, and is anything watching the disk?"


A fair objection at this point: if the totals are so small, why not just keep everything, raw, forever? Two honest reasons. First, "forever" compounds. Raw-everything grows without bound and crosses backup windows and database performance thresholds years later. It hands the problem to whoever inherits the system at its worst moment. Second, and less obviously, hoarded raw data does not actually serve the long-horizon questions better. Capacity planning across three years is easier against daily summaries than against eight hundred million raw samples. Every long-range query pays the wading cost. The ladder is not only a savings scheme; it is pre-computed answers at the resolutions the questions come in. Keep raw generously where investigations live, and let everything else age with dignity.


Think in metric classes, not metrics


A policy that names individual metrics is a maintenance burden that dies young — nobody updates the page every time a check is added. Instead, sort everything you collect into a handful of classes with shared retention needs, and let every current and future metric inherit its class's ladder. Four classes cover a small shop:


ClassWhat it containsIts retention personality
AvailabilityUp/down checks: ping, service, interface statusNeeds long history as percentages and outage events; raw detail matters for weeks, evidence for years
PerformanceUtilization, CPU, memory, latency, error ratesThe investigation workhorse: raw detail earns its keep; summaries carry baselines and seasonal trends
CapacityDisk fill, scope usage, repository sizes, growth countersSlow-moving; daily points for years beat minute detail for anything; longest horizons of all
EventsTraps, syslog, state-change and alert historyNot samples but records; kept verbatim for a window, then thinned to summaries or counts

If a metric does not obviously fit a class, it usually means you have not decided why you collect it — which is a scoping question worth answering before a retention one.


The per-class ladder: small-shop defaults


Here is a defensible default ladder for each class — the numbers to start from and bend to your answers to the three questions. (The ladder mechanics themselves — rollups, what survives each rung, and why averages-only rollups are a trap — are covered in rollups and retention. This table assumes you keep max alongside average wherever your tool allows.)


ClassRaw windowSummary rungsFinal horizon
AvailabilityNative interval, 30 daysHourly % up, 1 yearDaily % + outage events, 3 years
PerformanceNative interval, 30 daysHourly avg+max, 1 yearDaily avg+max, 3 years
CapacityNative interval, 60 daysDaily avg+max, 5 years— (dailies are the horizon)
EventsFull detail, 90 daysDaily counts by type, 1 year—

The reasoning, row by row. Availability: thirty raw days covers late-reported outages and monthly reviews. Hourly percentages for a year feed availability reporting. Three years of dailies plus the preserved outage event list is what lets you substantiate "99.7% last year" and answer "which afternoon was that?" Percentages alone cannot. Performance: thirty raw days is the investigation window — wide enough that "it's been slow for a couple of weeks" still has full-detail evidence. A year of hourly avg+max carries baselines and the seasonal comparisons. Three years of dailies shows the long arc for planning. Capacity: keep sixty raw days, because growth investigations ("when did this disk start filling faster?") look back further than outage ones. And the class is cheap at its slow polling cadence. Then go straight to five years of dailies, because capacity questions are years-scale. A daily max is all the resolution a fill curve ever needs. Events: ninety days verbatim covers investigations and postmortems. After that, daily counts by type preserve the trend ("flap storms are increasing") without the bulk.


The exceptions list


Classes do the bulk work; a short exceptions list handles the handful of metrics that earn more. A good small-shop example is the Tier-1 latency checks — the internet link, the path to the main application. These might keep raw data for 90 days instead of 30. That is because latency disputes and slow-burn degradations are exactly where fine history pays. Another example is the WAN links, keeping a full year at 5-minute resolution. They are the metrics you argue about with providers and size upgrades against. And a year of fine data on a handful of interfaces costs almost nothing. The discipline is the list's size: every exception is a standing storage commitment and a bit of configuration drift. So each entry must name its reason. Five entries is a healthy exceptions list; twenty-five is a second policy wearing a trench coat.


Tip: resist the symmetrical temptation — exceptions that keep less. Trimming the lab gear's retention below its class default saves pennies of disk and costs you uniformity, which is worth more. Let exceptions only ever add.


What goes in the policy document


The document itself is seven short parts — compact enough that filling it in is an afternoon, most of which is thinking, not typing:


  • Scope: which monitoring system(s) and sites the policy covers.
  • Classes: your class definitions with a couple of examples each, so future checks can be sorted at a glance.
  • Ladder table: the per-class raw window, summary rungs, and final horizon — the table above, with your numbers.
  • Exceptions: metric, what extra it keeps, and why. Reason required.
  • Storage projection: the arithmetic total, plus the alert threshold guarding it.
  • Owner: a name. Unowned policies are trivia.
  • Review date: when this page gets reread — the triggers are the section after next.

Enforcement: from paper to tool settings


A policy the tool does not implement is a wish. Translation is tool-specific, but retention settings live in one of a few usual places depending on the genre. Tools built on round-robin archives fix each metric's rungs when the metric is created. That means ladder changes may only apply to new metrics, or need a migration step — check before assuming. Database-backed tools expose retention windows and run scheduled rollup-and-purge jobs. Time-series backends take retention durations per resolution or per bucket. Find where your tool says it, and set it per class (device groups and check templates are the usual levers). And record the mapping in the policy so the next admin can see intent next to implementation.


Enforcement also has an onboarding half: every new check that gets added from now on must land in a class. If your tool applies retention through templates or groups, this happens for free — sorting the check into the right template is the policy decision. If retention is set per check, add one line to your device-onboarding routine: "assign retention class." The failure mode you are preventing is a quiet one. Two years of new checks accumulate on tool defaults while the policy page confidently describes a system that no longer exists.


Then verify, because between intent and disk sit defaults, per-check overrides, and the occasional bug. The spot-check is simple: pull up a day from ten months ago on a performance metric and look at the point spacing. Your policy says hourly, so thirty points for that day means something is off (and finding out now beats finding out mid-postmortem). Do the same at each rung boundary once, then quarterly. Finally, set the guard that protects the whole scheme: a disk-headroom alert on the monitoring server's data volume. It should warn early enough to act calmly. When the metrics disk fills, many tools start shedding exactly the history your policy promised to keep. In that case, some stop collecting altogether. The policy's worst enemy is a full disk discovering your priorities for you.


When to revisit the policy


Retention policies age well — the questions change slowly — so the review burden is light. Reread the page on four triggers. Read it annually, on the review date the document names (usually a five-minute confirmation that nothing changed). Read it on tool change, since a migration resets every retention assumption and old data's fate must be decided explicitly. Read it on disk pressure, where the policy tells you what to shrink last instead of letting urgency decide. And read it on new compliance or reporting asks, because a new SLA, customer, or auditor can silently extend your evidence horizons. Between triggers, leave it alone. A policy that needs monthly tending was written at the wrong altitude.


A worked example: the 100-device shop


Here is the whole policy computed end to end for a typical 100-device network. It has roughly 200 availability checks (a mix of 1- and 5-minute intervals). It has 1,200 performance series at 5 minutes, 400 capacity series at 15 minutes, and a modest event stream. Using the default ladder above and the storage arithmetic from the rollups guide (samples per day × window × a few tens of bytes each, summaries similarly):


ClassVolumeLadder appliedSteady-state estimate
Availability200 checks, ~100k samples/day30 d raw + 1 y hourly + 3 y daily~0.2 GB
Performance1,200 series, ~350k samples/day30 d raw + 1 y hourly avg+max + 3 y daily~1.3 GB
Capacity400 series, ~38k samples/day60 d raw + 5 y daily avg+max~0.2 GB
EventsTraps + syslog + alert history90 d full + 1 y daily counts~1 GB budgeted
Exceptions6 WAN/latency series, fine-grained for 1 y90 d raw / 1 y at 5-min~0.1 GB
TotalAll classesSteady state, defaults + exceptions~3 GB

To see where a line comes from, walk the biggest one. Performance: 1,200 series at 5-minute polling is 288 samples per series per day — about 346,000 samples daily. A 30-day raw window holds around 10.4 million samples; at a few tens of bytes each, call it 0.3–0.4 GB. The year of hourly summaries is 1,200 series × 24 points × 365 days ≈ 10.5 million points, kept as avg+max. That is similar again, with room to spare inside the 1.3 GB estimate once indexes and the three years of dailies are added. Every other row is the same three multiplications with smaller numbers.


Three gigabytes, steady state, generous everywhere it matters — on hardware where the coffee-fund USB stick holds ten times that. The estimates are deliberately rough (per-sample overhead varies by backend; double everything for comfort and it still fits). And that is precisely the lesson: at small-shop scale, retention is not a storage problem. It is a decision problem, and this shop has now made its decisions. Any outage from the last month is investigable in full detail. Any capacity question has five years of runway. Last year's uptime numbers can be proven. And the disk-headroom alert stands guard at 20% free.


The policy-in-a-page template


Copy, fill, done. Everything in this guide compresses to this page:


METRIC RETENTION POLICY — [site/org]     Owner: [name]   Next review: [date]

SCOPE
  Covers all data collected by [monitoring system] for [sites/networks].

CLASSES
  Availability : up/down checks (ping, service, interface status)
  Performance  : utilization, CPU, memory, latency, error rates
  Capacity     : disk fill, scope usage, growth metrics
  Events       : traps, syslog, alert and state-change history

LADDER
  Availability : raw [30 d] -> hourly % up [1 y] -> daily % + outage events [3 y]
  Performance  : raw [30 d] -> hourly avg+max [1 y] -> daily avg+max [3 y]
  Capacity     : raw [60 d] -> daily avg+max [5 y]
  Events       : full detail [90 d] -> daily counts by type [1 y]

EXCEPTIONS (each entry: metric — extra retention — reason)
  1. [WAN links — 1 y at 5-min — provider disputes, upgrade sizing]
  2. [Tier-1 latency — 90 d raw — slow-burn degradation forensics]

STORAGE
  Projected steady state: [~x GB]. Data volume alert at [20]% free.

VERIFICATION
  Quarterly: spot-check a [10-month-old] day renders at [hourly] resolution.

Print it, commit it to the wiki, and let it be boring. Boring is what a good retention policy feels like right up until the day it quietly saves an investigation. For the mechanics underneath every line of it, the Polling Intervals & Data Granularity pillar has the full set: choosing the intervals that feed the ladder, what they cost, what they miss, and how the data ages.


Put this into practice with SysPeep Monitor

A retention policy needs a data store you can reason about. SysPeep Monitor keeps its metric history in your choice of SQLite, MySQL/MariaDB or SQL Server. Its reports and dashboards make spot-checking old data's resolution a two-minute job. The free trial installs on any Windows machine in minutes.

Download Free TrialLearn About SysPeep