Home › Topics › Monitoring Scope: What to Watch (and Skip) › Anti-Hoarding

Data Hoarding: When Monitoring Everything Makes You Blind


Every hoarder has the same reason, and it always sounds sensible: we might need it. In monitoring the reasoning goes like this. Storage is cheap, the tool can collect it, and the one metric you skip will be the one you need during next month's mystery outage. So you tick everything. Every counter on every interface, every process on every server, every sensor the vendor exposes. Each decision feels prudent on its own. I know, because I made every one of them. The sum was a wall of four hundred graphs where the one line that mattered looked exactly like the three hundred and ninety-nine that never would.


This guide is built on one uncomfortable fact: collection is not insight. A metric nobody looks at answers no questions, catches no outages, and prevents nothing. It sits there costing you attention, polling capacity, and database space, and it feels like diligence the entire time. That is what makes hoarding hard to stop: it never once feels like a mistake. By the end of this guide you will be able to recognize the real costs of over-collection. You will be able to diagnose whether you are hoarding (there are measurable signs). You will be able to prune safely using your own monitoring system's usage data rather than gut feel. And you will be able to answer the "but what if we need it?" objection with something better than a shrug.


In this guide

  • The hoarder's logic, and why it feels right
  • What hoarding actually costs
  • The diagnostic: signs you are hoarding
  • The pruning method: data-driven, not vibes
  • A worked example: one estate's first prune
  • The “just in case” rebuttal script
  • What healthy collection looks like
  • The quarterly pruning ritual
  • Where to go next

The hoarder's logic, and why it feels right


Nobody sets out to hoard. It happens one reasonable decision at a time. You enable a template that adds forty counters per server, because unticking boxes takes effort and who knows which ones matter. You keep collecting metrics for the app you retired, because deleting feels risky. You add a dashboard for last quarter's incident, and never remove it after the incident stops being interesting. Adding a check has a visible upside (coverage!) and an invisible cost; removing one has a visible risk (what if?) and an invisible benefit. Every incentive points the same direction: accumulate.


The flaw in "we might need it" is that it treats collection as free and consultation as automatic. It is as if data, once stored, will raise its hand at the right moment. It will not. During the mystery outage, nobody scrolls through four hundred graphs. They look at the five they trust and ignore the rest, exactly as they do every other day. Data you never look at does not help you in a crisis. It helps you feel prepared, which is a different thing, and considerably cheaper to achieve.


What hoarding actually costs


"Storage is cheap" is the hoarder's entire cost model, and it is wrong on all four counts that matter.


Attention

This is the expensive one. Every graph on a dashboard competes for eyes, and human attention does not scale the way disk does. A dashboard with twelve charts gets read; a dashboard with four hundred gets skimmed, then avoided, then forgotten. The important anomaly — the disk trending toward full, the error counter starting to climb — hides in plain sight among hundreds of flat lines. Hiding a signal among noise is functionally the same as not collecting it. Dashboard sprawl is how a monitoring system becomes a place data goes to be ignored. Think of a control room. A wall with ten gauges gets watched. A wall with a thousand gets wallpapered over. And your team's eyes are the scarcest resource your monitoring spends.


Signal

Checks breed alerts. Enable enough metrics with default thresholds and some of them will fire — not because anything is wrong, but because defaults are wrong for your network somewhere. Each pointless alert trains everyone to trust alerts a little less. That erosion is precisely the mechanism behind the "monitor everything" myth dismantled in monitoring myths. A hoarded estate does not just hide signals on dashboards; it actively manufactures noise that buries the alerts you genuinely need.


Performance and storage

Every metric is polled, transmitted, written, indexed, retained, and backed up — forever, on a schedule, whether anyone cares or not. Multiply thousands of metrics by polls every minute by years of retention. The "cheap storage" becomes a bloated database, sluggish dashboards, reports that time out, and polling cycles that lag behind their own schedule. There is a grim irony available here: a monitoring server so overloaded with collection that it fails to notice problems promptly — the monitoring that monitored itself into the ground. Your monitoring system is itself a system with capacity limits, and hoarding spends those limits on nothing.


Tollgate Foods had sixty switches, forty-eight ports on each, and a discovery template that enabled forty-eight counters per port. That came to a little over a hundred and thirty thousand series polled every minute. Of those, the number anyone had ever put on a graph was four, all on the two WAN uplinks. The database's nightly maintenance job grew until it ran into the morning. The morning polls lagged behind their own schedule by eleven minutes. The WAN outage that eventually happened was reported by a user before the monitoring noticed it. The review described the collection as "thorough". The per-port counters went to rollups in one afternoon. The lag went to zero, and nobody has asked for the other hundred and thirty thousand since.


Maintenance

Every check is a small forever-commitment. It will one day misfire, and someone will spend twenty minutes deciding whether that matters. It will reference a server that got renamed, and someone will fix it or silence it. It will appear in an audit, a migration, or a tool upgrade. Someone will have to answer "do we still need this?" without knowing who created it or why. Multiply that recurring pinch of confusion by a thousand hoarded checks and you have a real, permanent tax on the team. That tax is paid in the currency of decisions, which is the same currency you troubleshoot with.


The diagnostic: signs you are hoarding


Hoarding is measurable. Look for these four signs in your own environment — each comes with a way to check for it honestly.


  • Checks nobody can explain. Pick ten checks at random and ask the team what each one is for and what you would do if it fired. To find them systematically: sort checks by creation date and look at the oldest — the ones that predate everyone's memory are prime suspects.
  • Dashboards nobody opens. If your tool records dashboard views, sort by last-viewed and read the bottom of the list. No view data? Rename a suspect dashboard with a prefix like "ARCHIVE —" and see whether anyone mentions it within a month. Silence is your answer.
  • Metrics with zero queries against them. Data collected continuously since forever that no graph displays, no report includes, and no alert threshold watches. Many tools can list metrics not referenced by any dashboard or alert rule — that list is your hoard, itemized.
  • Alerts that get acknowledged and never acted on. Pull the last few months of alert history and ask of each rule: did anyone ever do anything because of this? A rule whose every firing was closed without action is not earning its place.

The fourth sign is the one people argue about, so here is the plain picture. An alert rule that is always acknowledged and never acted on is a doorbell nobody answers. If all four signs come up clean, congratulations — skip to the healthy-collection section and carry on. Most estates that have grown for a few years without deliberate pruning will find plenty.


The pruning method: data-driven, not vibes


The wrong way to prune is by argument: people defend checks emotionally ("I set that up during the big outage!") and debates go nowhere. The right way is to let usage data decide. Your monitoring system is itself a system that generates metadata. This is data about the data, such as which dashboards were opened and which metrics an alert rule reads. That metadata tells you what is actually earning its keep. Nobody can argue with a view count. (They will try.)


Step 1: the usage audit

Gather three lists from your tool. The first is which dashboards were viewed in the last quarter (and how often). The second is which metrics are referenced by at least one dashboard, report, or alert rule. The third is which alert rules led to a human action. These three lists split your entire estate into "demonstrably used" and "candidate for pruning." You are no longer asking "might we need this?" — a question that cannot be answered — but "did anyone use this?", a question your own database answers precisely. If your tool tracks none of this metadata, use proxies. Alert history exports, the dashboard-rename trick from the diagnostic section, and a simple team poll of "name every dashboard you opened this month" get you most of the way.


Step 2: the keep / aggregate / drop triage

Walk the candidate list and give every metric one of three fates. "Aggregate" is the middle path hoarders need: many metrics deserve to exist as a coarse long-term trend but not at full one-minute resolution forever. A rollup (sometimes called downsampling) is one summary number — an hourly or daily average, say — that replaces many raw samples as they age. Most tools do this automatically, and the triage decides which metrics deserve full resolution at all. Expect to find at least one truth being collected four different ways, and keep one of them.


FateWhen it appliesExamples
Keep at full resolutionWatched by an alert, consulted during incidents, or on a dashboard people actually openTier 1 server disk space, WAN interface traffic, DNS response time
Keep as rollup onlyUseful as a trend over months, never consulted minute-by-minutePer-interface traffic on access switches, office temperature, printer page counts
Stop collectingNo query, no alert, no plausible question it answers — or the device is gonePer-process memory on the retired app server, sensors on decommissioned gear, a metric that duplicates one already kept

Step 3: the safety nets

Pruning feels dangerous, so build the nets that make it reversible and the fear evaporates. First, archive before deleting: disable collection but keep the historical data through a grace period (a quarter is plenty). If nobody misses it, delete the history in the next ritual. Second, leave a re-enable note wherever the check lived. That way, the person who wants it back next year restores it in two minutes instead of rebuilding it from scratch:


# Pruned 42 per-process counters on APP-03 after usage audit
# Reason: zero dashboard/alert references for 2+ quarters
# History archived until next quarterly review; template saved as
#   "app03-process-counters" — re-enable takes ~2 minutes
# Questions: ask the monitoring owner (see runbook)

Third, prune in reversible batches: one device class or one dashboard at a time, then wait a week. If something screams, you know exactly which batch to revert. Nobody has ever regretted pruning slowly; plenty of people have regretted the heroic Saturday purge.


A worked example: one estate's first prune


Here is how the method plays out on a fictional but painfully typical estate. It has sixty servers, forty network devices, and one monitoring server that has been "temporarily" slow for a year. It has a check inventory nobody has reviewed since it was built. The admin runs the usage audit and finds a few thousand metrics being collected, of which only a few hundred are referenced by any dashboard, report, or alert rule. Of thirty-odd dashboards, six were opened in the last quarter. Of the alert rules that fired, roughly a third produced any human action. The noisiest single rule — a default CPU threshold on a batch server that legitimately runs hot every night — accounts for an absurd share of all alert volume by itself.


The triage takes one afternoon. Everything referenced or Tier 1 stays at full resolution. Per-port traffic on access switches, environmental sensors, and the long tail of "nice trend, never urgent" metrics drop to rollups. Collection stops entirely for two retired servers still being polled into the void. It stops for the per-process counters someone enabled fleet-wide during a single investigation years ago, and every metric belonging to devices that no longer exist. Each batch gets a re-enable note; history goes to the archive shelf for a quarter.


The results are unglamorous and immediate. The database stops growing like a teenager. Dashboards load fast enough that people start opening them again. The nightly CPU alert gets a schedule-aware threshold and goes quiet. And — this is the part worth writing down — over the following months, nobody asks for any of the pruned data back. The "we might need it" fear, tested against reality, loses. It almost always does.


The “just in case” rebuttal script


Every pruning effort meets the same three objections. Each contains a legitimate need wrapped around a bad conclusion, and each has a cheaper, better answer than "collect everything forever."


“We need it for capacity planning”

Capacity planning is real, and it needs long-term trends, not per-minute detail: nobody plans next year's storage purchase from sixty samples an hour. The cheaper alternative is targeted retention. Keep the dozen capacity-relevant metrics (disk growth, link utilization, memory trend) as rollups for years, and let the rest age out. You get better planning data than the hoard gave you, because now the trends are findable.


“We need it for audits”

Audits need specific, named evidence — who logged in, what changed, whether backups ran — kept intact for a defined period. That is a job for deliberate audit logging, designed against the actual requirement, with proper retention and protection. A heap of performance counters is not an audit trail; it fails the auditor's needs while bloating your database. I once watched a year of CPU graphs being offered to an auditor as evidence. She was polite about it. If audit requirements exist, read them and build for them; "keep everything" satisfies neither the auditor nor you.


“That one time, the obscure metric saved us”

Sometimes true! Deep diagnostic data genuinely shines during a hard investigation. But the lesson is not "collect everything always" — it is collect deeply on demand. When a problem is under investigation, turn on the detailed counters, the debug logging, the packet capture, for that system, for that week. You get richer data than the always-on hoard would have held, aimed exactly where the problem is, and it turns itself off when the investigation ends. The always-on version of that one lucky save costs you every other week of the year.


Tip: when someone resists a specific pruning, do not argue — delegate the cost. "Happy to keep it: whose dashboard should it be on, and what alert threshold do you want?" If they will not claim it, they have answered the question. Data worth keeping always has a keeper.


What healthy collection looks like


The alternative to hoarding is not minimalism for its own sake; it is purposeful collection. The standard is the question test. Every metric you collect at full resolution should trace to a question someone actually asks — "is the ERP up?", "will this disk fill this quarter?", "is the WAN link degrading?". If you cannot name the question, the metric is decoration. This is the same discipline, applied at the metric level, that monitoring goals and scope applies to the whole program. Monitoring exists to answer questions and trigger actions, and everything it collects should serve one or the other.


A healthy estate, in practice, has a small set of dashboards that people open daily and could describe from memory. It has alert rules whose firings almost always produce action. It has deep, full-resolution data on the systems where it earns its place (your Tier 1 estate — see criticality tiers). It has rollups for the long tail and a documented not-to-monitor list for the things deliberately left silent. Notice what this buys you during a real incident. The graph you need is one of forty, not one of four thousand, and you already know its name.


The quarterly pruning ritual


Hoarding is not a one-time cleanup problem; it is a gas that refills any container. The permanent fix is a small recurring ritual — an hour a quarter keeps the estate lean forever. The checklist:


  • Run the usage audit: pull dashboard views, unreferenced metrics, and no-action alert rules for the quarter.
  • Delete last quarter's archived prunings that nobody asked about — the grace period has done its job.
  • Triage this quarter's candidates: keep, roll up, or stop — one batch, using the decision table above.
  • Retire one dashboard from the bottom of the view list (rename to ARCHIVE first if you are nervous).
  • Review the noisiest alert rule of the quarter: fix its threshold, or admit it is not actionable and remove it.
  • Write the re-enable notes for everything pruned, and book next quarter's hour before leaving the room.

The first ritual is the big one; after that it is genuinely an hour. Teams that adopt it notice the same pattern: nobody ever asks for the pruned data back — but everyone notices the dashboards got readable. If you run structured coverage reviews already, the ritual slots neatly alongside them: one meeting asks "what are we missing?", this one asks "what are we carrying?"


Where to go next


The hoarder's instinct never fully goes away. "We might need it" will whisper at you during every new deployment, and it will still sound sensible, because it always does. The counterweight is scope discipline. Rank what matters with criticality tiers. Write down what earns silence in a not-to-monitor list. And give every new system a deliberate scope decision at intake so the hoard never re-forms. The rest of Monitoring Scope: What to Watch (and Skip) treats all of these as one connected habit. Collect on purpose, prune on schedule, and let every metric answer a question. I still have a cupboard of counters I have not got round to. The difference is that I now know it is a cupboard.


Put this into practice with SysPeep Monitor

A lean monitoring estate needs a tool that helps you stay lean. SysPeep Monitor's thresholds, dependencies, and alert rules are designed to keep alert noise down rather than manufacture it. Its dashboards live in a single web console. So the graphs you keep are the graphs you actually see. Try it free on any Windows machine and start with the checks that answer real questions.

Download Free TrialLearn About SysPeep