Home › Topics › Retention and Granularity: How Long to Keep What › Retention Tiers

Designing Retention Tiers: Raw, Hourly, Daily, and Gone


Somewhere in your monitoring product there is a page with three or four boxes on it. Keep raw samples for this many days, keep hourly for this many, keep daily for this many. The boxes look harmless. They are the most consequential settings in the system. They decide what you will be able to see about last month when last month turns out to have mattered. And they decide it in advance, quietly, every night.


Those boxes are the retention tiers. Designing them well is not hard. But it does need three things most people skip. You need to know where each boundary comes from. You need to know which numbers must survive when raw samples become a summary. And you need to know how to change the settings on a live system without deleting the thing you were trying to keep.


By the end of this guide you will be able to define a tier. You will be able to build the usual ladder from the needs of the people described in the three stakeholders guide. You will be able to choose the fields that survive each rollup. You will know how to align the tiers with the graphs people actually open. And you will be able to carry out a tier change with a procedure instead of a hope. There is a tier table at the end you can fill in and adopt.


In this guide

  • What a tier is
  • The usual ladder, and where each boundary comes from
  • What survives a rollup: the average is not enough
  • What should never be rolled up
  • Aligning tiers with polling and with graphs
  • The tier table template
  • Changing tiers safely on a live system
  • Products differ: check the documentation
  • War story, checklist, where to go next

What a tier is


A tier is a resolution paired with an age. Resolution is how far apart the data points are; age is how old a sample must be before it moves to that resolution. "Hourly, from four weeks old" is a tier. So is "daily, from six months old." The set of tiers, from finest to coarsest, is the ladder. And the last rung of every ladder is "gone", which is a tier too, with a resolution of nothing.


Data moves down the ladder by rollup. A scheduled job reads the raw samples that have crossed an age boundary. It writes one summary row per hour (or per day), then deletes the samples it summarised. The rollups and retention guide explains that mechanism and what a rolled-up graph looks like; I will take it as read. This guide is about deciding what the rungs should be and what each summary row should carry.


The ladder has a shape worth naming: each rung is cheaper than the one above it, and each rung answers fewer questions. Raw answers "what happened at 14:35?" Hourly answers "what was the afternoon like?" Daily answers "was Tuesday busier than Monday?" Gone answers nothing, at zero cost. Designing tiers is choosing, for each age of data, which questions you are still willing to pay to answer.


The usual ladder, and where each boundary comes from


The ladder nearly everyone ends up with is raw for weeks, hourly for months, daily for years, then gone. The numbers vary; the shape does not. What matters is that each boundary is set by a reason, and the reasons come from the stakeholders.


The raw boundary comes from forensics. Ask how long after an incident someone might still need to see individual samples. And add the time it takes for the incident to reach you in the first place. Two weeks is the least I would accept; four is comfortable; six covers the user who mentions it at the monthly meeting. This is the expensive boundary, and the storage math guide shows exactly how expensive. Every extra week costs a full week of raw growth for the whole estate.


The hourly boundary comes from the graphs. Hourly data is what a month-long graph is drawn from. So keep it for as long as people compare months: a quarter at least, six months if they compare "this quarter with last." Hourly is also what a threshold review looks at. The review asks whether an alert threshold has been set sensibly. So make sure the review period fits inside the hourly tier.


The daily boundary comes from the capacity planner and the auditor. The planner needs a year at minimum to see a seasonal pattern once, and two years to see it twice. That is when it becomes believable. The auditor's period is set by whatever you answer to. Take the longer of the two. Daily rows are so cheap that this is the one boundary where generosity costs almost nothing.


The "gone" boundary is the one people forget to set, and a database with no last rung grows forever. Set it, even if it is far out. Before anything reaches it, the monthly summary should already have been written to the archive, so that "gone from the database" never means "gone".


What survives a rollup: the average is not enough


A rollup row that carries only the average destroys the one thing forensics and capacity planning both need: the peak. An average is a diplomat. It wants every sample to get along. It will happily report a quiet hour even if a link sat at 100 percent for ten minutes. Watch what happens to twelve five-minute samples when eleven of them are calm and one is not.


samples (percent busy):  20 20 20 20 20 20 98 20 20 20 20 20
average               =  (11 x 20 + 98) / 12  =  318 / 12  =  26.5
maximum               =  98
minimum               =  20
count                 =  12

The average says 26.5 percent, which is a number nobody would look at twice. The maximum says 98, which is the number that explains the tickets. Keep the maximum. Not "on the important counters": on all of them, because you do not know in advance which counter the incident will be about. Keep the minimum too. It is the number that shows a counter dropped to zero. That is how a stopped service or a dead interface looks in a summary.


Keep the count as well. The count is how many raw samples went into the summary row, and it is your confidence. An hourly row built from twelve samples is trustworthy. An hourly row built from two samples, because the poller was down for fifty minutes, is a guess wearing the same clothes. Without the count you cannot tell them apart, and a year later nobody will remember the poller was down.


Four fields, then: average, maximum, minimum, count. Some products add the sum, or the last value, or a percentile; those are useful when you have them, but the four above are the floor. A rollup row with all four is perhaps twice the width of a raw row. And there are twelve to 288 times fewer of them. The disk will not notice. The planner will.


Tip: when building the daily rollup from hourly rows rather than from raw, check its maximum. Make sure the daily maximum is the maximum of the hourly maximums, not the maximum of the hourly averages. The second version quietly flattens every peak a second time, and the graph will look calmer than the year was.


What should never be rolled up


Some data has no sensible summary, and some data is too important to summarise. Status changes are the first kind. A device going down at 02:14 and up at 02:31 is two rows; there is no hourly version of that which means anything. Alert history is the same: the fact that an alert fired, when, and who acknowledged it, is a record, not a measurement. Keep both streams whole for as long as the longest stakeholder wants them. That stakeholder is usually the auditor. Archive these streams rather than delete them when the time comes.


Incident windows are the second kind. When something happens that will be investigated, export the raw samples for that window at full resolution on the day. Do this before the raw boundary reaches them. The samples should be stored outside the database. This is the incident snapshot, and the procedure for taking one is in archiving old monitoring data. The rollup does not know which Thursday was the important one. You do, and you have about four weeks to act on it.


Monthly summaries are the third. The auditor will ask for a short report at month end, with availability per service, peak utilisation per link, and alert counts. That is also what the planner will graph in three years' time. It should be produced while the hourly data still exists. And it should leave the database. The database is the one place with a scheduled job whose purpose is deletion.


Aligning tiers with polling and with graphs


A tier's resolution should be a whole multiple of the polling interval. With a five-minute poll, an hourly rollup summarises exactly twelve samples, every hour, and the count column is a tidy twelve. With a seven-minute poll, some hours get eight samples and some get nine. With that interval, every summary row is slightly different from its neighbours for no reason anyone will remember. Poll at intervals that divide an hour cleanly: one, two, five, ten, fifteen, thirty minutes. If the raw tier is too expensive at your interval, the polling interval itself is the other lever. The polling load budget guide is where to weigh it.


A tier should also match the graphs people actually open, and a graph can only show as many points as it has pixels. A graph 720 pixels wide showing one day has 288 five-minute samples to place, one every two and a half pixels: raw data, fully visible. The same graph showing a month has 720 hours across 720 pixels: hourly rows, one per pixel. Raw samples would be averaged by the graph anyway. Showing a year, it has 365 days across 720 pixels: daily rows, with room to spare, and hourly rows would be twelve to a pixel.


So the alignment writes itself. Day and week graphs need raw. Month and quarter graphs need hourly. Year graphs need daily. A tier that is finer than the widest graph anyone draws from it is wasted disk. A tier that is coarser than the narrowest graph drawn from it is a graph that lies by smoothing. Look at which time ranges people actually pick in the console. It is nearly always "last day", "last week", "last month", and once a year "last year". Make sure each one lands on a tier built for it.


The tier table template


The table is the design, written down. One row per rung, plus rows for the streams that are never rolled up. The values are a starting point for a shop that polls every five minutes. Change them to match your stakeholders and your disk. And keep the "used by" column honest, because it is the column that stops the next person from shortening a tier that a graph depends on.


TierResolutionKept forFields keptUsed by
RawPolling interval (5 minutes)4 weeksValueForensics; day and week graphs; incident snapshots
Hourly1 hour6 monthsAverage, maximum, minimum, countMonth and quarter graphs; threshold reviews
Daily1 day2 yearsAverage, maximum, minimum, countYear graphs; capacity planning; auditor summaries
GoneNoneAfter 2 yearsNothing in the database; monthly summary in the archiveNobody, by design
Status changesAs they happen2 years live, then archivedWhole rowForensics; availability reports; auditors
Alert historyAs they happen2 years live, then archivedWhole rowAuditors; alert reviews
Incident snapshotsRaw, exported on the dayIndefinitely, in the archiveEvery sample in the windowPost-incident reviews; disputes

Changing tiers safely on a live system


Changing a tier on a running system is a small change with an asymmetry in it. Lengthening a tier is safe and slow: the database keeps more and grows until the new steady state. Shortening a tier is fast and permanent: the next purge deletes everything past the new boundary, and there is no undo except a backup. Adding a field to the rollup, such as the maximum, applies only to rollups built from now on. The old summary rows will not grow a maximum they never had. Treat the change as a change, with a procedure.


  • Write down the current settings, exactly, with a date. If the change goes wrong, this is the only way back to where you were.
  • Run the storage math for the new ladder. Check it against the free space on the database volume, including the growth before the new steady state.
  • Take a backup of the monitoring database and confirm it restores. The monitoring backups guide covers how.
  • Export incident snapshots for any window that the new, shorter boundary would delete. This is the step people regret skipping.
  • If you can, apply the change to a restored copy first and watch one purge cycle run against it.
  • Tell the people who look at graphs what is changing and when, in one sentence: "From Monday, raw data is kept for four weeks instead of eight."
  • Change one tier at a time, on a quiet day, not on a Friday.
  • Watch the free space and the rollup job for a week. A purge that deletes months of raw data in one night can take hours and generate a large transaction log. The housekeeping guide explains what to expect.
  • Open one graph at each resolution and confirm it still draws. Then update the tier table with the new values and the date.

The single most common mistake is shortening the raw tier to free space during an emergency, on the day of the emergency. It works. It also deletes the raw samples of the emergency. Free space some other way first, then design the tiers on a calmer afternoon.


Products differ: check the documentation


Everything above is the general shape, and the general shape is what you should design to. How a product implements it varies more than you would expect. Some products have a fixed ladder with only the ages adjustable. Some let you set tiers per metric class, so that interface traffic and disk space can age differently. Some store each tier in its own table. Some keep rollups alongside raw rows with a resolution flag. Some compute summaries when a graph is drawn rather than storing them at all. Some roll up to averages only and cannot be told otherwise, which is worth knowing before you build a capacity plan on their year graph.


So: design the ladder on paper first, using the tier table. Then open the product documentation and find out which parts of it the product can enforce and which you will have to handle with an export. The documentation is also where you find the name of the rollup job and how to tell whether it ran. That is the question the housekeeping guide starts with.


A war story from Northgate Retail


Northgate Retail's stores each had a small link to the payment processor. The store on the high street reported that card payments failed for a few minutes at closing time, most evenings. The network team pulled up the month graph, which was drawn from hourly averages, and saw the link peaking at about 35 percent. They told the store manager the link was fine, twice, in writing. The raw data had aged to hourly after seven days, so nobody could look closer at any evening more than a week old. And the next week nobody remembered to look at all. Eventually a new administrator exported the raw samples for one evening on the day it happened. The administrator found the link flat at 100 percent from 17:50 to 18:02, every day, while the tills uploaded the day's takings. The hourly average had been telling the truth, in the way an average does.


Checklist


  • Every tier boundary has a reason written next to it, and the reason names a stakeholder or a graph.
  • The ladder has a "gone" rung, and the monthly summary is archived before anything reaches it.
  • Every rollup keeps average, maximum, minimum and count.
  • The daily maximum is the maximum of the hourly maximums, not of the hourly averages.
  • Status changes, alert history and incident snapshots are never rolled up.
  • Each tier's resolution divides evenly into the next, and the polling interval divides an hour cleanly.
  • Day, week, month and year graphs each land on a tier built for them.
  • The tier table is filled in, dated, and kept with the retention matrix.
  • Tier changes follow the procedure: record, calculate, back up, snapshot, announce, change one tier, watch.
  • You have read what your product actually does with rollups, rather than assuming.

Where to go next


Tiers are the design; the rest of the pillar is the operation. The housekeeping guide makes sure the rollup and purge jobs actually run and that the disk they free is really freed. The archiving guide handles the snapshots and summaries that leave the ladder on purpose. Both live in Retention and Granularity: How Long to Keep What. Design the ladder once, with reasons, and the boxes on that settings page stop being a guess.


Put this into practice with SysPeep Monitor

A tier design is only as good as the graphs that draw from it. SysPeep Monitor keeps its history in SQLite for small installs, or MySQL/MariaDB or SQL Server for larger ones. It lets you look at a counter's history, graphs and reports from one web console at whatever range you pick. For how it ages and rolls up data, check the product documentation. The free trial installs on any Windows machine.

Download Free TrialLearn About SysPeep