Home › Topics › Retention and Granularity: How Long to Keep What › Storage Math

Storage Math: What Your Monitoring Data Will Cost in Disk


Monitoring data grows in a way nothing else on your servers does. A file share grows when people save things. A monitoring database grows every five minutes, all night, all weekend, whether or not anyone is at work. It grows faster every time you add a device. You do not notice, because each row is tiny. Then one day the disk is full and the graphs stop, usually during the week you needed them.


The good news is that this growth is the most predictable thing in your infrastructure. It is multiplication: sources, times counters, times samples per day, times bytes per sample, times days kept. Know those five numbers and you know, within a factor of two, how big the database will be. The factor of two is something you can measure rather than guess.


By the end of this guide you will be able to run the formula on your own numbers. You will understand why the real file is bigger than the formula says. You will be able to measure the real growth in a week. And you will be able to fill in a one-page storage estimate you can hand to whoever buys the disks. The people who decide the "days kept" input are covered in the three stakeholders guide; this guide is about what their answers cost.


In this guide

  • The formula in plain words
  • A worked example: raw, hourly, daily
  • Steady state versus growth
  • The second stream: status and event history
  • Why the real file is about double
  • Measuring your actual growth
  • A short script and a worksheet
  • Logs and flows are separate sums
  • War story, checklist, where to go next

The formula in plain words


The size of a monitoring database is five numbers multiplied together. A source is anything you poll: a switch, a server, a firewall, a printer you have been asked to watch against your better judgement. A counter (also called a metric) is one thing you measure on that source. Examples include CPU percentage, bytes in on port 12, free space on drive D. A sample is one measurement of one counter at one moment. Each sample becomes one row in a database table, and a row costs some number of bytes on disk.


bytes  =  sources  x  counters per source  x  samples per day  x  bytes per sample  x  days kept

Two of the inputs are easy to read off your monitoring system: how many sources and how many counters. One is set by your polling interval: a sample every five minutes is 288 per day, every minute is 1,440. (The guide on choosing polling intervals is where that number comes from.) One is set by your retention policy: days kept. The last, bytes per sample, is the one nobody knows and everybody guesses; I will come back to it.


The product of the first two, sources times counters, has its own name: the number of series. A series is one counter on one source, followed over time. Two hundred devices with thirty counters each is six thousand series. Almost everything in monitoring storage scales with the series count, so know yours first.


A worked example, step by step


Take an example shop. The inputs below are examples, chosen to be round; replace them with yours. Two hundred devices. Thirty counters per device, a typical mix of CPU, memory and disk counters plus traffic, error and discard counters on a handful of interfaces. A sample every five minutes. Raw samples kept for four weeks. And, for the row size, an assumed forty bytes. That covers a typical raw row of a timestamp, a series identifier and a value, with some per-row overhead from the database engine. That last number is an assumption; the section on measuring shows how to replace it. Multiply one step at a time, so the intermediate numbers mean something.


series          =  200 devices  x  30 counters       =      6,000
samples per day =  86,400 seconds  /  300 seconds     =        288
rows per day    =  6,000 series  x  288 samples       =  1,728,000
bytes per day   =  1,728,000 rows  x  40 bytes        = 69,120,000   (about 69 MB)
raw tier        =  69,120,000 bytes  x  28 days       =  1,935,360,000   (about 1.9 GB)

So the raw tier, the expensive one, is about two gigabytes at steady state and grows by about seventy megabytes a day until the first purge. I use "gigabyte" to mean a thousand million bytes. Your operating system may count in multiples of 1,024 and show a slightly smaller number for the same file. Do not let that argument eat your afternoon. It is a ten percent difference and the overhead factor swallows it whole.


Every input is a straight multiplier: double the devices, halve the interval, or double the raw retention, and the answer doubles. There is no clever setting that gets you out of the arithmetic, only settings that pick which multiplier you would rather pay.


The rollup tiers: hourly and daily


After the raw period, most monitoring systems replace the raw samples with rollups. A rollup is one summary row that stands in for many raw ones. An hourly row stands in for twelve five-minute samples, a daily row for 288 of them. The rollups and retention guide explains the mechanism and what it loses. What matters for the arithmetic is that a rollup row is wider than a raw row. A useful one holds the average, the maximum, the minimum and the count of samples it summarised. Assume sixty-four bytes for one of those, and again, measure yours.


The same formula applies to each tier, with that tier's rows per day and its own retention. Keep hourly for six months and daily for two years, which are common choices, and the example works out like this.


hourly rows per day  =  6,000 series  x  24        =    144,000
hourly bytes per day =  144,000  x  64 bytes       =  9,216,000   (about 9 MB)
hourly tier          =  9,216,000  x  183 days     =  1,686,528,000   (about 1.7 GB)

daily rows per day   =  6,000 series  x  1         =      6,000
daily bytes per day  =  6,000  x  64 bytes         =    384,000   (about 0.4 MB)
daily tier           =  384,000  x  730 days       =  280,320,000   (about 0.3 GB)

Two things in that result surprise people. The first is that the hourly tier is nearly as big as the raw tier, even though it has twelve times fewer rows per day. It is kept six and a half times longer, and each row is wider. If you want to save disk, hourly retention is the second lever, not a rounding error. The second surprise is how cheap the daily tier is: two years of it costs less than five days of raw. That is the arithmetic behind giving the capacity planner their two years without argument. All three tiers together come to about 3.9 gigabytes of data.


Steady state versus growth


A monitoring database has two sizes, and you need both. The steady state is the size it settles at once every tier has filled and the purge deletes as much as the pollers add. In the example, that is about four gigabytes of data. The growth rate is how fast it climbs before it gets there: the raw bytes per day, about seventy megabytes in the example. Size the disk for the steady state with room to spare. Watch the growth rate, because it is the number that tells you when something has changed.


The growth curve is a staircase that flattens. The database grows at the raw rate for four weeks, then the first purge runs and growth drops to the hourly rate. Six months in, the hourly purge starts and growth drops to the daily rate, which is close to flat. If your database is a year old and still climbing at the raw rate, the purge is not running. The housekeeping guide is where to go next.


The second stream: status and event history


Alongside the samples, your monitoring system keeps a second, much smaller stream: status history, the record of things changing state. A device went down, a device came back, a check failed, an alert was raised or acknowledged. These rows are written only when something happens, so there are far fewer of them. Each carries a little text (a device name, a message), so they are wider. Estimate them the same way, with events per day in place of samples per day.


checks           =  200 devices  x  5 status checks   =  1,000
events per day   =  1,000 checks  x  2 changes (assumed) =  2,000
bytes per day    =  2,000  x  100 bytes (assumed)      =  200,000   (0.2 MB)
one year         =  200,000  x  365 days               =  73,000,000   (about 73 MB)

A year of status history for the example shop is seventy-odd megabytes, about a day of raw samples. There is no reason to roll it up and every reason to keep it, because it is the stream the forensic investigator and the auditor both ask for first. The one caution is flapping. A device that goes up and down every poll writes 288 change rows a day on its own. Ten flapping devices out-write the other hundred and ninety combined. Flapping is a monitoring problem before it is a storage problem, but it shows up in this table first.


Why the real file is about double


The formula gives you the data. The database gives you a file, and the file is bigger. The main reason is indexes. An index is a second, sorted copy of some columns, kept so the database can find "series 4,112 between Tuesday and Thursday" without reading every row. A time-series table almost always has an index on the series identifier and the timestamp, so those columns are stored twice. That alone can add half again to the raw size.


Then there is everything else. That includes per-row headers for the engine's own bookkeeping and pages that are only partly full. There is free space left by deleted rows that has not been reclaimed yet. And, in some engines, there is a transaction log beside the data file. Add it up and the honest rule of thumb is that the file on disk will be around twice what the formula says. In the example, that turns four gigabytes of data into roughly eight of disk. Treat two as a placeholder, then measure and replace it.


Measuring matters more than the formula, because the formula cannot know your product's schema. Some products store a value as an eight-byte number; some store it as text. Some pack an hour of samples into one row. Some compress. The formula gets you to the right order of magnitude, which is enough to buy a disk. Only the measurement gets you to the right number, which is enough to set an alarm on.


Tip: measure bytes per row by dividing the whole file size by the total rows in the big tables. That folds indexes and overhead into the per-row figure, and the formula becomes honest without a separate overhead factor.


Measuring your actual growth


The method is the same for every database: record the size once a day, at the same time, for a week, then subtract and divide. Seven readings give six daily differences, enough to see the average and to notice an odd day. Where you read the size depends on the backend family; the exact command is in your database's documentation, but the shape is this.


  • SQLite, the usual choice for small installs: the database is one file. Read its size in the file system, and add the write-ahead log file that sits next to it if there is one.
  • MySQL or MariaDB: the server keeps a catalogue of every table's data size and index size. Query it and add the two together for the monitoring tables, or measure the data directory on disk.
  • SQL Server: the database has one or more data files and a log file. Read their sizes from the file properties or the built-in management views. And record data and log separately, because the log has its own habits.

While you are in the database, count rows per day. It is the most useful query in the whole pillar, because it answers "is the purge running?" as well as "how fast are we growing?". The names below are illustrative; your product's table and column names will differ.


-- Illustrative: table and column names differ by product.
SELECT DATE(sample_time) AS day, COUNT(*) AS rows_written
FROM   metric_samples
WHERE  sample_time >= CURRENT_DATE - 7
GROUP  BY DATE(sample_time)
ORDER  BY day;

Compare the rows per day with the formula's prediction, 1,728,000 in the example. If the query says 3,400,000, something is polled twice as often as you thought, or there are counters you did not count. If it says 400,000, half the estate is not being polled at all, which is a more interesting finding than anything to do with storage.


A short script that does the arithmetic


Once you have done the multiplication by hand, let a script do it every time after that. This one takes the inputs from the worked example; change them to yours. It prints each tier, the data total, the total with overhead, and the growth per day before the first purge. That growth is the number to compare with your measurement.


# storage_estimate.py : rough disk estimate for monitoring metric history
devices      = 200      # sources you poll
counters     = 30       # counters per source
interval_s   = 300      # seconds between samples (300 = five minutes)
raw_bytes    = 40       # bytes per raw row: measure yours
rollup_bytes = 64       # bytes per rollup row (avg, max, min, count)
raw_days, hourly_days, daily_days = 28, 183, 730
overhead     = 2.0      # indexes, headers, free pages: measure yours

series  = devices * counters
per_day = 86400 // interval_s
raw     = series * per_day * raw_bytes * raw_days
hourly  = series * 24 * rollup_bytes * hourly_days
daily   = series * 1 * rollup_bytes * daily_days
total   = raw + hourly + daily

def gb(b):
    return b / 1e9

print(f"series {series:,}   raw rows per day {series * per_day:,}")
print(f"raw    {gb(raw):6.2f} GB for {raw_days} days")
print(f"hourly {gb(hourly):6.2f} GB for {hourly_days} days")
print(f"daily  {gb(daily):6.2f} GB for {daily_days} days")
print(f"data   {gb(total):6.2f} GB; with overhead x{overhead}: {gb(total * overhead):.2f} GB")
print(f"growth before first purge: {series * per_day * raw_bytes * overhead / 1e6:.0f} MB per day")

With the example inputs it reports about 1.9 gigabytes raw, 1.7 hourly, 0.3 daily, and 3.9 of data. It reports 7.8 gigabytes with overhead, and about 138 megabytes of growth per day before the first purge. Change the interval to sixty seconds and the raw tier becomes 9.7 gigabytes. That single edit is the most persuasive argument you will ever make for five-minute polling on things that do not need better.


The storage estimate worksheet


The worksheet is the formula laid out as lines you can fill in and show to someone. Each line names its inputs by letter so the arithmetic can be checked. The example column carries this guide's worked example; the last column is yours.


LineInput or resultExampleYours
ASources polled200
BCounters per source (average)30
CSeries (A x B)6,000
DSamples per series per day (86,400 / interval in seconds)288
ERaw rows per day (C x D)1,728,000
FBytes per raw row (measured)40
GRaw bytes per day (E x F)69 MB
HRaw days kept28
IRaw tier size (G x H)1.9 GB
JBytes per rollup row (measured)64
KHourly tier size (C x 24 x J x hourly days)1.7 GB at 183 days
LDaily tier size (C x 1 x J x daily days)0.3 GB at 730 days
MStatus and event stream (events per day x bytes x days)0.07 GB at 365 days
NData total (I + K + L + M)4.0 GB
OOverhead factor (measured; assume 2 until measured)2
PDisk at steady state (N x O)8 GB
QGrowth per day before first purge (G x O)138 MB
RDays of headroom on the volume (free space / Q)
SDate measured, and by whom

Line R is the one to put an alarm on. If free space divided by daily growth is less than the time it takes you to notice and act, the disk is going to win the negotiation. And it is going to win it at a weekend.


Logs and flows are separate sums


Metrics are the stream this pillar is about, and often the smallest of the three a monitoring shop keeps. Syslog messages are text, variable in length, and arrive in bursts; their arithmetic is in log retention. Flow records, one per conversation across a router, can outgrow both metrics and logs on a busy network; their sums are worked in flow storage math. Do each sum separately and give each stream its own volume if you can. That way, the one that surprises you does not take the other two down with it.


A war story, at my own expense


Early in my career, at Harbourview Clinic, I installed a monitoring system on a retired desktop with a forty-gigabyte disk. I set every retention value to "unlimited", on the grounds that more history could only be better. I did not do the multiplication, because I did not know there was multiplication to do. The database took nine months to fill the disk, which I took as proof that the disk was fine. When it filled, the collector stopped writing, and every graph became a perfectly flat line at whatever value it had last recorded. The practice manager remarked that the network had never been so stable. Most of what I learned that quarter is in this guide.


Checklist


  • You know your series count and your samples per day.
  • You have run the formula for each tier and for the status stream, with assumptions labelled as assumptions.
  • You have measured the database size daily for a week and know your real growth per day.
  • You have measured bytes per row (file size divided by row count) and replaced the guessed overhead factor.
  • Rows per day from the database roughly match the formula, or you know why not.
  • The worksheet is filled in, dated, and stored with the retention matrix.
  • Line R, days of headroom, has an alarm on it.
  • Logs and flows have their own sums and, ideally, their own volumes.

Where to go next


The arithmetic tells you what a retention decision costs; it does not make the decision. Designing retention tiers turns the days-kept inputs into a ladder your product can enforce. The rest of Retention and Granularity: How Long to Keep What keeps the real file close to the estimate. Multiply once, measure once, and the disk stops being a stakeholder who surprises you.


Put this into practice with SysPeep Monitor

The arithmetic is easier to check when the database is one you can open and count. SysPeep Monitor stores its history in SQLite for small installs, or MySQL/MariaDB or SQL Server for larger ones. It serves history, graphs and reports from one web console. For the retention settings that set the days-kept inputs, check the product documentation. The free trial installs on any Windows machine.

Download Free TrialLearn About SysPeep