The Storage Math of Flow Data
Flow monitoring has a surprise ending that nobody mentions during setup. You configure export, the reports light up, and everyone is briefly impressed. Then, months later, the collector's disk fills, queries crawl, or the oldest data quietly vanishes exactly when you finally need it. The surprise has a simple cause hiding in plain sight: flow data is made of records, and records multiply.
Compare the monitoring you already know. An interface counter costs one number per interface per poll — flat, predictable storage no matter how wild the traffic gets. A flow record, by contrast, exists for every conversation your network has. Busy days literally manufacture more data to store; a new chatty application manufactures more; every device you add manufactures more. Storage grows with activity, not with device count, and nobody's intuition from metric monitoring survives first contact with that fact.
This guide makes the arithmetic friendly. You will learn to estimate how many records your network produces and what they cost in disk. You will learn how collectors survive the flood with rollups and retention tiers. The guide shows how to size a collector box with honest headroom. It shows how to recognize an undersized one before it silently loses data. It also covers what to shrink first when storage pressure finally arrives. Everything here is estimation technique with stated assumptions — not benchmarks — so swap in your own numbers and the method still holds.
In this guide
- Why flow storage surprises people
- The estimation arithmetic, in three steps
- A month of records vs. what the collector keeps
- How collectors survive: rollups and retention tiers
- Sampling as a storage lever
- Sizing the collector box
- When storage pressure hits
- The sizing worksheet
Why flow storage surprises people
The core fact: records arrive per conversation, not per byte. A user downloading one enormous file creates a huge amount of traffic but only a handful of flow records. The same user browsing the morning news creates almost no traffic but a blizzard of records. Every page pulls content from the site, its content-delivery networks, its analytics, and its advertisers. Each connection becomes at least one record. Light traffic can be heavy records. This is why a busy office's edge router can produce thousands of records per minute all day long while the bandwidth graph looks perfectly calm.
A second multiplier hides in the export mechanism itself. Exporters do not wait for a conversation to finish before reporting it. A long conversation is sliced into pieces by an active timeout (commonly around a minute) so the collector stays current. An hour-long video call is therefore not one record — it can be dozens, one per slice. Long, steady conversations quietly multiply themselves.
There is a third twist, and it matters for sizing: record volume peaks on exactly the days you least want data loss. A host scanning the network makes a record for every probe. A malware outbreak, a misbehaving application, or a busy incident day all manufacture record floods far above your average. In other words, the day you most need the flow data is the day there is the most of it to store. So a collector sized for the average day fails on the interesting one. Keep that in mind when we get to headroom.
Each individual record is tiny — tens of bytes — which is exactly what makes the trap comfortable to walk into. Tiny feels free. But millions of tiny things are big, and flow monitoring produces millions of tiny things on a schedule. The way out is arithmetic, so let us do some.
The estimation arithmetic, in three steps
The whole estimate is three multiplications: records per day, times bytes per record, times days of retention. The skill is choosing defensible inputs and writing them down, so that when reality differs you know which assumption to correct.
Step 1: Records per day
Anchor the estimate on people and their habits. Here is a reasonable planning assumption for an ordinary knowledge-work office. Each active person's device averages on the order of one new flow per second across the working day. Bursts run far higher while browsing, with long quiet stretches in meetings. That is a deliberately round figure for estimation, not a law of nature. Some networks run leaner, while chatty SaaS-heavy ones run hotter. Then add background traffic that exists whether or not humans are present as a percentage on top. That includes servers syncing, cameras streaming to a recorder, guest Wi-Fi, and update agents.
Step 2: Bytes per record
On the wire, flow records really are small. A classic NetFlow v5 record is 48 bytes. And v9/IPFIX records are in the same neighborhood depending on the fields exported. What lands on the collector's disk is bigger, because databases add indexes and structure so the records can be searched. Unless your collector's documentation says otherwise, about 100 stored bytes per record is a fair planning figure — roughly double the wire size to cover the bookkeeping.
Step 3: Multiply it out
Here is the full worked estimate for a made-up 60-person office. Every input is an assumption you should replace with your own:
Assumptions (estimation technique - swap in your own numbers): 60 people, ~10 active hours each per workday ~1 new flow per second per active person (round planning figure) background traffic (servers, cameras, guests): add ~30% stored size: ~100 bytes per record, indexes included Records/workday = 60 x 36,000 seconds = 2,160,000 + 30% background = ~2,800,000 records/day Disk per day = 2,800,000 x 100 bytes = ~280 MB Disk per month = ~280 MB x 30 = ~8-9 GB of raw records
Read the result two ways. First, the reassuring way: 8–9 GB a month for a 60-person office is survivable on almost any modern disk. Second, the cautionary way: 2.8 million records a day means every "search the whole month" query walks something like 80 million rows. That is where small collectors start to wheeze long before the disk complains. Now scale the same arithmetic up — a larger organization or a busier edge averaging ten times the flows. In that case, you get roughly 28 million records and 2.8 GB per day, over 80 GB a month, and a terabyte-scale year. Records are small; millions of small things are big.
Tip: the best estimate is a measurement. Once export is live, your collector's own statistics — records received per second, per day, per exporter — replace every assumption above. Run the system for one full week (weekdays and weekend look very different), read the real numbers, and run the multiplication again. Estimation gets you a sensible first disk; measurement tells you the truth.
A month of records vs. what the collector keeps
Second worked example, same office, different question: what does the collector actually hold after a quarter? If it kept every raw record, the math above says 25–27 GB and roughly a quarter of a billion rows. Every long-range report would then grind through them. Real collectors almost never do this. A typical policy looks like:
Raw records, kept 7 days: ~280 MB/day x 7 = ~2 GB Hourly summaries (top talkers, top conversations, protocol mix, per-interface totals), kept 90 days: a few MB per day = a few hundred MB Total on disk = ~2-3 GB versus keeping all raw for 90 days = ~25-27 GB
A tenth of the disk, and long-range queries run fast because they read the small summary tables instead of the record mountain. The catch — and it is a big one — is what the word "summary" quietly discards. That trade is the next section, because it is the single most important thing to understand before you accept any collector's default settings.
How collectors survive: rollups and retention tiers
Rollups: raw records age into summaries
A rollup is the process of boiling detailed records down into pre-computed answers as they age. If you monitor metrics, you already know this trick. Monitoring systems keep per-minute counter samples for days, then keep only hourly averages for months. Flow collectors do exactly the same thing, just with records instead of samples. After the raw window passes, the per-conversation records are reduced to hourly or daily summaries. Those include totals per interface, top talkers, top conversations, and protocol mix. The raw records are deleted.
What survives a rollup: "who were the top ten talkers between 2 and 3 PM on the 14th?" What evaporates: "show me every conversation host 10.0.8.44 had at 2:13 PM on the 14th." The summary keeps the leaderboard and discards the game footage. And note the word deleted: rolled-up detail is not compressed or archived somewhere — it is gone. No later cleverness recovers it.
Retention tiers: a sensible small-shop default
Put rollups on a schedule and you get retention tiers: full detail for a short window, summaries for a long one. Collectors name the tiers differently — raw and archive, detail and history, granular and aggregated — but the shape is always the same. Every collector ships with defaults chosen by its vendor, not by you. Find the retention settings on day one and read them, because a default raw window is often shorter than you would have guessed. The first anyone learns this is usually mid-investigation. Here is a starting policy that suits most small networks — adjust it with the test below:
| Tier | What is kept | Keep for | Can still answer |
|---|---|---|---|
| Raw records | Every conversation, full detail | 7–14 days | Full drill-down, incident timelines, per-host history |
| Hourly summaries | Totals and top-N per interface per hour, protocol mix | About 90 days | Trends, "was last month like this?", capacity questions |
| Daily summaries | Daily totals and top-N | 1–2 years | Growth over quarters, budgeting evidence |
The "last Tuesday" test
How long should the raw window be? Apply one test: can I still investigate last Tuesday? Full drill-down — the kind that names a host, a destination, and a start time — exists only inside the raw window. And investigations rarely start the day something happens. The user mentions the weirdness at the end of the week. The abuse notice arrives after the weekend. The finance question comes at month-end. Set the raw window to your realistic investigation lag plus margin. If you typically hear about problems within a couple of days, seven days of raw is comfortable. If your reviews are monthly, either extend the raw window and pay the disk, or accept that month-old questions get summary-grade answers.
Feel the difference through one scenario. An abuse notice arrives saying a machine on your network contacted a malicious address — ten days ago. With a 14-day raw window, you pull that host's complete conversation history. That includes the destination, the first contact, every session since, and the other hosts that talked to it. With a 7-day window, the summaries can still tell you the host was unusually busy that day — but not with whom. Same collector, same alert; the raw window is the difference between an answer and an apology.
Sampling as a storage lever
Everything so far manages records after they arrive. Sampling refuses to create them in the first place. The exporter examines only one packet in every N and builds records from that sample. So the record stream shrinks dramatically at the source. Every downstream cost — disk, ingest load, query time — shrinks with it. That makes sampling a genuine storage lever, and on very busy gear it may be the only lever big enough. But the price is paid in fidelity: small conversations vanish from the data entirely, and small numbers become estimates. On a modest network, prefer winning the storage battle with rollups and windows before you spend accuracy. The full trade is covered in Sampling: What 1-in-1000 Does to the Truth.
Sizing the collector box
A collector box has two separate ceilings, and both must clear. The first is ingest: how many records per second the software can absorb and index in real time. That is a rate limit, usually stated in the collector's documentation. The second is storage: how many records it can hold for how long — the disk math from above. A machine can have terabytes free and still drop records because its CPU cannot keep pace with the busy hour. Or it can ingest happily while its disk fills in a month. With that in mind, the sizing checklist:
- Records per second, estimated then measured. Start from the arithmetic above (2.8 million a day averages out to roughly 32 per second). But remember averages flatter you. The busy hour runs far above average, and the collector must swallow the peak. After a week live, read the collector's own statistics and replace your estimate with reality.
- Raw window, from the last-Tuesday test. Days of full detail, chosen deliberately rather than inherited from a default.
- Summary retention. How long the hourly and daily tiers live; cheap enough that generosity is fine.
- Headroom: multiply the disk answer by two. The doubling covers growth, indexes, and the extra interfaces you will export next year. It also covers the incident during which you will desperately want to extend the raw window. It covers all the futures you cannot see from here.
Just as important is recognizing an undersized collector, because it rarely announces itself. The symptoms follow in ascending order of harm. First are slow queries: reports that took seconds now take minutes, so people quietly stop looking. Next are disk alarms and shrinking retention: emergency purges age data out earlier than the policy promises. And worst are dropped records. Flow export usually travels over UDP, the network's fire-and-forget postal service — there is no retry and no receipt. An overwhelmed collector simply loses records, and the reports do not display an error; they display less traffic than reality, with total confidence. Silent undercounting is the failure mode that lies to you.
Warning: because export is fire-and-forget, the only honest witnesses are the collector's own counters. Most collectors publish received, processed, and dropped statistics. Check the drop counter after go-live, after adding an exporter, and any time totals look suspiciously low. A useful cross-check: compare the flow totals for one link against the SNMP interface counters for the same period. A large gap means missing records or unaccounted sampling. Monitor the monitor — the collector is a server, and its disk, CPU, and services deserve the same watching as any other.
When storage pressure hits
Sooner or later the disk-space alarm fires anyway — growth is what networks do. When it happens, shrink in the right order. Shrink the raw window before you shrink coverage. Cutting raw retention from fourteen days to seven halves the biggest tier while still recording every interface and every conversation. You keep a complete, shorter memory. Turning off export from a segment to save disk does the opposite. It creates a permanent hole in the record — a blind spot of exactly the kind cataloged in The Eight Blind Spots. Holes have a way of being where the next incident happens. A short memory beats a partial one.
Second moves: roll up sooner or more aggressively, and consider sampling at the source if the volume is truly beyond the hardware. And do not overlook the unglamorous fix — flow storage is ordinary disk, and buying more of it is often the cheapest solution on the table. One special case deserves its own path. If policy or regulation requires long history, do not hold a year of raw records in the live database. Export them periodically to compressed archive files on cheap storage instead. Flow records compress very well. The archive does not slow your daily queries, and it can be reloaded if an auditor or investigator ever asks.
The sizing worksheet
Everything above, reduced to seven lines you can fill in. Do the multiplication on paper before the collector does it to your disk.
| Line | Your number | Where it comes from |
|---|---|---|
| A. Average records/second | ____ | Collector statistics after a trial week, or the Step 1 estimate |
| B. Records/day | ____ | A × 86,400 |
| C. Stored bytes/record | ____ | Collector documentation; ~100 bytes if unknown |
| D. Raw window (days) | ____ | The last-Tuesday test: investigation lag plus margin |
| E. Raw tier disk | ____ | B × C × D |
| F. Summary tiers disk | ____ | Collector documentation; usually small next to E |
| G. Disk to provision | ____ | (E + F) × 2 for headroom |
Where to go next
The storage math is the least glamorous part of flow monitoring. It is also the part that decides whether the data is there on the day you need it. Estimate before you deploy, measure after, roll up deliberately, watch the drop counters, and when pressure comes, sacrifice window before coverage. The rest of Flow Monitoring (NetFlow, sFlow, IPFIX) completes the picture: setting up export end to end, and what the resulting data can reveal. It also covers the fidelity price of sampling when you pull that lever.
Put this into practice with SysPeep Monitor
A flow collector is a server like any other, and its disk always fills at the worst possible moment. SysPeep Monitor can watch the collector box itself. That includes disk space, CPU, memory, and service health over SNMP and WMI. It can alert you while the storage math is still a plan rather than an emergency. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep