Sampling: What 1-in-1000 Does to the Truth
Somewhere in your flow monitoring setup, there is probably a line that reads sampling-rate 1000. It means exactly what it appears to mean. The device examines one packet out of every thousand and ignores the other 999 entirely. Your collector multiplies everything it sees by a thousand. The first time an admin truly absorbs this, the reaction is somewhere between betrayal and panic. My reports are a 0.1% survey with a straight face?
Yes — and it is more respectable than it sounds. Election pollsters do not call every voter. They ask a sample and get remarkably close, because sampling mathematics genuinely works when the thing being measured is big enough. That last clause is the entire subject of this guide. Sampled flow data answers some questions almost perfectly, answers others with confident-looking nonsense, and the difference between the two is knowable in advance. By the end you will be able to say which of your questions survive 1-in-1000. You will be able to pick a rate without superstition, and verify your multiplied numbers against ground truth.
In this guide
- Why sampling exists at all
- Big things stay true
- Small things turn to folklore
- Rare events all but vanish
- What survives sampling (and what doesn't)
- Choosing a rate without agonizing
- The multiply-and-compare sanity check
- When you need unsampled truth
- A survey, not a census
Why sampling exists at all
Full flow accounting is honest work: the device must touch every single packet, identify which conversation it belongs to, and keep a running table of every active conversation. On a firewall handling an office uplink, that is easy money — the firewall tracks connections anyway. On a fast switch, it is a different proposition entirely. A loaded multi-gigabit link can carry millions of packets per second. The chips that forward packets at that speed are built to forward, not to bookkeep. Asking a switch to account for every packet at line rate means expensive dedicated hardware, a strangled CPU, or both.
Sampling is the deal that makes visibility affordable: examine 1 packet in N, and the accounting workload drops by a factor of N. At N = 1000, a job that was millions of operations per second becomes thousands. That is cheap enough that switch makers build it into the forwarding silicon. That is why sampled export (sFlow, and sampled NetFlow on busy routers) is what fast gear actually offers. One craftsmanship detail is worth knowing. Good samplers pick packets randomly, averaging one in N, rather than metronomically taking every thousandth. A fixed rhythm could accidentally sync with rhythmic traffic and skew the picture. Randomness is what makes the multiply-back arithmetic legitimate.
Figure 1: The sampling bargain. The device examines about 1 packet in N and the collector multiplies by N. Big flows contribute so many samples that their estimates land close to truth; tiny flows are sampled once or never, so their estimates are folklore.
Big things stay true
Time for the friendly version of the statistics, using nothing beyond arithmetic. Suppose sixty million packets cross your core uplink during ten busy minutes. A nightly backup — the classic elephant flow — accounts for 30% of them: eighteen million packets. At 1-in-1000, the sampler examines roughly 60,000 packets in total. Since the backup is 30% of the stream, roughly 18,000 of the samples belong to it. Multiply back: 18,000 × 1000 = eighteen million. The estimate lands on top of the truth.
"Roughly 18,000" is doing some work in that sentence, so look at it squarely. Random chance means the real sample count wobbles. A couple of hundred either side is the natural spread for a count that size. Multiply the wobble back out and the estimate is off by around one percent. For the question anyone actually asks — "what is eating a third of my uplink?" — a one-percent error is indistinguishable from perfection. This is the survey principle: ask 60,000 randomly chosen packets what they are carrying, and the big stories in the traffic cannot hide. The elephant is unmissable precisely because it is an elephant: it contributes so many samples that chance averages out.
The same logic protects every question about large aggregates. Those include which hosts are the top talkers and what fraction of traffic is web versus backup versus video. They include whether the link's busy-hour load is trending up month over month. All of these are answered by thousands of samples apiece, and thousands of samples do not lie by much.
Rankings are even sturdier than totals, which is worth a moment because rankings are what you actually use. Suppose the true top three consumers of your uplink are the backup at 30%, a video service at 12%, and a file sync at 8%. For sampling to get the order wrong, chance would have to push one host's sample count past another's. And when the gap between them is thousands of samples wide, the wobble of a few hundred cannot bridge it. The bottom of a top-fifty list may shuffle between hosts that were nearly tied anyway; the podium does not move. So the answer to "who should I go talk to about the slow uplink" is reliable. That is true even when the exact byte figures beside each name carry a percent or two of fuzz.
Small things turn to folklore
Now run the same arithmetic on a small conversation: an SSH login session of about 50 packets. At 1-in-1000, its expected number of samples is 50 divided by 1000 — 0.05, one-twentieth of a single sample. In practice that means: most of the time, zero of its packets are sampled and the conversation simply does not exist in your data. Once in a while, one packet happens to be picked. Then the multiply-by-1000 rule dutifully reports a 50-packet conversation as a 1,000-packet conversation, overstating it twenty-fold.
Sit with how strange that is. The same conversation, on different nights, is reported as either nothing at all or twenty times its true size. Neither report is a bug. The arithmetic is doing exactly what it promised; it just has nothing to average over. A sampled estimate of any individual small flow is folklore — a story the numbers tell that happens not to be true. There is a consolation: small flows in aggregate behave like an elephant. Ten thousand DNS lookups of 5 packets each are 50,000 packets, contributing about 50 samples. That is enough that "DNS overall is about such-and-such of my traffic" stays trustworthy even while every individual lookup is invisible. Categories survive; individuals dissolve.
Once you know this, you will start recognizing the artifact in your own reports. Those are long tails of conversations that all claim suspiciously similar sizes. Each entry is one sampled packet, multiplied by N, dressed up as a round number. Those entries are not measurements; they are lottery winners. Train your eye to read the tail of a sampled report as "small stuff existed down here" rather than as individual facts. Reserve your trust for entries built from many samples. Collectors that display sample counts alongside estimates make this easy; if yours does, that little column is the honesty meter.
Rare events all but vanish
Push one step further and you reach events consisting of a handful of packets total — and here sampling goes from fuzzy to functionally blind. The canonical example is a port scan: an attacker probing your address space sends roughly one small packet per port or host probed. Each probe has a 1-in-1000 chance of being sampled. A brisk 1,000-port scan? Expect about one sampled packet — a single unremarkable record, easily lost in the noise. A cautious scan of 100 ports, spread over an evening? The odds favor your sampler seeing none of it whatsoever. The attacker did not evade your monitoring; your monitoring statistically declined to look.
This is the honest boundary of sampled data as a security sensor, and it cuts both ways. Loud badness still shows. A flood that saturates your link is an elephant, and bulk data exfiltration moves enough packets to be sampled repeatedly. Even a stealthy malware beacon accumulates samples over weeks until the pattern surfaces in long-window reports. That is a tiny check-in every five minutes, a few hundred packets a day. But single-packet events and short whispers fall through the mesh, tonight and every night. What sampled and unsampled flow data can each realistically catch is treated properly in Flow Data as a Security Sensor. The summary is: sampling trades away the whispers to make watching the roar affordable.
What survives sampling (and what doesn't)
The pattern behind all three sections is one sentence: sampling preserves questions about shares of traffic and destroys questions about individual small conversations. Whenever you are about to trust a sampled number, ask which kind of question you are asking.
Questions that survive comfortably include who the top talkers are, what the protocol mix looks like, and how capacity is trending. They include whether one department, VLAN, or application is crowding out the rest. These are all fractions of a large whole, answered by thousands of samples. Questions that do not survive include the exact size of one modest conversation and byte-accurate accounting for billing or chargeback. They include whether a specific short exchange happened at all, and anything hinging on one packet. For those you need unsampled records — or a different tool entirely.
Choosing a rate without agonizing
The rate knob is a dial with device cost on one side and fidelity on the other. Sample more often (smaller N) and estimates sharpen while the device works harder. Sample less often (bigger N) and the device relaxes while small-flow blindness creeps upward into medium-flow blindness. Two principles turn the dial-fiddling into a five-minute decision.
First, match the starting point to the link speed, because what matters is samples per second, and faster links generate plenty even at high N. The following is character guidance rather than vendor law. On modest edge links up to around a gigabit, prefer unsampled export if the device offers it. For those links, prefer something gentle like 1-in-100 if the device needs relief. Around 10-gigabit, 1-in-1000 is the classic starting point. On faster backbone and fabric links, 1-in-2000 up to 1-in-10000 still yields a firehose of samples. Second, sample harder only when the device budget forces you to. The reason to raise N is a measurable one — exporter CPU climbing, the vendor's stated limits — not tidiness. If the device is comfortable, keep the fidelity.
How do you know when the device is not comfortable? Watch two things after enabling or tightening sampling. The first is the exporter's CPU and memory, which you should be monitoring anyway as part of basic device health. A sustained climb that began when sampling did is the device telling you the rate is too generous for its budget. The second is subtler: many devices keep a counter of samples they intended to take but dropped under load. A climbing drop counter means your effective rate is quietly worse than your configured rate. The device is sampling 1-in-more-than-N and nobody updated the multiplication. Raise N deliberately yourself rather than letting an overloaded device raise it for you at random.
Tip: the rate is a per-device decision, not a network-wide vow. An unsampled edge firewall and a 1-in-2000 core switch living in the same collector is not messy. It is correct design, with each device spending what it can afford. Just make sure the collector knows each exporter's rate, for reasons the next section makes vivid.
The multiply-and-compare sanity check
Sampled data earns trust the same way any witness does: by agreeing with independent testimony. Your independent witness is the humble interface counter, which counts every byte crossing a port with no sampling involved. Here is the ritual. Pick one exported interface and one recent busy hour. Ask the flow collector for total estimated bytes on that interface in that hour. Pull the same hour's byte total from the interface counters. Divide the larger by the smaller. You met this check as part of deployment validation in Setting Up Flow Export End to End. It deserves repeating whenever a sampling rate changes, and the verdicts read like this:
- Within a few percent: healthy. Counters include some low-level framing overhead that flow records do not, so a small, stable gap is normal.
- Off by about your sampling factor (flow totals absurdly low): the multiplier is not being applied — the collector does not know this exporter samples, or was told the wrong rate.
- Off by the factor in the other direction (flow totals absurdly high): the multiplication happened twice — the device scaled its numbers before export and the collector scaled them again. This one has embarrassed veterans.
- Off by some unprincipled factor: usually structural — traffic crossing interfaces that are not exporting, or a configured rate that does not match what the hardware actually does.
Five minutes, once per exporter, and your multiplied numbers graduate from plausible to verified. Then make the ritual part of change control: rerun it after firmware upgrades, after collector migrations, and after anyone touches the sampling configuration on either end. Rate settings have a way of reverting to defaults during upgrades. Because sampled export keeps producing plausible-looking reports when the multiplier is wrong, nothing will look broken. The comparison is the only witness that notices, so call it back to the stand whenever the cast changes.
When you need unsampled truth
Some questions refuse to be surveyed: billing-grade byte accounting, dependency audits where a tiny weekly conversation is exactly the thing you must not miss, security forensics on short exchanges. The answer is not to force your fastest switch to count every packet — it is to put the census where the census is affordable. This is the tiered design most mature networks converge on. The internet edge firewall runs full, unsampled NetFlow or IPFIX. The speeds there are modest, and the firewall tracks every connection anyway. It is precisely where accounting and security questions concentrate. Meanwhile, the core and fabric run sampled sFlow for statistically faithful breadth across east-west traffic. Each tier answers the kind of question it is good at, and the collector merges both into one view. (Which devices speak which protocol, and why that maps onto this design so neatly, is covered in NetFlow vs. sFlow vs. IPFIX.)
A survey, not a census
Sampled flow data is a survey, not a census — and surveys are excellent instruments when you respect their error bars. Pollsters do not apologize for sampling; they publish the margin and stand behind the results it supports. Do the same. Lean confidently on sampled data for shares, rankings, mixes, and trends. Refuse to let it testify about individual small conversations. And keep one unsampled vantage point where exactness matters.
Respecting error bars also changes how you report, and this is a professional skill worth naming. When a sampled estimate says the backup moved 4.2 TB, say "about four terabytes" — rounding to match your actual certainty. Never let a decimal point imply precision the data does not have. When a small-flow figure is folklore, say "too small to measure at our sampling rate" rather than repeating the folklore with a straight face. Colleagues and managers forgive honest approximation instantly; they remember confidently wrong numbers forever. The admins people trust with capacity money are the ones whose numbers come pre-labeled with how much to trust them. For quick reference, the rules of thumb:
| Question | Survives 1-in-N sampling? | Notes |
|---|---|---|
| Who are the top talkers? | Yes | Elephants contribute thousands of samples; rankings are solid |
| What is the protocol/application mix? | Yes | Aggregates stay honest even when members are invisible |
| Is utilization trending up? | Yes | Trends smooth the sampling noise further |
| Is one VLAN crowding the others? | Yes | Shares of a large whole — sampling's home turf |
| How big was that 50-packet conversation? | No | Sampled once or never; the estimate is folklore |
| Exact bytes for billing or chargeback | No | Use unsampled export at the edge |
| Did anyone port-scan us last night? | Mostly no | One-packet events slip through the mesh almost every time |
| Is that workstation beaconing? | Over weeks, maybe | Steady heartbeats accumulate samples in long windows |
Respect those boundaries and sampling stops being a compromise and becomes what it actually is. It is the engineering trick that bought you visibility your hardware could never otherwise afford. The rest of the Flow Monitoring (NetFlow, sFlow, IPFIX) guides build on this foundation — from reading the reports your survey produces to the storage math of keeping it all.
Put this into practice with SysPeep Monitor
Sampled flows are the survey; interface counters are the census — the ground truth you just learned to check every estimate against. SysPeep Monitor keeps that census for you. It discovers your devices and graphs every interface's real byte counts over SNMP. It watches exporter health and CPU, and alerts when something drifts. The free trial installs on any Windows machine in minutes.
Download Free TrialLearn About SysPeep