Catching the Intermittent: Ring Buffers and Long-Running Captures
The worst network problems are the shy ones. The freeze that happens "sometimes, in the morning, I think." The call that drops once a day. The transfer that fails every night — but never while you are watching. You start a packet capture, stare at it for an hour, see nothing, stop it. The problem returns twenty minutes later, smug. Troubleshooting an intermittent fault with a hand-run capture is like photographing lightning by pressing the shutter when you hear thunder.
The fix is to stop chasing and start staking out. Leave a capture running for days or weeks, in a way that cannot fill your disk. Arrange things so that when the problem finally performs, the evidence is guaranteed to be on file. This guide covers the whole stakeout kit: ring buffers, the disk math, and snaplen strategy. It covers trigger scripts that preserve evidence automatically, and the unglamorous discipline of keeping a long capture alive. It ends with a real-shaped case where day nine paid for the whole setup.
In this guide
- Why intermittent problems defeat ordinary captures
- Ring buffers: capture forever in a fixed slice of disk
- Snaplen: trade payloads for weeks of history
- The event happened — which file has it?
- A watchdog that preserves the evidence for you
- Placement, and keeping the stakeout alive
- The companion habit: cheap continuous evidence
- Case file: the Tuesday-morning freeze, caught on day nine
- The stakeout checklist
Why intermittent problems defeat ordinary captures
A normal capture session assumes you can be present for the event. Intermittent problems break that assumption twice. First, timing: the fault picks the moment, not you, and its favorite moment is when nobody qualified is looking. Second, disk: a capture on a moderately busy link writes gigabytes per hour. So "just leave it running" ends with a full disk by tomorrow. Usually the capture then dies quietly, hours before the fault finally shows up.
So the stakeout has two requirements that pull against each other: run indefinitely, and use bounded disk. The tool makers solved this long ago with the ring buffer. It is the single most useful capture feature you have not been using.
Ring buffers: capture forever in a fixed slice of disk
A ring buffer writes the capture into a fixed number of rotating files. When the current file reaches a size limit, the tool starts the next one. When all of them have been used, it loops back and overwrites the oldest. Disk usage stays constant forever. At any moment you hold the most recent N files of history — a conveyor belt of evidence with the oldest end always falling off. Both of the standard capture engines do this natively:
# dumpcap (Wireshark's capture engine, runs headless; also on Windows): # 48 files of 100 MB each, oldest overwritten forever. filesize is in kB. dumpcap -i eth0 -w /captures/ring/stakeout.pcapng -b filesize:102400 -b files:48 # tcpdump equivalent: -C is file size in millions of bytes, -W is file count tcpdump -i eth0 -w /captures/ring/stakeout.pcap -C 100 -W 48
That is the whole trick: 4.8 GB of disk, spoken for once, never exceeded. On Windows, dumpcap installs alongside Wireshark and takes the same flags from a command prompt. On Linux it is in the wireshark-common package. Prefer dumpcap for stakeouts — it is the lightweight engine Wireshark itself uses, without the memory-hungry user interface attached.
The question that actually matters: how much history does your ring hold? That depends entirely on the traffic rate at your capture point. And it is worth doing the arithmetic before the stakeout starts rather than discovering the answer during the incident:
history (seconds) = (files x MB per file x 8) / average rate in Mbps
Example: 48 files x 100 MB = 4,800 MB of ring
link averages 40 Mbps = 5 MB written per second
4,800 / 5 = 960 seconds = 16 minutes of history
Sixteen minutes! On a busy segment, a comfortable-sounding ring holds far less past than you would guess. If the freeze happens at 08:10 and the first human looks at 09:00, sixteen minutes of history preserves nothing. You have four levers. The first two are more files and bigger files. The third is a narrower capture filter. Capturing only the suspect host or subnet stretches the ring enormously — see the tcpdump filters worth memorizing. The biggest lever of all is snaplen.
Snaplen: trade payloads for weeks of history
The snaplen (snapshot length) is how many bytes of each packet the capture keeps. By default you keep everything, including full payloads. But most diagnosis questions — who talked to whom, when, how fast, with what flags, windows, and retransmissions — are answered entirely by headers. These live in roughly the first 60 to 100 bytes of the packet. Cap the snaplen and every stored packet shrinks:
# keep only the first 128 bytes of each packet dumpcap -i eth0 -s 128 -w /captures/ring/stakeout.pcapng -b filesize:102400 -b files:48
Consider links dominated by full-size packets: backups, file transfers, media, most bulk traffic. Their frames run about 1,500 bytes on the wire. On these links, storing 128 bytes per packet shrinks the recording by a factor of ten or more. On such links your ring's history stretches ten to twenty times. The sixteen-minute ring becomes three to five hours. Add a sensible filter and the same disk holds days. Chatty small-packet traffic gains less, because its packets were already small.
The price is real: you cannot read payloads you did not keep. If the question is "what did the server actually reply," "what did the DNS answer contain," or "which error text came back," headers-only will not answer it. For DNS specifically, a snaplen of 512 or so keeps typical answers intact for cheap. The honest strategy is to default to short snaplen for long stakeouts, because the intermittent question is nearly always about timing, loss, resets, and stalls. Go full-length only when the payload itself is the suspect. Buy the shorter history that entails knowingly.
The event happened — which file has it?
Week two. A user calls: "it froze again this morning." You have 48 files. Which one holds the freeze? This is where stakeouts are won or lost, and the answer is timestamp discipline — three unglamorous habits set up on day one.
First: the capture box's clock must be right. Point it at your NTP (Network Time Protocol) source and verify it before the stakeout starts. Every ring file's name is stamped by dumpcap with a serial number and its starting date and time. So finding the event is just "open the file whose stamp brackets 08:10" — but only if the box's 08:10 is the world's 08:10. A clock minutes off sends you spelunking through the wrong file. Also decide up front whether your notes are in local time or UTC, and stick to it.
Second: the humans must bring timestamps. Brief the affected users before the stakeout: "when it happens, do not troubleshoot — look at the clock and write down the exact minute, then tell me." A report of "this morning sometime" forces you to skim hours of packets; "08:09, maybe 08:10" takes you to the scene in seconds.
Third: let your monitoring timestamp it for you. Alerts from your monitoring system — ping loss, interface errors, a service check failing — carry exact times and no human forgetfulness. Correlating an alert time against ring file stamps is the fastest "which file" answer there is. And once an event's file is identified, copy it out of the ring immediately and log it. Preserved evidence is subject to the storage, privacy, and retention rules covered in capture hygiene.
A watchdog that preserves the evidence for you
There is a nasty race hiding in the ring design. The ring overwrites oldest-first, by design. So if nobody copies the interesting files out before the belt comes around, the evidence self-destructs. On a ring holding three hours, an 08:10 event survives until about 11:10 — fine on a workday, fatal on a Saturday. The cure is a watchdog: a small script that detects the symptom itself and preserves the ring the moment it happens. The logic is always the same, whatever the symptom:
#!/bin/sh
# every 30 s: test for the symptom; on failure, preserve the ring
while true; do
if ! ping -c 3 -W 1 app-server >/dev/null 2>&1; then
d=/captures/incident-$(date +%m%d-%H%M)
mkdir -p $d
cp /captures/ring/* $d/
echo "symptom at $(date), ring preserved to $d" >> /captures/watchdog.log
sleep 900 # one incident, one copy -- not twenty copies
fi
sleep 30
done
The ping test is a placeholder for whatever your symptom looks like. Swap in a port check, a small HTTP request, or a search of an application log for its error line. The shape is the whole idea — loop, test, preserve, cool down — and it translates to PowerShell on Windows in a dozen lines. A loop around Test-Connection and Copy-Item is the same watchdog in different clothes. There are two details worth knowing. The newest ring file is still being written when you copy it, which may leave its last packet truncated. That is harmless for evidence, and the file becomes clean after the next rotation if you copy again. And give the preserved folder a generous cooldown so one long incident does not clone the ring twenty times.
Tip: fire-drill the watchdog before trusting it. Simulate the symptom: block the port for ten seconds or write the error line into the log by hand. Confirm a preserved folder actually appears. Triggers fail silently, and there is no worse moment to learn yours never worked than the morning after the event you waited two weeks for.
Placement, and keeping the stakeout alive
Where does the stakeout stand? Start near the victim: capture at or beside the machine that experiences the symptom. That vantage records exactly what the sufferer saw — including everything that failed to arrive. When two systems disagree ("we sent it" / "it never came"), run both-ends stakeouts simultaneously and let the packets referee. The mechanics of mirror ports, taps, and capturing on the host itself are covered in SPAN, mirror ports, and taps.
Then comes the discipline nobody warns you about: a long capture is a small production service, and it will fail like one. Every veteran has the same scar — the three-week stakeout that turned out to have died on day two. The cause might be a patch-night reboot no one told the capture box about. It might be a colleague parking a disk image on the "empty" drive until the ring starved. Or it might be an interface renamed by an update so dumpcap exited at 03:00 with nobody listening. Three weeks of confident waiting, zero packets. The defenses are boring and completely effective:
- Check it daily. One glance: are the ring files' timestamps advancing? A capture that stopped looks identical to a quiet network unless you look at the file dates.
- Alert on disk headroom. The ring itself cannot fill the disk, but everything else on the box can — and a starved ring stops silently.
- Make it survive reboots. Run the capture as a service or scheduled-at-boot task, not a command in someone's logged-in session.
- Add the capture box to your monitoring. It is infrastructure now. Watch its disk, its uptime, and if you can, the capture process itself. Then the stakeout watches the network while something watches the stakeout.
The companion habit: cheap continuous evidence
Packets tell you what happened on the wire, but a capture alone floats in space. You will want context: was the path lossy all morning or just at 08:09? Did latency climb before the freeze or exactly at it? So alongside every stakeout, run cheap continuous evidence. That means a timestamped ping log against the victim and a couple of waypoints, and interface counters collected on schedule. It costs nothing and turns "the capture shows a burst of retransmissions" into "the capture shows a burst of retransmissions at the exact minute the ping log shows 12% loss, and both recovered together."
ping -D app-server >> /captures/pinglog.txt # -D stamps every line with the time
If the intermittent symptom specifically is packet loss, there is a whole workflow for cornering it with layered evidence — see diagnosing intermittent packet loss. In that case, the stakeout capture becomes the final, decisive layer of that hunt rather than the only one.
Case file: the Tuesday-morning freeze, caught on day nine
The complaint. Every Tuesday around 08:10, the warehouse order-entry app froze for everyone — thirty to ninety seconds, then fine for a week. Monitoring graphs at five-minute resolution showed only a faint blip on the app server's uplink. Nobody could reproduce it on demand, because nobody can schedule a Tuesday.
The setup (day one, a Monday). A spare desktop with a large disk went on a mirror of the app server's switch port. The dumpcap ring had 48 files of 200 MB, snaplen 256, filtered to the app server's address. Call it ninety minutes of history at that segment's rate. Clock NTP-checked. A watchdog probed the app's web port every 15 seconds, preserving the ring after two consecutive failures. Ping log running. Users briefed to note the minute.
The miss (day two, the first Tuesday). The freeze came at 08:11 — and the watchdog never fired. Its trigger was too strict: single probes kept squeaking through, so two consecutive failures never happened. By the time a human checked at 10:30, the ring had rotated past 08:11 entirely. Painful, instructive: the trigger was loosened to "any two failures within a minute." This time it was fire-drilled by blocking the app port for twenty seconds and watching a preserved folder appear.
The catch (day nine, the second Tuesday). 08:09: the watchdog fired and preserved the ring. The capture told the story in two acts. Act one began at 08:09:12 with a burst of hundreds of parallel connections to the app's database tier. They all came from one HQ host — a weekly inventory-sync job. Act two, everywhere else: warehouse clients' SYNs going unanswered and retransmitting with the classic stretching backoff:
08:09:14.201 client17.51844 > app.443: Flags [S] (hello?) 08:09:15.212 client17.51844 > app.443: Flags [S] (again, 1 s later) 08:09:17.240 client17.51844 > app.443: Flags [S] (again, 2 s -- SYN backoff) meanwhile: hq-sync opens connections 40112, 40114, 40117 ... by the hundreds
The finding and the fix. The sync job's connection storm was exhausting the app tier's capacity to accept new connections. For about a minute, everyone else's hellos waited in line or died. The fix was an application change — cap the sync's parallelism and move it to 06:30. The following Tuesday passed without a freeze. Nine days of patience, one preserved ring, and "the network is flaky on Tuesdays" became a scheduling bug with a first and last name.
The stakeout checklist
Before you walk away and let it run, verify the list. Every line exists because someone (fine: because everyone) has lost a stakeout to it.
| Check | Why it matters |
|---|---|
| Clock NTP-synced, timezone decided | Every "which file has it" answer depends on timestamps agreeing |
| Ring sized with the math, not vibes | History must outlast the delay before anyone reacts |
| Snaplen chosen deliberately | Headers-only stretches history 10× or more; payloads only when content is the question |
| Capture filter as narrow as safety allows | Less noise, far longer history, smaller privacy surface |
| Watchdog fire-drilled | Triggers fail silently; prove it preserves before you rely on it |
| Preserve procedure copies out of the ring | The ring destroys oldest evidence by design |
| Capture runs as a service, disk headroom alerted | Stakeouts die of reboots and full disks, quietly |
| Daily is-it-alive glance scheduled | A dead capture looks exactly like a quiet network |
| Users briefed to note exact times | "This morning" is not a timestamp |
| Companion ping/counter log running | Context that dates and corroborates the packets |
| Stakeout logged with owner, purpose, end date | Long captures collect sensitive traffic; hygiene rules apply |
That is the whole craft: bounded disk, honest math, a tested trigger, and the humility to check on your own trap every day. Set it up once, properly, and the shy problems stop winning. For the rest of the toolkit — choosing capture moments, vantage points, filters, and reading what you caught — the Packet Capture & Analysis pillar has a guide for each step.
Put this into practice with SysPeep Monitor
A stakeout only works if it is still running when the problem returns — and if something tells you the moment the symptom appears. While you chase packets, SysPeep Monitor covers that ground. It watches your devices, interfaces, disks, and services around the clock, including the capture box itself. Its timestamped alerts hand you the exact minute to look for in the ring. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep