Home › Topics › Packet Loss › Intermittent Loss

Chasing Intermittent Loss: A Field Workflow


The ticket says "the network drops out sometimes." You walk to the user's desk, run a ping, open a few sites, watch for ten minutes. Everything is perfect. The user swears it was terrible an hour ago — and they are not making it up. You close the ticket as "could not reproduce," and next Tuesday it is back: same vague symptom, same flawless network.


Intermittent packet loss — packets that leave one end and never arrive at the other, but only sometimes — is the most demoralizing fault in networking. Not the most difficult; the most demoralizing. It is shaped to defeat the way we naturally troubleshoot: show up, test now, conclude from what we see. A problem living in a few five-minute windows scattered across the week is nearly invisible to a spot check.


The cure is a change of role. Stop being a hunter and become a trapper. This guide is the field workflow that does it. Characterize the pattern, plant recording instruments, correlate timestamps, and bisect the path. And check a shortlist of usual suspects. None of it needs expensive gear — most of it is two small scripts and the discipline to leave them running. By the end you will know how to catch a problem that never has the courtesy to happen while you watch.


In this guide

  • Why the problem vanishes when you look
  • Step 1: Characterize before you chase
  • Step 2: Instrument to catch it in the act
  • Step 3: Correlate the timestamps with everything
  • Step 4: Bisect the path with waypoints
  • Step 5: Round up the usual suspects
  • A worked case: the backup that ate the morning
  • Patience, evidence, and the long game

Why the problem vanishes when you look


Start by absorbing why spot checks fail. Suppose the loss appears for five minutes, three times a day, at unpredictable moments — twenty-five bad minutes in an eight-hour day. A ten-minute visit has roughly a one-in-twenty chance of overlapping a bad window. Visit five days in a row and you will quite likely see nothing at all. Meanwhile, the users — who are on the network all day — keep hitting it. Nobody is lying. You are sampling; they live there.


There is a second trap: faults that genuinely retreat under observation. Plug your laptop into the closet switch and the act of moving gear flexes a marginal patch cable back into contact for the afternoon. Investigating changes the system. Recording instruments dodge both traps: they are always present, they never wiggle a cable, and they never get bored. Think camera trap, not stakeout.


Step 1: Characterize before you chase


Resist the urge to start pinging things immediately. An hour spent establishing the shape of the problem saves days of watching the wrong wire. You want defensible answers to three questions.


  • When does it happen? Time of day (mornings? just after lunch?), day of week, and whether it tracks load — month-end processing, shift changes, the whole office joining a video call. For outdoor or building-to-building links, add weather and temperature: sun on a cabinet, rain, wind.
  • Who and what is affected? One user or a whole floor? One switch, one VLAN, wireless clients only? One application — typically voice, because real-time traffic feels loss first — or everything at once?
  • One path or many? Do the complaints share a route: the same uplink, the same WAN link, the same server? Traffic that stays on the local switch working fine while anything crossing the WAN suffers is a huge clue.

Your sources: ticket history, the users, and whatever monitoring history you already have. Mine the tickets first. Pull every "network was slow" complaint from the past month and list the timestamps. Patterns leap out of a list. Then interview, gently. Users round aggressively: "all afternoon" often means two bad patches after lunch. Ask what they were doing at the exact moment it hurt (a call? a file copy?), and whether the next desk felt it too.


Write the result down as a hypothesis specific enough to test. Not "the network is flaky," but: "Loss affects calls between the office and the branch, weekday mornings roughly 8:30 to 9:15; calls inside the office are fine." Vague in, vague out. Specific in, and the instruments practically place themselves.


Step 2: Instrument to catch it in the act


Now build the trap. There are three layers of instrument, from cheap-and-easy to deep; most intermittent problems fall to the first two.


Instrument 1: continuous ping logs with timestamps

The workhorse. A tiny script sends one ping every couple of seconds to a chosen target and writes a timestamped line to a file, forever. Crucially, you run several of them in parallel to different waypoints. These are the first switch, the far end of the suspect link, the firewall, and something beyond it. The comparison between them is what localizes the fault in Step 4. Run the loggers from a machine that shares the sufferers' path: a wired desktop near the affected users, plus one in the server room for contrast.


On Windows, save this as ping-log.ps1 and leave it running in a PowerShell window (create C:\pinglog first):

# ping-log.ps1 - one timestamped ping every 2 seconds
$target = "192.168.1.1"          # the waypoint to watch
$log    = "C:\pinglog\gateway.csv"
while ($true) {
  $stamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
  if (Test-Connection $target -Count 1 -Quiet) {
    Add-Content $log "$stamp,ok"
  } else {
    Add-Content $log "$stamp,LOST"
  }
  Start-Sleep -Seconds 2
}

The -Quiet switch returns a plain true or false on every PowerShell version. Round-trip times are also worth having. Latency creeping up before a loss window is itself a clue. If you want round-trip times, drop -Quiet and log the reply's ResponseTime property (newer PowerShell calls it Latency).


On Linux, the same idea:

#!/bin/bash
# ping-log.sh - one timestamped ping every 2 seconds
target=192.168.1.1
log=$HOME/pinglog/gateway.csv
mkdir -p $HOME/pinglog
while true; do
  stamp=$(date '+%F %T')
  rtt=$(ping -c 1 -W 1 $target | awk -F'time=' '/time=/ {print $2+0}')
  if [ -n "$rtt" ]; then
    echo "$stamp,ok,$rtt" >> $log
  else
    echo "$stamp,LOST" >> $log
  fi
  sleep 2
done

Let the loggers run for days. One ping every two seconds is about 43,000 samples a day — enough to catch even short windows — and a negligible load. The CSV opens in any spreadsheet, where loss windows appear as unmissable clusters of LOST rows.


Tip: before you start, make sure every machine running an instrument has its clock synchronized via NTP. Nearly everything does by default — verify anyway. The whole method lines up timestamps from different sources; two logs whose clocks disagree by four minutes will send you chasing a ghost.


Instrument 2: interface counter snapshots on a schedule

Ping tells you that packets vanished. Interface counters tell you where, and often why. Every switch and router port keeps running totals of its troubles. There are errors (frames that arrived damaged) and discards (packets the device dropped on purpose, usually because a queue was full). The counters are lifetime totals, so a single reading means little. But snapshot them every five minutes with a timestamp. Then the counter that jumps during a loss window is practically a signed confession. The guide to errors, discards, and drops decodes each counter; here we only need to record them.


A small script run from cron (or Windows Task Scheduler) does it. It uses SNMP to read the counters of the suspect port — here port index 10 on a switch:

# counters.sh - snapshot error/discard counters for one port
STAMP=$(date '+%F %T')
VALUES=$(snmpget -v2c -c COMM 192.168.1.2 -Oqv \
  IF-MIB::ifInErrors.10 IF-MIB::ifOutErrors.10 \
  IF-MIB::ifInDiscards.10 IF-MIB::ifOutDiscards.10 | paste -sd, -)
echo "$STAMP,$VALUES" >> $HOME/pinglog/sw-port10.csv

# in crontab: every 5 minutes
# */5 * * * * /home/you/counters.sh

Reading the log later is a matter of looking for the jump — the delta between rows:

08:30:00,41230,0,0,18211
08:35:00,41230,0,0,18211
08:40:00,41233,0,0,19874   <-- 1,663 new output discards
08:45:00,41233,0,0,21055   <-- still climbing during the window
08:50:00,41233,0,0,21055   <-- and it stops when the loss stops

If the ping log shows loss from 8:38 to 8:49 and one uplink's discard counter moved only in that window, you have the place and the mechanism in one glance.


Instrument 3: ring-buffer packet captures

Sometimes you eventually need the packets themselves — to see retransmissions, timeouts, or exactly which conversation died. The trick for intermittent problems is a ring buffer capture. The capture tool writes into a fixed set of files, overwriting the oldest. So it can run for days in a bounded amount of disk. When the event finally fires, you stop the capture and the last however-many minutes are preserved. Treat this as the third layer, deployed after ping logs and counters have narrowed the segment. Capture everywhere at once and you will drown in data. The packet capture guides elsewhere in the topics library cover the tooling in depth; for this workflow, it is enough to know the technique exists.


Step 3: Correlate the timestamps with everything


A week later you have something precious: a list of loss windows with exact start and end times. Line those windows up against every other timeline you can find:


  • The change log. What did IT change, and when? Firmware updates, config pushes, new equipment, patches. If loss began the week something changed, the something goes to the top of the list.
  • Scheduled jobs. Backups, offsite replication, antivirus scans, patch downloads, report generation, camera archive uploads, cloud sync clients. Anything that moves bulk data on a schedule can saturate a link on that same schedule.
  • The counter CSVs from Step 2 — which counters moved during the windows, and on which ports.
  • Building rhythms. HVAC cycles, the cleaning crew (equipment does get unplugged), lunch hour. For Wi-Fi problems, the office microwave oven is a serious suspect. It radiates in the same 2.4 GHz band Wi-Fi uses. A loss pattern that lives from noon to one o'clock near the kitchen is a classic. The Wi-Fi loss guide explains why wireless is its own world.
  • Weather and temperature, for outdoor links and hot closets. Loss that follows sunny afternoons is pointing a finger at a cabinet baking on a wall.

The correlation to trust is the one that repeats. One overlap is a coincidence; a job that aligns with nine of eleven recorded windows is a lead. Absence is evidence too. If the LAN counters never moved during any window, suspicion shifts outward to the WAN and the ISP — or inward to the endpoints.


Step 4: Bisect the path with waypoints


Because you ran ping logs to several waypoints in parallel, you can now do the single most powerful move in loss troubleshooting. That is finding where along the path the loss begins. During a bad window, every waypoint before the fault shows clean pings, and every waypoint at or beyond it shows loss. The first lossy waypoint marks the far edge of the broken segment.


WaypointWhere it sitsLoss during bad windows
Access switchFirst hop, user's closet0%
Core switchBuilding backbone0%
Firewall (inside address)LAN edge0%
ISP gateway (first outside hop)Across the WAN link4.2%
Public targetAcross the internet4.5%

Reading it: the path is clean through the firewall, and loss appears between the firewall and the ISP's gateway. The fault lives on the WAN link or just beyond, not in your LAN. The table turns "the network drops sometimes" into "our WAN link loses about 4% of packets during these windows." That is a sentence you can act on — and, if it comes to it, hand to the ISP.


Here is one caution. Pinging to a router or switch is answered by its management brain (the control plane), which many devices treat as low priority when busy. A waypoint showing mild loss while everything beyond it is clean is usually a lazy responder, not a broken segment. Trust tests that go through a device over tests that stop at it. Ping's blind spots are covered in the guide to measuring loss accurately.


Bisection works at every scale, closet to WAN. If the loss begins between two devices you own, the segment is yours — cable, optic, port, or configuration. If it begins past your handoff to the ISP, your logs just became the evidence pack for the escalation described in the remediation catalog.


Step 5: Round up the usual suspects


With a timeline and a localized segment, match what you see against the classic intermittent culprits. Each has a recognizable fingerprint:


Pattern in your logsLikely culpritHow to confirm
Same clock times, weekdaysA scheduled job saturating a linkJob schedules and run history; output discards climbing only in the windows
Follows temperature: afternoons, hot closets, sunny daysMarginal cable, optic, or outdoor radio failing when warmInput errors rising with temperature; swap the part and watch
Follows load, any time of dayCongestion, or a duplex mismatch collapsing under trafficUtilization graphs; late collisions and error counters; check duplex both ends
Brief total outages, seconds long, logged as port up/downLink flaps: a connection renegotiating over a marginal cableSwitch logs show the port bouncing; each flap matches an outage
Wireless users only, patterned by time or placeInterference or airtime congestionAccess point retry statistics; a walk of the affected area
Random, no pattern, slowly worsening over weeksFailing hardwareError trends across many ports; device logs; swap and compare
Started abruptly on a known dateWhatever changed that dayThe change log; roll back and retest

Two of these deserve expansion. First, consider the temperature-sensitive part. Cables, connectors, and optical transceivers can be fine cold and marginal warm. That is why "it only happens in the afternoon" is such a strong clue. Outdoor gear adds sun, water creeping into a connector, and wind flexing a run. When the fingerprint points here, swap the suspect part early — a patch cable or optic costs less than another week of investigating.


Second, consider link flaps. A port that drops and renegotiates takes the link down for a few seconds each time. That is felt as frozen calls and dropped sessions rather than "slowness." Your ping log shows short solid blocks of LOST; the switch log shows matching up/down events. The usual causes are a marginal cable or mismatched negotiation settings — the layer-by-layer tour of where packets die walks through the whole cast of physical-layer villains.


A worked case: the backup that ate the morning


Here is the workflow end to end, on a case every admin eventually meets. A forty-person office with a small branch starts logging complaints: phone calls sound robotic and choppy, "sometimes, in the morning." Nobody can demonstrate it on demand. The calls in question all cross the WAN between the sites.


Step 1 — characterize. Ticket timestamps cluster on Tuesdays, Wednesdays, and Fridays, between roughly 8:30 and 9:15. Only calls between the sites are affected; calls inside either office are clean. Hypothesis: something is hurting the WAN path on weekday mornings.


Step 2 — instrument. Two ping loggers on a wired desktop: one to the firewall's inside address (LAN waypoint), one to the branch router across the WAN. A cron job snapshots the WAN interface's counters every five minutes.


Step 3 — correlate. After a week, the ping log to the branch shows loss windows: Tuesday 8:32–9:11, Wednesday 8:36–8:58, Friday 8:31–9:14. The LAN waypoint is spotless throughout — that is Step 4's bisection done for free: the trouble is on or past the WAN link. The counter CSV shows output discards on the WAN interface climbing in exactly those windows. The link is overfull and the router is dropping the overflow. Now the timeline hunt: what runs weekday mornings? Nothing, on paper. But the nightly backup's job history tells a better story. The offsite replication starts at 1:00 AM. And its finish time has drifted later for months as the data grew. It was 5:40, then 7:15, lately past 9:00 on the three nights the big datasets run. The backup was overrunning into the workday and saturating the WAN uplink, and voice packets were losing the queue lottery.


Steps 4 and 5 — already done. Bisection localized the WAN egress. The culprit table's first row (same clock times, output discards in the windows, a bulk job on the calendar) matched perfectly.


The fix and the proof. The replication job got a bandwidth cap of half the uplink and an earlier start. It also got an alert if it ever runs past 8:00 AM. The following week, the same instruments — still running — showed zero loss windows and a flat discard counter. Tickets stopped. Total hands-on time: perhaps two hours, spread across ten days. Nobody ever witnessed the problem live, and nobody needed to.


Patience, evidence, and the long game


Intermittent problems select for a temperament, and it is not brilliance — it is patience. The fault has one obligation: to keep committing the crime. Your instruments have one job: to be running when it does. Every recorded day either catches an event or narrows where and when the next one can hide. A week of quiet logging beats a month of heroic random visits.


Tip: while the trap is set, resist the urge to "just try rebooting it" or swap parts on a hunch. Random changes mid-experiment reset your evidence. And if the problem pauses afterward, you will not know whether you fixed it or the microwave was in the shop that week. If you must change something, log the timestamp: a change is just one more event on the correlation board. When you move to deliberate fixes, the remediation catalog is built around the same one-change discipline.


And if the instruments come back clean while users still suffer? That is progress too. The path you measured is probably innocent. The trouble lives somewhere your instruments were not looking. That could be an endpoint, an application, a path you did not test, or one of the places networks routinely hide problems from their admins. The Network Blind Spots guides map those hiding places.


For the rest of the toolkit, head back to the Packet Loss pillar, where every guide in this series lives. It covers where loss comes from, how much matters for which traffic, and how to read the counters.


Put this into practice with SysPeep Monitor

This entire workflow exists because no human can watch a network around the clock — but software can. SysPeep Monitor pings your waypoints and polls interface counters continuously. It keeps the timestamped history, and alerts you when errors climb or responses start failing. So the next intermittent fault gets caught in the act while you sleep. The free trial installs on a Windows machine in minutes.

Download Free TrialLearn About SysPeep