Measuring Latency Honestly
Two admins are arguing about a slow application. One runs a ping and announces the network is fine: 2 ms. The other times an actual transaction and announces the network is terrible: 900 ms. Both measurements are correct. Neither admin is lying. They are answering different questions with different instruments. Until someone notices that, the argument can run for days, which is exactly how latency arguments usually go.
Measuring latency honestly does not require expensive gear. It requires knowing three things about every number you produce: what the tool actually measured, when and how often it measured, and how the results were summarized. Get any of the three wrong and you will produce numbers that are technically true and practically misleading. This guide builds you a layered test kit, from the humble ping up through TCP handshake timing and full application-level breakdowns. Then it covers the traps: one-way versus round-trip, averages that hide the pain, spot checks that miss the problem, and idle tests that flatter a link. It ends with a runbook table you can work through top to bottom the next time someone says "the network is slow, prove it isn't."
In this guide
- What "latency" are you actually asking about?
- Layer 1: ICMP ping, the smoke test
- Layer 2: TCP connect timing, closer to app truth
- Layer 3: application timing with curl
- One-way vs. round trip
- Percentiles beat averages
- Spot checks vs. continuous measurement
- Idle vs. under load
- The latency measurement runbook
- Where to go next
What "latency" are you actually asking about?
"Latency" is a stack of different waits wearing one name. At the bottom is the wire-level round-trip time (RTT): how long a small packet takes to reach a machine and come back. Above that sits connection latency: how long it takes to actually open a usable connection to a real service. Above that, application latency: how long until the first byte of a real answer arrives — which includes the server's think-time. And at the top, what the user calls latency: click to pixels, everything included. (The guide on the anatomy of a round trip dissects that whole stack with worked numbers.)
Each layer of the kit below measures one of those waits, honestly, as long as you keep the labels straight. The arguing admins earlier? One measured wire RTT, the other measured application latency, and the 898 ms gap between them was the server thinking. The tools were fine. The labeling was the failure.
Layer 1: ICMP ping, the smoke test
ping sends a tiny "are you there?" message using ICMP (Internet Control Message Protocol — the network's built-in signaling language), and times the reply. It is on every operating system, needs no setup, and answers the first question fast:
# Linux / macOS: 20 pings, one per second ping -c 20 app-server # Windows ping -n 20 app-server
The summary line at the end reports minimum, average, and maximum RTT. On Linux, it also reports mdev — a measure of how much the samples varied, your first glimpse of jitter. Ping is a genuinely good instrument, but honesty requires knowing its four caveats:
- It measures the wire, not the app. The reply comes from the operating system's network stack. A server can answer pings in 2 ms while the database on it takes 30 seconds per query. A clean ping rules out the path; it says nothing about the service.
- Network devices deprioritize it. Routers and switches answer pings addressed to them with their busy management CPU, as their lowest-priority chore. Meanwhile, traffic through them is forwarded in fast hardware. A router that pings at 80 ms may be forwarding your packets in microseconds. Ping through devices to endpoints for path truth; treat pings at network gear as rough health signals only.
- Some networks rate-limit or block it. A missing reply does not always mean down, and a few dropped pings to one target may mean a polite rate limiter, not loss. Corroborate before declaring an outage.
- It is small and occasional. Default pings are tiny packets once per second. They are gentle probes that can sail through moments of congestion that hammer real traffic, and that easily miss short bursts of trouble. You can at least test with realistic sizes:
-s 1400on Linux,-l 1400on Windows sends near-full-size packets.
Verdict: always start with ping. It is the fastest way to learn the rough RTT and whether the path is sane (compare against the baseline table for the path type). Just never let it be the only witness.
Layer 2: TCP connect timing, closer to app truth
Real applications do not speak ICMP; they open TCP connections to real ports. Timing that connection — the SYN out, the SYN-ACK back — measures a genuine round trip as forwarded like real traffic. It goes through the same queues and fast paths your application uses, to the actual service's doorstep. TCP connect timing also works where ICMP is filtered, and it proves the port is answering at all. Two good instruments:
Cross-platform, with curl (shipped with current Windows, macOS, and virtually every Linux). The handshake time is time_connect minus the DNS portion:
curl -o /dev/null -s -w "dns %{time_namelookup}s tcp-done %{time_connect}s\n" http://app-server/
If that prints dns 0.004s tcp-done 0.046s, the handshake took 42 ms. Since a TCP handshake costs exactly one round trip, you have measured the RTT to the service with the network treating you like a real client.
On Windows, with repetition and statistics. The built-in check proves reachability of a port:
Test-NetConnection app-server -Port 443
…but it reports success, not handshake timing. For timed, repeated TCP connects, the free Sysinternals tool psping is the workhorse:
psping -n 20 app-server:443 TCP connect statistics for 10.20.0.15:443: Sent = 20, Received = 20, Lost = 0 (0% loss), Minimum = 38.21ms, Maximum = 44.90ms, Average = 39.77ms
Twenty real handshakes, with spread. If ping said 2 ms and TCP connect says 40, believe both. In that case, the wire is short but something on the connection path — commonly an overloaded middlebox or the far host's stack — is adding cost to real connections. One honesty caveat here too: the operating system can complete handshakes on the application's behalf while the application itself is wedged. So a fast connect plus a hanging service means the problem sits above the network. Which is exactly what the next layer measures.
Layer 3: application timing with curl
The top layer times a real request end to end, with a stopwatch on every phase. curl's write-out flag prints a running clock at each milestone:
curl -o /dev/null -s -w "DNS done: %{time_namelookup}s
TCP done: %{time_connect}s
TLS done: %{time_appconnect}s
First byte: %{time_starttransfer}s
Total: %{time_total}s
" https://app-server/api/status
Example output, and what each subtraction reveals:
DNS done: 0.032s TCP done: 0.074s TLS done: 0.158s First byte: 0.601s Total: 0.688s
| Phase | Compute as | This example | It measures |
|---|---|---|---|
| DNS lookup | time_namelookup | 32 ms | Name resolution (cold caches pay more) |
| TCP handshake | connect − namelookup | 42 ms | One true round trip: your RTT |
| TLS handshake | appconnect − connect | 84 ms | Encryption setup: one to two more RTTs |
| Server think-time | starttransfer − appconnect − one RTT | ~400 ms | The application working: not a network cost |
| Transfer | total − starttransfer | 87 ms | Response size, bandwidth, and slow-start round trips |
One command, and the slow-app argument is over. This path has a 42 ms RTT behaving exactly as its handshakes predict, and a server spending 400 ms thinking. Run it several times and discard the first (it pays cold DNS and session tolls the rest skip). Browser developer tools show the same phases graphically for every page resource — the anatomy guide walks through reading that waterfall.
One-way vs. round trip
Everything above measures round trips, and it is worth being honest about what that hides. The two directions of a path are not twins. The routes can differ entirely (asymmetric routing is normal on the internet), and the congestion usually does differ. A site whose uplink is saturated by a cloud backup will have a bloated outbound direction and a clean inbound one. The RTT blends them into one number that blames nobody in particular.
The obvious fix — measure each direction separately — runs into an unglamorous wall: clock synchronization. To time a one-way trip you subtract the send timestamp on one machine from the arrival timestamp on another. So both clocks must agree to well under a millisecond. Ordinary NTP time sync keeps clocks within a few milliseconds at best across a network. That is error the same size as the thing you are measuring. True one-way measurement needs hardware-grade time (GPS-disciplined clocks, or precision time protocols on a controlled network) plus measurement daemons at both ends. It exists, and it is how carriers verify their promises. It is rarely worth the ceremony inside a business network.
What to do instead: report RTT and never silently divide it by two. Write "84 ms RTT," not "42 ms each way," because the split may be 70/14. When you suspect asymmetry, hunt it with indirect evidence. Traceroute from both ends when you control both (different routes are a clue). Also load one direction at a time while watching RTT. If saturating the uplink inflates the RTT but saturating the downlink barely moves it, you have found which direction owns the queue.
Percentiles beat averages
Now the summarizing trap. Suppose 100 pings return 95 samples of 20 ms and 5 samples of 800 ms. That is a link that is usually fine and occasionally awful, which is exactly how real congestion behaves. The average is 59 ms: a number that describes not one single packet that actually occurred, flattering the spikes and slandering the typical. Averages are how a link that ruins four phone calls an hour passes its health check.
Percentiles keep both truths. Sort the samples; the median (50th percentile) is the middle one — here 20 ms, the honest "typical." The 95th percentile (p95) is the value 95% of samples sit at or below. Here it is 800 ms, the honest "how bad are the bad moments." Report the pair: "20 ms typical, but the worst 5% hit 800" describes this link perfectly, and no single number can. Users, meanwhile, experience the percentiles you did not fix: spikes are what a frozen cursor and a garbled word of audio feel like. The math is a sort and an index, no statistics package required:
# 1. Log 300 RTT samples, one per second (Linux/macOS)
ping -c 300 -i 1 app-server | sed -n 's/.*time=\([0-9.]*\).*/\1/p' > rtt.txt
# 2. Sort them and pull the median, p95, and p99
sort -n rtt.txt | awk '{a[NR]=$1} END {print "median:", a[int(NR*0.50)];
print "p95:", a[int(NR*0.95)]; print "p99:", a[int(NR*0.99)]}'
On Windows, PowerShell does the same trick — collect, sort, index:
$t = (Test-Connection app-server -Count 300).ResponseTime | Sort-Object "median: " + $t[[int]($t.Count * 0.50)] "p95: " + $t[[int]($t.Count * 0.95)]
A useful reading habit: compare p95 to the median. Close together means a steady path. A p95 several times the median means something is queuing intermittently. The times of day when the gap opens tell you when to go looking.
Spot checks vs. continuous measurement
Every command so far shares a quiet flaw: it measures now. But latency problems are creatures of habit — the 10 AM swell, the backup window, the Tuesday report run. A spot check only catches one if you happen to run it during the misery. The classic failure: a user reports slowness at 2 PM, you test at 4 PM, everything is clean, ticket closed, user seething, problem intact and punctual.
The honest fix is continuous measurement: the same modest probes, run on a schedule, with the results kept. A ping per minute per target is invisible on any network and produces 1,440 samples a day. That is enough to draw the daily shape, catch the recurring spike, and compute honest percentiles per hour. This is precisely the job description of a monitoring system. And it is how you build the baseline — your network's documented normal — that turns future arguments into ten-second lookups. (The baselines guide covers what normal should look like per path type and how to write yours down. The free-tools guide shows measurement options for all the core metrics.) Spot checks still matter — they are how you chase a live problem — but they answer "what is happening?", never "what changed?". Only history answers that.
Idle vs. under load
The final honesty trap: when you measure relative to load. An idle link is a flattering photographer. Queues are empty, so ping shows you the physics floor and hides how the link behaves the moment it earns its living. Since queuing is the delay component that varies (the four delay types guide explains why), an idle test literally cannot see the problem users complain about.
So test both states, deliberately. The recipe: start a continuous ping to a target beyond the link you are testing. Note the idle RTT for a minute. Then saturate the link — a large file download, then separately a large upload — while the ping keeps running. Watch what the RTT does. A healthy, well-managed link rises modestly under load: a few extra milliseconds, maybe tens. A link whose RTT balloons from 15 ms to 400 ms the moment a transfer starts has oversized, unmanaged buffers. That is the bufferbloat signature, worth its own investigation in the bufferbloat guide, which also covers the queue-management fixes. Either way, the under-load number is the truthful one for predicting how voice calls and remote sessions will feel while the network is actually being used. That is, after all, the network's whole job.
Tip: whatever you measure, write down the context next to the number: tool, target, port, time of day, wired or Wi-Fi, idle or loaded. "42 ms" is trivia. "42 ms TCP connect to the app server, wired, 2 PM under normal load, p95 of 20 samples" is evidence. Future-you, reading the ticket six months later, will not remember. Numbers without context are how measurement dishonesty starts, even among the honest.
The latency measurement runbook
The whole kit, in working order. Start at the top; each row's answer decides whether the next row matters:
| Question | Tool | Command | What good looks like |
|---|---|---|---|
| Is the path alive, and what is the rough RTT? | ping | ping -c 20 host (Windows: -n 20) | RTT matches the baseline for the path type; small min–max spread |
| Does a real port answer, and how fast? | psping / curl | psping -n 20 host:443 | Close to ping's RTT; consistent across runs |
| Is it the server or the network? | curl timing | the -w breakdown above | Handshakes ≈ whole-number RTT multiples; think-time small |
| Where along the path does delay start? | traceroute / mtr | mtr -rwc 100 host | Steps match geography and persist to the final hop |
| How bad are the bad moments? | logged pings | the sort-and-percentile one-liner | p95 near the median |
| Does load wreck it? | ping + a big transfer | the idle-vs-load recipe | RTT rises by little, not by hundreds of ms |
| What is normal here, and did it change? | continuous monitoring | scheduled checks, kept history | A baseline per path, and alerts only on drift from it |
Work it top to bottom and you will notice something pleasant. By the time you reach the bottom rows, you are no longer measuring — you are diagnosing. The runbook's answers hand you straight to the right fix-it guide. A step in the traceroute goes to the high-RTT troubleshooting tree. A load-triggered balloon goes to the bufferbloat guide. And a fat server think-time goes to whoever owns the application, with your evidence attached.
Where to go next
Honest measurement comes down to five habits. Name which latency you are measuring. Use a tool that speaks that layer. Summarize with percentiles. Measure continuously enough to know normal, and test under load as well as idle. With those in place, every other guide in this topic gets sharper. For example, the round-trip anatomy tells you what the phases mean. Meanwhile, the baselines reference tells you what the numbers should be. And the troubleshooting decision tree puts the kit to work on a live problem. The full topic lives on the Network Latency pillar page.
Put this into practice with SysPeep Monitor
Every command in this guide is a spot check - useful the moment you run it, silent the rest of the week. SysPeep Monitor runs the equivalent checks on a schedule: response times to your servers and gateways, and device health over SNMP and WMI. It records them all so percentiles and baselines come free with the history. Thresholds and dependency-aware alerting mean you hear when a path drifts from its normal, not every time one ping hiccups. The free trial installs on a Windows machine in minutes.
Download Free TrialLearn About SysPeep