Home › Topics › Storage Network Monitoring › Storage Latency

Storage Latency: The Number Applications Feel


Every storage system gets sold on throughput. The pitch covers how many gigabytes per second it can move, how wide the pipes are, and how fast the numbers look on the box. Then it goes into production, and the first complaint you get has nothing to do with throughput. The database is timing out. The application freezes for two seconds when someone clicks Save. The VMs feel like they are wading through glue. The pipes are nowhere near full, and everything is still slow.


That is because applications do not experience throughput. They experience latency — the time between asking the storage for something and getting the answer back. An application performing thousands of small reads and writes per second feels every millisecond of that round trip, multiplied thousands of times. Meanwhile, the throughput graph sits at a serene fifteen percent. If you learn to watch one storage number for the rest of your career, make it this one.


This guide explains why latency is the metric that matters. It walks through the life of a single I/O so you can see where the time goes. It shows you how to read latency at three different levels — the host, the network, and the array. Most usefully, it shows how comparing those three readings tells you which layer is guilty. Along the way we will demystify queue depth with a deli counter and be honest about which workloads actually care. We will set you up with baselines so the next "everything is slow" ticket meets a graph instead of a shrug.


In this guide

  • Why latency, not throughput, is what applications feel
  • The life of one I/O: where the time goes
  • Reading latency at three levels: host, network, array
  • The comparison trick: finding the guilty layer
  • Queue depth: the line at the deli counter
  • Different workloads, different pain
  • Baselines, spikes, and the "everything is slow" ticket
  • Where to go next

Why latency, not throughput, is what applications feel


Picture a highway with a badly designed on-ramp. The highway itself is eight lanes of glorious empty asphalt — enormous capacity, barely used. But the on-ramp has a stop sign, and every single car must halt, wait, and merge before it can use any of that capacity. If your journey is one car hauling one big trailer, the stop sign barely matters; you pay it once and then cruise. If your journey is two thousand separate little deliveries, you pay the stop sign two thousand times. Your day is then defined by the ramp, not the highway.


Throughput is the highway: how much data can move per second when it is really flowing. Latency is the on-ramp: how long each individual request takes, start to finish. Storage workloads that people actually complain about — databases, virtual machines, anything interactive — are the two thousand little deliveries. Each operation is small. Each one must complete before the application takes its next step. And many of them are synchronous: the application stops and waits for the answer. A modest rise in per-operation latency multiplies across every operation into misery, while the bandwidth graph shows nothing wrong at all. The same distinction applies to networks in general. The guide to latency versus bandwidth is worth internalizing once, because it keeps explaining things forever. But storage is where the lesson bites hardest, because nothing waits as impatiently as an application waiting on a disk.


One vocabulary note before we move on: you will also meet IOPS, input/output operations per second — how many individual requests the storage completes. IOPS, throughput, and latency form a triangle: lots of small operations means high IOPS and low throughput; a backup stream means the reverse. Latency is the one of the three that measures experience rather than volume, which is why it leads this guide.


The life of one I/O: where the time goes


To read latency intelligently, follow one write on its journey. An application on a server saves a record; the storage lives on an array across the network. Here is the round trip, and every stop on it is a place where time can be lost.


  • 1. The application asks. It hands the write to the operating system and — if the write is synchronous — stops right there, holding its breath.
  • 2. The OS queues it. The request joins the line of other outstanding I/Os for that disk. If the line is long, waiting in it may take longer than everything else combined — remember this for the deli counter section.
  • 3. The network carries it. The request is packaged (for iSCSI, into TCP packets). It crosses the host's network card and one or more switches, then arrives at the array's port. Each switch adds a little; a congested queue on a switch adds a lot; a lost packet and its retransmission adds an eternity.
  • 4. The array does the work. The array receives the write, lands it in cache or on disk, and generates an acknowledgment. The time this step takes is the array's service time — the number the array's own dashboard reports.
  • 5. The answer travels home. Back across the network, up through the host's driver stack, to the application — which finally exhales and takes its next step.

The latency the application feels is the whole loop: queueing plus network out, plus array service, plus network back. And here is the insight the rest of this guide is built on: different tools measure different portions of that loop. The host sees the whole thing. The network tools see only step three and five. The array sees only step four. Read all three and subtract, and the journey itself tells you where the time went.


One wrinkle worth knowing early: reads and writes age differently, so track them separately. Arrays usually acknowledge writes the moment the data lands in protected cache. That is fast, until the cache fills under sustained load and write latency suddenly cliff-dives off its usual number. Reads are the opposite: a read the array happens to have in cache comes back instantly. One that must be fetched from disk takes many times longer, so read latency naturally wobbles more. A write-latency spike often means "something is flooding the array"; a read-latency creep often means "the working set outgrew the cache." Two counters, two stories — do not average them into one.


Reading latency at three levels: host, network, array


Level one: the host — the truth serum

The host's operating system tracks how long each disk operation takes from request to completion. And this number is the ground truth, because it is measured exactly where the application suffers. On Windows, the counters live in Performance Monitor under the LogicalDisk and PhysicalDisk objects. The ones you want are Avg. Disk sec/Read and Avg. Disk sec/Write. They are reported in seconds, so 0.012 means 12 milliseconds. You can watch them live from a prompt:


typeperf "\LogicalDisk(*)\Avg. Disk sec/Read" "\LogicalDisk(*)\Avg. Disk sec/Write" -si 5

On Linux, iostat -x 5 prints extended statistics every five seconds. Read the columns r_await and w_await (average milliseconds per read and write, queue time included). Also read aqu-sz (the average queue length):


Device     r/s   w/s  rMB/s  wMB/s  r_await  w_await  aqu-sz  %util
sdb       210   480    3.2   12.1     1.8      2.4     1.9    64.0
sdc        12   890    0.1   88.4     2.1     38.7    31.2    99.6

Read that sample like an admin: sdb is having a fine day — a couple of milliseconds per operation, short queue. sdc is in trouble: writes are averaging almost forty milliseconds. Thirty-plus operations are stacked up waiting. And the device is pinned at full utilization. Whatever application lives on sdc is feeling this. Hypervisors expose the same idea per datastore or virtual disk under names like "guest latency" or "device latency". The label varies; the meaning does not.


Level two: the network — the round trip alone

Next, measure just the pipe. A simple, continuous ping from host to array across the storage VLAN tells you the network round-trip time (RTT) with no storage work included. On a healthy switched storage network this should be well under a millisecond, boringly flat, with zero loss. That is exactly why it is worth graphing continuously rather than running once during a crisis. The value of the measurement is the moment it stops being boring. A one-off ping at 3 PM cannot tell you what the path looked like at 2:14.


Be honest with yourself about what ping cannot see. It sends one small packet at a time, so it sails through a path that is mangling large, back-to-back storage frames. And it can miss brief congestion bursts between probes. So pair it with the counters on the storage switch ports — utilization, errors, and above all discards. That counter ticks up when a switch queue overflows and a frame is thrown away. A switch quietly discarding frames is the classic way a "fast" network manufactures huge storage latency. Every discarded frame becomes a TCP retransmission and a stall. The interface metrics that matter guide covers reading those counters properly.


Level three: the array — its side of the story

Almost every storage array reports its own service time: how long it took to complete operations, measured from its own front door. Find this in the array's management interface, per volume or per LUN (logical unit — one served slab of block storage). Get it into your monitoring if the array offers any way to export it. The array's number is measured after the network, which is exactly what makes it useful. It is the piece of the journey the network cannot explain.


The comparison trick: finding the guilty layer


Now the payoff. One number tells you that storage is slow; three numbers tell you where. The host feels the whole loop, so when it reports pain, subtract the layers. If the array says it is answering quickly and the network round trip is clean, the missing time is on the host itself. If the array's own service time is high, no amount of network tuning will help. Here is the decision table to keep taped inside your skull:


Host latencyNetwork RTT / errorsArray service timeMost likely culprit
HighCleanLowThe host: deep queues, a driver or HBA issue, or one VM flooding the disk queue
HighElevated, or discards climbingLowThe network path: congestion, a failing link, retransmits on the storage VLAN
HighCleanHighThe array: overloaded disks, cache exhausted, a rebuild running, or a noisy neighbor volume
HighElevatedHighEverything is drowning — usually one flood (backup, replication, migration) saturating the path and the array at once
NormalAnythingAnythingStorage is innocent. Give the ticket back to the application — politely, with the graph attached

That last row deserves respect. Half the value of measuring storage latency at the host is exoneration. The application team may be sure the storage is slow while your host-level graph shows two flat milliseconds all afternoon. In that case, the investigation moves on and you saved everyone a day. Bring evidence, not vibes.


Queue depth: the line at the deli counter


Queue depth is the number of storage operations in flight at once — sent, but not yet answered. The deli counter explains everything about it. The clerk slices sandwiches at a fixed pace: that is service time. The line of customers is the queue. Your experience — the time from joining the line to holding your sandwich — is latency, and it is service time plus waiting. When the line is two people, waiting is trivial. When the line is thirty, your sandwich takes half an hour even though the clerk never slowed down. Deep queues inflate latency without anything being broken; the system is simply busier than it can serve.


Now the honest twist: some queue is good. A clerk with an empty line is wasted capacity. Storage arrays are like clerks who work faster with a short line in front of them. They reorder and batch the work cleverly. So do not alarm on queue depth alone. The pattern that matters is the pairing: queue depth rising while latency rises means demand has passed what the system can serve. In that case, every new request will wait longer than the last. Queue depth steady with latency low means a healthy, busy system. On Windows the counter is Avg. Disk Queue Length. On Linux it is the aqu-sz column you met earlier. Hypervisors report queue depth per datastore. Watch it as latency's context, not as a headline number.


Tip: averages hide pain. A five-minute average of eight milliseconds can contain a thirty-second stall that froze every VM, smoothed into invisibility. Where your tools offer peaks or percentiles, watch those alongside the average. When a user says "it froze for a moment at 2:14," believe them and zoom in.


Different workloads, different pain


Honesty requires saying that not everything cares about latency equally, and knowing who cares saves you from chasing the wrong ghosts. Databases are the most sensitive tenants you have. They perform small, random, synchronous operations. Their transaction logs must be written — and confirmed — before a transaction completes. A few extra milliseconds per write translates directly into slower commits for every user of the application. When storage latency spikes, databases scream first and loudest.


File shares are human-paced. Opening a document performs a burst of operations, then the human reads it for ten minutes. Latency has to get genuinely bad — tens of milliseconds sustained — before people feel it. Even then the complaint is a mild "the share feels slow today." Backups and other bulk jobs barely care at all: they move huge sequential streams, tolerate latency happily, and want throughput — the highway, not the on-ramp. The cruel joke of shared storage is that these tenants live together. The backup floods the path and the array with throughput it barely appreciates. The database — which feels every millisecond — pays for it. That collision is common enough that it has its own guide: Backup Traffic: The Nightly Flood That Tests Everything.


Baselines, spikes, and the "everything is slow" ticket


What counts as bad latency? It depends — genuinely — on the storage and the workload. All-flash systems are commonly expected to answer in around a millisecond or less. Spinning-disk systems are commonly expected to answer in the several-millisecond range. Sustained readings in the tens of milliseconds mean someone, somewhere, is suffering. But rules of thumb are where the thinking starts, not where it ends. The number that actually matters is your normal. That is why the practical move is to baseline per datastore or LUN. Record latency continuously for each one, and learn its typical value and daily rhythm. Alert on sustained departure from that normal rather than on a universal magic number. The database LUN that lives at two milliseconds has a problem at eight; the archive share that lives at twelve does not. Collect each one separately — a blended average across all storage mostly measures nothing. The general craft of learning normal before alerting on abnormal is covered in the baselining guides.


Then put the baseline to work on the oldest ticket in the book. When "everything is slow" arrives, pull up host-level latency for the affected systems and look at the timeline: did latency spike when the complaints started? If yes, you have converted a vague complaint into a storage investigation with a timestamp. In that case, go find what else happened at that moment. Look for a backup starting, a migration, a replication burst, or a flood on a shared link. Run the three-level comparison to place the blame. If latency stayed flat through the complaint window, you have converted it into a not-storage investigation. That is just as valuable and much rarer to see proven. Either way, the graph did the arguing. Keep a few weeks of history; patterns that repeat daily at the same hour are practically a confession.


Where to go next


The short version to carry with you: applications feel latency, not throughput, because they wait on every operation and they wait synchronously. Measure that wait in three places. The host measures the whole journey — the truth. The network measures the pipe alone. The array measures its own service time. Let the comparison point at the guilty layer. Understand queue depth as the deli line that inflates latency when demand outruns service. Baseline each LUN and datastore so you know what its normal looks like. Meet every "everything is slow" ticket with a timeline instead of a guess.


If your block storage runs over Ethernet, your next stop should be Monitoring iSCSI: Sessions, Paths, and Retransmits. TCP retransmissions are the network's favorite way to manufacture latency spikes. If you are still building the mental model of what storage traffic is and why it gets its own network, step back to Storage on the Network: iSCSI, NAS, and SAN in Plain Language. And the rest of the Storage Network Monitoring guides carry on from here — isolation, backup floods, and all.


Put this into practice with SysPeep Monitor

Latency baselines only help if something is actually collecting them around the clock. SysPeep Monitor gathers disk and interface metrics from your hosts and switches over SNMP, WMI, and SSH. It graphs them per device so each LUN's normal becomes visible. It alerts when a number leaves its lane. So the next "everything is slow" ticket meets a timeline, not a shrug. The free trial runs on a single Windows machine.

Download Free TrialLearn About SysPeep