Monitoring iSCSI: Sessions, Paths, and Retransmits
Here is an uncomfortable truth about iSCSI environments: "the storage is up" is the least reassuring green light in your whole monitoring system. An iSCSI setup can be up — every LUN mounted, every VM running — while one of its two network paths has been dead for months. Its remaining path may be quietly retransmitting packets. The entire thing may be one flaky cable away from an outage that takes every server down at once. Nothing about "up" captures any of that.
That is because iSCSI is a layered thing: real storage riding on ordinary TCP riding on ordinary Ethernet. Each layer can degrade in ways the layer above papers over — right up until it cannot. Monitoring iSCSI well means watching the layers separately. Are the sessions logged in? Are all the paths alive? Is TCP clean? Is the frame size consistent end to end? None of those checks is hard. Most shops simply never set them up, because nobody told them which questions to ask.
This guide tells you which questions to ask. You will learn what iSCSI actually is in one breath and how to watch session state per initiator-target pair. You will learn why the silently dead multipath link is the classic iSCSI time bomb. You will learn why TCP retransmits are the storage killer and where to catch them. You will also learn how jumbo frames go wrong. It ends with a health-check table you can turn into your monitoring configuration more or less line by line.
In this guide
- iSCSI in one breath: a disk that thinks it has a cable
- Sessions: the login that must never lapse
- Multipath: the silently dead path
- TCP retransmits: the storage killer
- Jumbo frames: all-or-nothing, no exceptions
- The iSCSI health-check table
- Where to go next
iSCSI in one breath: a disk that thinks it has a cable
Computers talk to disks in a command language called SCSI: blunt little orders like "read these blocks" and "write these blocks." iSCSI wraps those commands in TCP — the same reliable transport that carries web pages — so they can cross an ordinary Ethernet network. The server runs an initiator (built into every major operating system). The storage array exposes a target. And the served slab of blocks is a LUN (logical unit number). Once connected, the operating system sees a disk. Not a network service that resembles a disk: a disk, indistinguishable from one screwed into the chassis. If this is new territory, the plain-language storage guide builds the full picture; this one assumes the elevator version.
The design is elegant, and it has one sharp consequence you must internalize before anything else in this guide makes sense. TCP promises delivery, and it keeps that promise by waiting. When a packet is lost, TCP retransmits it. Everything behind that packet queues up until the replacement arrives. For a web page, that pause is an invisible hiccup. For iSCSI, that pause happens inside a disk operation. The operating system asked its "disk" to write some blocks, and the disk has apparently wandered off. So the application that issued the write stands frozen, mid-transaction, until TCP sorts itself out. Network loss does not slow iSCSI storage down; it stalls it, in hard little stutters. Enough stalls in a row and the OS concludes the disk is failing, with all the drama that implies. Hold that thought; the retransmit section builds on it.
Sessions: the login that must never lapse
Everything in iSCSI happens inside a session — a logged-in relationship between one initiator and one target, carried over one or more TCP connections. A session is the storage equivalent of a phone call that must never hang up. While it is established, SCSI commands flow. The moment it drops, every LUN it carried vanishes from the server like a disk yanked from its bay. The initiator will normally try to log back in, and often succeeds quickly. That is exactly why session drops sneak past casual monitoring. A session that drops and recovers in twenty seconds may leave nothing behind but one cryptic event-log entry and a database that crashed mid-write.
So the first monitoring target is plain: track session state per initiator-target pair, continuously. Every host should hold a known, constant number of sessions to each target it uses — a number set by your design, not by luck. On a Linux host, list them like this:
iscsiadm -m session tcp: [1] 10.20.50.11:3260,1 iqn.array.example:vol-datastore01 tcp: [2] 10.20.50.12:3260,2 iqn.array.example:vol-datastore01
Two sessions to the same target, through two different array addresses — that is a multipath setup, which the next section dissects. The long name is an IQN (iSCSI qualified name), the unique identifier initiators and targets present to each other. On Windows, PowerShell answers the same question:
Get-IscsiSession | Select-Object TargetNodeAddress, IsConnected, NumberOfConnections TargetNodeAddress IsConnected NumberOfConnections ----------------- ----------- ------------------- iqn.array.example:vol-datastore01 True 1 iqn.array.example:vol-datastore01 True 1
Your monitoring should assert two things about this output: the count matches the design, and the count is stable. A session count that dips and recovers — flapping — is a health signal even though it always ends at the right number. Each flap was a moment when storage disappeared. Alert on any drop below design, and keep enough history to spot the 3 AM flap pattern that would otherwise stay invisible.
Turning that into automation is refreshingly unglamorous. A scheduled check on each host runs the command and counts the sessions for each target. It compares the count with the number you wrote down when you designed the thing. Monitoring systems that execute scripts or remote commands can run it directly and alert on a shortfall. And check the same fact from the other end, too: the array's management interface lists which initiators are currently logged in. So make a quick assertion of "every host that should be logged in, is" from the array side. That catches the case where a host — or the host's own monitoring agent — has quietly gone missing. Two vantage points, one truth; when they disagree, that disagreement is itself the alert.
Multipath: the silently dead path
Serious iSCSI deployments never rely on a single network path to storage, because that path becomes the single point of failure for everything. Instead they run multipath I/O (MPIO). The host establishes two or more independent sessions to the same LUN through separate NICs and, ideally, separate switches. Multipath software in the OS merges them into one logical disk. Lose a path and the software instantly shifts all traffic to the survivors; the disk never blinks. Spread across both paths in normal operation and you get the bandwidth of both. Redundancy and speed — genuinely one of the better bargains in infrastructure.
And now the trap, which is severe enough to justify this entire guide. Failover is silent by design. When one path dies, the multipath software does its job flawlessly. The cause may be a cable, a NIC, a switch port, or a VLAN mistake during maintenance. Traffic moves, nothing crashes, no user notices. If nothing is watching path counts, that flawless failover is also a flawless cover-up. From that day forward you are running on one path, meaning half the bandwidth your design assumed and zero redundancy left. And every green light on your dashboard still glows. Months later the surviving path takes a hit, and what should have been a non-event failover becomes an every-server-down outage. The post-incident review discovers the first path died some forgotten Tuesday. This is the classic iSCSI failure story, and it is entirely preventable with one monitored number. That number is the count of live paths per LUN, compared with the design.
Both major platforms will show you the paths. On Windows, MPIO reports per-disk path counts (mpclaim -s -d lists MPIO disks; add the disk number to see each path's state). On Linux, multipath -ll lays out the tree — healthy paths show active ready, dead ones show failed faulty. Hypervisors display paths per datastore in their storage views. Wherever you read it, the monitoring rule is identical: the path count for every LUN equals the design count. Anything less is an alert — a serious one. Treat "storage running on its last path" with the urgency of "storage down," because it is one incident away from becoming exactly that.
One more habit separates shops that trust their multipath from shops that merely hope: test the failover on purpose. During a scheduled maintenance window, disable one path. Administratively shut its switch port, or pull the cable like an honest pessimist. Watch two things happen. First, storage should keep flowing without a hiccup, which proves the multipath software is doing its job. Second, and just as important, your alert should fire, which proves the monitoring is doing its job. A failover test that only checks the first half misses the point of this whole section. The disaster scenario is not "failover breaks," it is "failover works and nobody hears about it." Then bring the path back and confirm the count returns to design before you leave.
Tip: after any storage-adjacent maintenance — switch reboots, cable moves, NIC replacements, VLAN changes — verify path counts before you close the ticket. Post-maintenance is precisely when a path fails to come back and precisely when everyone stops looking. Thirty seconds of checking beats discovering it during the next failure.
TCP retransmits: the storage killer
Remember the sharp consequence from the first section: a lost frame means a retransmission, and a retransmission means a stalled I/O. In the ugly case, TCP only recovers a lost packet via its retransmission timeout. A single drop can then freeze a storage conversation for a large fraction of a second. String a few of those together during a busy period and you get the signature iSCSI complaint. Latency spikes and applications freeze on a network whose utilization graphs look perfectly reasonable. Even a tiny proportion of lost frames, far too small to dent a throughput chart, is enough to make storage feel broken. Retransmits are how you catch it.
Watch for them in two places, because each tells you something different. On the host, the OS counts every TCP segment it had to send again. On Windows, netstat -s shows "Segments Retransmitted" under the TCP section. On Linux, netstat -s or nstat reports the equivalent counters. These counters are host-wide rather than per-disk. Your hosts should use dedicated storage NICs on a dedicated storage VLAN (see the isolation guide). If they do, the counter on those hosts is dominated by storage traffic and becomes a wonderfully honest signal. Sample it on a schedule and graph the rate of increase. A healthy storage network retransmits essentially never, so any sustained climb is a finding.
On the switches, you cannot see retransmits by name — a switch does not know one TCP segment from another. But you can see their cause. Two port counters on the storage ports carry the story. Discards count frames the switch deliberately threw away, almost always because a queue overflowed on a congested port. Discards on one side become retransmits on the other. So a discard counter climbing on a storage port is the smoking gun for congestion-induced stalls. Errors (CRC and friends) count frames that arrived damaged — a bad cable, a dying transceiver, a duplex mismatch. It is a physical-layer problem with a physical-layer fix. The full craft of reading these counters lives in the interface metrics that matter. For iSCSI, the summary is blunt: on storage ports, both counters should sit at zero. And "a little bit of errors" is not a thing you tolerate, because you now know what each one costs upstairs. When retransmits climb and latency spikes at the same timestamps, you have found your storage killer. Line the graphs up against your latency baselines.
Jumbo frames: all-or-nothing, no exceptions
Standard Ethernet frames carry up to 1,500 bytes of payload — the MTU, or maximum transmission unit. Jumbo frames raise that to around 9,000 bytes, and storage networks are their natural habitat. Storage moves big blocks. Packing them into fewer, larger frames means less per-frame overhead for the hosts and array to process. Enabled properly, jumbo frames are a modest, real efficiency win. Enabled improperly, they are one of the great sources of weird iSCSI misery. That is because of one iron rule: the MTU must match on every device along the path. Host NIC, host switch port, every switch and inter-switch link in between, the array's ports — all of them, the same value. Jumbo frames are all-or-nothing along the path; there is no partial credit.
Mismatch produces a uniquely confusing failure because small packets still work. Sessions log in fine (login traffic is small), pings succeed (pings are small), and light I/O may behave. Then a large read or write emits full-size 9,000-byte frames. They arrive at the one switch port still set to 1,500 and are silently dropped as oversized. The result is storage that "works" but stalls horribly under real load, and every basic connectivity test comes back clean. The test that actually settles it is a ping forced to carry a jumbo payload with fragmentation forbidden. Use 8,972 bytes of payload, leaving room for the packet headers:
# Windows: -f forbids fragmentation, -l sets payload size ping -f -l 8972 10.20.50.11 # Linux: -M do forbids fragmentation, -s sets payload size ping -M do -s 8972 10.20.50.11
Replies mean the whole path honors jumbo frames. Failures or timeouts mean something along the path does not — go find it. Run this test from every host to every array address when the network is built, and again after any change touching storage switching. After all, "replacement switch shipped with default MTU" is a beloved classic of the genre. If you cannot commit to keeping every link consistent, running standard frames everywhere is a perfectly honorable choice. Consistency matters far more than the efficiency gain.
The iSCSI health-check table
Here is the guide condensed into a checklist you can hand to your monitoring system. Poll cheap things frequently; the point of every row is to catch degradation while it is still boring.
| Check | Where to read it | Healthy looks like | Alert when |
|---|---|---|---|
| Session count per initiator-target pair | iscsiadm -m session, Get-IscsiSession | Matches design, steady | Any drop below design; flapping |
| Live path count per LUN | MPIO status, multipath -ll, hypervisor storage view | Equals design count, all active | Fewer than design — treat one-path-left as critical |
| Host disk latency | OS disk counters per volume | At its own baseline | Sustained rise above baseline |
| TCP retransmits on storage hosts | netstat -s counters, sampled | Flat, near zero growth | Any sustained climb, especially during busy windows |
| Discards on storage switch ports | Switch port counters | Zero | Any steady increase — congestion is stalling I/O |
| Errors (CRC) on storage switch ports | Switch port counters | Zero | Any steady increase — physical-layer fault |
| Storage link utilization | Switch port counters | Visible headroom at peak | Flat-topped saturation during production hours |
| Jumbo-frame consistency | Do-not-fragment ping at jumbo size | Passes host to array, every path | Fails after any switch or NIC change |
Where to go next
The mental model to keep: iSCSI is a disk that thinks it has a cable. And your job is to monitor the cable it actually has. Sessions are the logins that must never lapse — watch the count per initiator-target pair. Multipath fails silently on purpose — watch path counts against the design, and treat a lost path like the emergency it quietly is. TCP retransmits are how a slightly lossy network becomes a stalling disk. Watch host counters for the symptom and storage-port discards and errors for the cause. And jumbo frames are a one-rule technology — consistent everywhere, or not at all.
Two neighboring guides finish the picture. Storage Latency: The Number Applications Feel covers the metric all of these problems eventually surface in. It shows how to localize blame across host, network, and array. And Keeping Storage Traffic Isolated covers the network design that makes every check in this guide cleaner. A storage VLAN that carries only storage is a storage VLAN whose counters have no excuses. The rest of the Storage Network Monitoring guides are one click up.
And if you were sent here by a mystery — VMs freezing, a database timing out at odd hours — start with the timestamps. Pull host latency, retransmit counters, and storage-port discards for the complaint window and see which ones moved together. In iSCSI environments the answer is very often a short chain. A congested or faulty link dropped frames. TCP stalled while it recovered, and the application upstairs felt a disk hiccup. Find the first link in that chain and the rest of the mystery usually confesses on the spot.
Put this into practice with SysPeep Monitor
Every row of that health-check table needs something patient enough to poll it around the clock. SysPeep Monitor watches your storage switch ports over SNMP and your hosts over WMI and SSH. It watches utilization, errors, discards, and disk latency, with custom checks for the counters you care about. Thresholds and dependencies are tuned to alert you before a degraded path becomes an outage. The free trial installs on one Windows machine in minutes.
Download Free TrialLearn About SysPeep