Drivers and Offloads: When the Smart NIC Does Something Dumb
Some network problems behave. The cable is bad, the errors climb, you swap it, done. And then there is the other kind: the file server that moves data at a crawl over a link that tests perfectly. The one application that stalls while everything else on the same server flies. The packet capture full of alarming red "bad checksum" warnings on a network that works fine. These tickets do not respond to normal logic. And they have a common ancestor more often than you would guess: the network card itself, trying to be helpful.
A modern NIC (network interface card) is not a dumb socket. It is a small computer with its own processor, memory, and firmware. It volunteers to do chores the operating system would otherwise do. It computes checksums, slices big sends into packet-sized pieces, and bundles small arrivals into big deliveries. These chores are called offloads, and when they work (which is nearly always) they save real CPU and everyone wins. But an offload can misbehave: a driver bug, a bad interaction with VLANs or virtualization, a firmware quirk. That failure happens inside the data path, below every log your OS writes. It produces symptoms that seem designed to confuse you.
This guide teaches you the offloads in plain language. It shows you the signature symptoms of offload trouble (including the famous capture false alarm that has fooled generations of admins). It gives you a safe, boring, one-change-at-a-time method for testing your way to the culprit. It also offers habits for keeping drivers and firmware out of trouble in the first place.
In this guide
- The NIC that does chores
- The three offloads you will actually meet
- Why offload bugs look so weird
- The capture false alarm
- Driver counters beyond the basics
- Safe experimentation, one toggle at a time
- Keeping drivers and firmware sane
- When to suspect the driver: a checklist
- Where to go next
The NIC that does chores
For every packet a server sends or receives, somebody has to do the paperwork. That means computing the little error-detection sums each packet carries and cutting application data into pieces small enough for the network. It also means reassembling arriving pieces and interrupting the CPU to say "you have mail." Historically the operating system did all of it, spending CPU cycles on every single packet. On a busy server, that adds up to a lot of cycles that could have gone to the actual application.
So NIC designers made an offer: hand those chores to the card. The NIC has silicon that can compute checksums and slice data far more cheaply than general-purpose CPU code. Doing the work on the card means fewer interruptions for the OS. The interruptions matter more than the arithmetic, by the way. Every packet the OS handles individually costs a fixed overhead — an interrupt, a context switch, some bookkeeping. At high packet rates that fixed cost dwarfs the useful work. Offloads win mostly by turning thousands of small handoffs into a few big ones.
Collectively these delegated chores are the offloads. On any modern server several of them are on by default without anyone ever deciding anything. That is worth pausing on: your servers are already relying on this machinery, chosen by nobody, right now. Think of offloads like a dishwasher: a genuinely great appliance that saves you real time. That lasts right up until the day it starts quietly eating a spoon per load. You spend a week blaming the spoon drawer.
The three offloads you will actually meet
Checksum offload
Most packets carry checksums — small computed sums that let the receiver detect corruption. With checksum offload on, the OS hands the packet to the NIC with the checksum field effectively blank. The card fills it in on the way out the door. Inbound, the card verifies arriving checksums so the OS does not have to. Remember the "blank on the way out" detail — it stars in the false-alarm story below.
Segmentation offload (large send)
Networks move data in packets no bigger than the link's MTU (maximum transmission unit — the largest parcel a link accepts). Applications think in much bigger chunks. Someone has to slice. With segmentation offload, the OS hands the NIC one big chunk plus a template. You will see names like "large send offload" in adapter settings. The card cuts the chunk into wire-sized segments itself. One handoff instead of dozens of per-packet trips: a big CPU saving on servers that send a lot.
Receive coalescing
The mirror image. When a stream of related packets arrives back-to-back, the card (or driver) glues them into one large delivery before bothering the OS. That means one interrupt and one big buffer, instead of many small ones. Great for bulk transfer throughput; occasionally unhelpful for traffic that cares about every millisecond, since packets sit briefly while the card waits for their siblings. Different systems use different names for the same idea; adapter settings usually say something like "receive segment coalescing."
There are more exotic residents of the advanced-settings tab. But these three — checksums, segmentation, coalescing — carry almost all real traffic and cause almost all the interesting trouble.
Why offload bugs look so weird
Here is the key insight: an offload failure is a helper failing, selectively, inside the pipeline — not a wire failing. That single fact explains every strange symptom on the list.
The link tests clean, but throughput is terrible. Cables and switch counters measure the wire, and the wire is fine. But suppose a segmentation offload mishandles certain packets. Receivers then discard or misorder data, and senders retransmit and back off. Your big transfer limps — over a physically perfect link. Everything in the switch error counters playbook comes back innocent, because the crime scene is inside the server.
One application suffers while others fly. Offloads apply to specific kinds of traffic. A bug might bite only large sends (so bulk copies crawl but small chatty queries are fine). It might affect only a particular protocol, or only VLAN-tagged or tunneled packets. Encapsulation is where many driver bugs live. That is also why virtual machines meet more than their share — see the VM edition of this story. Selective damage looks like an application problem, and the app team will correctly point out their code is unchanged.
The problem tracks the server, not the network path. Move the server to another switch port, swap cables, even another switch — the weirdness follows, because it lives in the server's driver. That portability is itself diagnostic gold, and it is worth doing deliberately just to prove it.
The CPU quietly pays a tax. Sometimes an offload does not fail loudly — it silently stops being used, and the OS shoulders the chores again. Nothing breaks; the server just burns noticeably more CPU on interrupts and packet processing at the same traffic level. Your utilization baseline is what catches this one, which is one more argument for knowing your servers' normal.
Put those traits together and you can see why offload trouble earns its reputation. A concrete miniature: backups from one particular server run at a tenth of their usual speed. Ping is perfect, the switch port is clean, small transfers are snappy — only sustained bulk sends crawl. Cable swapped, port moved: no change. That combination — clean link, one traffic type, follows the server — is the offload signature. In this story, the culprit turns out to be a segmentation offload stumbling over that server's recently added VLAN tagging. Nothing in any log said so; the shape of the symptom did. Learning to read that shape is the skill, and the checklist at the end of this guide compresses it.
The capture false alarm
Now the classic. One day you run a packet capture on a server — troubleshooting something unrelated, perhaps. The analysis tool lights up: bad checksum on packet after packet. Thousands of them. All corrupt? Is the NIC destroying data? Before you draft the incident report, look closer: the "corrupt" packets are all outgoing, from the machine you are capturing on. And they are almost certainly fine.
Remember where capturing happens: the capture engine sits between the operating system and the NIC. It photographs packets on their way to the card — and with checksum offload enabled, that is before the card fills the checksum in. The capture faithfully records a checksum field that is blank or wrong at that instant. The hardware would have completed it a microsecond later, just past the camera. Your analyzer, checking the math, flags every one. Same story with segmentation offload: local captures show impossible giant packets, far larger than the MTU, photographed before the card sliced them for the wire. Both are artifacts of where you stood, not of what the network carried.
The verification rule is simple: a checksum error on your own machine's outgoing packets, in a capture taken on that machine, is almost always cosmetic. To find out what really crossed the wire, capture somewhere else — on the receiving host or a switch mirror port. If the receiver's capture shows valid checksums, the alarm was false and the offload was just doing its job out of frame. (If the receiver sees bad checksums too, now you have something real.) Most analysis tools also offer an option to stop validating checksums for exactly this reason. The broader craft of captures and vantage points has its own pillar. For today, carry away this one inoculation — it eventually saves every admin a very embarrassing escalation.
Driver counters beyond the basics
The standard interface counters — bytes, errors, discards, covered end-to-end in the host counters guide — are the public ledger. Drivers also keep a private one: dozens or hundreds of extended statistics with vendor-chosen names, and this is where offload trouble leaves fingerprints. On Linux the conventional window into it is the statistics option of the standard NIC tool:
$ ethtool -S eth0 | grep -i -e err -e drop -e restart
rx_csum_offload_errors: 41288
tx_restart_queue: 912
rx_no_buffer_count: 3
tx_timeout_count: 2
rx_long_length_errors: 0
Names differ per vendor, but the reading strategy is universal. Anything containing error, drop, timeout, restart, or reset that is moving deserves attention. A climbing checksum-offload error count means the card is detecting (or causing) bad sums in real traffic. Queue restarts and transmit timeouts mean the driver is stalling and recovering. That is a driver quietly crashing in miniature, many times a day, without a single line in the application logs. On Windows, vendors expose their extended sets as additional performance counter objects and adapter status pages. The standard-counter layer is covered in Windows network performance counters, and whatever your vendor adds sits alongside it. Check the OS event logs too: repeated "adapter reset" or "link flap" messages with no cabling explanation are the driver confessing.
Can you monitor these extended counters continuously? Sometimes, and selectively. Standard SNMP and WMI collection usually stops at the public ledger. Reaching the private one generally means an agent or a scheduled script (over SSH on Linux). It reads the driver statistics and reports a handful of chosen values. Do not ship all three hundred into your monitoring system. Pick the few that have burned you or that the reset-and-timeout family covers, and watch those as rates. The rest can stay on the card until a troubleshooting day, now that you know where they live.
Safe experimentation, one toggle at a time
Suppose the symptoms fit and you suspect an offload. The internet's favorite advice — "just turn all the offloads off" — is a blunt instrument. It changes many variables at once, costs permanent CPU, and teaches you nothing about which chore was broken. Do it like an experiment instead:
- 1. Define one measurement. Pick a single number that captures the pain and is repeatable. For example: "copying this specific test file to that server takes N seconds," or "this query returns in N milliseconds." No measurement, no experiment — just vibes.
- 2. Record the baseline. Run the measurement a few times with everything as-is; note the numbers, the time, and the current settings.
- 3. Get a maintenance window. Toggling offloads can blip the interface for a moment, and on a teamed or production NIC a blip is an event. Windows exist for this.
- 4. List the current state, and save it. You want the exact before-picture to restore later.
- 5. Change exactly one offload. Start with the one that matches the symptom. Bulk sends slow → segmentation offload. Capture-visible checksum weirdness confirmed at the far end → checksum offload. Latency-sensitive receive trouble → coalescing.
- 6. Re-run the same measurement. Dramatically better? You have found your suspect. No change? Revert the toggle — leaving it off buys nothing and costs CPU — and try the next candidate.
- 7. Write down everything — what you toggled, when, the before/after numbers. This log is the difference between troubleshooting and wandering.
On Linux the toggles are the feature flags of the same standard tool (ethtool -k eth0 lists them; ethtool -K eth0 tso off style commands change one). On Windows they live in the adapter's advanced properties and the equivalent PowerShell cmdlets. Either way the vocabulary is the same three chores you now know by name.
Warning: a disabled offload that makes the problem vanish is a diagnosis, not a cure. The cure is a fixed driver or firmware from the vendor. The disabled offload is a documented workaround with a CPU price tag. Note it, report it to the vendor, and revisit after the next driver release. Otherwise your fleet slowly accumulates mystery settings nobody remembers the reason for.
Keeping drivers and firmware sane
Most offload misery is avoidable with unglamorous hygiene. The principles are evergreen:
- Stability over novelty. On a server, the best driver is rarely the newest one. It is the one the vendor recommends for your card that has been boring for a long time. Update for a reason — a fixed bug that bites you, a security advisory — not for recreation.
- Driver and firmware are a couple. The card runs firmware; the OS runs the driver; vendors test them in matched pairs. Updating one far ahead of the other is how you manufacture the very bugs this article is about. Follow the vendor's pairing guidance.
- Change one thing, then watch. Never bundle a driver update with an OS upgrade and a settings change. When something weird starts a week later, you will want to know which lever moved.
- Document versions and settings. A simple record per server — NIC model, driver, firmware, non-default settings, and why — turns "did anything change?" from archaeology into a lookup.
- Keep the fleet consistent. When your similar servers share NIC models and driver versions, one server behaving differently becomes a signal. A zoo of one-off combinations hides everything.
- Keep the rollback path warm. Know where the previous known-good driver lives and how to reinstall it before you need to at 2 AM.
When to suspect the driver: a checklist
Pin this mental checklist next to your other troubleshooting reflexes. The more rows that match, the harder you should look at the driver and its chores:
| Symptom | Why it points here |
|---|---|
| Throughput poor, but link clean at both ends | The wire is fine; the damage is in the data path software |
| Problem follows the server across ports and cables | Drivers travel with the server; bad cables do not |
| Only one app, protocol, or traffic type affected | Offloads apply selectively; wires do not |
| Trouble only with VLAN-tagged, tunneled, or VM traffic | Encapsulation paths are where offload bugs cluster |
| Capture on the host shows checksum errors or giant frames | Usually the cosmetic capture artifact — verify at the far end |
| Started after a driver, firmware, or OS update | The strongest hint there is; check the change log |
| Adapter resets or transmit timeouts in system logs | The driver is stalling and recovering behind your back |
| Toggling one offload clearly changes behavior | That is not a hint; that is a confirmation |
And the mirror-image rule, to keep you honest: the switch port may show climbing physical errors. Or the problem may stay with the cable path when the server moves. In either case, it is not the driver — go back to the physical-layer playbook. Drivers deserve suspicion for weird problems, not all problems.
Where to go next
You now hold the decoder ring for the strangest tickets in server networking. You know the offloads and their chores, the selective way they fail, and the capture artifact that fools everyone once. You know the extended counters where drivers leave fingerprints. You also have a one-toggle-at-a-time method that turns spooky symptoms into a named culprit. The habits — matched driver/firmware pairs, one change at a time, write everything down — are what keep you from ever needing the rest.
One natural companion to this guide is the host counters guide, which covers the standard ledger these extended counters extend. Another is the virtual NIC story, where offloads meet hypervisors and get even stranger. The rest of this pillar lives at Where Server Meets Network: NICs, Teaming & Drivers.
Put this into practice with SysPeep Monitor
No monitoring tool can fix a driver — but the before/after evidence lives in monitoring. SysPeep Monitor graphs per-interface throughput, errors, and CPU from your servers over WMI, SNMP, or SSH. When you toggle one offload in a window, you can see exactly what changed and prove it. The free trial runs on any Windows machine.
Download Free TrialLearn About SysPeep