Monitoring Queue Drops: Seeing QoS Work (or Not)
You designed the classes, set the trust boundaries, shaped the WAN edge, and turned on the priority queue. The phones sound fine today. Is that because the policy is working, or because nobody has run a backup during business hours yet? A QoS policy that is never observed is a belief, not a fact. Beliefs about networks have a way of being tested at the worst possible moment.
The good news is that a QoS policy is one of the most observable things on a network. Every device that queues by class keeps counters: how many packets each class matched, how many it sent, and — the important one — how many it dropped. Read correctly, those counters tell you whether the classes are seeing traffic at all, and whether the priority queue is protecting what it should. They also tell you whether the link has quietly become too small for the day's work. This guide shows you where the counters live, what each one means in each queue, and how to graph them against the policy's allocations. It also shows how to alert on them without being paged every time a backup gets throttled.
In this guide
- The counters that exist, and where they live
- Reading per-queue output: an annotated example
- What drops mean in each queue
- Zero matches: when a class sees nothing
- Graphing per-class usage against its allocation
- Alert design: loud for priority, quiet for scavenger
- The counter interpretation table
- Where to go next
The counters that exist, and where they live
A device applying a QoS policy keeps a small set of numbers per interface, per direction, per class. The names vary by platform, but the ideas are universal:
- Matched (or classified) packets and bytes. How much traffic the classifier placed in this class. Sometimes shown as "pre-policy," meaning before any drops.
- Sent (or transmitted) packets and bytes. What actually left the interface from this class. Sometimes "post-policy."
- Dropped packets and bytes. Matched minus sent, roughly: traffic that was discarded because the queue was full, or because a priority-class policer said no. Some platforms split the two.
- Offered rate and sent rate. The same thing as a speed rather than a total, averaged over a short window. The gap between them is the drop rate in bits per second.
- Queue depth. How many packets are waiting right now, against the queue's maximum. A queue pinned at its maximum is a queue about to drop.
There are two ways to read them. The command line shows them for one interface on demand: perfect during a test, useless for a week of history. SNMP is the Simple Network Management Protocol, the question-and-answer protocol monitoring tools use to poll devices. It lets a monitoring system collect these counters on a schedule and graph them. Two kinds of SNMP data matter here. The standard interface table that every device supports includes an output discards counter per interface. It counts packets thrown away on the way out without any error. On most platforms, that is exactly what a full output queue produces. The output-discards counter is not per class, but it is universal, cheap, and available even on devices with no QoS configured. That makes it your early warning for congestion you did not know about. The per-class counters live in vendor-specific parts of the SNMP tree. Walk the device's QoS subtree once to find them. Expect the row indexes to change when you edit the policy, so re-discover after every change.
All of these are counters: running totals since the device booted or the policy was last applied. A single reading means nothing. Read twice, subtract, divide by the seconds between reads, and you have a rate. That arithmetic is what every graph in this guide is made of. Prefer the 64-bit versions where a device offers them. A 32-bit byte counter on a busy link wraps back to zero in minutes and produces nonsense graphs.
Reading per-queue output: an annotated example
Here is what the counters look like on a WAN router during a busy afternoon with a backup running. The format is neutral, close to what most platforms print. The policy is the four-class design from Designing a Simple QoS Policy: a priority class capped at a quarter of the link, guaranteed shares for business-critical and scavenger, best effort taking the remainder.
show qos interface wan0 outbound
Interface wan0, policy WAN-EDGE-OUT (outbound), shaper 95000 kbps
Class REALTIME (priority, limit 25% = 23750 kbps)
matched: 14,205,118 packets 3,011,485,016 bytes
offered: 2,940 kbps sent: 2,940 kbps
dropped: 0 packets 0 bytes
Class BUSINESS (guarantee 30%)
matched: 41,882,905 packets 38,617,220,410 bytes
offered: 18,310 kbps sent: 18,310 kbps
dropped: 1,204 packets 1,613,360 bytes
queue depth: 12 / 256 packets
Class SCAVENGER (guarantee 5%)
matched: 96,004,331 packets 140,102,455,900 bytes
offered: 61,200 kbps sent: 4,750 kbps
dropped: 4,118,377 packets 6,177,565,500 bytes
queue depth: 256 / 256 packets
Class BEST-EFFORT (remainder, fair-queue)
matched: 188,410,777 packets 201,556,913,204 bytes
offered: 71,900 kbps sent: 66,900 kbps
dropped: 402,118 packets 603,177,000 bytes
queue depth: 190 / 256 packets
Read it top to bottom and the afternoon tells its own story. The priority class is carrying a modest stream of voice, a fraction of its cap, with zero drops. Every voice packet that arrived left immediately. Business-critical is sending everything offered to it, with a handful of drops from bursts and a nearly empty queue: healthy. Scavenger is the backup, offering far more than the link can take. It is being held to exactly its guaranteed sliver, with its queue pinned full and millions of drops. That is not a problem; that is the design doing precisely what you asked. Best effort is feeling the squeeze, sending most of what it is offered but dropping some and running a deep queue. Add up the sent rates and you get roughly the shaped rate: the link is full, and the policy is deciding who waits.
Now the version of the same output you never want to see:
Class REALTIME (priority, limit 25% = 23750 kbps)
matched: 22,918,404 packets 4,858,701,648 bytes
offered: 27,150 kbps sent: 23,750 kbps
dropped: 391,220 packets (exceeded priority limit)
The offered rate is above the cap. The sent rate is pinned exactly at the cap. And the difference is being dropped by the policer that protects the other classes. Every one of those dropped packets was a piece of somebody's call. The cause is one of two things, and the next section is about telling them apart.
What drops mean in each queue
A drop is not good or bad on its own. It is good or bad depending on which queue it happened in. The whole point of the policy is to move drops out of the queues that cannot tolerate them and into the queues that can.
Drops in the priority queue: very bad
The priority queue is served first, so it only drops when its own traffic exceeds its cap. There are exactly two ways that happens. One possibility is that the real-time class is genuinely oversubscribed. That means more simultaneous calls or video streams than the cap was sized for. This often follows a new site, a new phone rollout, or a video system that was not there when the cap was set. The other is that something is mis-marked into it. Examples include an application labeling its screen sharing or file transfers EF, a device in "gaming mode," or a monitoring probe someone set to EF for a test and forgot. To tell them apart, compare the offered rate with what the real call count should need. Capture EF traffic at the router's LAN side to see who is actually sending it. Oversubscription means raising the cap (and, since the cap protects the other classes, probably the circuit). Mis-marking means fixing the source and tightening the trust boundary; DSCP Markings in the Wild lists the usual suspects.
Drops in the business-critical queue: investigate
Occasional drops with an offered rate below the guaranteed share are bursts hitting a short queue: harmless. Sustained drops with the offered rate above the guarantee mean one of two things. The share is too small for what the business actually does over this link. Or something bulky has been marked into the class that does not belong (nightly replication marked "important" is the classic). Look at what is in the class before you raise the share, because raising it to accommodate a mis-marked backup steals from best effort for nothing.
Drops in best effort under load: working as designed
When the link is full, best effort is where the waiting and the dropping are supposed to happen. Drops here during a backup or a big transfer are the policy protecting the classes above it. The exception is duration. If best effort drops heavily all day, every day, with the link pinned at the shaped rate, QoS is doing its job and the circuit is simply too small for the daytime workload. Users will report "the internet is slow" and they will be right. That is a capacity conversation, and these counters are the evidence for it.
Drops in scavenger: informational
Scavenger exists to be dropped first. Heavy drops during the day mean a backup or a sync client is being held back to make room, which is the entire purpose of the class. Two things are worth a glance anyway. One is whether the overnight window is long enough for the throttled jobs to finish. The other is whether scavenger shows no traffic while backups are running. In that case, the backups are unmarked and fighting it out in best effort.
Tip: drops on a link that averages 20 percent utilization are not a contradiction. Traffic arrives in bursts far shorter than any polling interval, and a small queue overflows during a burst while the five-minute graph shows calm. If a class drops with no visible congestion, look at queue depth and at utilization over the shortest interval your tools allow.
Zero matches: when a class sees nothing
The quietest failure in QoS is a class whose matched counter stays at zero, or barely moves, while the traffic it was built for is obviously flowing. Phones are in use all day and the real-time class has matched a few hundred packets. This is not a queuing problem; it is a classification problem, and it means the policy is protecting nothing. The causes, in order of likelihood:
- Marks are stripped before they reach this device. An untrusted switch port or uplink between the source and this router has rewritten them to 0. Everything lands in best effort.
- The class matches the wrong value. The phones mark EF, the class was typed to match something else; or one site uses AF31 for business-critical and the router here expects AF21.
- The policy is on the wrong interface or direction. Applied inbound when the congestion is outbound, or on the LAN interface instead of the WAN one.
- A tunnel is hiding the marks. The policy sees the outer header, which carries 0, because the inner DSCP is not being copied.
The diagnostic is the same walk used for stripped markings, but with counters instead of captures. Check per-class matches on each device along the path, from the access switch to the WAN edge. And find the hop where the class goes dark. That is where the marks stop arriving. Make "every class shows matches proportional to its expected traffic" a check you run after every change. And treat a class at zero during business hours as a fault.
Graphing per-class usage against its allocation
Counters answer "is something wrong right now." Graphs answer the questions that actually drive decisions. Is the priority cap still big enough, is the business share sized right, and is the circuit itself running out? Three graphs per policed interface cover it:
- Per-class sent rate, stacked. All classes on one graph, stacked to the shaped rate. This shows the composition of the link over the day. When the top of the stack flattens at the shaped rate, the link is full, and you can see which class is getting squeezed.
- Offered rate against the allocation, per class. For the priority class, offered rate against the cap: the gap is your headroom, and it should stay comfortably positive. For guaranteed classes, the graph shows offered rate against the guarantee. A class that regularly offers more than its guarantee during congestion is either under-allocated or over-populated. The graph tells you when to look.
- Drop rate per class. Drops per second, or dropped as a fraction of offered, per class. Priority should be a flat zero. Scavenger should be noisy during the day. Best effort should be noisy only when the stack is full.
Two habits make these graphs trustworthy. Poll at the shortest interval the device tolerates on the interfaces that matter, because a one-minute sample catches bursts a five-minute sample averages away. And draw the allocations on the graphs as reference lines, so "offered is above the guarantee" is visible at a glance. Keep months of per-class history. The priority class's slow climb toward its cap is the earliest warning that the circuit needs to grow. And that climb is invisible in a week of data.
Alert design: loud for priority, quiet for scavenger
The temptation is to alert on "drops greater than zero" for everything. That pages someone every time a backup gets throttled and trains everyone to ignore QoS alerts within a week. Severity must follow the queue. A drop in the priority queue is a damaged call and deserves attention now. A drop in scavenger is the design working and deserves a graph. Alert on rates derived from the counters, never on cumulative totals. Require the condition to persist for a short window so a single burst does not page anyone.
| Condition | Severity | Why |
|---|---|---|
| Priority-class drops above zero, sustained for a minute | High | Every drop is a piece of a call. Oversubscribed or mis-marked; either way, now. |
| Priority-class offered rate above roughly four-fifths of its cap | Warning | Headroom is running out. Find the new traffic before it becomes drops. |
| Business-critical drops sustained during business hours | Medium | Share too small, or something bulky mis-marked into the class. |
| Real-time or business-critical class matching nothing during business hours | Warning | Classification is broken; the policy is protecting nothing. |
| Best-effort drops sustained across most of the working day | Low | Capacity planning input, not an incident. QoS is working; the circuit is small. |
| Scavenger drops | None (graph only) | The design working as intended. |
| Output discards rising on an interface with no QoS policy | Warning | A queue is forming somewhere you have no policy. A congestion point you did not know about. |
| Per-class counters reset to zero | Informational | The device rebooted or the policy was edited. Useful context for the change log. |
Two refinements keep the list quiet without making it blind. Tie the business-critical and best-effort alerts to a schedule, because drops during the overnight backup window are expected and drops at ten in the morning are not. And add a dependency so that when the WAN interface itself is down, the QoS alerts for it are suppressed rather than piling on top of the outage alert. A monitoring system with persistence windows, schedules, and dependencies can implement this whole table. One that cannot needs the priority-drop alert at minimum, because that is the one that matters.
The counter interpretation table
When you are looking at a screen of per-class numbers and need a verdict, use this table. Read the offered rate next to the drop counter every time; drops without the rate are half a fact.
| What you see | What it means | What to do |
|---|---|---|
| Priority: matches climbing, drops zero | Real-time traffic is flowing and fully protected | Nothing. This is the goal. |
| Priority: drops above zero, sent rate pinned at the cap | Oversubscribed or mis-marked into the class | Compare offered rate with expected call load; capture EF at the LAN side to find the sender |
| Priority: matches near zero while phones are in use | Marks not arriving, or wrong value in the class | Walk per-class matches hop by hop; check trust on ports and uplinks |
| Business: occasional drops, offered below guarantee | Bursts hitting a short queue | Nothing, or lengthen the queue slightly if the application is sensitive |
| Business: sustained drops, offered above guarantee | Share too small, or a bulk transfer mis-marked into the class | Inspect what is in the class before raising the share |
| Best effort: drops during backups or transfers | Working as designed | Nothing. Confirm scavenger is absorbing the backup. |
| Best effort: drops most of the day, link pinned at the shaped rate | The circuit is too small for the daytime workload | Capacity conversation, with these graphs as the evidence |
| Scavenger: heavy drops, queue pinned full | Bulk traffic being held back to protect the rest | Nothing, unless overnight jobs stop finishing on time |
| Scavenger: no matches while backups run | Backups are unmarked and competing in best effort | Mark them at the backup server's switch port |
| Any class: matches zero on this device, non-zero upstream | Marks stripped between the two devices | Fix trust on the link between them |
| Output discards rising, no policy on the interface | Hidden congestion point | Decide whether it needs policy or capacity |
| Queue depth pinned at maximum | Sustained congestion; drops imminent or already happening | Read the drop counter next; check offered rate against allocation |
| Drops on a link averaging low utilization | Micro-bursts overflowing a short queue | Poll at shorter intervals; consider a longer queue for that class |
Where to go next
A QoS policy explains itself through its counters, if you know which queue you are looking at. Matches tell you whether classification is reaching the device. Drops tell you who is waiting when the link is full. The queue the drops appear in tells you whether that is the design working or failing. Priority drops are an incident, business-critical drops are an investigation, best-effort drops under load are the plan, and scavenger drops are a graph.
Counters show the policy working on a busy afternoon; they do not prove it will hold when you need it to. For that, generate the load yourself and measure each class while it runs, which is the subject of Proving QoS Works End to End. The mechanics behind every counter in this guide are explained in QoS in Plain Language. And the whole series lives on the QoS: Designing, Verifying & Monitoring It pillar page.
Put this into practice with SysPeep Monitor
The counters in this guide only help if something is reading them every minute, all week, and telling you when the priority queue starts dropping. SysPeep Monitor polls your routers and switches over SNMP and graphs interface traffic and discard counters. It lets you set thresholds and alert rules with persistence windows and dependencies. That way, the scavenger queue never pages you and the priority queue always does — all from one web console.
Download Free TrialLearn About SysPeep