Home › Topics › QoS: Designing, Verifying & Monitoring It › Proving QoS End to End

Proving QoS Works End to End


"QoS is configured" and "QoS works" are different sentences. Sooner or later someone with budget authority will ask which one you mean. The configuration is easy to show. The proof is harder. That is because a QoS policy only does anything when a link is congested. Links are rarely congested on demand while you happen to be watching. So most networks run for years on QoS that has been configured, inspected, and never once observed doing its job.


This guide is the test that settles it. You will deliberately fill a link and send small marked probe streams through it in each class. You will measure what happens to each one. If the policy works, the priority stream comes through clean while the best-effort stream suffers. Then you will have numbers to show for it. If it does not, the same test tells you which hop failed. You will also learn how to run it without breaking anything. You will learn how to package the results for the people who asked. You will also learn why the test has to be repeated after every change to the path. If you have not yet met the four QoS jobs, start with QoS in Plain Language.


In this guide

  • The proof method: a flood and a witness
  • Building the witness: marked probe streams per class
  • The test plan: five runs, one afternoon
  • Reading the results: what "working" looks like
  • Hop by hop: marks survive, queues engage
  • The before-and-after evidence pack
  • Scheduling the test safely
  • Re-verify after every change
  • Where to go next

The proof method: a flood and a witness


The test has two actors. The flood is a deliberate attempt to fill the link. It is a throughput test between a host on each side, pushing more traffic than the link can carry. The traffic is unmarked. So it lands in best effort like any backup would. Its job is to create the line that QoS exists to manage. Running a throughput test without harming anything is a craft of its own, covered in Throughput Testing Done Right. Here it is simply the crowd at the checkpoint.


The witness is a small, steady probe stream that crosses the same link at the same time. It is marked with the DSCP value of the class you are testing. It runs at roughly the rate of a real phone call. The receiving side counts what arrives and checks sequence numbers for loss. It measures how evenly the packets arrive (jitter) and how long they took (latency). The witness is a stand-in for a voice call that you can measure precisely and repeat identically. You cannot do that with a real call.


The claim you are testing is simple: under flood, a witness marked for the priority class stays clean while a witness marked best effort does not. Both halves matter. If you only run the priority witness and it comes through clean, you have shown nothing. That is because perhaps the link was never really full. The best-effort witness is the control: it proves the flood hurt. It also proves that the only difference between suffering and not suffering was the mark.


Building the witness: marked probe streams per class


Most throughput tools have a UDP mode that sends at a fixed bit rate and reports loss and jitter at the receiver. Most allow you to set the DSCP or the whole type-of-service byte on the packets they send. That is the witness. Choose parameters that resemble one call: small packets, fifty per second, around a hundred kilobits per second, for sixty seconds. Run the receiver on a host beyond the congested link, so the stream actually crosses it in the direction under test. In neutral command form:


# Far side: a receiver for witness streams
probe-recv --udp --port 5201 --report-interval 1

# Near side, run 1: witness marked EF (DSCP 46), call-like rate, 60 seconds
probe-send --udp --to 10.20.5.50 --port 5201 --rate 100k --size 200 --dscp 46 --time 60

# Near side, control: the identical witness marked best effort (DSCP 0)
probe-send --udp --to 10.20.5.50 --port 5201 --rate 100k --size 200 --dscp 0 --time 60

# The flood: unmarked TCP, several parallel streams, to a different test host
flood-send --tcp --to 10.20.5.51 --parallel 8 --time 90

Some tools take the value as the whole byte instead (EF is 0xb8, AF31 is 0x68, CS1 is 0x20). Check which form yours expects, because a value in the wrong form silently marks something else. A marked ping is a useful second witness: many ping utilities let you set the type-of-service byte. And a marked ping running alongside the flood gives you round-trip latency per class with no setup. Use dedicated test hosts at both ends, not production servers, and confirm before the window that a witness with no flood arrives clean. That no-load run is your baseline and everything else is compared to it. Run the witnesses one class at a time rather than all at once. A single stream per run keeps the result unambiguous. The whole plan still fits in an afternoon.


Tip: the sending host must actually put the mark on the wire. Some operating systems ignore an application's requested DSCP unless a policy permits it. A test host on an untrusted switch port will have its mark rewritten to 0 before the first hop. Capture the witness leaving the sender once, before the test, and confirm the DS field is what you asked for. A test that fails because the witness was never marked proves nothing about the policy.


The test plan: five runs, one afternoon


Five short runs cover everything you need to prove, and they take well under an hour including setup. Each run is one witness, with or without the flood, for sixty to ninety seconds. Record loss, jitter, and latency for the witness, and the throughput the flood achieved.


RunWitness markFloodExpected resultIf it fails, it means
1. BaselineEF (priority)NoneClean: no loss, low jitter, latency at the path's normal valueThe path has a problem unrelated to QoS: errors, Wi-Fi, routing. Fix that first.
2. The proofEF (priority)YesStill clean, latency within a few milliseconds of baselineMarks are not reaching the queue, or the queue is not engaging. Go hop by hop.
3. The control0 (best effort)YesVisible loss, high jitter, latency well above baselineThe link was not actually congested: flood too small, or it took a different path. The test proved nothing yet.
4. The middle classAF31 (business-critical)YesBetter than run 3, worse than run 2: little loss, moderate jitterThe business class is not being classified; it is falling into best effort.
5. The negative testEF, with the flood also marked EFYes, marked EFWitness suffers, and the priority class shows policer drops on the routerThe priority cap is not enforced; a mis-marking application could starve the whole link.

Run 5 deserves a warning label. It deliberately oversubscribes the priority queue, which means any real call in progress during those ninety seconds will be damaged. Run it only in a window with no calls, keep it short, and skip it entirely on a first pass if you cannot guarantee an empty phone system. Its value is that it proves the cap exists. And it shows the people watching exactly what an over-eager application marking everything EF would do to their phones.


Reading the results: what "working" looks like


The receiver reports three numbers per run. "Working" is a relationship between the runs rather than any single value. Loss is the share of witness packets that never arrived. Jitter is how much the gap between arriving packets wandered from the steady rate they were sent at. A queue that sometimes holds a packet and sometimes does not produces jitter. Latency is the time the packets took. A deep queue adds delay to everything that waits in it. That is why best effort under flood is not just lossy but slow. Most tools report one-way delay only if both hosts' clocks are tightly synchronized. So unless you have arranged that, use the round-trip time from the marked ping. In that case, compare runs against each other rather than against an absolute number.


A working policy produces a clear pattern. The priority witness under flood looks almost identical to the baseline. Loss is at or near zero, jitter is small, and latency is a few milliseconds above baseline at most. That is because it waited only behind other priority packets. The best-effort witness under flood is visibly worse on all three, and often dramatically worse on latency. That is because it sat in the deep queue with the flood. The business-critical witness lands in between. And the flood's own throughput settles close to the shaped rate, which is its own small proof. It shows the line formed on your router, where the policy lives, rather than at the provider. If the flood achieves more than the shaped rate, the shaper is not on this path. The absolute values that count as "clean enough" for voice are laid out in QoS for real-time traffic. Those values specify how much delay, jitter, and loss a call tolerates. Compare run 2 against those, and run 2 against run 3 for the proof that the policy is what made the difference.


Three failure patterns cover nearly every disappointing result. If the priority witness suffers exactly like the best-effort one, the mark is not reaching the queue or the policy is not applied where the line forms. The hop-by-hop walk below finds which. If every witness is clean, including best effort, the link was never congested. In that case, the test needs a bigger flood or a check that the flood used the same path. If everything is bad, including the no-flood baseline, the path has a problem QoS cannot fix. In that case, you have learned something more urgent than what you set out to learn.


Hop by hop: marks survive, queues engage


When run 2 fails, or simply to make the proof complete, watch the witness at each hop while the flood is running. There are two questions at every device: did the mark arrive intact, and did the queue do something with it?


  • Marks, at the LAN side of the WAN router. Capture the witness there and read its DS field. It should still carry the value you sent. If it is 0, a switch between the sender and the router rewrote it, and DSCP Markings in the Wild lists where to look.
  • Queues, on the WAN router. Read the per-class counters during the flood. The priority class's matched counter should climb at the witness rate, around fifty packets a second, with zero drops. Best effort's drop counter should climb steadily; that is the flood being held back. A priority class that matches nothing while the witness runs is the classification failure in its purest form. Monitoring Queue Drops explains every counter on that screen.
  • Marks, on the WAN side. Capture the outside interface. For a tunnel, this is where you see whether the inner mark was copied to the outer header; for a plain circuit, this is the last point you control.
  • Marks and queues, at the far site. Check what arrived. On the public internet expect 0, and confirm the far site's fallback classification handles the return direction anyway. On a private WAN, check whether the provider preserved the value it promised to honor.

Your monitoring system's interface graphs are a useful fifth witness here. During each flood run the WAN interface should climb to the shaped rate and the output discards should start counting. That is the picture of a queue forming exactly where you intended. Then turn around and run the whole plan from the far site toward you. The return direction is protected by the far router's policy, not yours. "I can hear them but they can't hear me" is what a one-directional QoS deployment sounds like. A path is only proven when both directions have a clean run 2 and a suffering run 3.


The before-and-after evidence pack


The people who asked whether QoS works do not want the counters; they want one page. Give them the same five runs twice: once with the policy removed from the WAN interface, once with it applied. Use the same window with the same flood. The table below is the whole pack. Fill in your numbers and add a sentence of method. Attach the router's per-class counter output from run 2 as the appendix nobody reads but everybody is reassured by.


Witness classLoss, no policyJitter, no policyLatency, no policyLoss, with policyJitter, with policyLatency, with policy
Priority (EF), no flood      
Priority (EF), under flood      
Business-critical (AF31), under flood      
Best effort (0), under flood      
Flood throughput achieved  

Under the table, three short lines finish the job. State what was proven: the priority class stayed clean under a full link, in both directions, on this path. State what was not: inbound from the internet is not under your control, and upstream Wi-Fi marking depends on the client device. And state what to do if the numbers ever move: the test plan, scripted, so the next run takes fifteen minutes. That last line is what turns a one-time demonstration into an operational practice.


Scheduling the test safely


A flood is, by design, the thing your policy exists to survive, and the best-effort class will genuinely suffer while it runs. That is fine in a window and rude at ten in the morning. The checklist:


  • 1. Pick a window when best-effort suffering costs little, but keep a real user or a real phone available to confirm what the witness reports.
  • 2. Announce it, including the fact that web and file traffic will crawl for a few minutes. A flood that surprises the help desk becomes an incident.
  • 3. Keep runs short. Sixty to ninety seconds per run is plenty to fill queues and gather statistics; a long sustained flood can also resemble an attack to a provider's automated protections.
  • 4. Use dedicated test hosts at both ends. Never make a production server the flood endpoint.
  • 5. Check the circuit's terms. A metered link or a cellular backup path is not where you run floods. Know which path the test will take.
  • 6. Watch the priority counters live during every run, and stop immediately if the priority class starts dropping in any run other than the negative test.
  • 7. Reserve the negative test for a window with no calls in progress, or skip it.
  • 8. Write down the abort step before you start: stopping the flood tool is usually enough, and removing the policy from the interface is the fallback.

Re-verify after every change


A passing test describes one path on one day, and the path does not stay still. Each of the following has quietly undone a working policy on real networks. And each is a reason to run the five-run plan again before anyone reports a problem:


  • New firmware on a switch, router, or firewall. Default trust behavior and marking defaults change between releases, and the release notes rarely mention it.
  • A new or upgraded circuit. The shaper is set to the old rate. Either the new capacity is wasted, or, after a downgrade, the queue has moved back to the provider and QoS has stopped engaging entirely.
  • A new provider. Different handling of DSCP, a different set of honored classes on a private WAN, and a different policer at their edge.
  • A new site, which needs the mirror-image policy and its own two-direction proof.
  • A new tunnel, switch, access point, or proxy on the path: every one is a potential mark-stripper with its own defaults.
  • A phone or conferencing system change. New endpoints mark differently, or not at all, and the classes match nothing.
  • Any edit to the policy itself. A retuned share, a new class, a changed cap. The edit that was supposed to help is the one most likely to have broken something else. And the five runs are the only way to know.

Script the plan so that a re-run is a fifteen-minute chore rather than an afternoon. Run it on a schedule a few times a year even when nothing has changed. That is because "nothing has changed" is the most common false statement in networking. Keep every result next to the last one. Watch for a slow drift in run 2's latency, or a run 3 that hurts less each time because the flood no longer fills the link. Either tells you about the circuit and the workload long before anyone complains.


Where to go next


Proof is a flood and a witness: fill the link on purpose and send a marked probe through it in each class. Show that the priority class stays clean while best effort suffers. The control run makes it evidence rather than luck. The hop-by-hop walk finds the failure when there is one. The negative test proves the cap. And the before-and-after table turns it all into one page. Then you run it again after every change to the path. A policy is only proven for the network that existed on the day of the test.


This is the last step in the series. The mechanics behind the test are in QoS in Plain Language. The policy it verifies is built in Designing a Simple QoS Policy for a Small Network. And the day-to-day counters that keep proving it between tests are in Monitoring Queue Drops. All of it lives on the QoS: Designing, Verifying & Monitoring It pillar page.


Put this into practice with SysPeep Monitor

A flood and a witness prove the policy on the day of the test. Monitoring proves it every day after. SysPeep Monitor polls your WAN routers and switches over SNMP. It graphs interface utilization and discards on one dashboard so you can watch the link fill while the test runs. It alerts when it fills for real. The free trial installs on a Windows machine in minutes.

Download Free TrialLearn About SysPeep