Proving Where Call Problems Live: LAN, WAN, or Provider
"The calls are bad" is where the argument starts. The phone team says it is the network. The network team says it is the provider. The provider says everything looks fine on their side and would you like to check your firewall. Everybody is sincere, nobody has evidence, and the calls stay bad.
Voice problems are unusual because three different parties can own the fault. They are your LAN (the switches and cabling inside a building), your WAN (the links between your sites), or the provider on the far side of the trunk. The way out is not louder arguing. It is isolation: shrinking the problem to one segment, with measurements taken on each side of it, until only one suspect remains.
This guide is that playbook. You will scope the complaint properly. You will use the fastest split there is (internal calls versus external calls). You will gather evidence segment by segment. You will reproduce the problem with synthetic probes. You will build an evidence file the provider cannot wave away, and have the conversation with them. The guide assumes the mental model from What Makes VoIP Different: signaling and media take different paths, and loss, jitter, latency, and a quality score describe a call.
In this guide
- Step 1: Scope before touching anything
- The fastest split: internal versus external calls
- Evidence from the phone's switch port
- Evidence from the voice VLAN and the LAN core
- Evidence from the WAN for site-to-site calls
- Evidence from the trunk and the provider
- Reproducing with synthetic probes
- The evidence file for the provider
- Decision table: symptom patterns to segments
- The conversation with the provider
- Where to go next
Step 1: Scope before touching anything
The urge is to start looking at switches. Resist it for five minutes, because the shape of the complaint is the first piece of evidence and it is free. Ask, and write down the answers:
- Who? One user, one floor or switch, one site, or everyone?
- Which calls? Internal on the same site, between sites, external inbound, external outbound, or all of them?
- Which direction? Could they hear the far end? Could the far end hear them? Media flows separately in each direction, and one bad direction is a strong clue.
- When? Constant, at certain times, or since a specific day? "Since Tuesday" means "what changed Tuesday?"
- What kind of bad? The words users choose map surprisingly well onto causes.
| What the user says | What it usually means | Where to look first |
|---|---|---|
| "Choppy," "cutting out," "words missing" | Packet loss, usually in bursts | Errors and discards on the path; congestion |
| "Robotic," "underwater," "like a bad radio" | Jitter beyond the buffer, with the phone concealing the gaps | Queuing on a congested link; QoS not applied |
| "We keep talking over each other," "long pauses" | High latency | A long or congested path; a large jitter buffer; the WAN |
| "Echo" | Latency exposing an acoustic or analog reflection | The far end's equipment, then latency on the path |
| "I can hear them but they can't hear me" | One-way audio: media blocked in one direction | Firewall, NAT, or routing for RTP |
| "It drops after exactly a few minutes every time" | A timer: a NAT or firewall session expiring, or a SIP session refresh failing | Firewall session timeouts; SIP timer settings |
| "It won't connect," "no ringing," "phone says no service" | Signaling, not media | Registration, the PBX, the trunk |
Finally, get the currency of everything that follows: two or three specific bad calls, with the time, the extension, and the number dialed. From those the PBX log gives you the call IDs. A call ID is what turns "the calls are bad" into a fact you can look up on every device along the path, including the provider's.
The fastest split: internal versus external calls
Remember where media goes. On an internal call between two phones on the same site, the audio flows phone to phone across the LAN. It never touches the WAN, the trunk, or the provider. On a site-to-site call it crosses both LANs and the WAN. On an external call it leaves through the PBX or border device, out the internet link, over the trunk, and into the provider's network. There are three kinds of call, with three different sets of segments. That means three test calls from the complaining phone are the most informative thing you can do:
- 1. Call a phone on the same switch. Talk for a minute, in both directions. Note how it sounds.
- 2. Call a phone at another site. Same test.
- 3. Call an outside number, ideally a mobile, and have the far end call you back.
Then read the pattern. All three bad: the phone, its cable, its port, or its access switch. Only the site-to-site and external calls bad: the site's uplink, its edge, or the WAN. Only the external call bad: the internet link, the firewall, the trunk, or the provider. Only the inbound external call bad, or only one direction: something direction-specific, such as NAT, the asymmetric internet link, or the provider's route toward you. Two minutes of testing, and most complaints are already narrowed to one segment.
Evidence from the phone's switch port
When the scope is one phone or one small group, the first segment is the access port. Find it from the switch's neighbor table or the MAC address table, and pull its counters. Most phones announce themselves over LLDP, the Link Layer Discovery Protocol. Errors such as CRC failures point at cabling or a duplex mismatch. Discards point at congestion on the port. A link that has gone up and down repeatedly points at a cable or a power problem. And check the PoE state and draw. A phone that reboots mid-call because the switch is short of power looks like a call drop. The guide PoE Power Budgets explains how a switch runs out of watts.
The trick is to read the counters twice, before and after a bad call, because a counter that is merely large may have been large for a year. One that grows during the call is the culprit:
# before the test call snmpget -v2c -c labread sw-floor2 IF-MIB::ifInErrors.14 IF-MIB::ifInDiscards.14 IF-MIB::ifOutDiscards.14 IF-MIB::ifInErrors.14 = Counter32: 18220 IF-MIB::ifInDiscards.14 = Counter32: 0 IF-MIB::ifOutDiscards.14 = Counter32: 41 # after a two-minute call that sounded choppy IF-MIB::ifInErrors.14 = Counter32: 18391 IF-MIB::ifInDiscards.14 = Counter32: 0 IF-MIB::ifOutDiscards.14 = Counter32: 41
A hundred and seventy new input errors in two minutes on the phone's port is a cabling or duplex problem. No amount of provider tickets will fix it. Remember also the PC plugged into the back of the phone. The phone has a tiny switch inside, and a PC pushing large transfers through a port negotiated at a low speed can starve the phone's own traffic.
Evidence from the voice VLAN and the LAN core
When several phones on a site suffer, look one layer up. First, confirm the phones are actually in the voice VLAN. A phone that landed in the data VLAN shares a broadcast domain with every PC and gets none of the QoS treatment designed for it. Second, confirm that QoS is doing its job. Confirm that voice packets are marked, the access switch trusts the marking, and the uplinks give the voice queue priority. Pull the queue-drop counters on the uplinks if the switch exposes them; drops in the voice queue during the complaint window are conclusive. Third, look at uplink utilization at the exact minutes of the bad calls, and at the spanning-tree topology-change count. A loop or a flapping uplink causes exactly the kind of brief, severe loss that users describe as "it cut out for ten seconds."
The evidence you want from this segment is a pair. One part is a counter or utilization graph that misbehaves at the cluster's minutes. The other is a probe between two points on the same site that shows loss at the same minutes. Together they say "the LAN" with a specific device attached.
Evidence from the WAN for site-to-site calls
If same-site calls are clean and site-to-site calls are not, the WAN is the suspect and it has a short list of ways to fail. Utilization on the link, in both directions, at the cluster's minutes — a link saturated by a replication job will queue voice behind bulk traffic unless QoS holds a place for it. The QoS policy at the WAN edge: is voice in the priority queue, is that queue large enough for the number of concurrent calls, and is it dropping? The tunnel, if the WAN rides a VPN: tunnel devices under CPU strain drop small packets first, and voice packets are small. And the latency, jitter, and loss reported by the synthetic probe on that site pair, compared with the same probe on a healthy pair.
One reminder that saves days: the circuit between sites is usually supplied by a different provider than the voice trunk. If the WAN probe shows loss while both LANs are clean, the conversation belongs with the circuit provider. In that case, the evidence file described below works just as well for them.
Evidence from the trunk and the provider
If only external calls are bad, the path is the PBX or border device, the firewall, the internet link, the trunk, and the provider. Take them in order. The trunk's own heartbeat first: was it registered, and what was the OPTIONS round-trip time doing at the time? A keepalive that normally answers in 20 ms and was answering in 190 ms during the complaint window is a path problem with a timestamp. Then the internet link: utilization in both directions, especially upstream, and error counters. Then check the firewall: session table near its limit, CPU pegged. Also check the feature that causes more one-way audio than any other, the SIP ALG (Application Layer Gateway). This is a firewall function that rewrites SIP messages to "help" with NAT and frequently mangles them instead. Most voice engineers disable it. Then the PBX: was it transcoding between codecs, which costs CPU and quality?
The most useful evidence on this segment is per-direction. In one case, the stream you sent left your network cleanly (your side's counters and probe say so). Yet the provider's edge reports receiving it with loss. Or the stream they sent arrived at your edge already damaged (your border device's receive statistics say so). The provider's edge is the frontier. Your measurements stop there and theirs begin, so for external calls you will always need their view of the same call IDs to finish the job.
Tip: keep an internal call running while you test an external one, from phones on the same switch. If the internal call stays clean at the very minutes the external one degrades, you have just ruled out the LAN in real time. In that case, that sentence belongs at the top of the evidence file.
Reproducing with synthetic probes
A synthetic probe is a simulated call. Two agents exchange a voice-like packet stream, marked exactly as real voice is. They measure loss, jitter, and latency continuously. For isolation, place them so that each probe covers exactly one segment. Use two agents on different access switches of the same site for the LAN. Use one agent at each site for the WAN. Use an agent at the internet edge sending toward the provider's edge, or the trunk's OPTIONS round trip standing in for it, for the external path. Run them through the complaint window and compare. The first path that degrades names the segment. A path that stays clean while calls on it are bad says the problem is not the path at all but the phones, the PBX, or the codec.
Probes matter most for intermittent problems, which are most of them. You cannot ask a user to place a test call at four minutes past nine every morning. But a probe will, forever, and it will still be watching the day the problem finally recurs. In a pinch, a continuous ping with small packets, marked with the same DSCP value as voice, approximates the voice queue well enough to catch a saturation event. DSCP is the header field QoS uses to recognize voice packets. The ping is not a voice probe, but it is running now.
The evidence file for the provider
A provider ticket that says "our calls are bad" gets the reply "everything looks fine here." A ticket with an evidence file gets an engineer. The file is short, factual, and organized so that their engineer can look up the same calls you did. The Exoneration Evidence guide covers the general craft of proving a network innocent; this is the voice-specific version.
Account / trunk: provider-main, trunk ID 4471-A
Symptom: choppy audio heard by the OUTSIDE party; inside party hears fine
Scope: all outbound external calls, all sites; internal calls clean
Window: 09:00-09:40 local (UTC-5), Tue; recurring 3 of last 5 weekdays
Sample calls (SIP Call-ID, local time, from -> to, direction):
a83f2c...@pbx.example 09:04:12 ext 204 -> +1555xxx0142 outbound
b1049e...@pbx.example 09:11:47 ext 318 -> +1555xxx7710 outbound
c77d10...@pbx.example 09:26:03 +1555xxx0080 -> DID 5500 inbound (clean)
Our measurements in the window:
Internal calls, same switches, same minutes: loss 0.0%, jitter 3 ms
LAN probe (floor2 -> floor4): loss 0.0%, jitter 2 ms
Internet link utilization (upstream): 31% peak; errors 0
Trunk OPTIONS round trip: baseline 21 ms; window 24-190 ms
Our sent stream (RTCP from PBX, outbound legs): 0 loss reported by our side
Far-end reports (RTCP receiver reports): 4-9% loss on outbound legs
Firewall: session table 38% of limit, SIP ALG disabled, CPU 22%
Ruled out: LAN (internal calls and probe clean), PoE, VLAN, upstream saturation
Request: media statistics from your edge for the three Call-IDs above;
loss/jitter on your route toward our public IP in the window;
any route or edge changes on your side since last week
Every line is a measurement, a timestamp, or a call ID. Nothing is an adjective. Note the inbound call included as a clean control, and the "ruled out" line that pre-empts the firewall question. Note, too, that the far-end receiver reports say the outbound audio arrived at the provider's side with loss while your side saw none. That is precisely the frontier, and the file asks them for the one thing only they can see.
Decision table: symptom patterns to segments
| Pattern | Most likely segment | First evidence to pull |
|---|---|---|
| One phone, every call bad | The phone, its cable, or its port | Port errors before and after a call; PoE state; swap the cable |
| One floor or switch, mixed calls bad | The access switch's uplink or PoE budget | Uplink utilization and discards; PoE headroom |
| One site, internal calls bad too | That site's LAN core or voice VLAN | Core uplink drops; topology changes; VLAN membership; QoS trust |
| One site, only site-to-site and external bad | The site's WAN link or edge device | WAN utilization both ways; QoS queue drops; probe on that site pair |
| All sites, only external bad | Internet link, firewall, trunk, or provider | OPTIONS round trip; upstream utilization; RTCP per direction; firewall state |
| Only inbound external bad | Provider's route toward you, downstream link, or NAT | Downstream utilization; your border's receive statistics; provider's edge view |
| Only outbound external bad | Upstream link or QoS on the edge | Upstream utilization at the minutes; far-end receiver reports |
| One-way audio on external calls | Firewall, NAT, or SIP ALG | Firewall logs for RTP ports; ALG setting; NAT session state |
| Drops at a fixed duration | Firewall session timeout or SIP session timer | UDP timeout on the firewall; timer settings on the PBX and trunk |
| Same minutes every day, any scope | A scheduled job saturating a link | Utilization graph at those minutes; job schedules |
| Bad calls, clean probes on every path | Phones, PBX, or codec transcoding | PBX CPU; codec mismatches in the call records; phone firmware |
The conversation with the provider
Open with facts, in the order their engineer needs them. Start with what you observed, when, the call IDs, and what you measured. Then give what you have ruled out and how, and the specific things you are asking them to check. Ask for their edge's media statistics for your call IDs, per direction. Ask whether the route from their edge toward your public address changed, and whether other customers on the same edge reported anything in the window. Ask for the ticket to go to someone who can read a call trace, and say so politely. First-line scripts exist to close tickets. And an evidence file is how you get past them.
Keep the ticket log in one place. When the problem recurs, update the same file with new call IDs rather than starting over. A second occurrence with matching evidence is what turns a ticket into an escalation. And be fair. Sometimes their view shows your stream arriving at their edge already damaged, and that means the fault is on your upstream path or your internet provider's. Accept it, thank them, and redirect the file. The goal of isolation is the truth about where the problem lives, not a victory over the provider.
Where to go next
Isolation is a sequence, and the sequence is the whole method. Scope the complaint, and split internal from external. Gather evidence at the port, the LAN, the WAN, and the trunk. Reproduce with probes that each cover one segment, and write the file. Most of the evidence is ordinary network monitoring — port counters, utilization, latency history — that is only useful if it was already being collected when the bad call happened.
That is why Building a Call-Quality Monitoring Program comes before the next incident. And SIP Trunk Health keeps the trunk's heartbeat and round-trip history ready for the file. Both live in VoIP & Call-Quality Monitoring.
Put this into practice with SysPeep Monitor
An evidence file is only as good as the counters that were already being collected when the bad call happened. SysPeep Monitor keeps that standing evidence. That includes port errors and discards, uplink and WAN utilization, PoE state, and latency and loss history. It is all gathered over SNMP and similar collectors from one Windows console. There are graphs you can drop straight into the file. The free trial installs in minutes.
Download Free TrialLearn About SysPeep