Home › Topics › Application Slowness vs. Network Slowness › Chatty Applications

Chatty Applications: When the App Makes the Network Look Slow


This is the incident that makes a network administrator reconsider the profession. The branch office says the order-entry application takes ten seconds to open an order. You check everything: no packet loss, round trip steady at forty milliseconds, the WAN link cruising at a third of its capacity, the server idle. Headquarters opens the same order in under a second. Every dashboard you own is green, and the users are still right. The application is simply chatty, and chattiness turns latency into time by multiplication.


I will show the multiplication with a worked example. I will explain why a chatty application feels fine on the LAN and collapses over a WAN or VPN. And I will give three ways to detect it in captures, flows, and browser waterfalls. Then come the classic culprits, the fixes worth proposing and who owns each, and the honest conversation with developers and vendors. Finally, I will show how to measure whether a fix worked.


In this guide

  • Round-trip multiplication, in plain language
  • Why chatty apps feel fine on the LAN and collapse over the WAN
  • Detecting chattiness
  • The classic culprits
  • Fixes worth proposing
  • The honest conversation with developers and vendors
  • Measuring the improvement
  • Where to go next

Round-trip multiplication, in plain language


A chatty application is one that gets its work done through many small requests made one after another. The application waits for the previous answer before sending the next request. Imagine grocery shopping by phoning the store once per item: "Do you have eggs?" — wait — "Add them." — wait — "Do you have milk?" Each call is short and the store answers instantly, but a forty-item list takes forty calls. Now give the phone line a two-second delay on every call. Your list has not changed, the store is no slower, and your shopping takes minutes of pure waiting.


That is the whole mechanism. Every request across a network costs at least one round trip. That is the time for a packet to reach the server and the answer to come back. A chatty application pays that toll once per request, and because the requests are sequential, the tolls add up instead of overlapping. The rule fits in your head:


time spent waiting  =  number of sequential requests  x  (round trip + per-request server time)

A worked example with illustrative numbers. An order has 40 lines. To display it, the application fetches the order header (1 request), then each line (40 requests). Then it fetches the product details for each line (40 more), then the stock level for each product (40 more). That makes 121 sequential requests. Suppose the server answers each in 5 ms. The only thing that changes between locations is the round trip.


Where the user sitsRound tripNetwork waiting (121 × RTT)Server time (121 × 5 ms)Total
Same building as the server1 ms0.12 s0.6 sabout 0.7 s
Branch office over the WAN40 ms4.8 s0.6 sabout 5.4 s
Home office over VPN80 ms9.7 s0.6 sabout 10.3 s

Look at what did not change. The server did the same work in the same 0.6 seconds everywhere. The data was tiny: 121 responses of a couple of kilobytes each. Any of those links would carry that data in a fraction of a second if sent all at once. The branch user's wait is made of nothing but 121 forty-millisecond pauses, stacked end to end. Everyone's dashboard is green because every dashboard measures one round trip, and one round trip is fine. What nobody measures is how many of them the application needs.


Why chatty apps feel fine on the LAN and collapse over the WAN


Applications are built, tested, and demonstrated a millisecond or two from their servers. At that distance, 121 round trips cost a tenth of a second, so nobody counts them. A design that makes hundreds of small calls then ships unnoticed. The application is not badly written by the standards of its own desk. It is written on the assumption that round trips are free, and that assumption holds right up to the edge of the building.


This is also why the usual network fix fails. Bandwidth is how much data a link can move per second. Latency is how long one trip takes. And no amount of the first reduces the second, a distinction covered in latency versus bandwidth. Upgrading the branch link from a modest circuit to a fast one leaves the round trip at forty milliseconds. That is because the round trip is mostly distance and the number of devices in the path. The chatty application will be exactly as slow on the new link.


Two things make WAN and VPN cases worse than the plain multiplication suggests. First, if the application opens a new connection for each request rather than reusing one, every request also pays the handshake. That means one round trip for TCP and one or two more for TLS. So the multiplier becomes three or four round trips per request. Second, VPN paths are often longer than they look. A home user's traffic may travel to a concentrator at headquarters, out to a cloud-hosted service, and back the same way. Wi-Fi adds variability on top, so the multiplication is not just large but jittery, which users describe as "sometimes it just hangs."


At Bellweather Logistics the depot's order-entry screen took nine seconds, and the ticket said "WAN latency" for months. The circuit was upgraded, and the screen took nine seconds on the new one. A capture from the depot showed 214 sequential exchanges per order; from headquarters the same order took under a second. The developers batched the lookups within a fortnight. The invoice for the circuit is still described, in meetings, as an investment.


Detecting chattiness


The signature of a chatty application is not any single slow thing; it is a large count of small, quick, strictly sequential exchanges. You can see it from three places, and the third is the proof.


Request counts in a browser waterfall

For a web application, open the browser's developer tools on the network tab and perform the slow action. Ignore any one row's timing and look at the shape of the whole list. Look for hundreds of short rows arranged as a staircase, each beginning only after the previous one ends. That staircase is serialization; a healthy page shows a few wide rows and many that overlap. The row count per action is the number you will carry into every later conversation.


Many tiny packets in captures and flow records

In a packet capture of the conversation, packet sizes skew small and the byte count is modest while the packet count is high. Flow records from your routers show the same thing at a distance. They show many short flows, or one long flow with low bytes per second and a high packet rate, to the same server. A link carrying a chatty application is busy in packets and idle in bytes, which is why its utilization graph looks fine.


Serialized waits, and the arithmetic that proves it

Filter a capture to one client-server conversation and look at the pattern of direction. The client sends a little, and the server answers a little. After a gap of about one round trip, the client sends a little again. No overlap, no bursts, just alternation. Count the exchanges, multiply by the round trip you measured with the handshake, and compare with the total duration of the action. If the product is close to the duration, the network has been convicted of nothing except distance, and the request count is the finding. Reading the conversation in detail is covered in reading TCP conversations; this test needs only the count and the round trip.


# Conversation summary from a capture at the branch (illustrative)
client 10.20.5.31  <->  server 10.0.0.40:1433
duration                 8.9 s
handshake round trip     41 ms
request/response pairs   214
bytes, both directions   388 KB
largest gap between any two packets   62 ms

# The arithmetic
214 exchanges x 41 ms = 8.8 s   (matches the 8.9 s duration)
388 KB over 8.9 s     = about 44 KB/s  (the link is nearly idle)

Two confirmations finish it. Time the same action from headquarters. The same number of exchanges in a small fraction of the time means the time scales with round trip. That is the definition of chattiness. And decompose one request from the middle of the conversation using the method in response-time decomposition: every segment is small. Nothing is slow. There is just a great deal of it.


Tip: the "largest gap" line is worth adding to your routine. In a chatty conversation no single gap is large; the delay is hundreds of gaps of about one round trip each. If instead you find one gap of several seconds, you are looking at server think time, not chattiness. In that case, this topic's guide on server think time is the one to open.


The classic culprits


File-share metadata storms

Opening a network folder is not one request. The file-sharing protocol asks for the directory listing, then queries attributes, sizes, and permissions, often one file at a time. The client's shell may then open each file briefly to read its icon or summary. A folder with a thousand files can generate thousands of exchanges before a single document opens. Opening the document adds dozens more: locks, header reads, small-chunk body reads, checks that the file has not changed. Redirected folders and roaming profiles put this storm on every login. On the LAN it is invisible; over a VPN it is the ticket that says "the file server is slow."


Query storms from data-access layers

Many applications use a library that translates program objects into database queries automatically. By default, such a library fetches a list with one query and then each item's details with one more query per item. Developers call that pattern the "N plus one" problem. A screen showing a hundred rows issues a hundred and one queries. When the application server and the database sit in the same rack, that is a hundred round trips of a millisecond and nobody notices. When they are separated, the same screen pays a hundred WAN round trips per view. That might mean an application in one data center and a database in another. Or it might mean a service in the cloud querying a database on premises.


Legacy client-server applications

The worst offenders are thick-client applications designed for a single office. There, the program on the user's PC talks directly to the database server with no application tier between. Every screen refresh, every dropdown, every keystroke-triggered lookup is a database round trip from the desktop. Some use file-based databases, where the client reads index pages across the share one block at a time. These applications were correct in the world they were built for. Moving their users to a branch or a home connection changes the world without changing the application.


CulpritWhat the chatter isHow it shows upWho can change it
File-share browsing and document opensPer-file attribute queries, small-chunk reads, lock checks"The file server is slow" from remote sites and VPN onlyYou (caching, placement, protocol tuning); the vendor for the client behavior
Data-access query stormsOne query per row instead of one per screenSlow when app and database are far apart; fast in the same rackDevelopers (batching, eager loading); you (placement)
Thick clients talking straight to a databaseEvery screen element is its own database round tripSlow everywhere except the office the app was built inVendor (rarely); you (publish the app next to its database)
Per-request connection setupNew TCP and TLS handshake for each callMultiplier of three or four round trips per requestDevelopers (connection reuse); sometimes a proxy in front
Authentication and lookup chatterDirectory or DNS lookups on every operationSlow when the directory or resolver is far awayYou (local directory or resolver at the site)

Fixes worth proposing


Every fix for chattiness attacks one of the three factors in the formula. Those are the number of requests, the round trip each one pays, or the per-request cost. Knowing which factor a proposal touches tells you who owns it and how much it can help.


  • Batching — reduces the count. One request that returns a hundred rows instead of a hundred requests returning one. In web applications this is a bulk endpoint. In data-access libraries it is a setting that fetches related records together. In file protocols it is compound operations, which current protocol implementations support when both ends are current. The fix with the biggest ceiling, and it belongs to the developers or the vendor.
  • Parallelism — hides the count. If independent requests are sent at the same time instead of one after another, a hundred round trips overlap into a few. Newer transport features let many requests share one connection at once, but the application has to be written to use them. Developer-owned.
  • Caching — removes repeat requests. The options include a client-side cache, a branch-office cache for file shares, offline copies of rarely changing folders, and a local DNS resolver. Each turns a WAN round trip into a local one for anything asked twice. Mostly yours, and often the fastest thing you can deploy.
  • Moving the server closer — reduces the round trip. Put the application tier next to its database so the chatty conversation happens over a one-millisecond path. And let users talk to the application over one long hop instead of a hundred. For thick clients, use the same idea in reverse: publish the application on a remote-desktop or virtual-desktop host in the data center. That way, the chatter stays on the LAN and only screen updates cross the WAN. The classic rescue for legacy applications, and yours to propose.
  • Moving the data closer — also reduces the round trip. A replicated file share or a database read replica at the branch, for data read far more often than it is written.
  • Application delivery tweaks — reduce per-request cost. These include persistent connections instead of a handshake per request, and WAN optimization appliances that pre-fetch and compress file-share traffic. They also include routing that sends VPN users directly to a cloud service rather than through headquarters. Useful, but each is a percentage, not a multiple; none turns 214 round trips into 12.

FixFactor it attacksOwnerCeiling
Batching, bulk endpoints, eager loadingRequest countDevelopers or vendorVery high: hundreds of requests become a handful
Parallel requests, multiplexed connectionsRequest count, effectivelyDevelopersHigh, for requests that do not depend on each other
Caching, offline files, local resolverRepeat requestsYouMedium to high for read-heavy work
Publish the app next to its data; virtual desktopsRound tripYouVery high for legacy client-server apps
Replicas and branch copiesRound tripYou and the data ownerHigh for read-mostly data
Connection reuse, WAN optimization, direct routingPer-request costYou, or developers for reuseModest: percentages, not multiples

The honest conversation with developers and vendors


At some point the evidence is in and someone has to say the sentence out loud: this application was never designed for the WAN. Said badly, it is an insult. Said with the multiplication table in hand, it is a statement about geography, and geography is nobody's fault. The framing that works: the application assumes round trips are free, and users forty milliseconds away have broken the assumption. Nothing is defective; a design constraint has been violated by a floor plan.


With developers, bring three things. Bring the request count per action from the waterfall or capture and the round trip from each location. Also bring the multiplication that turns those into the seconds users report. Ask, not tell: "What would it take to fetch these in one call?" and "Are these requests dependent on each other, or could they go in parallel?" Developers who can see 214 exchanges on a screen usually know the fix before you finish the sentence. That is because the pattern has a name in their world too. Offer your side in the same breath: placement, caching, and a harness that will show the improvement.


With vendors, the conversation is about supported topology. Ask what round-trip latency the product is designed for and whether the vendor publishes a deployment guide for remote sites. Also ask whether a published-application or virtual-desktop deployment is supported. Send the capture summary with the request count. A vendor's support team may not know their product makes two hundred calls per screen until a customer counts them. Where the answer is "it must be on the LAN," that is a decision, not a dead end. In that case, publish it centrally, replace it eventually, or accept the wait and say so in writing. The political side, and the writing that keeps these conversations from becoming rivalries, is in this topic's guide on working with app owners.


Measuring the improvement


The metric that matters is the one in the formula: sequential requests per transaction, alongside total time from the affected site and from near the server. Measure all three before any change, with the same action and vantage points. Ideally, measure from a scheduled synthetic check so the numbers are nobody's opinion. Then measure again afterwards. The illustrative table shows a successful batching change from the branch.


Open order 88213 from the branch (round trip 41 ms; illustrative)
                              before        after
request/response pairs          214           12
bytes transferred             388 KB       402 KB
total time, branch             8.9 s        1.1 s
total time, headquarters       0.9 s        0.6 s
time / round trip (exchanges)   217           27

Read the last row as the report card. Dividing total time by the round trip gives the number of round trips the action effectively paid. Before the change it matched the exchange count almost exactly, which was the diagnosis. Afterwards it fell by an order of magnitude, which is the cure. Notice that bytes went slightly up: batching often moves a little more data in far fewer trips. Anyone who expected bandwidth to be the problem now has a number showing it never was. Headquarters improved only slightly, because its round trip was already tiny. If the count dropped and the branch time did not, the remaining time is somewhere else. In that case, decompose one of the new, larger requests and see which segment grew.


Where to go next


Chattiness is the case where every measurement comes back clean and the users are still right. Its vocabulary is three words: request count, round trip, and the multiplication between them. The layered diagnosis workflow shows where "one site, one app, connect segment normal" falls in the sequence. The guide to working with app owners has the dashboard and harness behind the before-and-after table. The Application Slowness vs. Network Slowness pillar collects all of it.


Put this into practice with SysPeep Monitor

The multiplication only works if you know the round trip from each site. And the number that convinces people is a graph, not a one-off ping. SysPeep Monitor's optional remote probes measure latency and loss from your branch offices toward your application servers around the clock. They put those measurements on the same dashboard as link utilization. So when the branch says the app is slow, you can show the round trip that the application is paying two hundred times per screen. The trial is free.

Download Free TrialLearn About SysPeep