Where to Run Transaction Tests From (and Why It Changes the Answer)
The warehouse says the ordering system is unusable. Head office says it is fine. Both are telling the truth, and the argument that follows will eat your afternoon unless you can prove where the slowness lives. A scripted user journey, run from one place, can tell you that a step is slow. The same journey run from several places can tell you why — because an application problem looks the same from everywhere. And a network problem only looks bad from the side of the network that has it.
That is the whole idea behind vantage points: the places you run a journey from. Choose them well and every result becomes a comparison instead of a lonely number. This guide explains the five vantage points worth having, the logic for reading differences between them, and what each one costs. It shows how to schedule runs so that your own testing does not become the load problem. It assumes you already know what a journey is; if not, start with scripted user journeys.
In this guide
- A vantage point is a witness
- The five vantage points and what each one sees
- Reading the differences: the localization logic
- Comparing step timings across points
- What a vantage point costs, and how to prioritize
- Keeping runners honest
- Scheduling and staggering so you do not hammer the app
- Where to go next
A vantage point is a witness
Think of each runner — the machine that executes the journey — as a witness standing somewhere on the map. A witness in the head-office LAN sees the application across the core switches and the data-center firewall. A witness in a branch sees it across the branch switch, the WAN link, and whatever the carrier does in between. A witness on the public internet sees it across your ISP, your edge firewall, your public DNS, and your certificate. Every witness reports the same thing: how long each step of the journey took, and whether it passed. The difference is what lies between the witness and the server.
This is why a single vantage point is so easy to misread. A journey that runs only from head office and reports "search took four seconds" has told you that search is slow from head office. Whether the four seconds was spent in the database or on a congested link is invisible, because there is nothing to compare against. Add a second witness in a different place and the number becomes a comparison. If the branch also sees four seconds, the path is probably innocent. If the branch sees twelve, the path is probably not.
One witness matters more than the others: the one standing next to the server, on the same network segment. There is almost no network between it and the application. Its results are the control, in the laboratory sense — the measurement with the variable removed. Whatever the control reports is the application's own cost. Everything a farther witness adds on top of the control is the cost of getting there.
The five vantage points and what each one sees
You do not need dozens of vantage points. Five kinds cover almost every question an administrator is asked, and most organizations start with three of them.
Next to the server (the control)
A small runner on the server VLAN, or on the same hypervisor host. It exercises the application, the database, and their dependencies, with the network reduced to a switch hop. When the control is slow, nobody needs to look at a WAN graph.
The office LAN
A runner on an ordinary user subnet at head office, ideally on the same access switch and VLAN as real desks. It adds the campus network, the data-center firewall, internal DNS, and any proxy between users and the application. This is the "most users" view and usually the first vantage point people build.
Each branch
A runner at every site that matters, on a user subnet there. It adds the branch LAN, the WAN link, and the carrier. Branches are where the differences show up, because each branch has its own link, its own utilization pattern, and its own way of being misconfigured. If you cannot afford a runner at every branch, start with the busiest and the one that complains most, which are rarely the same place.
A remote-worker representative
A runner that reaches the application the way home workers do: through the VPN client or the remote-access gateway, from a residential-style connection. It adds the VPN concentrator, split-tunnel rules, and an internet path you do not control. A small machine at an employee's home, or a cloud instance that connects through the VPN, does the job. Without this vantage point, "it's slow from home" is unfalsifiable.
An outside point
For applications that are reachable from the internet — a customer portal, a public web app — a runner outside your network entirely. It adds public DNS, your ISP, the edge firewall, the certificate chain as the world sees it, and any load balancer in front of the app. Simple external web checks from many locations are a subject of their own, covered in the guide on external multi-location checks. Here the outside point runs the full journey, not a page fetch.
| Vantage point | Path it exercises | A failure only here implicates | Typical runner |
|---|---|---|---|
| Next to server (control) | Application and its dependencies only | The application, database, or identity provider itself | Small VM on the server VLAN |
| Office LAN | Campus network, DC firewall, internal DNS, proxy | Core network, firewall rules, internal DNS, proxy | Small VM or mini PC on a user subnet |
| Each branch | Branch LAN, WAN link, carrier, branch DNS | That branch's link, router, or local config | Mini PC or container on a user subnet at the site |
| Remote-worker representative | Residential internet, VPN client and concentrator, split-tunnel rules | VPN gateway, tunnel policy, home connectivity | Mini PC at a home, or cloud instance on the VPN |
| Outside point | Public DNS, ISP, edge firewall, public certificate, load balancer | The edge, public DNS, certificate, or ISP | Cloud instance or hosted runner |
Reading the differences: the localization logic
Once the same journey runs from several witnesses, diagnosis becomes pattern matching. The patterns are few and worth memorizing, because they turn "the app is slow" into an assignment for a specific team within one look at a table.
- Slow or failing everywhere, including the control. The application or something it depends on. The network is exonerated in one glance. Hand it to the application owner with the step name and the control's timing.
- Slow from every remote site, but the control is fine. Something every path shares: the data-center firewall, a core switch, internal DNS, a proxy, or the WAN aggregation. Look at what all the slow witnesses have in common.
- Slow from one branch only. That branch's path: its WAN link, router, local switch, or local DNS setting. Compare against the branch's link utilization for the same minutes.
- Slow from the remote-worker witness only. The VPN concentrator, the tunnel policy, or the home connection. If a second remote witness on a different ISP agrees, it is your gateway, not their broadband.
- Slow or failing from outside only. The edge: public DNS, the ISP, the edge firewall, a certificate the world does not trust, or a load balancer with a sick member. Internal users will not complain, which is exactly why you need this witness.
- One step slow everywhere, other steps fine. A dependency behind that step (search service, report engine, identity provider), not the path, since the path is shared by all the steps.
Notice the last pattern, because it is the one only a journey can produce. A network problem taxes every step roughly equally, since every step crosses the same path. An application problem taxes the one step that touches the sick component. When the search step alone is slow from every vantage point, the WAN is not the suspect no matter how convincingly someone says the word "network." The single-probe version of this reasoning asks where to put simple ping and port checks so that each one implicates one segment. That is the subject of the guide on probe placement. Journeys add the per-step dimension on top.
Tip: Before blaming a branch, check that its runner is healthy. A runner starved of CPU by a backup job, or one whose clock has drifted, will report slowness that no user is experiencing. Monitor the runners like any other server.
Comparing step timings across points
The most useful artifact this whole approach produces is a small table: steps down the side, vantage points across the top, the latest timing in each cell. Here is one for the customer-records journey during a busy Tuesday morning, with the control's timing as the reference column.
| Step | Control | Office LAN | Branch North | Branch South | Remote (VPN) | Outside |
|---|---|---|---|---|---|---|
| Open app | 0.2 s | 0.3 s | 0.6 s | 2.9 s | 0.8 s | 0.7 s |
| Log in | 0.9 s | 1.0 s | 1.3 s | 4.1 s | 1.6 s | 1.4 s |
| Search | 3.8 s | 3.9 s | 4.2 s | 7.5 s | 4.5 s | 4.3 s |
| Open record | 0.4 s | 0.5 s | 0.8 s | 3.4 s | 1.0 s | 0.9 s |
| Submit note | 0.6 s | 0.7 s | 1.0 s | 3.8 s | 1.2 s | 1.1 s |
| Log out | 0.2 s | 0.2 s | 0.5 s | 2.7 s | 0.7 s | 0.6 s |
Read it in two passes. First, down the control column: search takes 3.8 seconds even with no network in the way, against a baseline of about half a second. That is an application problem, and every other column inherits it. The search row is slow everywhere, by roughly the same amount over the control. Second, across each row: Branch South adds two to three seconds to every step, including trivial ones like "log out" that transfer almost nothing. A uniform tax on every step is the signature of a path problem, and it belongs to Branch South's WAN link. Two different problems, two different teams, visible in one table without a packet capture.
The Branch South number also gives you a useful conversation with the branch. "You are seeing about three seconds of extra delay on everything. Search is slow for everyone today. The first is your link and we are on it. The second is the application team's." That is a far better answer than "we are looking into it."
Two habits make the table trustworthy. Compare timings from runs that happened within the same few minutes. Comparing a branch's 09:00 run against the control's 09:20 run mixes in whatever changed in between. And always subtract the control: the interesting quantity in each cell is the difference from the control, not the raw number. Some teams show only that difference, which makes a path problem jump off the page.
What a vantage point costs, and how to prioritize
Every vantage point needs a runner: a machine that lives there, executes the journey on schedule, and reports results somewhere central. That is the honest cost, and it is not zero. A runner needs hardware or a VM, power and a network port at the site, and an operating system to patch. It needs the scripts and credentials kept in sync with every other runner. It needs a way to ship results home, and someone who notices when it dies. Browser-automation journeys also need memory to spare. Multiply that by the number of sites and it is clear why "a runner everywhere" is a goal, not a starting point.
Prioritize by the questions you actually get asked, in roughly this order:
- 1. The control. Cheapest to build, and without it no other result can be interpreted. Do this first, even before the office LAN.
- 2. The office LAN. The largest user population, and the reference point for "normal."
- 3. The branch that complains most. Not necessarily the biggest; the one whose tickets you cannot currently answer.
- 4. The remote-worker representative. If a meaningful share of staff works remotely, this one settles a whole category of arguments.
- 5. The outside point. Essential for anything customers reach; optional for purely internal applications.
- 6. The remaining branches, in order of user count and link quality. Sites on the shakiest links are worth more than sites with the most people.
Keep runner hardware boring and identical. A fleet of the same small machine with the same image is far easier to keep honest than a mix of retired desktops. When one fails, you can post a replacement to the site and have a receptionist plug it in.
Keeping runners honest
A vantage point is only as useful as its resemblance to the users it stands in for. The branch runner should use the branch's DNS servers, the branch's default gateway, and the branch's proxy settings. That is what the branch's users do. A runner that has been "helpfully" pointed at the data-center DNS, or plugged into the router's management VLAN, is measuring a path no user takes. It will stay green while the branch is on fire. The same applies to the remote-worker runner. If it does not connect through the same VPN profile as real remote staff, it is a second office-LAN witness with extra steps.
Runners also need to be stable in the ways users are not. Give them a dedicated purpose and nothing else — no backup jobs, no file-server duties, no one's spare desktop. Keep their clocks synchronized, because comparing timings across witnesses depends on knowing when each run happened. Monitor each runner's own health (CPU, memory, disk, reachability) as a first-class device, and make "runner unreachable" a different alert from "journey failed." A journey that stops reporting is not a failed journey. It is a missing witness, and treating the two the same way produces confident conclusions from no data.
Finally, keep runner configuration identical except for the things that are supposed to differ. Same script version, same timeouts, same test account (or an equivalent one per site), same result format. When a branch reports differently, you want the only possible explanation to be the branch.
Scheduling and staggering so you do not hammer the app
Here is the arithmetic that surprises people. One journey every five minutes from one runner is twelve logins an hour. The same journey from eight vantage points is ninety-six logins an hour, plus ninety-six searches and ninety-six writes, all against production, all day, forever. For a large application that is background noise. For a small line-of-business system with a fussy database, or an identity provider that rate-limits logins, it is a self-inflicted denial of service. The first person to notice will be the application owner, in a tone.
Three rules keep the load reasonable. First, choose the frequency per journey based on how fast you need to know, not on what the scheduler allows. The guide on synthetic test frequency works through that trade-off. Second, stagger the runners so that they do not all start at the top of the minute. If eight runners each run every ten minutes, offset them by a minute or so each. That way, the application sees one synthetic login roughly every minute rather than eight at once followed by nine minutes of silence. Third, cap concurrency: a runner should never start a new run while its previous one is still going. Otherwise, a slow application turns into a pile-up of overlapping synthetic sessions that makes the slowness worse.
# journey: records-core, interval 10 min, eight runners staggered runner offset runs at (minutes past the hour) control-dc :00 00 10 20 30 40 50 hq-lan-01 :01 01 11 21 31 41 51 branch-north-01 :02 02 12 22 32 42 52 branch-south-01 :03 03 13 23 33 43 53 branch-east-01 :04 04 14 24 34 44 54 remote-vpn-01 :05 05 15 25 35 45 55 outside-01 :06 06 16 26 36 46 56 branch-west-01 :07 07 17 27 37 47 57 # max one run in flight per runner; skip, do not queue, if the previous run is still going
Two courtesies go a long way with application teams. Send synthetic traffic with an identifiable user-agent string or a header. That lets them filter it out of their usage reports and spot it in logs. And agree the schedule with them before turning it on. Include a quiet period during their backups or batch jobs if those already stress the system. A journey that runs during the nightly index rebuild will faithfully report that the index rebuild is slow, every night, at three in the morning, to your phone.
Where to go next
Vantage points turn a journey from a stopwatch into a diagnosis. The control tells you what the application costs on its own. Each farther witness tells you what its path adds. The pattern across witnesses — everywhere, one branch, remote only, outside only, one step only — names the team that should be looking. Build the control first, then the office LAN, then the branch that complains most, and keep every runner boring, identical, and monitored.
From here, the guide on alerting on experience shows how a second vantage point doubles as confirmation before anyone gets paged. The guide on keeping synthetic tests honest covers the care and feeding of a runner fleet over time. If the WAN itself is the suspect, throughput testing done right explains how to measure a path without breaking it. All of these start from the Synthetic Transactions & End-to-End Testing pillar.
Put this into practice with SysPeep Monitor
When a journey is slow from one branch and fine everywhere else, the next question is what that branch's link was doing at the time. SysPeep Monitor's optional remote probes sit at your sites and measure latency, loss, and WAN interface utilization from there. So the vantage-point table and the network graphs line up minute for minute in one console. The free trial installs on any Windows machine.
Download Free TrialLearn About SysPeep