Home › Topics › Active vs. Passive Monitoring › Synthetic Probes

Designing Synthetic Probes That Mimic Real Users


Every admin has one of these mornings. The dashboard was green. The users were not. A synthetic check was doing exactly what it had been told. The port answered, so the check passed. Meanwhile, behind that port the application had quietly stopped letting anyone log in. The check was not broken. It was measuring the wrong promise. Mine was a webmail port check. It stayed green for most of a morning while the directory server behind it rejected every password in the building.


A synthetic probe is a scripted fake user. The craft of building one has a goal that is easy to state. It takes real care to hit: the probe should fail when your users would fail, pass when they are happy, and do nothing else. A probe that fails while users are fine produces false alarms, and false alarms teach everyone to ignore alerts. A probe that passes while users suffer turns monitoring into a decoration. This guide walks the craft end to end. It covers what to test, how to make the test realistic, and how to keep the test account safe. It also covers where the probe should run from, why speed matters as much as success, and the habits that keep a probe trustworthy. It closes with a complete worked example for a small company you will probably recognize.


In this guide

  • What a synthetic probe is
  • Target the path users take, not the path that's convenient
  • Make the transaction real
  • Test accounts and test data, handled safely
  • Vantage honesty: where the probe runs from
  • Time budgets, not just pass/fail
  • Probe hygiene: patience, confirmation, and manners
  • Worked example: one small company's probe set

What a synthetic probe is


A synthetic probe (also called a synthetic check or synthetic transaction) is active monitoring's precision instrument. Your monitoring system generates traffic purely so it can measure what happens to it. If the active-versus-passive split is new to you, the two modes are built from scratch in the five-scenario guide. The one-line recap is that active monitoring makes test traffic on a schedule, while passive monitoring watches the real traffic that already exists. Probes are the active mode at its most refined — and precision cuts both ways. A well-aimed probe discovers at 2 a.m. what users would have discovered at 8. A badly aimed one verifies, faithfully, around the clock, something nobody cares about.


Every probe is the answer to three design questions, and this guide takes them in order. First comes what to test (the target and how deep to go). Next comes from where (the vantage point). Last comes how (timing, thresholds, and manners). Skip any of the three and you get a probe that runs on time, reports faithfully, and is wrong about the one thing that matters.


Target the path users take, not the path that's convenient


The most common probe mistake is committed at birth: monitoring what is easy to reach instead of what users actually touch. It is a natural mistake — the monitoring tool shows you a list of servers, so you attach checks to servers. But your users do not experience servers. They experience a journey. It starts with a laptop, Wi-Fi, and maybe a VPN tunnel (the encrypted connection that lets a remote laptop behave as if it were in the office). Next comes a DNS lookup of a name. Then perhaps a proxy or a load balancer comes into play (the front-door machines that pass each user to whichever server is free). Only then comes the application, which itself leans on a database, a directory service, and a file store. A probe pointed at the middle of that chain certifies the middle of the chain. Every hop it skips is a hop it can never catch failing.


So the first design step is not technical at all. Sit next to a user (or be one for ten minutes) and write down the actual journey, hop by hop, name by name. Two rules fall straight out of the exercise. First, probe the name users type, not the IP address you happen to know. If the portal is portal.example.com, the probe should look that name up itself every time. DNS is part of the journey and famously fond of breaking. A probe aimed at a bare IP address will cheerfully pass while every user stares at a name that no longer resolves. Second, enter through the front door. If users reach the app through a load balancer or proxy, a probe aimed directly at an app server behind it skips the front door. In that case, the front door can fall off without a single check noticing.


Then choose depth. Checks stack into a ladder of realism: a ping at the bottom, then a port check, then a real protocol conversation, then a full logged-in transaction at the top. The ladder — and how to pick the right rung for each service — has its own guide, ping up vs. service up. The short version: low rungs are cheap, fast, and shallow; high rungs prove what users actually care about but cost more to build and run. The pattern you will see in the worked example below is to use both. A cheap low rung runs often as a tripwire. A deep transaction runs at a gentler pace to catch "up, but broken."


Make the transaction real


"Port 443 answers" and "a user can read their mail" are separated by a long list of things that can be independently broken. That could mean a web server dutifully serving an error page or a login system rejecting everyone since last night's directory change. It could mean a dead session store or an application waiting forever on a stuck database. A high-rung probe closes that gap by doing what a person does. For webmail: log in with a test account, open the inbox, confirm the folder list appears, log out. For a file portal: log in, download a known test document, confirm its contents. For a line-of-business app: log in, run a canned search, confirm the expected record comes back.


The last clause of each recipe is the part beginners skip: content verification. That means asserting that the response contains something only success can produce. Examples include the phrase "Signed in as" or a marker sentence inside the test document. Web applications have a real talent for failing politely. A crash can produce a beautifully formatted error page, delivered promptly, with a status code of 200. That is the three-digit result number that means "everything is fine." A probe that checks only the status code is taking the server's word for it. The server is an unreliable narrator about its own health. Check for evidence of success, never for the mere absence of catastrophe.


Keep transactions read-only wherever possible. A probe on a five-minute schedule runs 288 transactions a day. You do not want 288 test orders per day accumulating in the sales system. When only a write can prove the thing that matters — say, an order actually saving — write into a sandbox area built for the purpose. In that case, tag the records unmistakably. Then have the probe (or a scheduled job) clean up after itself.


Test accounts and test data, handled safely


A realistic transaction needs credentials, and credentials are where probe projects go quietly wrong. A probe account is a real account with a real password that logs in hundreds of times a day. So it deserves the same care as any other service account, plus a little extra for the schedule. The rules that keep it boring:


  • Least privilege. The account gets permission to do exactly what the probe does and nothing more. That means logging in, running the one search, and opening the one document. Never reuse a person's account, and never an admin account. Probe credentials end up stored in monitoring configurations, and configurations travel — backups, exports, migration copies. Grant nothing you would mind traveling with them.
  • Obvious naming. Name the account so nobody could mistake it: svc-probe-webmail, not jsmith2. Name the test data even louder. A file called MONITORING-TEST-do-not-delete.txt survives cleanup sweeps that would have deleted test.txt. Obvious names also mean whoever reviews security logs recognizes probe activity at once.
  • Excluded from business numbers. Exclude probe accounts from usage reports, active-user counts, license consumption where the platform allows it, and any analytics a decision will ever be based on.
  • Real credential care. Store the password in the monitoring tool's credential store, not pasted into a script body. Rotate it on the same calendar as other service credentials. Pay attention to password-expiry policy, for the reason in the tip below.

Handled sloppily, a probe account is a security hole with a schedule. Handled carelessly in the other direction, it becomes your most engaged user. Northgate Retail's file-portal probe ran under an account called jsmith2, cloned in a hurry from a real person's account. For a year it logged in 288 times a day and never read anything. The usage report that justified renewing the portal counted it as the most active member of staff. A security reviewer spent an afternoon on it and found a monitoring server. The review described the account as "highly engaged". Future-you, presenting adoption numbers, will be grateful for the exclusion rule.


Tip: if your directory's password policy applies to the probe account, then on expiry day every probe using it fails at once. That looks exactly like the entire company falling over at 9 a.m. Most admins perform this stunt precisely once. Exempt probe accounts from automatic expiry and rotate them deliberately instead. And adopt the reflex: when all probes fail simultaneously, suspect the probe's own account, host, or network before declaring an apocalypse.


Vantage honesty: where the probe runs from


Picture the standard setup. The monitoring server lives in the server room, one quiet wired hop from the application servers — no Wi-Fi, no VPN, no proxy, no distance. Its probes stroll down an empty private hallway. Users, meanwhile, ride Wi-Fi to an access point, a VPN tunnel over home broadband, and a proxy. Their experience includes every one of those hops. When the VPN concentrator (the box that terminates all those tunnels) chokes, users are down and the server-room probe stays green. That is because a probe measures the path it rides, and it is riding a different path. "Probe passed" and "users are fine" are only the same sentence when the probe travels the way a user travels.


Vantage honesty is two habits. First, for every probe, be able to finish this sentence: "when this passes, it proves…" For a server-room login probe, the honest ending is "…that the app, database, and directory work; it proves nothing about Wi-Fi, the VPN, or the branch line." Write that sentence into the check's description field, because at 2 a.m. nobody remembers what a green light actually guarantees. Second, run at least one probe from where users really sit. Put a small probe host on the office LAN (local area network: the wired and wireless network inside the building), another at the branch. That way, the full user path gets exercised end to end. Disagreement between vantages then becomes free diagnosis. Server-room probe green plus branch probe red equals "the problem lives between them," before anyone picks up a phone. Placement patterns include inside versus outside the firewall, per-VLAN, and per-site. A VLAN is a virtual LAN, one physical switch carved into several separate networks. These patterns get their own guide in probe placement.


Time budgets, not just pass/fail


A probe that reports only pass or fail is a smoke detector: priceless for fires, silent about everything smoldering. Users feel slow long before they see broken, and slow is the phase where problems are cheapest to fix. So a good transaction probe records how long every step took, on every run:


probe: portal-login          result: PASS     total: 2.84 s
  resolve portal.example.com                   0.02 s
  tcp connect 10.0.8.40:443                    0.01 s
  tls handshake                                0.14 s
  get /login (first byte)                      0.31 s
  submit credentials, follow redirect          1.92 s
  load dashboard, verify "Signed in as"        0.44 s

This run "passed." It also just reported a problem. On this probe, the credential step normally takes about 0.6 s. Today it took 1.92 s — three times its usual self. That pattern, login slowing while every other step holds steady, points a finger at the directory server or the database behind it, before a single user has called. Step timings turn one probe into half a dozen tripwires, each pointing somewhere different. Name resolution slow means DNS. Connect slow means the network path. TLS slow (the encryption handshake that puts the "s" in https) suggests certificate chain trouble or an overloaded server. First byte slow means the application. Download slow means bandwidth. One probe, six diagnoses, zero extra checks.


Give each step a budget: a threshold based on its own observed normal, not on hope. Collect a week or two of timings first, then warn when a step runs at, say, twice its typical value for fifteen sustained minutes. The overall shape is "warn on slow, alert on broken." Degradation warnings arrive while users are still only grumbling. Hard alerts arrive when the transaction genuinely fails. The entire point is to meet your problems during the grumbling phase, when the fix is a calm afternoon task instead of an outage call.


Probe hygiene: patience, confirmation, and manners


The remaining craft is behavioral: the habits that decide whether the team trusts a probe or mutes it. Each one below exists because a probe without it has woken someone for nothing.


  • Timeouts that mirror human patience. How long would a user wait for the portal before declaring it broken — ten seconds? Fifteen? Set the probe's timeout in that neighborhood. A two-second timeout on a transaction that normally takes three files false alarms all day. A 120-second timeout means real outages are discovered two minutes late, every time. The probe should give up roughly when a human would give up.
  • One polite retry. A user who hits a glitch reloads the page once before complaining. Give the probe the same reflex: one quick retry, and the attempt counts as failed only if the retry also fails. That absorbs one-packet bad luck without hiding anything real.
  • Confirmation before alerting. Require two consecutive failed cycles — or two of the last three — before waking anyone. Blips self-heal; outages do not. Confirmation does add minutes of detection delay. Tuning that trade among speed, noise, and cost is the whole subject of synthetic test frequency.
  • Maintenance-window awareness. The probe must know when patch night is. Otherwise Sunday 2 a.m. alerts become routine, and routine alerts train people to ignore alerts — the fatigue that kills monitoring systems. Suppress or downgrade alerts during declared windows. Make sure the probe alarms loudly if the service is still down when the window closes. A reboot that never comes back is a common failure.
  • Manners toward the service. From a security tool's point of view, a login probe is a machine hammering a login endpoint at metronome intervals. Make sure repeated logins cannot trip account lockout (the defense that freezes an account after repeated attempts) or brute-force detection. Do that either by pacing the probe well below those limits or by a documented, deliberate exception. Where the platform allows it, tag probe requests with an identifying user-agent string so its activity is instantly recognizable in logs.
  • Suspect the probe first. When every probe fails at the same moment, the odds overwhelmingly favor the probe host, its network connection, or its credentials over a simultaneous collapse of every service you run. Dependency rules in your alerting — "if the probe host is down, hold everything downstream" — encode that reflex so it does not depend on anyone remembering it.

Two of those rules earn their place more often than the rest. The maintenance-window rule matters in both directions, because "the reboot never came back" is a genuinely popular failure. A probe that was politely silent all through patch night is no use if it stays silent afterwards. And the dependency rule exists so that 3 a.m. you does not have to remember to suspect the probe host first. I have met 3 a.m. me, and he does not remember things.


Worked example: one small company's probe set


Time to assemble the pieces. The company: forty people, one office, one small branch. Three services matter enough to probe deeply: hosted webmail, an internal file portal, and the line-of-business order app the branch depends on. Probes run from two vantage points — a small probe host on the office LAN, where most users sit, and another at the branch. Here is the set, then the reasoning behind it:


ServiceProbe actionVantageFrequencyAlert rule
Webmail (shallow)Resolve name, connect to port 443, read certificate days-to-expiryOffice LAN1 minAlert after 2 consecutive failures; warn 30 days before cert expiry
Webmail (deep)Log in, open inbox, verify folder list, log outOffice LAN5 minAlert on 2 of 3 failures; warn when login step >2× baseline for 15 min
File portal (shallow)Ping host, connect to port 443Office LAN1 minAlert after 2 consecutive failures
File portal (deep)Log in, download test file, verify marker text inside itOffice LAN10 minAlert on 2 consecutive failures; warn when download time >2× baseline
Order app (deep)Log in, run canned search, verify the known test record appearsOffice LAN10 minAlert on 2 of 3 failures; warn on step budgets
Order app (branch view)Same transaction as aboveBranch probe10 minCompare with office result: branch-only failure flags the WAN, not the app

Notice the two-rung pattern on each service. The shallow check is the tripwire: it runs every minute because it is nearly free, cannot trip lockouts, and catches hard-down failures fast. The deep transaction runs at a gentler pace. It runs often enough that "up but broken" is caught within minutes. It is spaced enough that the login endpoint is not being hammered. Between them, a total outage is known in about two minutes and a subtle one in about ten to fifteen. That is a fine bargain for six checks' worth of configuration.


The duplicated order-app probe at the branch is doing the vantage work from earlier. Office green plus branch red means the WAN line (the wide area network link that joins the branch to the office). Both red means the app. Both green means the next branch complaint is probably a local machine. One duplicated probe replaces an entire guessing phase of troubleshooting. And because every deep probe carries step budgets, slowness surfaces as calm warnings days before it matures into a hard failure.


Six probes, two vantage points, and every single one exists because a specific user pain would trip it. When a probe in this set fails, somebody real is having a bad time — which is exactly the property that makes an alert worth waking up for.


Where to go next


The craft fits in one paragraph. Probe the journey users actually take, at the rung that proves what they need. Verify content, not just connections. Give the test account nothing worth stealing and a name nobody can mistake. Be honest about what each vantage point proves. Budget the timings so slow gets caught before broken. Give every probe the patience and manners of a polite human. Do all six and a green light means what the users think it means. Skip one and you are back to the morning in the first paragraph, measuring a promise nobody asked for. What probes structurally cannot see — and how passive monitoring covers those gaps — is cataloged in the blind spots of each mode. The full architecture that combines both modes on a small budget is assembled in the blended monitoring design. The rest of the topic lives at Active vs. Passive Monitoring.


Put this into practice with SysPeep Monitor

Transaction probes are a craft; the layer underneath them is routine, and routine is what software is for. SysPeep Monitor keeps watch on the foundations your probes stand on. It discovers the devices on your network and tracks health and performance over SNMP, WMI, and SSH. Its lightweight remote probes give you monitoring vantage points at branch sites and other segments from one console. Thresholds and dependency rules keep alerting calm. The free trial installs on any Windows machine in minutes.

Download Free TrialLearn About SysPeep