Keeping Synthetic Tests Honest: Maintenance, Drift, and False Alarms
A synthetic test is born honest. On the day you build it, it does exactly what a user does, checks exactly the right thing, and fails for exactly the right reasons. Then the application changes, the test account's password expires, someone deletes the test record, and the login page grows a consent dialog. Six months later, the same test fails every night for a reason nobody has looked at. Or it passes every night while checking nothing at all. Neither version is monitoring. Both are theater.
Infrastructure checks age slowly, because a ping is a ping. Journeys age fast, because they are tied to an application that other people are paid to change. This guide is about keeping them truthful. It covers the ways tests rot, the two traps that turn a test into a liability, and the ownership and review habits that catch drift early. It explains how to version scripts alongside the application and how to get application teams to warn you before they move the login button. It shows how to retire a test with the feature it was watching. It assumes you have journeys running; if not, start with scripted user journeys.
In this guide
- How synthetic tests rot
- The test that always passes
- The test that always fails
- Ownership and the review ritual
- Versioning scripts with the application
- Pre-release coordination with application teams
- Retiring tests for retired features
- The maintenance checklist
- Where to go next
How synthetic tests rot
Rot is not a bug in your script; it is the normal consequence of a script depending on things other people control. Knowing the common forms is half the cure, because most of them announce themselves in a recognizable way.
- The application changed. A redesigned login page, a renamed field, a search box moved into a menu, a new confirmation step. Browser-automation scripts that locate elements by name or position break instantly; HTTP scripts break when a URL, parameter, or response format changes. Symptom: a step fails with "element not found" or "unexpected response" right after a release.
- The test account expired or got locked. Password policies apply to service accounts too. An account locks after the runner retries a stale password a few times. Or it gets disabled by an inactivity sweep. It might get swept up in a directory cleanup because nobody knew what it was for. Symptom: every vantage point fails at login simultaneously, while real users are fine.
- Certificates and DNS changed. The application moved to a new address and the runner has a stale hosts entry. The certificate was renewed with a new chain the runner's trust store does not know. A name the script hard-coded stopped resolving. Symptom: the first step fails with a connection or trust error.
- Someone cleaned up the test data. The test customer the journey searches for was deleted as "junk" by a well-meaning administrator, or an archiving job moved it out of the live table. Symptom: login passes, search returns zero results, everything after it is skipped.
- The application added a dialog. A cookie banner, a consent prompt, a "what's new" pop-up, an unexpected multi-factor challenge. The page the script expects is now behind one more click. Symptom: a step times out waiting for an element that is present but covered.
- The runner drifted. An operating system update changed the browser, a proxy setting was pushed by policy, the runner's clock drifted, or the machine ran out of disk. Symptom: failures from one vantage point only, with no application change to blame.
| Cause | Typical symptom | Fix | Prevention |
|---|---|---|---|
| UI or API change | Step fails right after a release | Update the script; re-verify the proof of success | Release notifications; scripts versioned with the app |
| Account expired or locked | All vantage points fail at login at once | Reset, unlock, rotate the stored secret | Documented account owner; expiry reminders; stop retrying on auth failure |
| Certificate or DNS change | First step fails with trust or connection error | Update trust store or name; remove hard-coded addresses | Use names users use; monitor certificate expiry separately |
| Test data removed | Search returns nothing; later steps skipped | Recreate tagged records | Tell data owners; the journey creates what it needs |
| New dialog or prompt | Step times out on a covered element | Handle or dismiss the dialog in the script | Pre-release run against staging |
| Runner drift | One vantage point fails, others fine | Repair the runner; compare with a healthy one | Identical images; monitor runners as servers |
The test that always passes
A test that never fails feels like good news, and sometimes it is. More often it means the test is checking the wrong thing. The classic examples are embarrassing precisely because they are so easy to write. A login step that asserts "status 200" passes when the application returns a 200 with "invalid credentials" in the body. A search step that checks "the page loaded" passes when the page loaded with zero results. A step that looks for the word "Welcome" passes on the error page that also says "Welcome, please try again." Or suppose a journey's login step silently lands back on the login form. Its later steps all fetch that same form. It passes six steps in a row without ever being logged in.
The cure is a habit: test the test. When you build or change a journey, break the application on purpose. Do this in a staging environment, or by pointing the test at a deliberately wrong record, a wrong password, a stopped service. Confirm that the test fails, at the right step, with a useful message. If it passes anyway, it was never watching. Repeat the exercise at every review. An assertion that was sharp on day one can be dulled by an application change that made the expected text appear everywhere.
Two design rules make always-pass tests rarer. First, assert on the result, not on the page. Look for a specific customer name, a reference number, a row count, a timestamp you wrote in an earlier step. Second, record what each step actually received, even on success. A journey that keeps the response title or a screenshot from every run lets you audit it. Pick a random passing run from last week and check that it really did show the record. A journey that keeps only "PASS" cannot be audited, and an unauditable test drifts toward theater.
The test that always fails
The opposite failure is louder and, strangely, less likely to get fixed. A journey that has failed every night for three weeks has already taught everyone on the on-call rotation that its alerts mean nothing. That lesson generalizes. Soon all journey alerts are read as "probably that broken test again." One permanently red test poisons the credibility of every honest one beside it, and the day it fails for a real reason, nobody looks.
The rule is simple and must be enforced without exceptions. A failing test is fixed or silenced within a short, agreed time — a few working days at most. Silencing is never permanent. If nobody can fix it now, disable it with a note that says why and who owns it. Include a date on which the silence expires and the test either returns or is retired. A muted test with no expiry is a deleted test that still costs money. Keep the list of muted journeys visible on the dashboard. A quietly muted test is how "we were monitoring that" turns out to be false during a post-incident review.
Watch for the flaky test as well, the one that fails one run in twenty for no reason anyone can find. It is the always-fail test in slow motion: each individual alert is dismissed, and the habit of dismissing spreads. Treat intermittent failures as bugs in the test until proven otherwise. Look for a timeout set too tight, a step that does not wait for the page to settle, a runner short of memory. Fix the cause rather than raising the confirmation count until the noise disappears.
Tip: Make auth failures a stop, not a retry. A runner that keeps retrying a stale password will lock the test account within minutes. That turns a small maintenance task into an outage of every journey that shares the account.
Ownership and the review ritual
Every journey needs two names on it. One is an IT owner who is responsible for the script and the runners. The other is an application-side counterpart who can answer "is this still how users do it?" Without the first, nobody fixes the test. Without the second, the test slowly measures a workflow nobody uses anymore. Record both names, and the test account's owner, where the alert can display them.
Then hold a short review on a fixed cadence. Monthly is right for most shops; a busy application with frequent releases may need it after every release instead. The review is not a deep audit; it is a fifteen-minute walk through a checklist per journey:
- 1. Does the journey still reflect a real user path? Ask the counterpart. Features move, shortcuts appear, and the "search" users do today may be a different screen from the one the script drives.
- 2. Are the thresholds still right? Compare each step's current baseline with the thresholds in the alert rules. Applications get faster after an upgrade, leaving thresholds so loose that a real slowdown never trips them. Or they get slower, leaving thresholds that fire every afternoon. The guide on alerting on experience describes how thresholds are derived; the review checks that they were re-derived.
- 3. Did it fail this month, and were the failures true? Walk the failure list. Every false alarm is a maintenance task hiding as an incident; every true failure nobody acted on is a routing problem.
- 4. Is the test account healthy? Password age against policy, lockout history, remaining privileges, and whether the account still appears in the directory's list of known service accounts.
- 5. Does the test data still exist, and is it still tagged? Open the test record by hand.
- 6. Is every runner healthy and identical? Same script version, same image, same configuration; clocks in sync; no runner quietly missing.
- 7. Test the test. Break something and confirm the journey notices.
Write down the outcome in one line per journey, including "no change." A review log that shows a journey has been checked twelve times and adjusted three is evidence that the monitoring can be trusted. That is exactly what you will need when someone asks why the journey did not catch last Tuesday.
Versioning scripts with the application
Journey scripts are software, and they should live where software lives: in version control. They need a history, a change description on every commit, and a way to roll back. The step up from that is versioning them with the application. When the application ships a release, the journey script that matches it gets a tag with the same release name. If the application is rolled back, the matching script rolls back with it. If you are asked "what did the test look like when this incident happened?", the answer is one command away.
Keep the runner configuration — schedule, vantage-point list, thresholds, timeouts — in the same repository. That gives "what is the test doing right now?" one authoritative answer rather than one per runner. Deploy scripts to runners from that repository, never by editing a file on the runner by hand. A hand-edited runner is a runner that disagrees with the others. A disagreement among witnesses is the one thing a vantage-point comparison cannot survive. A layout that has worked for many teams:
synthetic-tests/
journeys/
records-core/
journey.yml # steps, proofs of success, per-step budgets
script.sh # or the browser-automation script
README.md # owner, counterpart, test account name, test data tags
runners/
hq-lan-01.yml # schedule offset, vantage-point label, proxy settings
branch-north-01.yml
alerts/
records-core.yml # thresholds, confirmation, severities, routing
CHANGELOG.md # one line per change: what, why, which app release
# tag the repository to match the application release it was verified against
git tag records-app-release-spring-refresh
The credentials do not live in the repository. The script reads them from a secrets store or an environment the runner is given at start. And the review checks that nothing secret has crept into the history.
Pre-release coordination with application teams
Most test rot arrives on release day, which means most of it is preventable by knowing about the release beforehand. The message to application teams is short: tell us before you change the login page. In practice that means three arrangements.
First, get the journey owner onto the release notification list. Get the journey listed as a consumer of the application in whatever the team uses to track dependencies. Second, add a line to the release or change template: "Does this change any screen, URL, or field used by synthetic monitoring? If yes, notify the journey owner." The application team usually cannot answer that without seeing the journey's design table. So give it to them. A one-page list of steps, URLs, and expected results is the best coordination document you will ever write. Third, where a staging environment exists, run the journey against the new build there before release day. A journey that passes on staging and fails in production after the deploy is telling you something about the deploy, which is a bonus.
For the deploy window itself, suppress the journey's alerts rather than letting them fire and be ignored. Planned quiet time is a normal part of alerting. The guides under Maintenance Windows & Suppression cover how to do it without losing track of what was silenced. Then run the journey by hand immediately after the deploy, before the window closes. That way, the first failure of a changed screen is found by you and not by the on-call phone at midnight.
Retiring tests for retired features
Features die. The report export that had its own journey is replaced by a dashboard. The legacy portal is decommissioned; the file share moves to a document system. A journey for a retired feature has exactly one future: it fails, forever, and becomes the permanently red test from earlier. Retire it on purpose, the day the feature goes, using the same discipline as adding one:
- 1. Confirm with the counterpart that the feature is really gone and nobody is quietly still using it.
- 2. Disable the journey and its alert rules at the same time, so no orphaned rule keeps firing on missing data.
- 3. Remove it from dashboards, or a grey cell will sit there implying something is unmonitored.
- 4. Archive the script in version control with a retirement note; do not delete the history.
- 5. Remove or repurpose the test account and data, and tell the directory and application owners you did.
- 6. Record the retirement in the review log, so the next reviewer does not go looking for it.
If a feature is replaced rather than removed, retire the old journey and build a new one for the replacement. Do that rather than editing the old one into something it was never designed to be. Journeys that have been patched through three generations of an application tend to test the seams between them rather than what users do today.
The maintenance checklist
Everything above, as a schedule. Assign each row to a named person, and put the recurring ones in a calendar. The whole point of maintenance is that it happens whether or not anything is on fire.
| When | Task | Owner | Evidence it was done |
|---|---|---|---|
| Weekly | Scan the failure list; every failure is classified true, false, or runner | Journey owner | Classification noted on each failure |
| Weekly | Check the muted-journey list; anything past its expiry is fixed or retired | Journey owner | Muted list has no expired entries |
| Monthly | Review ritual: real path, thresholds, account, data, runners, test the test | Journey owner + app counterpart | One line per journey in the review log |
| Monthly | Confirm test account password age and lockout history against policy | Account owner | Directory report attached to review log |
| Per application release | Run journey against staging; update and tag the script; suppress during deploy; run by hand after | Journey owner, notified by app team | Script tag matches release; post-deploy run recorded |
| Per runner change | Rebuild from the standard image; verify identical configuration | Runner owner | Runner inventory updated |
| Quarterly | Audit a sample of passing runs: did they really show the expected result? | Journey owner | Audit note in review log |
| Quarterly | Retire journeys for features that no longer exist; check for journeys that should exist and do not | Journey owner + app counterpart | Retirement entries; new-journey requests |
| After any incident | Ask: did the journey catch it? If not, why not, and what assertion would have? | Journey owner | Post-incident note; script change if needed |
Where to go next
A synthetic test is only as honest as its last review. Tests rot because applications, accounts, certificates, data, and runners all change under them. The always-pass test checks the wrong thing, and the always-fail test teaches people to ignore you. The cure for both includes named owners and a monthly ritual that includes deliberately breaking things. It also includes scripts versioned with the application and release coordination that warns you before the login page moves. It includes the discipline to retire a test with its feature.
The rest of this topic covers what the maintenance is protecting. Read scripted user journeys for building tests worth maintaining. Read where to run transaction tests from for the runner fleet. Read alerting on experience for the thresholds the review keeps honest. All of them start from the Synthetic Transactions & End-to-End Testing pillar.
Put this into practice with SysPeep Monitor
Journeys need hands-on maintenance because applications change; the infrastructure monitoring beside them should mostly look after itself. SysPeep Monitor auto-discovers devices and interfaces, so a moved server or a new link shows up without a script edit. Its dependency and alert rules keep the network side quiet during the deploy windows you have just coordinated. The free trial installs on any Windows machine.
Download Free TrialLearn About SysPeep