Us or Them? A Fast Isolation Playbook for SaaS Outages
The CRM (the customer relationship management system, where sales keeps its records) stops loading at 9:12 on a Tuesday. Within three minutes you have two tickets and a chat message from the sales director. A colleague is leaning over the partition asking whether "the internet is down." Everyone wants the same thing from you, and it is not a fix. It is a verdict: is this us or is this them? Until you can answer, you cannot say who is working on it or tell users what to expect. You cannot decide whether the next ten minutes belong to your firewall or to the vendor's support line. I have given those ten minutes to the wrong one more than once.
The verdict is almost always reachable in about ten minutes, without any special tooling, by asking a fixed series of questions in a fixed order. Each question rules out one piece of the road between your users and the hosted service. By the end, the fault has been squeezed into one segment: your network, your ISP (internet service provider), the shared identity provider, or the vendor. Then you know whose phone to pick up.
This guide is that sequence. It assumes you have a list of the services the business depends on (the dependency inventory). Ideally, you also have synthetic checks running from more than one place. The playbook works without them, just more slowly. By the end you will have the seven steps and a decision table that maps results to verdicts. You will know what evidence to capture while it is fresh. You will also have a worked example and the checks to run after the vendor says "resolved."
In this guide
- Before you start: what "us or them" really asks
- The ten-minute sequence
- The decision table
- Evidence to capture while it is happening
- A worked example
- The reset: what to check after "resolved"
- Where to go next
Before you start: what "us or them" really asks
Between a user's browser and a hosted service sits a chain of segments. It includes the user's device, your LAN (local area network), and your DNS resolvers (the servers that turn a name into an address). There are your firewall and web filter, your ISP, and the wider internet. Then come the vendor's DNS host and content-delivery network (the vendor's rented cache servers near your users). Add the identity provider that signs the user in, and finally the vendor's application. "Us or them" is shorthand for a more precise question: which segment is broken, and who owns it? Your side ends at your ISP's edge. Their side begins at the vendor's front door. In between are things neither of you fully controls: upstream networks, and shared services like the identity provider, which is "them" too, just a different them.
Two habits make the sequence work. First, do not start with the vendor's status page. It is late by design, green by default, and looking at it first anchors you on whichever answer it happens to show. It is step seven for a reason. Second, do not start by rebooting anything. A reboot destroys evidence and rarely fixes a reachability problem. Start by asking questions.
A little preparation makes the ten minutes shorter. Keep a printed or offline copy of the inventory with each service's real hostnames, your tenant identifiers, and the vendor support numbers. That way, none of it depends on a service that might be the one that is down. Know your office's public IP address (the address the rest of the internet sees your office as) in advance. The vendor will ask for it, and looking it up during the outage costs a minute you would rather spend elsewhere.
At Ashgrove Housing the resident portal stopped loading one Monday morning. Three people started on it at once with no agreed order. One rebooted the firewall, which cleared its logs. One changed the DNS forwarders, which changed nothing. One opened a vendor ticket that said "the portal is down for us" and got back a request for the public IP address. Forty minutes in, the receptionist loaded the portal on her phone and it opened at once. The cause was a web-filter category update that had reclassified the vendor's new address range. The review noted that the firewall reboot had "limited the available evidence". The seven steps are now on a laminated card by the help desk, and the phone test is step two.
The ten-minute sequence
The minute markers are a budget, not a stopwatch. If a step gives you a clear answer early, skip ahead. If it is ambiguous, move on rather than dwelling, because a later step usually resolves the ambiguity.
Step 1 (minutes 0–2): Scope it
One user, one site, or everyone? Ask the reporters three things: exactly what fails (which address, which app, what the error says), from where (office, branch, home), and since when. Glance at the ticket queue and the chat channel for other reports, and at your monitoring dashboard if you have synthetic checks. The answer sorts the outage immediately. One user points at their device or account. One site points at that site's path. Everyone, including remote workers, points at a shared dependency or the vendor.
Step 2 (minutes 2–3): Test from a second network
The fastest "outside" you own is a phone. Turn on a mobile hotspot, connect a laptop to it (Wi-Fi off, so it is not still using the office), and open the service. This single test bypasses your LAN, your DNS resolvers, your firewall, your web filter, and your ISP all at once. If the service works from the hotspot and fails from the office, the problem is somewhere on your side. If it fails from both, the problem is beyond your network. It is the vendor, or something shared such as the identity provider or the vendor's own DNS. Keep the tethered laptop handy; you will use it again.
Step 3 (minutes 3–5): Resolve the name from inside and outside
DNS is the first thing every connection needs and the most common thing to break on your side. Look the service's real hostname up against your internal resolver, then against a public resolver, and compare. On Windows:
nslookup app.crm.example.com <- your default (internal) resolver nslookup app.crm.example.com 10.0.0.5 <- a specific internal resolver nslookup app.crm.example.com 1.1.1.1 <- a public resolver, for comparison Resolve-DnsName app.crm.example.com -Server 10.0.0.5 <- PowerShell alternative
No answer internally but a good answer publicly means your resolver or its forwarders: us. A wrong or stale answer internally (an address that no longer responds) points at a cached record, a manual override, or a filtering resolver: still us. No answer from anywhere means the vendor's DNS host is failing: them, and a bad enough version that everything they run is probably down. Matching, sensible answers everywhere mean DNS is innocent; move on. If you want the deeper version of this test, Proving It Isn't DNS covers it.
Step 4 (minutes 5–6): Is the identity provider healthy?
Your services may use single sign-on (SSO, one login that every application trusts). If so, one sick identity provider (the service that performs that login) makes every application look broken while every application is fine. Open the identity provider's login page directly, from the office and from the hotspot. Then open one hosted service that does not use single sign-on, if you have one. If the login page fails or hangs and the non-SSO service works, you have found the in-between case: the identity provider is down. That is a vendor problem with a blast radius the size of your whole company. If the login page loads but sign-in fails after the redirect, the trouble is somewhere in the handshake between the two vendors. Auth Path Troubleshooting walks that chain.
Step 5 (minutes 6–7): Are other hosted services working?
Open two unrelated Tier 1 services from your inventory, from the office. If they all fail, you are not looking at a vendor problem. In that case, you are looking at your internet path, your DNS, or the identity provider. The earlier steps should already be pointing at which. If only the original service fails, the fault is specific to that vendor or to the road between you and that vendor. The next step separates those cases.
Step 6 (minutes 7–9): Compare vantage points
Now read the pattern. Which places can reach the service: the office, each branch, a remote worker, the phone hotspot, an outside check point if you run one? The rule is simple: the fault lives where the failing paths overlap and the passing paths do not. Office fails alone: the office path. Office and branch fail while remote and outside pass: something those sites share, such as a common DNS forwarder, a firewall policy, or one ISP. Everything fails everywhere: the vendor or its dependencies. If you have synthetic checks from several vantage points, this step takes thirty seconds. The synthetic checks guide shows how to set them up so that next time it does.
Step 7 (minutes 9–10): Now, and only now, the status page
With a verdict in hand, open the vendor's status page. You are not asking it whether there is a problem; you already know. (It will tell you anyway, eventually.) You are asking it to confirm, to give you the vendor's vocabulary and incident identifier for your ticket, and to reveal whether the incident is regional. If the page is red, your verdict is confirmed and your first user message can quote it. If the page is green and your evidence says "them," the page is late, which is normal; open the ticket anyway. Status Pages: Trust, but Verify explains why the page belongs last.
The decision table
The steps produce a handful of observations. This table turns the common combinations into a verdict and a first suspect. "Hotspot" means the second-network test in step 2. "Outside" means any vantage point that crosses none of your network. "IdP" is the identity provider from step 4.
| Scope | Hotspot | DNS | IdP | Other services | Verdict | First suspects |
|---|---|---|---|---|---|---|
| One user | — | — | — | Work for others | Us (the device or account) | Browser cache, expired session, locked account, missing multi-factor device, local proxy setting, wrong clock |
| One site | Works | Differs or fails at that site | Fine | Fail at that site | Us (that site's path) | Site DNS forwarder, site firewall, site link or ISP |
| All sites | Works | Fails or wrong internally | Fine | Fine | Us (DNS) | Internal resolvers, forwarders, a filtering resolver, a stale record |
| All sites | Works | Fine | Fine | Fine | Us (a rule) | Firewall or web-filter rule, an allow-list that missed a new vendor address, a proxy inspecting TLS |
| All sites | Works | Fine | Fine | All fail | Us (the path) | Internet link, ISP, firewall health, an upstream route |
| All sites + remote | Fails | Fails everywhere | — | Fine | Them (their DNS) | The vendor's DNS host or domain |
| All sites + remote | Fails | Fine | Fails | SSO ones fail, non-SSO fine | In between (identity) | The identity provider; open a ticket with that vendor |
| All sites + remote | Fails | Fine | Fine | Fine | Them (the provider) | Vendor application, region, or their cloud platform |
| All your sites | Works | Fine | Fine | Fine; outside and remote pass | In between (your ISP) | Your ISP's route to that vendor; still your call to make |
Two notes on the in-between rows. An identity provider outage is a vendor problem, but not the vendor your users are complaining about. The ticket goes to the identity vendor. The message to users must explain why "everything" is down at once. An ISP routing problem is technically outside your building, but the ISP is your supplier. So the call is yours to make, with the traceroute evidence below in hand. That is a trace of the hops a packet takes toward the vendor, and where it stops.
If the observations contradict each other, repeat steps 2 and 3 before trusting the table. A phone hotspot has its own DNS resolvers and its own carrier network. So a hotspot that also fails can occasionally be the carrier's problem rather than the vendor's. A second outside vantage point, or a colleague on a different carrier, settles it.
Evidence to capture while it is happening
Whatever the verdict, spend two minutes saving what you saw. Evidence gathered during the outage is worth ten times what you can reconstruct afterward. It serves two purposes: convincing the vendor's support desk quickly, and convincing your own organization later about what actually happened. The principle is the same one used to prove the network was innocent in an on-premises incident, laid out in Exoneration Evidence. Here it is aimed outward.
Timestamps first check failure (per vantage point); first user report; status page changes; recovery Scope who is affected (users, sites, remote); which paths (browser, client app, API) DNS nslookup output from inside and outside, with the addresses returned Path tracert / pathping to the returned address, from office and from hotspot Connection curl -v output: TCP connect, TLS handshake, certificate subject and expiry, HTTP status Errors exact error text and codes as users see them; a screenshot of one Identity does the IdP login page load; do non-SSO services work Status page component, region, phase, and incident ID when it appears Your side your public IP address; tenant ID; any change made in the last 24 hours
A vendor ticket that opens with this gets escalated: "Since 09:12, we have received HTTP 503 from your login endpoint, from three of our sites and an outside check point. DNS resolves to these addresses. Traceroute completes, and the TLS handshake succeeds. Our public address is X, and our tenant ID is Y." A ticket that says "the CRM is down for us" gets a request for exactly that information, an hour later.
The path trace deserves one extra sentence. A trace that completes all the way to the vendor's address proves the road is open and puts the fault behind their front door. Suppose a trace dies inside your ISP's network, from every vantage point on your side but not from the hotspot. That evidence turns "we think it is the ISP" into a call the ISP has to take seriously.
Tip: save the raw command output, not your summary of it. Redirect each command to a text file named with the service, the vantage point, and the time, for example crm-office-0915.txt. Summaries drift; raw output does not, and the vendor's engineer may see something in it you did not.
A worked example
Tuesday, 09:12. The sales team at the main office reports the CRM "spinning forever." The branch office, when you ask, is fine. Here is the sequence in practice.
- Scope (09:13). Three reports, all from the main office. The branch and a remote worker on chat both say the CRM is fine. Your synthetic checks show the office check failing since 09:09 with a TCP connection timeout; branch, remote, and outside checks are green. Scope: one site.
- Hotspot (09:14). A laptop tethered to your phone loads the CRM immediately. The problem is on your side of the office's internet connection.
- DNS (09:16). The internal resolver and the public resolver return the same addresses for the CRM's hostname. DNS is innocent, but you notice the addresses are not the ones you remember from the firewall allow-list.
- Identity provider (09:17). The login page loads from the office. Other SSO services work. Not identity.
- Other services (09:17). Email, the file service, and the ticketing tool all work from the office. The internet link is fine; this is specific to the CRM from this site.
- Vantage points (09:18). Office fails; everything else passes. The fault lives in the office path, and DNS and the link are already cleared. That leaves the firewall and the web filter.
- Status page (09:19). Green. Consistent with your verdict: this is us.
Verdict at 09:19: us, most likely a firewall rule. The firewall log confirms it: outbound connections to the CRM's new addresses are being dropped. The vendor moved its front end into a new address range overnight, and your outbound allow-list still names the old one. You add the new range at 09:24, and the office check goes green at 09:25. And the whole thing is over in thirteen minutes with the evidence saved. Note what did not happen: nobody rebooted the firewall or spent twenty minutes refreshing a green status page. Nobody opened a vendor ticket for a problem the vendor could not have fixed.
For contrast, imagine the same morning with different answers. The hotspot fails too, DNS resolves everywhere, and the identity provider is fine. Other services are fine, and the outside check point is red. That is the bottom-right of the table: them. The right moves are the opposite ones. Stop touching your network, open the ticket with the evidence, send the first user message, and start tracking their incident.
The reset: what to check after "resolved"
The vendor's status page turns green and says "resolved." That is a claim, not a fact, and the ten minutes after it are where a tidy incident turns messy if you let it. Run this reset before you tell anyone it is over.
- 1. Re-run every check from every vantage point. Recovery is often regional, and "monitoring" or "resolved" on the page can arrive before your tenant catches up. Your own checks going green, from the office and the branches, is the only recovery signal that counts.
- 2. Test the client applications, not just the browser. Sync clients, desktop mail clients, and softphones may sit in a failed state and need a restart or a fresh sign-in. Check one of each before users discover it for you.
- 3. Re-run the jobs that failed. Scheduled API integrations that ran during the outage failed silently. Find them on the inventory and re-run or verify each one, or tomorrow's reports will be wrong.
- 4. Undo your workarounds. A temporary DNS override, a relaxed firewall rule, a bypassed web filter: every one of them must be reverted now. You still remember why each was made.
- 5. Record the recovery time from your data. The vendor's "resolved" timestamp and your first green check are two different numbers. Write both down; the gap is part of the record.
- 6. Update your internal status page and send the all-clear. Only after the checks agree, and with the honest caveat that client apps may need a restart.
- 7. Book the review. Ask the vendor for their post-incident report, and put fifteen minutes on the calendar with the service's business owner. The outage communication guide covers what that conversation should include.
Where to go next
With this sequence in your back pocket, "is it us or them?" stops being a debate and becomes a ten-minute procedure with a written answer. Run the reset afterward as well, because the workaround you forget today is next year's mystery. Each step leans on something else in Is It Us or Is It Them? Monitoring SaaS Reachability. The inventory tells you which services and hostnames to test. The synthetic checks make step six instant, and the status-page guide explains step seven. The communication guide takes over the moment the verdict is "them." The next time the CRM stops at 9:12, you will have a verdict by 9:22 and a folder of evidence to go with it.
Put this into practice with SysPeep Monitor
Steps one and six of this playbook are instant if you already have checks running from more than one place. SysPeep Monitor runs from a central Windows machine with lightweight remote probes at your other sites. It alerts through email and other channels. It keeps the timestamped alert history that becomes your evidence file when the vendor asks "since when?" The free trial installs in minutes.
Download Free TrialLearn About SysPeep