Home › Topics › Website & Web-Service Uptime Monitoring › Website Outage Triage

Website Down: A Triage Sequence for the First Ten Minutes


The message arrives the way it always does: "Is the website down?" Your pulse goes up, three browser tabs open at once, and somebody suggests restarting the server before anyone knows what is wrong. The first ten minutes of a web outage are where the most time is either saved or lost. And the difference is almost never technical skill. It is whether you have a sequence and follow it.


This guide gives you that sequence. It starts by confirming the outage from outside. Then it walks the layers a customer's request passes through in the order most likely to find the fault fastest. That order is DNS, certificate, hosting, the edge, the application, the backend. Along the way you get a table that maps the symptom you see to the first thing to check. You get a way of thinking about what changed last and a list of who to call at each layer. You get a note on what to say to the people waiting. And you get a routine for collecting evidence before anyone reboots anything. No prior incident experience is assumed; the point is that you should not need it.


In this guide

  • Minute zero: confirm from outside
  • The layered sequence
  • The decision table: symptom to first check
  • Rollback thinking: what changed last?
  • Who to call at each layer
  • Talking to people while it is down
  • The evidence file
  • Where to go next

Minute zero: confirm from outside


Before touching anything, answer one question: is it down for everyone, or for the person who reported it? A surprising share of "the website is down" reports are one office's internet link, one laptop's proxy, or one browser's cache. Restarting a healthy server to fix a broken Wi-Fi access point is a bad start to the morning.


If you have external checks running from several locations, look at them first. They answer the question in seconds, and the pattern of which locations fail is your first clue. The guide on multi-location checks explains how to read that pattern. If you have no external checks yet, improvise one that is genuinely outside. That could be a phone on mobile data with Wi-Fi turned off, or a remote host on a different network. Do not use the office network; it may be the thing that is broken.


While you are there, capture two things. Note the exact time, because you will need it for every conversation that follows. And copy the exact error text, word for word, from the outside view. It might be "server not found," "connection timed out," "your connection is not private," a numbered error page, a blank white page. That text is the single best clue you will get in the next ten minutes. And each version of it points at a different layer.


The layered sequence


A customer's request passes through layers, and an outage lives in one of them. The sequence below walks those layers from the outside in, for two reasons. The outer layers (DNS, certificate, reachability) can be tested in seconds from anywhere without any special access. And they include the failures that are embarrassingly common: an expired domain, an expired certificate, a firewall change. Ruling them out first costs almost nothing. The guide on the layers of a single web check explains each stage in depth; here the emphasis is on speed.


Layer 1: DNS and the registrar (minutes one to two)

Ask a public resolver for your name from outside, and compare the answer with what it should be:

nslookup www.example.com          # uses your current resolver
nslookup www.example.com 9.9.9.9  # asks a specific public resolver instead

Three outcomes matter. No such name for every name under the domain means the domain itself is gone. That means an expired registration, or a delegation change that removed your name servers. Check the registrar account immediately; an expired domain is fixable within hours if you act, and a nightmare if you wait. The wrong address means a record was changed, deleted, or pointed at a retired host; find the change and reverse it. Server failure or timeout means the authoritative name servers are unreachable, which is a DNS hosting problem. Remember that DNS changes take time to spread, governed by the record's TTL (time to live, the number of seconds resolvers may cache an answer). So a fix here may take minutes to reach everyone, and so may the damage from a bad change made an hour ago.


Layer 2: the certificate (minutes two to three)

If the outside view shows a security warning, the outage is the certificate, and nothing further down the stack needs attention until it is fixed. The warning text tells you which: expired, wrong name, or an issuer the browser does not trust (usually an incomplete chain). A command-line check confirms it in one line, and adding -v shows the certificate's names and dates:

curl -sSI https://www.example.com/ 2>&1 | head -3
curl: (60) SSL certificate problem: certificate has expired

The fix is to renew and deploy, or to deploy the certificate that was renewed but never installed, which is the more common story. Check every hostname that matters, including the bare domain and the API, because they may carry different certificates.


Layer 3: hosting and reachability (minutes three to five)

The name resolves and there is no certificate warning, but nothing answers. Test whether anything is listening on the web port from outside:

curl -sS -o /dev/null --connect-timeout 5 -w "%{http_code}\n" https://www.example.com/
curl: (28) Connection timed out after 5001 milliseconds

A timeout means packets are being dropped: the host is off, the network path is broken, or a firewall is discarding traffic. Check the hosting provider's status information and console, and check whether a firewall rule changed. Connection refused means the host is up but the web service is not listening. The service crashed or was stopped, or it is bound to a different port or address. That is good news of a sort, because you can reach the machine and restart the service. If you cannot reach the machine at all and the provider reports no problem, the fault is between you and them. In that case, it is the border router, the internet link, the address translation, or a firewall.


Layer 4: the edge (minutes five to six)

Many sites have something in front of the server: a content delivery network, a web application firewall, a reverse proxy, or a load balancer. Each can fail on your behalf. And each tends to produce a recognizable error page of its own, often with a request identifier on it. An error page in the edge's own style that says the origin is unreachable means the edge is fine and your server is not. A "forbidden" or "blocked" page for every visitor after a rule change means the firewall is rejecting everyone. A "bad gateway" or "no healthy upstream" from a load balancer means it has no working backend to send traffic to. The quickest test is to bypass the edge and hit the origin directly:

# Talk to the origin server by address while still presenting the right hostname
curl -sS -o /dev/null -w "%{http_code}\n" --resolve www.example.com:443:203.0.113.10 https://www.example.com/

If the origin answers and the edge does not, the edge is the layer. If the origin fails too, keep going.


Layer 5: the application (minutes six to eight)

The server is reachable, and it is answering with error pages or with nothing. Now the questions are: was there a deploy? Did a configuration change? Is the disk full? Is the process running? Look at the application log for the last few minutes; the first error after the last normal line usually names the cause. Restarting the application service is a reasonable early action at this layer, after you have copied the log and noted the state. A restart erases the evidence that would tell you why it happened.


Layer 6: the database and backend (minutes eight to ten)

The application is running, but every page that touches data fails or hangs while static pages work. That is the signature of a backend problem. The database is down, its disk is full, or its connection limit is exhausted. Or a long-running query is holding locks. Or a service the application depends on (a cache, a queue, an authentication provider) is unreachable. The application log will usually say "cannot connect" or "timeout" with the backend's name in the message. This is the layer where you most often hand off. And the evidence you have gathered on the way down is what makes the handoff quick.


The decision table: symptom to first check


The layers give you the sequence; the symptom lets you skip ahead. Match what the outside view showed you to the row below and start there.


What the outside seesMost likely layerFirst check
"Server not found," "can't resolve"DNS or registrarLook the name up from a public resolver; check the registrar account for expiry
Security warning, "not private," certificate errorCertificateRead the warning: expired, wrong name, or untrusted chain; check expiry dates for every hostname
Spinning forever, then "timed out"Hosting, network, or firewallTCP connect from outside; provider status; last firewall change
"Connection refused" instantlyWeb service on the hostIs the service running? Is it listening on the right port and address?
Branded error page from the edge or CDNEdge, or origin as seen by the edgeRead the page's message; bypass the edge with a direct request to the origin
"Forbidden" or "blocked" for everyoneWeb application firewall or access rulesLast rule change; the request identifier on the block page
"Bad gateway," "service unavailable," "gateway timeout"Load balancer or proxy with no healthy backendBackend health checks; application process state
Application error page, stack trace, "internal server error"ApplicationApplication log; last deploy; disk space; configuration change
Static pages fine, anything with data fails or hangsDatabase or backend serviceDatabase reachability and connections; the dependency named in the log
Page loads but is wrong or emptyApplication data or a third partyContent checks; third-party script reachability; recent content change
Fine from most places, down from one regionNetwork path or a regional edge nodeCompare locations; do not touch the server

Rollback thinking: what changed last?


Sites that were working yesterday and are broken today were, in most cases, changed in between. Somewhere in the ten minutes, and ideally at the start, ask out loud: what changed last? A deploy. A DNS edit. A certificate renewal. A firewall rule. A configuration tweak. An operating system update. A new third-party script. A change at the hosting provider. If your organization keeps a change log or a deployment history, open it before you open a terminal. The guide on change correlation covers the habit in depth.


Then apply a simple rule: if the last change can be undone in two minutes, undo it before you debug it. Roll back the deploy, revert the DNS record, restore the previous configuration file, disable the new firewall rule. If the site comes back, you have found the cause and bought yourself daylight to understand it. If it does not, you have eliminated the most likely suspect at almost no cost. Debugging a live outage forward, while customers wait, is the expensive option, and it should be the choice you make after the cheap one fails.


Tip: the change that broke the site is often not the one you think of first, because it was made by someone else. Ask in the team channel, "did anyone change anything today?" and mean it kindly. The answer "I only updated the WAF rules" has ended more outages than any command.


Who to call at each layer


Half of triage is knowing whose problem it is. Every layer has an owner, inside or outside the organization, and the sequence above is also a routing table. Keep the contacts in the same document as the sequence, because the moment you need them is the moment you do not want to search for them.


LayerWho owns itWhat to have ready when you call
DNS and registrarWhoever holds the registrar and DNS hosting accounts; often one person, sometimes a marketing agencyThe name, the answer you got, the answer you expected, the resolver you asked
CertificateWhoever renews and deploys certificates; sometimes automated, sometimes a calendar reminder nobody ownsThe hostname, the error text, the expiry date shown
Hosting and networkThe hosting provider's support; your own network team for the border and firewallTimestamps, the public address, outputs of connect and trace tests from outside
Edge and CDNThe edge provider's support; whoever administers the WAF rulesThe request identifier from the error page, the time, the affected hostnames
ApplicationDevelopers or the application vendorThe log excerpt, the last deploy time, the exact error page
Database and backendDatabase administrator or platform team; the vendor of any external serviceThe connection error from the application log, connection counts, disk state

If the answer to "who owns this?" is "nobody" for any row, you have found a risk worth fixing in daylight. The DNS row and the certificate row are the usual empty ones.


Talking to people while it is down


While you work the sequence, other people are waiting: the support desk, the sales team, management, customers. Silence makes every one of them assume the worst and start calling you, which slows the fix. So the first message goes out within the first few minutes, before you know the cause. It says three plain things. We know the site is down, we are working on it, and the next update comes at a stated time. Then keep that promise, even if the update is "no change yet." The guide on the first fifteen minutes of an incident covers roles and communication in detail.


Two things to avoid. Do not guess at the cause in public; a wrong guess repeated by the sales team becomes a fact you spend a week correcting. And do not promise a time to fix until you have found the layer. Saying "we expect to restore service within the hour" is a promise you cannot keep from minute three. If you have a status page hosted somewhere other than the site that is down, use it. A status page that goes down with the site is decoration.


The evidence file


Whichever layer the fault lands in, you will probably need to convince someone outside your team: a hosting provider, an edge vendor, a developer, or your own management afterward. The provider's first response to "your platform is down" is nearly always "it looks fine from here." The only reply that moves the conversation is evidence with timestamps. Collect it as you go, and collect it before restarting anything, because a restart destroys the most useful evidence there is.


A small capture routine, run from an outside vantage point and saved to a file named with the time, gathers most of what anyone will ask for:


date -u                                                    # timestamp in UTC
nslookup www.example.com 9.9.9.9                           # what the world resolves
curl -sS -o /dev/null --connect-timeout 5 -m 20 \
  -w "dns %{time_namelookup} connect %{time_connect} tls %{time_appconnect} status %{http_code}\n" \
  https://www.example.com/                                  # which stage fails
curl -sSI https://www.example.com/ 2>&1 | head -20        # headers, error text, request identifiers
traceroute www.example.com                                 # where the path stops (or tracert on Windows)

Add to that file the check results from every external location with their times. Add screenshots of the error page as customers see it and the request identifier from any edge error page. Add the relevant application log excerpt and a one-line list of what changed in the last day. State every time in UTC (Coordinated Universal Time, the single reference clock that avoids time-zone confusion) or say which zone you mean. A provider in another time zone cannot search their logs for "about ten past nine."


When you open the ticket, lead with the evidence: "From three external locations since 09:12 UTC, TCP connects to 203.0.113.10 port 443 time out; the trace stops at hop 7, inside your network; nothing changed on our side today; outputs attached." That ticket gets escalated. "Our site is down, please help" gets a template reply asking for exactly the things you already collected. The same file becomes the raw material for the post-incident review, which is where the ten-minute sequence gets improved for next time.


Where to go next


The first ten minutes of a web outage go best when you refuse to guess. Confirm from outside and copy the exact error. Walk the layers from the outside in: DNS and registrar, certificate, reachability, the edge, the application, the backend. Let the symptom jump you to the right row. Ask what changed last and undo it if you can. Call the owner of the layer with evidence in hand. Tell the waiting people what you know and when you will speak again. Save everything before anyone reboots. None of that requires brilliance; it requires a sequence written down before it is needed, and the discipline to follow it when the phone is ringing.


Most of the checks this sequence leans on should already be running before the outage. That is the subject of the rest of the Website & Web-Service Uptime Monitoring topic. It covers multi-location checks to confirm from outside in seconds. It covers per-stage web checks that name the failing layer for you. And it covers critical-path monitoring so the alert arrives before the phone call does.


Put this into practice with SysPeep Monitor

Layers three through six of this sequence are your own servers, firewalls, links, and databases. Those are what SysPeep Monitor watches: reachability, services, disk, CPU, and interface health over SNMP, WMI, and SSH. Its dependency-aware alerts say which layer fell over first. When the outside view says the site is down, the inside view should already know why. The free trial installs on any Windows machine in minutes.

Download Free TrialLearn About SysPeep