Tag Archives: ddos or outage

DDOS or Outage? How to Tell in 10 Minutes

DDOS or Outage? How to Tell in 10 Minutes

The question every on-call engineer eventually asks, somewhere between the first alert and the first angry customer email, is whether they are looking at a DDoS or an outage. It matters more than almost anything else you decide in the first hour, because the two answers send you in opposite directions: one escalates to a scrubbing provider and an attack evidence pack, the other escalates to a supplier status page and your own change log. Guess wrong and you lose the hour. Ten minutes of disciplined triage is usually enough to avoid that, provided you know which signals actually discriminate and which ones fit a hundred unrelated faults.

Why the first ten minutes decide the rest of the incident

Paging a distributed denial-of-service (DDoS) mitigation provider for a failed deployment wastes the one team who could have rolled it back. Rolling back clean code while a botnet chews through your search endpoint wastes the window in which mitigation would have been cheapest to apply. Both mistakes are common, and both come from the same root cause: a single person looking at a single browser tab and reasoning outwards.

Cloudflare’s global outage on 18 November 2025 is the cleanest recent illustration. Large parts of the web returned errors simultaneously, social media immediately assumed a record-breaking attack, and Cloudflare’s own published incident report attributed it to an internal failure involving an oversized bot management configuration file, not to malicious traffic. From a UK office, a worldwide provider failure and a worldwide attack looked identical for the first several minutes. Plenty of teams spent that time on the wrong phone call.

Set the ground rule before the next incident, not during it. One named person owns the diagnosis and says out loud what they currently believe and why. Everyone else gathers evidence and reports it to that person. Diagnosis by group chat produces four confident theories and no decision.

The evidence that actually separates a DDoS from an outage

Vague symptom lists are useless here. “The site is slow”describes a volumetric flood, a saturated database connection pool, a mis-sized instance and a bad ORM query equally well. What separates them is the shape of the traffic and the fingerprint of the errors.

Traffic shape before the failure. In an attack, request or packet rate rises first and errors follow. In an outage or a capacity fault, traffic is flat or falling while errors climb, because the same visitors are arriving and being turned away. Pull the request-rate graph and the error-rate graph on the same time axis and look at which line moved first. That single overlay resolves more arguments than any other artefact.

Error fingerprints. Connection timeouts, SYNs with no completed handshake and half-open sessions point at the network and transport layers. Clean 502s and 504s served promptly by a healthy edge point upstream, at your origin or at something between the two. A wall of 403s or interstitial challenge pages is frequently your own mitigation: a rate limit set too tight, or a WAF rule someone shipped on a Friday afternoon.

Who is affected. Everybody at once, or one mobile network, or one geography, or one point of presence? An outage confined to a single POP or a single ASN is a provider routing problem far more often than it is an attack. Global and total, with no gradient, tends to mean something above the web stack: DNS, certificates, domain registration or routing.

Source diversity against payload sameness. This is the signature no capacity problem can imitate. Take a worked contrast. Forty thousand requests per second arriving from six thousand distinct autonomous systems, nearly all of them aimed at one expensive search endpoint with near-identical headers, is an attack; organic traffic never distributes like that. Flat request volume, clean 502s at the edge and a normal origin process count is an upstream or deploy failure wearing an attack’s clothing.

Screenshot early. Edge dashboards frequently aggregate fine-grained data after a few hours, and the per-second view you need for an evidence pack may be gone by the time you write the report.

Check three surfaces in order: DNS, network, application

Order matters, because a failure in the first surface produces symptoms in all three.

Authoritative DNS first

Authoritative DNS is the layer most often left unmanaged, inherited from whoever registered the domain, and it fails in a distinctive way: mail, APIs, SSO callbacks and third-party integrations all break at the same second as the website. If your payment webhooks and your Microsoft 365 mail flow died alongside the site, stop looking at the web tier.

Resolve your zone against several public resolvers (for example 1.1.1.1, 8.8.8.8 and 9.9.9.9), then query each of your authoritative nameservers directly by IP address and compare SOA serial numbers. Check the delegation the registrar actually publishes against the NS records in the zone; the two disagree more often than people expect after a migration. Teams that have not thought about this surface before should read a proper treatment of authoritative DNS resilience and what to demand from a provider while nothing is on fire.

Network and transport second

Is your prefix still being announced? A public looking glass or a route-collector view answers that in under a minute, and a withdrawn or leaked prefix produces total, instant, global loss that feels exactly like a terabit-class flood. If the prefix is up, run traceroutes from at least two networks outside your estate. Packets dying several hops upstream, at the same router, for everyone, is a transit problem. A saturated link and a dark link look different on an interface graph: one is pinned at line rate, the other is at zero.

Application layer third

Thread starvation is the great impersonator. A worker pool exhausted by one slow database query produces queueing, timeouts and rising latency that reads like an application-layer flood until you put request concurrency next to origin CPU. Attack: concurrency high, CPU high, request mix skewed to one or two endpoints. Capacity fault: concurrency high, CPU idle, everything waiting on a lock or a downstream call.

The cross-check that settles most disputes is the edge-versus-origin comparison. Does the origin see far more traffic than the edge forwarded? Or does it see almost nothing while the edge reports normal volumes? Either mismatch tells you more in thirty seconds than an hour of log grepping.

Outages that get misreported as attacks, and attacks mistaken for outages

DNS provider failure versus DNS flood. Both produce the same maddening pattern: some users fine, others seeing nothing, depending on whose resolver cache still holds a valid answer. The test is to query each authoritative nameserver directly from more than one network. Responses from some anycast locations and timeouts from others suggests attack traffic or a partial platform fault; uniform timeouts everywhere, from every vantage point, suggests the platform itself. There is more detail on staying reachable through a DNS provider outage or an attack on your zone for teams building that plan properly.

Origin IP leakage. If the proxy reports normal traffic while the origin is being hammered, the attacker has your real IP address. This is a more common route to a bypassed proxy than sheer volume, and the causes are mundane: historical DNS records, a mail server on the same host, an unproxied staging subdomain, a certificate transparency log entry. The fix is re-IPing the origin and restricting its firewall to the edge provider’s published ranges. Buying more scrubbing capacity does nothing, because the attack is not touching the scrubber.

The boring catastrophes. An expired TLS certificate and a lapsed domain registration both produce total, simultaneous, global failure that is indistinguishable from a massive attack for the first few minutes. Check the certificate chain and WHOIS expiry before you dial anyone. Control-plane and dashboard failures deserve the same scepticism: if you cannot log in to change anything, the platform may be broken even when the data plane is serving traffic fine.

Self-inflicted downtime. On-demand mitigation creates its own outage window. Time-to-mitigate is not just the vendor’s detection number; it includes human authorisation and BGP convergence, and a diversion triggered mid-incident can look to users like a second, separate failure while routes settle. Anyone weighing that trade-off should understand what the seconds in a time-to-mitigate SLA actually cover before treating the figure as a promise.

Low-and-slow attacks. Application-layer attacks that hold connections open barely register on a bandwidth graph, so they get filed as performance degradation and sit in a backlog for days. Watch concurrent connection counts and time-in-request rather than megabits per second. If connection counts are at ceiling while throughput is unremarkable, you are not looking at a capacity problem.

What to do once you have the answer

If it is an attack: find out, now, who is contractually permitted to trigger mitigation. “Managed”varies wildly. Some contracts let a named engineer on the provider’s side change policy without waking you; others require the customer to raise a ticket and confirm the attack first, which means your protection is only as fast as whoever is asleep with their phone on silent. If a human on the provider’s side cannot change policy at 03:00 without your sign-off, you have bought monitoring, not management. Know your resale chain too: if your supplier resells someone else’s scrubbing network, your escalation carries an extra hop and your evidence has to survive being retold by a first-line agent. That question belongs in a structured comparison of vendor claims at contract time, but it bites during triage.

If it is a provider outage: work out whether the failing component is the one that publishes your failover. Changing nameservers during a DNS provider outage is slow and sometimes impossible. A zone with 3600-second records means a nameserver change can take up to an hour to reach resolvers, which is precisely why TTLs are lowered days before a planned cutover and not during an incident.

If it is you: roll back cleanly, one change at a time, and resist the temptation to describe a bad deploy as “a sophisticated attack”in the status update. Customers forgive mistakes. Engineers at your customers’ companies read traceroutes, and they will notice.

Tell the board and your customers what you know, what you do not know yet, and when you will next update. Then insist on a written post-incident report. A credible attack report contains a timeline in UTC, a vector breakdown, peak figures in bits per second, packets per second and requests per second, the moment mitigation engaged, and how much traffic was dropped against how much was passed. A credible provider root cause analysis names the change, the blast radius and the preventative action. “Elevated error rates were observed”is not an RCA.

Build the runbook before you need it

The five checks, in order, each with an owner and a command or dashboard written next to it:

  1. Resolve the zone from three public resolvers and query each authoritative nameserver directly; compare SOA serials.
  2. Compare the registrar’s published delegation against the NS records in the zone, and check WHOIS expiry and the certificate chain.
  3. Confirm the prefix is announced, then traceroute from two external networks.
  4. Overlay edge request rate against origin request rate, and note any mismatch in either direction.
  5. Put application concurrency next to origin CPU and the top endpoints by request count.

Host that card somewhere outside the failure domain, along with out-of-band monitoring from an external vantage point; monitoring that shares a CDN, a DNS provider or a cloud region with production will go dark exactly when you need it. Keep phone numbers, not just email addresses, for the scrubbing provider, the host, the registrar and the DNS operator, and confirm annually that the out-of-hours numbers still reach a human. The UK’s National Cyber Security Centre publishes denial of service guidance that is a sensible baseline for preparation and reporting expectations.

Finally, work out your downtime cost per hour. That number decides whether always-on protection is worth its premium over an on-demand arrangement with a convergence delay, and it sets how much evidence and contractual commitment you are entitled to demand at purchase. Answering “DDoS or outage?”in ten minutes is mostly a matter of having decided, in advance, which five things you will look at and who will be looking at them.

Frequently Asked Questions

How can I tell if my site is down from a DDoS or an outage?

Put request rate and error rate on the same time axis. If traffic climbed before errors appeared, and the sources are spread across thousands of networks hitting a narrow set of endpoints, treat it as an attack. Flat or falling traffic with rising errors points at a provider failure, a capacity fault or your own recent change.

What are the first three checks to run when a website goes down?

Resolve your domain against several public resolvers and query your authoritative nameservers directly. Confirm your prefix is still announced and traceroute from outside your own network. Then compare what the edge says it forwarded with what the origin actually received. Certificate expiry and domain registration status are worth a ten-second look in the same pass.

Can a DNS provider outage look the same as a DDoS attack?

Yes, and the user-visible pattern is nearly identical: some visitors keep working on cached answers while others get nothing at all. Query each authoritative nameserver directly from two different networks. Uniform timeouts from every vantage point suggest a platform failure; partial responses that vary by location lean towards attack traffic or a degraded anycast footprint.

Why does my site go down when my DDoS proxy says traffic is normal?

Usually because the attacker found your origin IP address and is going around the proxy entirely. Old DNS records, mail hosts, staging subdomains and certificate transparency logs all leak it. Re-IP the origin and restrict its firewall to your edge provider’s published ranges; extra scrubbing capacity will not help with traffic that never reaches the scrubber.

Who should I contact first, my hosting provider or my DDoS provider?

Contact whoever owns the surface your triage implicates. Clean 5xx errors from a healthy edge with no traffic spike means the host or your own deploy; a verified flood means the mitigation provider. Check in advance who on the provider side is contractually permitted to trigger mitigation without your written confirmation, and whether they resell another network’s scrubbing capacity, because that adds a hop to every escalation.