Breaking an AWS and WordPress retry storm

A production incident looked like a traffic spike. The real failure was a feedback loop between uncached WordPress requests, shared application fragments and an origin that could no longer recover.

A feedback loop between edge traffic, WordPress, shared fragments and retriesA trigger became an outage only after the system started amplifying its own load.Edge traffic01WordPress origin02Shared fragments03Retry pressure04
A trigger became an outage only after the system started amplifying its own load.

01 / The field note

The useful question during an incident is rarely “which server is broken?” It is “which dependency is turning a recoverable problem into a loop?” In this case, an AWS-hosted WordPress origin was receiving expensive requests while another application repeatedly fetched shared page fragments from the same struggling origin.

Based on a real production incident. Client names, traffic values and infrastructure identifiers are withheld.

01

The failure was a system, not a server

The first signal was familiar: CPU saturation, slow PHP workers and pages that stopped completing. Restarting the worker pool restored service, but that did not explain why the origin could not recover on its own.

Two traffic shapes mattered. Uncached requests created the initial pressure. At the same time, a dependent application fetched shared WordPress fragments. As the origin slowed, those fetches timed out, failed to populate cache and were attempted again. The dependent application became an amplifier.

02

Build one timeline across the stack

CDN logs, origin access logs, PHP slow logs and infrastructure metrics were aligned on one timeline. That separated the trigger from the amplifier and ruled out several plausible distractions, including a deploy, a cache flush and an administrative action.

The health check also told an incomplete story. It observed a cached public page, so it stayed green while the uncached origin path was saturated. A health check is only useful when it exercises the failure mode that matters.

03

Remove the feedback loop

The shared fragments were given an origin cache with locking, background refresh and stale-on-error behaviour. Edge controls challenged the abusive request pattern before it reached PHP. The recovery path was then tested against the origin and through the CDN.

This reduced repeated rendering work and allowed the system to serve a last-known-good fragment while WordPress refreshed in the background. The incident response moved from restarting a process to changing the load path.

04

The operating lesson

A traffic spike is not a complete root cause. Production recovery needs a causal model: trigger, amplifier, failed guardrail and safe intervention. The same model applies to queues, webhooks, API retries and cache stampedes.

The strongest fix removes amplification first. Capacity can still matter, but more capacity only delays the next failure when the architecture keeps multiplying work.

03 / Working principles

The reusable part

What to carry into the next system.

  1. 01

    Correlate edge, origin, runtime and application evidence on one incident timeline.

  2. 02

    Treat retries and uncached shared dependencies as potential load multipliers.

  3. 03

    Probe an uncached origin path when a cached homepage can hide saturation.

  4. 04

    Prefer cache locking, stale serving and edge controls over repeated manual restarts.

05 / Contact

AI · AWS · DevOps · WordPress · Software

Need this kind of decision in your system?

A discovery call is enough to map the constraint, identify the evidence still missing and decide on the smallest useful intervention.

Start with the problem

Tell me what needs to move.

A short description is enough. I will review it personally and reply with a useful next step.

By sending this enquiry, you confirm that you have read how the information is handled in Legal & privacy.