Metastable Failures: Why Your System Stays Down After the Cause Is Fixed
The trigger lasted two minutes. The incident lasted nine hours. Here's the failure mode your postmortem template can't see.
Short answer: you hit a metastable failure. The event that started the outage (the trigger) and the thing keeping you down (the sustaining effect) are two different mechanisms, and the second one doesn't need the first. In most real incidents the sustaining effect is your own retry traffic. Removing the trigger changes nothing, and a system in this state cannot recover on its own — it needs work removed, not rescheduled, which is why exponential backoff doesn't end it and load shedding does.
If you've ever rolled back a deploy in two minutes and then watched dashboards stay red for hours, this is the mechanism, and it's why the root-cause field in your postmortem was misleading.
Key takeaways
Trigger ≠ root cause. The trigger is brief and unpreventable. The sustaining effect is a feedback loop your system runs against itself, and it's the thing you can actually fix.
Retries are the most common sustaining effect — present in more than half the real-world incidents catalogued in the OSDI '22 study.
Goodput collapses while utilisation stays pinned at 100%. Your servers are busy processing requests whose clients already gave up.
Backoff and jitter are necessary but not sufficient. Jitter fixes synchronisation; neither fixes amplification.
The exits are retry budgets, deadline propagation, edge shedding, and circuit breakers — all of which reduce total work rather than spreading it out.
Most production systems run in the "vulnerable" state deliberately, because headroom is expensive.
What is a metastable failure?
A metastable failure is an outage that persists after its trigger is removed, sustained by a feedback loop inside the system itself.
The framework comes from Bronson et al., Metastable Failures in Distributed Systems (HotOS '21). It separates an outage into two mechanisms:
| What it is | Duration | Can you prevent it? | |
|---|---|---|---|
| Trigger | Deploy, load spike, cache flush, network blip | Usually brief | Not reliably — the supply is unbounded |
| Sustaining effect | A self-feeding loop: work amplification or capacity degradation | Lasts until forcibly broken | Yes, and this is the real work |
The paper's central point: incidents like this get blamed on the trigger, but the true root cause is the sustaining effect. Failures that do resolve when the trigger is removed — a DoS that stops, a livelock — are explicitly not metastable.
Metastable Failures in the Wild (Huang et al., OSDI '22) went looking for these in public incident reports and found 22 confirmed cases across 11 organisations, including AWS, Google Cloud, Azure, IBM, Spotify, Elasticsearch and Cassandra. At least 4 of the 15 major AWS outages of the previous decade were metastable. Recovery durations ranged from 1.5 to 73 hours.
The most common sustaining effect, in over half the catalogued incidents, was the retry policy.
How do retries sustain an outage?
Every arrow in this loop is a feature behaving as designed:
requests slow down
-> requests exceed the client timeout
-> clients retry
-> offered load increases
-> queues grow, latency climbs
-> requests exceed the client timeout (loop closes)
Nothing in that cycle references the trigger. Once it's turning, it supplies its own fuel.
Run the arithmetic
A service with 1,000 rps of capacity, normally serving 700 rps. Something pushes p99 past the 2-second client timeout. The client library retries twice — an entirely ordinary default.
baseline offered load 700 rps
attempts per request 3 (1 initial + 2 retries)
------------------------------------------------
offered load under retry 2,100 rps
served capacity 1,000 rps
deficit 1,100 rps <- does not depend on the trigger
Roll back the deploy. The deficit is unchanged, because the retries are generated by the timeouts, the timeouts are generated by the queueing, and the queueing is generated by the retries.
Amplification compounds across layers
The number is usually worse than people expect, because retry logic lives at several layers that don't know about each other:
SDK / client library 2 attempts
service mesh sidecar 2 attempts
application-level retry 2 attempts
------------------------------------
total attempts 2 x 2 x 2 = 8, not 6
Retry layers multiply. Audit yours before you assume it's 3x.
Goodput vs throughput
The metric that makes this legible is goodput — throughput of work that's still useful to someone.
During a retry storm, CPU utilisation sits at 100% while goodput approaches zero, because a large share of the work is being spent on requests whose client timed out seconds ago and has already sent a replacement. Every resource dashboard reports a healthy, busy system.
Your servers are at 100% CPU doing work nobody is waiting for anymore.
If you only alert on utilisation and error rate, this failure is close to invisible until customers tell you.
Why is my system vulnerable in the first place?
The framework describes three states:
Stable — enough headroom that the loop can't sustain itself. Remove the excess load and the system recovers unaided.
Vulnerable — the loop is possible but hasn't been started. Nothing looks wrong. A system can sit here for years.
Metastable failure — the loop is running and self-sustaining. Requires a strong corrective action to exit.
The uncomfortable finding is that many production systems choose to run in the vulnerable state permanently, because it is substantially more efficient than the stable one.
That's a rational trade. Stable means idle capacity, and idle capacity is a line item. But it means every autoscaling target you tightened and every over-provisioning review you passed moved you closer to the boundary — deliberately, and usually correctly. The problem isn't the trade. It's making it without knowing you made it.
Why doesn't exponential backoff fix it?
Because backoff reschedules work; it doesn't remove it.
This is worth separating carefully, because two distinct mechanisms get conflated:
Jitter solves synchronisation. Without it, everything that failed at the same instant retries at the same instant, producing a thundering herd in phase. Jitter smears arrivals out. Genuine problem, genuine fix.
Backoff does not solve amplification. Three attempts spread across eight seconds is still three attempts. If the deficit is structural (2,100 offered against 1,000 served), spreading arrivals doesn't close it — the queue drains slower than it fills, and it will keep filling.
Backoff slows the loop down. It doesn't turn it off. A metastable system doesn't need the loop to be fast — only self-sustaining.
Ship backoff and jitter. Just don't file them under "retry storms: handled."
What actually breaks the loop?
Anything that reduces total offered work. Four mechanisms, roughly by implementation cost:
1. Retry budgets (highest leverage)
Not per-call retry counts — a fleet-wide cap. Permit retries only while they remain under a fixed fraction of total request volume:
# Token-bucket retry budget: retries capped at ~10% of traffic.
# Converts worst-case amplification from 3x to ~1.1x by construction.
class RetryBudget:
def __init__(self, ratio=0.1, capacity=100):
self.ratio, self.capacity = ratio, capacity
self.tokens = capacity
def on_request(self):
self.tokens = min(self.capacity, self.tokens + self.ratio)
def allow_retry(self) -> bool:
if self.tokens >= 1:
self.tokens -= 1
return True
return False # budget exhausted: fail fast, do not queue
Per-call limits cannot do this. A max_retries=2 setting has no idea what the other 900 requests per second are doing.
2. Deadline propagation
Pass the client's remaining time budget through the call chain and drop work already past it:
async def handle(request, deadline_ms: int):
if deadline_ms <= 0:
raise DeadlineExceeded() # never start doomed work
start = now_ms()
result = await db.query(request, timeout_ms=deadline_ms)
remaining = deadline_ms - (now_ms() - start)
return await downstream.call(result, deadline_ms=remaining)
Most systems will happily spend a full database query on a request that expired four seconds ago. This is the cheapest goodput you will ever recover.
3. Shed at the edge, fast
Return 429 at the load balancer instead of queueing. A rejected request costs microseconds; a queued one holds a connection, a worker, and a pool slot that something useful needed — the same mechanism described in Your Database Is Fine. Your App Is Dying Anyway.
Queueing is not politeness. It's deferred rejection with a resource cost attached.
4. Circuit breakers on the client
When a dependency is clearly failing, stop calling it. The non-obvious benefit is to the caller: you stop burning your own threads and connections waiting on something that won't answer.
The prerequisite
Retries are only safe if the operation is idempotent. A retry storm against a non-idempotent endpoint produces an outage and a data corruption incident — the mechanism behind an API that charges a customer twice.
Three questions to answer before the next incident
What is my maximum amplification factor? Multiply out every retry layer, including the SDK you forgot retries. If the answer is above 3x, that's your priority.
Can I shed load without a deploy? If dropping traffic requires shipping code, you can't do it during an outage — the pipeline is one of the things that's broken. It needs to be a config flag or an LB rule.
Where does expired work get dropped? In most systems the honest answer is nowhere.
The postmortem field that catches this
Replace one question.
Not "what caused this outage." Ask: "what kept it going after the cause was gone?"
If the honest answer is "we scaled the fleet 3x" or "we drained traffic and let it settle," you didn't fix a metastable failure — you overwhelmed one. It's still there, waiting for a different trigger. And next time the trigger may be a cache eviction or a routine restart rather than a deploy you can roll back in two minutes.
You will never run out of triggers. You can absolutely run out of sustaining effects.
The trigger is the part with a timestamp and a name attached, which is why it ends up in the root-cause field. The loop is the part you can actually eliminate.
FAQ
What is a metastable failure in distributed systems?
A metastable failure is an outage that persists even after the triggering condition is removed, because a feedback loop inside the system — usually retry-driven work amplification — supplies enough load to keep the system in a degraded state. Goodput stays near zero while utilisation stays at capacity. Exiting requires a strong external corrective action, typically load shedding.
Why is my system still down after I rolled back the bad deploy?
Almost certainly because the retries generated during the incident are now sustaining it on their own. The rollback removed the trigger, but offered load is still amplified above capacity, so requests keep timing out and clients keep retrying. Nothing in that loop depends on the bad deploy still being deployed.
Does exponential backoff with jitter prevent retry storms?
Jitter prevents synchronised retry bursts (thundering herd), which is a real and separate problem. Neither backoff nor jitter reduces the total number of attempts, so neither closes a structural capacity deficit. Use both, but pair them with a retry budget, which does cap total work.
What is a retry budget and how do I set one?
A retry budget is a fleet-wide cap on retries as a fraction of total request volume, usually implemented as a token bucket. A common starting point is 10%, which bounds worst-case amplification at roughly 1.1x instead of 3x. Unlike max_retries, it accounts for what the whole fleet is doing rather than one call site.
What is goodput and why does it matter during an outage?
Goodput is throughput of work that is still useful to a waiting client, as opposed to raw throughput which counts everything processed. In a retry storm the two diverge sharply: CPU utilisation stays at 100% while goodput approaches zero, because much of the work is on requests that already timed out. Alerting on utilisation alone will miss this entirely.
How do I tell a metastable failure from ordinary overload?
Remove the extra load and watch. Ordinary overload recovers on its own once demand drops below capacity. A metastable failure doesn't, because the system is now generating the load itself. If recovery required scaling up or draining traffic rather than simply waiting, it was metastable.
Related reading
Your Database Is Fine. Your App Is Dying Anyway. — connection pool exhaustion, a classic sustaining effect
The idempotency piece — when an API charges twice — the prerequisite for safe retries
Why Your App Is Fast in Dev and Dying in Production — N+1 queries, a common trigger
Start Here — the full Simply Explained map
Sources: Bronson, Charapko, Aghayev & Zhu, Metastable Failures in Distributed Systems, HotOS '21 · Huang et al., Metastable Failures in the Wild, OSDI '22 · Isaacs et al., Formal Analysis of Metastable Failures in Software Systems, arXiv:2510.03551
What's the longest an incident has outlived its cause on a system you've worked on — and did the postmortem name the loop, or stop at the trigger?
