Liveness vs Readiness Probes: What Your Health Check Should Never Check

Short answer: a liveness probe should never check your database, and a readiness probe usually shouldn't either. Shared dependencies fail on every instance at the same moment, and each probe's reader responds by acting on every instance at the same moment. Dependency health belongs in monitoring.

The rest of this post is the why, the platform behavior behind it, and copy-paste starting points.

The handler that looks responsible

@app.get("/health")
def health():
    db.execute("SELECT 1")
    cache.ping()
    return {"status": "ok"}

This passes review almost everywhere. The trouble is that it answers "is everything okay?", and nothing that polls it is asking that.

Who reads the check, and what they do with a failure

Same 503, four outcomes. Only the last row was designed for "the database is down."

Walking through a 45-second database failover

An illustrative scenario: 12 pods, the handler above wired into the liveness probe, Kubernetes default timing (probe every 10 s, 1 s timeout, act after 3 consecutive failures).

1. t = 0 s: the database starts failing over. /health returns 503 everywhere.
2. t ≈ 30 s: the third consecutive miss. Each kubelet restarts its pods, independently, with no fleet-wide view.
3. t = 45 s: the database is back. Your pods aren't. They're booting.
4. Boot: 12 pods open connection pools at once. At 20 connections each, that's 240 new connections hitting a freshly failed-over database, plus client retries.
5. If that overloads it: probes fail again, and the loop repeats.

Without the liveness check, the damage was ~45 seconds of failed DB-backed requests while cached and static paths kept working. With it, you get a full outage that outlives its trigger: a textbook metastable failure, with a reconnect flood that looks a lot like connection pool exhaustion.

Readiness instead? No restarts, which is better, but every pod goes not-ready together and the Service has nothing to route to. Henning Jacobs ("Liveness Probes are Dangerous") describes this with a Spring Redis health indicator, enabled by default, that marked every pod down during a brief Redis blip.

ALB fail-open? It only triggers when all targets fail. A partial failure, like one availability zone losing its path to the database, unroutes half the fleet and doubles the load on the other half. AWS's Builders' Library article on health checks says Amazon is skeptical of relying on fail-open in partial failures and keeps fast load balancer checks mostly local.

Shallow checks fail too, just in the other direction

Returning 200 whenever the process is up isn't the fix:

• The black hole. The same AWS article describes a server rendering blank error pages. Its check passed, errors were fast, so it looked like the quickest server and got more traffic.
• The side-door check. Jacobs describes a readiness probe on a separate management port that stayed green while the main server was stuck. A health handler on its own thread does the same while request workers are blocked.

Too deep kills healthy servers. Too shallow protects broken ones. The goal is a check its reader can act on.

One question per probe

The Everyone-Fails Test

For each line in the handler:

1. Who reads it, and what does it do on a fail? Restart, unroute, or page.
2. Can that action fix the cause? A restart fixes a deadlock. It can't fix a database.
3. What if every instance fails it in the same second? If the answer is "nothing serves," it's in the wrong place.
4. Move it to the reader that can act on it.

SELECT 1 in liveness fails step 2 and step 3. A worker-loop heartbeat in liveness passes both: a restart usually clears a deadlock, and if a bad release deadlocks every pod, restarting isn't making it worse.

A starting implementation

Separate endpoints (the kube-apiserver itself exposes /livez and /readyz), with dependency checks moved to a background task that only exports metrics:

import asyncio
from fastapi import FastAPI, Response

app = FastAPI()
state = {"warmed_up": False, "draining": False, "inflight": 0, "last_beat": 0.0}
MAX_INFLIGHT = 200

@app.get("/livez")            # reader: kubelet -> "restart me?"
def livez(resp: Response):
    if not worker_heartbeat_recent(seconds=30):
        resp.status_code = 503
    return {}

@app.get("/readyz")           # reader: Service/LB -> "route to me?"
def readyz(resp: Response):
    if not state["warmed_up"] or state["draining"] or state["inflight"] > MAX_INFLIGHT:
        resp.status_code = 503
    return {}                 # a database outage does NOT flip this

async def watch_dependencies():   # reader: monitoring -> "page a human?"
    while True:
        DEP_UP.labels("db").set(int(await probe_db(timeout=1)))
        DEP_UP.labels("cache").set(int(await probe_cache(timeout=1)))
        await asyncio.sleep(5)

Probe settings that give the app room to breathe. Treat the numbers as a starting point to tune against your own boot times and pauses:

startupProbe:              # holds off liveness until the app has booted
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30     # up to ~150 s to start
livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 6      # ~60 s of misses before a restart
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2

Three rules that keep this honest under load:

• Answer from memory. Probes read state; they never open a new DB connection. AWS recommends a background checker that updates a flag, plus monitoring of that checker thread itself.
• Same server, priority lane. Serve probes through the real serving path so a stuck server fails them, but reserve capacity so they're always admitted. Otherwise a busy pod misses a one-second timeout, restarts, and pushes its load onto the rest.
• Degrade, don't disappear. With the DB down, serve cached reads and static pages, and return 503 with Retry-After on writes. That instance is doing the best any instance can, so it stays ready.

Probe thresholds are config, and config ships with less scrutiny than code. That's the same gap behind production breaking with nothing deployed.

Key Takeaways

• A health check is an instruction to a reader, not a status report. Name the reader.
• Liveness: only conditions a restart fixes. Never shared dependencies.
• Readiness: whether this instance can serve. A shared database outage shouldn't unroute the fleet.
• Dependency health goes to metrics and alerts, where a human or rate-limited automation decides.
• Kubernetes Services don't fail open. AWS ALB does, but only when every target fails.
• Run the Everyone-Fails Test on every line of your handler.

FAQ

Should a Kubernetes liveness probe check the database?

No. If the database fails, every pod fails the probe and the kubelet restarts all of them. A restart can't fix the database, and the simultaneous reboot adds a reconnect flood the moment the database recovers.

Should a readiness probe check dependencies?

Only dependencies that are local to the instance, or ones where failing readiness actually moves traffic to a healthier instance. A shared database fails for every pod at once, which leaves the Service with zero endpoints and turns a partial outage into a total one.

What is the difference between liveness and readiness probes?

A failed liveness probe makes the kubelet restart the container. A failed readiness probe removes the pod from Service endpoints without restarting it. Liveness asks "would a fresh process fix me?"; readiness asks "should I get traffic right now?"

What does "fail open" mean for a load balancer health check?

When every registered target is unhealthy, an AWS Application Load Balancer routes requests to all targets anyway instead of dropping traffic. It protects against a shared dependency failing every check, but it doesn't trigger when only some targets fail.

What are the default Kubernetes probe timings?

By default a probe runs every 10 seconds with a 1-second timeout and acts after 3 consecutive failures, so roughly 30 seconds of failures before a restart or removal. A startup probe can hold off liveness checks while a slow app boots.

What is a deep health check, and when should I use one?

A deep health check tests dependencies like databases, caches, and downstream services. Run it in a background task that exports metrics and drives alerts, not in liveness or readiness, so a shared outage pages a human instead of restarting or unrouting your whole fleet.

Related reading

• The Cause Was Fixed. The Outage Wasn't.: why systems stay down after the trigger is gone
• Metastable Failures: Why Your System Stays Down After the Cause Is Fixed: retry storms and load shedding
• Your Database Is Fine. Your App Is Dying Anyway.: connection pool exhaustion
• Nobody Deployed Anything. Production Broke Anyway.: config as the unreviewed fast lane

The bottom line

Before you add a line to a health handler, ask who acts on it and whether their action can fix what you're checking. If the answer is "a restart, and no," that line will turn your next database blip into an outage.

Too deep, and the check kills healthy servers. Too shallow, and it protects broken ones.

More from Simply Explained: Start Here · Connection pool exhaustion · Metastable failures Get one of these in your inbox each week → subscribe to the newsletter

Adam Jaber is a software engineer who writes Simply Explained: complex topics, made simple, with no jargon and no hype.