Timeouts Don't Cancel Work: Deadlines, Cancellation, and the Unknown Outcome

Here's a pattern that shows up in a lot of codebases:

try {
  const res = await fetch(`${PAYMENTS}/charge`, {
    method: "POST",
    body: JSON.stringify(order),
    signal: AbortSignal.timeout(5000),
  });
  return await res.json();
} catch (err) {
  // timeout, network error, whatever: treat it as failed and retry
  return retryCharge(order);
}

It looks defensive. It has a timeout. It retries. It's also a double-charge bug waiting for a slow afternoon.

The problem isn't the retry logic. It's an assumption buried in the catch: that a timeout means the work didn't happen. It doesn't mean that. A timeout ends the wait. It doesn't end the work.

What a timeout actually tells you

When your client times out, you've learned one thing about your own side: you stopped waiting. You've learned nothing about the other side. The request might never have arrived. It might be halfway done. It might have finished one millisecond after you gave up.

So every call has three possible outcomes, not two:

For reads, "unknown" is cheap: ask again. For writes, you have two honest options:

1. Find out. Query the downstream system for the result before acting again.

2. Make repeats safe. Send an idempotency key so the other side recognizes the retry and returns the original result instead of doing the work twice. (Deep dive: Why Your API Charges People Twice (And the One Idea That Fixes It).)

Treating timeout as failure is a third option, and it's a guess.

What keeps running after a 504

Take a slow report endpoint. A load balancer in front of the app gives up at 60 seconds (AWS load balancers default to a 60-second idle timeout). The query takes 90.

Here's what each layer does when the load balancer gives up:

The result: 30 seconds of a worker slot, a database connection, and CPU spent on an answer nobody can read. Call it zombie work.

One zombie is noise. Under load, zombies compound. More requests cross the limit, users and clients retry, each retry adds a new copy while the old ones still run, the database slows further, and more requests time out. Every zombie also holds a connection, which is how you end up with a healthy database and an app that's dying anyway, and how a short slowdown becomes an outage that outlives its cause.

Why raising the timeout makes it worse

Bumping 30s to 120s feels like a fix. It isn't:

• The work isn't faster. You moved the cliff.

• More work is in flight. Little's Law: requests in progress = arrival rate × time per request. Let slow requests hold workers twice as long and the same traffic ties up twice as many workers during a slowdown.

• Layers stop agreeing. If the load balancer still quits at 60s, a 120s app limit only guarantees more zombie work.

Build a shrinking ladder with deadlines

Every request path is a ladder of timeouts. The rule: each layer gives up before its caller does.

The cleanest way to get there is to stop configuring per-hop timeouts and propagate a deadline instead: one budget set at the edge, with every hop using whatever is left.

// Edge: one budget for the whole request
const deadline = Date.now() + 25_000;

function remainingMs(marginMs = 500) {
  return deadline - Date.now() - marginMs;
}

async function callDownstream(url) {
  const budget = remainingMs();
  if (budget <= 0) throw new Error("Out of time. Don't start the work.");

  return fetch(url, {
    signal: AbortSignal.timeout(budget),
    headers: { "x-request-budget-ms": String(budget) }, // let the next hop do the same
  });
}

gRPC has this built in: a server can pass the client's deadline to its own outgoing calls, and gRPC transmits the remaining time (elapsed time already subtracted) rather than a timestamp, so clock skew between machines doesn't matter.

Wire cancellation all the way down

A deadline nobody acts on is just a number. The stop signal has to reach the code doing the work.

Go: pass the request context into the database call.

ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
defer cancel()

rows, err := db.QueryContext(ctx, reportSQL)

gRPC servers: the docs put the responsibility on the server application to stop its own activity. Long handlers should check for cancellation.

PostgreSQL: put a ceiling where the work runs, regardless of what happened to the client.

BEGIN;
SET LOCAL statement_timeout = '20s';  -- scoped to this transaction
SELECT ...;                            -- the expensive report query
COMMIT;

Optionally, set client_connection_check_interval so long queries abort sooner when the client is gone (it relies on kernel support on Linux, macOS, illumos and the BSDs).

PHP: know what max_execution_time measures. On non-Windows systems it counts only script execution time. Time spent waiting on database queries, streams, and system calls is excluded. A 30s limit won't stop a script that spends 90s waiting on a query, so the database-side ceiling matters more.

Loops: check for cancellation between batches. The cheapest work to cancel is work you never start.

When the answer is "not a request"

If the honest timeout would be minutes, the work belongs in a background job:

POST /reports/export
→ 202 Accepted
{ "jobId": "exp_123", "status": "queued", "statusUrl": "/jobs/exp_123" }

The request covers only the quick part. The job gets its own lifetime, retries, and status. Double-clicks are easy to absorb: if an identical job is already running for that user, return it instead of starting another. The trade-off is real (a queue, a job table, a status flow to monitor), but the alternative is unowned work running on borrowed time.

The Timeout Audit

Pick your slowest endpoint and answer:

• [ ] Who stops waiting first? List every timeout on the path, browser to database.

• [ ] Does the stop signal reach the work? If the work continues after the caller leaves, it's zombie work.

• [ ] Is a timeout treated as unknown? No blind retries on writes without an idempotency key or status check.

• [ ] Does each layer quit before its caller? No inner limit longer than an outer one; no unbounded bottom rung.

• [ ] Should this be a request at all? Minutes of work means a job, not a request.

Key Takeaways

• A timeout ends the caller's wait. The work below keeps running unless cancellation reaches it.

• "Timed out" is a third outcome: unknown. For writes, find out or make repeats safe before retrying.

• Raising a timeout moves the cliff and increases work in flight during slowdowns.

• Timeouts form a ladder. Each layer should give up before its caller; propagate a deadline instead of fixed per-hop numbers.

• Put a hard ceiling where the work runs (e.g. statement_timeout), and check what your limits actually measure.

• Work longer than anyone will wait belongs in a background job with a status.

FAQ

Does a request timeout cancel the work on the server?

Not by itself. A timeout ends the caller's wait. The server keeps working unless it detects the disconnect or receives a cancellation signal, and many stacks only notice when they next try to write a response.

Does PostgreSQL stop a query when the client disconnects?

Not by default. PostgreSQL only notices a lost client when it next interacts with the connection, often when sending results. The client_connection_check_interval setting (default 0, off) makes it poll during long queries. statement_timeout caps execution time regardless.

Should I retry a request that timed out?

For idempotent reads, usually yes, with backoff. For writes, only if repeating is safe: use an idempotency key or check the result first, because the original request may have succeeded.

What is deadline propagation?

Setting one time budget at the edge of a request and passing the remaining time to every downstream call, so each layer gives up before its caller. gRPC supports it natively; with HTTP you pass the budget yourself, for example in a header.

Why doesn't PHP's max_execution_time stop slow database queries?

On non-Windows systems, max_execution_time counts only the script's own execution time. Time spent waiting on database queries, streams, and system calls isn't included, so use a database-side limit such as statement_timeout.

When should a long request become a background job?

When the work routinely takes longer than a user or caller will reasonably wait. Return 202 Accepted with a job ID, run the work in a worker, and expose a status endpoint.

Related reading

• Why Your API Charges People Twice (And the One Idea That Fixes It): idempotency keys, the safe way to retry an unknown outcome.

• Your Database Is Fine. Your App Is Dying Anyway.: how stuck work drains a connection pool.

• The Cause Was Fixed. The Outage Wasn't.: retry amplification and failures that sustain themselves.

The bottom line

Most systems have plenty of timeouts. What they lack is agreement on what a timeout means. The load balancer thinks "I'm done", the user thinks "it failed", and the database, never told, thinks nothing at all.

A timeout is a fact about waiting. Treat it like a fact about work, and production will correct you.

Adam Jaber is a software engineer who writes Simply Explained: complex topics, made simple. No jargon, no hype.