Out-of-Order Webhooks: Why Deduping Event IDs Isn't Enough

You already dedupe webhooks by event ID. Good. Here's the bug that check can't catch.

A customer's renewal fails at 09:14:07. They update their card, and the retry succeeds at 09:14:41. The first delivery of the failure event times out and waits in the provider's retry queue, so your endpoint receives invoice paid first and payment failed a minute later. Two different events, two different IDs. Both pass your dedupe check. The second one marks a paying customer past due.

Every handler was correct. The final state is wrong. This post is about why that happens and the handler shape that makes arrival order irrelevant.

TL;DR: Treat a webhook as a doorbell, not a delivery. Store and acknowledge it fast, dedupe the delivery, then fetch the object's current state from the provider (for state events) or record the occurrence once by its own ID (for payments, refunds, disputes). Run a scheduled sweep to catch events that never arrived.

What providers actually guarantee

Read the docs side by side and the pattern is clear: nobody promises once, in order, and on time.

On ordering specifically, Stripe's docs are explicit: delivery order isn't guaranteed, your endpoint shouldn't depend on it, and the second-resolution created timestamp shouldn't be used to decide order or detect duplicates.

Even if a provider did deliver in order, you'd break it yourself. Retries put old events behind new ones, and more than one worker draining a queue means whoever finishes last writes last.

Why "save the payload" fails

A webhook payload is a snapshot of an object at the moment the event was created. A handler that writes that snapshot to your database is trusting a photograph that could be a minute old, or days old after retries.

# The package handler: correct for each event, wrong in total
def handle(event):
    sub = event.data.object
    db.execute(
        "UPDATE subscriptions SET status = %s WHERE id = %s",
        (sub.status, sub.id),
    )

Feed this handler events in creation order and it's fine. Feed it the order the network chooses and the last write wins, even when the last write is the oldest news.

Fetch-on-notify: make order irrelevant

The doorbell handler reads two things from the payload, the object ID and the event type, then asks the provider for the object's current state.

# The doorbell handler: every event triggers a look at the present
def handle(event):
    current = provider.subscriptions.retrieve(event.object_id)
    save_subscription(current)

Replay the 09:14 scenario. Invoice paid arrives: fetch, active, save. Payment failed arrives late: fetch, still active, save. The late event can't rewind anything because it no longer carries state.

Stripe has leaned into this design. Its "thin" event format carries minimal data plus the related object's ID, the SDK exposes a method to fetch the latest version of that object, and the dashboard labels thin as the recommended payload format. Snapshot events still exist, and the docs note you can retrieve the resource from the API for the up-to-date version.

Sort your event types first: state vs occurrence

Fetch-on-notify is right for some events and wrong for others. Before writing a handler, put every event type you subscribe to into one of two columns.

State events converge. Occurrence events accumulate. Mixing them up produces one of two bugs: acting on a state change as if it were fresh news (the 09:14 bug), or overwriting an occurrence with the "latest" one and losing a refund.

Side effects follow the same rule. Don't send "your payment failed" straight from the event. Send it when your saved status transitions from active to past due, and record that it was sent, so a late or duplicate event can't trigger it.

The production shape: inbox, worker, lock

1. The endpoint: verify, store, acknowledge

The slow handler is a self-inflicted wound: slow responses time out, timeouts trigger retries, retries create more duplicates and more reordering. Keep the request path to three steps.

CREATE TABLE webhook_inbox (
  event_id     TEXT PRIMARY KEY,   -- provider's event ID: dedupe for free
  event_type   TEXT NOT NULL,
  object_id    TEXT NOT NULL,
  payload      JSONB NOT NULL,
  received_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
  processed_at TIMESTAMPTZ          -- NULL until a worker finishes it
);

-- after signature verification on the RAW body:
INSERT INTO webhook_inbox (event_id, event_type, object_id, payload)
VALUES ($1, $2, $3, $4)
ON CONFLICT (event_id) DO NOTHING;
-- then return 2xx

2. The worker: two columns, one lock per object

def process(event):
    if event.type in STATE_EVENTS:
        with lock_object(event.object_id):          # one writer per object
            current = provider.fetch(event.object_id)
            save_state(current)                     # the present, not the photo
    elif event.type in OCCURRENCE_EVENTS:
        record_once(event.payload)                  # unique on payment/refund ID
    mark_processed(event.event_id)

The lock matters. Without it, worker A fetches "active," the customer cancels, worker B fetches "canceled" and saves, then A saves its older "active" on top. That's a lost update with an HTTP call in the middle. Fetching and saving inside a per-object lock (row lock, advisory lock, or a queue partitioned by object ID) guarantees the later save comes from the later fetch.

Fetch from the provider, not from your own mirrored table or a read replica. Reading your own lagging copy brings the stale-read problem straight back.

3. Coalesce fetches to protect your rate limit

When several events for one object are pending, one fetch covers all of them:

-- group pending events by object (Postgres)
SELECT object_id, array_agg(event_id) AS event_ids
FROM webhook_inbox
WHERE processed_at IS NULL
GROUP BY object_id
LIMIT 50;

Fetch each object once, save, and mark every claimed event processed. If the fetch fails, leave processed_at NULL and retry from your inbox with backoff. You already acknowledged the delivery, so retries are yours now, on your schedule.

4. No read API? Use a version that only goes up

If you can't fetch, you need a per-object version from the provider that only increases, and a write that refuses to go backward:

UPDATE subscriptions
SET status = :status, version = :incoming_version
WHERE id = :id AND version < :incoming_version;

A one-second timestamp is not a version. Ties and clock differences let old data through.

Catch the webhooks that never arrive

Duplicates and late events are noisy once you look for them. Missing events are silent: an endpoint down past the retry window, a deploy that broke signature checks for an afternoon, a subscription that got removed. Shopify's own docs recommend a reconciliation job for exactly this.

def sweep(object_type):
    checkpoint = load_checkpoint(object_type)
    started = now()
    repaired = 0
    for obj in provider.list_changed_since(checkpoint - OVERLAP):
        with lock_object(obj.id):
            if differs(obj, load_state(obj.id)):
                save_state(obj)                     # same path as the worker
                repaired += 1
    save_checkpoint(object_type, started)
    metrics.gauge(f"webhook_sweep_repairs.{object_type}", repaired)

The OVERLAP margin re-reads a little of the previous window so nothing slips through a boundary. The repair count is the metric to alert on: near zero means webhooks are doing their job; a jump means a doorbell went quiet.

The Shuffle Test: prove it converges

Record a realistic batch of events from a staging account or test mode, then replay it three ways and assert your state matches the provider.

@pytest.mark.parametrize("mode", ["shuffled", "doubled", "holed"])
def test_webhooks_converge(mode, recorded_events, provider_state):
    events = mutate(recorded_events, mode)   # random order / every event twice / drop 10%
    for e in events:
        ingest(e)
    drain_worker()
    if mode == "holed":
        sweep("subscription")
    assert db_state() == provider_state
    assert all(n == 1 for n in emails_sent_per_transition().values())

• Shuffled changes the result: a package handler is hiding somewhere.
• Doubled changes a total: an occurrence isn't keyed by its own ID.
• Holed survives the sweep: the sweep doesn't cover that object type.

Key Takeaways

• Providers promise at-least-once delivery, not order. Stripe states order isn't guaranteed; GitHub doesn't retry failed deliveries automatically.
• Deduping event IDs stops the same event twice. It does nothing about an old event arriving after a new one.
• For state events, use the webhook as a signal and fetch the object's current state from the provider.
• For occurrence events (payments, refunds, disputes), insert once keyed by the object's own ID.
• Acknowledge in milliseconds: verify, write to an inbox table, return 2xx. Process in a worker with a per-object lock.
• Run a reconciliation sweep and alert on how many rows it repairs.

FAQ

Do webhooks arrive in order?

Not reliably. Stripe's documentation states it doesn't guarantee events are delivered in the order they were generated. Even with in-order sending, retries put older events behind newer ones, and multiple workers on your side reorder processing. Design handlers so order doesn't matter.

Is deduplicating webhook event IDs enough?

No. Event-ID dedupe catches the same event delivered twice. Out-of-order bugs involve two different events with different IDs, so both pass the check. Stripe also notes that sometimes two separate event objects are generated for one change, which you identify by object ID plus event type.

Can I sort webhooks by timestamp to fix ordering?

Not safely. Stripe's created field has one-second resolution, so distinct events can share a timestamp, and the docs say not to use it for ordering. You also can't know how long to wait for a late event. Use fetch-on-notify, or a per-object version that only increases.

Should I use the webhook payload or fetch the object from the API?

For state (status, fulfillment, membership), fetch the current object and save that. The payload is a snapshot from when the event was created. For occurrences like payments and refunds, the payload is fine, but insert each one once, keyed by its own ID.

How fast should a webhook endpoint respond?

As fast as possible, before any business logic. Stripe says to return a 2xx before complex logic that could time out, GitHub allows 10 seconds, and Shopify fails a request after 5 seconds. Verify the signature, insert into an inbox table, return 2xx, and process asynchronously.

How do I detect webhooks that never arrived?

Run a scheduled reconciliation job: list objects that changed since your last checkpoint from the provider's API, compare with your database, and repair differences through the same code path your worker uses. Track the number of repairs per run as a health metric.

Related reading

• Why Your API Charges People Twice (And the One Idea That Fixes It): idempotency from the sending side.
• Your Database Transaction Didn't Save You From This Bug: the lost update behind the per-object lock.
• Your Worker Read the Past. Then It Saved It.: why the fetch must hit the source, not a replica.
• There's No Safe Order for Two Writes: the other half of integrating with a system you don't own.
• Metastable Failures: Why Your System Stays Down After the Cause Is Fixed: what a retry backlog does to a recovering endpoint.

New here? Start Here: a plain-English map of everything Simply Explained covers. Get one of these in your inbox each week: subscribe to the newsletter.

The bottom line

A webhook is a true statement about the past, delivered at an unknown time, maybe twice, maybe never. Build the handler that assumes exactly that, and the network can deliver events in any order it likes.

You don't need the events in order. You need a handler that stops caring.

Adam Jaber is a software engineer who writes Simply Explained: complex topics, made simple. No jargon, no hype.