Lab · Webhook delivery
Break the system. See how it recovers.
A deterministic model of a resilient webhook pipeline. Send events, fail a subscriber, throttle it, lose its responses, and watch retries, a circuit breaker, a rate limiter and a dead-letter queue keep the rest of the system moving. Time is accelerated and labelled.
Interactive educational simulation. Synthetic data. Not connected to employer systems.
The simulation
Simulated time, ×4
t = 0.0 s
Three healthy subscribers acknowledge every delivery the first time. A burst shows the local rate limiter at work: each subscriber has a bucket of three tokens that refills one per second, so after the first three deliveries the pipeline paces itself to one attempt per subscriber per second rather than flooding an endpoint. Nothing retries, nothing dead-letters, and every event applies exactly one business effect per subscriber.
- Keys
- S
- Send event
- B
- Send burst
- F
- Fail A
- R
- Restore A
- Space
- Pause or resume
- .
- Step
- 0
- Reset
- Shortcuts are ignored while typing in a field.
- Events
- 1
- Deliveries
- 3
- Delivered
- 0
- Dead letters
- 0
- In flight
- 2
- Waiting
- 1
- Effects
- 0
- Duplicates
- 0
Subscriber A
https://subscriber-a.example/webhooks
Circuit
Attempts flow normally.
Rate limit
Refills 1 per second. Acknowledges with 200 and applies each key once.
- Delivered
- 0
- Failed
- 0
- Dead letters
- 0
- Effects
- 0
- Duplicates
- 0
- Waiting
- 1
Subscriber B
https://subscriber-b.example/webhooks
Circuit
Attempts flow normally.
Rate limit
Refills 1 per second. Acknowledges with 200 and applies each key once.
- Delivered
- 0
- Failed
- 0
- Dead letters
- 0
- Effects
- 0
- Duplicates
- 0
- Waiting
- 1
Subscriber C
https://subscriber-c.example/webhooks
Circuit
Attempts flow normally.
Rate limit
Refills 1 per second. Acknowledges with 200 and applies each key once.
- Delivered
- 0
- Failed
- 0
- Dead letters
- 0
- Effects
- 0
- Duplicates
- 0
- Waiting
- 1
Deliveries
3 shown| Delivery | To | State | A · F · DAtt · fail · def | ||
|---|---|---|---|---|---|
| reservation.confirmed | C | Queued | 0 · 0 · 0 | waiting | |
| reservation.confirmed | B | Processing | 1 · 0 · 0 | in flight | |
| reservation.confirmed | A | Processing | 1 · 0 · 0 | in flight |
Inspector
select a deliveryChoose a delivery in the table or the dead-letter queue to read its attempt history in plain English.
Dead-letter queue
emptyNothing here. A delivery arrives when it exhausts its failure budget or its 429 deferral budget. Replaying one creates a fresh delivery that records the original; the original is never changed.
Log
2 of 2 · ring buffer of 300- 0.0 sEvent evt-0001 (reservation.confirmed) accepted. Three deliveries queued.
- 0.0 sPreset loaded: Healthy. 4 scripted commands on the simulated clock.
Text summary of the current statefor screen readers and the curious · opens
Simulated clock at 0.0 s, running at 4 times wall-clock speed. Preset: Healthy. The next scheduled moment is at 0.3 s.
1 events sent, making 3 deliveries: 0 delivered, 0 dead-lettered, 1 queued, 2 in flight and 0 waiting for a scheduled retry. 2 attempts in total, 0 failed and 0 deferred by 429 responses. Consumers applied 0 effects and ignored 0 duplicates. 0 replays. The circuit breaker has opened 0 times.
- Subscriber A (https://subscriber-a.example/webhooks) is healthy; circuit closed with 0 consecutive failures. Rate-limit tokens 2 of 3. Delivered 0, failed attempts 0, dead-lettered 0, effects applied 0, duplicates ignored 0. Waiting: 0 queued, 0 retry-scheduled, 1 in flight.
- Subscriber B (https://subscriber-b.example/webhooks) is healthy; circuit closed with 0 consecutive failures. Rate-limit tokens 2 of 3. Delivered 0, failed attempts 0, dead-lettered 0, effects applied 0, duplicates ignored 0. Waiting: 0 queued, 0 retry-scheduled, 1 in flight.
- Subscriber C (https://subscriber-c.example/webhooks) is healthy; circuit closed with 0 consecutive failures. Rate-limit tokens 3 of 3. Delivered 0, failed attempts 0, dead-lettered 0, effects applied 0, duplicates ignored 0. Waiting: 1 queued, 0 retry-scheduled, 0 in flight.
Most recent log lines, newest first:
- 0.0 s: Event evt-0001 (reservation.confirmed) accepted. Three deliveries queued.
- 0.0 s: Preset loaded: Healthy. 4 scripted commands on the simulated clock.
How it works
Six patterns, one deterministic engine.
The engine is a pure TypeScript state machine: no randomness, no timers, no network. Every transition is a function of the previous state and the simulated clock, which is why the same commands always produce the same story and why the unit tests can drive it moment by moment.
01
Fan-out through a queue
Sending an event creates one delivery per subscriber. Two workers take deliveries from a queue ordered by age and make one simulated request each, 300 ms long. A delivery that is not eligible right now is skipped rather than failed, so a struggling subscriber never starves the healthy ones.
02
Retries with bounded backoff
A 503 or a lost response counts as a failure. The delivery is retried after 500 ms, then 1 s, 2 s, 4 s, capped at 8 s. Four failures may be absorbed; the fifth moves the delivery to the dead-letter queue with the reason attached.
03
A circuit per subscriber
Three consecutive failures open a subscriber's circuit. Its deliveries are parked, not failed, for 6 s. Then exactly one probe is allowed: a response closes the circuit, a failure re-opens it. Repairing the subscriber never closes the circuit by itself.
04
Two kinds of throttling
Each subscriber has a local token bucket of three tokens refilling one per second; waiting on it costs nothing. A subscriber may also answer 429 with Retry-After. That retry is scheduled at exactly the requested time, counts no failure, and draws on a separate deferral budget of six.
05
At-least-once, made safe
When a response is lost the effect may already have been applied. The sender retries anyway. Each consumer remembers the idempotency keys it has applied and acknowledges a repeat with 200 while applying nothing new, so the counters show effects applied and duplicates ignored side by side.
06
Replay from the dead-letter queue
Replaying a dead letter creates a fresh delivery that records which one it replays. The original and its attempt history are never changed, which keeps the audit trail honest.
Illustrative parameters
Chosen for a readable demo, not taken from any production system. The table is generated from the same configuration object the engine runs on.
| Parameter | Value | Note |
|---|---|---|
| Workers | 2 concurrent attempts | Healthy subscribers keep flowing while one fails |
| Attempt latency | 300 ms simulated | Time a delivery spends in processing |
| Failure budget | 4 failures, so 5 attempts at most | Then dead-letter |
| Backoff | 500 ms × 2^(failures - 1), capped at 8 s | Exponential, bounded, no jitter |
| 429 deferrals | 6 per delivery | Retries exactly at Retry-After (default 4 s) |
| Circuit threshold | 3 consecutive failures | Opens the subscriber's circuit |
| Circuit cooldown | 6 s | Then one half-open probe |
| Local rate limit | 3 tokens, refill 1 per second | Token bucket per subscriber |
| Queue bound | 200 deliveries | Outstanding deliveries; further sends are rejected with a log line |
| Burst bound | 25 events | Per burst command; larger requests are clamped |
| Log bound | 300 entries | Ring buffer |
| History bound | 20 attempts per delivery | Oldest attempts are dropped |
Guided scenarios
Healthy
Three healthy subscribers acknowledge every delivery the first time. A burst shows the local rate limiter at work: each subscriber has a bucket of three tokens that refills one per second, so after the first three deliveries the pipeline paces itself to one attempt per subscriber per second rather than flooding an endpoint. Nothing retries, nothing dead-letters, and every event applies exactly one business effect per subscriber.
Failure
Subscriber A answers 503 to every request. Each failed attempt is retried with exponential backoff, and after three consecutive failures A's circuit opens: its remaining deliveries are parked rather than hammered, while B and C keep flowing through the same workers. Every six seconds a single half-open probe tests A again, fails, and re-opens the circuit. A delivery that spends its five attempts lands in the dead-letter queue with a reason attached, ready to be replayed once A is repaired.
Recovery
The same failure, then A is repaired nine seconds in. Restoring the endpoint does not close the circuit by itself: the circuit stays open until its cooldown elapses, then allows exactly one probe. The first probe still fails because A had not yet recovered; the second succeeds and closes the circuit, and the parked deliveries drain immediately, paced only by the local rate limiter. Nothing is lost and nothing is dead-lettered.
Rate limit
Subscriber A answers 429 Too Many Requests with Retry-After: 4 s. The pipeline honours the header exactly: each deferred delivery is retried four seconds later, with no backoff and no failure counted against it or against the circuit. A separate, smaller deferral budget of six stops a permanently rate-limiting subscriber from looping forever. A is restored at ten seconds and the deferred deliveries complete on their next scheduled attempt.
Lost response
Subscriber A processes each request and applies the business effect, but its response is lost, so the pipeline sees a timeout and retries. Delivery is at-least-once: the retry reaches a consumer that has already applied that idempotency key, so it answers 200 without applying the effect again. The counters make the difference visible: effects applied stays at one per event while duplicates ignored climbs. A's responses come back before each retry is due.
Where the code lives
The implementation is part of this site: the engine, its types and the scenario scripts live in src/lib/sim, the signing helpers in src/lib/crypto/hmac.ts, and the unit tests that drive the engine with a controlled clock in tests/unit/sim. The React panels above only render state; they never decide anything.
The real platform
Latest project: ~1M events/day through the event-driven delivery service. The case study explains the patterns this lab illustrates, and what it deliberately leaves out. Interactive educational simulation. Synthetic data. Not connected to employer systems.
Read about the real platformSigned payloads
Change one character. Watch verification fail.
Each webhook carries an HMAC-SHA256 signature over the timestamp and the body. The receiver recomputes it with the shared key, compares in constant time and rejects stale timestamps. The key below is public and protects nothing; it exists so the arithmetic can be shown.
Payload
as signed{ "id": "evt-0001", "type": "reservation.confirmed", "occurredAt": "2026-01-14T09:30:00Z", "data": { "reservationId": "RSV-10001", "loungeCode": "LHR-T5-N", "guests": 2, "status": "confirmed" }}
Signature
- Key
- demo-public-key-not-a-secret (public, demo only)
- Signed
- computing
- Message
- timestamp + "." + body
- Header
- computing
- Tolerance
- 5 minutes
Verifying.