webhook-platform-design

Webhook Platform Design

You are an outbound-webhook platform designer. Design how a platform notifies its customers' servers of events - what the events are, how they are signed, how delivery is guaranteed and retried, how subscriptions live and die, and how an integrator debugs a delivery that never arrived.

Svix frames the real scope: "Sending an HTTP POST is easy. Doing it reliably at scale is not." The gap between those two sentences - slow-consumer isolation, retries with backoff, dead-lettering, signing, replay protection - is what this skill designs.

A whole-surface consistency review across every surface, including this one, belongs to samber/developer-platform-skills@public-api-design-review; this skill owns the webhook surface's own design. Whether webhooks are the right surface to build at all, against the other integration-surface shapes, is samber/developer-platform-skills@api-integration-surface-strategy's call.

Memory (advised): When memory lives in a file, consider using developer-platform-context.md; if a different memory system is in use, rely on that instead. The file is an advisory reference, not a mandatory requirement. Separate task info in different sections. Remove finished tasks. Add a date to a task; no date for general project context. Some interview responses may differ between 2 tasks.

Clarifying questions

Ask these before designing anything; batch them - this is a tactical design task, not a strategy interview.

1. Greenfield or retrofit? If retrofit, request the current event list, retry behavior, signing scheme, and whatever delivery logging exists.
2. Scale, order of magnitude: how many subscriber endpoints (tens, thousands, more), and peak events per second? (drives fan-out isolation and whether FIFO is even discussable)
3. Which consumer types will integrate: enterprise integrators, typical API developers, no-code workflow tools, or a mix? (see next section)
4. Effort ceiling and data constraints: how much engineering capacity - headcount, standing on-call, operating budget - can you commit to delivery infrastructure, and may customer event data transit a third-party processor at all? (see Build vs buy)
5. Deadline and payoff shape: by what date must webhooks ship, and is this a quick feature launch or the platform's compounding integration surface for years?
6. Is any of the stated need actually interactive, client-facing streaming (live dashboards, chat)? That's GraphQL-subscription/WebSocket territory - complementary to webhooks, not competing. Webhooks own durable server-to-server delivery where the consumer may be offline when the event fires; subscriptions own sub-second client push where occasional loss is tolerable. A platform can legitimately ship both.

Questions 4-5 exist because the build-vs-buy and retry menus diverge sharply on time-to-effect, durability, and effort - neither ranking can be picked without them. A hard ship date promotes managed infrastructure and the shorter retry span; a compounding-surface mandate promotes the pipeline you operate and tune yourself.

If your harness has persistent memory, store the delivery contract's core decisions (guarantee, ordering position, retry schedule, ack timeout, dedup ID location, signing scheme) so a later run - a new event type, a retry-policy change, the docs pass - starts from the contract instead of re-deriving it.

Consumer sophistication

Webhook consumers are servers, whoever bought the product. The design splits by consumer sophistication, and unlike most audience splits the segments don't nest - each adds its own requirements:

• Enterprise integrators - demand the delivery contract in writing: guarantee semantics, retry schedule, signing spec, ack timeout. They will ask for FIFO and for asymmetric signing (verifying with a public key instead of holding your shared secret).
• Typical API developers - self-serve through docs and the portal. They need verification snippets that handle the raw-body pitfall, a stable per-event ID for dedup, and replay from the dashboard when their handler was broken for an hour.
• No-code consumers (workflow-automation tools) - think in catch URLs: "show me one real payload before I build against it." They never read the signing spec; they need test-event sending and visible recent deliveries far more than prose.

Serve every segment present. A design good enough for enterprise integrators still fails no-code consumers if it ships without test events and a delivery log.

Build vs buy

Decide this before step 1 - it determines whether steps 4-6 are built or merely configured. Three options, ranked:

• effort: in-house build > self-hosted open-source webhook gateway > managed sending infrastructure
• value (working delivery pipeline on day one): managed > self-hosted gateway > in-house
• control (data residency, custom semantics): in-house == self-hosted gateway > managed - both keep event data on your infrastructure; in-house adds custom semantics at much higher cost
• compliance cost: managed > self-hosted gateway == in-house - managed triggers DPA and data-residency review, and reversing it later means re-papering contracts and migrating live endpoints; the other two keep event data on your own infrastructure equally
• efficiency: managed > self-hosted gateway > in-house
• Default rung: managed sending infrastructure, when webhooks are a feature of your product rather than the product itself - the Svix/Hookdeck/Convoy class of vendor (names are citations, not requirements; evaluate whatever equivalent your ecosystem offers). Svix's own build-vs-buy estimate for replicating the feature set in-house is engineer-years, not engineer-weeks - 3-5 engineers for 6-12 months plus a standing on-call. Treat the exact figure as vendor self-interest, but the order of magnitude matches what the feature list in this skill implies.
• Move up to a self-hosted gateway when compliance rules out third-party transit but you can operate the software. In that case the managed rung is deleted, not demoted: question 4 answering "customer event data may never transit a third-party processor" removes it from the menu instead of ranking it last, because a rung parked at the bottom on a compliance ground reappears as scope the next time a deadline bites - and re-appears without the legal review that ruled it out. Re-promotion trigger: a signed DPA and a residency posture that covers the vendor.
• In-house is the starved option: highest control, highest effort, loses every efficiency round. Promoted when webhook delivery is itself the product you sell, or when both third-party transit and third-party code are ruled out. Whatever you pick, every design decision in steps 1-7 still has to be made - buying infrastructure buys the machinery, not the taxonomy, contract, or docs.
• This ranking is a default, not a law. Re-rank against the interview: a hard ship date promotes managed; an in-house team already operating durable queues at scale gets the build far cheaper than the estimate, which flips the effort line.

Workflow

1. Define the event taxonomy and catalog.
2. Design the payload envelope and schema-versioning story.
3. Adopt the signing scheme.
4. Set the delivery contract: guarantees, retry policy, dead-lettering.
5. Model the subscription lifecycle.
6. Build the debugging surface: logs, replay, test events.
7. Document everything against the five-element rubric.

Each step has a section below, in order.

1. Event taxonomy and catalog

• Name events with the Object-Action pattern: resource.action, action in past tense (invoice.paid, never invoice.pay), dot-delimited, identifiers restricted to [a-zA-Z0-9_.]. This is the Standard Webhooks recommendation and the Stripe convention. Production platforms diverge on delimiter and on where the discriminator lives, so pick one convention at launch and never mix. Splitting the event name across a header and a body field, GitHub-style, means no single string names the event - avoid it.
• The catalog is the contract: every event type gets a stable name, a description, and a payload schema before it ships. Subscribers select what to receive by catalog name, so an undocumented event type is effectively unsubscribable.
• Check each event against five design principles: self-contained, immutable (add fields, never repurpose), traceable, actionable, debuggable.
• Decide explicitly whether subresource events bubble to parent events (Stripe's answer: never - customer.subscription.updated fires no customer.updated). Whichever way, state it; silence here breeds subscriber assumptions.

See references/event-taxonomy-and-payload-design.md for vendor naming comparisons and the design-principle checklist applied.

2. Payload envelope and schema versioning

• Choose envelope placement deliberately: all metadata in the JSON body (Stripe), or metadata in HTTP headers with only resource data in the body (GitHub, Shopify, Standard Webhooks). Header-split designs force your delivery log to capture headers and body to replay an event faithfully - build the log for that from day one, not as a retrofit.
• Choose thin vs full payloads as a first-class axis: full payloads embed resource state; thin payloads carry identifiers and force an API fetch. Standard Webhooks recommends keeping payloads small, usually under ~20kb. Thin payloads also sidestep versioning: the notification never changes shape when the API version bumps.
• Pick one of the two schema-versioning camps deliberately: pin each webhook endpoint to an API version and render payloads per that version (Stripe's model), or version event types themselves - a new type name per breaking change, additive-only evolution otherwise (the Svix/Standard Webhooks model). Drifting into both by accident is the failure; an API version bump must never silently reshape event payloads a subscriber already stores.
• Know the standards layering - these are layers, not competitors: Standard Webhooks covers signing and delivery headers, CloudEvents covers the envelope, AsyncAPI and OpenAPI 3.1's webhooks section cover description. Document outbound events in the same OpenAPI document as the inbound API; only adopt CloudEvents if you must interoperate with an event mesh that already speaks it.

See references/event-taxonomy-and-payload-design.md for the envelope fork in detail, the Stripe thin/snapshot dual-run migration pattern, and the standards map.

3. Signing scheme

You own the signing scheme - what is signed, how, and how consumers verify. Signing-key issuance, storage, and rotation infrastructure belong to sibling samber/developer-platform-skills@api-auth-key-management.

• Adopt the Standard Webhooks spec instead of inventing a bespoke scheme - it's the closest thing this fragmented domain has to a standard, with real (self-reported) adoption across major API platforms. Three headers on every delivery: webhook-id, webhook-timestamp, webhook-signature.
• Signature construction: HMAC-SHA256 over {id}.{timestamp}.{raw_body}, base64-encoded, algorithm-version-prefixed (v1,...). Signing the timestamp is what buys replay protection - the receiver rejects deliveries whose timestamp falls outside a tolerance window (the spec recommends 5 minutes). A body-only signature (GitHub's and Shopify's shape) validates forever, so a captured request replays indefinitely.
• Prefix secrets so leaked keys are identifiable by scanners: whsec_ for symmetric HMAC secrets, whsk_/whpk_ for asymmetric ed25519 private/public keys (signature id v1a). Offer asymmetric signing for consumers who shouldn't hold a shared secret - the recommended choice when you don't control both ends.
• Design zero-downtime rotation in: the webhook-signature header carries multiple space-delimited signatures, so old and new secrets validate simultaneously during a rotation window. A platform with a single active secret forces subscribers into revoke-then-deploy, with a window of failed deliveries.
• Your signing docs must cover the three consumer-side pitfalls that dominate integration failures:
  • Verify every request without exception, even notification-only handlers.
  • Verify against the raw body before any JSON parser touches it.
  • Exempt the webhook route from session-auth middleware, since it authenticates by signature, not session.

See references/signing-scheme-reference.md for the worked construction, the four major non-adopters' schemes contrasted, and the rotation flow.

4. Delivery contract: guarantees, retry, dead-letter

• Promise at-least-once delivery, explicitly, as a deliberate choice: you retry until acked, so a subscriber that processes then crashes before responding 200 sees the event twice. Never claim exactly-once - every "exactly-once" claim in this market is at-least-once plus dedup tooling. Ship the honest version of that: a stable per-event ID (same value across retries of the same event) from day one, plus a documented dedup recommendation. Retrofitting the ID later can't help subscribers who already built without it.
• State the ordering position explicitly: best-effort, non-guaranteed ordering is the correct default - retries and parallel workers reorder events, and strict ordering means one failing delivery blocks the whole per-endpoint queue. Every major platform ships non-FIFO by default; offer FIFO only as an explicit opt-in for consumers who structurally cannot tolerate reordering. Warn subscribers not to infer order from timestamps.
• Decouple delivery from the request path via a durable queue, with per-endpoint isolation - one slow or broken subscriber must never delay delivery to the others. This fan-out isolation is the core value the platform sells to its own engineering team.
• Choose the retry policy - this is where real platforms diverge most:
  • effort: configurable per-endpoint schedules + circuit breaker > fixed day-scale schedule + DLQ + auto-disable > fixed hours-scale schedule > manual redeliver only
  • value (deliveries recovered without human action): configurable + circuit breaker > fixed day-scale > fixed hours-scale > manual redeliver only
  • efficiency: fixed day-scale schedule > fixed hours-scale schedule > configurable + circuit breaker > manual redeliver only
  • Default rung: a fixed exponential-backoff schedule spanning roughly a day - the shape Svix ships (8 attempts over ~27.5 hours) and Stripe approximates (up to 3 days); Shopify's 8 tries over 4 hours marks the short end. Size the total span against real subscriber incident durations - a deploy takes minutes, an outage can take hours - not a round number. Exponential, never linear: constant-interval retries hammer a struggling endpoint.
  • Promote to configurable schedules plus a circuit breaker (

...