Webhooks — Shared Inbox and Delivery Archive

Platform-agnostic plumbing every inbound-webhook module can share: an idempotency inbox that makes replays free, and an archive of every delivery to a /webhooks/* route — including the ones we reject.

Source: api-modules/webhooks (registered via WebhooksModule.forRoot() in apps/api/src/app.module.ts).

Nothing here is payment-specific. Shipping and marketing webhooks use it on the same terms.

Idempotency inbox

Backed by the unique (provider, external_event_id) index on processed_webhook_event, which is the actual enforcement point — concurrent deliveries of one event collide on insert.

const claim = await inbox.claim({ provider, externalEventId, eventType });
if (!claim.isNew) return ack(claim.previousOutcome);   // replay

try {
  await inbox.markProcessed(claim.id, await dispatch());
} catch (err) {
  await inbox.markFailed(claim.id, err);
  throw err;
}

claim() returns { isNew, id, previousOutcome }, so a replay can be acknowledged with whatever the first delivery actually reached.

A delivery recorded as failed is re-admitted on the provider's next retry: claim() flips the row back to claimed, increments attempts, and returns isNew: true. A settlement blocked by a transient outage therefore heals itself when the provider retries, instead of being acked forever. The failed guard is applied inside the UPDATE, so concurrent retries can't both win — the loser still sees a duplicate.

Row lifecycle

attempts counts how many times the event has been admitted, so a row repeatedly failing is visible rather than looking like a single bad delivery.

Recording the outcome — rather than only the fact of receipt — is what makes a failed delivery recoverable instead of silently skipped forever.

Providers without an event id

Some gateways ship no stable event id. Compose one from the entity plus its terminal state, so retries of a delivery collide while genuinely distinct events do not. PhonePe, for example, uses `${orderId}-${state}` for order events and `${refundId}-${state}` for refunds.

Delivery archive

A single path-scoped interceptor captures every request under /webhooks/, so new webhook endpoints are covered with no work and existing ones needed no changes.

Rejected deliveries are archived deliberately: a forged-signature attempt is precisely what an operator needs to see, and previously it left no trace at all.

What is never stored

Headers use an allowlist, not a denylist — a denylist silently leaks the first credential-bearing header a future provider invents. Authorization is excluded outright, because for some providers (PhonePe) that header is a reusable static credential rather than a per-request MAC. Only its presence is recorded.

Bodies pass through the same redactBounded() used by the audit store, so keys like client_secret, signature, card and vpa are replaced with [REDACTED].

Storage and retention

ClickHouse table webhook_deliveries — MergeTree, partitioned by month, TTL received_at + INTERVAL 90 DAY. Deliberately a shorter window than audit_events (24 months), because these rows carry raw provider payloads.

The write rides the existing audit outbox with eventType: "webhook.received"; the audit worker routes that event type to this table instead of audit_events. That reuses the durable outbox, batching and 2-second drain rather than duplicating them.

Best-effort by design

Archiving is fire-and-forget inside a try/catch, and a failure is logged rather than raised — a ClickHouse outage must never fail a webhook, and providers time out in seconds. If AuditModule is not mounted there is no sink, and the interceptor logs a warning once at startup saying archiving is disabled.

Adding a provider

1. Mount the route under /webhooks/… — archiving then needs no further work.
2. Verify the provider's signature over req.rawBody (captured globally in main.ts).
3. Claim through the inbox, dispatch, record the outcome.
4. Ack 2xx once authenticated, even for events you ignore, or the provider will retry forever.

Related

• payment-phonepe.md and webhooks/payment-phonepe.md — the reference consumer.
• marketing-klaviyo and shipping-clickpost keep their own domain-specific idempotency for now, but are archived by the shared interceptor.