Webhooks Module — Admin

Archive of every inbound provider webhook delivery — the request we received and the exact response we sent back, including deliveries we rejected and ones rate-limited before routing. Stored in ClickHouse with a 90-day TTL, joined to the idempotency ledger, and served as a filterable, paginated admin read API with an operator resend that recomputes the provider credential rather than storing it.

Source: api-modules/webhooks/src/query/admin-webhook.controller.ts.

Optional plugin. Removing WebhooksModule.forRoot() from app.module.ts disables it entirely — the capture sink is never wired (so the Fastify hook degrades to a pass-through), the shared idempotency inbox stops being provided, the ClickHouse migrations don't register, and /admin/webhooks leaves the OpenAPI surface. Provider webhooks keep working.

Why it exists

When a provider says "we delivered the event and it failed", the operator needs to see both halves of the exchange to decide whether the fault is ours. Recording the status code alone is not enough — the body carries the reason (Invalid signature, a validation error, an unhandled 500).

Architecture

Fastify lifecycle                                    Worker role
─────────────────                                    ───────────
onRequest  ── stamps startedAt
preParsing ── tees rawBody (see raw-body.ts)
   ↓
Nest guards → handler → interceptors / exception filter
   ↓
onSend     ── builds the capture ──► audit_outbox ──► AuditProcessor ──► webhook_deliveries

Capture runs at onSend, not in an interceptor. That is the last lifecycle step before the socket write, and the only point that observes what we actually replied:

Being path-scoped to /webhooks/* rather than per-controller means a new provider route is archived with no extra work.

Best-effort, always. Every step is wrapped so a capture failure can never fail a webhook — providers ack-timeout in seconds. Writes ride the existing audit outbox and drain, so they never block the response.

Redaction — request headers are copied by allowlist, never denylist (WEBHOOK_ARCHIVED_HEADERS). authorization is deliberately excluded: PhonePe's webhook auth is that header, so its value is a reusable credential. Signature headers are recorded as a boolean presence flag only. Response headers use their own allowlist (content-type, location, retry-after).

Bounds — the request body is capped at 64 KiB (WEBHOOK_BODY_MAX_BYTES) and our response at 8 KiB (WEBHOOK_RESPONSE_MAX_BYTES); the true size and a truncated flag are stored alongside. The request cap is sized for resend rather than for reading: a truncated body cannot be resent faithfully, so it bounds what stays replayable.

Retention — ClickHouse native TTL of 90 days on received_at, far shorter than audit's 24 months precisely because these rows carry raw provider payloads.

Event ids come from Postgres

webhook_deliveries.external_event_id and event_type are written empty — the capture hook cannot know them, since the provider's event id is parsed inside the controller. The read layer instead joins processed_webhook_event on request_id, which yields the provider event id, the event type, and the dedup outcome (status, attempts, error). WebhookArchiveInterceptor exists solely to stamp the audit requestId onto the request so both sides agree on one value.

Only providers that claim through WebhookInboxService (Razorpay, PhonePe, CCAvenue notify) produce a row to join against; for the others — and for requests rejected before any handler ran — inbox is null.

Endpoints

All require a Better-Auth admin session. The three reads need a role granting webhook: view; the resend needs webhook: replay, so a read-only operator can inspect the log without being able to re-fire side effects.

All four return 503 with an explanatory message when CLICKHOUSE_ENABLED=false, rather than surfacing a driver error as an opaque 500.

GET /admin/webhooks

Paginated, filterable list (newest first). Standard offset/limit pagination (limit, offset) plus filters:

Response: { data: WebhookDelivery[], metadata: { total, limit, offset, hasMore } } (the canonical ApiWrappedPaginatedResponse shape).

GET /admin/webhooks/providers

Distinct provider names present in the archive, for populating a filter. Declared before :id so the param route does not swallow it.

GET /admin/webhooks/:id

Fetch a single delivery by id, with its idempotency-inbox outcome. 404 if not found.

POST /admin/webhooks/:id/replay

Requires webhook: replay (not view). Resends the archived delivery — see Resending a delivery. Returns { sourceId, statusCode, responseBody, clearedInboxClaim }. 400 when the provider has no signer or the body was archived truncated; 404 when the delivery is unknown.

WebhookDelivery shape

id, provider, receivedAt (ISO), method, path, route, statusCode, requestId, signaturePresent, ip, durationMs, requestBody, requestHeaders, requestBytes, requestTruncated, responseBody, responseHeaders, responseBytes, responseTruncated, replayedFrom (set when this row is itself a resend), and inbox — either null or { externalEventId, eventType, status, attempts, error, processedAt } where status is claimed | processed | skipped | failed.

Configuration

Resending a delivery

An operator can resend an archived delivery so the handler runs again — the usual case being an event that failed on our side and needs re-processing once the bug is fixed.

The credential is recomputed, never stored. The archive deliberately holds no authorization or signature value, so a resend cannot replay the original one. Instead each provider plugin registers a WebhookReplaySigner that mints a genuine credential over the same body from the same admin settings the verifier reads:

The resent request therefore passes the real verifier; no signature-bypass path exists for an operator to abuse. A provider with no signer is refused with a 400 rather than resent unverified.

Signers self-register. Each plugin adds its signer to its own providers: [] (where its config service is visible) and the signer registers itself with the global WebhookReplayRegistry on init. Removing a plugin's forRoot() line removes its resend support with no other edit.

Dispatch goes through fastify.inject() — the full lifecycle in-process, including the content-type parsers, the rawBody tee, signature verification, the handler, and this module's own archive hook. No socket, no base URL, no TLS to configure. The resend is consequently archived as its own delivery carrying replayedFrom, so it reads as a resend rather than an unexplained duplicate.

The idempotency claim is cleared first. WebhookInboxService.claim() is insert-or-nothing, so leaving the processed_webhook_event row in place would make the resend ack and skip — a no-op for exactly the events worth retrying. The row is deleted before dispatch, and clearedInboxClaim in the response reports whether one existed. Side effects run again, which is the point; the confirm dialog says so.

Refused when the body was truncated, since a partial body cannot be resent faithfully. The claim is left untouched in that case.

POST /admin/webhooks/:id/replay is a mutating admin route and is not in AUDIT_HTTP_PATH_IGNORE, so every resend lands in the audit log attributed to the staff member who triggered it.

Admin UI

System → Webhooks (/system/webhooks), served by @sc/admin-webhooks. The list filters by provider, outcome and method with a path search; the detail view (/system/webhooks/:id, deep-linkable) shows three panels — Request, Our response, and Processing outcome. The nav entry carries permission: { resource: "webhook", action: "view" }, which gates the whole route subtree.

Resend sits on the detail view behind <Can resource="webhook" action="replay">, so read-only operators never see it. It is disabled with an explanation when the body was truncated, and confirms before dispatch that side effects will run again. Rows that are themselves resends are marked in the list and link back to their original.

Idempotency inbox

Separate from the archive, and available to any module receiving webhooks: WebhookInboxService.claim() inserts-or-nothing on (provider, external_event_id) in processed_webhook_event, so a replayed delivery is a no-op with exactly-once side effects even under the verify-vs-webhook race. Record the result with markProcessed / markSkipped / markFailed; a row left in claimed means a crash mid-dispatch and is visible to ops rather than silently skipped forever.