Platform Event Webhooks: Limiting Queue Consumer Credential Blast Radius

The page says platform_event_delivery_stalled: the oldest unacknowledged event is still waiting, and one consumer's authorization failures are climbing. The leaked-key drill has just become an operations problem. The on-call needs to know which credential was exposed, which deliveries used it, and whether revoking it will stop one consumer or the whole fan-out path.

The answer is to accept each platform event once at a single webhook, durably enqueue an immutable envelope, and give every internal consumer an independent delivery record, credential, retry schedule, and dead-letter destination. A leaked key can then be revoked at one boundary while unrelated consumers continue. Don't share the ingress secret with consumers, and don't use one queue message as a race that only one consumer can win.

This is an isolation decision, not a transport-fashion decision. I've been paged by missed jobs and duplicate deliveries; both get much harder to explain when an acknowledgment means different things to different parts of the system. Define those meanings before choosing a broker or writing the handler.

How should one platform event reach several internal consumers through a single webhook and queue?

Treat receipt and delivery as separate state machines. The webhook authenticates the sender, applies a size limit, assigns or preserves an event identifier, stores the envelope durably, and acknowledges only after that durable handoff. A fan-out dispatcher then creates one delivery record per subscribed consumer. Each worker leases its own record and calls exactly one internal destination.

That distinction matters. If the webhook loops over five consumers inline, its response time inherits the slowest consumer and a retry from the sender can repeat work already completed by the other four. If five workers compete for one undifferentiated queue item, one of them gets the event and the others do not. Fan-out requires five addressable delivery obligations, not five hopeful readers.

A compact data model is enough:

The uniqueness constraint on event_id + consumer_id is the idempotency reflex. Enforce it in storage, not merely in a process-local cache. The consumer also needs an idempotency key because a worker can lose its lease after the destination commits but before the acknowledgment is recorded. Exactly-once language tends to hide this gap. Design for at-least-once attempts and idempotent effects instead.

The envelope should contain the event ID, type, occurrence time supplied by the producer, receipt time, schema version, and payload. Keep delivery metadata outside it. Retries must not rewrite the received event, and credential identifiers belong on the delivery side so a rotation can be audited without changing history.

Here is the shape of a generic Go handler. The interfaces are intentionally local: broker choice is less important than the acceptance boundary.

package ingress

import (
    "context"
    "errors"
    "io"
    "net/http"
)

type Envelope struct {
    EventID      string
    EventType    string
    SchemaVersion string
    Payload      []byte
}

type Verifier interface {
    Verify(header string, body []byte) error
}

type EventStore interface {
    PutIfAbsent(ctx context.Context, event Envelope) (created bool, err error)
}

type Publisher interface {
    PublishAccepted(ctx context.Context, eventID string) error
}

type Handler struct {
    Verify Verifier
    Store  EventStore
    Bus    Publisher
}

func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
    body, err := io.ReadAll(io.LimitReader(r.Body, 1<<20))
    if err != nil {
        http.Error(w, "invalid body", http.StatusBadRequest)
        return
    }
    if err := h.Verify.Verify(r.Header.Get("Webhook-Signature"), body); err != nil {
        http.Error(w, "invalid signature", http.StatusUnauthorized)
        return
    }

    event := Envelope{
        EventID: r.Header.Get("Event-ID"),
        EventType: r.Header.Get("Event-Type"),
        SchemaVersion: r.Header.Get("Event-Schema"),
        Payload: body,
    }
    if event.EventID == "" {
        http.Error(w, "missing event id", http.StatusBadRequest)
        return
    }

    created, err := h.Store.PutIfAbsent(r.Context(), event)
    if err != nil {
        http.Error(w, "acceptance unavailable", http.StatusServiceUnavailable)
        return
    }
    if created {
        if err := h.Bus.PublishAccepted(r.Context(), event.EventID); err != nil {
            http.Error(w, "dispatch unavailable", http.StatusServiceUnavailable)
            return
        }
    }
    w.WriteHeader(http.StatusAccepted)
}

var ErrLeaseLost = errors.New("delivery lease lost")

In a real implementation, storing the envelope and publishing the accepted notification need one atomic boundary, commonly a transactional outbox. The sketch leaves that storage detail behind interfaces rather than pretending two separate calls are atomic. This is the catch with a short example: copied literally, it doesn't settle the dual-write failure. A database-backed outbox, or a broker transaction with equivalent guarantees, must close it.

Work backward from the page

Start at the action the on-call can take. The page should identify a consumer, a credential version, the oldest pending delivery age, and the failure class. It should link to a runbook that can disable that subscription, revoke the credential, issue a replacement, replay a bounded delivery range, and verify recovery. Paging on a total webhook request count gives none of that context.

Now move one step earlier. Before the page, a warning should show that one consumer's authorization-failure ratio has departed from its own baseline while ingress acceptance remains healthy. The useful split is between sender authentication at the public webhook and destination authentication at each internal consumer. Combining those counters can make a compromised consumer key look like an ingress outage, which sends the responder to the wrong owner.

Instrument each state transition with low-cardinality outcome labels: accepted, delivery leased, acknowledged, retry scheduled, dead-lettered, and subscription disabled. Put event and delivery identifiers in structured logs or traces, not metric labels. Measure queue age per consumer, pending count per consumer, attempts by outcome class, duplicate suppressions, dead-letter growth, and credential version usage. The last measure is what turns rotation from a hopeful configuration change into a checkable operation.

One signal is especially valuable: deliveries authenticated with the retiring credential after the rotation deadline. It should fall to zero before that credential is destroyed. I'm not sure a universal time window exists here; traffic shape, retry policy, and consumer criticality change the right answer. Resolve that uncertainty with a full rotation rehearsal and an observed maximum delivery age, then write the measured bound into the runbook.

Keep the page narrow. A retryable timeout may justify backlog monitoring, while an authentication rejection during a leaked-key drill should stop attempts for that consumer and trigger credential action. Retrying a rejected secret at high frequency adds noise and can obscure whether the replacement is actually in use.

Run the leaked-key drill without widening impact

Use a synthetic consumer and an event that cannot mutate production data for the first rehearsal. Record the starting credential version, enqueue a known event, confirm one successful delivery, then declare that consumer key leaked. Disable only its subscription or revoke only its outbound credential. Other consumer delivery ages should remain within their normal operating range.

Then rotate. Create a distinct replacement credential, update the one consumer reference, re-enable delivery, and replay the pending synthetic delivery by its stable key. Confirm the destination deduplicates a repeated attempt, the retiring version records no later use, and the queue drains for that consumer. Finally, remove the old credential according to the organization's retention and recovery policy. OWASP's secrets-management guidance treats creation, rotation, revocation, and expiration as parts of a secret lifecycle; the drill needs evidence for each transition, not a checkbox saying that rotation exists.

The sharp edge is shared identity. If three consumers use one outbound token, revocation interrupts all three and audit logs cannot attribute use cleanly. If the same secret authenticates both webhook ingress and internal delivery, a response to an internal leak can also stop new platform events. That blast radius is an architecture property. No queue setting repairs it.

Use a decision record before rollout:

• One ingress verification key is scoped to one sender integration and can overlap with its replacement during rotation.
• One outbound credential reference is scoped to one consumer subscription.
• A delivery is uniquely addressable by event and consumer IDs.
• Disabling a subscription does not delete its pending deliveries.
• Replay is bounded by identifiers or time and preserves the original idempotency key.
• Logs can answer which credential version authorized a delivery without recording the secret.

The catch is that a single webhook plus queue is not suitable when the event must complete synchronously before the sender can proceed, or when every consumer must commit in one cross-service transaction. Stick with a synchronous orchestrated call for the former. For the latter, reconsider the service boundary rather than claiming an asynchronous fan-out provides atomic commit. Stick with direct handling for a truly single-consumer, low-rate integration when the sender already supplies durable retry and the receiver's recovery objective permits it. Simpler can be right.

Set thresholds that buy action, not anxiety

The first threshold should express user or system harm: oldest pending delivery age exceeding the consumer's recovery objective, sustained dead-letter growth, or absence of expected progress for a known event stream. Attempt count alone is weak because retries may be normal and fast. A fixed global queue-depth limit is weak too; ten pending events can be severe for a stream that emits hourly and irrelevant for one that emits thousands per minute.

Test the alert in the leaked-key drill. The responder should receive it after the authorization signal is credible but before the delivery-age objective is breached. Verify that its labels select exactly one runbook branch and one credential owner. Also test a quiet consumer, because rate-based alerts behave badly when the denominator approaches zero. Your mileage may vary, so thresholds should come from the consumer's event rate and recovery objective rather than a number copied from another service.

False positives have a direct reliability cost — they train the on-call to distrust the next page and can provoke unnecessary credential rotations, which themselves interrupt delivery if overlap is misconfigured. Tune warnings for early, ambiguous movement and reserve pages for conditions with an immediate action. After every drill, record detection time, isolation time, last use of the retiring credential, replay completion, and any duplicate suppression. If the alert fired but no safe action was available, the instrumentation change is unfinished.

Short pages win.

The final design test is blunt: can the responder revoke one leaked consumer credential, prove which events were retried, and leave every unrelated consumer moving? If any answer depends on reading source code during the incident, add that state to the delivery record or telemetry before production.

References

• OWASP, "Secrets Management Cheat Sheet": https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html

Further reading

• https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html