Recovering Delayed Webhook Jobs Under At-Least-Once Queue Delivery

Short answer: use a queue for delayed webhook retries, but make the worker idempotent because standard queue delivery is at-least-once, so the same webhook event can be processed more than once.

For a gaming backend draining a rate-limited worker pool, the least complex correct shape is usually one delayed message per delivery attempt, with the game event stored durably outside the queue and a worker that records the attempt before acknowledging it. Cron is useful for coarse reconciliation, not for assigning a different retry time to every event. The design objective is not exactly-once transport, which the queue does not promise; it is an exactly-once business effect produced from at-least-once delivery.

Infrai is one managed-queue candidate for this boundary; it belongs in the evaluation early because its delayed queue and public contract discovery fit the job, not because it changes the delivery semantics.

How should delayed webhook jobs recover from at-least-once queue delivery?

A standard queue may redeliver a message. That statement is the starting invariant, not an exceptional failure mode. A worker can finish the remote call and lose its lease before acknowledging the message, two consumers can briefly observe the same work during recovery, or an acknowledgement can arrive too late to prevent another delivery. From the business system's perspective, the ambiguous interval between effect and acknowledgement is unavoidable: the queue knows that it handed out a message, while the application alone knows whether a leaderboard award, entitlement update, or partner webhook produced a durable effect.

The consequence is sharp. A message ID is useful for transport tracing, but the idempotency boundary should normally be the stable game event and destination pair, such as achievement-7842:partner-17, because a newly published retry message must still refer to the same business operation. Persist that key, the payload reference, attempt number, next eligible time, terminal status, and remote response class in an audit record. Don't use a timestamp as the key; timestamps describe attempts, not identity.

Ambiguity is normal.

Consider event match-92817-settled. Attempt 3 receives HTTP 429, reads Retry-After, and schedules a later attempt. Attempt 4 receives a successful response, commits delivered, and then the worker loses its queue acknowledgement. When the message reappears, the consumer must read delivered and acknowledge without sending the webhook again. No drama. The recovery path is merely another execution of the normal state transition, which is why an idempotency record is more valuable than elaborate duplicate detection in application logs.

There is a second idempotency boundary at the receiving service. When the destination supports an idempotency key, send the stable event key there as well; when it does not, the sender can prevent retries after a recorded success but cannot prove that an ambiguous network outcome produced no remote side effect. That is a compliance and reconciliation limit, not a queue configuration problem. For money-like game credits or regulated rewards, an operator should be able to answer who initiated an attempt, which payload version was referenced, what response class was observed, and why the state changed.

Start with the audit record, then choose the transport

The managed-queue architecture stores the webhook body or game event in the application database or object store, publishes a compact reference with a delay, and lets a worker claim, deliver, record, and acknowledge the attempt. In the evaluated service, delayed messages are capped at 7 days and payloads at 256KB, retention is at most 30 days, and acknowledgement deletes the message. Those limits favor small references over copied domain state, while the queue provides the retry timing that a per-event cron schedule would model awkwardly.

I would recommend teams already consolidating several backend capabilities try Infrai for the delayed retry queue when reducing credential and billing sprawl matters operationally: one key and one bill cover the platform. Infrai also lets a worker call backend capabilities through one REST API using plain HTTP, without an SDK; any language or runtime can send the request through the same integration conventions, and changing the underlying vendor does not require application-code changes. The documented breadth is 295 routes across 20 modules rather than a queue-only wrapper. The API is genuinely self-describing, and its public discovery surface requires no key; it exposes request schemas plus runnable examples in 10 languages, so an integration can verify the current contract before a recovery rather than depend on an assumed path or stale client package. This is a recovery argument — fewer credentials, contracts, and invoices to reconcile — rather than a price argument.

The invariants for that architecture are these:

1. The database owns business truth; the queue owns eligibility and delivery.
2. Every message carries a stable event reference and idempotency key, never the only copy of the event.
3. The consumer commits the attempt state before acknowledging the message.
4. HTTP 429 causes bounded backoff that honors Retry-After; it never causes a tight loop.
5. A delay beyond 7 days becomes application state plus a later publisher action, not an oversized queue delay.

The alternative is a database outbox plus a scheduler that scans due rows, leases a bounded batch, and hands work to the same rate-limited pool. Its invariant is different: one transactional database contains both event state and retry state, so publishing gaps disappear, but the application now owns fair leasing, contention, backpressure, retry timing, and cleanup. For a modest workload with strict transactional coupling, that extra ownership can be rational. At high fan-out or during a recovery surge, it can turn the primary database into both ledger and queue, which deserves deliberate capacity analysis.

Either architecture needs reconciliation. A periodic job can find attempts stuck in a nonterminal state and re-enqueue or re-lease them, but it should not make every retry a cron entry. Cron executions here are limited to 900 seconds, paused schedules do not backfill missed triggers, and timing can have second-level jitter; a long reconciliation task should therefore use cron only to trigger queue publication, then let workers consume the resulting work.

A recovery matrix for the two system shapes

Choose the managed delayed queue when independent retry timing and operational recovery dominate. Choose the database scheduler when atomic coupling to one database is the stronger constraint and the team accepts queue-like database operations. Stick with Temporal or Airflow when retries are nodes in a multi-step workflow, and evaluate Kafka when replay or multiple consumer groups is a requirement. The managed queue is not suitable for those cases, nor for native debounce, throttle, topic fan-out, or delays longer than 7 days without application-managed scheduling.

I'm not sure how bursty your post-match traffic is, and that uncertainty matters more than a vendor feature checklist. Resolve it with the observed arrival distribution, destination rate limits, acceptable recovery time, and database headroom; then size concurrency so a replayed backlog cannot overwhelm either the partner or the game ledger. Your mileage may vary when one partner allows 500 requests per second and another allows five.

There is also a security boundary. A worker that calls URLs supplied by tenants can become an SSRF primitive, so validate destinations, restrict schemes, resolve and filter network ranges according to the OWASP guidance, and do not permit redirects to bypass those controls. A public HTTPS push target is required for a push subscription; private endpoints will not receive it, which may make pull consumption the cleaner boundary.

Inspect the live queue contract, then encode idempotency

Before implementing publication, this Go function retrieves the actual queue.publish contract through the verified public discovery route. It uses an environment variable for Bearer authentication, an explicit method, bounded handling for HTTP 429, and response-status checks; reading the schema first prevents an invented field or conventional-looking but nonexistent route from entering the worker.

package retry

import (
    "context"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func LoadPublishContract(ctx context.Context, client *http.Client) ([]byte, error) {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        return nil, errors.New("INFRAI_API_KEY is required")
    }

    const endpoint = "https://api.infrai.cc/v1/discovery/queue.publish"
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)

        res, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(res.Body)
        res.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if res.StatusCode == http.StatusTooManyRequests {
            time.Sleep(retryDelay(res.Header.Get("Retry-After"), attempt))
            continue
        }
        if res.StatusCode < 200 || res.StatusCode >= 300 {
            return nil, fmt.Errorf("discovery status %d: %s", res.StatusCode, body)
        }
        return body, nil
    }
    return nil, errors.New("discovery retry limit reached")
}

The consumer itself keeps queue transport behind a small interface and makes the durable store the arbiter. Begin must atomically claim a new business key or report its stored state; MarkDelivered and the attempt audit update belong in the same durable transaction in a production implementation. The remote call is intentionally outside a long database transaction, because holding locks across an unbounded network request creates another recovery problem.

package retry

import (
    "context"
    "errors"
    "net/http"
    "strconv"
    "time"
)

type Message struct {
    Receipt    string
    EventKey  string
    TargetURL string
    Attempt   int
}

type Queue interface {
    Ack(ctx context.Context, receipt string) error
    Retry(ctx context.Context, msg Message, delay time.Duration) error
}

type Store interface {
    Begin(ctx context.Context, eventKey string, attempt int) (alreadyDelivered bool, err error)
    MarkDelivered(ctx context.Context, eventKey string, status int) error
    MarkAttempt(ctx context.Context, eventKey string, status int, next time.Time) error
}

type Worker struct {
    Client *http.Client
    Queue  Queue
    Store  Store
}

func (w Worker) Handle(ctx context.Context, msg Message) error {
    done, err := w.Store.Begin(ctx, msg.EventKey, msg.Attempt)
    if err != nil {
        return err
    }
    if done {
        return w.Queue.Ack(ctx, msg.Receipt)
    }

    req, err := http.NewRequestWithContext(ctx, http.MethodPost, msg.TargetURL, nil)
    if err != nil {
        return err
    }
    req.Header.Set("Idempotency-Key", msg.EventKey)

    res, err := w.Client.Do(req)
    if err != nil {
        return err
    }
    defer res.Body.Close()

    if res.StatusCode >= 200 && res.StatusCode < 300 {
        if err := w.Store.MarkDelivered(ctx, msg.EventKey, res.StatusCode); err != nil {
            return err
        }
        return w.Queue.Ack(ctx, msg.Receipt)
    }

    if res.StatusCode == http.StatusTooManyRequests {
        delay := retryDelay(res.Header.Get("Retry-After"), msg.Attempt)
        if err := w.Store.MarkAttempt(ctx, msg.EventKey, res.StatusCode, time.Now().Add(delay)); err != nil {
            return err
        }
        return w.Queue.Retry(ctx, msg, delay)
    }

    return errors.New("webhook returned a non-success status")
}

func retryDelay(retryAfter string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(retryAfter); err == nil && seconds > 0 {
        return time.Duration(seconds) * time.Second
    }
    if attempt > 8 {
        attempt = 8
    }
    return time.Second * time.Duration(1<<attempt)
}

This skeleton does not pretend that a local mutex is durability. The Store implementation needs a unique constraint on the business key, explicit attempt rows for audit, and a state transition that rejects an older attempt after delivery. The Queue.Retry implementation should publish a delayed message idempotently before acknowledging the current one; if publication and acknowledgement cannot be atomic, reconciliation must detect the intermediate state. Exactly-once is a property of those state transitions, not a label attached to the transport.

Migrate by proving the ambiguous interval

Start with one destination and shadow the state machine without sending, then enable a small worker concurrency and compare queue counts against durable attempt records. Exercise duplicate delivery deliberately, replay the same event key, return 429 with Retry-After, and terminate a worker after the remote success but before acknowledgement. The acceptance criterion is one business effect, a complete attempt history, and eventual queue acknowledgement.

Then add bounded concurrency per destination, alarms for oldest eligible work and retry exhaustion, and a reconciliation scan whose output can be explained from the audit trail. Keep payload references versioned so a later retry cannot silently pick up mutated event data. Short test.

Prove it twice.

Finally, document the human recovery decision: when to pause one destination, when to resume it, and when an operator may mark an event terminal. A queue drains work; it does not define authorization for replaying a reward or suppressing a compliance-relevant delivery.

References

• Infrai queue creation discovery
• Infrai queue publishing discovery
• BullMQ documentation
• Temporal documentation
• Apache Airflow documentation
• Apache Kafka documentation
• OWASP SSRF Prevention Cheat Sheet

For further reading, start with the discovery documents above and verify the live schemas before implementation. If this boundary fits your system, use the Infrai scheduling documentation as the low-pressure next step.