Retention Architecture for Node.js SaaS: Scheduled API or Work Queue?

Short answer: use cron to start a bounded cleanup API, keep selection and deletion idempotent in the database, and add a queue only when one invocation cannot safely finish the eligible work or each record needs an independent retry.

The scheduler is not the correctness boundary. A timer can fire late, twice, or while an earlier run is still active; a queue can deliver the same message again when processing outlasts its visibility timeout. The durable data model therefore has to make repetition harmless and leave enough evidence to reconcile every destructive write.

That distinction prevents a small retention task from becoming an accidental distributed system. For a modest SaaS table, a cron-triggered API that claims and processes a limited batch may be the best simple design. For a large backlog, variable-duration external calls, or per-record retry requirements, the same API should claim work and publish stable job identities to workers. The migration from one shape to the other should not change the deletion contract.

Define the retention boundary before choosing the trigger

“Old” is not a database predicate until someone defines the clock, the eligible states, the exceptions, and the evidence that must survive. An account marked inactive thirty days ago may still own unsettled activity; an expired session may have no such dependency. Treating both as created_at < cutoff hides a policy decision inside a convenient query, which is precisely where audit and reconciliation become difficult.

Write the rule as versioned data: a rule identifier, a cutoff calculation, an eligibility predicate, a disposition such as delete or redact, and an owner who can approve changes. Record the evaluated cutoff with each run rather than recomputing it during retries. If policy version retention-v3 selected a row under cutoff T, a replay must not silently reinterpret that row under retention-v4.

Deletion is a write.

That short statement carries the main architectural consequence. A cleanup operation belongs in the same correctness discipline as a payment mutation: stable idempotency keys, transactional state changes, immutable audit evidence, and reconciliation between intended and completed effects. “The job succeeded” is too coarse. The useful evidence says which rule selected which record, which disposition occurred, and which idempotency key prevented a repeated effect.

Compliance boundaries should remain configuration and policy inputs, not numbers copied into application code. Retention periods vary with data class and jurisdiction, and engineering cannot infer them from the age of a table. The application can enforce an approved limit, but legal and data-governance owners must supply that limit and its exceptions. This separation also makes a dry run meaningful: the system can report what a policy would affect before permission to destroy data is enabled.

How should a Node.js SaaS choose a scheduled cleanup API, cron, or queue?

Choose by retry unit and workload shape, not by framework preference. Cron answers when to inspect. An API provides an authenticated control boundary. A queue answers how to distribute independently retryable work. These mechanisms overlap operationally, but they do not solve the same problem.

Stick with the bounded API when the upper bound is demonstrably below the invocation budget and all effects can be committed locally. A queue is not an upgrade in correctness by itself; it adds delivery state that must be reconciled with database state. Conversely, use workers when a single record may wait on another system, when throughput must be controlled independently of the web process, or when retrying the entire batch would repeat too much completed work.

There is no universal batch size. I'm not sure one can be chosen responsibly without measuring row-lock time, transaction duration, downstream latency, and backlog growth in the actual deployment — and those measurements should be taken under a dry-run or non-destructive disposition first. The invariant is more durable than any number: every invocation has a strict work limit, and another invocation can resume without guessing what happened previously.

Measure first.

The application language does not alter this decision. A Node.js service may expose the control endpoint while a worker uses another runtime; the contract is the stable claim and idempotency model, not an in-process timer. In-process scheduling is suitable only when process lifetime, replica count, and overlap behavior are deliberately controlled. If those assumptions are implicit, a deploy or horizontal scaling event changes retention behavior without changing retention code.

Make the database own idempotency and the audit trail

An exactly-once mindset does not require pretending the transport delivers exactly once. It means defining one durable place where an effect can win once, then making every repeated attempt observe that decision. Queue documentation makes the risk concrete: with Amazon SQS, a received message becomes temporarily invisible, but it can become visible again if it is not deleted before the visibility timeout. A consumer must therefore tolerate another delivery.

Retries happen.

The cleanest unit is a retention event with a uniqueness constraint over the rule version, record identity, and disposition. In one database transaction, lock or conditionally claim the record, verify that it still satisfies the rule, apply the deletion or redaction, and insert the event. A repeated worker sees the existing event and returns success without repeating an external or destructive effect. One transaction cannot atomically cover an ordinary database and an external queue, so use a transactional outbox when publication must follow a claim: commit the claim and an outbox row together, then let a relay publish the job with the outbox identity as its deduplication key. This closes the database-to-publication gap without asking either system to participate in a distributed transaction, although it also creates a relay whose lag and failures must be monitored. The worker still checks the retention event because transport deduplication windows and database correctness are different concerns; a published message may be observed more than once, while the database effect remains singular. The interfaces below make those boundaries visible. The timer calls Start; Claim selects a bounded set and records a stable run; Apply is responsible for the transaction that changes data and writes evidence. Nothing in the worker trusts delivery uniqueness, and every component can be restarted after its last durable write rather than after its last attempted action.

package retention

import (
    "context"
    "time"
)

type Claim struct {
    RunID     string
    RuleID    string
    RecordID  string
    CutoffUTC time.Time
}

type Store interface {
    Claim(ctx context.Context, ruleID string, cutoff time.Time, limit int) ([]Claim, error)
    Apply(ctx context.Context, claim Claim) (alreadyApplied bool, err error)
}

type Publisher interface {
    Publish(ctx context.Context, claim Claim, idempotencyKey string) error
}

func JobKey(c Claim) string {
    return c.RuleID + ":" + c.RecordID + ":" + c.CutoffUTC.Format(time.RFC3339Nano)
}

Be careful with the word “delete.” Hard deletion can erase the very columns needed to explain eligibility, while retaining a complete pre-delete snapshot can defeat the retention policy. The audit event should contain the minimum non-sensitive evidence approved for the policy: identifiers or keyed digests where appropriate, rule version, cutoff, disposition, and completion time. The exact fields are a governance decision. The architecture merely guarantees that the evidence and effect cannot disagree inside the database transaction.

Secure the control plane and classify retry outcomes

A cleanup endpoint is an administrative control, even if it is invoked automatically. Keep it outside user-facing authorization paths, restrict who can reach it, and authenticate the request. RFC 2104 defines HMAC as keyed hashing for message authentication; an implementation can sign a canonical representation of the method, path, timestamp, and body, then reject a signature that does not verify or a timestamp outside the accepted replay window.

Canonicalization is part of the protocol. Both sides must agree on byte encoding, field order, timestamp form, and whether the body is hashed before signing. Don't sign a parsed object on one side and raw JSON on the other. Compare message authentication codes without data-dependent early exit, rotate secrets through an explicit key identifier, and log the key identifier rather than the secret.

The handler should acknowledge acceptance, not claim that every record has already been removed. Its state machine can stay small:

package retention

type Outcome int

const (
    Completed Outcome = iota
    AlreadyCompleted
    RetryLater
    Rejected
)

func Classify(status int, timedOut bool) Outcome {
    switch {
    case timedOut:
        return RetryLater
    case status == 409:
        return AlreadyCompleted
    case status == 429:
        return RetryLater
    case status >= 400:
        return Rejected
    default:
        return Completed
    }
}

Those status codes are an example of an internal contract, not a universal HTTP law. The important part is that retry decisions are explicit and audited: 429 means respect back pressure; 409 may mean the idempotent effect already exists if the API defines it that way; a permanent validation rejection goes to review instead of cycling forever. Avoid logging payloads merely because an operation failed. Cleanup often touches precisely the data that logs should not preserve.

Observability should connect four counts: eligible, claimed, published, and applied. A success counter alone cannot distinguish “nothing was eligible” from “the selector stopped finding records.” Track the age of the oldest eligible item, claim age, retry count, and the difference between claimed and applied work. Reconcile audit events back to policy runs. If the queue has a dead-letter path, it needs an owner and a replay procedure that retains the original idempotency key.

Roll out from evidence to deletion

Start with the policy evaluator in report-only mode. Capture counts by rule and data class, inspect the oldest candidates, and compare the result with the system that owns the business state. Then enable claims without destructive application, exercise duplicate triggers and overlapping runs, and confirm that every claim has one stable identity.

Next, apply the least destructive disposition to a deliberately limited cohort, reconcile eligible, claimed, and applied counts, and test replay with the same keys. Increase the limit only while lock duration, backlog age, and downstream pressure remain within the service's operating constraints. Rollback means disabling new claims; completed retention effects generally cannot be restored, so approval and dry-run evidence matter more than a conventional application rollback.

Keep the inline path until measurement shows it no longer fits. The migration to queue workers should replace execution after the claim boundary, not rewrite selection rules or audit semantics. That gives a team a genuinely simple starting point while preserving the one property scheduled cleanup cannot negotiate away: after any retry, overlap, or worker loss, the database can still explain exactly which effect happened once.

References

• AWS SQS visibility timeout documentation
• RFC 2104: HMAC keyed-hashing for message authentication