callumreed2198Short answer: put one policy decision in front of every email and SMS attempt, snapshot that decision...
Short answer: put one policy decision in front of every email and SMS attempt, snapshot that decision with the event, and give the application team ownership of password-reset templates while the delivery layer owns channel rendering constraints.
The page says password-reset messages are late. On-call can see a healthy event consumer and a rising queue age, but not whether each message was blocked by a user preference, a suppression entry, an expired reset token, or a delivery attempt that never reached an accepted state. That is an observability failure before it is a transport failure. The least complex fix is a small, explicit decision record between the event and the sender.
This matters more for a short-expiry reset than for a newsletter. A message can be delivered exactly as designed and still be useless because its token expired in the queue. Retrying blindly can make the page look better while sending duplicates. I've been paged by missed jobs and duplicate deliveries; both incidents teach the same habit: prove the state transition before touching the retry button.
Start with the user-visible objective: a valid reset message reached at least one permitted channel before the token expired. Queue depth is supporting evidence, not the objective. A useful page groups events by terminal reason and shows the age of the oldest event that is still eligible to send. The terminal reasons should distinguish at least delivered, suppressed, opted_out, expired, and permanent_recipient_rejection. Those are application states in this design, not raw provider strings.
The signal that should fire earlier is an increasing count of eligible events whose remaining token lifetime is approaching the system's normal end-to-end delivery time. No universal threshold is honest here. Your queue latency, token lifetime, and traffic shape determine it, and I'm not sure a static threshold is even appropriate for a system with sharp class-change peaks. A rolling baseline may fit that workload; a fixed service-level threshold is easier to explain in a runbook. Either way, record the inputs so the alert can be defended after the page.
Instrument the decision boundary, not just the network call. For every notification event, emit one transition when policy evaluation completes and another when the channel adapter returns an accepted or rejected outcome. Attach an internal event ID, template revision, preference revision, suppression decision, channel, token expiry, attempt number, and a stable reason code. Do not put the reset token, email address, or phone number in metrics labels or logs.
One line in the runbook should be blunt: a suppressed message is not a failed send. It is a successful policy decision. Paging on it trains responders to bypass consent controls under pressure, which is the opposite of what an incident process should encourage.
Evaluate policy once per attempted channel, as close as possible to dispatch, using a consistent snapshot of the user's channel preferences and the relevant suppression lists. The event says what happened. The policy result says whether this message may be attempted now. The adapter says how to ask a transport to deliver it. Keeping those statements separate makes a Node.js event notification system easier to test even when the reference state machine is expressed in another language.
Order matters.
First reject an event that is already past its useful expiry. Then load the user's current email and SMS preferences. Next apply global and channel-specific suppression. Finally choose a permitted channel according to the product rule and create an idempotent attempt. A preference is a user choice about a class of messages; a suppression entry is an operational or compliance control on a destination. They can produce the same immediate outcome, but they should not share a reason code or an audit trail because they answer different questions.
| Decision input | Allowed outcome | Recorded reason |
|---|---|---|
| Token is no longer useful | Do not dispatch | expired |
| User disabled the channel | Do not dispatch | opted_out |
| Destination matches suppression | Do not dispatch | suppressed |
| Current policy permits the channel | Claim one attempt | allowed |
Here is the boundary I want the dispatcher to expose. It is deliberately boring. The stores can sit behind a database or service, and the transports can change, but the policy result remains stable.
package notify
import (
"context"
"time"
)
type Channel string
const (
Email Channel = "email"
SMS Channel = "sms"
)
type ResetRequested struct {
EventID string
UserID string
Email string
Phone string
Token string
ExpiresAt time.Time
TemplateRevision string
}
type Decision struct {
Channel Channel
Allowed bool
Reason string
PreferenceRevision string
SuppressionVersion string
}
type Policy interface {
Evaluate(ctx context.Context, event ResetRequested, channel Channel) (Decision, error)
}
type AttemptStore interface {
Begin(ctx context.Context, idempotencyKey string, decision Decision) (created bool, err error)
Complete(ctx context.Context, idempotencyKey string, outcome string) error
}
type Transport interface {
SendReset(ctx context.Context, event ResetRequested) (outcome string, err error)
}
The idempotency key should identify the event, channel, and template revision. A retry of the same logical attempt then finds the existing record instead of creating a second message. Do not key only on user ID: two legitimate reset requests from one learner would collide. Do not key only on a provider request ID either, because that ID does not exist until after the boundary at which the application needs duplicate protection.
There is a subtle race between preference changes and queued work. Suppose a learner requests a reset, then disables SMS before the worker reads the event. For delivery permission, the dispatch-time preference should win; for incident reconstruction, retain the preference revision that produced the decision. This is why copying a boolean such as smsEnabled into an event and trusting it hours later is weak. The event is immutable evidence. Permission is current policy. Opt-out processing needs its own idempotency key and audit record too. Repeating the same opt-out must converge on the same disabled state, while new dispatch decisions must observe that state before sending. The CTIA messaging interoperability and compliance material belongs in the design review for SMS controls; local counsel and your actual messaging program determine the final policy. An engineering article cannot classify a password-reset message for every jurisdiction or program.
The application team should own the password-reset subject, wording, token placement, expiry language, localization keys, and template revision. That team owns the user flow and can deploy a matching UI and message change together. The delivery layer should own channel constraints such as encoding, safe substitution, destination normalization, and the transport-specific envelope. This split keeps product meaning out of a shared communications service without forcing each feature team to learn delivery mechanics.
Store a template revision on the notification event and attempt. Render from a structured model, not a caller-supplied block of HTML or text. For a reset request, that model might contain a reset URL, an expiry timestamp, locale, and product display name. The template decides how those fields appear; the policy layer never edits copy, and the transport never invents it. Preview tests should render every supported locale with missing optional fields, long display names, and a fixed clock. The fixed clock is important because expiry copy that changes during a snapshot test creates noise and encourages engineers to ignore diffs.
The catch is organizational.
Application ownership is not suitable when dozens of teams can publish arbitrary templates without review, localization support, or a compatible deployment process. In that environment, a central communications team may need to own publication and schema validation, while application teams own reviewed content specifications. Stick with centralized template ownership when regulatory review or brand approval must gate every change. You give up some deployment independence, but the approval boundary becomes real instead of ceremonial.
Do not let this choice leak into the event schema. The event should reference a logical template and revision regardless of which team publishes it. That preserves the operational model if ownership changes later.
The worker below shows the critical ordering. It checks usefulness, evaluates both channels, claims an idempotent attempt, and records the outcome. A production implementation also needs durable queue acknowledgement and retry classification, but those mechanisms belong around this function rather than inside the policy rules.
package notify
import (
"context"
"fmt"
"time"
)
type Worker struct {
Policy Policy
Attempts AttemptStore
Transports map[Channel]Transport
Now func() time.Time
}
func (w Worker) Handle(ctx context.Context, event ResetRequested) error {
if !w.Now().Before(event.ExpiresAt) {
return nil
}
for _, channel := range []Channel{Email, SMS} {
decision, err := w.Policy.Evaluate(ctx, event, channel)
if err != nil {
return fmt.Errorf("evaluate %s policy: %w", channel, err)
}
if !decision.Allowed {
continue
}
key := event.EventID + ":" + string(channel) + ":" + event.TemplateRevision
created, err := w.Attempts.Begin(ctx, key, decision)
if err != nil {
return fmt.Errorf("begin %s attempt: %w", channel, err)
}
if !created {
return nil
}
outcome, err := w.Transports[channel].SendReset(ctx, event)
if err != nil {
return fmt.Errorf("send reset by %s: %w", channel, err)
}
return w.Attempts.Complete(ctx, key, outcome)
}
return nil
}
Keep the adapter outcome vocabulary small and map provider responses at the edge. The worker should decide retries from an internal classification, not from free-form error text. It also needs a deliberate multi-channel rule. The example stops after the first allowed channel because sending both email and SMS can create duplicate user experiences even though the transport attempts are technically distinct. If the product requires fallback, wait for a defined terminal outcome before trying the next channel; do not treat a slow callback as proof of non-delivery.
Stop there.
Test the decision table before testing adapters. Cover email allowed and suppressed, SMS allowed and opted out, both blocked, token expired, preference changed after enqueue, duplicate event delivery, and two distinct reset events for one user. Then run contract tests for template inputs and adapter outcome mapping. The dispatch test should assert that a blocked channel never invokes its transport. That negative assertion catches more consent regressions than a happy-path mock returning success.
Deployment needs the same discipline. Introduce new reason codes before dashboards depend on them. Deploy readers that understand both old and new template revisions before publishing events with the new revision. During rollback, keep idempotency records and policy audit rows; deleting them trades a clean dashboard for duplicate risk.
The false-positive cost is real. If the early warning threshold fires whenever a handful of events approach expiry, normal queue jitter will page the team, responders will learn to wait, and the first genuinely broad delay will arrive to a muted audience. Start the signal as a dashboard and ticket, compare it with user-impacting outcomes, then page only when it reliably identifies actionable risk. Quiet is not the goal. Trust is.
Use the two primary sources above during design review: the delivery documentation for adapter boundaries, and the CTIA material for the SMS policy discussion. Recheck both when transport behavior or messaging-program requirements change.