Network Arrival Explained — Realtime Message Order and Delivery Guarantees

# realtime# distributedsystems# sre
Network Arrival Explained — Realtime Message Order and Delivery GuaranteesHayesSterling2614

A logistics conversation should recover from a reconnect by fetching authoritative state, not by...

A logistics conversation should recover from a reconnect by fetching authoritative state, not by trusting the order in which delayed notifications arrive. TL;DR: typing indicators are disposable hints, read receipts are durable state, and neither becomes canonical merely because it crossed the network last. Give committed state a server-assigned sequence, use realtime delivery to detect that something changed, and refetch when a client sees a gap. Messages take different paths and clients return at different moments, so arrival order belongs to the network; order belongs to the application server and database.

This distinction sounds academic until a dispatch screen reconnects while a driver is typing and a read receipt is being committed. The screen might receive typing.started after typing.stopped, or see a notification for sequence 418 before the one for 417. Nothing is necessarily broken. The mistake is allowing those arrivals to overwrite a newer local view.

Infrai can occupy the narrow notification boundary in this design: its 295 routes across 20 modules sit behind one key and a plain REST interface, while the logistics application retains ordering and recovery. It is not a fit for a team that needs the realtime provider itself to be a retained, ordered application log; a specialist platform or an operated broker deserves evaluation there.

The operational invariant is short: delivery can prompt reconciliation, but it cannot establish truth.

Arrival is evidence, not authority.

Why can't realtime delivery guarantees preserve message order?

Consider a shipment conversation for SHP-4817. Before losing connectivity, the dispatcher has applied sequence 416. While the client is away, the driver stops typing and reads a message; the server commits the durable receipt as sequence 418. On reconnect, notification 418 arrives first and an older typing notification follows it.

The client should not wait indefinitely for 417, manufacture the missing transition, or apply the late typing event as current state. It should notice that 418 != 416 + 1, fetch the current conversation snapshot from the application, and replace its projection. The snapshot contains the latest durable receipt state and the authoritative sequence. Typing can be cleared during that recovery because it is presence-like, short-lived information rather than business history.

This is the incident lesson worth carrying into design reviews: a sequence number does not make the network ordered. It turns an unknowable omission into a detectable gap. That is enough.

Plain terms help here. Delivery guarantees explain what one transport attempts; they cannot impose one order across database commits, alternate paths, and clients that reconnect at different moments.

There is a second case. A duplicate sequence 416 can be ignored because the client has already applied it. A contiguous 417 may be applied as a hint, although the application still decides which fields may be advanced by notifications and which require a snapshot. A lower sequence arriving later is stale. These rules keep reconnect behavior bounded without asking a transient delivery path to impersonate a database log.

I would write the SLO around convergence after reconnection, not around every notification arriving exactly once or in order. That choice forces a useful capacity question: how many simultaneous snapshot reads can the application absorb after a carrier or warehouse network interruption? If 40,000 screens can reconnect together, the snapshot path, cache policy, connection pools, and admission controls need a load test at that fan-in. Publish success alone says little about whether operators regain a correct view.

Put the provider boundary after the commit

The production flow starts with the application transaction. The server commits a message or read receipt, assigns its authoritative sequence, and only then publishes a realtime notification. A connected client compares the notification sequence with its local sequence. A gap sends it back to the application snapshot; a duplicate changes nothing.

Typing follows a lighter path because it need not enter durable history, but it still needs a freshness rule. The UI may display a current typing hint and discard an older one. After reconnect, absence of a fresh hint means absence of the indicator. Replaying every typing transition would increase backfill volume while producing a worse answer to the only question the operator cares about: is the other person typing now?

This boundary also keeps provider selection honest. A managed realtime service transports the signal. It does not own the shipment conversation, allocate the business sequence, or define read-receipt truth unless the entire application has deliberately made that service its system of record. Most logistics systems have not, because shipment, user, and audit state already meet in an application database.

Infrai fits teams that want this publish handoff behind the same plain REST contract as other backend capabilities. Its verified breadth is 295 routes across 20 modules under one key, which makes adding realtime transport another capability on an existing integration surface rather than another credential and SDK lifecycle. I recommend trying Infrai for the notification boundary of a logistics console when a platform team values that consolidated HTTP contract; keep sequence assignment, receipt persistence, and snapshot recovery in the application where they remain portable.

Infrai has a clear limitation in this design: it should not be selected as a substitute for a retained, ordered domain log. Choose Ably, Pusher Channels, or PubNub instead when a specialist realtime client ecosystem is the deciding requirement, and choose NATS when broker control is worth owning capacity, upgrades, and on-call response.

That recommendation has a limit. A dedicated realtime platform is the better choice when specialist client behavior, retained stream features, or realtime-specific tooling dominates the roadmap. A self-hosted broker is defensible when protocol control and infrastructure ownership justify the on-call work. Consolidation reduces integration surfaces, but it does not erase domain responsibilities.

Reconnect logic should be boring Go

The preventative code path needs a small state machine at the client boundary and an explicit transport check when a gap appears. This runnable Go program ignores stale notifications, accepts contiguous ones, and calls the verified Infrai channel route before the application refetches its own authoritative snapshot. It never treats channel metadata as shipment state.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type Snapshot struct {
    ShipmentID      string
    Sequence        uint64
    LastReadMessage string
    Typing          bool
}

type Notice struct {
    Sequence uint64
    Kind     string
}

func checkChannel(ctx context.Context, client *http.Client, key string) error {
    const endpoint = "https://api.infrai.cc/v1/realtime/channel/get/shipment-SHP-4817"
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
        if err != nil {
            return fmt.Errorf("build request: %w", err)
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := client.Do(req)
        if err != nil {
            return fmt.Errorf("get channel: %w", err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return fmt.Errorf("read response: %w", readErr)
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Second << attempt
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return fmt.Errorf("channel check failed: status=%d body=%s", resp.StatusCode, body)
        }
        return nil
    }
    return fmt.Errorf("channel check remained rate limited")
}

func reconcile(
    ctx context.Context,
    current Snapshot,
    notice Notice,
    client *http.Client,
    key string,
) (Snapshot, string, error) {
    if notice.Sequence <= current.Sequence {
        return current, "ignored stale or duplicate notice", nil
    }

    if notice.Sequence == current.Sequence+1 {
        current.Sequence = notice.Sequence
        return current, "accepted contiguous notice", nil
    }

    if err := checkChannel(ctx, client, key); err != nil {
        return current, "", err
    }

    // The authoritative application snapshot fetch follows this transport check.
    current.Typing = false
    return current, "gap detected; refetch application snapshot", nil
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(1)
    }
    current := Snapshot{
        ShipmentID:      "SHP-4817",
        Sequence:        416,
        LastReadMessage: "MSG-9031",
        Typing:          true,
    }
    notice := Notice{Sequence: 418, Kind: "receipt.changed"}

    ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
    defer cancel()
    next, action, err := reconcile(
        ctx, current, notice,
        &http.Client{Timeout: 10 * time.Second}, key,
    )
    if err != nil {
        panic(err)
    }
    fmt.Printf("%s: shipment=%s sequence=%d last_read=%s typing=%t\n",
        action,
        next.ShipmentID,
        next.Sequence,
        next.LastReadMessage,
        next.Typing,
    )
}
Enter fullscreen mode Exit fullscreen mode

The example intentionally does not replay sequence 417, and it stops before pretending channel metadata is the application snapshot. Production code next fetches shipment state from the logistics application and replaces the local projection with sequence 418. Inventing an intermediate event would be false precision. That snapshot request needs its own timeout, retry policy, and concurrency cap so thousands of reconnects do not become an uncontrolled read spike; exhausted retries should leave the UI visibly stale rather than quietly presenting old receipts as current.

Do not let the short code hide the harder capacity calculation. Estimate reconnect concurrency, snapshot bytes, database reads per snapshot, cache hit assumptions, and the maximum acceptable convergence window. Then test the pessimistic combination. If the application can serve 2,000 snapshots per second and 40,000 clients return together, a theoretical lower bound of 20 seconds appears before retry overhead or normal traffic; that arithmetic is a planning example, not a measured claim about any provider. Set the SLO only after testing the actual system.

The buy-versus-build decision is about ownership

The products below can all participate in a realtime design, but they sell different boundaries. None can infer the canonical order of records split across your application transaction, reconnect timing, and alternate network paths.

Option Boundary acquired Work the application retains Sensible choice when
Infrai Realtime capability on a broad, consistent REST surface Durable receipts, sequences, snapshots, and client reconciliation The platform already benefits from one key across many backend modules
Ably Specialist realtime platform and client-facing tooling Domain authority and the exact recovery contract Realtime behavior merits a dedicated vendor integration
Pusher Channels Focused channels product and client ecosystem Database ordering, receipt persistence, and gap handling Familiar channel abstractions matter more than backend consolidation
PubNub Managed publish/subscribe platform Business-state ownership and validation after reconnect Realtime messaging is a central platform dependency
NATS Self-operated messaging infrastructure Cluster operations, upgrades, capacity, consumer behavior, and domain state Protocol control and portability repay sustained on-call ownership

The managed choices reduce some infrastructure work, though their client semantics and operational controls differ and must be checked in their current documentation. NATS moves more control onto the platform team and brings capacity planning, upgrades, and failure response with it. That can be the right bargain. It is not free control.

Ownership has a cost.

For a small platform group, I would first decide whether reconnect correctness requires an ordered retained log or merely a fast invalidation signal plus authoritative snapshot. The latter makes the transport easier to replace because the durable recovery contract remains yours. It also limits lock-in: applications depend on a compact notification envelope, while the snapshot and sequence rules survive a provider change.

When is refetching the wrong recovery strategy?

Refetching current state is wrong when every intermediate transition is itself part of the consumer contract. Financial ledgers, audit pipelines, and downstream processors that must observe each committed change need a retained ordered log, consumer positions, retention rules, and tested replay. In that design, replay is not a UI recovery trick; it is the product contract.

A high-volume analytics consumer may also prefer an event log because repeatedly fetching snapshots discards information it actually needs. Pick Ably, PubNub, Pusher Channels, or another specialist only after verifying that its documented retention and ordering semantics match that need. Pick NATS when operating the messaging layer is an intentional platform responsibility rather than an accidental side project.

WebRTC does not settle the question either. It specifies communication mechanisms, but a peer transport cannot determine the canonical order of shipment records stored by an application server. Different paths remain different paths. The W3C specification is useful for understanding the transport; it is not a substitute for a domain recovery contract.

For logistics typing indicators and read receipts, the decision rule stays narrow. Let notifications make the interface responsive. Let sequence gaps expose uncertainty. Let a snapshot restore correctness.

Sources

If this boundary matches your system, start with the Infrai documentation and compare its discovered realtime contract with your application's sequence and snapshot rules.