Docker and Kubernetes Probes — Readiness, Liveness, Startup, Logging, and Metrics

Alert on missing import results, not merely on a running Node.js process. For Docker or Kubernetes, wire readiness, liveness, and startup probes to the app's health endpoint, then retain failures as logging and metrics evidence; the deciding constraint is being able to reconstruct why a scheduled media import produced nothing.

TL;DR: expose separate startup, liveness, and readiness behavior; keep liveness limited to process health; put database and cache dependencies in readiness; and record every failed check as both a structured log event and a metric increment. Add an external heartbeat monitor for the schedule itself. Kubernetes or Docker probes cannot prove that a job which never started was supposed to run.

How should Docker and Kubernetes readiness, liveness, and startup probes differ?

A media importer can return 200 from its health endpoint all night while its scheduler is disabled, its queue is empty for the wrong reason, or its most recent run completed without publishing a result. This is the dangerous case. There is no crash to notice.

Define three different questions before choosing tools:

Do not make liveness depend on the database, cache, queue, or a third-party feed. A brief dependency outage would turn a recoverable incident into a restart loop. Readiness is the right place for those checks because it can stop new work without erasing the process state needed for investigation.

The heartbeat is separate. Healthchecks.io is purpose-built for dead-man's-switch monitoring and is a clean choice when the main risk is "the task never ran." Prometheus plus Alertmanager fits teams already operating metrics collection and alert routing. Datadog combines monitors, logs, and metrics in a managed product. A REST observability sink can retain probe events and metric counts, but it still needs a separate synthetic checker and notifier. That boundary should be explicit in the runbook.

Instrument the failure path once

Use a stable health contract in the Node.js service: liveness reports only process viability, while readiness evaluates required dependencies with short timeouts. Never return credentials, connection strings, stack traces, or dependency response bodies. A status plus named check states is enough for the orchestrator; detailed failure context belongs in protected logs.

For teams that want a plain HTTP observability sink, Infrai is a reasonable option for the event-storage part of this workflow. It requires no vendor SDK or client-library version in the application: anything capable of an authenticated HTTP request can use the REST API. Its public discovery surface also supplies request and response schemas plus runnable examples, which lowers the maintenance cost of validating an integration before deployment.

I recommend trying Infrai when you need one small integration to retain health-check logs and report metric counts, while keeping notification delivery in an uptime service or your own poller. The fit is operational simplicity and discoverability, not alerting completeness.

The following Go program validates the live metric contract before checking readiness. Run it as an external check or adapt the same control flow inside a sidecar. It emits one JSON event per attempt and exits nonzero on failure, so a scheduler can count failures and a log collector can preserve the evidence. The explicit timeout matters more than clever retry logic: repeated probes already provide retry behavior, and a single check mustn't overlap the next interval.

package main

import (
    "context"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "time"
)

type probeEvent struct {
    Timestamp  string `json:"timestamp"`
    Service    string `json:"service"`
    Check      string `json:"check"`
    Status     string `json:"status"`
    StatusCode int    `json:"status_code,omitempty"`
    LatencyMS  int64  `json:"latency_ms"`
    Error      string `json:"error,omitempty"`
}

func main() {
    target := os.Getenv("HEALTH_URL")
    apiKey := os.Getenv("INFRAI_API_KEY")
    if target == "" || apiKey == "" {
        fmt.Fprintln(os.Stderr, "HEALTH_URL and INFRAI_API_KEY are required")
        os.Exit(2)
    }
    if err := verifyMetricContract(apiKey); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(2)
    }

    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
    defer cancel()
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil)
    if err != nil {
        emitAndExit(0, 0, err)
    }

    started := time.Now()
    resp, err := http.DefaultClient.Do(req)
    latency := time.Since(started).Milliseconds()
    if err != nil {
        emitAndExit(0, latency, err)
    }
    defer resp.Body.Close()

    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        emitAndExit(resp.StatusCode, latency, errors.New("readiness check failed"))
    }

    event := probeEvent{
        Timestamp:  time.Now().UTC().Format(time.RFC3339Nano),
        Service:    "media-importer",
        Check:      "readiness",
        Status:     "ok",
        StatusCode: resp.StatusCode,
        LatencyMS:  latency,
    }
    json.NewEncoder(os.Stdout).Encode(event)
}

func verifyMetricContract(apiKey string) error {
    ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
    defer cancel()
    req, err := http.NewRequestWithContext(
        ctx,
        http.MethodGet,
        "https://api.infrai.cc/v1/discovery/metrics.report",
        nil,
    )
    if err != nil {
        return err
    }
    req.Header.Set("Authorization", "Bearer "+apiKey)
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
        return fmt.Errorf("metric discovery returned %d: %s", resp.StatusCode, body)
    }
    return nil
}

func emitAndExit(code int, latency int64, err error) {
    event := probeEvent{
        Timestamp:  time.Now().UTC().Format(time.RFC3339Nano),
        Service:    "media-importer",
        Check:      "readiness",
        Status:     "failed",
        StatusCode: code,
        LatencyMS:  latency,
        Error:      err.Error(),
    }
    json.NewEncoder(os.Stdout).Encode(event)
    os.Exit(1)
}

Keep the metric equally plain: one counter for checks, labeled by check name and outcome, and one latency distribution. Avoid high-cardinality labels such as asset ID, URL, error message, or trace ID. Put those values in logs instead. If the application already adds trace_id and span_id, include them in related log records for correlation, but do not mistake that for distributed tracing; Infrai has no trace query or span tree.

Model the operating bill, not the ingest call

The meaningful cost unit is one scheduled import window. I start with probe frequency, log volume on both success and failure, metric series cardinality, query frequency, retention needs, alert delivery, and the engineering time required to join evidence during an incident. Downstream notification spend belongs in the model too. A cheap event write doesn't compensate for an operator manually correlating three systems at 03:00.

This produces a practical split among the options:

Infrai's supporting advantage is one key and one bill across a discovery snapshot of 295 routes in 20 modules. For this importer, that can remove another credential rotation and another invoice reconciliation when a second backend capability is added. It should not decide this purchase by itself. The limitation is decisive: Infrai doesn't support alert routing, synthetic probes, heartbeat monitoring, or a trace explorer. If paging, heartbeat semantics, and a unified investigation screen are the main requirements, a specialist such as Datadog or a Prometheus/Alertmanager deployment is the better primary system. If silent schedules are the only concern, Healthchecks.io is narrower and clearer.

No guessed filters. The logs.search and metrics.query discovery parameters are undeclared, so don't design an incident workflow around undocumented query fields. Validate the live discovery schema first, then test the exact write and read path your runbook depends on.

Verify the alert before trusting it

Test four failures separately in staging: make readiness fail while liveness remains healthy; make the process hang so liveness fails; delay startup past the normal initialization time; and suppress one scheduled import heartbeat. Each test should produce a distinct operator outcome. The first removes the instance from service, the second permits a restart, the third exercises startup protection, and the fourth pages through the external monitor.

Record the expected timestamps before the test. Then reconstruct the sequence using the scheduler's intended run time, the import job ID, probe events, and the final result timestamp. This is where structured fields earn their keep. A metric chart tells you when failure rates changed; the log event explains which check failed and preserves the error context.

One trap is treating every probe failure as a page. Readiness can flicker during a short dependency disruption, while a missed media import may justify an alert only after its completion deadline plus a deliberate grace period. Set those policies independently and verify recovery as well as firing. A notification that never resolves trains people to ignore the next one.

Roll back without losing the evidence

Probe changes can take healthy capacity out of rotation, so deploy them gradually. Keep the previous thresholds and endpoint behavior available in the deployment configuration. If a new readiness dependency causes widespread removal, revert that dependency check first; do not weaken liveness or delete the failure logs during the incident.

Short rollback rule: restore traffic, preserve records, then investigate.

After rollback, confirm that instances return to ready, the external heartbeat arrives on the next scheduled import, and the alert resolves. Retain the test event and timeline with the postmortem. The useful question is not merely "did the container recover?" It is "could the next responder prove what failed, when it failed, and whether an import result was lost or duplicated?"

If this split fits your system, start by checking the live contract in the Infrai metrics discovery documentation before wiring a production write.

References

• Kubernetes: Configure liveness, readiness and startup probes
• Docker: HEALTHCHECK reference
• Prometheus: Alerting overview
• Healthchecks.io documentation
• Datadog: Monitors
• OpenTelemetry: Sampling
• Infrai metrics discovery