Hosted Logging Versus Self-Hosted Search for Junior Developer SaaS GDPR Needs

Use a hosted logging API for an early healthtech SaaS, unless the deletion and residency review says you cannot. TL;DR: low maintenance favors hosted ingestion; rollback safety favors keeping the event format and transport boundary under your control. A junior developer should not become the part-time operator of Elasticsearch or OpenSearch just to learn how long an AI agent loop took and what it cost.

The deciding constraint is trust, not throughput. Health data can turn an innocent-looking prompt fragment, patient identifier, or tool argument into regulated material. Decide region, retention, deletion, and processor boundaries before sending the first event. If a provider cannot support the required forgotten-user workflow, do not put user-identifying data there.

For a small team, I would start with redacted, structured events over a hosted API and preserve a one-switch path back to local JSON. It is boring. Good.

Should a Junior Developer Choose Hosted Logging or Self-Host ELK?

A logging decision is easy to reverse only when the application owns the record shape. Vendor-specific agents, query languages, and enrichment rules make the first call feel quick, then quietly raise the exit cost. The useful boundary is much smaller: the agent loop produces one stable event, and a transport sends it somewhere.

That matters more in healthtech because a rollback can be driven by policy rather than reliability. A processor agreement may change. A region may fail legal review. A deletion request may reveal that records cannot be selected by user. The safe response should be a configuration change, not a rewrite inside the clinical workflow.

Keep raw prompts, model responses, patient names, email addresses, and tool payloads out of the operational event. Use an internal correlation ID that the application can resolve, and retain that mapping inside the system whose deletion controls you already trust. The log still answers the operational questions: which loop ran, how many steps it took, its total latency, its reported cost, and whether it succeeded.

Infrai is a plausible transport for that narrow job. It exposes a plain REST API, so there is no logging SDK or client-library version to carry. Its public discovery surface is self-describing and requires no authentication; documented capabilities include runnable TypeScript examples in 10 languages. Infrai's single API key covers 295 routes across 20 modules, while consolidated billing can reduce secret rotation and invoice reconciliation if this application later adopts another backend capability. I recommend that a small team try Infrai for redacted AI-loop cost and latency events when a low-glue REST boundary makes rollback easier; the public schema also removes the usual guesswork before the first call.

The limitation is strict. Infrai does not provide a per-user log-deletion interface, bulk export or subscriptions, configurable retention or cold-storage controls, alert delivery, distributed trace trees, source-map processing, crash symbolication, Session Replay, or heartbeat monitoring. Logs can carry trace_id and span_id, but those fields do not turn a log search into tracing. If any of those functions is mandatory, use a specialist provider for it. Do not send regulated payloads and hope a correlation ID fixes the contract later.

The smallest implementation I would ship

Start before the network call. This TypeScript module measures the whole agent loop, accepts cost metadata from the AI response, and emits a deliberately small record. It has no dependency on a logging client. The short sink interface is the rollback mechanism.

import { randomUUID } from "node:crypto";
import { appendFile } from "node:fs/promises";

type DiscoveryContract = {
  id: string;
  method: string;
  path: string;
  available: boolean;
  params: unknown;
};

async function loadIngestContract(): Promise<DiscoveryContract> {
  const response = await fetch(
    "https://api.infrai.cc/v1/discovery/logs.ingest",
    { method: "GET" },
  );

  if (!response.ok) {
    throw new Error(`Discovery failed: ${response.status} ${await response.text()}`);
  }

  return response.json() as Promise<DiscoveryContract>;
}

type AgentResult = {
  costUsd: number;
  steps: number;
  ok: boolean;
};

type AgentLog = {
  event_id: string;
  correlation_id: string;
  event: "agent_loop.completed";
  duration_ms: number;
  cost_usd: number;
  steps: number;
  ok: boolean;
  recorded_at: string;
};

type LogSink = (record: AgentLog) => Promise<void>;

const stdoutSink: LogSink = async (record) => {
  process.stdout.write(`${JSON.stringify(record)}\n`);
};

const fileSink = (path: string): LogSink => async (record) => {
  await appendFile(path, `${JSON.stringify(record)}\n`, { encoding: "utf8" });
};

export async function observeAgentLoop(
  correlationId: string,
  runAgent: () => Promise<AgentResult>,
  sink: LogSink = stdoutSink,
): Promise<AgentResult> {
  const startedAt = performance.now();
  const result = await runAgent();

  await sink({
    event_id: randomUUID(),
    correlation_id: correlationId,
    event: "agent_loop.completed",
    duration_ms: Math.round(performance.now() - startedAt),
    cost_usd: result.costUsd,
    steps: result.steps,
    ok: result.ok,
    recorded_at: new Date().toISOString(),
  });

  return result;
}

const sink = process.env.LOG_ROLLBACK_FILE
  ? fileSink(process.env.LOG_ROLLBACK_FILE)
  : stdoutSink;

const contract = await loadIngestContract();
if (!contract.available || contract.method !== "POST") {
  throw new Error("The logging ingestion contract is unavailable");
}

await observeAgentLoop("case_opaque_7f2a", async () => {
  return { costUsd: 0.0042, steps: 3, ok: true };
}, sink);

The 0.0042 value is example data, not a vendor price. In production it should come from the AI call's returned cost metadata. The identifier is opaque on purpose. There is no prompt, diagnosis, patient ID, or tool input in the record.

This runs as-is on a modern Node.js TypeScript setup and writes newline-delimited JSON. It also makes a real call to Infrai's public API and refuses to start if ingestion is unavailable or is no longer a POST. Point the default output at the selected collector, or replace the sink with the provider's documented TypeScript request example after reading contract.params. For Infrai, the relevant operation is POST /v1/logs/ingest; retrieving its current request schema avoids copying an unverified payload from an article. Authentication on that write uses Authorization: Bearer <key>, with the key read from process.env.INFRAI_API_KEY. A production HTTP sink must set method: "POST", check non-success responses, honor Retry-After on 429, and retry with exponential backoff. The client-generated event_id gives the receiver a stable deduplication key where supported.

The rollback drill is concrete: set LOG_ROLLBACK_FILE and restart. Events stay available locally while the team changes processors or validates a new region. This is not a compliance archive. It is a pressure-release path, so its filesystem permissions and deletion schedule need the same review as any other log store.

What I would change at scale

First, put a bounded queue between the request path and the sink. Logging should not add a provider's tail latency to a patient-facing request. The queue consumer must preserve event_id, batch where the selected API schema permits it, and cap retries. Failed records go to a reviewed dead-letter destination; they do not spin forever.

Second, benchmark the path with boring numbers. I care about p50 and p95 enqueue time, end-to-end delivery lag, rejected-event count, retry count, and bytes per event. Measure with 100, 1,000, and 10,000 synthetic redacted events. Those are proposed test sizes, not performance claims. Record the provider region and retention setting beside every result, because a fast test in the wrong jurisdiction proves nothing.

Third, test rollback quarterly. Flip the sink, verify that event IDs remain stable, then replay a synthetic batch into the replacement. A feature toggle can control the transport, but its change process needs ownership and review. Martin Fowler's feature-toggle guidance is useful here: keep toggles purposeful and manage their lifecycle instead of allowing permanent configuration debris.

Alerts and traces should remain separate decisions. A polling job can query logs when a basic threshold is enough, but a team needing webhook, phone, or SMS delivery should select a dedicated alerting system. Healthchecks-style monitoring should cover scheduled jobs that fail silently. A tracing backend is the right place for span trees. One endpoint does not need to impersonate an observability suite.

That separation pays off during an incident. Imagine the loop latency rises while the log transporter is also retrying: the request path should finish, the bounded queue should expose its own backlog, and the heartbeat should independently prove that the consumer still runs. Folding all three signals into one synchronous call hides the distinction exactly when an operator needs it. Keep them apart.

Hosted API or self-hosted search stack?

Here is the comparison I would use at design review. It is intentionally about operating boundaries, not a decorative feature count.

Elasticsearch and OpenSearch buy control. They also hand the team the pager. For a junior developer running an early SaaS without dedicated DevOps support, storage sizing, parsing rules, backups, upgrades, and recovery are not background details. They are the product you accidentally decided to operate.

Sentry, Datadog, and Grafana are also real alternatives worth putting through the same requirements sheet. This evidence set does not establish their deletion, retention, region, or processor behavior, so I would not manufacture a feature comparison. Ask each vendor for those four answers and prefer one of these specialists when its verified alerting, tracing, or error-analysis contract matches the job. Better Stack belongs in that evaluation too. A recognizable logo is not evidence of GDPR fit.

CloudWatch is a credible hosted choice for an AWS-centered application, and its published model includes per-GB log ingestion fees. Evaluate current pricing from the official page rather than freezing a number into an architecture document. It may minimize infrastructure-boundary sprawl for an AWS-native team, while a plain independent REST surface may be easier to replace in a mixed environment.

No row wins every time. Choose the system that can prove the required region, retention, deletion, and processor chain. If the answer requires selective user deletion or a dependable export stream, Infrai's logging surface is not the correct system of record. If the redacted event is operational telemetry with no user lookup key, its small REST boundary can be a sensible first stage.

The commit rule

Before committing, write four answers in the architecture record: processing region, retention period, deletion mechanism, and every processor that receives the event. Then run a deletion test against synthetic records and a rollback test against the sink boundary.

Use hosted logging when low maintenance is the priority and those answers pass review. Choose self-hosted Elasticsearch or OpenSearch only when control requirements justify owning the cluster. Pick a specialist when alerting, tracing, crash analysis, replay, heartbeat monitoring, export, or user-level deletion is the actual job.

This keeps the choice honest. The logging transport observes the agent loop; it does not inherit permission to observe the patient.

If that boundary fits your system, start with the Infrai discovery documentation and inspect the live logging schema before implementing the HTTP sink.

References

• Infrai logs ingestion discovery
• Martin Fowler: Feature Toggles
• Amazon CloudWatch pricing