I Built an AI-Agent Guardrail, Then Discovered My Observer Was Watching Itself

Reactive AI cost monitoring has an awkward limitation: by the time a chart shows a runaway agent, the tokens are already gone. I wanted to test a different idea for the Agents of SigNoz hackathon: make a small execution contract before an AI coding agent starts work, then measure whether its actual behavior stays inside that contract.

That became Preflight. Given a coding task, it predicts a model tier, effort level, and expected file footprint. Those predictions become observable expectations: when the agent writes files, Preflight measures scope drift, emits OpenTelemetry, and surfaces the result in SigNoz. It is deliberately a prototype, not a universal security sandbox. The useful part is that the prediction, the observed actions, and the misses are all inspectable.

The most valuable lesson did not come from the happy path. It came from discovering that my external observer was counting its own tooling metadata as agent work.

The contract comes before the agent

A task such as 'rename one local variable' should not automatically receive the same model budget or repository-wide scope as 'introduce tenant-aware authorization across billing, reporting, migrations, tests, and rollout documentation.' Preflight classifies the task first and creates a compact contract:

I use a cheap gpt-5.6-luna structured classification call for that small JSON decision, capped at 250 completion tokens. This is intentional: the guard should practice the same 'predict cheap when appropriate' principle it recommends to other agents.

Preflight then has two honest runtime modes:

• In-process interception evaluates actions from Preflight's built-in agent while it runs. When drift gets high, it can pause and propose a human-confirmed remediation.
• External watch predicts before launching another coding CLI, recursively observes writes and deletes in a selected workspace, then calculates drift after the process exits. This mode is post-hoc; it cannot preempt the child process.

That distinction matters. A filesystem watcher cannot prove semantic safety, see short-lived files, or protect paths outside the watched workspace. Calling those modes equivalent would make the demo look stronger than it is.

What SigNoz sees

I exported stable OpenTelemetry span names instead of putting task text or model names into span names:

preflight.predict
  -> preflight.execute
    -> preflight.action.evaluate
    -> preflight.drift.check
    -> preflight.signal      (only when something is wrong)
    -> preflight.complete    (successful session end)

Attributes carry the changing details: session ID, predicted tier, runtime mode, action target, verdict, and drift score. The metrics cover classifier cost, modeled savings versus an always-large baseline, tier distribution, action count, and drift. 'Modeled savings' is explicitly a comparison assumption, not a billing claim.

I used those signals across several SigNoz features:

• a Preflight dashboard for classification cost, drift, routing tiers, policy verdicts, session cost, and modeled savings;
• three warning alert rules for scope drift, cost spikes, and the hard ceiling;
• a trace funnel with preflight.predict -> preflight.execute -> preflight.drift.check -> preflight.complete;
• Query Builder aggregation to find the tiers actually showing high drift.

The dashboard query was not just a screenshot. In the saved v5 query, I calculated avg(preflight.drift_score), grouped by preflight.predicted_tier, and applied HAVING avg_drift_score > 0.5. In the current data, only the medium tier surfaced, at 0.542 average drift. That is a real query result, not a synthetic metric.

A three-session live batch reached every stable funnel stage: predict, execute, drift check, and complete. The classification cost for that batch was $0.001852. I also hit two practical SigNoz integration details while building it: dashboard group-by needed groupBy.key rather than an older groupBy.name shape, and v5 aggregation aliases must be supplied through the alias field instead of inline SQL-style aliasing. Those are the kinds of details that only show up when a dashboard is actually wired to data.

The bug: my observer was watching itself

For the external-watch proof, I wrapped the installed Codex CLI and asked it to create exactly one Markdown file. The first raw result looked like a success: the watcher saw changed files and drift rose.

But the requested file had not been created.

The culprit was a globally installed Codex plugin named Foresight. It wrote its own .foresight contract and history files inside the temporary workspace. My watcher dutifully counted those internal files as if the external coding agent had edited user code.

That was invalid evidence. I kept the failed attempts in the repository, but I did not call them successful runs.

The fix is intentionally narrow:

_INTERNAL_WATCH_DIRECTORIES = frozenset({".foresight"})

relative_path = path.relative_to(self.workspace)
if any(part in _INTERNAL_WATCH_DIRECTORIES for part in relative_path.parts):
    continue

I also made the proof runner fail closed. A run is accepted only when the external CLI returns zero, the exact requested output file exists, and its contents exactly match the expected text. Every rerun receives a fresh workspace. This is not a broad hidden-file exclusion; it filters one verified plugin metadata directory and leaves ordinary project files observable.

After the correction, the committed result contains two real Codex CLI runs. Each produced exactly one requested user file, each returned exit code 0, and each had one observed file write with 0.33 drift against a three-file contract. The two live Luna predictions used 203 input tokens each, 50 and 54 output tokens, and cost $0.001030 total.

Evidence instead of a perfect-looking score

I wanted the evaluation story to be more useful than a single flattering percentage. The initial 18-task live benchmark was 61.11% accurate on both tier and blast-radius labels. That exposed a real weakness: short requests underweighted authentication, validation, regression coverage, and durable operational work.

I applied one scope-rubric calibration and reran the same corpus. The result was 18/18, or 100%, but I label it clearly as a same-corpus calibrated result rather than independent validation. Then I froze a separate eight-task held-out set before running it. That held-out run was 8/8 for tier and blast radius. It is encouraging, not a universal guarantee.

The repository keeps v1, v2, held-out data, raw token counts, costs, failed external attempts, and successful external results. The interesting engineering outcome is not 'the model never misses.' It is that a miss, a drift event, or a broken observer leaves a trail another person can inspect.

What I would tell my past self

1. Predict before spending. A cheap preflight decision can bound a much more expensive agent run.
2. State runtime guarantees precisely. In-process interception and post-hoc external watch are useful for different reasons.
3. Treat failed evidence as evidence. The Foresight metadata bug made the project stronger only because I kept the bad run, explained it, and tightened the test.
4. Make observability falsifiable. Dashboards are more valuable when they connect to committed raw data, stable spans, and explicit boundaries.

Preflight is my attempt at making agent scope and cost visible before they become cleanup work. The full repository, Foundry deployment files, SigNoz dashboard evidence, and benchmark artifacts are available at AmanM006/Preflight.