I Built an AI Agent That Watches Itself and Heals Itself - Here's How

# signoz# opentelemetry# ai# devops
I Built an AI Agent That Watches Itself and Heals Itself - Here's HowDebjyoti

I've been doing DevOps long enough to know that the scariest production failures are not the ones...

I've been doing DevOps long enough to know that the scariest production failures are not the ones that crash everything. They are the quiet ones. Latency creeping up. Error rates ticking higher. Costs climbing while your dashboards show green because you never instrumented the right things.

AI agents make this worse. They chain LLM calls, invoke tools, query databases, make decisions on their own. When something breaks in that chain there is no obvious stack trace. The failure lives somewhere in the sequence of calls and if you did not instrument it, you are flying blind.

For the Agents of SigNoz hackathon I built Sentinel - a system where one AI agent is fully observed by SigNoz, and a second agent monitors that telemetry and heals the first one automatically when something goes wrong. No human in the loop.

Here is exactly how I built it and what I learned.

The Problem I Was Trying to Solve

Take a simple AI agent: it receives a question, searches for context, calls an LLM, returns an answer. When it works, great. When it breaks, what do you actually see?

Without proper observability: a 500 error in your logs. Maybe a timeout. Nothing about which step failed, how long each step took, how many tokens got burned, whether the failure was in the tool or the LLM.

With OpenTelemetry traces in SigNoz: you see every step as a span in a waterfall. The tool call took 20ms. The LLM call took 2.5 seconds. Token count was 850 in, 180 out. The third retry hit a different model. Everything is there.

SigNoz trace waterfall showing worker-agent spans

That is the observability part. But I wanted to go further. Once you can see the problem, can the system fix itself?

The Architecture

Two agents. One observability platform. One healing loop.

User question
      ↓
WORKER AGENT  (FastAPI :8001)
  → search tool → LLM call → answer
  → every step traced to SigNoz
      ↓
SIGNOZ  (:8080 UI / :4317 collector / :8000 MCP)
  → stores traces in ClickHouse
  → fires alerts when things break
      ↓
SENTINEL AGENT  (FastAPI :8002)
  → receives alert webhook
  → diagnoses with LLM
  → heals Worker via /control endpoint
  → its investigation is ALSO traced in SigNoz
Enter fullscreen mode Exit fullscreen mode

The thing that makes this different from just "instrument your app and make dashboards": Sentinel is itself instrumented. In SigNoz you see two services — worker-agent and sentinel-agent. You can watch the healer being observed. The investigation, the diagnosis, the healing action — all of it shows up as spans.

Setting Up SigNoz with Foundry

I had not used Foundry before this. Turns out it is the right way to deploy SigNoz now.

Before Foundry you had to clone the repo, edit Docker Compose files by hand, run bash scripts that worked on some machines and not others. Not great for a project where reproducibility matters.

With Foundry I wrote twelve lines of YAML:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    mode: docker
    flavor: compose
  mcp:
    spec:
      enabled: true
Enter fullscreen mode Exit fullscreen mode

Then:

foundryctl gauge -f casting.yaml   # validates your environment
foundryctl cast -f casting.yaml    # deploys everything
Enter fullscreen mode Exit fullscreen mode

Two commands. Two minutes. SigNoz running at localhost:8080 with six containers — ClickHouse for storage, the OTel Collector on port 4317 receiving spans, the web UI, and the MCP server on port 8000.

That enabled: true under mcp is what exposes your telemetry to AI agents. One line turns SigNoz into a data source that Sentinel can query.

SigNoz home page showing traces ingestion is active

Foundry also generates casting.yaml.lock — a lock file that pins the exact resolved configuration. Anyone cloning my repo runs foundryctl cast and gets the identical stack. For a hackathon where judges re-run your deployment, this is not optional.

Instrumenting the Worker

Auto-instrumentation handles the HTTP layer with zero code:

FastAPIInstrumentor.instrument_app(app)
Enter fullscreen mode Exit fullscreen mode

Every request to /ask now gets a root span automatically. HTTP method, URL, status code — all captured without me writing anything.

The interesting part is the manual spans for business logic that auto-instrumentation cannot see:

with tracer.start_as_current_span("agent.llm.call") as span:
    span.set_attribute("gen_ai.system", "groq")
    span.set_attribute("gen_ai.request.model", model)

    resp = groq_client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}]
    )

    span.set_attribute("gen_ai.usage.input_tokens", resp.usage.prompt_tokens)
    span.set_attribute("gen_ai.usage.output_tokens", resp.usage.completion_tokens)
    span.set_attribute("gen_ai.response.finish_reason", resp.choices[0].finish_reason)
Enter fullscreen mode Exit fullscreen mode

Those gen_ai.* names follow the OpenTelemetry GenAI semantic conventions. SigNoz reads them natively — token counts appear in traces without any custom configuration. Get the names right from the start and the tooling just works.

One thing I got wrong early: I assumed raising an exception would automatically mark the span as ERROR. It does not. You have to do it yourself:

if _state["flaky_api"] and random.random() < 0.4:
    span.set_status(trace.StatusCode.ERROR, "LLM timeout (injected)")
    span.set_attribute("error.type", "injected_timeout")
    raise HTTPException(500, "LLM API timeout")
Enter fullscreen mode Exit fullscreen mode

Without that set_status call, the span shows as OK in SigNoz even when a 500 was returned. The alert rule counting error spans never fires. I only caught this because I was actually looking at the trace attributes — another reason to instrument deeply.

What the Traces Actually Show

After sending some traffic, the waterfall in SigNoz:

GET /ask                    2.52s
  agent.request             2.52s
    agent.tool.search       0.02ms
    agent.llm.call          2.50s
      gen_ai.system = groq
      gen_ai.request.model = llama-3.1-8b-instant
      gen_ai.usage.input_tokens = 850
      gen_ai.usage.output_tokens = 180
      gen_ai.response.finish_reason = stop
Enter fullscreen mode Exit fullscreen mode

The LLM call is 2.5 seconds out of 2.52 total. The tool call is 20ms. If you want to reduce latency you focus on the LLM, not the tooling. That is immediately actionable. Without this trace you would be guessing.

Worker agent GET /ask trace waterfall with agent.llm.call spans

Building the Self-Healing Loop

The Worker has a /control endpoint that changes its behavior at runtime:

@app.post("/control")
def control(action: dict):
    action_type = action.get("type", "")

    if action_type == "inject_flaky":
        _state["flaky_api"] = True    # 40% of LLM calls fail

    elif action_type == "reset":
        _state["flaky_api"] = False   # back to normal
        _state["slow_tool"] = False

    elif action_type == "switch_model":
        _state["model"] = action.get("model", "llama-3.1-8b-instant")

    return {"ok": True, "state": dict(_state)}
Enter fullscreen mode Exit fullscreen mode

Sentinel calls this endpoint to apply healing. The state change is instant — no restart needed.

When a SigNoz alert fires, Sentinel receives a webhook POST and runs this:

diagnosis = investigate(alert_name, payload)   # LLM call
action = PLAYBOOK.get(alert_name, {"type": "log_only"})  # deterministic lookup
requests.post(f"{WORKER_URL}/control", json=action)  # heal
Enter fullscreen mode Exit fullscreen mode

The playbook is intentionally simple:

PLAYBOOK = {
    "HighErrorRateWorker": {"type": "reset"},
    "HighLatencyWorker":   {"type": "switch_model", "model": "llama-3.1-8b-instant"},
}
Enter fullscreen mode Exit fullscreen mode

The LLM diagnoses. The playbook acts. The LLM is not trusted to invent healing actions — that would be unpredictable in a real system. It provides human-readable diagnosis; the code provides the deterministic response. This was an intentional design decision.

The Docker Networking Problem

Setting up the SigNoz alert webhook to point at Sentinel kept giving me connection refused. SigNoz on localhost:8080, Sentinel on localhost:8002 — should work, right?

Wrong. SigNoz runs inside Docker containers. Inside a container, localhost means the container itself, not your Mac. Took me longer to figure this out than I would like to admit.

The fix is Docker's special DNS name for the host machine:

http://host.docker.internal:8002/webhook
Enter fullscreen mode Exit fullscreen mode

From inside any Docker container, this resolves to the host machine's IP. Once I updated the notification channel URL to use this, the test notification went through immediately.

SigNoz notification channel test success message

The Dashboards

Four panels in the Sentinel Operations dashboard, all built with SigNoz Query Builder:

  • Request Rate — requests per minute over time
  • p99 Latency — 99th percentile latency in milliseconds
  • Error Rate — percentage of requests failing
  • LLM Calls — count of agent.llm.call spans per minute

Sentinel Operations dashboard with 4 panels

When you inject a failure, the Error Rate panel climbs visibly. When Sentinel heals, it drops back to zero. That visual is more convincing than any explanation.

Watching It Actually Work

The full demo sequence:

# 1. Inject failure
curl -X POST localhost:8001/control \
  -H "Content-Type: application/json" \
  -d '{"type": "inject_flaky"}'

# 2. Send traffic — errors appear in SigNoz
for i in {1..15}; do
  curl -s "localhost:8001/ask?q=test" > /dev/null
  sleep 0.5
done

# 3. Trigger Sentinel
curl -X POST localhost:8002/webhook \
  -H "Content-Type: application/json" \
  -d '{"alertname": "HighErrorRateWorker", "severity": "critical"}'
Enter fullscreen mode Exit fullscreen mode

Sentinel's response:

{
  "alert": "HighErrorRateWorker",
  "diagnosis": "High error rates indicate LLM API failures...",
  "action": {"type": "reset"},
  "outcome": {"ok": true, "worker": {"flaky_api": false}},
  "report": "--- incident report ---\nalert: HighErrorRateWorker\naction: reset\noutcome: healed"
}
Enter fullscreen mode Exit fullscreen mode

Error rate drops to zero. System healed. From failure injection to recovery: under 30 seconds. No human touched anything after the initial injection.

Sentinel healing trace with 9 spans

The Part That Made It Click

When I opened the Traces explorer and filtered for sentinel-agent, I saw Sentinel's investigation as a full trace — 9 spans covering every step of the healing process. The investigation took 3.7 seconds. The actual healing took 20ms. The LLM diagnosis was the bottleneck, not the fix itself.

That is the detail I did not expect. You can watch the healer work. You can see how long diagnosis takes, whether it succeeded, what action was applied. If Sentinel fails to heal something, that shows up as an ERROR span — which could itself trigger another alert.

The healer is observed. That changes how you think about operational tooling.

SigNoz services tab showing worker-agent and sentinel-agent

What I Would Do Differently

The tool call is mocked — it returns a canned search result. A real version would hit an actual vector database, and that is where you would see the most interesting traces. Retrieval failures and cache misses are a huge source of agent degradation that almost nobody instruments properly.

I would also add cooldown periods to Sentinel. Right now if an alert keeps firing, Sentinel keeps responding. In production you need deduplication — if the same alert fires three times in two minutes, you do not want three simultaneous healing attempts.

And tail sampling in the OTel Collector. At real scale you do not want every trace stored — you want all errors, all slow requests, and maybe 1% of normal ones. The architecture supports this without code changes; it is a Collector configuration.

What I Actually Learned

Foundry is the right way to deploy SigNoz now. The lock file approach means your deployment is a versioned artifact you can put in git, share with teammates, and have judges reproduce exactly. Treat your observability infrastructure as code the same way you treat your application code.

GenAI semantic conventions matter from day one. Name your attributes correctly and every OTel-compatible tool understands your LLM data without custom plugins. gen_ai.usage.input_tokens is not just a name — it is a contract with the tooling ecosystem.

Instrument the healer, not just the system being healed. If your operational tooling fails silently, you have the same problem you were trying to solve. Sentinel being fully traced means I have the same visibility into the healing system as I do into the Worker.

The docker networking gotcha will get you. localhost inside Docker is not your machine. host.docker.internal is. Write it on a sticky note.

Links

AI tools used: I used Groq's llama-3.1-8b-instant as the LLM for both agents, and Claude as a coding assistant during development. Declared per hackathon rules.