The Metric I Didn't Know I Needed: Debugging Silent Failures in My Chess Coach Agent

My AI chess coach was dying quietly. No crash, no error in the logs — the frontend would just sit there waiting, the user assuming it was "thinking," until it eventually timed out. Nothing in my dashboard told me anything was wrong. That gap is what I actually learned building ChessWiz for the WeMakeDevs x SigNoz "Agents of SigNoz" hackathon.

The setup

ChessWiz has two halves. The engine — C++, negamax with alpha-beta pruning, Zobrist-hashed transposition table, quiescence search — is deterministic and easy to reason about. The Coach Agent, which calls Groq's LLaMA 3.3 70B to explain moves, behaves like any network call to someone else's GPU: fast most of the time, unpredictable the rest.

I fed both into a self-hosted SigNoz stack (ClickHouse + OTel Collector, Docker Compose). The engine got 7 metrics almost immediately, because I already knew what "healthy" looked like. The Coach Agent got 3 — added reactively, one gap at a time.

Where it broke

Early on, the Coach Agent's only signal was "did the Groq call return 200." During testing, a request to the LLaMA endpoint hung well past a normal response window — and from the frontend's side, nothing looked wrong. No error, no log line to grep for, no panel that moved. It just never came back.

The real problem wasn't that the agent was slow — I'd already accepted LLM calls are slower and spikier than a local search. It was that a stalled call looked identical to a slow-but-working one. I couldn't tell "Groq is just having a moment" apart from "this request is never returning."

The fix

I added coach_agent_call_status, a counter labeled success / slow / timeout, wrapped around a span with an explicit 15-second boundary:

js
const span = tracer.startSpan("coach_agent_llm_call");
const timeout = new Promise((_, reject) =>
setTimeout(() => reject(new Error("timeout")), 15000)
);

try {
const result = await Promise.race([callGroq(prompt), timeout]);
meter.createCounter("coach_agent_call_status").add(1, { status: "success" });
span.setStatus({ code: SpanStatusCode.OK });
return result;
} catch (err) {
const status = err.message === "timeout" ? "timeout" : "error";
meter.createCounter("coach_agent_call_status").add(1, { status });
span.recordException(err);
} finally {
span.end();
}

Once this shipped, silent failures showed up as a spike in the timeout label on the panel instead of a confused user staring at a spinner.

Noise vs. signal

"LLM call duration" alone was noise-adjacent — it told me things were slow, which I expected. What actually helped was a metric shaped around the specific failure mode I'd watched happen. I also resisted copying the engine's timeout alert thresholds onto the Coach Agent — LLM latency is naturally spikier, so the same threshold would've either spammed false alarms or missed real ones. Different subsystem, different threshold.

Takeaways
You can't design a good metric for a failure you haven't watched happen yet.
"Slow" and "broken" need to look different on the dashboard, or it's lying to you by omission.
Don't copy-paste alert thresholds across subsystems with different latency profiles.
Reactive instrumentation isn't sloppy planning — it's how you find out what you actually needed.
Conclusion

The useful signal wasn't more dashboards — it was one metric shaped exactly around a failure I'd already seen. If you're wiring an LLM into anything, ask early: what does "silently stuck" look like from the outside, and is anything actually watching for it?

Built for the WeMakeDevs x SigNoz "Agents of SigNoz" hackathon.