Observability for Email Agents

You can't watch an email agent work, but everything it did yesterday is one API call away:

curl --request GET \
  --url "https://api.us.nylas.com/v3/grants/<GRANT_ID>/messages?limit=50" \
  --header "Authorization: Bearer <NYLAS_API_KEY>"

That's the strange, underrated property of building agents on email. Most autonomous systems need observability bolted on — tracing, structured logs, replay tooling. An agent that lives in a mailbox gets three observability primitives for free, because the medium is the record. Here's how to use each one, drawn from how Agent Account mailboxes (currently in beta) actually behave.

The event stream: webhooks

Every inbound message fires message.created — typically within seconds of the SMTP handoff — with a payload that includes the thread_id your agent needs to reconstruct conversation state. That's your input-side event stream, no instrumentation required.

One wrinkle to handle: when a message body exceeds roughly 1 MB, the trigger becomes message.created.truncated and the body is omitted from the payload — fetch the full message by ID in that case, or your agent will silently reason over nothing.

The output side is richer than most people expect, because the platform owns the SMTP path end-to-end and reports back on every send:

Pipe those three into whatever metrics system you already run and you have per-message delivery telemetry for an autonomous sender. A climbing send_failed count is your earliest signal that something upstream — a rule, a quota, a reputation problem — is throttling the agent.

Two refinements. If your workflow is batch rather than real-time, you don't need webhooks at all for the input side — GET /messages with received_after polls fine; webhooks earn their keep in near-real-time agent loops. And aggregate the outbound triggers per domain, not just per grant: sender reputation is shared across every Agent Account on a given domain, so one misbehaving agent's bounce rate quietly degrades its siblings' deliverability. Fleet observability is a domain-level concern wearing per-account clothes.

State you can read: folders

An agent mailbox comes with six system folders — inbox, sent, drafts, trash, junk, archive — and they double as a state machine you can inspect. junk shows you what spam filtering and mark_as_spam rules decided to divert; if real customer mail is landing there, you'll see it by listing one folder. Custom folders extend the pattern: rules that route invoices or VIP senders into named folders turn "what kind of mail is the agent getting?" into a folder-counts query.

The drafts folder earns special mention in human-in-the-loop designs. If your agent proposes replies as drafts and a reviewer approves them, the drafts folder is your approval queue — its count is your queue depth, and a draft that's been sitting there for hours is a stalled approval you can detect with a folder listing.

The governance layer is observable too. Every rule that fires on inbound mail is logged as a rule evaluation you can fetch afterward — so "why did the agent never see that message?" has a queryable answer (a block rule rejected it at the SMTP layer) instead of a shrug.

The audit log: sent mail

Here's the primitive that ordinary agent architectures genuinely lack. Every action this agent takes in the world is an email, and every email it sends is preserved in sent — addressed, timestamped, threaded to its context. The audit log can't drift from reality because it is reality.

Threading makes the log legible. Replies group by standard RFC 5322 headers, so reviewing an incident means fetching one thread and reading the whole exchange in order — what came in, what the agent said, what came back.

And because IMAP access exposes the identical mailbox the API sees, a non-engineer can audit the agent by opening it in Outlook or Apple Mail. There's no separation between protocol traffic and API traffic: one mailbox, one record, two ways to read it. Try giving your compliance team that kind of access to your LLM's tool-call logs.

A Tuesday-morning incident, walked through

Here's how the three primitives compose under pressure. Tuesday, 9:40 a.m.: your dashboard shows the support agent's reply rate dropped to zero overnight, but inbound volume looks normal. Where do you look?

First, the event stream. message.created events are still arriving, so mail is landing — input is healthy. But message.send_failed started climbing at 11 p.m. The agent has been drafting replies and failing to deliver them for ten hours.

Second, the governance record. A send that fails before it reaches the recipient is typically an outbound rule block or a policy limit, and rule evaluations are logged per grant:

curl --request GET \
  --url "https://api.us.nylas.com/v3/grants/<GRANT_ID>/rule-evaluations" \
  --header "Authorization: Bearer <NYLAS_API_KEY>"

The evaluations show which rule matched and what action it took. In this story, a recently enabled outbound rule is matching more broadly than intended.

Third, the mailbox itself. Fetch the affected threads and read them in order — what came in, what the agent tried to say, where it stopped. Total diagnostic surface: one webhook chart, one API call, one folder read. No log aggregator, no trace sampling, and the postmortem writes itself from artifacts that can't disagree with each other.

The blind spots

Honest limits, so you don't design around capabilities that aren't there. Native open and click tracking — message.opened, message.link_clicked — isn't emitted for messages sent through the API on these accounts, so "did a human read it?" is not an observable event; delivery signals are where your visibility ends. And a send_success only means the recipient's server accepted the message — recipient-side filtering afterward is invisible to you, as it is to every sender on earth.

There's also a subtler gap: the webhook stream tells you what happened, not why the agent chose it. Mailbox observability covers actions; you still own logging the reasoning (prompts, classifications, decisions) that produced each send.

Wire the outbound three first

If you instrument nothing else this sprint, subscribe to message.send_success, message.send_failed, and message.bounce_detected and chart them. Input observability fails loud — the agent stops responding and someone notices. Output observability fails quiet: the agent keeps cheerfully sending into a rising failure rate, and the webhook stream is how you find out in minutes instead of weeks.

What's on your email-agent dashboard today — and if the answer is "nothing yet," which of the three primitives would you wire up first?