I Broke My Own Serverless App on Purpose, Then Asked AWS DevOps Agent What Happened

A hands-on field report: real account, real incident, real investigation output — graded claim by claim against CloudWatch, including the two things the agent got wrong.

Table of contents

1. Why another DevOps Agent post
2. What AWS DevOps Agent actually is
3. The five nouns you need: Agent Space, Topology, Skills, Journal, Goal
4. Pricing you can predict before you switch it on
5. The lab: a serverless order API engineered to fail in a diagnosable way
6. Onboarding in 20 minutes — and the four things that bite
7. Wiring alarms to the agent: two paths, and which one to pick
8. Breaking production on purpose: the incident, minute by minute
9. The evidence chain: what the agent actually has to work with
10. Asking the agent — and the moment it refused to answer
11. The investigation, graded claim by claim
12. The second root cause, and what it proves
13. Reading the agent's mind: the audit journal over the API
14. Fixing it, and what prevention should give you
15. Closing the loop: EventBridge, tickets, Slack, and the handoff to Kiro
16. Making the agent yours: skills, instructions, triggers, custom SRE agents
17. What this cost, and how to tear it down
18. Honest scorecard
19. Appendix: copy-paste checklist

1. Why another DevOps Agent post

Most posts about AWS DevOps Agent follow the same arc: quote the "up to 75% lower MTTR" figure, paste the architecture diagram, show a screenshot of a root cause analysis, conclude that on-call is solved.

The exceptions are worth your time. The AWS Networking & Content Delivery and Security blogs have genuinely good walkthroughs that deploy a broken workload, wire CloudWatch alarms into the agent, and step through real scenarios. I reference them in §7 rather than pretending I found that path first.

But they're all written from the outside in: here is the feature, here is it working. This post is written from the inside out — the questions you'll actually have at 09:00 on the Monday you're asked to evaluate it:

• What breaks during onboarding?
• My alerts are CloudWatch alarms. How do those reach the agent? (There's no native trigger. AWS ships samples for two different bridges, and they are not equivalent — §7.)
• How do I tell whether the thing is working, or just sitting there?
• What does it cost per incident, in dollars?
• When its root cause analysis is wrong, how would I know?

That last one is the reason this post exists. I built an application designed to fail with more than one plausible cause, broke it, then handed the agent the aftermath and checked every single number it produced against CloudWatch. It got the verdict right and two mechanism details wrong — and one of those two would have sent an engineer looking for a configuration setting that does not exist.

Everything below is from one real AWS account in us-west-2 on 31 July 2026. Every timestamp, capacity figure and error count is copied out of CloudWatch or the agent's own output. Nothing is reconstructed or illustrative. Account IDs, the Agent Space ID and the API ID are redacted as <account-id> and <agentSpaceId>; everything else is verbatim.

2. What AWS DevOps Agent actually is

AWS DevOps Agent went GA on 31 March 2026, after a preview announced at re:Invent 2025. It's one of the AWS "frontier agents" — the family that also includes Kiro and the security testing agent — and it's built on Amazon Bedrock AgentCore, with dedicated infrastructure for memory, policies, evaluations and observability, rather than being a thin wrapper over a model.

It splits into two halves.

Production operations (GA). This is the part worth your attention today.

Release management (preview, us-east-1 only). Release readiness review and autonomous release testing — it reviews a change for blast radius and permission expansion before merge, then generates change-specific tests rather than running a static suite. Free during preview. Genuinely interesting, but evaluate it separately: different region, different maturity.

Where it runs. Eleven regions at the time of writing: us-east-1, us-west-2, ca-central-1, sa-east-1, ap-south-1, ap-southeast-1, ap-southeast-2, ap-northeast-1, eu-central-1, eu-west-1, eu-west-2.

The bit people miss: an Agent Space monitors resources in any region of an associated account, regardless of where the Agent Space itself lives. You pick the Agent Space region for data residency and team proximity, not to match your workloads. You do not need one per region.

How it differs from things you already have

• vs. Amazon DevOps Guru — DevOps Guru is ML anomaly detection: it learns normal operating bounds from metrics, logs, events and traces, then flags deviations and aggregates them into insights. DevOps Agent runs an investigation: it forms hypotheses, queries logs, code and deployment history to test them, and writes a conclusion with a mitigation plan. Detection vs. diagnosis. Different jobs, and the overlap is smaller than the names suggest.
• vs. your coding agent + a CloudWatch MCP server — the honest comparison, and the answer is context and governance. Your coding agent can query CloudWatch if you wire it up. What it can't do is hold a continuously-refreshed cross-account topology, share it with your whole team without per-engineer setup, enforce IAM boundaries on what the agent may touch, or keep an immutable audit journal of every reasoning step. §12 is the concrete example of why that matters.
• vs. an SRE — not a replacement. A very fast first responder that never sleeps, and that you still have to check. §11.

3. The five nouns you need

Skip these and the docs will feel like fog.

Agent Space — a logical container defining what the agent can see: which AWS accounts, which third-party tools, which MCP servers, which users. Isolation is real: AWS-account isolation via dedicated IAM roles, user access isolation, and data isolation — investigations, chat history and recommendations do not leak between spaces. Use separate spaces for prod vs non-prod, or per business unit.

Topology — the auto-discovered map of your resources and how they relate. A background learning agent scans infrastructure, telemetry and code to infer application and service boundaries. In my account it correctly grouped resources into applications purely from CloudFormation stack boundaries, and explicitly discarded CDKToolkit and aws-sam-cli-managed-default as scaffolding. That's the difference between "I can query CloudWatch" and "I know what this system is".

Skills — modular Markdown instruction sets, an implementation of the open Agent Skills spec (non-executable files only: Markdown, PDFs, images, data). Three tiers:

• AWS-provided — built in, maintained by AWS
• User-defined — yours; runbooks, standards, correlation rules, skip rules
• Learned — the agent writes these itself from your environment and past investigations

Each skill has frontmatter with name and description, and the description is load-bearing: the agent decides whether to activate the skill by reading it. "RDS skill" gets ignored. "Use this skill when investigating database latency, connection errors, or query timeouts for Amazon RDS instances" does not.

Journal — the immutable, append-only record of every message, tool call and reasoning step in an execution. The agent cannot rewrite it. This is your audit trail and, as §13 shows, your only real debugging tool.

Goal — a scheduled objective. Every new space gets one by default: an ONCALL_REPORT goal on rate(7 days) producing weekly post-incident analysis with prevention recommendations.

4. Pricing you can predict before you switch it on

One number to remember: $0.0083 per agent-second, billed per second, for all three categories (investigations, evaluations, on-demand SRE tasks). Nothing for idle time. No commitment.

That's $0.50 per agent-minute, which makes mental math easy:

A small team doing 10 investigations a month: ~$40. An active team at 80 investigations plus 100 chats: ~$344. Enterprise scale in AWS's own example: ~$2,366.

Free money, in order of how much you should care:

1. AWS Support credits. On a paid support plan you get monthly DevOps Agent credits as a percentage of the prior month's support charge: 100% for Unified Operations, 75% for Enterprise, 30% for Business Support+. Credits are issued by the 10th, apply within that month, and expire at month end. For a lot of Enterprise Support customers this makes DevOps Agent effectively free. Check this before building a business case from scratch.
2. Two-month free trial for new customers, starting at your first operational task: up to 10 agent spaces, 20h investigations, 15h evaluations, 20h on-demand tasks per trial month.
3. Included in the AWS Free Tier free plan for new AWS customers.
4. Release management is free during preview.

The cost that isn't on the pricing page: the agent pays your CloudWatch bill by proxy. Every Logs Insights query and trace retrieval it runs is billed by CloudWatch at standard rates. On a chatty log group this is not nothing. Budget for it.

Check your own consumption any time:

aws devops-agent get-account-usage --region us-west-2

{
  "monthlyAccountInvestigationHours":   { "limit": -1, "usage": 0.0 },
  "monthlyAccountEvaluationHours":      { "limit": -1, "usage": 0.0 },
  "monthlyAccountSystemLearningHours":  { "limit": -1, "usage": 0.0 },
  "monthlyAccountOnDemandHours":        { "limit": -1, "usage": 0.0 },
  "usagePeriodStartTime": "2026-07-01T00:00:00+00:00"
}

limit: -1 means no cap configured. Note the fourth category: system learning hours are metered separately, and the background learning agent runs whether or not you've had an incident.

What this post actually cost, measured

Here is the real meter after the chat session in §10 and the investigation in §11:

Two things in there surprised me, and they're both worth planning around.

The conversation cost 3.3× more than the investigation. Every pricing example — AWS's included — is built around investigations as the billable unit. In practice the investigation ran for 3.2 minutes, well under the 8-minute figure the pricing page uses, while the handful of chat turns that led up to it burned 10.5 minutes of agent time. Chat feels free because each reply is quick; it isn't, and it accumulates silently. If you roll this out to a team, the line item that surprises you will be monthlyAccountOnDemandHours, not investigations.

Investigations may be cheaper than you budget for. At 3.2 minutes this one cost less than a coffee, and it correlated four services and reconstructed a two-cause failure chain (§11). If cost is the thing making you hesitate on investigations, the number is probably smaller than your instinct says.

One caveat on reading your own meter: usage resets on the calendar month, and usagePeriodStartTime tells you which period you're looking at. My incident ran late on 31 July UTC and the investigation landed on 1 August, so the two showed up in different billing periods — easy to misread as "nothing was billed" if you only check one.

5. The lab

To evaluate an investigation agent you need an incident with more than one plausible cause. A Lambda that throws KeyError on line 12 proves nothing — any log search finds it. I wanted a failure where the symptom and the cause sit in different services.

Architecture

              POST /orders
                   │
        ┌──────────▼──────────┐
        │  API Gateway HTTP   │   access logs → CloudWatch
        │   API (prod stage)  │
        └──────────┬──────────┘
                   │
        ┌──────────▼──────────┐
        │  Lambda             │   X-Ray tracing on
        │  devops-lab-orders  │   structured JSON logs
        │  arm64, py3.13      │   env: WRITE_FANOUT
        └──────────┬──────────┘
                   │
        ┌──────────▼──────────┐
        │  DynamoDB           │   PROVISIONED
        │  devops-lab-orders  │   1 RCU / 1 WCU   ← the trap
        └─────────────────────┘

  4 CloudWatch alarms ──► SNS ──► Lambda bridge ──► DevOps Agent webhook
  EventBridge (source: aws.aidevops) ──► CloudWatch Logs

The whole trick, in two parameters

WriteFanout:          # items written per API request. 1 = healthy.
  Default: '1'
TableWriteCapacity:   # provisioned WCU. Deliberately tiny.
  Default: 1

The Lambda branches on WRITE_FANOUT:

if WRITE_FANOUT <= 1:
    _table.put_item(Item=_order_item(order_id))          # healthy
    return _response(201, {"orderId": order_id, "writes": 1})

# the "optimisation" that broke production
with _table.batch_writer() as batch:
    for line_no in range(WRITE_FANOUT):
        batch.put_item(Item=_order_item(f"{order_id}#{line_no}", line_no))

Three details make this behave like a real incident rather than a toy.

Padded items. Each item carries a 900-byte notes field so it costs a full write capacity unit. Without padding, DynamoDB's per-item rounding hides the problem.

A deliberately short retry budget.

_ddb = boto3.resource("dynamodb",
    config=Config(retries={"max_attempts": 2, "mode": "standard"}))

The boto3 default of ~10 adaptive retries would absorb the throttling and turn a hard failure into mysterious latency. Two attempts means throttling surfaces as a real error, a real Lambda invocation failure, and a real API 5xx.

One BatchWriteItem, not 25 calls. batch_writer() buffers and flushes in batches of 25, which is the BatchWriteItem API maximum. With WRITE_FANOUT=25 each invocation issues exactly one request carrying 25 items. Remember this detail — it becomes the cleanest proof in §11.

Everything is in one SAM template with structured JSON logging, X-Ray on, 7-day log retention, and four alarms whose AlarmDescription fields contain actual runbook text. That last point matters more than it looks; see §7.

6. Onboarding in 20 minutes

Four supported paths: Console, AWS CLI, CloudFormation/CDK (AWS::DevOpsAgent::AgentSpace + AWS::DevOpsAgent::Association), and Terraform (awscc_devopsagent_agent_space). If you have a choice, use IaC — this is production access configuration and it belongs in version control.

The CLI path, condensed:

REGION=us-west-2
ACCOUNT=$(aws sts get-caller-identity --query Account --output text)

# 1. role the service assumes to read your account
cat > trust.json <<EOF
{"Version":"2012-10-17","Statement":[{
  "Effect":"Allow",
  "Principal":{"Service":"aidevops.amazonaws.com"},
  "Action":"sts:AssumeRole",
  "Condition":{
    "StringEquals":{"aws:SourceAccount":"$ACCOUNT"},
    "ArnLike":{"aws:SourceArn":"arn:aws:aidevops:$REGION:$ACCOUNT:agentspace/*"}}}]}
EOF

aws iam create-role --role-name DevOpsAgentRole-AgentSpace \
  --assume-role-policy-document file://trust.json
aws iam attach-role-policy --role-name DevOpsAgentRole-AgentSpace \
  --policy-arn arn:aws:iam::aws:policy/AIDevOpsAgentAccessPolicy

# 2. role your humans use in the web app
#    (same trust policy, plus sts:TagSession)
aws iam attach-role-policy --role-name DevOpsAgentRole-WebappAdmin \
  --policy-arn arn:aws:iam::aws:policy/AIDevOpsOperatorAppAccessPolicy

# 3. the space
aws devops-agent create-agent-space --name "prod-ops" --region $REGION

# 4. let it discover the account
aws devops-agent associate-service --agent-space-id $SPACE --service-id aws \
  --configuration "{\"aws\":{\"assumableRoleArn\":\"arn:aws:iam::$ACCOUNT:role/DevOpsAgentRole-AgentSpace\",\"accountId\":\"$ACCOUNT\",\"accountType\":\"monitor\"}}" \
  --region $REGION

# 5. turn on the web app
aws devops-agent enable-operator-app --agent-space-id $SPACE --auth-flow iam \
  --operator-app-role-arn "arn:aws:iam::$ACCOUNT:role/DevOpsAgentRole-WebappAdmin" \
  --region $REGION

The four things that bite

1. Your AWS CLI is probably too old. The first wall:

$ aws --version
aws-cli/2.32.22 Python/3.13.11 Windows/11 exe/AMD64

$ aws devops-agent help
aws: [ERROR]: argument command: Found invalid choice 'devops-agent'

winget upgrade --id Amazon.AWSCLI --accept-source-agreements
# or download https://awscli.amazonaws.com/AWSCLIV2.msi

The CLI and the SDK move independently, and that bites in both directions. On the same machine, at the same moment: the CLI rejects devops-agent outright, while boto3 1.43.6 / botocore 1.43.7 has a working devops-agent client. So every aws devops-agent … command in this post has a two-line Python equivalent that works before you upgrade anything:

import boto3
boto3.client("devops-agent", region_name="us-west-2").get_account_usage()

The subtler version of the same problem: that boto3 build has get_operator_app and it returns HTTP 200 — but its response model is missing the operatorAppUrl field a newer botocore returns. You get a successful call with silently absent data, no error. If a field the docs promise isn't in your response, upgrade before you debug anything else.

(If you're stuck: the web app URL is deterministic — https://<agentSpaceId>.aidevops.global.app.aws.)

2. iam:CreateServiceLinkedRole is not in the managed policy. Topology discovery uses AWS Resource Explorer, and the agent role needs to create Resource Explorer's service-linked role. AIDevOpsAgentAccessPolicy doesn't grant it:

{"Version":"2012-10-17","Statement":[{
  "Sid":"AllowCreateServiceLinkedRole",
  "Effect":"Allow",
  "Action":["iam:CreateServiceLinkedRole"],
  "Resource":["arn:aws:iam::ACCOUNT:role/aws-service-role/resource-explorer-2.amazonaws.com/AWSServiceRoleForResourceExplorer"]}]}

Skip it and discovery quietly under-performs.

3. The IAM auth link into the web app expires in 10 minutes. Fine for a first look, useless for an on-call rotation. Wire up IAM Identity Center (or a direct OIDC IdP — Okta, Entra ID) before handing this to a team.

4. Two consoles, and the split is not intuitive. Admins configure Agent Spaces in the AWS Management Console. Operators do all actual work — investigations, chat, topology, recommendations — in a separate web app on its own domain, outside the AWS console. If you're hunting for the investigation UI in the AWS console, you won't find it.

Confirm it's actually alive

Onboarding "succeeding" and the agent working are different things. I wrote a status script (lab/scripts/agent_status.py) that pulls the whole picture in one go. Its real output against my space, before I did anything:

=== account agent-hours this month (us-west-2) ===
  investigations : 0.00h (no cap)
  evaluations    : 0.00h (no cap)
  learning       : 0.00h (no cap)
  on-demand/chat : 0.00h (no cap)

=== my-first-devops-agent  (<agentSpaceId>) ===
  web app     : https://<agentSpaceId>.aidevops.global.app.aws
  association : aws <account-id> monitor -> valid
    webhooks  : none  <-- nothing can page this agent
  tasks       : 1
    [FAILED   ] SYSTEM_LEARNING  Create Agent Space Understanding Skills  <-- FAILED
  prevention recommendations : 0
  goal        : ONCALL_REPORT [ACTIVE] rate(7 days) (ENABLED)

Read that carefully, because it's the honest state of a "successfully onboarded" Agent Space:

• association valid ✓
• zero webhooks — nothing in the world could trigger an investigation
• the initial learning task FAILED three weeks earlier and nothing told me
• zero recommendations, zero agent-hours consumed

This space had been sitting there doing nothing since 6 July. That's less a criticism of the product than a warning about the onboarding experience: there is no green light. Run something like this on a schedule.

7. Wiring alarms to the agent

The user guide lists three ways an investigation starts:

1. Built-in integrations — a ServiceNow ticket, a Dynatrace problem, a Datadog event, a Splunk or New Relic alert, a PagerDuty incident
2. Webhooks — generic HTTP POST, HMAC or bearer-token authenticated
3. A human clicking Start investigation in the web app

A CloudWatch alarm is not on that list, and neither is SNS. There's no alarm action, no built-in CloudWatch integration, no EventBridge-as-input. If CloudWatch alarms are your alerting — which for most AWS-native serverless shops they are — you have to build glue.

I want to be careful here, because when I first drafted this section I got it wrong twice, and both errors are instructive.

Correction 1: this is well documented, just not in the user guide. AWS has published this pattern at least four times — the Networking & Content Delivery blog's automated network incident response post (with a deployable CloudFormation template on aws-samples), the Security blog's Network Firewall troubleshooting post, and two repost.aws articles. One of those repost articles opens by noting CloudWatch Alarms are the most commonly requested integration. So: a real gap in the product, thoroughly covered in the blogs. Search them before you write anything — the samples are good and will save you an afternoon.

Correction 2: there is an API path, and I missed it. I originally wrote that no StartInvestigation API exists, which is true of the name and false in substance:

aws devops-agent create-backlog-task \
  --agent-space-id $SPACE \
  --task-type INVESTIGATION \
  --priority HIGH \
  --title "devops-lab-api-5xx in ALARM" \
  --description "$ENRICHED_CONTEXT" \
  --client-token "cw-devops-lab-api-5xx-2026-07-31T16:56:30" \
  --region us-west-2

--task-type accepts INVESTIGATION, EVALUATION, RELEASE_READINESS_REVIEW and RELEASE_TESTING. create-backlog-task was sitting in the command surface I'd already enumerated and I read it as ops-backlog bookkeeping rather than an investigation entrypoint. It's the mechanism behind the repost.aws pattern that bridges Incident Detection and Response alarms via EventBridge → Lambda → CreateBacklogTask.

That matters, because it means there is a path with no webhook, no shared secret, and no console step.

The two paths, side by side

Which to pick. If you want the whole thing in version control with no click-ops and no secret rotation, Path B. If you're already receiving alerts from Datadog or PagerDuty and want CloudWatch alarms arriving through the same door in the same shape, Path A. Path A is also the better-trodden route — it's what the aws-samples templates use.

One thing I have not verified, and neither AWS post addresses: whether a {% raw %}CreateBacklogTask investigation passes through the same triage stage that webhook-delivered incidents do — the stage that decides LINKED / SKIPPED / PROCEED and dedupes related alarms inside a ~20-minute window (§8). That behaviour is documented for incoming incidents, not for API-created tasks. If you depend on triage correlation to avoid paying for three investigations of one outage, test this before committing to Path B. I'd rather flag the unknown than guess.

Path A, and the parts the samples leave to you

I built Path A, because I wanted the same payload shape a Datadog or PagerDuty alert would arrive in. It's about 40 lines.

Step 1 — generate a generic webhook. This step genuinely has no API. In the console: your Agent Space → Capabilities → Webhook → Configure → Generate webhook, and choose HMAC or API key.

Choose HMAC, and know why:

Store the secret in Secrets Manager. You cannot retrieve it again.

Step 2 — SNS → Lambda → webhook. Signing is stdlib-only:

timestamp = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.000Z")
signature = base64.b64encode(
    hmac.new(secret.encode(),
             f"{timestamp}:{body.decode()}".encode(),
             hashlib.sha256).digest()
).decode()

headers = {
    "Content-Type": "application/json",
    "x-amzn-event-timestamp": timestamp,
    "x-amzn-event-signature": signature,
}

The signed string is timestamp:payload, not the payload alone. Get that wrong and you get a 4xx with no useful detail.

Step 3 — translate properly. Getting a 200 back is the easy part, and it's where the published samples stop. Translation quality is what separates a useful investigation from four paid minutes of the agent rediscovering your account layout. The schema:

{
  "eventType": "incident",
  "incidentId": "string",                  // dedupe key
  "action": "created|updated|closed|resolved",
  "priority": "CRITICAL|HIGH|MEDIUM|LOW|MINIMAL",
  "title": "string",
  "description": "string",                 // the agent reads this first
  "timestamp": "2026-07-31T16:56:31.000Z",
  "service": "string",
  "data": { }                              // raw upstream event
}

Three things a naive JSON.stringify(alarm) gets wrong:

CloudWatch alarms have no severity. The triage stage uses priority to decide whether to investigate independently, link, or skip. Map it deliberately:

PRIORITY_RULES = (
    ("api-5xx",             "CRITICAL"),   # customers see this
    ("ddb-write-throttled", "HIGH"),
    ("errors",              "HIGH"),
    ("latency",             "MEDIUM"),
)

incidentId must be stable per transition, not per delivery. SNS is at-least-once. Key on alarm name plus state-change time truncated to the second, so redeliveries dedupe instead of spawning duplicate paid investigations:

f"cw-{alarm_name}-{alarm['StateChangeTime'][:19]}"

The description is your prompt. A raw alarm gives the agent a metric name and a threshold. Give it identifiers instead — and this is where those AlarmDescription runbooks pay off:

Threshold Crossed: 1 datapoint [161.0 (31/07/26 16:55:00)] was >= threshold (1.0).

Alarm: devops-lab-ddb-write-throttled
Intent: Orders table is rejecting writes. Customer order submissions are failing.
        Runbook: check ConsumedWriteCapacityUnits against provisioned WCU and
        recent deploys to devops-lab-orders.
Account: <account-id>
Region: US West (Oregon)
Metric: AWS/DynamoDB/ThrottledRequests
Dimensions: {"TableName": "devops-lab-orders", "Operation": "BatchWriteItem"}
Application: devops-agent-lab (serverless order API: API Gateway HTTP API ->
             Lambda devops-lab-orders -> DynamoDB devops-lab-orders)

That's the actual payload my bridge produced during the incident, copied from CloudWatch Logs. Every line is something the agent would otherwise burn paid agent-seconds discovering — or guess wrong about.

Practical note: write your bridge so it works before the webhook exists. Mine takes the webhook URL as a template parameter, and when it's empty it logs the exact payload it would have sent under {"event": "webhook_not_configured", "wouldHaveSent": {...}}. That let me verify the entire pipeline — priority mapping, dedupe keys, description assembly — without spending a cent, and gave me the payloads quoted above. The AWS samples take the equivalent precaution differently: their bridge skips delivery while the secret still holds a placeholder.

One piece of advice in the AWS post I'd push back on. Its "further considerations" section suggests adding correlation logic to the Lambda — buffering alarms in DynamoDB on a 60-second window, grouped by application — to stop one root cause spawning duplicate investigations. The problem is real: my incident fired three alarms 71 seconds apart for one cause. But the agent already has a triage stage that correlates incoming incidents within roughly a 20-minute look-back and marks them LINKED, which is a wider window and a richer comparison than name-matching in your own Lambda. Before you build and operate a DynamoDB dedupe table, send the alarms through unbuffered and watch for Investigation Linked on the EventBridge bus (§15). If triage is already doing the job, you've saved yourself a stateful component in your alerting path — and if it isn't, you'll have the evidence to justify building one.

8. Breaking production on purpose

Deploy healthy, prove it's healthy, ship the bad change, apply load. All timestamps UTC, 31 July 2026.

Baseline — 60 requests, 10 concurrent:

sent      : 60 in 6.0s (10.0 rps)
status    : {201: 60}
p50 / p95 : 0.79s / 2.03s
error rate: 0.0%

The bad deploy — WriteFanout: 1 → 25. Nothing else changes. No IAM change, no new resource, no infrastructure diff. Exactly the kind of one-line change that sails through review.

16:54:52  deploy starts
16:55:34  deploy completes   (UPDATE_COMPLETE, OrdersFunction updated)

Load — 400 requests, 40 concurrent:

16:55:47  load starts
sent      : 400 in 13.6s (29.3 rps)
status    : {201: 88, 500: 52, 503: 260}
p50 / p95 : 0.87s / 2.55s
error rate: 78.0%
16:56:00  load ends

78% of customer requests failed. Note the split: 52 × 500 and 260 × 503. Two different failure modes, and that turns out to matter (§12).

A note on latency numbers, because two sets appear in this post and they disagree. The p50 / p95 figures above are client-side, measured by my load generator, so they include TLS handshake and connection setup from a cold Python client. The p99 figures the agent reports in §11 are API Gateway's server-side Latency metric. Server-side p99 at baseline was 1,228 ms against my client-side p95 of 2,030 ms — not a contradiction, just a different measuring point. When you cross-reference an agent's output against your own numbers, check you're comparing the same vantage point before you conclude it's wrong.

The pages, from CloudWatch alarm history:

16:55:51  first ProvisionedThroughputExceededException in Lambda logs
16:56:30  devops-lab-api-5xx            OK -> ALARM   (55 5xx in the 16:55 minute)
16:56:31  bridge posts CRITICAL incident to DevOps Agent
16:56:56  devops-lab-orders-errors      OK -> ALARM   (50 Lambda errors at evaluation
                                                     time; the minute finalised at 52)
16:56:56  bridge posts HIGH incident
16:57:41  devops-lab-ddb-write-throttled OK -> ALARM  (161 throttled BatchWriteItems)
16:57:41  bridge posts HIGH incident

Two numbers worth writing down:

• 39 seconds from first customer-visible error to first page. That's the floor for a 1-minute-period, 1-datapoint CloudWatch alarm, and no agent can beat it — detection latency is yours to own.
• ~1 second from alarm state change to the agent having a signed, enriched incident. The bridge is not the bottleneck.

Also: the three alarms fire 71 seconds apart for one root cause — exactly the pattern the triage stage exists to handle.

9. The evidence chain

An investigation is only as good as what's on the table. Everything the agent could reach, straight from CloudWatch.

DynamoDB consumed write capacity (Sum per minute, against 1 provisioned WCU = 60 WCU/minute):

Throttling — and this is where you need to be careful which metric you read:

4,025 = 161 × 25, and 125 = 5 × 25. The ratio between the two metrics is exactly the fanout factor. That's not a coincidence, and the DynamoDB docs explain why: a single request produces multiple write events, WriteThrottleEvents increments once per throttled event, and for BatchWriteItem the ThrottledRequests metric is only incremented when all the individual put/delete events in the batch are throttled. So 161 fully-throttled batches × 25 items each = 4,025 throttled item-events.

Two implications:

• Pick your metric on purpose. ThrottledRequests masks the true extent of throttling — the AWS docs say so explicitly and recommend comparing it against the event-level metrics. My alarm used ThrottledRequests, which is fine for detection but understates blast radius by 25×.
• This arithmetic is also the proof of what the code did — one BatchWriteItem per invocation carrying 25 items. Hold that thought until §11.

Lambda: Errors 52, Throttles 260, both in the 16:55 minute. Average duration 52.6 ms → 625.2 ms, max 2,917 ms.

The log line that names the culprit:

{
  "event": "order_write_failed",
  "level": "ERROR",
  "requestId": "BYX_lg6wvHcEPjw=",
  "route": "POST /orders",
  "errorType": "ProvisionedThroughputExceededException",
  "error": "An error occurred (ProvisionedThroughputExceededException) when calling
            the BatchWriteItem operation (reached max retries: 2): The level of
            configured provisioned throughput for the table was exceeded...",
  "table": "devops-lab-orders",
  "writeFanout": 25
}

That "writeFanout": 25 is deliberate, and it's the single highest-leverage thing in this lab. Log the feature flags and config values that changed behaviour, in the error path. It converts "correlate the deployment timeline against metric anomalies and hypothesise" into "read one field". It costs nothing and saves an entire investigative branch.

The chain the agent needs to assemble:

customer 5xx
  → Lambda invocation errors
    → ProvisionedThroughputExceededException on BatchWriteItem
      → 2,200 WCU consumed vs 1 WCU provisioned
        → BatchWriteItem where there used to be PutItem
          → CloudFormation UPDATE_COMPLETE on OrdersFunction at 16:55:34,
            12 seconds before the first error
            → WriteFanout parameter 1 → 25

Seven hops across four services, symptom three services away from cause. This is the shape of problem DevOps Agent is built for.

10. Asking the agent

The DevOps Agent web app chat panel. Left sidebar: Pages (Incidents, Improvements, Artifacts, Topology), Chats, Agents, Knowledge, Settings, and the account/space footer <account-id> / my-first-devops-agent. The conversation shows the stack-history question and the agent's tool count.

I started in chat rather than through the webhook, because that's what an engineer actually does when they notice something odd. First question, deliberately mundane:

Me: so can you say about my devops-agent-lab cloudformation stack — how many times it is updated and with what changes

Agent: Let me look into the CloudFormation stack history for devops-agent-lab — I'll check the update history and what changes were made each time.

10 tools · 1 agent used

It came back with a table:

All four rows match my deploy logs. But the part I didn't expect was the closing note:

Notable Context — The stack description identifies it as "a deliberately fragile serverless order API used to generate real, diagnosable production incidents" — so the fan-out injection was intentional lab activity, not an accidental misconfiguration.

It read the CloudFormation Description field and used it to reason about intent. Nobody told it this was a lab. That's a small thing, but it's the difference between a metrics query and an operational teammate: it distinguished "someone broke this on purpose" from "this is an incident", which is exactly the judgement call you want before it starts paging people.

The moment it refused

Then I pushed:

Me: so does that load thing happened on dynamodb can you investigate

And the agent declined to answer in chat:

Agent: This sounds like open-ended root-cause work — figuring out what actually happened to DynamoDB during that write fanout spike is the kind of multi-signal investigation (metrics, logs, traces, changes) that the Investigation agent is built for.

Want me to open a tracked investigation into the DynamoDB load impact from the WRITE_FANOUT spike, or keep digging here in chat?

Me: Open an investigation

Agent: Let me get that started right away!

1 tool · 1 skill used

The investigation is up and running! 🎉 DynamoDB load impact from WRITE_FANOUT spike on devops-agent-lab — It's scoped to the key questions: DynamoDB throttling, Lambda errors during the fanout window, whether the billing mode switch resolved things, and any downstream blast radius. It'll correlate metrics, logs, and the change timeline from July 31st.

This is worth dwelling on, and it's the single most useful product behaviour I saw.

The chat agent recognised that my question exceeded what a conversational answer should cover, named the reason, and offered to escalate to a tracked, billable investigation rather than silently producing a confident-sounding chat reply. It asked before spending my money. It also pre-scoped the investigation — throttling, Lambda errors, whether the fix worked, blast radius — which is a better scope than the one-line question I'd typed.

Two practical notes for anyone budgeting this:

• Chat and investigations are metered separately (monthlyAccountOnDemandHours vs monthlyAccountInvestigationHours) at the same $0.0083/agent-second. The chat exchanges above are on-demand seconds; the investigation is investigation seconds.
• This is a third way to start an investigation that isn't in the docs' list of three — a human escalating from chat. Practically it's the same as clicking Start investigation, but the ergonomics are much better because the agent writes the scope for you.

11. The investigation, graded claim by claim

The investigation detail view with **Investigation timeline* and Root cause tabs, showing the agent's generated latency charts: average latency peaking at 245 ms during throttling, p99 spiking to 2,401 ms, and a backend-latency chart used to confirm Lambda/DynamoDB as the bottleneck. Note the chart x-axis renders in browser-local time (IST) while the narrative text is UTC.*

Here's what it produced. I've quoted it, then checked every number against CloudWatch myself.

What it said

Impact 1 — DynamoDB OrdersTable massive write throttling during WRITE_FANOUT=25 window

DynamoDB table devops-lab-orders experienced 4,025 WriteThrottleEvents at 16:55 UTC and 125 at 16:56 UTC when WRITE_FANOUT was changed from 1 to 25. ConsumedWriteCapacityUnits spiked 36.7x from baseline 60 to 2,200. The provisioned capacity was completely overwhelmed. Throttling dropped to 6 residual events at 17:00 and ceased after the PAY_PER_REQUEST billing mode change at 17:01. Impact lasted approximately 5 minutes.

Impact 2 — Lambda OrdersFunction errors (52) and throttles (260) during fanout burst

52 errors (ProvisionedThroughputExceededException after 2 retries on BatchWriteItem) and 260 invocation throttles at 16:55 UTC. Duration spiked 12x from 52ms to 625ms average (max 2917ms). Concurrency was pegged at 10 (reserved limit). The combination of longer-running invocations (due to DDB retry backoff) and capped concurrency caused massive throttling. Recovery by 17:04.

Impact 3 — API Gateway returned 312 5xx errors (78% error rate) at peak

312 out of 400 requests as 5xx at 16:55 UTC (78% error rate). p99 latency peaked at 2.4 seconds (from ~1.2s baseline). No 4xx errors. Full recovery by 17:04.

Root cause

At 16:55 UTC the WRITE_FANOUT environment variable on Lambda function devops-lab-orders was changed from '1' to '25'. This caused each Lambda invocation to execute 25 BatchWriteItem operations to DynamoDB instead of 1, resulting in a 36.7x increase in consumed write capacity units (60 → 2,200 WCU/min).

Key finding 2 — Lambda concurrency exhaustion due to increased invocation duration from DDB retry backoff

With DynamoDB throttling, the Lambda SDK retried BatchWriteItem with exponential backoff, causing average duration to spike from 52ms to 625ms (max 2917ms). With a reserved concurrency limit of 10 and invocations taking 12x longer, the function could only serve ~16 requests/second. This resulted in 260 Lambda throttles at 16:55 — the vast majority of incoming requests were rejected before even reaching the function code.

What checks out

I re-pulled every metric. The impact numbers are not approximately right, they're exactly right:

The trailing-error breakdown deserves a callout. 5 at 16:59, 2 at 17:00, 15 at 17:04 is not a rounded summary — those are three individual CloudWatch datapoints reproduced exactly, and they're the fingerprint of my rollback (the 7 errors at 16:59–17:00) and my final verification run (the 15 at 17:04). It didn't just find the incident; it found the two things I did after the incident and folded them into the recovery narrative correctly.

More impressive than the numbers: the causal chain in Key Finding 2 is better than the one I had written down before running it. I had the 500s and the 503s as two separate root causes. The agent connected them: DynamoDB throttling → SDK retry backoff → 12× longer invocations → concurrency slots drained → invocations rejected before the code runs → 503s. The two failure modes weren't independent after all; the first amplified the second. That's a genuinely better explanation and I had to go back and correct my own draft.

Two of its methodological choices are worth stealing.

It chose WriteThrottleEvents over ThrottledRequests — the event-level metric AWS docs recommend for understanding true extent (§9). That's the right call, and not the obvious one; the request-level metric is the one most people alarm on.

It compared IntegrationLatency against Latency to localise the delay. API Gateway emits both: total round-trip and time spent waiting on the backend. If they track each other, the time is downstream; if Latency exceeds IntegrationLatency, the gateway itself is the problem. At 16:55 both p99s were 2,401 ms — identical — which is what let it state "confirming Lambda/DynamoDB as the bottleneck" rather than guessing. That's a textbook diagnostic technique applied unprompted, and it's the sort of thing that separates this from a metric dump.

What doesn't

Two mechanism errors. Both reach the right conclusion by a wrong route.

Error 1 — "execute 25 BatchWriteItem operations instead of 1." It's one BatchWriteItem carrying 25 items. BatchWriteItem maxes out at 25 items per request and boto3's batch_writer() flushes at exactly that boundary, so WRITE_FANOUT=25 produces a single request per invocation.

The proof is in the metrics the agent itself quoted. Across the two burst minutes (16:55 and 16:56): ThrottledRequests 161 + 5 = 166, WriteThrottleEvents 4,025 + 125 = 4,150 — a ratio of exactly 25. If there had been 25 separate BatchWriteItem calls per invocation, ThrottledRequests would have been in the thousands, not 166.

Impact: cosmetic here. It doesn't change the diagnosis or the fix.

Error 2 — "a reserved concurrency limit of 10." This one matters.

$ aws lambda get-function-concurrency --function-name devops-lab-orders
{}

$ aws lambda get-account-settings
{ "AccountLimit": { "ConcurrentExecutions": 10,
                    "UnreservedConcurrentExecutions": 10 } }

There is no reserved concurrency on this function — my template never sets any, and GetFunctionConcurrency returns empty. The limit of 10 is the account-level concurrency quota, where the Lambda default is 1,000. This account has never had a quota increase and is sharing 10 slots across 43 functions in the region.

The agent saw ConcurrentExecutions max out at 10 and inferred a function-level reservation. Understandable, and wrong in a way that changes what you'd do about it:

• If it were reserved concurrency: edit the function's concurrency configuration. Two minutes, self-service, in the template.
• Because it's the account quota: raise it through Service Quotas, which is a support request with a turnaround, and it affects all 43 functions in the region.

An engineer handed this RCA at 3 AM would go looking for a setting that isn't there.

The honest reading of this

AWS markets "94% root cause accuracy" for DevOps Agent. Having now graded one investigation line by line, I think I understand the shape of the residual, and it's not what I expected. The failure mode isn't "the agent hallucinates an unrelated cause". It's:

The verdict is right. The numbers are right, sometimes to the decimal. And a mechanism detail in the middle is stated with exactly the same confidence as everything else, while being wrong.

There's no hedging in the output, no confidence score on individual claims, nothing that flags "I inferred this rather than read it". Concurrency was pegged at 10 (reserved limit) reads identically to 4,025 WriteThrottleEvents at 16:55, and one is a direct metric read while the other is an inference that happens to be false.

Which is a completely workable tool, as long as you use it as "here is a correct conclusion and an evidence trail, check the middle before you act" rather than "here is the answer". That's also precisely why §13 matters more than any feature on the pricing page.

A small presentation nit while I'm here: the chart axes in the investigation view render in browser-local time while the narrative text is in UTC. My screenshots show a 22:23–22:34 axis for a 16:53–17:04 UTC window — correct, but +5:30, and easy to misread when you're cross-referencing against CloudWatch at speed.

12. The second root cause

This section originally argued that DevOps Agent would find the concurrency limit where a code-focused agent never would. It's worth reporting what actually happened, because the outcome was better than my prediction in one way and worse in another.

The setup: my load test returned 52 × 500 and 260 × 503. The 500s are DynamoDB throttling. The 503s are something else — Throttles, not Errors, meaning those invocations never ran. The cause is an account-level Lambda concurrency quota of 10, against a default of 1,000, shared across 43 functions. With 40 concurrent client threads and 10 available slots, most requests were rejected before my code executed.

I did not plant this. I found it while working out why the two error codes disagreed.

What I predicted: point any code-focused AI at this repository and it finds the WRITE_FANOUT bug immediately — it's right there in the diff. It never finds the concurrency limit, because the limit isn't in the code, isn't in the template, isn't in the logs and isn't in the alarm. It's an account-level quota that only exists in the environment.

What happened: DevOps Agent found the concurrency exhaustion, unprompted, and did better than I did by connecting it causally to the DynamoDB throttling via retry backoff. Then it mislabelled the source of the limit as reserved concurrency.

So the thesis holds, with a sharper edge than I'd have written on my own: the environment-level cause is exactly the class of thing only an environment-aware agent surfaces at all — and also exactly the class of thing it's most likely to get subtly wrong, because quotas, reservations and defaults all present identically in a metric graph. The metric shows you a ceiling at 10. It does not tell you which of four different mechanisms put it there.

That combination is the real argument for the audit journal.

One more finding, mine not the agent's: I had no alarm on Lambda Throttles, so 260 of 312 failed requests — 83% — never paged anyone. The alarms I did have fired on the smaller failure mode. Exactly the kind of gap the prevention evaluation is supposed to hand you.

13. Reading the agent's mind

Every execution writes an immutable journal you can read over the API:

aws devops-agent list-journal-records \
  --agent-space-id $SPACE --execution-id $EXECUTION --region us-west-2

Remember the FAILED learning task from §6? Here's what actually happened, reconstructed from its 119 journal records.

It started at 08:14 UTC with the prompt understanding-agent-space: Create understanding of the agent space. Its first move was to read its own instructions and check for code repositories:

"I need to create an understanding-agent-space skill, which falls within my capabilities. I'm starting by reading the relevant skill documentation and gathering context about the agent space by listing available repositories."

Finding no repositories, it reasoned about which workflow steps to skip:

"Looking at what's available: the repository list is empty so there's no codebase to discover, I have one AWS account active, and today's date is July 6, 2026. Most steps will be skipped — no prior state files exist, no observability tools are configured beyond AWS, and no member skills are loaded. The main work is discovering the topology for that AWS account."

Then it partitioned my account by CloudFormation stack:

"This is one AWS account hosting **multiple independent applications, concentrated in us-east-1 and us-west-2, with minimal footprint in ap-south-1. CloudFormation stacks give me strong boundary signals: **AiTravelConcierge* — 5 stacks (Api, Compute, Data, MockApi, Orchestration); HealthFormMvp, RetailMindAI, SelfHealingAiAgent, email-agent — 1 stack each. Scaffolding to ignore: CDKToolkit, aws-sam-cli-managed-default."*

All five applications, correctly bounded, scaffolding correctly discarded — from stack topology alone, with no repository access.

It then spawned seven sub-agents in parallel, wrote 11 container reference files, and self-validated its own output:

"All 11 container reference files and SKILL.md are written. Now let me validate the skill output before mirroring to memory. Let me check for template/placeholder leftovers and confirm every graph node has both icon and description."

The journal also exposes the machinery. Aggregated from the utilization records:

--- summary ---
peak context window : 74.7%
sub-agents spawned  : 7  [disc-aitravelconcierge, disc-emailagent, disc-healthform,
                          disc-retailmind, disc-selfhealing, disc-useast1-apsouth1,
                          scope-discovery]
tool calls          :
  fs_write                          13
  fs_read                           12
  list_resources                     5
  fs_tree                            4
  use_aws                            4
  fs_grep                            3
  list_repositories                  1
  get_topology_map                   1
  get_associations                   1
  get_account_cloudformation_stacks  1

So the agent has a working filesystem (fs_read/fs_write/fs_tree/fs_grep over paths like /aidevops/skills/learned/understanding-agent-space/SKILL.md), a generic use_aws escape hatch, and purpose-built topology tools. Context peaked at 74.7% with zero compactions.

And then — at 09:02 UTC, 48 minutes in, minutes after that final validation step — the task was marked FAILED.

Two conclusions, and I think they're the most useful paragraphs here.

The good one. After §11, this is no longer a nice-to-have. The agent produces confident prose containing occasional confidently-wrong details, and the journal is the only place you can see which claims came from a metric read and which came from an inference. No other agent product I've used lets me reconstruct a 48-minute autonomous run to this fidelity. Do not treat the journal as a compliance artifact. It's your primary verification surface, and you should know how to read it before an outage, not during one.

The bad one. The failure was invisible. The console gave me a working Agent Space. list-associations said valid. Nothing surfaced that the environment understanding the entire product is built on had never completed — and I only looked because I went digging with the API. Then, when I did find it, the journal recorded 119 records of successful work and no terminal error explaining the failure.

So: poll list-backlog-tasks and alarm on FAILED. Treat your agent as a production dependency with its own health check, because AWS hasn't given you one.

14. Fixing it

Mitigation 1 — roll back. The first thing any agent should recommend: revert the change, don't fix forward during an outage.

16:59:13  rollback starts  (WriteFanout 25 -> 1)
16:59:54  rollback complete

Result at 12.8 rps:

status    : {201: 113, 500: 2, 503: 5}
error rate: 5.8%

78% → 5.8%. But not zero, and that residual is the interesting part: at 12.8 rps with 1 KB items I need ~13 WCU and have 1. The rollback removed the 25× amplification, not the underlying fragility. It made the incident survivable; it did not fix the system. Any mitigation plan that stops at "reverted, closing the incident" is wrong.

Mitigation 2 — the durable fix. Switch the table to on-demand:

TableBillingMode:
  Type: String
  Default: PROVISIONED
  AllowedValues: [PROVISIONED, PAY_PER_REQUEST]

sent      : 200 in 6.6s (30.2 rps)
status    : {201: 185, 503: 15}
p50 / p95 : 0.78s / 0.93s
error rate: 7.5%

Zero 500s. Every DynamoDB error gone, at 2.4× the request rate that was still failing before. p95 improved from 1.83 s to 0.93 s — a 49% reduction, because the SDK is no longer burning retries against a throttled table. The agent independently confirmed this from the other direction: "Throttling … ceased after the PAY_PER_REQUEST billing mode change at 17:01."

The remaining 15 × 503 are the concurrency quota from §12 — which is the correct end state for this lab: one incident, two independent causes, one fixed by a deploy and one requiring a quota increase. Real outages look like this.

What prevention should produce

The ONCALL_REPORT goal runs on rate(7 days) across observability, infrastructure, pipelines and application resilience. Mine hadn't run against this incident at the time of writing (list-recommendations returns empty — I created the incident on the last day of the cycle). So rather than invent output, here's the list I derived by hand, which is the standard I'd hold it to:

If your evaluation returns items 1 and 2 — the ones that require looking at the environment rather than the diff — it's earning its $7.47. If it only returns 3 and 7, you've paid for a linter.

15. Closing the loop

Getting a root cause into the web app is half a workflow.

Read the agent's decisions with EventBridge. DevOps Agent publishes to the default event bus under source aws.aidevops:

Investigation lifecycle: Created, Pending Triage, Priority Updated, In Progress, Completed, Failed, Timed Out, Cancelled, Linked, Skipped
Mitigation lifecycle: In Progress, Completed, Failed, Timed Out, Cancelled

Capture all of it from day one — cheapest possible instrumentation of your agent, and how you build an MTTR dashboard that isn't vibes:

InvestigationEventRule:
  Type: AWS::Events::Rule
  Properties:
    EventPattern:
      source: [aws.aidevops]
    Targets:
      - Id: log
        Arn: !GetAtt InvestigationEventLogs.Arn

Then filter for what you act on:

{ "source": ["aws.aidevops"],
  "detail-type": ["Investigation Completed", "Investigation Failed"] }

Watch Investigation Skipped and Investigation Linked especially closely in the first month. Those are the triage stage making judgement calls on your behalf.

Two-way flow needs a ticketing system. Only the built-in integrations (ServiceNow, PagerDuty, Slack, Microsoft Teams) get updates written back. A generic webhook is one-way: you push an incident in, findings appear in the web app, and nothing lands in the tool your team watches. If you want the agent posting an investigation timeline into Slack and updating the ticket as it goes, you need one of those integrations. Budget for it in your evaluation.

The Kiro handoff. Investigation output includes agent-ready instructions: a mitigation spec another frontier agent can implement. For my incident that's a clean division of labour — DevOps Agent determines that WRITE_FANOUT=25 against a 1-WCU table is the cause, Kiro implements the template change and the missing throttle alarm, release management reviews the PR for blast radius before merge. Three agents with different context boundaries, handing work between them, humans approving at the transitions.

Worth noting after §11: the handoff is also where a wrong mechanism detail would propagate. "Reserved concurrency limit of 10" fed to a coding agent produces a template change setting reserved concurrency — which would make things worse, since reserving from a pool of 10 starves the other 41 functions. Verify at the transition, not after.

Access from anywhere. The Agent Space runs as a remote server speaking MCP, ACP and A2A, so your own tools and agents can trigger investigations or query operational health without going through the web app.

16. Making the agent yours

Defaults get you a generic SRE. These knobs get you your SRE.

Skills — reusable Markdown instruction sets, optionally targeted to a specific agent type (Generic, On-demand, Incident Triage, Incident RCA, Incident Mitigation, Evaluation). Targeting matters: a triage-only skill doesn't consume context during RCA.

The highest-value first skill is a triage skip rule, because it stops you paying for investigations you don't want:

---
name: skip-scheduled-maintenance
description: Skip low-priority incidents during a scheduled maintenance window.
  Use this skill to automatically filter MEDIUM and LOW severity alarms that
  fire during planned maintenance, avoiding unnecessary investigations for
  expected disruptions.
---

# Skip Scheduled Maintenance

Skip incidents meeting BOTH criteria:
1. Arrived between 2026-08-15 02:00 UTC and 2026-08-15 06:00 UTC
2. Severity is MEDIUM or LOW

Do NOT skip HIGH or CRITICAL, even during the maintenance window.

Target it at Incident Triage. Every new Agent Space ships this as an inactive sample — activate and adapt rather than starting from scratch.

The second skill I'd write is the one §11 taught me to write. Note that it exists specifically to prevent the mistake the agent made:

---
name: serverless-throttle-triage
description: Investigation procedure for API Gateway 5xx and 503 responses on
  serverless order-processing APIs. Use when investigating elevated 5xx, 503,
  Lambda invocation errors, or DynamoDB throttling on devops-lab-orders.
---

# Serverless Throttle Triage

Distinguish 500 from 503 before forming a hypothesis. They have different causes,
and in this application the first one amplifies the second.

## 503 responses
Check `AWS/Lambda Throttles`. Then establish WHICH ceiling was hit, because the
remediation differs and the metric graph looks identical either way:
  1. `lambda get-function-concurrency` - is reserved concurrency set on the function?
     If empty, it is NOT reserved concurrency. Do not report it as such.
  2. `lambda get-account-settings` - `ConcurrentExecutions` is the account quota.
     Default is 1,000. A lower value means this account has never had an increase,
     and the fix is a Service Quotas request, not a function config change.
Concurrency exhaustion produces 503 with no application log entry at all.

## 500 responses
Check Lambda `Errors`, then search the log group for `order_write_failed` and read
the `writeFanout` field - it records the write-amplification config active at
failure time.
Compare `ConsumedWriteCapacityUnits` against provisioned WCU
(provisioned WCU x 60 = per-minute budget).

## Reading DynamoDB throttling correctly
`ThrottledRequests` counts requests and only increments when EVERY item in a
BatchWriteItem is throttled. `WriteThrottleEvents` counts individual items.
Prefer `WriteThrottleEvents` for blast radius. If the ratio between them is a
round number, that number is the batch size - which tells you the fanout.
boto3 `batch_writer()` flushes at 25 items, and BatchWriteItem accepts at most
25, so N<=25 items is ONE request, not N requests.

## Always
Check CloudFormation stack events for `devops-agent-lab` in the 15 minutes before
the first error. A parameter-only update changes behaviour without changing
resources and will not appear as a resource diff.

That's institutional knowledge that normally lives in one senior engineer's head — and writing it as a skill is how you stop the agent repeating an error you've already caught once.

Instructions — the AGENTS.md equivalent, applied to every task or scoped to specific agents. Use it for standing context: which accounts are production, who owns what, what "customer-impacting" means to you.

Memories — learned knowledge, created either by directing the agent in chat or by the managed learning skill distilling past investigations. The mechanism behind "the third DynamoDB throttling incident investigates faster than the first".

Triggers and custom agents — schedule work with create-trigger (cron/rate conditions), and build custom SRE agents for recurring tasks. AWS's own example is a daily ops report: 2 minutes a day, $29.88/month.

Ask for artifacts in chat. Chat generates versioned documents — "generate a weekly operational health report", then "add a section on Lambda cold starts" — which beats screenshotting dashboards into Confluence. You can also attach files (dashboard screenshots, YAML, a runbook PDF; 3.75 MB each, 20 per message) and have the agent reason over them.

Give feedback on investigations. After §11 this is not a nicety. A completed investigation has an Add feedback action where you rate the root cause correct/incorrect, supply the actual root cause, and note whether steering was needed. Feedback feeds accuracy metrics you can query in chat. For my investigation the honest verdict is "root cause correct, mitigation detail incorrect — the concurrency limit is the account quota, not reserved concurrency", and recording that is how the next one gets better.

17. What this cost, and how to tear it down

Roughly $0.15 in AWS infrastructure charges. Everything is serverless and short-lived: ~500 Lambda invocations on 256 MB arm64 (many requests were throttled before they ever invoked), ~2,600 DynamoDB write capacity units consumed in total, four alarms at $0.10/month, some CloudWatch queries. The 7-day log retention on every log group is deliberate — it's the line item that quietly grows in labs like this.

DevOps Agent charges: $6.83, measured. 191 agent-seconds of investigation ($1.59) plus 632 agent-seconds of chat ($5.25) — see §4 for the full breakdown and why the chat, not the investigation, was the expensive half. Check your Support-plan credits before you treat that as a cost (§4); for Enterprise Support customers it's likely fully covered.

Teardown:

sam delete --stack-name devops-agent-lab --region us-west-2

# only if you're done with the agent entirely - irreversible, destroys all
# investigations, topology and recommendations in the space
aws devops-agent delete-agent-space --agent-space-id $SPACE --region us-west-2

The Agent Space itself costs nothing while idle — it's per-agent-second, not per-space. Leave it. But do delete the SAM stack: a DynamoDB table provisioned at 1 WCU is a footgun to leave lying around.

18. Honest scorecard

Where it clearly earns its keep

• Multi-service, multi-cause incidents. Mine had two causes in two layers, one of them not in the code at all. The agent found both and connected them causally better than I had.
• Environment knowledge that isn't written down. The concurrency ceiling is invisible to anything reading only the repository.
• Judgement, not just retrieval. It inferred from a CloudFormation Description that the fault injection was intentional. It declined to answer an open-ended RCA question in chat and asked permission to open a billable investigation instead. Both are the behaviour of a colleague, not a query engine.
• Auditability. The journal is the best I've seen in any agent product, and §11 is why it's essential rather than nice.
• Zero-setup onboarding for the team. One admin configures the space; every engineer inherits full context.
• The economics on Enterprise Support. 75% of your support spend back as credits changes the calculation entirely.

Where you should temper expectations

• It states inferences with the same confidence as measurements. Right verdict, exact numbers, and a mechanism detail in the middle that was confidently wrong and would have sent someone to a nonexistent setting. Verify the middle before you act on it. There is no per-claim confidence signal.
• It cannot see your alarms without glue. No alarm action, no CloudWatch integration, no EventBridge input. You will write a Lambda. A genuine product gap, though a well-documented one — AWS covers both bridge patterns with deployable samples (§7). Read those first, then spend your effort on translation quality.
• There is no health check. My learning task failed silently for three weeks in a space the console reported as fine.
• Detection latency is still yours. 39 of my seconds-to-page were CloudWatch alarm evaluation, and the alarm I most needed didn't exist.
• Garbage in, garbage out, and you pay per second either way. Structured logs that include changed config values are worth more than any feature on the pricing page.
• One-way by default. Without a ticketing integration, findings live in a web app nobody has open.
• Preview means preview. Release management is us-east-1 only.

Would I run it? For a single small serverless app with two alarms, no — a CloudWatch dashboard and a coding agent cover it. For a team on-call across a dozen services and multiple accounts, where the expensive part of every incident is the 30 minutes spent working out which service is broken: yes, and Support-plan credits probably make it free.

The honest summary: this is a very fast, very well-instrumented first responder that knows things about your account that aren't in your repository, and that occasionally gets a detail confidently wrong in the middle of a correct answer. Used as a starting point with an audit trail, it's genuinely good. Used as an oracle, it will eventually cost you an hour chasing a setting that doesn't exist. The work you put into your logs, your alarm descriptions and your webhook payload determines which one you get — and that part is entirely within your control.

19. Appendix: copy-paste checklist

Before you start

• [ ] Upgrade AWS CLI v2 and boto3/SDK. Verify with aws devops-agent help.
• [ ] Check your Support plan tier — you may already have credits covering all of this.
• [ ] Pick an Agent Space region for data residency, not to match your workloads.

Onboarding

• [ ] Create DevOpsAgentRole-AgentSpace with AIDevOpsAgentAccessPolicy.
• [ ] Add the iam:CreateServiceLinkedRole inline policy for Resource Explorer.
• [ ] Create DevOpsAgentRole-WebappAdmin with AIDevOpsOperatorAppAccessPolicy (trust policy needs sts:TagSession).
• [ ] Create the space, associate the account as monitor, enable the operator app.
• [ ] Do it all in CloudFormation, CDK or Terraform — not by hand.
• [ ] Wire up IAM Identity Center or an OIDC IdP before handing it to a team. The IAM link expires in 10 minutes.

Make it triggerable — pick a path first (§7)

• [ ] Read the two AWS sample posts before writing anything; don't reinvent the bridge.
• [ ] Path A (webhook): generate a generic webhook in the console, choose HMAC, store the secret in Secrets Manager, build SNS → Lambda, sign timestamp:payload.
• [ ] Path B (API): EventBridge → Lambda → create-backlog-task --task-type INVESTIGATION. No secret, no console step, fully IaC-able. Grant the Lambda role aidevops:CreateBacklogTask and use --client-token for idempotency.
• [ ] If you choose Path B, verify whether triage correlation (LINKED/SKIPPED) applies to API-created tasks.
• [ ] Map alarm names to CRITICAL/HIGH/MEDIUM deliberately — CloudWatch has no severity field.
• [ ] Make the dedupe key stable per alarm transition, not per delivery.
• [ ] Put account, region, metric, dimensions and the application's service chain in the description.
• [ ] Write real runbook text into every AlarmDescription and forward it.
• [ ] Send alarms through unbuffered first and watch for Investigation Linked before building your own dedupe table.

Make it observable

• [ ] EventBridge rule on source: aws.aidevops → CloudWatch Logs, from day one.
• [ ] Schedule list-backlog-tasks and alarm on FAILED.
• [ ] Check get-account-usage monthly — including learning hours.
• [ ] Know how to read list-journal-records before you need it.

Make it trustworthy

• [ ] Treat every mechanism detail in an RCA as a hypothesis until you've checked it. The verdict is usually right; a detail in the middle may not be.
• [ ] For any concurrency finding, confirm which ceiling: get-function-concurrency (reserved) vs get-account-settings (account quota). Different fixes.
• [ ] For DynamoDB throttling, read WriteThrottleEvents for blast radius, not just ThrottledRequests.
• [ ] Submit investigation feedback (correct/incorrect root cause) every time. It feeds accuracy metrics and future investigations.
• [ ] Verify at the agent-to-agent handoff. A wrong mechanism passed to a coding agent becomes a wrong code change.

Make it good

• [ ] Log changed config and feature-flag values in your error paths.
• [ ] Activate and adapt the sample triage skip skill.
• [ ] Write one skill per recurring incident class, with a description written from the agent's perspective — and encode the corrections you've already had to make.
• [ ] Add a ticketing or chat integration so findings reach where your team looks.