AI Engineering · Cloud Operations

cloud-incident-service

An AI-assisted service that investigates AWS incidents with a bounded, read-only agent and returns evidence-cited reports that keep facts, hypotheses and uncertainty separate.

Status: MVP · synthetic data
Discipline: AI Engineering
Key stack: Python, FastAPI
Repository: View on GitHub

Overview

What it does

Given a short incident report (service, environment, symptoms and a time window of up to 24 hours), the service collects evidence from read-only tools, correlates it by time and service, and returns a schema-validated report.

The report separates confirmed facts, hypotheses with confidence levels, uncertainties, next diagnostic steps and remediation proposals, and every statement cites the evidence behind it. The service never changes cloud resources. It is a portfolio MVP that runs locally against synthetic data and makes no claims about production scale.

Problem

Incident responders gather evidence by hand from metrics, alarms, logs, deployment records, infrastructure changes and runbooks, then correlate it mentally under time pressure. An AI assistant can reduce that overhead, but only if every claim stays traceable to real evidence and every change stays under human control.

Who it helps

AWS-focused DevOps, cloud operations and site reliability engineers investigating a degraded service or failed deployment, and reviewers who need to audit how an agent reached its conclusions.

Solution

Four read-only tools: CloudWatch signals, deployment history, infrastructure changes (with a 60-minute look-back) and runbook search.
Two orchestrators behind one protocol: deterministic correlation rules by default, or a model-driven loop capped at 8 tool calls, 6 turns and 60 seconds.
Evidence comes from the service, never from the model: the model cites source IDs, anything no tool returned is rejected, and hypotheses are checked against the correlation rules.
Every remediation proposal requires human approval; stale, malformed or unavailable sources are reported as uncertainty or as a partial report.
An offline evaluation of ten scenarios scores schema, citations, hypothesis, sources and safety. Investigation traces are redacted for access keys, account IDs and email addresses.

Technology stack

Backend: Python 3.12+
FastAPI
Pydantic
Uvicorn
AI / LLM: Amazon Bedrock (Converse API, tool use)
Anthropic Messages API
Offline rule-based mock
Cloud services: Amazon CloudWatch Logs (opt-in)
boto3
Testing & quality: pytest
Ruff
uv
DevOps: GitHub Actions
Dependabot
pip-audit

Infrastructure & architecture

A local-first service. There is no hosted deployment, container image or infrastructure as code; AWS is reached only through opt-in, read-only adapters.

incident request
bounded agent
read-only tools
validated report

Runtime: Runs locally with Uvicorn on 127.0.0.1, with optional API-key authentication and a 32 KiB request-body limit.
CI: GitHub Actions on Python 3.12, 3.13 and 3.14: formatting, linting, tests and both offline evaluations.
Supply chain: pip-audit dependency audit on pushes, pull requests and weekly; Dependabot updates for uv and GitHub Actions.
AWS access: Least-privilege IAM policy examples for Bedrock invocation and CloudWatch Logs reads. Live tests are opt-in.
Observability: Per-investigation JSON traces of tool calls, model turns and validation outcomes, with secret redaction.

Next project: estoe