AI-Powered Root Cause Analysis at Scale

sitereliabilityengineering devops/observability platformengineering ai/agenticai telemetry

AI-Powered Root Cause Analysis at Scale: From Theory To Production

Speaker: Letícia Mota & Yevgeny Gladun (Nubank)

AI-Powered Root Cause Analysis at Scale: From Theory To Production... Letícia Mota & Yevgeny Gladun

Tldr

• Scale & Context: Nubank, serving over 120 million clients across Latin America, manages thousands of microservices where a short downtime impacts thousands. They built an autonomous AI SRE agent to investigate incidents like a senior engineer and lower MTTR.

• Incredible Efficiency: Built in-house in two months by only two engineers, operating at an investigation cost of **under $0.20perrun\ast \ast (comparedtovendorquotesof$10+).

• Three Core Pillars: Any production-ready SRE agent relies on: Model Intelligence, Data Access (via Model Context Protocols/MCPs), and Institutional Knowledge, all tied together by a Stateful Orchestrator.

• Solving the “Dory” Problem: LLMs are stateless by default. The team built a custom, stateful orchestration loop to reconstruct the conversation context at each step.

• Advanced Data Compression: Implemented local statistical pre-analysis (computing mean, p95, and z-scores) and sub-agent summarization to shrink massive telemetry datasets down to fit within tight context limits.

• Expert Guides & specialized Flows: Introduced “Expert Guides” to tell the LLM how to query internal, custom telemetry (like Prometheus metrics) and categorized incident queries into tailored runbook “Flows”.

• Proactive Alerting: Integrated directly with alerting systems. The agent automatically runs when an alert fires, saving the investigation report and posting it directly into Slack alongside live observability links.

1. The Operational Gaps in SRE

Nubank’s massive scale (thousands of microservices, millions of transactions daily) is highly time-sensitive. During incidents, three core gaps hindered on-call engineers:

1. Training & Playbook Gap: Runbooks were often too general, outdated, or hard to adapt during active incidents.
2. Data Overload: Engineers were “drowning in data sources,” jumping between Prometheus, Honeycomb, Github, and custom dashboards.
3. Productivity Drain: High-stress incidents resulted in hours of lost productivity across multiple senior engineers.

The team’s objective was to transition incident investigation from hours of manual lookup to minutes or seconds of automated reasoning.

2. Build vs. Buy Evaluation

Before writing code, Nubank evaluated four commercial AI SRE vendors and established strict evaluation criteria:

Evaluation Criteria	Commercial Vendors’ Shortcomings
Predictable Pricing	Quotes around $10 per investigation were cost-prohibitive.
Telemetry Privacy	One vendor required sending all petabytes of daily telemetry outside Nubank’s secure network (a massive compliance blocker).
Custom Stack Integration	Vendors struggled to integrate with Nubank’s heavily customized in-house observability stack.
SOC2 Compliance	Hard requirement that many early-stage vendors could not meet.
Speed to Value	Needed to achieve a working proof-of-concept (PoC) with 2 engineers in under 2 months.

Ultimately, none of the vendors fit. Nubank chose to build a homegrown Python service, resulting in a production-ready agent costing less than $0.20 per run.

3. System Architecture & The 3 SRE Pillars

The system is designed around a central homegrown Python orchestrator running in Kubernetes, acting as the brain connecting three major pillars.

graph TD
    User([Engineer / Alert]) -->|Slack / UI| Orchestrator[Python SRE Orchestrator]
    Orchestrator <-->|Reasoning| Model[Model Intelligence: LLMs]
    Orchestrator <-->|Telemetry Context| DataAccess[Data Access: MCPs for Prometheus, Honeycomb, GitHub]
    Orchestrator <-->|Metadata| InstKnowledge[Institutional Knowledge: Internal APIs, Blueprint Service]

The SRE Agent Mental Model

“If you are going to take away only the mental model of any SRE agent, this is the slide to pay attention to. Any SRE agent consists of three main pillars: Model Intelligence, Data Access, and Institutional Knowledge, all tied together by the orchestrator.”

4. Overcoming Core Technical Challenges

Three fundamental problems had to be resolved to make the SRE agent reliable and cost-efficient:

A. The “Dory” Problem (Statelessness)

The "Dory" Analogy

“LLMs have no memory. Every new request or follow-up is just like talking to Dory; it has no idea what you talked about previously.”

• Solution: A Stateful Orchestration Loop managed by the Python service. The orchestrator tracks state, aggregates tool call outputs, and reconstructs the relevant historical context on each LLM turn.

B. The Context Window Limitation

Telemetry data is massive. For example, a single raw Honeycomb trace can easily exceed 26,000 characters. Sending raw datasets directly to the LLM overflows context limits and balloons costs.

• Solution 1: Local Statistical Pre-analysis. The Python orchestrator processes raw metrics before passing them to the LLM. For instance, rather than passing raw Prometheus timeseries data points, the orchestrator calculates the median, mean, p95, and z-score, and sends only those metrics.
• Solution 2: Sub-Agent Summarization. If data cannot be summarized mathematically (e.g., long text logs), a secondary, cheaper LLM sub-agent is spun up specifically to summarize the telemetry. The main reasoning loop only sees the clean summary.

C. Generic Tool Usage

Standard LLMs know how to execute API/database queries, but do not understand the tribal knowledge of custom environments (e.g., querying standard Prometheus metrics in a custom infrastructure returns empty datasets).

• Solution: Expert Guides. Every Model Context Protocol (MCP) server is deployed alongside explicit “Expert Guides.” These are system-level guides that instruct the LLM on exactly how to query metrics in Nubank’s environment (e.g., advising the model to ignore standard metrics and focus strictly on 14 high-signal custom SRE metrics).

5. The Stateful Orchestration Loop

The agent’s main execution loop operates in four sequential phases:

graph LR
    A[Phase 1: Discovery] --> B[Phase 2: MCP Selection]
    B --> C[Phase 3: Investigation]
    C --> D[Phase 4: Finalization]

1. Discovery Phase: The agent classifies the type of question (e.g., latency, crash, database error) and loads a specialized Flow (the AI equivalent of an SRE runbook).

2. MCP Selection: The agent dynamically limits active tools. For instance, if the query is a latency issue, it avoids loading the crash history tool, saving context space.

3. Investigation Phase: A recursive execution loop (capped at a configurable limit, default is 30 cycles). The LLM requests telemetry, reasons about the results, and dynamically requests more details until it reaches a conclusion.

4. Finalization Phase: A specialized prompt formats findings into a clean summary with live observability links to Nubank’s internal dashboards so human engineers can quickly verify the findings.

6. Key Production Lessons

• Model Quality is Parametric: Upgrading the underlying LLM model drastically improved investigation quality. With zero changes to code, guides, or prompts, the agent graduated from behaving like a “junior engineer” (grabbing the first stack trace line and stopping) to behaving like a thorough “senior SRE”.

• One Size Does Not Fit All: Moving away from a single general prompt to specialized, runbook-like Flows per incident type was a major breakthrough in accuracy.

• Decoupled Configuration: Prompts, flows, and expert guides are stored outside the main application codebase. This allows SREs to iterate on prompts, version them, and analyze performance shifts without executing code deployments.

• Observe the Agent: The SRE agent is treated like any other backend microservice. It uses the exact same SRE dashboard stack, meaning the agent can physically reason about its own resource limits (e.g., alerting when it runs out of CPU or memory).

• Streaming Updates Build Trust: Streaming real-time thoughts and tool calls back to the Slack channel keeps the engineer highly engaged. Often, watching the agent fetch data triggers an “aha! moment” for the engineer before the final report is even finished.

7. Beyond RCA: Platform Evolution

What started as an isolated SRE tool has evolved into a wider, company-wide Agentic Platform:

• Fraud Investigation: Nubank’s fraud team adapted the orchestrator’s framework to build automated fraud-investigation agents by swapping in custom MCPs and specialized flows.

• The “Marinator” Bot: An automated bot that tracks active Severity 1 and Severity 2 incidents, summarizing progress real-time for senior leadership.

• Proactive SREing: Hooked directly into live alerts. On-call developers receive a Slack page with the alert, coupled immediately with a pre-run diagnostic report generated by the agent.

Summary Mantra

“Always evaluate your pillars, invest in your guide rails, and ship fast/faster.”