The Operate blog
Field notes on AI SRE, root-cause analysis, and running self-hosted AI in production.
The Bug Was in the Recovery Code: What AT&T's 1990 Collapse Says About Fleets That Heal Themselves
AT&T's 1990 collapse is told as a missing break statement. The transferable lesson is that the defect was in recovery code, the self healing machinery was t
Context Deadline Exceeded Names the Service That Gave Up, Not the One That Was Slow
Context deadline exceeded names the caller that gave up, not the slow hop. A cross stack method for finding which service was actually slow, and when to raise t
Meta Runs 50,000 Automated Investigations a Day. The Number That Matters Is 2,000.
Meta's DrP runs 50,000 automated investigations a day across 300 teams. The overlooked number is 2,000 hand-authored analyzers, and what it says about the real
Your Triage Agent Solved Everything Nobody Needed Help With
Triage automation is scored on total cases handled, which is dominated by cases nobody needed help with. Score it on the residual instead, and the picture chang
Your Triage Agent Solved Everything Nobody Needed Help With
Triage automation is scored on total cases handled, which is dominated by cases nobody needed help with. Score it on the residual instead, and the picture chang
CrashLoopBackOff Is a Status, Not a Diagnosis: Triaging the Restart Loop in the Order That Actually Narrows It
CrashLoopBackOff tells you a container keeps restarting, not why. A production triage order: blast radius, exit code, crash versus kill, and the causes outside
Time to First Page: The Onboarding Number Engineering Leaders Do Not Report
Time to first page is a measurable onboarding metric with two distinct failure modes. How to define it, what to gate on, and the four numbers worth reporting.
Nothing Shipped and p99 Tripled: Three Ways Postgres Changes Your Plan Silently
Three documented ways Postgres changes an execution plan under an unchanged query: stale statistics, the generic plan switch, and lost partition pruning.
Runbook Automation Automates Execution. Execution Was Rarely the Slow Part.
Runbook automation compresses execution. Measure runbook match rate, verified recency and diagnosis share first, and find out if execution is your bottleneck.
The Recorder Ran Out Before Anyone Knew There Was an Incident: What the FAA's 25-Hour Rule Says About Your Retention Window
The FAA extended cockpit voice recorders from 2 hours to 25 because evidence kept getting overwritten. Your log retention has the same design flaw.
Incidents Per Pull Request Are Up 243 Percent: The AI Productivity Gain Has a Bill
Faros telemetry across 22,000 developers shows incidents per PR up 242.7 percent while DORA's survey finds no burnout. The operational bill for AI-written code.
Too Many Open Files Is Almost Never a ulimit Problem: A Production Field Guide to File Descriptor Exhaustion
Too many open files is rarely a limit problem. How to tell a descriptor leak from a sizing or latency issue, across Linux, nginx, Postgres and Kubernetes.
Seven Indexes, Larger Than the Table: Why Index Sets Rot and Nothing Forces a Review
One production table carried 17 GB of index over 11 GB of heap, and every index was justified when it was added. Why index sets only grow and how to run a revie
The Pipeline Was Green and It Deployed Nothing: A Taxonomy of Steps That Succeed by Doing Nothing
Pipeline config bugs almost always fail toward doing less, so they never raise an error. A taxonomy of steps that succeed by doing nothing and how to catch them
Configuration Drift Is Not a Hygiene Problem: Triaging the Drift That Actually Causes Incidents
Most configuration drift guides stop at infrastructure as code. This one covers the drift IaC never sees, how to tell legitimate drift from dangerous drift, and
Normal Accidents at Cloud Scale
Perrow's Normal Accidents predicted the Oct 2025 AWS cascade: tight coupling plus interactive complexity, triggered by the redundancy itself. An engineer's guid
Alarms Are the Cheap Half: An Ops Playbook for Teams of One
AI made shipping cheap; operating is still expensive. A triage order, minimum telemetry list, and one-page runbook for solo founders and two-person teams.
The Transactional Outbox Pattern on Day 2: Failure Modes the Tutorials Skip
Every outbox pattern tutorial stops at the happy path. A day-2 guide to relay stalls, outbox lag, table bloat, poison messages, and the metrics that catch them.
Closed Is Not Fixed: Why Closure Rate Is the Wrong Incident Metric
Ticket closure rate rewards silencing symptoms, and automation industrializes it. A replacement scorecard: recurrence rate, time-to-recurrence, verified-fix rat
Five Strangers, One Architecture: The Emerging Spec for AI in Production Operations
Six unaffiliated engineers building AI for operations converged on the same five constraints. What that consensus is, why it happened, and how to test vendors a
What Is p99 Latency? A Field Guide for Teams Whose p50 Looks Fine
What p99 latency is, why tails regress without a deploy, and a workflow for finding the cause: histograms, exemplars, per-route views, and SLOs that catch tails
Span of Control Is Five to Seven: What Fire Command Doctrine Actually Says About Running Software Incidents
Software took the incident commander title from fire service ICS and dropped the doctrine. What actually transfers - span of control, transfer of command - and
Everyone Is Building an Ops Copilot: Notes From the DIY Wave
Practitioners are assembling self-hosted LLM ops stacks. What the DIY wave gets right, where it stalls, and a build-vs-buy framework for AI operations tooling.
Your Dead Letter Queue Is Where Failures Go to Be Forgotten: An Operating Manual
Every broker doc explains configuring a dead letter queue. This is the missing operating manual: metrics, alert thresholds, failure triage, and safe redrives.
A Code Yellow Is a Lagging Indicator: The Signals That Fire Months Earlier
A Code Yellow is the bill for months of undiagnosed degradation. When to declare one, how to run it, and the signals that fire long before you need the protocol
ReadWriteOnce Does Not Mean One Pod: The Misconfig That Waits for a Node Drain
ReadWriteOnce limits a volume to one node, not one pod. Why this Kubernetes misconfig passes every test and fails during a node drain, and how to audit for it.
The N+1 Query Problem in Production: Why It Ships Silently and How to Catch It
The N+1 query problem rarely shows up in code review. How it ships silently, how to detect it in production telemetry, and how to fix it without regressions.
Green Dashboards, Dark Grid: The 2003 Blackout as a Silent-Failure Postmortem
A millisecond race condition silently killed the alarms before the 2003 blackout. The same failure mode lives in your cron jobs, queues, and alert pipelines.
30-40% of Your Engineering Capacity Is Missing. Here Is the Math.
Accelerate, Stripe, and real team data on unplanned engineering work, turned into capacity math you can run on your own org in two weeks.
Your Provider's Postmortem Is Not Your Postmortem
Four major AWS incidents in eight months, one public postmortem. A customer-side postmortem template for turning provider outages into your own reliability revi
Stuck state: the failure mode behind both July 2 postmortems
Railway and CircleCI both published July 2, 2026 postmortems with the same hidden failure mode: state captured during instability that never corrected itself.
Reduce MTTR by attacking the 80 percent: diagnosis, not repair
About 80 percent of MTTR is spent finding the cause, not fixing it. A diagnosis-first playbook for reducing MTTR across code, database, infra, logs, and CI.
Beating On-Call Alert Fatigue: Triage by Evidence, Not by Threshold-Tuning
On-call alert fatigue is a triage problem, not a threshold problem. A practical guide to cutting page volume by investigating alerts to root cause across the wh
Replay-Verified Is Not Production-Safe: Where the Agentic Database's Autonomy Ladder Breaks
A response to 'Agentic databases aren't agentic': replaying traffic on a fork proves answer-equivalence, not production safety. Why the reviewable diff, not the
Automating Root Cause Analysis Without Giving AI Write Access to Production
A practical, vendor-neutral guide to automated root cause analysis across the whole stack, and why the safe default is patch-file-only with no write access to p
Knight Capital's 97 Unread Emails: A $460M Lesson for the AI-Agent Era
Knight Capital lost $460M in 45 minutes. The SEC order reads like a checklist for teams deploying AI agents against production: unreviewed changes, unread signa
Idle in Transaction in Postgres: What It Means, How to Find It, and Why Autovacuum Can't Save You
What idle in transaction means in Postgres, why it silently blocks autovacuum while dashboards stay green, the 60-second diagnosis SQL, and the one-line timeout
The flaky test runbook: detect, quarantine, diagnose, verify
A practical runbook that treats flaky tests as incidents: detection thresholds, a five-cause taxonomy, quarantine exit criteria, and the math of verifying a fix
The 21% Problem: Why Second Opinions are Critical for Root Cause Analysis
When Mayo Clinic re-examined referred diagnoses, 21% changed completely. Root cause analysis has the same failure mode and none of the safeguards.
Every Component Is Green and the System Is Down
Pod healthy, service healthy, endpoint healthy - and the request times out. Four real incidents where every dashboard was green, and a cross-layer protocol for
When Recovery Doesn’t Recover: Railway, CircleCI, and Stuck State
Railway and CircleCI both went down on July 2, 2026. The interesting failures came after recovery: connections and workflows that silently kept bad state.
Recovery Is Not Convergence: Anatomy of an Outage That Outlived Its Root Cause
Railway's July 2026 outage report shows three systems that captured bad state during 20 minutes of instability and held it silently. Here's the failure class.
Every On-Call Model Has an Expiry Headcount: Lessons from Monzo’s Public Record
Monzo published three on-call models at three company sizes - with pay figures and honest failure notes. A teardown of the scaling cliffs every org hits.
Automating Root Cause Analysis Without Giving AI Write Access to Production
Manual RCA eats 30-40% of engineering time. Here's how to automate the investigation across your whole stack without giving AI write access to production.
Automated Root Cause Analysis in 2026: A Vendor-Neutral, Whole-Stack Guide for Engineering Leaders
A vendor-neutral guide to automated root cause analysis across the whole stack, and why the safe default is patch-file-only with no write access to production.
The Buyer's Guide to Automated Root Cause Analysis: What the AI Answers Leave Out
A vendor-neutral buyer's guide to automated root cause analysis: the features to look for, whole-stack coverage, and why the safe default is patch-file-only, no
The Monitoring Pilot Never Flies: Aviation's Rule for Verifying AI Diagnoses
Airbus builds redundant flight computers with different chips, teams, and languages. Aviation's independence principle is the standard AI root-cause analysis sh
They Added Capacity and It Did Not Help: What WorkOS's Connection-Pool Starvation Says About Hold-and-Wait Failures
WorkOS added capacity and nothing improved. Hold-and-wait connection-pool starvation, why it self-sustains, and the audit to run on your own critical path.
When to Hire an SRE (According to People Not Selling You One)
Everyone ranking for 'when to hire an SRE' is selling the answer. A disinterested framework: the real signals, what must exist before the hire, and the alternat
Toil Went Up, Not Down: The AI-Ops Paradox in the Survey Data
Survey toil fell for years, then reversed as AI adoption went mainstream. The mechanism: verification burden, new ops surfaces, and AI aimed at the wrong loop.
Zero-Downtime Database Migrations: The Playbook and the Failure Modes Nobody Writes Down
The expand-contract playbook plus what guides skip: foreign keys, stored procedures, backup mismatches, and how to verify a live migration is safe step by step.
AI-Assisted Infra Changes Need a Paper Trail, Not a Chat Log
AI-assisted terraform and pipeline changes ship fast, but the reasoning dies with the chat session. A practical convention for keeping the why in your repo.
Retries Are Eating Your Signal: Flaky Tests Are System Bugs Wearing a Costume
Google found 84% of CI pass-to-fail transitions are flakes. Retrying hides the race conditions and shared state that later cause production incidents.
Nobody Will Cut the Requests: A Blame-Safe Playbook for Rightsizing Kubernetes at 25 Percent Utilization
Your cluster runs at 25 percent because nobody wants to own the next latency blip. A rollout playbook for cutting padded Kubernetes requests safely.
The Delhi Data Center Fire Is a Blast-Radius Audit You Haven't Run Yet
A fire in one Delhi data center destroyed decades of customer data and degraded Google Cloud for weeks. Eight audit questions to ask about your own blast radius
The Pipeline Returned 200 OK and Did Nothing: The Agent Failure Class Your Monitoring Reports as Healthy
Agent pipelines fail while every dashboard stays green because 200 OK measures transport, not work. Five shapes of silent failure and how to detect each.
Schema Drift Is an Incident Waiting for a Deploy
Schema drift turns deploys, failovers, and restores into incidents. How drift happens, what Snowflake's 13-hour outage shows, and how to detect and triage it.
Your Average MTTR Is a Fiction: What 1.8 Million Outages Say About the Tail
Fresh data from 1.8 million outages: median resolution 1.9 minutes, mean 21.9. The gap explains why your MTTR dashboard misleads, and what to report instead.
Knight Capital's 97 Unread Emails: A $460M Lesson for the AI-Agent Era
Knight Capital lost $460M in 45 minutes. The SEC order reads like a checklist for teams deploying AI agents against production: unreviewed changes, unread signa
Your Observability Bill Is a Hedge Against Bad Investigations
Every cost guide says collect less telemetry. The real lever is why you store it: an investigation model that can only query pre-collected data. Fix that instea
How to Reduce MTTR: Fix the 80% of the Clock Nobody Instruments
Most MTTR guides optimize detection. The clock is lost in diagnosis. A phase-by-phase breakdown of where incident time goes and how to shrink the 80%.
Is It Us or Is It AWS? Triage for the First Fifteen Minutes of an Upstream Outage
AWS's July 16 CloudFront outage took 59 minutes to reach the status page. A triage protocol for deciding, in the first fifteen minutes, whether an incident is y
The Andon Cord Almost Never Stops the Line: What Software Got Wrong About Toyota's Most Borrowed Idea
Toyota's andon cord rarely stopped the line—it summoned help in seconds. What the real mechanics teach ops teams about escalation, freezes, and cheap signals.