Production engineering · AI agents · SRE
Production Notes
Build AI agents you can debug, evaluate, and operate.
Practical notes on runtimes, tool calls, incident investigations, and the unglamorous controls that make agent systems useful after the demo.
Featured by CNCF: You Can’t Debug What You Can’t See — Observability for AI Agents
Start with a production problem
No need to read the whole archive. Pick the problem in front of you and follow a focused path.
Debugging
Find where an agent run went wrong
Freeze a reproducible task, inspect the trace, and replay one controlled change.
Evaluation
Test whether an RCA is trustworthy
Score evidence and named unknowns before fluency, latency, or token cost.
Architecture
Understand the agent runtime boundary
Separate the execution loop from the model, chat UI, and multi-tenant platform.
Language choice
Compare Go and Python for AI agents
Make the trade-off using deployment, concurrency, libraries, and team experience.
Operations
Interrupt or contain an active agent
Apply corrections at safe boundaries without silently dropping operator feedback.
API selection
Diagnose API, model, and latency choices
Distinguish Chat Completions, Responses, reasoning effort, retries, and tool wait.
Choose a reading path
| Path | Best for |
|---|---|
| SRE on-call starter pack | Triage, RCA, observability — 5 posts |
| SRE as service starter pack | Investigation as a service product — 5 posts |
| Go agent runtime starter pack | Runtime definition, Go vs Python, platform split — 5 posts |
| ClickHouse on AWS starter pack | Self-managed ClickHouse from first principles — full series |
| Full enterprise AI agent series | Searchable archive in full reading order |
| Service Rendered Efficiently | AI investigation as a service product for on-call |
Pocket checklists
Copy these into an incident review or agent launch review. No proprietary schemas required.
- Debug a multi-step agent — reproduce, inspect, change one thing, replay
- Review evidence-gated RCA — claims, receipts, and unknowns
- Ask whether the agent task is done — independent verification and mutation safety
- Review SRE as a service — reuse, honest partials, and handoff
Browse by topic
- AI agent runtime — execution boundaries, debugging, testing, and Go architecture
- AI agent workflows — bring-up, evidence-gated RCA, and verification
- AI agents for SRE — triage, observability, and tokenomics
- AI incident triage — on-call triage versus demo theater
- Go AI agents — language choice and platform architecture
- ClickHouse on AWS — engines, S3 storage, spot readers, and cost
- Service Rendered Efficiently — investigation product design
About these notes
I’m Sabith K S, a Principal Engineer at StackGen. I write about the systems we build, the mistakes we make, and what we can learn from them. The examples often come from building Aiden, StackGen’s platform for running agents for multiple teams, and from site reliability engineering (SRE) work.
These are engineering notes from a particular production setting, not a claim that every team needs an agent—or needs to build one in Go.
Stay in the loop — production notes on AI agents, workflows, and SRE.
Low volume — new posts and curated reading lists. Unsubscribe anytime.
Connect
- Newsletter: Subscribe on Substack · Issue #1 — Start here · RSS
- GitHub: @sks
- LinkedIn: Sabith
- Company: StackGen