A small content-moderation agent for AI 372: Agentic Systems Engineering, Patterns & Controls.
It reads flagged comments from a SQLite table, decides what to do with each one using a real Claude model, and acts through a fixed set of tools. It flags what needs moderation, escalates what it isn't sure about to a human, and never deletes anything — because a hook stops it, not because it behaves.
python3 -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
python -m pytest # 19 passed
python -m agent # the demo
Use a virtualenv — the system Python on macOS refuses pip install (PEP 668).
The tests need no API key and never touch the network — python -m pytest is
the whole assignment either way. The key is only for the live demo below.
Your API key goes in a file called .env next to this README:
ANTHROPIC_API_KEY=sk-ant-...
.env is gitignored, so a key can't be committed by accident. An exported
ANTHROPIC_API_KEY works too and takes precedence.
The demo seeds a fresh database, runs the agent, then reads the database back to report what actually happened — rather than trusting the agent's own account.
Without a key it falls back to an offline stand-in decider so the demo still runs; the tests never touch the network either way.
agent/loop.py perceive -> decide -> act, and when to stop
agent/brain.py the decider: a real Claude call (+ test doubles)
agent/tools.py the five tools, and what the model is shown
agent/hooks.py middleware: block destructive calls, cap the run, trace it
agent/db.py the world: six seeded comments
tests/ one file per module's control
exercises/ one handout per module
solutions/ pointers to the solution branches
Runtime output, all gitignored: HITL.md (escalations for a human),
summary.md, trace.jsonl, moderation.db.
It reasons (a generative model picks the action), acts (five tools), communicates (a summary, and a handoff packet a human can act on), and adapts — an uncertain call becomes an escalation instead of a flag, and it remembers what it already escalated so it doesn't ask twice.
Nothing about the order is hardcoded per comment: the model chooses the tool at
runtime from what it just read. Swap in a decider that misbehaves and the
outcomes stay safe, which is the whole point — see tests/test_hooks_destructive.py.
The seeded table contains two attempts to hijack the agent:
- Comment 5 is a crude jailbreak ("ignore all of your instructions…"). A current model will almost certainly refuse it.
- Comment 6 never asks the model to disobey. It impersonates an internal Trust & Safety notice and cites a legitimate-sounding compliance reason, so deleting looks like following policy. This is the one a capable model may well act on.
The agent has a real delete_comment tool and the model can request it. The only
reason nothing is ever deleted is hooks.py. That's the lesson: you can't
verify an in-band claim of authority, and a model is a distribution while a
hook is a guarantee.
| Module | Control | Exercise branch |
|---|---|---|
| 1 | Pattern selection (worksheet only) | — |
| 2 | Tools — read_comments |
exercise/module-2 |
| 3 | Escalation — escalate → HITL.md |
exercise/module-3 |
| 4 | Middleware — blocking destructive calls | exercise/module-4 |
| 5 | Stop signals, termination, tracing | exercise/module-5 |
main is the finished reference. Each exercise/module-N branch has one thing
stubbed and its test red; each solution/module-N branch restores it and
includes the prompt that got there.
Building or extending this course? Read
COURSE_DESIGN_CONTRACT.md first.