DEV Community: Maaz Kazi

For an agent that runs on a schedule, silence has to mean no

Maaz Kazi — Wed, 07 Oct 2026 05:08:03 +0000

Locus gives you persistent, named AI teammates that work on a real computer of their own: a container with a browser, a filesystem and a terminal. Your own API key, your own hardware, no account and no central service. They can also run on a schedule, without you watching.

That last part is where the interesting design problem is.

The approval layer, and why it lets things through

Every tool call an Operative makes passes a classifier before it executes. Deletes, exfiltration, publishing, remote access and permission changes stop at high danger. Installs and system writes stop at medium. Reads and GET fetches go straight through.

That last line is a deliberate design decision and it is the one people argue with. It would be easy to ask about everything, and it would be worse. A prompt that fires constantly trains the person to dismiss prompts without reading them, and once that habit exists the layer is decorative. Letting reads through is what buys the right to interrupt on a delete.

Two smaller choices in the same spirit:

Denial is injected back as a tool result, not thrown as an error. The model sees that it was refused and reports what it could not do, instead of failing opaquely or trying a different route to the same place.
"Always allow" keys on the program name, not the full command. Approving an exact string means you are approving a string; the next invocation with one flag changed is a new decision the user never made.

Then you add a scheduler and the whole thing breaks

Routines run the same Operative, with the same tools, at 3am, with nobody there. So what happens when a scheduled run hits an approval?

The tempting answers are all wrong. Auto-approve, and the safety layer only exists when it is not needed. Auto-approve just the medium ones, and you have invented a second, laxer policy that nobody reads. Queue it and wait, and you have an agent holding a half-finished action open for eight hours.

Silence is refusal, not consent. A routine that needs approval and gets no answer within five minutes stops, records needed_approval, and reports. It does not act.

One turn path, so the policy cannot drift

The reason a routine cannot quietly behave differently from a person asking is that there is no second implementation for it to drift from. A human message, a scheduled run and a handoff from another Operative all drive the same turn code. Approval, tool dispatch, memory writes, error handling: one path.

This sounds like tidiness and it is actually the safety property. Any system where "unattended mode" is a separate code path will eventually have an unattended mode with weaker rules, because that is the path nobody is looking at.

Testing it the only way that counts

The test for this is not that the refusal was logged. Logs are what you check after something has already happened.

The test fires a routine that wants to delete something, lets the approval go unanswered, and then asserts that the command never reached the computer at all. Not that it was blocked downstream, not that it was recorded — that the container never saw it.

The handoff guard needs two halves

Operatives can hand work to each other, which introduces loops. I shipped two guards, not one: refuse a handoff to anyone already on the current chain, and cap the chain depth.

Either on its own leaves a hole. Cycle detection alone permits an unbounded chain of distinct Operatives. A depth cap alone permits a tight two-agent loop to burn the whole budget before it trips. You need both, and the second one is the one people skip.

Locus is Apache-2.0 and the repository opens at launch. Written by Maaz Kazi: https://maazkazi.com

When the prompt is the product, it does not belong in the code

Maaz Kazi — Wed, 30 Sep 2026 20:52:21 +0000

CASEY started from a complaint about resumes. A candidate gets filtered out on the strength of a document that is a static snapshot of them, written once, describing a person who has since changed. Judging someone by it is like running a predictive model on a frozen spreadsheet.

So the product thesis was: replace the document with a conversation, and replace the fixed profile with a plan that keeps changing as the person does. You talk about your background, you pick a direction, and you follow a roadmap built around who you actually are.

The hypothesis worth testing first

The first build existed to answer one question, and it was not "can we ship this". It was: can a spoken conversation carry a psychometric framework?

Career assessment instruments are questionnaires. They work, and almost nobody finishes them, because answering forty Likert items about yourself is miserable and the output feels like a horoscope with citations. The bet was that the same constructs — what someone is motivated by, what they are reaching for, where they might actually fit — could be drawn out in a conversation that does not feel like an assessment at all.

So the MVP was not a product. It was an instrument for finding out whether that held. Everything else was downstream of the answer.

The structure: phases, not a chatbot

A freeform assistant would have been easier and wrong. The journey is explicitly staged.

Discovery — situation, interests, what is actually blocking you.
Orientation — narrow to a direction.
Roadmap — a three-step plan: explore, validate, confirm.
Accountability — recurring check-ins against that plan.

Stages matter because the thing being built is a judgement about a person, accumulated over time. A model that has to infer where it is in that process from conversation history will drift, and it will drift differently for every user.

The backend owns progression. The model supports extraction, not control.

This is the rule I would keep in any product like this.

The model runs the conversation. It does not decide where the user is in the journey, and it does not decide when a stage is complete. Completion is a deterministic backend function with an explicit condition — all three roadmap steps done, for instance. The model may offer to move on. Only the function decides.

It is tempting to let the model handle this, because it can. It will read the transcript and make a sensible call most of the time. But "most of the time" is doing a lot of work in a system where the state it is judging determines what the user is shown next, what gets written to their record, and whether they are charged. A stage boundary is business logic wearing a conversational costume.

The corollary is that the model is excellent at the other half: pulling structured signal out of unstructured talk. Extraction, summarisation, insight — yes. Flow control — no.

Two models, two jobs

The coaching conversation and the memory layer are deliberately not the same model.

The conversation runs on a voice-native stack, because latency and prosody are the product when someone is talking about their career and feeling exposed. Post-session summarisation and key-insight extraction run separately, on a cheaper text model, writing into a summaries table that becomes cross-session memory.

That separation is a cost decision and an architectural one. The expensive real-time path does only what has to happen in real time. Everything reflective happens after the call ends, where latency does not matter and a smaller model is sufficient.

The title of this post

Session prompts and phase configuration live in database tables, not in application code.

This was the single best structural decision in the build, and it is the one that sounds least interesting. Coaching behaviour — how CASEY opens a Discovery session, what it is trying to establish before moving on, the tone it holds when someone says something difficult — is tunable without a deploy. An admin surface exposes it directly.

The reason is not developer convenience. It is that the prompt is the product. In a conventional app, copy is presentation and logic is behaviour. In this one, the prompt is the behaviour: it is the thing you iterate on daily, the thing a non-engineer has the best judgement about, and the thing you most want to change in response to a real session that went badly yesterday.

Putting that behind a pull request and a deploy cycle means the person with the best instinct about it needs an engineer to act, and iteration slows to the speed of releases. Putting it in a table means you fix a coaching failure the afternoon you find it.

The test for whether this applies to your product: if the thing you edit most often is a string in a prompt file, it is not code, it is content, and you are versioning it in the wrong place.

Written by Maaz Kazi, who built CASEY. More at https://maazkazi.com

Asking a vision model what and where in the same call makes it worse at both

Maaz Kazi — Fri, 25 Sep 2026 12:23:29 +0000

Handrail takes a screenshot of whatever you are stuck in, answers your question about it, and then draws an arrow on the real control you need to touch. The obvious way to build that is one call: here is the screen, here is the question, give me the answer and the coordinates.

That is how I built it first, and it is worse at both halves.

Two passes, one job each

Handrail now makes two vision calls against the same screenshot.

The answering pass. The screenshot, the question, the last few turns, and any attached files go to the model. It replies with a short answer, or a checklist when the job genuinely takes several steps, and it names the one control you need to touch.
The pointing pass. The same screenshot goes back with a single question: where is that control? Answer as coordinates.

Separating them helped for a reason that is obvious in hindsight. The answering pass is a reading and reasoning task over the whole screen. The pointing pass is a localisation task over one named target. Asking for both in one response makes the model hold a spatial answer in working memory while it composes prose, and the coordinates are the part that degrades.

It also makes failure legible. If the answer is right and the arrow is wrong, you know exactly which pass to fix.

Coordinates that survive the real world

The second pass does not return pixels. It returns a position on a 0-1000 normalised grid, which Handrail then maps onto actual screen pixels.

This sounds like a detail and it removes an entire category of bug. Screens differ in resolution, in DPI scaling, and in how many of them are plugged in. If the model returns pixels, every one of those becomes arithmetic you have to get right on someone else's hardware. Normalised coordinates are resolution-independent by definition, so DPI and multi-monitor never enter the maths at all.

The two passes also do not get the same image. Answering uses a 1600px JPEG, which is enough to read a screen and cheap to send. Locating uses a native-resolution PNG, because the thing you are pinpointing may be a 12px chevron.

The model choice this forced

Splitting the work also changed which model I could use. The default is now the cheapest one that reliably reads a screen. I tried going cheaper still, and the lite tier failed in the most dangerous way available to a screen assistant: it invented menu paths for UI that was not in the screenshot.

That is the failure mode worth designing against. A model that says it cannot see the control is recoverable. A model that confidently names a menu item that does not exist sends someone hunting through Settings for something that was never there, and the whole product exists to stop exactly that.

What stays on the machine

Since the thing is looking at your screen, the trust boundary matters more than the features.

The API key is encrypted at rest by the operating system's own keychain, not by me, and never crosses into the UI layer after setup.
Conversations and attachments are JSON on disk in the app's data directory. No database, no account, no server.
Web search is off by default, and the only thing that ever leaves is the screenshot and your question, to the model you chose.
One production dependency. Everything else is Electron and Node built-ins.

A screen assistant that phones home is a different product, and a worse one.

Handrail is open source, Apache-2.0, with tagged Windows and macOS releases: https://github.com/M19K/handrail

It agreed with the reference 100% of the time. It was right 75% of the time.

Maaz Kazi — Wed, 23 Sep 2026 20:41:20 +0000

Swap a cheap model in behind an expensive one and the obvious way to check it is shadow traffic: send the same request to both, compare the answers, count how often they agree. High agreement, ship it.

I measured that on 60 live calls while building SuperRouter. The routed model agreed with the reference 100% of the time. It was correct 75% of the time.

Both numbers are real. The gap between them is the whole problem.

Agreement is not correctness

Agreement asks whether two models produced the same answer. It never asks whether the answer was right. And two models from the same era, trained on overlapping data, fail in the same direction far more often than they fail independently, so agreement is highest exactly where it protects you least.

A shadow run reporting 100% is consistent with two different worlds: a cheaper model that is genuinely as good, and a cheaper model that is wrong in precisely the same places as the expensive one. Nothing in the agreement number separates them.

What I do instead

Score against ground truth, per failure mode, and never against the other model.

Ground truth is generated, not hand-labelled. Faults are planted deliberately, so the right answer is known before any model sees the case.
A planted defect has to move pixels. One of 18 fault classes returned success and changed nothing on screen. Every fixture is now gated against a healthy frame of the same screen, because a fault no model could possibly have caught was quietly inflating every score.
Both halves of the exam have to be hard. The faithful cases were verbatim copies of the source, so false-alarm rates sat at 0-3% across seven models and that axis measured nothing at all.
Runs from different versions of a set are never compared. Every run fingerprints the exact cases it sat. Without that, the table ranked a model measured on 90 easy cases above one measured on 592.

The part that surprised me

I assumed a published leaderboard could stand in for measuring your own product. Across two products, rank order mostly transfers when a model is judging (0.83), but every model dropped a median 22 points in absolute terms. When the task was pointing at the right control rather than judging, the order barely transferred at all (0.49).

So a leaderboard tells you roughly who is good in general. It does not tell you who is good at your thing, and the second question is the only one that decides your bill.

The takeaway

If you are routing to save money, agreement rate is the metric you will reach for first and the one that will mislead you hardest. Measure correctness against something you constructed, split by the ways your product can actually break. Otherwise you are measuring how similar two models are and calling it quality.

SuperRouter is open source, Apache-2.0, with no runtime dependencies: https://github.com/M19K/superrouter