Skip to content

feat(code-mode): add opt-in execution runner - #7

Merged
klauern merged 19 commits into
mainfrom
codex/code-mode-mvp
May 23, 2026
Merged

feat(code-mode): add opt-in execution runner#7
klauern merged 19 commits into
mainfrom
codex/code-mode-mvp

Conversation

@klauern

@klauern klauern commented May 22, 2026

Copy link
Copy Markdown
Owner

Add Code Mode preferences, generated stubs, examples, and an opt-in ynab_code_execute tool with read/write namespace gating.

Harden the MVP runner with AST checks, timeout and output truncation, and add replacement-mode filtering for the external MCP tool surface.

Tests cover preferences, runner behavior, resources, registration, and replacement-mode filtering.

Summary by CodeRabbit

  • New Features

    • Code Mode: run Python snippets with opt-in read/write gating, configurable timeout and output/result truncation, and optional replacement mode that limits visible tools to a small allowlist.
    • Discoverable generated stubs and curated examples via new resource endpoints.
  • Documentation

    • Detailed Code Mode README, quickstart, examples page, and design/plan notebook added.
  • Tests

    • Comprehensive tests for execution, stubs/specs, preferences, server behavior, and safety/audit rules.
  • Chores

    • Project version bumped to 2.0.0.

Review Change Stack

Add Code Mode preferences, generated stubs, examples, and an opt-in ynab_code_execute tool with read/write namespace gating.

Harden the MVP runner with AST checks, timeout and output truncation, and add replacement-mode filtering for the external MCP tool surface.

Tests cover preferences, runner behavior, resources, registration, and replacement-mode filtering.
@coderabbitai

coderabbitai Bot commented May 22, 2026

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds "Code Mode": an AST-audited in-process Python snippet runner with a gated ynab.read/ynab.write proxy, stub/spec generation, server/tool integration (execute/search), resources for stubs/examples, preferences, tests, and documentation.

Changes

Code Mode Sandbox and Tool Execution

Layer / File(s) Summary
Package initializer
src/mcp_ynab/code_mode/__init__.py
Re-exports CodeModeResult, run_code, run_search, build_spec, and generate_stubs as the package public API.
Sandbox runner with AST audit and safe execution
src/mcp_ynab/code_mode/runner.py
CodeModeResult and CodeModeAuditError; audits snippets (blocks imports, with/async-with, f-strings, dunder access, dunder-containing strings), builds ynab proxy splitting ynab.read/ynab.write, captures bounded stdout, enforces timeout, truncates logs/results, and returns structured results.
Type stub generation for IDE discovery & spec
src/mcp_ynab/code_mode/stubs.py
generate_stubs enumerates MCP tools, emits ReadNamespace/WriteNamespace async stubs with docstrings (skips execute/search), and build_spec returns a structured tool catalog; write tools can be omitted when mutations disabled.
Preferences model and env parsing
src/mcp_ynab/state.py
Adds code_mode_enabled, code_mode_mutations_enabled, code_mode_replace_tools, code_mode_timeout_s, and code_mode_max_output_chars with validation; env coercion now parses float timeouts.
MCP resources for stubs and examples
src/mcp_ynab/resources.py
Adds ynab://code-mode/stubs (returns generated stubs using prefs) and ynab://code-mode/examples (loads docs/code-mode-examples.md from candidate paths and errors when missing).
MCP tools and server integration
src/mcp_ynab/tools/code_mode.py, src/mcp_ynab/server.py
Adds execute (mutating) and search (read-only) tools, both preference-gated; execute clamps/derives timeout and forwards to run_code; search builds spec and forwards to run_search. Server imports code_mode at startup, re-exports resources and tools, and optionally restricts visible tools to {search, execute} when code_mode_replace_tools is enabled.
Comprehensive test suite
tests/test_code_mode.py, tests/test_server.py, tests/test_preferences.py, tests/test_tools.py
Tests cover run_code/run_search behavior, audit rejections, mutation gating, stdout/result truncation and metadata, generate_stubs/build_spec output, server registration and replacement-mode filtering, resource exposure and missing-file errors, and preference env parsing/defaults.
Design document and executable examples
docs/code-mode-plan.md, docs/code-mode-examples.md, src/mcp_ynab/code_mode/README.md, README.md
Design notebook, README updates, and examples document architecture, snippet contract, discovery endpoints, runner limits/risks, and include three executable snippet examples.
.beads metadata and issues
.beads/.local_version, .beads/issues.jsonl
Bumped local version and added issue entries tracking Code Mode follow-ups and hardening tasks; removed a block of internal memory notes.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Poem

🐰 I audit AST leaves for naughty imports,
I fence ynab.write with polite exhorts,
Stubs show paths and examples teach the way,
A bounded stdout, timeboxed play,
The rabbit hops — safe snippets saved today.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 25.64% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title clearly summarizes the main feature: adding an opt-in Code Mode execution runner, which is the primary objective of this changeset.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/code-mode-mvp

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2d2b37aa95

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/mcp_ynab/code_mode/runner.py Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (2)
docs/code-mode-plan.md (1)

347-351: ⚡ Quick win

Add language identifier to fenced code block.

The fenced code block is missing a language identifier. For consistency with the rest of the document and to satisfy markdown linting, specify the language.

📝 Proposed fix
 What the LLM would write today (multi-call):
-```
+```python
 get_transactions_needing_attention()
 # … LLM reads result, picks 14 entries, decides categories …
 bulk_categorize([...])
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/code-mode-plan.md` around lines 347 - 351, The fenced code block
containing get_transactions_needing_attention() and bulk_categorize([...]) lacks
a language identifier; update that code fence to include a language (e.g., add
```python) so it matches the rest of the document and satisfies markdown
linting, ensuring the block begins with ```python and ends with ``` without
changing the code inside.
src/mcp_ynab/code_mode/runner.py (1)

188-188: 💤 Low value

Avoid private _tool_manager._tools access in code mode

src/mcp_ynab/code_mode/runner.py (and src/mcp_ynab/code_mode/stubs.py) iterate mcp._tool_manager._tools. Since src/mcp_ynab/server.py already overrides mcp.list_tools to apply the code_mode_replace_tools filtering, enumerate tools in code mode via await mcp.list_tools() instead of relying on FastMCP internals.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/mcp_ynab/code_mode/runner.py` at line 188, The loop is iterating the
private FastMCP internals via mcp._tool_manager._tools; replace that with the
public async API by calling await mcp.list_tools() (and adjust the code in both
src/mcp_ynab/code_mode/runner.py and src/mcp_ynab/code_mode/stubs.py
accordingly). Specifically, stop using mcp._tool_manager._tools and instead
await mcp.list_tools(), then iterate the returned tool entries (adapting to the
returned structure) so the existing server-side code_mode_replace_tools
filtering is respected.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/mcp_ynab/code_mode/stubs.py`:
- Around line 34-38: The generated method stubs inside the class are missing the
implicit instance parameter; update the stub generator in
src/mcp_ynab/code_mode/stubs.py so the signature for instance methods includes
"self" as the first parameter (i.e., build the param list for the string for
f"async def {tool.name}(...)" by prepending "self" to params before joining),
and ensure the code handles the case where params is empty (result should be
"self" not "self, "). Use the existing symbols (tool.name, params, return_type,
description) when constructing the final stub lines.

In `@src/mcp_ynab/resources.py`:
- Line 61: The function _read_code_mode_examples() currently raises a generic
RuntimeError when the code-mode examples file isn't found; change that to raise
FileNotFoundError with the same informative message (including the searched
variable) so callers can explicitly handle missing-file cases and tests can
assert the specific exception type. Locate the raise in
src/mcp_ynab/resources.py (the line with raise RuntimeError(f"Code Mode examples
file not found; searched: {searched}")) and replace the exception type to
FileNotFoundError while preserving the message.

In `@src/mcp_ynab/tools/code_mode.py`:
- Around line 36-38: The timeout calculation can produce non-positive values
when timeout <= 0, which will make asyncio.wait_for raise immediately; in the
block that sets timeout_s (using timeout, prefs.code_mode_timeout_s) clamp the
computed value to a sensible positive minimum (e.g., max(timeout_s, 0.1) or
max(timeout_s, 1)) after applying the min(...) so timeout_s is always > 0 before
it is passed to asyncio.wait_for; update the logic that assigns timeout_s (and
any callers that use timeout or prefs.code_mode_timeout_s) to enforce this lower
bound.

---

Nitpick comments:
In `@docs/code-mode-plan.md`:
- Around line 347-351: The fenced code block containing
get_transactions_needing_attention() and bulk_categorize([...]) lacks a language
identifier; update that code fence to include a language (e.g., add ```python)
so it matches the rest of the document and satisfies markdown linting, ensuring
the block begins with ```python and ends with ``` without changing the code
inside.

In `@src/mcp_ynab/code_mode/runner.py`:
- Line 188: The loop is iterating the private FastMCP internals via
mcp._tool_manager._tools; replace that with the public async API by calling
await mcp.list_tools() (and adjust the code in both
src/mcp_ynab/code_mode/runner.py and src/mcp_ynab/code_mode/stubs.py
accordingly). Specifically, stop using mcp._tool_manager._tools and instead
await mcp.list_tools(), then iterate the returned tool entries (adapting to the
returned structure) so the existing server-side code_mode_replace_tools
filtering is respected.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5fe3c9a6-1e0c-4f5b-a97a-591d255c292f

📥 Commits

Reviewing files that changed from the base of the PR and between d9264aa and 2d2b37a.

📒 Files selected for processing (12)
  • docs/code-mode-examples.md
  • docs/code-mode-plan.md
  • src/mcp_ynab/code_mode/__init__.py
  • src/mcp_ynab/code_mode/runner.py
  • src/mcp_ynab/code_mode/stubs.py
  • src/mcp_ynab/resources.py
  • src/mcp_ynab/server.py
  • src/mcp_ynab/state.py
  • src/mcp_ynab/tools/code_mode.py
  • tests/test_code_mode.py
  • tests/test_preferences.py
  • tests/test_server.py

Comment thread src/mcp_ynab/code_mode/stubs.py Outdated
Comment thread src/mcp_ynab/resources.py Outdated
Comment thread src/mcp_ynab/tools/code_mode.py Outdated
Clamp truncated output to the configured maximum, include self in generated stub methods, raise FileNotFoundError for missing examples, and enforce a positive minimum timeout.
@klauern

klauern commented May 23, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai help

@coderabbitai

coderabbitai Bot commented May 23, 2026

Copy link
Copy Markdown

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

CodeRabbit commands

These commands are invoked using PR/Issue comments.

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai rate limit to show your current review rate limit status.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai evaluate custom pre-merge check --instructions <custom-checks-instructions> --name <custom-checks-title> [--mode <error|warning>] to validate the custom pre-merge checks instructions. Defaults to error when --mode is omitted.
  • @coderabbitai ignore pre-merge checks to override pre-merge checks and get an approval on PR.
  • @coderabbitai run pre-merge checks to run pre-merge checks on the pull request.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai generate unit tests to generate unit tests for this PR.
  • @coderabbitai emit path instructions to emit generated path-specific review instructions for this repository.
  • @coderabbitai resolve merge conflict to automatically resolve merge conflicts.
  • @coderabbitai autofix to automatically fix issues identified in unresolved review comments.
  • @coderabbitai resolve to resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai generate configuration to create a PR that adds the current resolved configuration as .coderabbit.yaml (or show it if already present).
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore or @coderabbit ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit configuration file (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • You can also validate your configuration using the online YAML validator.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, support, documentation and community

  • Visit our status page to check the current availability of CodeRabbit.
  • Create a ticket on our support page for assistance with any issues or questions.
  • Visit our documentation site for detailed information on how to use CodeRabbit.
  • Join our Discord community to connect with other users and get help from the community.
  • Follow us on X/Twitter for updates and announcements.

@klauern

klauern commented May 23, 2026

Copy link
Copy Markdown
Owner Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented May 23, 2026

Copy link
Copy Markdown
✅ Actions performed

Full review triggered.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (2)
docs/code-mode-plan.md (1)

347-351: 💤 Low value

Add language identifier to code fence.

The code block showing the multi-call approach is missing a language identifier. While this appears to be pseudo-code, adding a language tag (e.g., python or text) improves consistency and silences the markdown linter.

📝 Suggested fix
-```
+```text
 get_transactions_needing_attention()
 # … LLM reads result, picks 14 entries, decides categories …
 bulk_categorize([...])
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/code-mode-plan.md` around lines 347 - 351, The code block showing the
multi-call sequence is missing a Markdown language identifier; update the fenced
block containing get_transactions_needing_attention() and bulk_categorize([...])
to include a language tag (e.g., `text` or `python`) so the markdown linter is
satisfied and formatting is consistent—locate the fence around the calls to
get_transactions_needing_attention and bulk_categorize and add the chosen
language identifier immediately after the opening ``` fence.
tests/test_server.py (1)

205-214: ⚡ Quick win

Avoid brittle minimum-length coupling in examples-content test.

Line 210 (assert len(text.strip()) > 500) can fail on harmless docs edits; validating required namespace
snippets is enough for this contract.

Suggested diff
 def test_code_mode_examples_resource_uses_current_namespaces() -> None:
     content = server.get_code_mode_examples()
     assert len(content) == 1
 
     text = content[0].text
-    assert len(text.strip()) > 500
     assert "ynab.read.get_budgets" in text
     assert "ynab.read.get_transactions_needing_attention" in text
     assert "ynab.write.bulk_categorize" in text
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_server.py` around lines 205 - 214, The test
test_code_mode_examples_resource_uses_current_namespaces is brittle because it
asserts a minimum content length via assert len(text.strip()) > 500; remove that
length-based assertion in the block that calls server.get_code_mode_examples()
and instead rely on verifying required namespace snippets (e.g.,
"ynab.read.get_budgets", "ynab.read.get_transactions_needing_attention",
"ynab.write.bulk_categorize") are present in text; keep the existing checks that
the examples list has one item and that those namespace strings are in text to
validate the contract without coupling to document length.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/code-mode-examples.md`:
- Line 8: Update the docs line about the LIMIT constant to explicitly state its
purpose and default value: mention that LIMIT is used to keep returned data
small and that LIMIT defaults to 10 (or the actual default used in the codebase)
so readers know what to expect; reference the LIMIT constant in the sentence and
add a short parenthetical or clause like "LIMIT defaults to 10" to the existing
line in docs/code-mode-examples.md.

In `@src/mcp_ynab/code_mode/runner.py`:
- Around line 231-232: The current await asyncio.wait_for(main(),
timeout=timeout_s) inside the contextlib.redirect_stdout(logs_buffer) block
cannot abort synchronous/blocking user code; to enforce timeout_s reliably run
the user-executing main() in an isolated worker process (or a separate OS
thread/process via multiprocessing.Process or
concurrent.futures.ProcessPoolExecutor) and kill/terminate that worker if
timeout_s elapses. Change the runner to spawn the worker that executes main()
with its stdout/stderr captured (redirect logs_buffer into the child or pipe
stdout back), wait for the worker with a hard timeout, and on timeout terminate
the worker and return a timeout error; reference main(), timeout_s, and the
contextlib.redirect_stdout(logs_buffer) usage to locate where to replace the
await asyncio.wait_for call.
- Around line 102-118: The function _truncate_result currently returns the raw
result in the early-return paths which can include non-JSON-safe values; change
it to always use the JSON-safe serialized form from _serialize_result.
Specifically, call _serialize_result(result) for all cases (including when
max_chars < 0 and when len(serialized) <= max_chars) and return that serialized
value along with the truncated flag (False in non-truncated cases); keep the
existing truncated branch that returns the preview from _truncate(serialized,
max_chars). Ensure this makes CodeModeResult.result JSON-safe for
model_dump(mode="json").
- Around line 221-228: The audit currently parses the raw snippet via
_audit_code(code, ...) which can miss syntax errors introduced by wrapping;
change the flow to call _audit_code on the wrapped body instead by first
generating wrapped = _wrap_code(code) and then invoking _audit_code(wrapped,
mutations_enabled=mutations_enabled) before compiling; ensure you still pass the
same flags and that errors raised are CodeModeAuditError so invalid constructs
(e.g., break/continue) are caught during audit rather than at compile().

---

Nitpick comments:
In `@docs/code-mode-plan.md`:
- Around line 347-351: The code block showing the multi-call sequence is missing
a Markdown language identifier; update the fenced block containing
get_transactions_needing_attention() and bulk_categorize([...]) to include a
language tag (e.g., `text` or `python`) so the markdown linter is satisfied and
formatting is consistent—locate the fence around the calls to
get_transactions_needing_attention and bulk_categorize and add the chosen
language identifier immediately after the opening ``` fence.

In `@tests/test_server.py`:
- Around line 205-214: The test
test_code_mode_examples_resource_uses_current_namespaces is brittle because it
asserts a minimum content length via assert len(text.strip()) > 500; remove that
length-based assertion in the block that calls server.get_code_mode_examples()
and instead rely on verifying required namespace snippets (e.g.,
"ynab.read.get_budgets", "ynab.read.get_transactions_needing_attention",
"ynab.write.bulk_categorize") are present in text; keep the existing checks that
the examples list has one item and that those namespace strings are in text to
validate the contract without coupling to document length.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 00c5d12d-d193-48b1-bd76-8e9f1ca7432c

📥 Commits

Reviewing files that changed from the base of the PR and between d9264aa and d32eed2.

📒 Files selected for processing (12)
  • docs/code-mode-examples.md
  • docs/code-mode-plan.md
  • src/mcp_ynab/code_mode/__init__.py
  • src/mcp_ynab/code_mode/runner.py
  • src/mcp_ynab/code_mode/stubs.py
  • src/mcp_ynab/resources.py
  • src/mcp_ynab/server.py
  • src/mcp_ynab/state.py
  • src/mcp_ynab/tools/code_mode.py
  • tests/test_code_mode.py
  • tests/test_preferences.py
  • tests/test_server.py

Comment thread docs/code-mode-examples.md Outdated
Comment thread src/mcp_ynab/code_mode/runner.py
Comment thread src/mcp_ynab/code_mode/runner.py Outdated
Comment thread src/mcp_ynab/code_mode/runner.py
klauern added 3 commits May 22, 2026 19:52
- _truncate_result: always round-trip through JSON so CodeModeResult.result
  is JSON-safe for model_dump(mode="json"); non-serializable values are
  coerced via default=str and parsed back to native Python types
- run_code: audit the wrapped async body instead of the raw snippet so
  constructs like break/continue outside loops are caught as
  CodeModeAuditError rather than a bare SyntaxError at compile time
- docs/code-mode-examples.md: clarify LIMIT defaults to 100
- Track mcp-ynab-fkv (P3 bug): asyncio.wait_for cannot enforce timeout
  against synchronous blocking user code
- README.md: document ynab://code-mode/* resources and Code Mode section
- code_mode/README.md: usage guide, wiring, and runner limits
Create umbrella tracking for post-MVP Code Mode work:

- mcp-ynab-fsv (epic, P2): Code Mode (MVP follow-through)
  - fsv.1 (P2): Runner hardening — subprocess isolation + OS resource limits
  - fsv.2 (P3): Replacement-mode visible-toolset policy
  - fsv.3 (P3): Stub generator quality — return types and model schemas
  - fsv.4 (P3): Expand ynab://code-mode/examples with workflow snippets

The previously-filed mcp-ynab-fkv (asyncio.wait_for timeout bug from PR #7
review) is reparented under fsv.1; the subprocess rewrite that fsv.1
prescribes is a strict superset of the timeout fix, so fkv closes naturally
when fsv.1 lands.

Maps the four "Still open" bullets in docs/code-mode-plan.md §0 to concrete
beads tasks so the work is trackable after this PR merges.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an opt-in “Code Mode” execution path to the mcp-ynab server, letting MCP clients run gated Python snippets via a single ynab_code_execute tool and discover the API via generated stubs/examples resources.

Changes:

  • Introduces ynab_code_execute plus Code Mode runner, stub generation, and ynab://code-mode/* resources.
  • Adds preferences/env parsing for Code Mode enablement, write gating, replacement-mode filtering, and execution limits.
  • Adds tests covering Code Mode registration, filtering behavior, runner auditing/truncation, and new preferences.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
tests/test_server.py Verifies Code Mode resources/tools are exposed and replacement-mode filtering works.
tests/test_preferences.py Adds assertions for new Code Mode preference defaults and env float parsing.
tests/test_code_mode.py New tests for runner behavior, stub generation, and ynab_code_execute preference/timeout handling.
src/mcp_ynab/tools/code_mode.py Implements the ynab_code_execute MCP tool wrapper around the runner.
src/mcp_ynab/state.py Adds Code Mode preferences and float coercion for env/tool-set values.
src/mcp_ynab/server.py Registers Code Mode module and adds tool-surface filtering for replacement mode.
src/mcp_ynab/resources.py Adds ynab://code-mode/stubs and ynab://code-mode/examples resources.
src/mcp_ynab/code_mode/stubs.py Generates Python .pyi-style stubs from the tool registry.
src/mcp_ynab/code_mode/runner.py In-process audited snippet runner with read/write proxy, timeout, and truncation.
src/mcp_ynab/code_mode/README.md Documents Code Mode usage, preferences, discovery resources, and runner limits.
src/mcp_ynab/code_mode/init.py Exposes Code Mode public API (run_code, generate_stubs, result model).
README.md Documents new Code Mode resources and adds a Code Mode section.
docs/code-mode-plan.md Adds an MVP design/plan notebook describing scope, risks, and follow-ups.
docs/code-mode-examples.md Adds curated snippet examples served via the examples resource.
.beads/issues.jsonl Tracks new follow-up issues for runner hardening and related Code Mode work.
.beads/.local_version Bumps local beads version.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread README.md Outdated
Comment thread src/mcp_ynab/resources.py
Comment thread src/mcp_ynab/code_mode/runner.py Outdated
Comment thread src/mcp_ynab/code_mode/runner.py Outdated
Comment thread src/mcp_ynab/code_mode/README.md Outdated
Comment thread src/mcp_ynab/code_mode/stubs.py
- README.md: fix absolute local path link → relative repo path
- code_mode/README.md: document soft-timeout limitation (cannot stop
  blocking/CPU-bound code; see mcp-ynab-fkv)
- runner.py: replace unbounded io.StringIO with _BoundedStringIO that
  stops accepting writes at max_output_chars, bounding memory during
  execution rather than truncating after the fact; add soft-timeout
  comment on asyncio.wait_for call
- stubs.py: isolate mcp._tool_manager._tools access behind _iter_mcp_tools()
  adapter so FastMCP private API upgrades only need one fix point
- tests: update truncation assertion to check content + marker presence
  rather than exact byte count (bounded-buffer contract differs from
  the old post-hoc _truncate contract)
- beads: track docs packaging issue as mcp-ynab-h27 (P3 bug)

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/mcp_ynab/code_mode/runner.py (1)

219-231: 💤 Low value

Consider consolidating private MCP API access.

stubs.py now isolates mcp._tool_manager._tools access behind _iter_mcp_tools() to create a single fix point for MCP upgrades, but _build_ynab_proxy still accesses the private API directly (line 222). Consider reusing _iter_mcp_tools or extracting a shared helper to maintain a single fix point.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/mcp_ynab/code_mode/runner.py` around lines 219 - 231, _build_ynab_proxy
currently reads mcp._tool_manager._tools directly; change it to use the
centralized iterator helper (_iter_mcp_tools) provided in stubs (or extract a
shared helper) so there is a single fix point for MCP internals; specifically,
replace the for loop over mcp._tool_manager._tools in _build_ynab_proxy with
iteration from _iter_mcp_tools(mcp) and keep the existing logic that selects
target = write if _is_mutating_tool(tool) else read and calls _bind_tool(tool,
ctx), while preserving the mutations_enabled branch that may replace write with
_DisabledWriteNamespace.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/mcp_ynab/code_mode/runner.py`:
- Around line 219-231: _build_ynab_proxy currently reads
mcp._tool_manager._tools directly; change it to use the centralized iterator
helper (_iter_mcp_tools) provided in stubs (or extract a shared helper) so there
is a single fix point for MCP internals; specifically, replace the for loop over
mcp._tool_manager._tools in _build_ynab_proxy with iteration from
_iter_mcp_tools(mcp) and keep the existing logic that selects target = write if
_is_mutating_tool(tool) else read and calls _bind_tool(tool, ctx), while
preserving the mutations_enabled branch that may replace write with
_DisabledWriteNamespace.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 1ca307eb-8bc4-425f-9e59-4bf1daee9983

📥 Commits

Reviewing files that changed from the base of the PR and between d32eed2 and 20746b2.

📒 Files selected for processing (8)
  • .beads/.local_version
  • .beads/issues.jsonl
  • README.md
  • docs/code-mode-examples.md
  • src/mcp_ynab/code_mode/README.md
  • src/mcp_ynab/code_mode/runner.py
  • src/mcp_ynab/code_mode/stubs.py
  • tests/test_code_mode.py
✅ Files skipped from review due to trivial changes (4)
  • .beads/.local_version
  • README.md
  • src/mcp_ynab/code_mode/README.md
  • docs/code-mode-examples.md

Rename ynab_code_execute → execute and add a new search tool that runs
LLM-authored code against a spec/catalog object (no live YNAB access).
The visible surface now defaults to exactly {search, execute}; the full
34-tool surface is available via the escape-hatch preference
code_mode_replace_tools=false.

- runner.py: extract _run_snippet helper; add run_search injecting spec global
- stubs.py: add build_spec returning structured tool catalog
- tools/code_mode.py: rename execute, add search tool wiring run_search + build_spec
- server.py: _CODE_MODE_REPLACEMENT_VISIBLE_TOOLS → frozenset({"search","execute"})
- state.py: code_mode_enabled and code_mode_replace_tools default to True
- pyproject.toml: bump to 2.0.0
- README: add Quickstart with YNAB_API_KEY setup and search→execute workflow
- tests: rename refs, add search/build_spec tests, add visible-set regression

BREAKING CHANGE: server defaults to 2-tool surface; set
code_mode_replace_tools=false to restore the previous direct-tool surface.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
src/mcp_ynab/server.py (1)

226-241: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Preserve bootstrap tools in replacement-mode visibility.

Line 241 currently filters external tools to only search and execute. With code_mode_replace_tools
defaulting to True (src/mcp_ynab/state.py, Line 113), this drops bootstrap tools and conflicts with
the documented contract (“bootstrap tools plus search and execute”).

Suggested fix
+_CODE_MODE_BOOTSTRAP_VISIBLE_TOOLS = frozenset(
+    {
+        "get_preferences",
+        "set_preference",
+        "set_api_key",
+        "clear_api_key",
+    }
+)
-_CODE_MODE_REPLACEMENT_VISIBLE_TOOLS = frozenset({"search", "execute"})
+_CODE_MODE_REPLACEMENT_VISIBLE_TOOLS = (
+    frozenset({"search", "execute"}) | _CODE_MODE_BOOTSTRAP_VISIBLE_TOOLS
+)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/mcp_ynab/server.py` around lines 226 - 241, The current filter in
_list_tools_with_code_mode_filter only returns tools whose name is in
_CODE_MODE_REPLACEMENT_VISIBLE_TOOLS, which drops required bootstrap tools when
code_mode_replace_tools is enabled; update the filter so it preserves bootstrap
tools as well (e.g., return [tool for tool in tools if tool.name in
_CODE_MODE_REPLACEMENT_VISIBLE_TOOLS or getattr(tool, "is_bootstrap", False)]),
or alternatively expand _CODE_MODE_REPLACEMENT_VISIBLE_TOOLS to include known
bootstrap tool names; make this change inside _list_tools_with_code_mode_filter
and ensure it honors ynab_resources.preferences.code_mode_replace_tools.
src/mcp_ynab/code_mode/runner.py (1)

243-274: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Align log truncation with the existing max_output_chars contract.

This path now forwards max_output_chars into _BoundedStringIO, but that buffer does not follow the same rules as _truncate/_truncate_result: negative values behave like “no capacity” instead of “unlimited”, and successful runs can return logs longer than the configured cap because the suffix is appended after the buffer is full.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/mcp_ynab/code_mode/runner.py` around lines 243 - 274, The logs truncation
behavior is inconsistent because _run_snippet uses
_BoundedStringIO(max_output_chars) which treats negative caps as "no capacity"
and appends suffix after fill; instead, instantiate _BoundedStringIO with a
positive capacity only when max_output_chars >= 0 (or pass a sentinel/None for
unlimited) but do not rely on its truncated flag; after capturing full logs from
logs_buffer.getvalue(), call _truncate(logs, max_output_chars) (the same helper
used on timeout) and use its returned truncated boolean to set
CodeModeResult.truncated, and ensure result truncation still uses
_truncate_result; update references in _run_snippet to stop using
logs_buffer.truncated and to consistently use _truncate(...) so negative
max_output_chars means unlimited and suffix handling matches the contract.
🧹 Nitpick comments (1)
src/mcp_ynab/code_mode/stubs.py (1)

77-97: ⚡ Quick win

Give build_spec a concrete return type.

list[dict] hides the fixed schema you introduced here, so callers lose static checking for required keys like "name" and "namespace". A small TypedDict would make this contract much clearer.

Possible typing cleanup
+from typing import Any, Literal, TypedDict
-
-from typing import Any
+
+
+class BuildSpecEntry(TypedDict):
+    name: str
+    namespace: Literal["read", "write"]
+    signature: str
+    doc: str
+    returns: str
...
-def build_spec(mcp: Any, *, mutations_enabled: bool = True) -> list[dict]:
+def build_spec(mcp: Any, *, mutations_enabled: bool = True) -> list[BuildSpecEntry]:
     """Return a structured catalog of available tools for the search sandbox."""
-    entries = []
+    entries: list[BuildSpecEntry] = []
As per coding guidelines, "Use type hints consistently with modern Python typing".
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/mcp_ynab/code_mode/stubs.py` around lines 77 - 97, Define a TypedDict for
the fixed schema and update build_spec's return annotation to use it: create a
ToolSpec (TypedDict) with keys "name": str, "namespace": str, "signature": str,
"doc": str, and "returns": str; import TypedDict (and Optional if needed) and
replace the function signature def build_spec(...)-> list[dict] with def
build_spec(...)-> list[ToolSpec]; ensure the entries.append() payload created in
build_spec (which uses _iter_mcp_tools, _is_mutating_tool, inspect.signature,
_format_param, and _annotation_name) conforms to the TypedDict types so static
checkers can validate callers.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@README.md`:
- Around line 56-58: The fenced code block in README.md is missing a language
label and triggers MD040; update the unlabeled block (the triple-backtick block
containing "set_preference: code_mode_replace_tools = false") to include a
language identifier (e.g., add "text" after the opening ``` to become ```text)
so the markdown linter stops reporting `fenced-code-language`.

---

Outside diff comments:
In `@src/mcp_ynab/code_mode/runner.py`:
- Around line 243-274: The logs truncation behavior is inconsistent because
_run_snippet uses _BoundedStringIO(max_output_chars) which treats negative caps
as "no capacity" and appends suffix after fill; instead, instantiate
_BoundedStringIO with a positive capacity only when max_output_chars >= 0 (or
pass a sentinel/None for unlimited) but do not rely on its truncated flag; after
capturing full logs from logs_buffer.getvalue(), call _truncate(logs,
max_output_chars) (the same helper used on timeout) and use its returned
truncated boolean to set CodeModeResult.truncated, and ensure result truncation
still uses _truncate_result; update references in _run_snippet to stop using
logs_buffer.truncated and to consistently use _truncate(...) so negative
max_output_chars means unlimited and suffix handling matches the contract.

In `@src/mcp_ynab/server.py`:
- Around line 226-241: The current filter in _list_tools_with_code_mode_filter
only returns tools whose name is in _CODE_MODE_REPLACEMENT_VISIBLE_TOOLS, which
drops required bootstrap tools when code_mode_replace_tools is enabled; update
the filter so it preserves bootstrap tools as well (e.g., return [tool for tool
in tools if tool.name in _CODE_MODE_REPLACEMENT_VISIBLE_TOOLS or getattr(tool,
"is_bootstrap", False)]), or alternatively expand
_CODE_MODE_REPLACEMENT_VISIBLE_TOOLS to include known bootstrap tool names; make
this change inside _list_tools_with_code_mode_filter and ensure it honors
ynab_resources.preferences.code_mode_replace_tools.

---

Nitpick comments:
In `@src/mcp_ynab/code_mode/stubs.py`:
- Around line 77-97: Define a TypedDict for the fixed schema and update
build_spec's return annotation to use it: create a ToolSpec (TypedDict) with
keys "name": str, "namespace": str, "signature": str, "doc": str, and "returns":
str; import TypedDict (and Optional if needed) and replace the function
signature def build_spec(...)-> list[dict] with def build_spec(...)->
list[ToolSpec]; ensure the entries.append() payload created in build_spec (which
uses _iter_mcp_tools, _is_mutating_tool, inspect.signature, _format_param, and
_annotation_name) conforms to the TypedDict types so static checkers can
validate callers.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6ce06955-3dcf-4488-a207-eb9f8217a075

📥 Commits

Reviewing files that changed from the base of the PR and between 20746b2 and 72948d6.

⛔ Files ignored due to path filters (1)
  • uv.lock is excluded by !**/*.lock
📒 Files selected for processing (16)
  • .beads/issues.jsonl
  • README.md
  • docs/code-mode-examples.md
  • docs/code-mode-plan.md
  • pyproject.toml
  • src/mcp_ynab/code_mode/README.md
  • src/mcp_ynab/code_mode/__init__.py
  • src/mcp_ynab/code_mode/runner.py
  • src/mcp_ynab/code_mode/stubs.py
  • src/mcp_ynab/server.py
  • src/mcp_ynab/state.py
  • src/mcp_ynab/tools/code_mode.py
  • tests/test_code_mode.py
  • tests/test_preferences.py
  • tests/test_server.py
  • tests/test_tools.py
✅ Files skipped from review due to trivial changes (4)
  • pyproject.toml
  • docs/code-mode-examples.md
  • src/mcp_ynab/code_mode/README.md
  • docs/code-mode-plan.md

Comment thread README.md Outdated
klauern added 2 commits May 23, 2026 14:52
- Copy docs/code-mode-examples.md into src/mcp_ynab/code_mode/examples.md
  so it is included in the installed wheel (hatchling packages all files
  under src/ by default)
- resources._read_code_mode_examples() now tries importlib.resources first
  (works in installed wheel) with a filesystem fallback for dev environments
- README.md: add 'text' language to unlabeled fenced code block (MD040)
- Update missing-file test to also patch _resource_files

Fixes mcp-ynab-h27.
Missing blank line flagged by ruff format --check in CI.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/test_server.py`:
- Around line 250-255: The test block around the simulated missing package data
is misformatted for Ruff/Black; reformat the block to satisfy ruff format
(Black-compatible) by adjusting whitespace and indentation around the _raise
function, the monkeypatch.setattr call (targeting
server.resources._resource_files), and the assignment original_is_file =
Path.is_file so the function definition and subsequent statements follow Black
style; run `uv run ruff format --check src/ tests/` locally and apply the
formatter (uv run ruff format) until the test_server.py block with _raise,
monkeypatch.setattr(server.resources, "_resource_files", _raise), and
original_is_file = Path.is_file passes.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 47280543-c823-45a6-95a0-1864f2ef318a

📥 Commits

Reviewing files that changed from the base of the PR and between 72948d6 and 2e22ca8.

📒 Files selected for processing (4)
  • README.md
  • src/mcp_ynab/code_mode/examples.md
  • src/mcp_ynab/resources.py
  • tests/test_server.py
✅ Files skipped from review due to trivial changes (2)
  • README.md
  • src/mcp_ynab/code_mode/examples.md

Comment thread tests/test_server.py
klauern added 7 commits May 23, 2026 15:21
…ative output cap

- Expand _CODE_MODE_REPLACEMENT_VISIBLE_TOOLS to include the 6 bootstrap
  tools (ping, set_preferred_budget_id, clear_api_key, get_preferences,
  set_api_key, set_preference) so they remain callable when
  code_mode_replace_tools=True. Fixes the documented contract.

- Fix _BoundedStringIO to treat negative max_chars as unlimited instead
  of immediately truncating all writes. Maps negative values to a large
  sentinel so write() logic stays unchanged.

Addresses CodeRabbit review findings (major + minor).
…listing

FastMCP._setup_handlers captures bound method references at construction time,
so assigning to mcp.list_tools only affects direct Python callers (tests) but
not the MCP protocol layer, which routes through _mcp_server.request_handlers.
Replace both ListToolsRequest and CallToolRequest handlers so the code-mode
replacement filter is enforced on the actual MCP protocol path, not just the
monkey-patched instance attribute.
Wraps CategoriesApi.update_category() to expose category mutation via MCP.
Supports renaming, note update, and moving to a different category group.
At least one field must be provided; idempotent when same values applied.

Audit (mcp-ynab-zt1) confirmed YNAB API v1 does not support category
creation or goal-setting via API — only name/note/group updates are
writable through SaveCategory.
@klauern
klauern merged commit e44b710 into main May 23, 2026
2 checks passed
@klauern
klauern deleted the codex/code-mode-mvp branch May 23, 2026 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants