Skip to content

feat: htl::run, and program-scoped checker state, for calling Teal from Rust (#538) - #558

Merged
ynishi merged 2 commits into
mainfrom
feat/538-run
Oct 25, 2026
Merged

ynishi merged 2 commits into
mainfrom
feat/538-run

Conversation

@ynishi

@ynishi ynishi commented Oct 25, 2026

Copy link
Copy Markdown
Owner

What changed

The first half of #538: a runner a Rust host uses to call the functions of a Teal module, and the checker fix it needs. Two commits.

1. fix: program-scoped settings per program state. A checker shared by program states made with Htl::with_checker_lua kept everything that belongs to one program in its globals: package.path, the module-name env (begin_program replaced it, crates/htl-core/src/prelude.lua), the project model's view, [lang], the deps and the lint selection. Two program states alive at once resolved each other's modules. The test runner hid it by saving and restoring the checker path per file (crates/htl-core/src/testing.rs). Now each program state has a program in the checker:

  • begin_program returns one (the checker's default settings, a fresh env) and leaves the globals alone.
  • Every call a split state makes into the prelude goes through H.with_program: enter, call, write the globals back into the program on success (not on error, so a failing setter leaves no partial value), restore. Nested entry of the same program counts depth; a second program while one is entered is an error.
  • The model view goes in as one (H.set_model), its names kept in one list (MODEL_FIELDS).
  • Rust side: Htl reaches the prelude only through checker_call on a CheckerHandle (prelude + program); pkg::TealResolver and harvest_modules go through it too; the test runner's path save/restore is gone.

2. feat: htl::run (feature async, crates/htl-core/src/run.rs): start(RunnerConfig) -> (Runner, RunnerDriver), Runner::instance(module, extra_paths) -> Instance, Instance::call(field, args) -> R, Instance::close, RunnerDriver::shutdown, errors in one CallError enum (thiserror, with sources).

  • One VM thread (current-thread runtime + LocalSet) owns one checker; an instance is a program state of its own, so module-level state is not shared between instances. Requests go over a bounded channel with oneshot replies; arguments and results convert on the VM thread, so no Lua value leaves it.
  • Before require, the module is resolved and checked inside the instance's program, so NotFound and Check { diagnostics } are values, not parsed error strings; a later failure is Runtime with the traceback.
  • Deadlines: one watchdog OS thread per runner cancels the token of a call or load that runs past it, counted from when the VM thread starts it. No preempt: the Lua hook sees a token cancelled from another thread, also inside require. Dropping a call or instance future cancels its request.
  • Instances do not search the working directory (drop_cwd_search_path).

Why this shape, and not the alternatives in the design discussion on #538: the prototype described there timed calls with a timer task on the VM's own runtime plus preempt = 1, and loads with a separate thread, because a timer on the same thread never runs while Teal is in a CPU loop. One watchdog thread covers both and drops preempt, whose cost is tasks interleaving at points the program did not mark. The prototype also left "an instance's extra paths leak into the shared checker" as a known limit; tracing it showed the module-name env was replaced too, which is the bug commit 1 fixes at its cause instead of documenting it. Limiting the runner to one live instance at a time was rejected because a host that serves requests concurrently could not use it, and the limit would not show in the types. Reusing mlua-isle's AsyncIsle was rejected because it owns one Lua state, and the runner needs one checker and a state per instance.

Acceptance

Criteria from the proposed acceptance in the #538 comment that this PR meets (2 and 3); 1 and 4 are the generated client, under Not covered. Expected values were written from the criteria before running; actual values are the assertions of the named tests, which passed (cargo test --workspace: 1660 passed, 0 failed). No CLI prints these; they are library behaviour.

# Criterion (quoted from the issue) Expected Actual Fails before
2 A call whose Teal loops forever returns CallError::Timeout at the deadline; so does a module whose top-level body loops forever, on require. Timeout for both, after the 500 ms deadline, and the runner keeps serving an_endless_loop_in_a_call_is_cut_at_the_deadline: Err(CallError::Timeout { what: "m.spin", after: 500ms }), elapsed ≥ 500 ms, then add(1, 1) on the same instance returns 2. an_endless_module_body_is_cut_at_the_deadline_of_instance: Err(CallError::Timeout { what: "require(\"endless\")", after: 500ms }), then a new instance's add(1, 1) returns 2 (crates/htl-core/src/run.rs) n/a: the runner did not exist
3 Two instances of one module do not share module-level state. each instance's counter starts from its own 0 two_instances_of_a_module_share_no_state: bump returns 1, 2 on instance a, then 1, 2 on instance b n/a: the runner did not exist. The checker part has its own before/after: with the old behaviour simulated, three tests in crates/htl-core/tests/programs.rs and the half-failed-setter unit test fail

Checks

  • Tests added or changed: crates/htl-core/src/run.rs (18 tests: value, timeouts, isolation, extra paths, Runtime with traceback, a field read that raises, NotAFunction, Check incl. a dependency's type error, NotFound incl. declaration-only, plain .lua, working directory not searched, closed is Gone, a dropped future cancels, queue time not counted, shutdown joins); crates/htl-core/tests/programs.rs (4); crates/htl-core/tests/test_runner.rs (1); unit tests in crates/htl-core/src/lib.rs (3).
  • just pre-commit (fmt, tests, clippy): its parts pass: cargo fmt --all --check clean, cargo test --workspace 1660 passed / 0 failed, cargo clippy --workspace --all-targets 0 warnings.
  • just pre-push (full build, end-to-end cases): passed (exit 0).
  • End-to-end cases added or changed: none; no CLI output changes.
  • Run through the installed binary on a project (CONTRIBUTING.md, Verification): cargo install --path crates/htl-cli (debug, into a separate root) on a project of about 2,250 lines of Teal with 97 tests: htl check --no-cache . printed htl check: 25 file(s), 0 error(s), 0 warning(s), 0 lint(s); htl test --no-cache . printed htl test: 7 file(s), 97 passed, 0 failed, 0 file(s) with errors; htl run of its entry over 6 scenarios printed output byte-identical to a reference written before this change.
  • Docs updated (README, doc comments): module doc of htl::run; doc comments of Htl::with_checker / with_checker_lua and the programs comment in prelude.lua. README unchanged: htl::run belongs in it once the generated client lands with it.

Unintended changes

none found. htl check, htl test and htl run on the project above print what they printed before.

Not covered

Refs #538

…er's globals (#538)

A checker shared by program states made with `Htl::with_checker_lua` kept
everything that belongs to one program in its own globals: `package.path`, the
module-name env (`begin_program` replaced it), the project model's view, `[lang]`,
the deps and the lint selection. Two program states alive at once therefore
resolved each other's modules, and a setter called on one changed what the other
checked. Nothing in `htl check` shows it, because that path has one state. The
test runner hid it by saving the checker's path before each file and restoring
it after, which covers a path and nothing else, and only while the files run one
after another. The runner of #538 is the first caller to hold two states at once.

Each program state now has a program in the checker: a table of the values above.

- `begin_program` returns one, copied from the checker's default program (the
  settings the checker's globals hold) with a fresh env, and leaves the globals
  alone.
- Every call a split state makes into the prelude goes through one entry,
  `H.with_program`. It puts the program's values in the globals, runs the call,
  and on success writes the globals back into the program, so a setter's change
  is what the program keeps. On error nothing is written back, so a setter that
  fails halfway leaves no partial value. Either way the checker's own values are
  restored.
- Entering the program already entered counts depth, and only the outermost leave
  writes back. Entering a second program while one is entered is an error.
- The model's view is installed in one call, `H.set_model`, with its names kept
  in one list, `MODEL_FIELDS`. An unknown name is refused, so the program never
  holds half of one model and half of another.
- On the Rust side `Htl` holds a `CheckerHandle` (the prelude table plus the
  state's program) and reaches the prelude through `checker_call` only.
  `pkg::TealResolver` and `harvest_modules` go through the same entry, so a
  resolver's checks and path changes are its own state's.
- The test runner's save and restore of the checker path is gone, and so is the
  `split` flag, now `program.is_some()`.

Left out of a program on purpose: the check store, which is shared so a module is
checked once however it was reached, and the per-call stacks (`H.requirers`,
`H.envs`), which are empty at every call's boundary. The comment above
`H.with_program` gives the reason for each.

The behaviour of the CLI and of a checker used directly is unchanged: a handle
with no program calls straight through, as before. The public API is unchanged
too; `Htl::with_checker` and `with_checker_lua` keep their signatures, and their
docs now say what the state's program holds.

Tests:

- `tests/programs.rs`: two live states resolve one name each to its own file;
  they keep their own `[lang]` and lint selection; a new state starts from the
  checker, not from a sibling; a setter reaches the later calls of its own
  state. With the old behaviour simulated, the first three fail.
- `tests/test_runner.rs`: a test-runner case for files that must not see each
  other's directories now that the path is no longer restored by hand.
- unit tests in `lib.rs`: a half-failed setter leaves its program as it was (it
  fails with the old behaviour too); one program is entered at a time, with the
  depth and the refusal of a second; `set_model` refuses a name it does not keep.

Refs #538
)

`htl::run` (feature `async`) is the boundary a Rust host uses to call into Teal: `require` a module, call a function of the record it returns, read the result back as a Rust value. A Lua state is not `Send`, and a Teal function may loop for as long as it likes, so neither can sit on a task of the host's own runtime. This is the first part of #538. The generated typed client from `[rust] exports` is a later change; this one is the runner it will sit on.

Shape:

- One dedicated VM thread, with a current-thread tokio runtime and a `LocalSet`, owns one checker. `run::start` returns a `Runner` (cheap to clone, `Send + Sync`, every method `async`) and a `RunnerDriver`. Requests wait on a bounded mpsc channel (`RunnerConfig::queue`) and are answered over a oneshot, so a caller that outruns the VM waits instead of piling up work.
- `Runner::instance(module, extra_paths)` makes a program state per instance, on top of the per-program checker state of the previous commit. Module-level state is therefore not shared between instances. The checker is shared, so a module is checked once per runner. The directories in `extra_paths` are on that instance's program alone. Instances do not search the working directory.
- `instance` resolves and checks the module in that instance's program before it calls `require`, so the three ways a load fails come back as values: `NotFound` (also a module only a `.d.tl` declares), `Check { diagnostics }` for a `.tl` with errors (a dependency's type error included), and `Runtime` only when the `require` itself raises. A plain `.lua` module is not checked. `Htl::gen_lua_unlinted` is the crate-private helper that does the check without the lint pass.
- `Instance::call(field, args)` converts the arguments, makes the call and converts the result on the VM thread. No Lua value leaves it.
- Deadlines: one watchdog OS thread per runner cancels the `CancelToken` of a call or load that passes its deadline. The deadline is counted from when the VM thread starts the request, not from when it was queued, and for `instance` it covers the `require` and not the check before it. There is no `preempt`: the Lua hook sees a token cancelled from another thread, inside `require` as well, so a CPU loop in a function and in a module body are both stopped.
- Dropping the future of a `call` or `instance` cancels its request. A request still queued is skipped, a running one is stopped by the same hook.
- `Instance::close` is explicit, because Drop must not do I/O. A closed instance answers `Gone`. `RunnerDriver::shutdown` stops and joins the VM thread and the watchdog.
- `CallError` is a `thiserror` enum that keeps its sources. `thiserror` is added to the workspace dependencies (already in the graph through mlua-isle), and tokio gains the `sync` feature.

Known limits, stated in the module doc: the deadline rests on the instruction-count hook, so it cannot cut inside a native call; a script that catches errors with `pcall` can swallow the cancel; the VM serves one request at a time, so a long request holds up those behind it (the wait is not counted against their deadlines). A module that `require`s another one at run time meets the strict searcher there and fails as `Runtime`; a typed answer for that is left for later.

Tests, in `run.rs`, each against a real module on a real runner:

- a call returns a converted value;
- an endless loop in a call, and an endless module body in `instance`, are cut at the deadline;
- two instances of a module share no state;
- an extra path is seen by its own instance alone;
- a Teal error is `Runtime` with a traceback;
- a field whose read raises is `Runtime`;
- a field that is no function is `NotAFunction`;
- a type error is `Check`, a dependency's type error included;
- an absent module, and a declaration-only module, are `NotFound`;
- a plain `.lua` module loads without a check;
- the working directory is not searched;
- a closed instance is `Gone`, and closing twice is fine;
- dropping a call stops it and the next request is served, and a request dropped in the queue is never run;
- time in the queue is not counted against the deadline;
- `shutdown` joins both threads.

Refs #538
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant