Repository navigation
feat: htl::run, and program-scoped checker state, for calling Teal from Rust (#538) - #558
Merged
Merged
Conversation
…er's globals (#538) A checker shared by program states made with `Htl::with_checker_lua` kept everything that belongs to one program in its own globals: `package.path`, the module-name env (`begin_program` replaced it), the project model's view, `[lang]`, the deps and the lint selection. Two program states alive at once therefore resolved each other's modules, and a setter called on one changed what the other checked. Nothing in `htl check` shows it, because that path has one state. The test runner hid it by saving the checker's path before each file and restoring it after, which covers a path and nothing else, and only while the files run one after another. The runner of #538 is the first caller to hold two states at once. Each program state now has a program in the checker: a table of the values above. - `begin_program` returns one, copied from the checker's default program (the settings the checker's globals hold) with a fresh env, and leaves the globals alone. - Every call a split state makes into the prelude goes through one entry, `H.with_program`. It puts the program's values in the globals, runs the call, and on success writes the globals back into the program, so a setter's change is what the program keeps. On error nothing is written back, so a setter that fails halfway leaves no partial value. Either way the checker's own values are restored. - Entering the program already entered counts depth, and only the outermost leave writes back. Entering a second program while one is entered is an error. - The model's view is installed in one call, `H.set_model`, with its names kept in one list, `MODEL_FIELDS`. An unknown name is refused, so the program never holds half of one model and half of another. - On the Rust side `Htl` holds a `CheckerHandle` (the prelude table plus the state's program) and reaches the prelude through `checker_call` only. `pkg::TealResolver` and `harvest_modules` go through the same entry, so a resolver's checks and path changes are its own state's. - The test runner's save and restore of the checker path is gone, and so is the `split` flag, now `program.is_some()`. Left out of a program on purpose: the check store, which is shared so a module is checked once however it was reached, and the per-call stacks (`H.requirers`, `H.envs`), which are empty at every call's boundary. The comment above `H.with_program` gives the reason for each. The behaviour of the CLI and of a checker used directly is unchanged: a handle with no program calls straight through, as before. The public API is unchanged too; `Htl::with_checker` and `with_checker_lua` keep their signatures, and their docs now say what the state's program holds. Tests: - `tests/programs.rs`: two live states resolve one name each to its own file; they keep their own `[lang]` and lint selection; a new state starts from the checker, not from a sibling; a setter reaches the later calls of its own state. With the old behaviour simulated, the first three fail. - `tests/test_runner.rs`: a test-runner case for files that must not see each other's directories now that the path is no longer restored by hand. - unit tests in `lib.rs`: a half-failed setter leaves its program as it was (it fails with the old behaviour too); one program is entered at a time, with the depth and the refusal of a second; `set_model` refuses a name it does not keep. Refs #538
) `htl::run` (feature `async`) is the boundary a Rust host uses to call into Teal: `require` a module, call a function of the record it returns, read the result back as a Rust value. A Lua state is not `Send`, and a Teal function may loop for as long as it likes, so neither can sit on a task of the host's own runtime. This is the first part of #538. The generated typed client from `[rust] exports` is a later change; this one is the runner it will sit on. Shape: - One dedicated VM thread, with a current-thread tokio runtime and a `LocalSet`, owns one checker. `run::start` returns a `Runner` (cheap to clone, `Send + Sync`, every method `async`) and a `RunnerDriver`. Requests wait on a bounded mpsc channel (`RunnerConfig::queue`) and are answered over a oneshot, so a caller that outruns the VM waits instead of piling up work. - `Runner::instance(module, extra_paths)` makes a program state per instance, on top of the per-program checker state of the previous commit. Module-level state is therefore not shared between instances. The checker is shared, so a module is checked once per runner. The directories in `extra_paths` are on that instance's program alone. Instances do not search the working directory. - `instance` resolves and checks the module in that instance's program before it calls `require`, so the three ways a load fails come back as values: `NotFound` (also a module only a `.d.tl` declares), `Check { diagnostics }` for a `.tl` with errors (a dependency's type error included), and `Runtime` only when the `require` itself raises. A plain `.lua` module is not checked. `Htl::gen_lua_unlinted` is the crate-private helper that does the check without the lint pass. - `Instance::call(field, args)` converts the arguments, makes the call and converts the result on the VM thread. No Lua value leaves it. - Deadlines: one watchdog OS thread per runner cancels the `CancelToken` of a call or load that passes its deadline. The deadline is counted from when the VM thread starts the request, not from when it was queued, and for `instance` it covers the `require` and not the check before it. There is no `preempt`: the Lua hook sees a token cancelled from another thread, inside `require` as well, so a CPU loop in a function and in a module body are both stopped. - Dropping the future of a `call` or `instance` cancels its request. A request still queued is skipped, a running one is stopped by the same hook. - `Instance::close` is explicit, because Drop must not do I/O. A closed instance answers `Gone`. `RunnerDriver::shutdown` stops and joins the VM thread and the watchdog. - `CallError` is a `thiserror` enum that keeps its sources. `thiserror` is added to the workspace dependencies (already in the graph through mlua-isle), and tokio gains the `sync` feature. Known limits, stated in the module doc: the deadline rests on the instruction-count hook, so it cannot cut inside a native call; a script that catches errors with `pcall` can swallow the cancel; the VM serves one request at a time, so a long request holds up those behind it (the wait is not counted against their deadlines). A module that `require`s another one at run time meets the strict searcher there and fails as `Runtime`; a typed answer for that is left for later. Tests, in `run.rs`, each against a real module on a real runner: - a call returns a converted value; - an endless loop in a call, and an endless module body in `instance`, are cut at the deadline; - two instances of a module share no state; - an extra path is seen by its own instance alone; - a Teal error is `Runtime` with a traceback; - a field whose read raises is `Runtime`; - a field that is no function is `NotAFunction`; - a type error is `Check`, a dependency's type error included; - an absent module, and a declaration-only module, are `NotFound`; - a plain `.lua` module loads without a check; - the working directory is not searched; - a closed instance is `Gone`, and closing twice is fine; - dropping a call stops it and the next request is served, and a request dropped in the queue is never run; - time in the queue is not counted against the deadline; - `shutdown` joins both threads. Refs #538
This was referenced Oct 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
The first half of #538: a runner a Rust host uses to call the functions of a Teal module, and the checker fix it needs. Two commits.
1.
fix:program-scoped settings per program state. A checker shared by program states made withHtl::with_checker_luakept everything that belongs to one program in its globals:package.path, the module-name env (begin_programreplaced it,crates/htl-core/src/prelude.lua), the project model's view,[lang], the deps and the lint selection. Two program states alive at once resolved each other's modules. The test runner hid it by saving and restoring the checker path per file (crates/htl-core/src/testing.rs). Now each program state has a program in the checker:begin_programreturns one (the checker's default settings, a fresh env) and leaves the globals alone.H.with_program: enter, call, write the globals back into the program on success (not on error, so a failing setter leaves no partial value), restore. Nested entry of the same program counts depth; a second program while one is entered is an error.H.set_model), its names kept in one list (MODEL_FIELDS).Htlreaches the prelude only throughchecker_callon aCheckerHandle(prelude + program);pkg::TealResolverandharvest_modulesgo through it too; the test runner's path save/restore is gone.2.
feat:htl::run(featureasync,crates/htl-core/src/run.rs):start(RunnerConfig) -> (Runner, RunnerDriver),Runner::instance(module, extra_paths) -> Instance,Instance::call(field, args) -> R,Instance::close,RunnerDriver::shutdown, errors in oneCallErrorenum (thiserror, with sources).LocalSet) owns one checker; an instance is a program state of its own, so module-level state is not shared between instances. Requests go over a bounded channel with oneshot replies; arguments and results convert on the VM thread, so no Lua value leaves it.require, the module is resolved and checked inside the instance's program, soNotFoundandCheck { diagnostics }are values, not parsed error strings; a later failure isRuntimewith the traceback.preempt: the Lua hook sees a token cancelled from another thread, also insiderequire. Dropping acallorinstancefuture cancels its request.drop_cwd_search_path).Why this shape, and not the alternatives in the design discussion on #538: the prototype described there timed calls with a timer task on the VM's own runtime plus
preempt = 1, and loads with a separate thread, because a timer on the same thread never runs while Teal is in a CPU loop. One watchdog thread covers both and dropspreempt, whose cost is tasks interleaving at points the program did not mark. The prototype also left "an instance's extra paths leak into the shared checker" as a known limit; tracing it showed the module-name env was replaced too, which is the bug commit 1 fixes at its cause instead of documenting it. Limiting the runner to one live instance at a time was rejected because a host that serves requests concurrently could not use it, and the limit would not show in the types. Reusing mlua-isle'sAsyncIslewas rejected because it owns one Lua state, and the runner needs one checker and a state per instance.Acceptance
Criteria from the proposed acceptance in the #538 comment that this PR meets (2 and 3); 1 and 4 are the generated client, under Not covered. Expected values were written from the criteria before running; actual values are the assertions of the named tests, which passed (
cargo test --workspace: 1660 passed, 0 failed). No CLI prints these; they are library behaviour.CallError::Timeoutat the deadline; so does a module whose top-level body loops forever, onrequire.Timeoutfor both, after the 500 ms deadline, and the runner keeps servingan_endless_loop_in_a_call_is_cut_at_the_deadline:Err(CallError::Timeout { what: "m.spin", after: 500ms }), elapsed ≥ 500 ms, thenadd(1, 1)on the same instance returns2.an_endless_module_body_is_cut_at_the_deadline_of_instance:Err(CallError::Timeout { what: "require(\"endless\")", after: 500ms }), then a new instance'sadd(1, 1)returns2(crates/htl-core/src/run.rs)two_instances_of_a_module_share_no_state:bumpreturns1,2on instance a, then1,2on instance bcrates/htl-core/tests/programs.rsand the half-failed-setter unit test failChecks
crates/htl-core/src/run.rs(18 tests: value, timeouts, isolation, extra paths,Runtimewith traceback, a field read that raises,NotAFunction,Checkincl. a dependency's type error,NotFoundincl. declaration-only, plain.lua, working directory not searched, closed isGone, a dropped future cancels, queue time not counted, shutdown joins);crates/htl-core/tests/programs.rs(4);crates/htl-core/tests/test_runner.rs(1); unit tests incrates/htl-core/src/lib.rs(3).just pre-commit(fmt, tests, clippy): its parts pass:cargo fmt --all --checkclean,cargo test --workspace1660 passed / 0 failed,cargo clippy --workspace --all-targets0 warnings.just pre-push(full build, end-to-end cases): passed (exit 0).cargo install --path crates/htl-cli(debug, into a separate root) on a project of about 2,250 lines of Teal with 97 tests:htl check --no-cache .printedhtl check: 25 file(s), 0 error(s), 0 warning(s), 0 lint(s);htl test --no-cache .printedhtl test: 7 file(s), 97 passed, 0 failed, 0 file(s) with errors;htl runof its entry over 6 scenarios printed output byte-identical to a reference written before this change.htl::run; doc comments ofHtl::with_checker/with_checker_luaand the programs comment inprelude.lua. README unchanged:htl::runbelongs in it once the generated client lands with it.Unintended changes
none found.
htl check,htl testandhtl runon the project above print what they printed before.Not covered
[rust] exports, andhtl rswriting the same text): the second PR for Design: typed Rust handles for the functions a Teal module exports, so a host calls them with checked argument and return types #538.requires another module which fails its check comes back asRuntime(the strict searcher raises a string); typed errors for that case are not done.htl teston a cold cache builds one extra program state per harvested test file; a cost, not a behaviour change.with_programin its traceback.Htl::with_checkergiven a split state as its checker copies the root checker's defaults, not that state's settings.Refs #538