For architecture and design, see OVERVIEW.md. This document covers how
to build, run tests, read output, and debug failures.
Always use ./ct.sh or ./ct-automation.sh to run tests. Never use
cargo test directly. These scripts handle feature flags
(--features sim-server), output capture (--nocapture), log rotation, and
test ordering.
./cb.sh— Build the test binary without running tests. Passes extra args to cargo (e.g../cb.sh --release). Uses the same compilation profile as./ct.sh, so running./ct.shafter./cb.shdoes not recompile../ct.sh— Run sim tests../ct.sh— runs all tests in normal order. This is the default../ct.sh accept_finished— starts at the first test matchingaccept_finished, wraps around through all tests. The test you care about runs first, then you confirm nothing else broke../ct.sh -o accept_finished— runs only test(s) matchingaccept_finished. Useful for isolating a single test's output.- If the argument doesn't match any test name, you get an error listing all available tests.
./ct-automation.sh— Run the complete./ct.shsuite with quiet success output. This is the preferred full-suite command for automation and LLM agents. It captures stdout and stderr together, prints onlyAll tests passedon success, and replays the complete captured log to stderr on failure.
If you must bypass the scripts, replicate what ct.sh does:
cargo test --lib --features sim-server -- --nocaptureThe --lib flag skips doc-test compilation (which adds ~14s even when there
are no doc-tests).
To start from a specific test (wraparound):
SIM_TEST_FROM=accept_finished cargo test --lib --features sim-server -- --nocaptureTo run only matching test(s):
SIM_TEST_ONLY=accept_finished cargo test --lib --features sim-server -- --nocaptureWhen a test fails, use this sequence:
- Isolate:
./ct.sh -o failing_test— run only that test while debugging. Individual tests are fast (<1s for most, ~3s for on-chain), so iteration is quick. - Full suite from that test: Once the test passes, switch to
./ct.sh failing_test(no-o). This runs all tests starting from the one you just fixed, then wraps around. The tests before it in the default order already passed on the run that discovered the failure, so there's no reason to re-run them first — start from the fix and cover the rest.
| Scope | Typical time |
|---|---|
| Single simple test | <1s |
| Single on-chain test | ~3s |
| Full sim suite (parallel) | ~8s |
Build (./cb.sh) |
~1s after a Rust edit; ~0s for hex-only changes |
| Variable | Effect |
|---|---|
SIM_TEST_FROM=name |
Start the test rotation at the first test matching name, wrap around (./ct.sh name) |
SIM_TEST_ONLY=name |
Run only test(s) matching name (./ct.sh -o name) |
SIM_TIMING=1 |
Print detailed timing for each simulation step (farm_block, new_block, push_transactions, deliver_message) |
RUST_LOG=debug |
Enable log::debug! output (normally suppressed) |
Tests are registered via test_funs() functions that return closures. All
closures are collected and executed in parallel using std::thread::scope with
a shared work queue sized to available_parallelism(). To disable a test,
comment out its res.push(...) call in the relevant test_funs() function.
For game packages, rust/tests/mod.rs is optional. When it exists, it must
export test_funs() and the build automatically includes those closures in the
internal runner; packages using a non-Rust harness do not appear in that
aggregation. The registry.json test list is reserved for internal Rust test
packages such as debug, whose bespoke factory/probe hooks are also used by the
simulator.
The output from ./ct.sh is designed to be read directly. A passing run ends
with a line like All 195 tests passed in 8.19s. A failing run prints
PANIC IN TEST: inline as each failure occurs, then ends with a summary:
--- 3 FAILED TEST(S) ---
FAIL: test_foo
some error message
FAIL: test_bar
another error message
192 passed, 3 failed in 8.42s
All tests run to completion regardless of failures — a single panic does not abort the suite. This lets you see every broken test in one run.
For automated or LLM-driven full-suite runs, use ./ct-automation.sh instead
of manually filtering this output. The wrapper suppresses a successful log but
replays the complete unmodified log to stderr if any command fails, so failure
diagnostics remain searchable.
The exit code is reliable: nonzero means at least one test panicked. Each test
prints RUNNING TEST <name> ... when it starts and <name> ... ok (<time>)
when it finishes. Failed tests print PANIC IN TEST: <name> inline instead.
Each test body is wrapped in catch_unwind. When a test panics, the runner
prints PANIC IN TEST: <name> and panic payload: with the error message
inline, then continues running remaining tests. Example mid-run output:
PANIC IN TEST: test_notification_accept_finished
panic payload: tx include failed: move_number=10 tx_name=Some("false accept transaction") ...
The PANIC IN TEST: line identifies which test panicked (even when multiple
tests run in parallel). The panic payload: line has the error details.
All failures are collected and printed again in the summary at the end.
If you need to save output for later, pipe through tee:
./ct.sh 2>&1 | tee /tmp/test-output.logUse 2>&1 because some output goes to stderr.
The simulation stalled panic message includes move_number, can_move,
and next_action. These map directly to the structured diagnosis questions:
which action was the sim loop waiting to fire, and why didn't its trigger
condition become true?
Trace both the expected hash (what was curried/committed earlier) and the actual hash (what was revealed/reconstructed). The divergence between their input data is the bug.
Simulation tests exercise the full off-chain/on-chain game lifecycle by running
two GameSession instances against a local Simulator blockchain.
For the complete GameAction catalog, explicit GameID rules, ProposeTrigger
semantics, and test-writing reference, see SIMULATOR_TESTING.md.
The sim loop advances move_number only when the next action's trigger
condition is satisfied. If a test stalls, first compare the next pending action
against the event state in LocalTestUIReceiver:
- Proposal actions wait for
channel_createdor a terminal notification for the referencedGameID. - Move actions wait for
game_accepted_idsoropponent_moved_in_gameto contain the referencedGameID. AcceptProposalis two-phase: first it waits for the proposal to arrive, then it waits for membership inProposalAcceptedGroup,InsufficientBalance, orProposalCancelledafteraccept_proposalhas been called.- Global actions such as
GoOnChain,WaitBlocks,AcceptSettlement, andCleanShutdownare unconditional once they become the next scripted action.
The sim loop panics after 200 iterations with a diagnostic message including
move_number, can_move, and the next pending action. Use that message to ask:
what event would make the next action ready, and why did the event not happen?
Common causes are using the wrong explicit GameID, waiting on a proposal that
was never delivered, or expecting AcceptProposal to resolve before the player
gets the potato.
Note: This section covers the current state of chialisp debugging, which lacks print statements and stack traces. Once those are available, prefer them over the technique below.
CLVM programs have no print/log facility. When a program crashes (raises,
returns wrong values, or hits a type error like Requires Int Argument),
the error message gives you the CLVM opcode and a NodePtr but no source
location or call stack.
The only way to probe execution is to make the program fail at a known
point using (x "MARKER" values...). The x operator raises an
exception whose payload appears in the Rust error message as
Raise(NodePtr(...)).
Critical rule: You must put the (x ...) in the return path, not
in a side binding. The chialisp compiler optimizes away unused bindings,
so this does nothing:
; WRONG — compiler optimizes this away, never executes
(assign
_dbg (x "HERE")
real_result (some_computation)
real_result
)
Instead, replace the return expression itself:
; RIGHT — this is the return value, so it must execute
(assign
intermediate (some_computation)
(x "DBG_POINT" intermediate)
)
- Pick the midpoint of the suspected code path.
- Replace the return at that point with
(x "MID" relevant_values). - Rebuild (
./cb.sh; it runstools/build-chialisp.shfirst). - Run the failing test (
./ct.sh -o test_name). - Interpret:
- If the error changes to
Raise(...)with your marker → execution reached that point. The bug is downstream. - If the error stays the same (e.g.
PathIntoAtom) → the crash is before your marker. The bug is upstream.
- If the error changes to
- Repeat, narrowing the range by half each time.
Once you've found the crashing expression, you can probe sub-expressions:
; What is this value? Is it an atom or a pair?
(x "CHECK_VAL" (strlen some_value) some_value)
Or use conditional asserts to test specific properties:
; Only crash if the value is wrong
(if (= (strlen val) 32)
(real_computation val)
(x "BAD_LEN" (strlen val) val)
)
Each chialisp change requires:
./cb.sh(runs the content-addressed Chialisp build, then rebuilds the test binary)- Run the test
tools/build-chialisp.sh automatically rebuilds when source content changes or
when any generated .hex file is missing or modified. Do not delete cache files
or run cargo clean; ordinary Cargo commands intentionally do not compile
Chialisp.
Remove all diagnostic (x ...) calls after the bug is fixed. They are
not documentation. Search for your marker prefix (e.g. DBG_) to find
them all.
-
Don't use
cargo testdirectly. Use./ct.shwhile debugging or./ct-automation.shfor an automated full-suite run. The scripts handle feature flags, output capture, and test ordering. -
Don't manually filter failure output. Read the complete
./ct.shoutput, or let./ct-automation.shcapture it and replay it on failure. The middle can contain per-test diagnostics needed to find the cause. -
Don't run tests in the background. Run
./ct.shand./cb.shin the foreground and wait for them to finish. Background execution with sleep-based polling wastes time and makes output harder to capture. -
AI agents: use
./ct-automation.shfor the final full-suite check. Use./ct.sh -o test_namefor focused debugging when direct output is useful. Give either command enough foreground time to finish; the automation wrapper keeps successful runs concise and emits the full log on failure. -
Don't use
sleepto wait for processes. When waiting for a command to finish, setblock_until_msto a value higher than the expected runtime. The tool returns as soon as the process exits or the timeout elapses, whichever comes first. Usingsleepwastes time and blocks interruption. -
On macOS/Linux, use
kill -0instead ofsleepfor waiting.sleep Nalways waits the full N seconds even if the process finished immediately. This alternative checks once per second whether the process is still running and exits as soon as it isn't:# kill -0 sends no signal — it only checks whether the PID exists for i in $(seq 60); do kill -0 <pid> 2>/dev/null || break; sleep 1; done
This is strictly better than
sleep 60: identical worst case (60s if the process truly takes that long), but returns within 1 second of process exit instead of wasting the remaining time. Use a higher count when the expected runtime is longer.kill -0is POSIX and works on macOS and Linux, not Windows.