Skip to content

Latest commit

 

History

History
329 lines (252 loc) · 12.6 KB

File metadata and controls

329 lines (252 loc) · 12.6 KB

Debugging & Testing Reference

For architecture and design, see OVERVIEW.md. This document covers how to build, run tests, read output, and debug failures.

Building and Running Tests

Use the repository build and test scripts

Always use ./ct.sh or ./ct-automation.sh to run tests. Never use cargo test directly. These scripts handle feature flags (--features sim-server), output capture (--nocapture), log rotation, and test ordering.

  • ./cb.sh — Build the test binary without running tests. Passes extra args to cargo (e.g. ./cb.sh --release). Uses the same compilation profile as ./ct.sh, so running ./ct.sh after ./cb.sh does not recompile.
  • ./ct.sh — Run sim tests.
    • ./ct.sh — runs all tests in normal order. This is the default.
    • ./ct.sh accept_finished — starts at the first test matching accept_finished, wraps around through all tests. The test you care about runs first, then you confirm nothing else broke.
    • ./ct.sh -o accept_finished — runs only test(s) matching accept_finished. Useful for isolating a single test's output.
    • If the argument doesn't match any test name, you get an error listing all available tests.
  • ./ct-automation.sh — Run the complete ./ct.sh suite with quiet success output. This is the preferred full-suite command for automation and LLM agents. It captures stdout and stderr together, prints only All tests passed on success, and replays the complete captured log to stderr on failure.

Running tests directly (without scripts)

If you must bypass the scripts, replicate what ct.sh does:

cargo test --lib --features sim-server -- --nocapture

The --lib flag skips doc-test compilation (which adds ~14s even when there are no doc-tests).

To start from a specific test (wraparound):

SIM_TEST_FROM=accept_finished cargo test --lib --features sim-server -- --nocapture

To run only matching test(s):

SIM_TEST_ONLY=accept_finished cargo test --lib --features sim-server -- --nocapture

Test debugging workflow

When a test fails, use this sequence:

  1. Isolate: ./ct.sh -o failing_test — run only that test while debugging. Individual tests are fast (<1s for most, ~3s for on-chain), so iteration is quick.
  2. Full suite from that test: Once the test passes, switch to ./ct.sh failing_test (no -o). This runs all tests starting from the one you just fixed, then wraps around. The tests before it in the default order already passed on the run that discovered the failure, so there's no reason to re-run them first — start from the fix and cover the rest.

Typical durations

Scope Typical time
Single simple test <1s
Single on-chain test ~3s
Full sim suite (parallel) ~8s
Build (./cb.sh) ~1s after a Rust edit; ~0s for hex-only changes

Environment variables

Variable Effect
SIM_TEST_FROM=name Start the test rotation at the first test matching name, wrap around (./ct.sh name)
SIM_TEST_ONLY=name Run only test(s) matching name (./ct.sh -o name)
SIM_TIMING=1 Print detailed timing for each simulation step (farm_block, new_block, push_transactions, deliver_message)
RUST_LOG=debug Enable log::debug! output (normally suppressed)

Test registration

Tests are registered via test_funs() functions that return closures. All closures are collected and executed in parallel using std::thread::scope with a shared work queue sized to available_parallelism(). To disable a test, comment out its res.push(...) call in the relevant test_funs() function.

For game packages, rust/tests/mod.rs is optional. When it exists, it must export test_funs() and the build automatically includes those closures in the internal runner; packages using a non-Rust harness do not appear in that aggregation. The registry.json test list is reserved for internal Rust test packages such as debug, whose bespoke factory/probe hooks are also used by the simulator.

Reading Test Output

The output from ./ct.sh is designed to be read directly. A passing run ends with a line like All 195 tests passed in 8.19s. A failing run prints PANIC IN TEST: inline as each failure occurs, then ends with a summary:

--- 3 FAILED TEST(S) ---

  FAIL: test_foo
  some error message

  FAIL: test_bar
  another error message

192 passed, 3 failed in 8.42s

All tests run to completion regardless of failures — a single panic does not abort the suite. This lets you see every broken test in one run.

For automated or LLM-driven full-suite runs, use ./ct-automation.sh instead of manually filtering this output. The wrapper suppresses a successful log but replays the complete unmodified log to stderr if any command fails, so failure diagnostics remain searchable.

Pass/fail

The exit code is reliable: nonzero means at least one test panicked. Each test prints RUNNING TEST <name> ... when it starts and <name> ... ok (<time>) when it finishes. Failed tests print PANIC IN TEST: <name> inline instead.

Panics

Each test body is wrapped in catch_unwind. When a test panics, the runner prints PANIC IN TEST: <name> and panic payload: with the error message inline, then continues running remaining tests. Example mid-run output:

PANIC IN TEST: test_notification_accept_finished
panic payload: tx include failed: move_number=10 tx_name=Some("false accept transaction") ...

The PANIC IN TEST: line identifies which test panicked (even when multiple tests run in parallel). The panic payload: line has the error details. All failures are collected and printed again in the summary at the end.

Saving output

If you need to save output for later, pipe through tee:

./ct.sh 2>&1 | tee /tmp/test-output.log

Use 2>&1 because some output goes to stderr.

Project-Specific Diagnostics

Simulation stalls

The simulation stalled panic message includes move_number, can_move, and next_action. These map directly to the structured diagnosis questions: which action was the sim loop waiting to fire, and why didn't its trigger condition become true?

Puzzle hash mismatches

Trace both the expected hash (what was curried/committed earlier) and the actual hash (what was revealed/reconstructed). The divergence between their input data is the bug.

Simulation Test Infrastructure

Simulation tests exercise the full off-chain/on-chain game lifecycle by running two GameSession instances against a local Simulator blockchain. For the complete GameAction catalog, explicit GameID rules, ProposeTrigger semantics, and test-writing reference, see SIMULATOR_TESTING.md.

Debugging Stalled Simulation Tests

The sim loop advances move_number only when the next action's trigger condition is satisfied. If a test stalls, first compare the next pending action against the event state in LocalTestUIReceiver:

  • Proposal actions wait for channel_created or a terminal notification for the referenced GameID.
  • Move actions wait for game_accepted_ids or opponent_moved_in_game to contain the referenced GameID.
  • AcceptProposal is two-phase: first it waits for the proposal to arrive, then it waits for membership in ProposalAcceptedGroup, InsufficientBalance, or ProposalCancelled after accept_proposal has been called.
  • Global actions such as GoOnChain, WaitBlocks, AcceptSettlement, and CleanShutdown are unconditional once they become the next scripted action.

The sim loop panics after 200 iterations with a diagnostic message including move_number, can_move, and the next pending action. Use that message to ask: what event would make the next action ready, and why did the event not happen? Common causes are using the wrong explicit GameID, waiting on a proposal that was never delivered, or expecting AcceptProposal to resolve before the player gets the potato.

Debugging Chialisp (CLVM)

Note: This section covers the current state of chialisp debugging, which lacks print statements and stack traces. Once those are available, prefer them over the technique below.

The problem

CLVM programs have no print/log facility. When a program crashes (raises, returns wrong values, or hits a type error like Requires Int Argument), the error message gives you the CLVM opcode and a NodePtr but no source location or call stack.

Diagnostic asserts via (x ...)

The only way to probe execution is to make the program fail at a known point using (x "MARKER" values...). The x operator raises an exception whose payload appears in the Rust error message as Raise(NodePtr(...)).

Critical rule: You must put the (x ...) in the return path, not in a side binding. The chialisp compiler optimizes away unused bindings, so this does nothing:

; WRONG — compiler optimizes this away, never executes
(assign
    _dbg (x "HERE")
    real_result (some_computation)
    real_result
)

Instead, replace the return expression itself:

; RIGHT — this is the return value, so it must execute
(assign
    intermediate (some_computation)
    (x "DBG_POINT" intermediate)
)

Binary search technique

  1. Pick the midpoint of the suspected code path.
  2. Replace the return at that point with (x "MID" relevant_values).
  3. Rebuild (./cb.sh; it runs tools/build-chialisp.sh first).
  4. Run the failing test (./ct.sh -o test_name).
  5. Interpret:
    • If the error changes to Raise(...) with your marker → execution reached that point. The bug is downstream.
    • If the error stays the same (e.g. PathIntoAtom) → the crash is before your marker. The bug is upstream.
  6. Repeat, narrowing the range by half each time.

Checking specific values

Once you've found the crashing expression, you can probe sub-expressions:

; What is this value? Is it an atom or a pair?
(x "CHECK_VAL" (strlen some_value) some_value)

Or use conditional asserts to test specific properties:

; Only crash if the value is wrong
(if (= (strlen val) 32)
    (real_computation val)
    (x "BAD_LEN" (strlen val) val)
)

Chialisp rebuild cycle

Each chialisp change requires:

  1. ./cb.sh (runs the content-addressed Chialisp build, then rebuilds the test binary)
  2. Run the test

tools/build-chialisp.sh automatically rebuilds when source content changes or when any generated .hex file is missing or modified. Do not delete cache files or run cargo clean; ordinary Cargo commands intentionally do not compile Chialisp.

Cleanup

Remove all diagnostic (x ...) calls after the bug is fixed. They are not documentation. Search for your marker prefix (e.g. DBG_) to find them all.

Mistakes to Avoid

  • Don't use cargo test directly. Use ./ct.sh while debugging or ./ct-automation.sh for an automated full-suite run. The scripts handle feature flags, output capture, and test ordering.

  • Don't manually filter failure output. Read the complete ./ct.sh output, or let ./ct-automation.sh capture it and replay it on failure. The middle can contain per-test diagnostics needed to find the cause.

  • Don't run tests in the background. Run ./ct.sh and ./cb.sh in the foreground and wait for them to finish. Background execution with sleep-based polling wastes time and makes output harder to capture.

  • AI agents: use ./ct-automation.sh for the final full-suite check. Use ./ct.sh -o test_name for focused debugging when direct output is useful. Give either command enough foreground time to finish; the automation wrapper keeps successful runs concise and emits the full log on failure.

  • Don't use sleep to wait for processes. When waiting for a command to finish, set block_until_ms to a value higher than the expected runtime. The tool returns as soon as the process exits or the timeout elapses, whichever comes first. Using sleep wastes time and blocks interruption.

  • On macOS/Linux, use kill -0 instead of sleep for waiting. sleep N always waits the full N seconds even if the process finished immediately. This alternative checks once per second whether the process is still running and exits as soon as it isn't:

    # kill -0 sends no signal — it only checks whether the PID exists
    for i in $(seq 60); do kill -0 <pid> 2>/dev/null || break; sleep 1; done

    This is strictly better than sleep 60: identical worst case (60s if the process truly takes that long), but returns within 1 second of process exit instead of wasting the remaining time. Use a higher count when the expected runtime is longer.

    kill -0 is POSIX and works on macOS and Linux, not Windows.