Skip to content

Record wall time, CPU time and peak memory per test on every run, with history to catch slowdowns #1288

Open

Description

@Lillecarl

Record what each test costs on every run, so a slowdown shows as a number and not as a feeling.

Per test, on every run, local and CI:

  • wall time
  • CPU time (user and system, the test's own process and its children)
  • peak memory (RSS; for a guest suite, the guest's too)

Kept as history: a test's numbers compare with its own earlier runs on the same lane and machine class. A run reports each test that moved past a threshold, slower or faster, and the set's total against its budget (#1247).

Where it lives:

  • pytest suites: pytest-agent already writes per-test detail and history.jsonl. Extend that.
  • guest suites (vivarium): events.jsonl and report.json already time phases and commands. Add per-case CPU and memory.
  • One shape for both, so a person or an agent reads one report.

Open: where history is kept across machines (CI artifact, a branch, or a store path), and the threshold that counts as a regression with noisy hosts.

Refs #1247.


Assisted-By: Claude Opus 5.5

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    source: monorepojosh, lock-subtrees, shared CI, pins

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions