Skip to content

feat(ci,tools): add core workload and jobset verifier for clickhouse metrics job - #910

Merged
janzak-dev merged 3 commits into
mainfrom
feat/nightly-examples-suite
Sep 29, 2026
Merged

janzak-dev merged 3 commits into
mainfrom
feat/nightly-examples-suite

Conversation

@janzak-dev

@janzak-dev janzak-dev commented Sep 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Introduces core workload lifecycle and JobSet specification verification for the clickhouse_metrics_job example workload in the nightly testing pipeline. This PR forms the lean baseline for the nightly verification suite without introducing uncalled analytical endpoints.

Stack Context: Layer 1 of 3 (Core Workload & JobSet Spec Verifier)

Note

This PR is Layer 1 (Base) of a 3-PR stack targeting main:

Motivation & Context

Validating example workloads in CI requires ensuring that submitted experiments transition through their lifecycle and that Kubernetes JobSet specifications conform to expectations (correct queue, replica count, container images, and command arguments).

By isolating the core workload lifecycle and JobSet spec validation into this PR:

  1. The PR remains strictly minimal (3 files changed, 163 insertions, well below the 400 prod / 1,000 total limits).
  2. The verification logic is immediately exercisable without requiring analytical database port-forwarding or external database credentials.
  3. Zero dead code invariant is preserved: common.py, xmc_entities_validator.py, and xmc_entities_validator_test.py have zero diff against main.

Key Changes

  • Core Verifier: Added tools/nightly_tests/clickhouse_metrics_job_verify.py implementing Stages 1–3:
    • Stage 1: Experiment metadata and existence verification via XMC client.
    • Stage 2: WorkUnit state transition polling until COMPLETED on target queue (cpu-queue).
    • Stage 3: JobSet specification validation (container commands, replica count, and worker template).
  • CI Matrix Wiring: Added clickhouse_metrics_job matrix entry to .github/workflows/nightly-examples-e2e.yml targeting the newly added verifier binary.
  • Bazel Build Configuration: Defined py_binary target in tools/nightly_tests/BUILD.bazel.

Assumptions

None — requirements were fully specified.

Verification & Testing

  • Local Unit Testing:
    • bazel test //tools/nightly_tests/... passed (4/4 tests green).
    • ./.githooks/pre-commit passed cleanly (buildifier, gofmt, golangci-lint, addlicense).
  • Diff Sizing Check (./tools/analyze_diff_size.py --base-ref origin/main...):
    • Production code: 163 lines (limit: $\le 400$) — PASS
    • Total code: 163 lines (limit: $\le 1,000$) — PASS
  • Live CI Run:
    • Workflow: nightly-examples-e2e.yml
    • Run ID: 36554440273
    • Status: Completed / Success
    • Job Execution: clickhouse_metrics_job (XID 18002): PASSED in 1m 29s (Job #109360104271)
    • Aggregate Result: {"clickhouse_metrics_job":"18002","simple_job":"18014"} — ALL PASSED.

Related Issues / Bugs

Stacked directly on main (following merged #933).

TAG=agy
CONV=acf882de-1561-404c-ba2e-ae0940451165

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds several nightly test verification scripts for XMC workloads, including validation for chart artifacts, ClickHouse metrics, init containers, managed Xprof, and dashboards, along with corresponding unit tests and helper utilities. The review feedback highlights several critical issues that need to be addressed: a loose matching logic in '_match_job_for_validation' that could silently ignore job name typos; a polling loop in '_poll_dashboard_iteration' that prematurely terminates on 'AssertionError' or crashes on non-JSON responses; a port-stripping bug in 'resolve_dashboard_endpoint' for remote hosts; and duplicate manual endpoint extraction in 'clickhouse_metrics_job_verify.py' and 'metrics_dashboard_job_verify.py' that bypasses central resolution logic.

Comment thread tools/nightly_tests/common.py Resolved
@janzak-dev
janzak-dev added this pull request to stack #911 September 17, 2026 11:54
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-suite branch from cdc662e to eb5fe3b Compare September 17, 2026 12:07
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-suite branch from eb5fe3b to f3cfd38 Compare September 17, 2026 12:12
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-suite branch 2 times, most recently from 5136759 to e468c14 Compare September 17, 2026 12:56
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-suite branch from e468c14 to a59bd2d Compare September 17, 2026 13:06
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-suite branch from a59bd2d to 4feea68 Compare September 17, 2026 13:09
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-suite branch 3 times, most recently from 3d171be to 0971d88 Compare September 17, 2026 14:39
Base automatically changed from feat/nightly-examples-pipeline-base to main September 17, 2026 14:52
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-suite branch from 0971d88 to 49cd04b Compare September 17, 2026 14:52
@janzak-dev
janzak-dev removed this pull request from stack #911 September 21, 2026 11:49
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-suite branch from 4dcec85 to 0c586bc Compare September 21, 2026 11:58
@janzak-dev
janzak-dev changed the base branch from main to feat/nightly-infra-gaps-enablement September 21, 2026 11:58
@janzak-dev janzak-dev changed the title feat(ci,tools): expand nightly examples suite to all 12 example workloads feat(ci,tools): add full coverage verifier for clickhouse metrics workload Sep 21, 2026
@janzak-dev
janzak-dev added this pull request to stack #935 September 21, 2026 12:00
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-suite branch 2 times, most recently from 3c2c4bf to 0048037 Compare September 21, 2026 12:56
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-suite branch 2 times, most recently from 60f1326 to 67f43a9 Compare September 28, 2026 11:10
Base automatically changed from feat/nightly-infra-gaps-enablement to main September 29, 2026 09:31
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-suite branch 3 times, most recently from 0e10918 to 21dd461 Compare September 29, 2026 10:15
@janzak-dev janzak-dev changed the title feat(ci,tools): add full coverage verifier for clickhouse metrics workload feat(ci,tools): add core workload and jobset verifier for clickhouse metrics job Sep 29, 2026
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-suite branch from 21dd461 to 8715eb1 Compare September 29, 2026 10:51
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-suite branch from 8715eb1 to 2244297 Compare September 29, 2026 11:32
@janzak-dev
janzak-dev marked this pull request as ready for review September 29, 2026 11:42
@github-actions
github-actions Bot requested a review from pachopepe September 29, 2026 11:42
@janzak-dev
janzak-dev removed this pull request from stack #935 September 29, 2026 11:46
@janzak-dev
janzak-dev added this pull request to stack #1036 September 29, 2026 11:48
pachopepe
pachopepe previously approved these changes Sep 29, 2026

@pachopepe pachopepe left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed that this PR is part of a stack, feel free to ignore the comments that are not relevant or covered in a next PR.

@janzak-dev
janzak-dev added this pull request to the merge queue Sep 29, 2026
Merged via the queue into main with commit 1a3250c Sep 29, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants