Repository navigation
feat(ci,tools): add core workload and jobset verifier for clickhouse metrics job - #910
Conversation
There was a problem hiding this comment.
Code Review
This pull request adds several nightly test verification scripts for XMC workloads, including validation for chart artifacts, ClickHouse metrics, init containers, managed Xprof, and dashboards, along with corresponding unit tests and helper utilities. The review feedback highlights several critical issues that need to be addressed: a loose matching logic in '_match_job_for_validation' that could silently ignore job name typos; a polling loop in '_poll_dashboard_iteration' that prematurely terminates on 'AssertionError' or crashes on non-JSON responses; a port-stripping bug in 'resolve_dashboard_endpoint' for remote hosts; and duplicate manual endpoint extraction in 'clickhouse_metrics_job_verify.py' and 'metrics_dashboard_job_verify.py' that bypasses central resolution logic.
cdc662e to
eb5fe3b
Compare
eb5fe3b to
f3cfd38
Compare
5136759 to
e468c14
Compare
e468c14 to
a59bd2d
Compare
a59bd2d to
4feea68
Compare
3d171be to
0971d88
Compare
0971d88 to
49cd04b
Compare
4dcec85 to
0c586bc
Compare
3c2c4bf to
0048037
Compare
60f1326 to
67f43a9
Compare
0e10918 to
21dd461
Compare
21dd461 to
8715eb1
Compare
8715eb1 to
2244297
Compare
Summary
Introduces core workload lifecycle and JobSet specification verification for the
clickhouse_metrics_jobexample workload in the nightly testing pipeline. This PR forms the lean baseline for the nightly verification suite without introducing uncalled analytical endpoints.Stack Context: Layer 1 of 3 (Core Workload & JobSet Spec Verifier)
Note
This PR is Layer 1 (Base) of a 3-PR stack targeting
main:feat/nightly-examples-suite): Core Workload & JobSet Spec Verifier. Verifies experiment metadata, WorkUnit state transitions toCOMPLETEDoncpu-queue, and validates JobSet container/replica specifications. Leavescommon.pyandxmc_entities_validator.pyminimal with zero dead code.feat/nightly-clickhouse-metrics-sql): Analytical SQL & Dashboard Hydration. Introduces ClickHouse HTTP SQL client assertions for row arrival/aggregates and tests Dashboard ServiceGetChartDatahydration.feat/nightly-remaining-examples-suite): Full Suite Expansion. Extends the E2E verification suite across the remaining 10 example workloads (JobSets, Ray, MPIJob, Artifacts, Dashboards).Motivation & Context
Validating example workloads in CI requires ensuring that submitted experiments transition through their lifecycle and that Kubernetes JobSet specifications conform to expectations (correct queue, replica count, container images, and command arguments).
By isolating the core workload lifecycle and JobSet spec validation into this PR:
common.py,xmc_entities_validator.py, andxmc_entities_validator_test.pyhave zero diff againstmain.Key Changes
tools/nightly_tests/clickhouse_metrics_job_verify.pyimplementing Stages 1–3:COMPLETEDon target queue (cpu-queue).clickhouse_metrics_jobmatrix entry to.github/workflows/nightly-examples-e2e.ymltargeting the newly added verifier binary.py_binarytarget intools/nightly_tests/BUILD.bazel.Assumptions
None — requirements were fully specified.
Verification & Testing
bazel test //tools/nightly_tests/...passed (4/4 tests green)../.githooks/pre-commitpassed cleanly (buildifier, gofmt, golangci-lint, addlicense)../tools/analyze_diff_size.py --base-ref origin/main...):nightly-examples-e2e.ymlclickhouse_metrics_job(XID18002): PASSED in 1m 29s (Job #109360104271){"clickhouse_metrics_job":"18002","simple_job":"18014"}—ALL PASSED.Related Issues / Bugs
Stacked directly on
main(following merged #933).TAG=agy
CONV=acf882de-1561-404c-ba2e-ae0940451165