Skip to content

feat(ci): add unified nightly examples e2e pipeline workflow - #909

Merged
janzak-dev merged 5 commits into
mainfrom
feat/nightly-examples-pipeline-base
Sep 17, 2026
Merged

janzak-dev merged 5 commits into
mainfrom
feat/nightly-examples-pipeline-base

Conversation

@janzak-dev

@janzak-dev janzak-dev commented Sep 17, 2026 •

Copy link
Copy Markdown
Collaborator

🥞 Stack Navigation

  • #908 Layer 1 (Base): feat(tools): make simple_job verification exhaustive with full protobuf resource and priority assertions
  • #909 Layer 2: feat(ci): add unified nightly examples e2e pipeline workflow 👈
  • #910 Layer 3 (Apex): feat(ci,tools): expand nightly examples suite to all 12 example workloads

🏛️ Role in PR Stack Architecture

This PR serves as Layer 2 (Unified Nightly Examples Pipeline Architecture) within the nightly examples testing stack.


🔄 Workflow Architecture & Distinction

This repository maintains two distinct nightly workflows serving complementary roles:

Dimension 1. nightly-test-suite.yml (Single Simple Job Baseline Test) 2. nightly-examples-e2e.yml (Extensible Examples Suite Pipeline Stub)
Primary Purpose Fast Cluster Sanity Check: Standalone, dedicated single-workload verification running the core simple job. Full Examples Matrix Verification: Scalable multi-example test suite covering all example workloads across the repository.
Pipeline Architecture Sequential Workflow: Discrete sequential jobs (launch $\rightarrow$ verify $\rightarrow$ alert-on-failure). Matrix Pipeline: Parallel matrix runner with dynamic workload filtering, in-job verification, and centralized aggregation.
Result Aggregation Individual job logs and alert notifications. Structured per-workload JSON result artifacts aggregated into a unified Markdown summary table via aggregate-example-results.js.
Current Scope & Role in PR Existing production baseline (unmodified). Pipeline Stub: Introduced in PR #909 with simple_job as the initial baseline, ready to be expanded to all 12 example workloads in upstack PR #910.

Summary

Introduces the unified nightly examples E2E workflow pipeline (.github/workflows/nightly-examples-e2e.yml) and Markdown table aggregation scripts (.github/scripts/aggregate-example-results.js), running against xmc-nightly on GKE with a simple_job baseline matrix entry.

Motivation & Context

Key Changes

  • CI / Workflow (.github/workflows/nightly-examples-e2e.yml):
    • Adds scheduled (cron 0 3 * * *) and workflow_dispatch pipeline for nightly examples testing.
    • Configures simple_job baseline matrix entry running in a Python virtual environment.
    • Generates JSON result artifacts per matrix leg and aggregates them into the job summary.
  • Reporting Scripts (.github/scripts/):
    • aggregate-example-results.js: Aggregates per-workload execution JSON results into a Markdown summary table.
    • aggregate-example-results.test.js: Hermetic Node.js unit tests (11 tests).

Assumptions

None — requirements were fully specified.

Verification & Testing

  • Bazel Unit Tests: bazel test //tools/nightly_tests:all (2/2 test targets passed)
  • Node Unit Tests: node --test .github/scripts/aggregate-example-results.test.js (11/11 tests passing)
  • Pre-commit Checks: ./.githooks/pre-commit check (100% clean)
  • Live GHA Workflow Runs (Passed):
    • nightly-examples-e2e.yml: Run #35227569927 (Status: ✅ Success — simple_job matrix execution & aggregation verified)
    • nightly-test-suite.yml: Run #35227574323 (Status: ✅ Success — existing demo workload baseline verified)

TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enhances the GitHub Actions results aggregation script to support both trigger-only and E2E test suite results, dynamically formatting the Markdown summary table columns and allowing custom titles. It also updates the CLI execution to exit with a non-zero code if any workload fails, and adds comprehensive unit tests for these new features. Feedback on the changes highlights a potential runtime crash and invalid map keys in aggregateResults due to missing null/undefined checks on the results array, along with redundant status check logic. A code suggestion was provided to filter the input array and safely simplify these checks.

@janzak-dev
janzak-dev added this pull request to stack #911 September 17, 2026 11:54
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-pipeline-base branch from 54aa94b to 3712468 Compare September 17, 2026 12:07
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-pipeline-base branch from 3712468 to 969c9e4 Compare September 17, 2026 12:11
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-pipeline-base branch from 969c9e4 to bb53217 Compare September 17, 2026 12:51
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-pipeline-base branch from de15808 to d589673 Compare September 17, 2026 13:03
@janzak-dev
janzak-dev force-pushed the feat/nightly-examples-pipeline-base branch from d589673 to 638eb86 Compare September 17, 2026 13:09
Base automatically changed from feat/exhaustive-simple-job-verify to main September 17, 2026 13:28
@janzak-dev
janzak-dev marked this pull request as ready for review September 17, 2026 13:50
@janzak-dev janzak-dev added the priority::2-high top of mind. should not wait. label Sep 17, 2026
pachopepe
pachopepe previously approved these changes Sep 17, 2026
@janzak-dev
janzak-dev added this pull request to the merge queue Sep 17, 2026
Merged via the queue into main with commit d71d985 Sep 17, 2026
11 checks passed
@janzak-dev
janzak-dev deleted the feat/nightly-examples-pipeline-base branch September 17, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority::2-high top of mind. should not wait.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants