Skip to content

feat(examples): add multi-workunit simple job launch script and nightly workflow - #847

Merged
janzak-dev merged 21 commits into
mainfrom
feat/nightly-demo-workload
Sep 11, 2026
Merged

janzak-dev merged 21 commits into
mainfrom
feat/nightly-demo-workload

Conversation

@janzak-dev

@janzak-dev janzak-dev commented Sep 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Introduces a standalone, multi-WorkUnit simple job workload launch script (examples/simple_job/launch.py) and a GitHub Actions workflow (.github/workflows/trigger-demo-workload.yml) to trigger and verify end-to-end experiment creation against the live xmc-nightly GKE deployment.

Motivation & Context

With the automated nightly deployment workflow merged, we need an easy, automated way to trigger and verify real-world XMC experiment submissions across Kueue queues (such as cpu-queue), validate multi-WorkUnit scheduling and pod execution.

Screenshots

image

Next steps

In next PR I will write a test that will assert that the Experiments/WorkUnits created by this script are properly created in the environment.

Key Changes

  • Simple Job Launch Script (examples/simple_job/launch.py):
    • Multi-WorkUnit support (--num_work_units, defaulting to 3) creating distinct WorkUnits named demo-worker-1, demo-worker-2, and demo-worker-3.
    • Configured with --queue=cpu-queue, --gpus=0, and dynamic project/domain resolution.
    • Added explicit upstream remediation TODO comments for gRPC auth monkey patch.
  • Documentation (examples/simple_job/README.md):
    • Added comprehensive usage documentation and a prominent warning banner documenting temporary CI/CD workarounds.
  • GitHub Actions Workflow (.github/workflows/trigger-demo-workload.yml):
    • Provides a workflow_dispatch trigger with customizable inputs (queue, num_work_units, duration_seconds).
    • Authenticates via Workload Identity Federation against the xmc-nightly environment.
    • Automatically launches examples/simple_job/launch.py and logs the created Experiment XID, Web UI URL, and WorkUnit execution details.

Technical Debt & Upstream Remediation

This PR includes an interim monkey patch in examples/simple_job/launch.py that should be remediated once upstream dependencies are updated:

  • gRPC Authentication / WIF Stub Patching:

Assumptions

  • None — requirements were fully specified.

Verification & Testing

  • Local Pre-Commit Linters: Passed all hooks (buildifier, gofmt, golangci-lint, addlicense, ruff).
  • Live GitHub Actions CI Run: Successfully validated end-to-end execution on GitHub Actions runner:
  • Live GKE Deployment & Execution on xmc-nightly:
    • Successfully launched Experiment XID 16 (experiments/16) against cpu-queue in ....
    • All 3 WorkUnits completed successfully:
      • WorkUnit 1 (demo-worker-1) — Completed
      • WorkUnit 2 (demo-worker-2) — Completed
      • WorkUnit 3 (demo-worker-3) — Completed
    • Verified experiment and WorkUnit execution visible in XMC Web UI.

Related Issues / Bugs

TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a standalone demonstration workload launch script (launch.py) along with its Bazel build configuration and documentation, designed to showcase real-time Resource Observability Dashboard (ROD) metrics in XMC. The review feedback points out that environment variables from .env are loaded too late (inside main()) to be used as default values for module-level command-line flags. It is recommended to move the environment loading to the module level and remove the redundant call inside main().

Comment thread examples/simple_job/launch.py Resolved
Comment thread examples/demo_2min_job/launch.py Resolved
@janzak-dev
janzak-dev force-pushed the feat/nightly-demo-workload branch from ffc6c1d to b3972cc Compare September 11, 2026 07:54
Add multi-WorkUnit support to the 2-minute demo workload script (`examples/demo_2min_job/launch.py`), configure `BUILD.bazel` target, and introduce `.github/workflows/trigger-demo-workload.yml` to trigger and verify workloads against the live `xmc-nightly` environment via Workload Identity Federation.

TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339
@janzak-dev
janzak-dev force-pushed the feat/nightly-demo-workload branch from b3972cc to 1936f52 Compare September 11, 2026 07:58
Move `dotenv.load_dotenv()` to module level before `flags.DEFINE_*` declarations so command-line flags can properly read default values from `.env`.

Addresses review comments on PR #847 (Comment IDs 3987026071, 3987026085).

TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339
Bind `github.actor` to `ACTOR` environment variable in launch step to eliminate potential template injection.

TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339
TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339
TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339
TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339
TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339
TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339
…tity-token

TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339
TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339
@janzak-dev janzak-dev changed the title feat(examples): add 3-workunit demo launch script and nightly workflow feat(examples): add multi-workunit simple job launch script and nightly workflow Sep 11, 2026
@janzak-dev
janzak-dev marked this pull request as ready for review September 11, 2026 09:58
@janzak-dev
janzak-dev added this pull request to stack #853 September 11, 2026 10:12
@janzak-dev janzak-dev added the priority::2-high top of mind. should not wait. label Sep 11, 2026
andrzejfudro
andrzejfudro previously approved these changes Sep 11, 2026
Comment thread examples/simple_job/BUILD.bazel Resolved
@janzak-dev
janzak-dev added this pull request to the merge queue Sep 11, 2026
Merged via the queue into main with commit ef8c353 Sep 11, 2026
11 checks passed
@janzak-dev
janzak-dev deleted the feat/nightly-demo-workload branch September 11, 2026 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority::2-high top of mind. should not wait.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants