Repository navigation
feat(examples): add multi-workunit simple job launch script and nightly workflow - #847
Merged
Merged
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a standalone demonstration workload launch script (launch.py) along with its Bazel build configuration and documentation, designed to showcase real-time Resource Observability Dashboard (ROD) metrics in XMC. The review feedback points out that environment variables from .env are loaded too late (inside main()) to be used as default values for module-level command-line flags. It is recommended to move the environment loading to the module level and remove the redundant call inside main().
janzak-dev
force-pushed
the
feat/nightly-demo-workload
branch
from
September 11, 2026 07:54
ffc6c1d to
b3972cc
Compare
Add multi-WorkUnit support to the 2-minute demo workload script (`examples/demo_2min_job/launch.py`), configure `BUILD.bazel` target, and introduce `.github/workflows/trigger-demo-workload.yml` to trigger and verify workloads against the live `xmc-nightly` environment via Workload Identity Federation. TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
janzak-dev
force-pushed
the
feat/nightly-demo-workload
branch
from
September 11, 2026 07:58
b3972cc to
1936f52
Compare
Move `dotenv.load_dotenv()` to module level before `flags.DEFINE_*` declarations so command-line flags can properly read default values from `.env`. Addresses review comments on PR #847 (Comment IDs 3987026071, 3987026085). TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
Bind `github.actor` to `ACTOR` environment variable in launch step to eliminate potential template injection. TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
janzak-dev
had a problem deploying
to
xmc-nightly
September 11, 2026 08:11 — with
GitHub Actions
Failure
TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
janzak-dev
had a problem deploying
to
xmc-nightly
September 11, 2026 08:13 — with
GitHub Actions
Failure
TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
janzak-dev
had a problem deploying
to
xmc-nightly
September 11, 2026 08:16 — with
GitHub Actions
Failure
janzak-dev
had a problem deploying
to
xmc-nightly
September 11, 2026 08:18 — with
GitHub Actions
Failure
TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
janzak-dev
had a problem deploying
to
xmc-nightly
September 11, 2026 08:21 — with
GitHub Actions
Failure
TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
janzak-dev
had a problem deploying
to
xmc-nightly
September 11, 2026 08:23 — with
GitHub Actions
Failure
…tity-token TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
janzak-dev
had a problem deploying
to
xmc-nightly
September 11, 2026 08:25 — with
GitHub Actions
Failure
TAG=agy CONV=606f1e41-6785-4137-b85b-cd0f56237339
janzak-dev
temporarily deployed
to
xmc-nightly
September 11, 2026 08:28 — with
GitHub Actions
Inactive
janzak-dev
temporarily deployed
to
xmc-nightly
September 11, 2026 09:28 — with
GitHub Actions
Inactive
…tion comments
…h CI workaround warning
janzak-dev
temporarily deployed
to
xmc-nightly
September 11, 2026 09:45 — with
GitHub Actions
Inactive
janzak-dev
marked this pull request as ready for review
September 11, 2026 09:58
andrzejfudro
previously approved these changes
Sep 11, 2026
andrzejfudro
approved these changes
Sep 11, 2026
This was referenced Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces a standalone, multi-WorkUnit simple job workload launch script (
examples/simple_job/launch.py) and a GitHub Actions workflow (.github/workflows/trigger-demo-workload.yml) to trigger and verify end-to-end experiment creation against the livexmc-nightlyGKE deployment.Motivation & Context
With the automated nightly deployment workflow merged, we need an easy, automated way to trigger and verify real-world XMC experiment submissions across Kueue queues (such as
cpu-queue), validate multi-WorkUnit scheduling and pod execution.Screenshots
Next steps
In next PR I will write a test that will assert that the Experiments/WorkUnits created by this script are properly created in the environment.
Key Changes
examples/simple_job/launch.py):--num_work_units, defaulting to 3) creating distinct WorkUnits nameddemo-worker-1,demo-worker-2, anddemo-worker-3.--queue=cpu-queue,--gpus=0, and dynamic project/domain resolution.TODOcomments for gRPC auth monkey patch.examples/simple_job/README.md):.github/workflows/trigger-demo-workload.yml):workflow_dispatchtrigger with customizable inputs (queue,num_work_units,duration_seconds).xmc-nightlyenvironment.examples/simple_job/launch.pyand logs the created Experiment XID, Web UI URL, and WorkUnit execution details.Technical Debt & Upstream Remediation
This PR includes an interim monkey patch in
examples/simple_job/launch.pythat should be remediated once upstream dependencies are updated:_patch_grpc_auth_if_needed()monkey-patchesess_api._create_experiment_state_server_stubto fall back togcloud auth print-identity-tokenin CI/WIF environments.XMC_AUTH_TOKENandXMC_USER_EMAIL.Assumptions
Verification & Testing
buildifier,gofmt,golangci-lint,addlicense,ruff).arc-runner-setin environmentxmc-nightly).xmc-nightly:experiments/16) againstcpu-queuein....demo-worker-1) — Completeddemo-worker-2) — Completeddemo-worker-3) — CompletedRelated Issues / Bugs
TAG=agy
CONV=606f1e41-6785-4137-b85b-cd0f56237339