Run the tests of a GitHub repository on ALCF machines (Aurora) and get the result back on the commit and the pull request, without a self-hosted GitHub runner and without a hosted runner waiting for the batch job.
The repository is tested by ALCF GitLab CI on a mirror of the GitHub repository. This repository holds the shared parts of that bridge, so that a project only carries three short files:
| Shared here | Purpose | In the project |
|---|---|---|
.github/workflows/trigger.yml |
reusable workflow: start the GitLab pipeline, set the commit status to pending | .github/workflows/alcf.yml (example) |
.github/workflows/report.yml |
reusable workflow: set the final commit status, copy the job logs | .github/workflows/alcf-report.yml (example) |
gitlab/aurora-julia.yml |
GitLab CI template: Julia on Aurora, build on the login node, test in PBS | .gitlab-ci.yml (example) |
The two workflows work for any GitLab CI pipeline on the ALCF instance. The GitLab template
is for Julia packages on Aurora; a project with other needs writes its own .gitlab-ci.yml
and still uses the workflows.
The setup was extracted from the copies in JuliaGPU/oneAPI.jl and exanauts/SparseDirectSolver.jl.
- How it works
- Requirements
- Setting up a project
- Reference: trigger workflow
- Reference: report workflow
- Reference: GitLab template
- Day-to-day use
- Maintenance
- Limits
- Trust
- Versioning
- Migrating a project that carries its own copy
push / pull request / label on a fork pull request
│
▼
alcf.yml → trigger.yml (about a minute) GitLab pipeline on Aurora (minutes to hours)
mirror sync, or push of the fork head ────► build (login node) → test (PBS batch job)
commit status: pending │ every status change
▼
project webhook (pipeline events)
│ workflow_dispatch
▼
alcf-report.yml → report.yml (about a minute,
final states only)
read the pipeline back from GitLab
commit status: success / failure / error
copy the job logs into the run
The result is a commit status (default name ALCF / Aurora) on the tested commit. It
shows up in the checks of a pull request and can be made a required check.
Properties of the design:
- No runner waits. Both workflows finish in about a minute. The batch job can sit in the PBS queue for hours without holding a GitHub runner or hitting a job timeout.
- Fork code is opt-in. GitLab CI jobs run as an ALCF user on the machine. Pushes and
pull requests from branches of the repository run automatically; a pull request from a
fork runs only after a maintainer applies a label (default
alcf-ci), and again only when the label is re-applied after new commits. The trigger workflow only moves git refs; it never checks out or executes pull-request code. - The webhook cannot forge a result. The dispatch inputs are only a hint; the report workflow reads the commit and the status back from GitLab.
- Only the newest pipeline of a commit speaks for it, so a late report of an old pipeline does not replace the result of a re-run.
- Infrastructure failures are not test failures. A pipeline whose failed jobs never ran
their script (no runner online, a batch job PBS never started, a runner error) is retried
once with a fresh pipeline, and reported as
errornaming the infrastructure if the retry fails the same way. - Logs are readable without an ALCF login. The GitLab instance needs an ALCF account, so the report run contains the job logs and the commit status links to that run.
Two things about ALCF's Jacamar runners shape the token setup: they refuse pipelines attributed to a bot user ("bot account token used for job, unsupported by runner"), and a project access token is a bot user. Pipelines therefore start in one of two ways only: by the mirror sync (attributed to the user who configured the mirror) or through a pipeline trigger token (attributed to the user who created it). The project access token is used for API calls and for moving refs, never to create a pipeline.
- An ALCF account with an allocation on the machine, access to ALCF GitLab CI, and the right to create a project there. CI jobs run as the user who owns the mirror and the trigger token.
- A public GitHub repository (the mirror and the fork push fetch from it without credentials) in which you can add workflows, secrets and labels.
- The GitHub organization must allow reusable workflows from
anlsys/alcf-ci(Settings → Actions → General → "Allow actions and reusable workflows"; nothing to do if all actions are allowed). - For the GitLab template: a shared filesystem path for the Julia depot and a site
juliamodule of the wanted version.
Placeholders: ORG/REPO is the GitHub repository, USER/PROJECT the GitLab mirror project
on gitlab-ci.alcf.anl.gov.
-
Create the project
USER/PROJECT(private is fine). -
Settings → Repository → Mirroring repositories: add
https://github.com/ORG/REPO.git, direction Pull, with- Overwrite diverged branches on (force-pushed branches follow),
- Trigger pipelines for mirror updates on (this is what starts a pipeline),
- Only mirror protected branches off (pull-request branches must be mirrored).
Do this as the ALCF user the jobs should run as: mirror pipelines are attributed to the user who set up the mirror.
-
If the jobs' git fetch fails on the instance's certificate, add the CI/CD variable
GIT_SSL_NO_VERIFY=true(Settings → CI/CD → Variables). The existing mirror projects set it.
- Project access token (Settings → Access tokens): role Maintainer, scopes api
and write_repository. Used to start a mirror sync, read pipelines, jobs and logs,
and push fork heads as
gh-pr-<N>branches. Note the expiry date. - Pipeline trigger token (Settings → CI/CD → Pipeline trigger tokens), created by the ALCF user the jobs should run as. Used for fork pull requests, re-runs after a failed pipeline and the automatic retry of infrastructure failures.
In ORG/REPO (Settings → Secrets and variables → Actions), or as organization secrets:
| Secret | Value |
|---|---|
ALCF_GITLAB_TOKEN |
the project access token |
ALCF_GITLAB_TRIGGER_TOKEN |
the pipeline trigger token |
Create the label alcf-ci:
gh label create alcf-ci -R ORG/REPO --description "Run this fork pull request on ALCF"Copy and fill in the placeholders:
| From | To |
|---|---|
examples/github/alcf.yml |
.github/workflows/alcf.yml |
examples/github/alcf-report.yml |
.github/workflows/alcf-report.yml |
examples/gitlab/.gitlab-ci.yml |
.gitlab-ci.yml |
gitlab_project and context must be the same in both workflows. alcf-report.yml has to
be on the default branch before the webhook can call it, so merge these files first; the
first run of alcf.yml then leaves a pending status until step 5 is done.
-
Create a GitHub fine-grained personal access token: resource owner
ORG, repositoryORG/REPOonly, permission Actions: read and write, nothing else. This permission can start workflows but cannot push code. Note the expiry date. -
In the GitLab project, Settings → Webhooks → Add new webhook:
-
URL:
https://api.github.com/repos/ORG/REPO/actions/workflows/alcf-report.yml/dispatches -
Trigger: Pipeline events only; SSL verification enabled
-
Custom headers:
Authorization=Bearer <the token>andAccept=application/vnd.github+json -
Custom webhook template (
mainis the default branch ofORG/REPO):{"ref":"main","inputs":{"pipeline_id":"{{object_attributes.id}}","status":"{{object_attributes.status}}"}}
The token lives in the webhook configuration only, so CI jobs, including those of label-gated fork pull requests, never see it.
-
-
Test → Pipeline events. "Recent events" must show a 2xx response and a run must appear under ALCF CI report in the Actions tab of
ORG/REPO.
-
Report a finished pipeline by hand; the commit it tested gets the status:
gh workflow run alcf-report.yml -R ORG/REPO -f pipeline_id=<id> gh api repos/ORG/REPO/commits/<sha>/status --jq '.statuses[] | {context, state, description}'
-
Open a pull request from a branch of the repository:
ALCF CIfinishes within a few minutes, the status is pending and links to the pipeline, and the final status arrives when the pipeline finishes. -
Once the status has been reported, it can be added as a required status check to the branch ruleset under the name given as
context.
anlsys/alcf-ci/.github/workflows/trigger.yml, called from a workflow that triggers on
push (the branches to test), pull_request and pull_request_target with
types: [labeled].
| Input | Default | |
|---|---|---|
gitlab_project |
required | path of the mirror project, USER/PROJECT |
gitlab_host |
gitlab-ci.alcf.anl.gov |
GitLab instance |
context |
ALCF / Aurora |
name of the commit status |
label |
alcf-ci |
label that runs a fork pull request |
report_workflow |
alcf-report.yml |
file name of the caller's report workflow |
| Secret | |
|---|---|
gitlab_token |
required; the project access token |
trigger_token |
the pipeline trigger token; without it fork pull requests and re-runs after a failed pipeline fail with a message |
Caller permissions: statuses: write, actions: write.
What it does, by event:
| Event | Action |
|---|---|
push, pull_request from a branch of the repository |
asks GitLab to sync the mirror now, which creates the pipeline |
pull_request from a fork |
nothing (the job is skipped) |
pull_request_target, the label applied to a fork pull request |
checks that the head is still the labeled commit, pushes it to GitLab as gh-pr-<N> with ci.skip, creates the pipeline with the trigger token |
| re-run of a run whose newest pipeline failed, was canceled or skipped | creates a fresh pipeline with the trigger token, if the commit is still the tip of its branch on GitLab |
It then waits up to 10 minutes for the pipeline to exist, sets the pending status with a link to it (or to the run, if the mirror has not synced yet), and dispatches the report workflow itself if the pipeline had already finished. A failed mirror-sync request is only a warning: the scheduled sync still creates the pipeline, later.
Runs for the same pull request or branch cancel each other, so only the newest commit is handled.
anlsys/alcf-ci/.github/workflows/report.yml, called from a workflow that triggers on
workflow_dispatch with the inputs pipeline_id and status.
| Input | Default | |
|---|---|---|
pipeline_id |
required | GitLab pipeline id |
status |
empty | status sent by the webhook; a run with an empty status always reports |
gitlab_project |
required | path of the mirror project |
gitlab_host |
gitlab-ci.alcf.anl.gov |
GitLab instance |
context |
ALCF / Aurora |
name of the commit status |
label |
alcf-ci |
named in the failure summary |
publish_logs |
true |
copy the GitLab job logs into the run |
| Secret | |
|---|---|
gitlab_token |
required; the project access token |
trigger_token |
the pipeline trigger token; without it an infrastructure failure is reported, not retried |
Caller permissions: statuses: write.
How a pipeline maps to the commit status:
| Pipeline | Status | |
|---|---|---|
success |
success |
|
failed, a job ran and failed |
failure |
the description names the failed jobs |
failed, every failed job has an infrastructure reason |
pending, then error |
retried once as a fresh pipeline; error if that fails the same way |
canceled |
error |
|
skipped ([skip ci]) |
unchanged | |
| still running (manual run) | pending |
links to the pipeline |
| not the newest pipeline of its commit | unchanged |
The infrastructure reasons are stuck_pending_no_matching_runners, no_updates_running,
runner_system_failure, stuck_or_timeout_failure, scheduler_failure and
runner_unsupported. Jobs with allow_failure do not count.
The webhook fires on every status change of a pipeline. Runs for non-final states skip their job and use no runner, but appear as skipped runs in the Actions tab. The run itself is attached to the default branch and is green whenever reporting worked, also for a failed pipeline; the pipeline's result is in the commit status.
gitlab/aurora-julia.yml, included with
include:
- remote: 'https://raw.githubusercontent.com/anlsys/alcf-ci/v1/gitlab/aurora-julia.yml'It includes ALCF's runner definitions (anl/ci-resources/defaults), so the project does
not have to.
Variables the project sets globally (top-level variables:):
| Variable | |
|---|---|
JULIA_VERSION |
version of the site julia module, e.g. "1.13" |
JULIA_DEPOT_PATH |
persistent depot on a filesystem the compute nodes mount; one per project and Julia version |
Variables the template sets:
| Variable | Value | |
|---|---|---|
JULIA_CPU_TARGET |
generic;icelake-server;sapphirerapids,-avx512fp16 |
package images for login and compute nodes; must be the same in every job |
JULIA_JIT_TARGET |
native,-avx512fp16 |
for julia -C "$JULIA_JIT_TARGET"; JULIA_CPU_TARGET does not constrain the runtime JIT |
Job templates:
| Template | Runs | Use |
|---|---|---|
.aurora-julia-build |
login node (shell runner, has network), stage build, timeout 2 h |
Pkg.instantiate, Pkg.precompile, anything that downloads; pass manifests on as artifacts |
.aurora-julia-test |
PBS batch job (no network), stage test, timeout 5 h including the queue wait |
the tests; set ANL_AURORA_SCHEDULER_PARAMETERS |
.aurora-env |
script fragment, already the before_script of both templates |
loads julia/$JULIA_VERSION, fails early if the two variables are missing |
.aurora-drop-site-oneapi |
script fragment | removes the site oneAPI libraries from LD_LIBRARY_PATH; for packages whose SYCL runtime comes from Julia artifacts or Conda |
Also set: workflow: rules (pipelines for push, web, api and trigger sources),
default: interruptible: true and stages: [build, test]. Both job templates use the
resource group aurora-ci, which serializes the jobs of a project across its pipelines.
A job that overrides before_script must include the environment itself:
before_script:
- !reference [.aurora-env, script]
- export MY_SETTING=1Several Julia versions: define one hidden job per version that sets JULIA_VERSION and
JULIA_DEPOT_PATH, and spell the jobs out (extends: [.aurora-julia-test, .julia-1.13],
needs: ["build:1.13"]). parallel: matrix does not work here, because GitLab does not
expand variables in needs:parallel:matrix.
| Want | Do |
|---|---|
| Run a pull request from a fork | apply the alcf-ci label after reading the diff; re-apply it after new commits |
| Run a commit again | re-run the ALCF CI run of that commit; a fresh pipeline is created if the newest one did not succeed |
| A status is stuck on pending | gh workflow run alcf-report.yml -f pipeline_id=<id> (the id is in the status link or on GitLab) |
| Skip a commit | [skip ci] in the commit message skips both sides |
| Read the logs | the commit status links to the report run, which contains the GitLab job logs |
Three credentials expire, each with a different symptom:
| Credential | Where | Symptom when expired |
|---|---|---|
| GitHub fine-grained token | GitLab webhook header | statuses stay pending; GitLab disables the webhook after repeated failed deliveries |
| GitLab project access token | secret ALCF_GITLAB_TOKEN |
the trigger run warns that there is no pipeline yet, the report run fails with "could not read GitLab pipeline" |
| GitLab trigger token | secret ALCF_GITLAB_TRIGGER_TOKEN |
fork pull requests and re-runs fail; trigger tokens do not expire by default, but die with their owner's access |
After renewing the webhook token, re-enable the webhook and report the missed pipelines by hand. GitLab does not repeat a failed delivery.
Expiry dates of the GitLab side:
glab api projects/USER%2FPROJECT/access_tokens | jq '.[] | {name, expires_at, active}'
glab api projects/USER%2FPROJECT/hooks | jq '.[] | {url, alert_status}'- One queued job per user and queue. The Aurora debug queues admit a single queued job
per user; a second
qsubis refused, which shows up as a failed job. The resource group serializes the jobs of one project, but not of two projects of the same ALCF user. Give such projects different queues (debug,debug-scaling), or expect occasional refused submissions when both have pipelines at the same time. - 1 h walltime in the debug queues. Keep downloads and precompilation in the build job.
- Public GitHub repositories only. The mirror and the fork push fetch without credentials.
- Job logs become as visible as the repository's Actions runs. Set
publish_logs: falseif the logs may contain something that should stay behind the ALCF login. - The check appears twice in a pull request: the short
ALCF CIworkflow run (did the pipeline start) and the commit status (the result). Only the commit status says whether the tests passed. - Pull requests from forks are not updated automatically; each new head needs the label again.
Both sides run code from this repository with the project's credentials:
- the reusable workflows receive the GitLab tokens of the calling project, and
- the GitLab template is executed as the ALCF user of the mirror project.
Whoever can push to the ref a project uses can therefore act as that project on ALCF. A
project that does not want to follow v1 pins both references to a commit:
uses: anlsys/alcf-ci/.github/workflows/trigger.yml@<commit sha> - remote: 'https://raw.githubusercontent.com/anlsys/alcf-ci/<commit sha>/gitlab/aurora-julia.yml'v1 is a moving tag: fixes and compatible additions are released by moving it, so projects
on @v1 get them without a change of their own. A change that needs action in the projects
(a renamed input, a changed commit-status name, a different token setup) gets a new major
tag. main is where changes land first; test them from one project with @main before
moving the tag.
For a repository that already has the full alcf.yml and alcf-report.yml:
- Replace both files with the examples. Keep the file names, the
contextthe project already uses (e.g.ALCF / LTS stack (Aurora)), and setgitlab_project. The numeric project id is no longer needed. - Nothing changes on GitLab: the mirror, the tokens, the secrets and the webhook stay as they are.
- In
.gitlab-ci.yml, replace the include ofanl/ci-resources/defaults, theworkflow,default, CPU-target variables and.aurora-envwith the include of the template, and let the jobs extend.aurora-julia-buildand.aurora-julia-test. Project-specific stages (e.g. coverage) are declared by overridingstages. - The job of the trigger run is now named
<caller job> / Start pipeline. The commit status keeps its name, so a ruleset that requires the status needs no change; one that requires the old job name does.