Repository navigation
[CI] Escape $BUILDKITE_PARALLEL_JOB in flakiness parallel-fanout dispatch - #150223
Conversation
The parallel-fanout step used `${BUILDKITE_PARALLEL_JOB}` (single `$`)
in step.command. Buildkite's pipeline-upload interpolation pass treats
that as an upload-time variable and substitutes empty (the value is
only injected by the agent at job-run time). The runtime command
becomes:
VARNAME="BATCH_COMMAND_"; eval "${!VARNAME}"
Indirect lookup of an unset `$BATCH_COMMAND_` expands to empty, eval
runs nothing, the job exits 0 in ~2 seconds with no JUnit XML, and BK
reports it green. See e.g. build 150689 job 019e73e0-1758.
Escape as `$$BUILDKITE_PARALLEL_JOB` to defer interpolation past the
upload pass; the agent then resolves it correctly to 0, 1, … per job.
Bug has existed since the modularization commit (9003f92) but was
masked: missing test results look identical to "no flakiness found".
Found while verifying the `--foreground` wrapper fix on PR 150209.
|
Pinging @elastic/es-delivery (Team:Delivery) |
|
Verified by #150225: parallel-fanout dispatch fix works in CI — 3 parallel unit-test batches ran real Gradle, uploaded JUnit XML, and analyze reported 13300 iterations. |
* upstream/main: In file watching tests, add 2s to guarantee a timestamp change (elastic#149937) Add query filter to more query types (elastic#150127) [DOCS] Fix setting error and edit reindex-indices.md (elastic#150139) Rename ResumeBulkByScrollRequest to ResumeBulkByPaginatedSearchRequest (elastic#150210) Fail search shards in resharding disruption tests (elastic#150124) Tests: Fix ShardSearchPhaseAPMMetricsTests (elastic#150200) ES|QL PromQL: implicit type coercion for counter/gauge (elastic#149985) Fix OTelMetricsBufferingIT: explicitly produce multiple delta batches (elastic#150136) [Test] Fix testActionStats off-by-one on powers of two (elastic#150218) Escape BUILDKITE_PARALLEL_JOB in batch dispatch (elastic#150223) Handle patterns that require resource on every project (elastic#149980) Mute org.elasticsearch.xpack.inference.external.http.sender.RequestExecutorServiceTests testChangingCapacity_DoesNotRejectsOverflowTasks_BecauseOfQueueFull elastic#150226 Add timestamp accumulator for cache quota tracking (elastic#149890)
Post-merge verification: confirmed working on a real PR buildFollowing up on the open test-plan item ("observe next flakiness-detection parallel-fanout step on a real PR"). The earlier check was on the synthetic verification PR #150225; this confirms the fix on a genuine post-merge PR build running Build: elasticsearch-pull-request #153989 (unrelated contributor PR, 2026-06-16) The
No overlap across jobs → each parallel job ran its own batch, which is only possible if Analyze step: 019ed02b-7fe9-4531 — downloaded all 6 distinct XMLs and reported real data: This is not a one-off: scanning the last ~20 passed post-merge builds, 7 had a genuine parallel flakiness fan-out (153989, 153974, 153951, 154017, 154014, 154004, 153978), all passed with (Note: the echoed |
What & why
step.commandintoBuildkitePipeline(the parallel-fanout dispatcher) was emitting:${BUILDKITE_PARALLEL_JOB}(single$) is a per-job runtime variable, but Buildkite's pipeline-upload interpolation pass runs before the agent injects per-job env. BK can't resolve it at upload time, so it substitutes empty. The runtime command becomes:Bash indirect lookup
${!VARNAME}of the unset$BATCH_COMMAND_expands to empty,eval ""is a no-op, the job exits 0 in ~2s with zero JUnit XML, and BK reports it green.Bug has been present since the flakiness-detection modularization commit
9003f9233a6c. It was masked because the analyzer treats "no artifacts" the same as "no flakiness found" — silent no-op all the way through. It surfaced today only after #150209 (the wrapper-hang fix) let parallel jobs finish quickly enough to inspect the log end-to-end.Example before the fix: build 150689 job
019e73e0-1758Change
$$BUILDKITE_PARALLEL_JOBdefers interpolation past BK's upload pass; the agent then resolves it correctly to0, 1, …per parallel job.Test plan
bun test scripts/flakiness-detection— 109 pass / 208 expect()step.commandmust contain$${BUILDKITE_PARALLEL_JOB}and must not contain the unescaped formVARNAME="BATCH_COMMAND_0"(or 1, 2, …) and produce real JUnit XML.