Skip to content

[CI] Pin the wheel-test torch to a shipped PG build - #4440

Merged
alogfans merged 1 commit into
mainfrom
fix/ci-wheel-torch-pg-pin
Oct 7, 2026
Merged

[CI] Pin the wheel-test torch to a shipped PG build#4440
alogfans merged 1 commit into
mainfrom
fix/ci-wheel-torch-pg-pin

Conversation

@he-yufeng

@he-yufeng he-yufeng commented Oct 1, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Collaborator

Description

Fixes #4439.

The wheel test legs of CI are red on every PR right now. scripts/test_installation.sh installs torch unpinned, so PyPI's latest (2.14.1) lands in the test venv, while the wheel only ships PG builds for EP_TORCH_VERSIONS (currently up to 2.14.0). mooncake.pg then looks for mooncake.pg_2_14_1, and all six test_pg_* modules fail at import.

The wheel is already installed a few lines earlier in the same script, so it now enumerates the mooncake.pg_* modules the wheel ships and installs the newest matching torch. Wheels without PG builds (non-cuda variants, or any detection failure) keep the previous unpinned behavior, so the change cannot make a green run worse.

Module

  • CI/CD

Type of Change

  • Bug fix

How Has This Been Tested?

Test commands:

bash -n scripts/test_installation.sh
# detection snippet run against a stub mooncake package with five fake
# pg_*.so builds (2.11.0 .. 2.14.0), against one with no PG modules, and
# against an environment where import mooncake fails outright

Test results:

  • stub with five PG builds: resolves torch==2.14.0, the newest shipped variant
  • stub with no PG modules: empty result, falls back to unpinned pip install torch numpy
  • import failure: || true keeps set -e intact, same fallback
  • the end-to-end validation is this PR's own test-wheel-ubuntu legs, which should go green again

Checklist

  • I have performed a self-review of my own code
  • I have run pre-commit on the files changed in this PR and all hooks pass (shell script, verified with bash -n; no pre-commit hook covers it)
  • I have updated the documentation (not applicable, no behavior change for users)
  • I have added tests to prove my changes are effective (the CI pipeline itself is the test: this leg is red on every open PR today)

AI Assistance Disclosure

  • AI tools were used (specify below)

Kimi Code drafted the change; verification steps above were actually run as described, and the submitter has reviewed every changed line.

@he-yufeng
[CI] Pin the wheel-test torch to a shipped PG build
d1f6acc 
test_installation.sh installs torch unpinned, so a fresh PyPI release
(2.14.1) lands in the test env while the wheel only ships PG builds for
EP_TORCH_VERSIONS (up to 2.14.0). mooncake.pg then looks for
mooncake.pg_2_14_1, and all six test_pg_* modules fail at import on
every PR (#4439).

The wheel is already installed at that point, so enumerate the
mooncake.pg_* modules it ships and install the newest matching torch.
Wheels without PG builds (non-cuda) keep the previous unpinned
behavior.

Signed-off-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com>
@he-yufeng he-yufeng mentioned this pull request Oct 2, 2026
Open
12 of 14 tasks
@he-yufeng

he-yufeng commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator Author

Status update: the end-to-end validation the description was waiting for is in. Both test-wheel-ubuntu legs pass on this PR (py312 in 20m58s, py310 in 6m22s), while the same job fails on every other open PR with ImportError: Mooncake PG was not built against torch==2.14.1 (example: job 110676867503 on #4450).

Since this unblocks CI for the whole open queue, a review when someone has a moment would help. @Aionw you merged most of the recent CI fixes, maybe you have context on this one.

This was referenced Oct 2, 2026
Open
Open
Open
Merged
Open
Merged
@he-yufeng

he-yufeng commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

Third day of repo-wide red from this gap: every open PR's test-wheel legs fail with the same missing pg_2_14_1 module (latest run 37178068463 this morning). This PR still passes both test-wheel legs end to end and the change is a one-line pin. If the pin direction looks right, merging it unblocks the whole queue; happy to adjust if a bounded torch range is preferred over a pin.

quantz8a pushed a commit to quantz8a/Mooncake that referenced this pull request Oct 5, 2026
zhangyupeng
Pin wheel-test torch to a shipped PG build
3331b36 
scripts/test_installation.sh installed torch unpinned, so PyPI's latest
(2.14.1) landed in the test venv while the wheel only ships PG builds for
EP_TORCH_VERSIONS (up to 2.14.0); mooncake.pg then looked for
mooncake.pg_2_14_1 and all test_pg_* modules failed at import.

Enumerate the mooncake.pg_* modules the installed wheel ships and install
the newest matching torch. Wheels without PG builds keep the previous
unpinned behavior.

Mirrors kvcache-ai#4440 (fixes kvcache-ai#4439) so this PR's
test-wheel-ubuntu legs go green; drop on rebase once kvcache-ai#4440 lands.
@UNIDY2002 UNIDY2002 self-assigned this Oct 6, 2026
@UNIDY2002

UNIDY2002 commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

I think getting #3673 merged is the ultimate fix.

@he-yufeng

he-yufeng commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Agreed, #3673 is the real fix. This PR is deliberately only an interim pin: it touches nothing outside the wheel-test workflow, it can't conflict with the JIT rework, and it keeps the test-wheel legs meaningful for the other open PRs until #3673 lands (right now all of them fail at import on the missing pg_2_14_1 module). Once #3673 merges I'll close this. If you'd rather not take an interim pin at all, say so and I'll close it now; the lane just stays red until then.

@LingYi-Liang LingYi-Liang mentioned this pull request Oct 7, 2026
Open
9 tasks done
@he-yufeng

he-yufeng commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator Author

Agreed that #3673 is where this should end up. In the meantime the wheel job fails on every open PR: test_installation.sh installs an unpinned torch, which now resolves to 2.14.1, while the shipped wheels only carry PG builds up to 2.14.0, so all six test_pg_* modules die at import. That failure is currently noise on top of every PR's real signal.

This pin just ties the test env to a torch the shipped PG actually covers, so CI reads clean until the JIT adapter removes the coupling. Once #3673 lands I can rebase this down to nothing or close it outright, whichever is cleaner.

@he-yufeng he-yufeng mentioned this pull request Oct 7, 2026
Merged
alogfans
alogfans approved these changes Oct 7, 2026
View reviewed changes

@alogfans alogfans left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alogfans
alogfans merged commit b079bfb into main Oct 7, 2026
28 checks passed
pjdurden added a commit to pjdurden/Mooncake that referenced this pull request Oct 7, 2026
@pjdurden
Merge main to pick up the wheel-test torch pin (kvcache-ai#4440)
d926b51
pjdurden added a commit to pjdurden/Mooncake that referenced this pull request Oct 7, 2026
@pjdurden
Merge main to pick up the wheel-test torch pin (kvcache-ai#4440)
a80b681
LingYi-Liang added a commit to LingYi-Liang/Mooncake that referenced this pull request Oct 8, 2026
@LingYi-Liang
Merge upstream wheel-test fix (kvcache-ai#4440)
2af3ab3
@fcczzz fcczzz mentioned this pull request Oct 9, 2026
Open
10 of 13 tasks
This was referenced Oct 9, 2026
Open
Merged
Open
Open
Open
This was referenced Oct 9, 2026
Open
Open
@fcczzz fcczzz mentioned this pull request Oct 9, 2026
Merged
8 of 32 tasks
@he-yufeng he-yufeng mentioned this pull request Oct 9, 2026
Open
6 of 32 tasks
This was referenced Oct 9, 2026
Open
Open
UNIDY2002 added a commit that referenced this pull request Oct 10, 2026
@UNIDY2002
Revert "[CI] Pin the wheel-test torch to a shipped PG build (#4440)"
673f1e8 
This reverts commit b079bfb.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alogfans alogfans alogfans approved these changes

Assignees

@UNIDY2002 UNIDY2002

Labels

run-ci Tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug]: test-wheel CI legs red on all PRs: unpinned pip install torch pulls 2.14.1, wheel only ships PG builds up to 2.14.0

3 participants

@he-yufeng @UNIDY2002 @alogfans