Skip to content

cuda-cccl 1.2.0 on Windows: access violation in call_build on first make_reduce_into (regression from 1.1.1) #11472

Closed

Description

@leofang

Every Windows job in CuPy's post-merge FlexCI matrix started segfaulting on 2026-09-17 at the moment cuda-cccl 1.2.0 was published to PyPI (16:01 UTC). The last passing PR-CI run against the identical CuPy code (SHA 4da8723d5, Windows CUDA 13.0) started at 15:50 UTC — 11 minutes before 1.2.0 shipped — and installed 1.1.1. Every post-1.2.0 run installs 1.2.0 (the CI script does pip install "cuda-cccl[minimal-sysctk1x]>=1.1.1" unpinned) and crashes.

Symptom

Exit code -1073741819 (0xC0000005, ACCESS_VIOLATION). The Python fatal-error handler dumps the interpreter stack; the C stack is unavailable. Identical stack on every failing job (13 CUDA × Python combinations, Python 3.10 through 3.14):

Windows fatal exception: access violation

Current thread <tid> (most recent call first):
  File ".../cuda/compute/_cccl_interop.py", line 329 in call_build
  File ".../cuda/compute/_cccl_interop.py", line 376 in build_for_ccs
  File ".../cuda/compute/algorithms/_reduce.py", line 129 in <lambda>
  File ".../cuda/compute/_caching.py", line 267 in _cache_single_flight
  File ".../cuda/compute/_caching.py", line 454 in cache_build_results
  File ".../cuda/compute/algorithms/_reduce.py", line 120 in __init__
  File ".../cuda/compute/algorithms/_reduce.py", line 275 in make_reduce_into
  File ".../cuda/compute/_caching.py", line 577 in inner
  <cupy fusion test calling x.sum()>

Full console for two representative jobs: job 227662 (Windows CUDA 13.0, Python 3.14) and job 227657 (CUDA 12.0, Python 3.10). Both show Successfully installed cuda-cccl-1.2.0 in the public build log (227662 build log).

The fault is inside build_impl_fn(*args, common_data, **kwargs) at _cccl_interop.py:329 — the C++ HostJIT compile entry point for the first make_reduce_into call. CUPY_ACCELERATORS=cuda_compute,cub is set on this lane, so CuPy's reduction accelerator is what invokes it, but the fault is inside cuda.compute; the CuPy caller has been unchanged since PR cupy/cupy#10158 merged on 2026-09-14 and was green on 1.1.1 as recently as bdc4a0f31.

Environment

cuda-cccl 1.2.0 Windows wheels (cp310–cp314) installed via cuda-cccl[minimal-sysctk1x]>=1.1.1 at test time. Windows Server 2022 (FlexCI runner), CUDA 12.0–13.3 across the matrix, NumPy 2.0–2.5. Reproduces on every one of the 13 CUDA × Python combinations.

Bisect vs 1.1.1

Downgrading to 1.1.1 restores the same runners to green — the last PR-CI Windows CUDA 13.0 job on the same CuPy tip 4da8723d5 took 1h9m and passed. The _cccl_interop.py diff between 1.1.1 and 1.2.0 adds precompiled-header plumbing (_ensure_pch_configured, _evict_pch, new _pch.py) around the build call and replaces the {cc_key: build_result} dict with a _PerCCBuildResults wrapper. Both are candidates for how the crash reaches build_impl_fn, though a Python faulthandler dump doesn't recover the C-side stack, so this isn't yet a proven root cause on cuda.compute's side.

Related

#10802 tracks intermittent Windows HostJIT access violations across algorithms (including reduce and histogram) with a stack-size hypothesis. This report is a deterministic 100 %-reproducible variant that started with 1.2.0 rather than an intermittent one, so I'm filing separately in case root causes differ; happy to consolidate if triage finds otherwise.

-- Leo's bot

Activity

  1. added theissue type on Sep 18, 2026
  2. added
    cuda.computeFor all items related to the cuda.parallel Python module
    on Sep 18, 2026
  3. self-assigned this
    on Sep 18, 2026
  4. moved this from Todo to In Progress in CCCLon Sep 18, 2026
  5. moved this from In Progress to In Review in CCCLon Sep 18, 2026
  6. moved this from In Review to Done in CCCLon Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

cuda.computeFor all items related to the cuda.parallel Python module

Type

Projects

  • Status
    Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions