TransformerKernelsTest#testReductionOneBlockWithLayer is non-deterministic: inter-workgroup race in the test kernel #1059

Describe the bug

TransformerKernelsTest#testReductionOneBlockWithLayer is non-deterministic: it fails about half
the time on the OpenCL backend and consistently on the CUDA backend, on develop, with a plain
value mismatch such as

expected:<0.8730353> but was:<0.9308163>

The input is deterministic — private final Random random = new Random(7), and JUnit gives each
test method a fresh instance, so fillRandomData produces the same array every run. The variation
is therefore in the device-side computation, not the data.

The cause is an inter-workgroup race in the test kernel (reductionOneBlockWithLayer). Each
workgroup reduces its own slice into local memory and writes one partial sum:

if (lid == 0) {
    output.set(groupId + 1, localX[0]);   // one slot per workgroup
}

if (gid == 0) {
    float ss = 0.0f;
    for (int i = 1; i < output.getSize(); i++) {
        ss += output.get(i);              // reads every OTHER workgroup's partial sum
    }
    ...
    output.set(0, ss);
}

Thread gid == 0 reads the partial sums of all eight workgroups, but the only synchronisation in
the kernel is context.localBarrier(), which orders threads within a workgroup. There is no
barrier across workgroups in either OpenCL or CUDA, so nothing guarantees that groups 1..7 have
written output[2..8] before group 0's thread 0 reads them. Whatever those slots happen to contain
at that moment goes into the sum.

That fully accounts for the observed behaviour: it depends on the order the scheduler happens to
run the workgroups in, which is why OpenCL lands on either side of the assertion and CUDA, with a
different scheduling policy, gets it wrong every time.

The same test class already contains two variants that do this correctly and pass —
testReductionOneBlockTwoStepApproach and testReductionOneBlock2WithLayer — which suggests this
one was left behind rather than intended as a different scheme.

How To Reproduce

make BACKEND=opencl
tornado-test uk.ac.manchester.tornado.unittests.compute.TransformerKernelsTest -V

Run it repeatedly; testReductionOneBlockWithLayer alternates between PASS and FAIL. Measured over
20 consecutive runs on one machine, on develop:

Expected behavior

The test should either pass deterministically or not exist in this form. Two options:

1. Split into two kernels — one per-workgroup reduction, then a second task combining the
   partial sums. This is what testReductionOneBlockTwoStepApproach already does, and it is the
   only way to get a cross-workgroup dependency ordered correctly.
2. Assert only what one workgroup can compute. If the intent is to exercise the local-memory
   reduction alone, check output[groupId + 1] for each group and drop the combining step.

Reporting rather than fixing it because which of those matches the intent is a call for whoever
wrote the test — the kernel is exercising a real feature (allocateFloatLocalArray plus
localBarrier), and that part works; it is only the cross-workgroup combine that is unsound.

Computing system setup

• OS: Ubuntu (Linux 6.8.0-58-generic), x86_64
• GPU: NVIDIA GeForce RTX 4090, driver 565.57.01
• OpenCL: NVIDIA CUDA platform (OpenCL 3.0) and Intel UHD 770; failure observed on the NVIDIA device
• CUDA: 12.6, NVRTC 12.6
• JDK 21.0.2
• TornadoVM commit: 7fcd11b10 (develop)

Note

Found while checking backend test parity for #1058. It is unrelated to that PR — it reproduces on
develop with no changes, and the measured flake rate there (9/20 pass) is statistically
indistinguishable from the rate on that branch (13/20 pass).