Repository navigation
Tags
Tags
gobackend: ARM64 NEON/SME half-precision & transformer optimizations (#… …82) ## Summary This PR optimizes transformer / embedding model execution on ARM64 (NEON & Apple M-series SME) across both native half-precision (`Float16` / `BFloat16`) and `Float32` execution paths: 1. **ARM64 NEON Half-Precision Slice Converters (`internal/gobackend/neon`, `dtypes/{float16,bfloat16}`)**: - Registers hardware NEON `FCVTL`/`FCVTL2` and `FCVTN`/`FCVTN2` (`Float16` $\leftrightarrow$ `Float32`) and `SHLL`/`SHLL2` + round-to-nearest-even (`BFloat16` $\leftrightarrow$ `Float32`) slice converters via `gobackend.RegisterHalfSliceConverters`. - Raises `ParallelHalfConvThreshold` and `convertDTypeMinParallelElems` to `262144` elements and skips `ParallelizeChunks` when the scheduler is already busy with concurrent workers. 2. **ARM64 NEON/SME `MultiRHS` MatMul & Fused Epilogues (`internal/gobackend/dot/matmul/neon`)**: - Implements `neonLargeMulti{Float32,BFloat16,Float16,Float64}` and `smeLargeMulti{Float32,BFloat16,Float16,Float64}` and registers them via `matmul.RegisterMultiRHSImplementation`, enabling shared LHS panel packing across Q/K/V in `FusedAttentionQKVProjection` and cache-resident tile epilogues (bias + activation) in `FusedDense`. - Adds ARM64 NEON `ApplyEpilogueTile` bias + GELU/ApproxGELU kernels (`internal/gobackend/activations/neon`). - Optimizes partial 16/32-row strip packing in `neonPackLHS` using `clear(...)` + 8-row NEON sub-strip assembly (`neonPackLHSKernelRows8Strided*Asm`), reducing scalar remainder copies to at most 7 rows. - Software-pipelines 2 K-steps in `smeLargeKernel{Float16,BFloat16}Asm_32x32` and `neonSubTile8x8Strided{Float16,BFloat16}Asm` to hide `LD1H`/`FCVT`/`FCVTL`/`SHLL` latency. 3. **Half-Precision `FusedLayerNorm` & Graph Fusion (`internal/gobackend/fusedops`, `internal/gobackend/executable.go`)**: - Adds native `Float16` and `BFloat16` support to `FusedLayerNorm` (accumulating in `float32` per row in L1 cache and supporting in-place buffer reuse). - Extends `fuseSingleConsumerResidualAddIntoLayerNorm()` to fuse `Add(a, b) -> ConvertDType(Half -> Float32) -> FusedLayerNorm(Float32) -> ConvertDType(Float32 -> Half)` into a single `FusedLayerNorm` node with an optional residual input, eliminating 3 intermediate ops and buffer allocations per LayerNorm block. - Adds ARM64 NEON assembly for `FusedLayerNorm` (`internal/gobackend/fusedops/neon`). 4. **ARM64 NEON Binary Broadcast Kernels (`internal/gobackend/ops/neon`, `internal/gobackend/ops/binaryops_simd.go`)**: - Adds ARM64 NEON fast paths for `Float16`, `BFloat16`, and `Float32` binary ops (`Add`, `Sub`, `Mul`, `Div`, `Max`, `Min`) across all broadcast patterns (`BroadcastNone`, `BroadcastLeading`, `BroadcastMid`, `BroadcastTrailing`, scalar). 5. **ARM64 NEON Fused SDPA & Softmax (`internal/gobackend/dot/matmul/neon`, `internal/gobackend/fusedops/neon`)**: - Adds NEON vectorization for `FusedScaledDotProductAttention` (`sdpaNEONBlockFloat32`, `sdpaScaleMaskSoftmaxRowNEON`, `sdpaSoftmaxRowNEON`, `sdpaValueAccumBlockNEON`) and `FusedSoftmax`. 6. **SME Cluster Lock & Scheduler Contention Reduction (`internal/gobackend/simdarch`, `internal/gobackend/scheduler`)**: - Caches `CurrentSMECluster()` (`ReadTPIDR_EL0`) and spins on a relaxed L1-cached `.Load() < limit` check in `AcquireSMEWithSpin` before attempting atomic `.Add(1)`. - Caches `runtime.GOMAXPROCS(0)` in `Scheduler` so `UnscaledParallelism()` avoids global `runtime.sched.lock` contention on every op. ## Benchmark Results (Apple M6, `All-MiniLM-L6-v2-Embedding-GGUF`, `f16`, 6 embedders, 10K passages) | Mode | Warmup (Before $\rightarrow$ After) | Benchmark Wall Time | Throughput (Passages/s) | Throughput (Tokens/s) | | :--- | :--- | :--- | :--- | :--- | | **Native F16 (`-force_f32=false`)** | `1.20s` $\rightarrow$ **`663.2ms` (1.81x)** | `19.2s` $\rightarrow$ **`16.2s`** | `521/s` $\rightarrow$ **`616/s` (+18.2%)** | `40.2K/s` $\rightarrow$ **`47.7K/s` (+18.7%)** | | **Forced F32 (`-force_f32=true`)** | `1.00s` $\rightarrow$ **`610.6ms` (1.64x)** | `15.6s` $\rightarrow$ **`14.2s`** | `641/s` $\rightarrow$ **`705/s` (+10.0%)** | `49.5K/s` $\rightarrow$ **`54.6K/s` (+10.3%)** |
gobackend, shapeinference: support dynamic RNGBitGenerator and intege… …r ConvGeneral validation (#77) ## Summary - **`gobackend` (`RNGBitGenerator`)**: Resolve dynamic output dimensions at execution time via `f.DynamicAxisValue`, enabling dynamic-shape random number generation (e.g. ONNX `RandomNormalLike`). - **`support/backendtest`**: Add `DynamicRNGBitGenerator` compliance test verifying multiple dynamic batch sizes (including zero-sized outputs) on a single compiled executable. - **`shapeinference` (`ConvGeneral`)**: Validate that `input` and `kernel` share a matching numeric, non-packed `DType` (supporting `Int32` convolutions for ONNX `ConvInteger`).
gobackend/dot/matmul: fix AVX-512 multi-worker useAccum copy-back til… …ing and AVX2 half-precision/GoSIMD kernels (#67) ## Summary Fixes a multi-threaded AVX-512 `LargeMatMul` regression introduced in #63 (`v0.1.26`), along with two latent AVX2 `LargeMatMul` kernel bugs uncovered while adding multi-dtype regression tests. ### 1. AVX-512 Multi-Worker `useAccum` Copy-Back Tiling Mismatch (`internal/gobackend/dot/matmul/avx512`) - **Bug**: When #63 ported L2 row sub-chunking (`maxMPanelsPerSubChunk`) from `avx2/large.go` to `avx512/large.go`, `Loop 3` (`avx512LargeMatrixSlice*`) was updated to advance `lhsPanelEnd` in relative `lhsPanelRowIdx + Mc` increments when `len(cachedLHSPanels) == 0`, while the `if useAccum` L2 accumulator copy-back loop still used global `Mc`-aligned grid boundaries (`(rowPanelIdx + 1) * Mc`). - **Impact**: Whenever multi-worker row splitting (`GOMAXPROCS >= 2`) produced a worker slice starting at `rowStart % Mc != 0` with unaligned tail rows `rowEnd % Mr != 0` and `numKPanels > 1` (e.g., `mistralai/Timestral-123M` `QuantileHead` `[36, 2560] @ [2560, 672]` at `rowStart = 24, rowEnd = 36` on `H=576`, or `wO` `[810, 640] @ [640, 640]` at `rowStart = 712, rowEnd = 810` on `H=864`), `useAccum` either skipped copy-back (`24..32` looked `Mr`-aligned) or read from the wrong `accumBuffer[mIdx * panelSize]` slot, leaving uninitialized pooled-buffer values in output rows from `rowStart` onward (forecast steps `192..575`). - **Fix**: Updated the `if useAccum` copy-back loop in `avx512/large.go` (and regenerated `gen_large_{bf16,f16,f64}.go`) to compute `lhsPanelEnd` identically to `Loop 3` and `avx2/large.go`, and ensured `params.LHSL1KernelRows = 4` is set whenever `!AVX512UseAsm`. ### 2. AVX2 Go-SIMD Direct-Output BCE Check & Half-Precision ASM `Y13` Clobber (`internal/gobackend/dot/matmul/avx2`) - **Go-SIMD BCE bounds check (`avx2/large.go`, `avx512/large.go`)**: Fixed `avx2LargeKernel*` checking `packedOutput[lhsActiveRows*rhsPanelCols-1]`, which panicked with `index out of range` under `GOMLX_GO_AVX2_ASM=false` when `canDirectOutput == true` wrote to a non-zero column tile with `rhsActiveCols < rhsCrossSize`. Aligned both `avx2LargeKernel*` and `avx512LargeKernel*` to check `(maxRow-1)*rhsPanelCols + maxCol - 1`. - **AVX2 `BFloat16` / `Float16` ASM (`large_amd64_{bfloat16,float16}.s`)**: Fixed `VPERMILPS $0xFF, X10, X13` executing two instructions before Row 3's `VFMADD231PS Y8, Y13, Y6` / `VFMADD231PS Y9, Y13, Y7` in `k_pair_loop` and `k_last_pair`, which clobbered `Y13` when `K >= 2`. ## Testing - Added `TestAVX512LargeParallelUnalignedPanels` (`internal/gobackend/dot/matmul/avx512/internal_test.go`) covering `workers = 2, 3, 4, 8`, pre-poisoned backend buffer pools, `[36, 2560] @ [2560, 672]`, `[810, 640] @ [640, 640]`, batched/2D-unaligned shapes, `Uncached` / `CachedRHS` / `CachedLHS` / `MultiRHS`, ASM + Go SIMD, and `Float32` / `BFloat16` / `Float16` / `Float64`. - Added `TestAVX2LargeParallelAndGoSIMDBounds` (`internal/gobackend/dot/matmul/avx2/internal_test.go`) covering multi-column-panel direct-output slices and `Float32` / `BFloat16` / `Float16` / `Float64` under ASM and Go SIMD. - Verified `GOEXPERIMENT=simd go test -race ./...` across `compute` and end-to-end `mistralai/Timestral-123M` vs PyTorch across `GOMAXPROCS = 1, 2, 3, 4, 8, 32` (`Max Abs Diff = 1.5e-05` on all thread counts).
Document, validate, and test negative Start/End margins in Pad and Dy… …namicPad (#64) In support of gomlx/onnx-gomlx#104. ### Summary - **Documentation**: Document in `PadAxis`, `StandardOps.Pad`, `DynamicPadAxis`, and `DynamicOps.DynamicPad` (`ops.go`, `ops_dynamic.go`) that `Start`/`End` (and `StartValue`/`EndValue`) accept negative values to truncate (crop) elements from the start or end of an axis (applied after interior padding), while `Interior` (and `InteriorValue`) must be non-negative (`>= 0`). - **Shape Inference**: - Update `shapeinference.Pad` and `shapeinference.DynamicPad` with full docstrings, examples, operand dimension validation, and clearing static axis names when static padding modifies the dimension size without a new `TargetAxisName`. - Add unit tests and testable examples (`TestPad`, `ExamplePad`, `TestDynamicPad`, `ExampleDynamicPad`). - **Compliance Tests**: - Expand `Pad` and `DynamicPad` compliance tests in `support/backendtest/special.go` to cover negative `Start`/`End`, combined interior and negative edge padding, cropping all elements, 2D negative margins, and partial `axesConfig`.
gobackend: ARM64 SME/NEON & AMD64 AVX2/AVX-512 optimizations for Time… …stral, SDPA, SwiGLU, binary broadcasts, and bufferPool (#63) ## Summary This PR optimizes the pure-Go backend (`internal/gobackend`) across **ARM64 (NEON + SME)**, **AMD64 AVX2**, and **AMD64 AVX-512**, covering both `DotGeneral` (`matmul`) scheduling/topology and non-MatMul Transformer/Time-Series operations (`FusedScaledDotProductAttention`, `FusedActivation` / `SwiGLU`, binary broadcasts, reductions, `Iota`, `Where`, and backend buffer pooling). Full profiling analysis and step-by-step benchmark logs are documented in: - `docs/OptimizeTimestral.md` - `internal/gobackend/simdarch/Neon.md` - `docs/GoBackendScheduler.md` --- ## Key Changes ### 1. Centralized `simdarch`, 1-Cycle `TPIDR_EL0` 2-L2 / 2-SME Topology, 5-Tier Hybrid Scheduling & Two-Regime Scheduler - **`internal/gobackend/simdarch`**: Centralized SIMD feature flags (`IsNEONAllowed`, `IsSMEAllowed`, `IsAVX2Allowed`, `IsAVX512Allowed`, `NEONUseAsm`, `AVX2UseAsm`, `AVX512UseAsm`) and 1-cycle `MRS TPIDR_EL0, R0` SME cluster detection/locking (`AcquireSME`, `AcquireSMEWithLimit`, `AcquireSMEWithSpin`, `ReleaseSME`, `CurrentSMECluster`). - **5-Tier Heterogeneous Worker Model & Fused In-SME Small/Medium GEMM (`internal/gobackend/dot/matmul/neon`)**: - 3-zone tapered 2D work partitioning (`smeAppendHybridWorkItems`: coarse $64 \times 128$, medium $32 \times 64$, fine $32 \times 32$ tiles) and 5-tier dual-ended claiming (`claimItem`). - Split multi-worker execution into `//go:noinline func neonLargeParallelWithParams*` + `neonScratchPool`, added hardware $32 \times 32$ `ZA0..ZA3` transpose (`smePackLHS32Float32Asm`), and fused single-`SMSTART` strip-transpose + direct unpacked strided RHS (`smeDirectNonTransposedFloat32Asm_32x32`), accelerating $64^3$–$256^3$ GEMMs by **1.92×–6.77×** and reaching **2,482–2,543 GFlops/s** on large GEMMs (**1.05×–1.14× faster** than Apple `Accelerate` `cblas_sgemm`). - **Two-Regime Inter-Call / Inter-Op Scheduling (`internal/gobackend/scheduler`)**: - Added `Scheduler.AcquireWholeChip()` / `ReleaseWholeChip()` (FIFO 1-slot channel gate for large ops $> 16\text{M}$ MACs vs. concurrent per-cluster execution for small/medium ops $\le 16\text{M}$ MACs), separated `interOpQueue` workers from `intraOpQueue` workers, and made `RHSConstantData` cached panel count invariant across `maxWorkers` transitions. ### 2. Multi-Row `FusedScaledDotProductAttention` (`SDPA`) Across ARM64 SME/NEON, AVX2, and AVX-512 - **ARM64 SME + 4-Row NEON (`internal/gobackend/dot/matmul/neon/sdpa_arm64.go`)**: - Unlocked `sdpaSMEBlockFloat32` ($4..32$ query rows) and `sdpaNEONBlockFloat32` ($1..4$ query rows) with `booleanMask` and `causal` masking, lazy once-per-KV-head `ZA` $K$ transpose (`smePackLHS32Float32Asm`), fused `ZA` probability transpose + direct strided $V$ `FMOPA` (`smeDirectStridedFloat32Asm_32x32`), per-phase SME lock release during parallel NEON softmax, and trailing masked key pruning (`sdpaActiveKVLenForBlock`) (**3.02×–3.26× faster** SDPA on Apple M6). - **AMD64 4-Row AVX2 & 4-Row AVX-512 Assembly (`internal/gobackend/fusedops/{sdpa.go,avx2,avx512}`)**: - Added 4-query-row register-blocked $Q K^T$ and $P V$ assembly kernels (`sdpaQKT4RowsAVX2Asm`, `sdpaValueAccum4RowsAVX2Asm`, `sdpaDot4RowsFloat32AVX512`, `sdpaWeightedAccum4RowsFloat32AVX512`) with `headDim == 64` in-register accumulators, vectorized boolean mask blending inside fused Estrin softmax, `isAllTrueBoolMask` fast path, causal active-KV pruning, and dynamic `qBlockSize` targeting `4 * numWorkers` tasks (**11.74× faster** SDPA on AVX2; **2.33× faster** SDPA on AVX-512). ### 3. Handwritten `SwiGLU` (`FusedActivation`), `Sigmoid`, and `Tanh` Assembly - Split `SwiGLU` into dedicated `swiglu.go` and `swiglu*.s` files in `internal/gobackend/activations/{neon,avx2,avx512}/`: - **ARM64 NEON (`swigluFloat32NEONAsm`)**: 4× unrolled (16 floats/iter) with hoisted `V19`–`V31` constants and overlapped `FDIV` + `FRECPE`/`FRECPS` Newton-Raphson fast reciprocal (**4.3×–4.5× faster**). - **AMD64 AVX2 (`swigluFloat32AVX2Asm`)**: 2× YMM (16 floats/iter) with overlapped `VDIVPS` + `VRCPPS` (**4.51× faster**). - **AMD64 AVX-512 (`swigluFloat32AVX512Asm`, `sigmoidFloat32AVX512Asm`, `tanhFloat32AVX512Asm`)**: 2× ZMM (32 floats/iter) with hoisted `Z16`–`Z29` constants and overlapped `VDIVPS` + `VRCP14PS` (**5.19× faster**). ### 4. Binary Mid/Leading/Trailing Assembly, 3-Segment Broadcasts, `Iota`, `Where`, and AVX-512 MatMul - **Zero-Spill Binary Mid/Leading/Contiguous Assembly (`internal/gobackend/ops/{avx2,avx512}/binary_mid_amd64.s`)**: Eliminates Go 1.27 `simd` slice-header stack spills on `amd64` with register-cached `C == 64` (and `C == 128` on AVX-512) fast paths; 4× unrolled AVX-512 `binary_trailing_amd64.s`; parallelized `tryBinaryTrailingArch` and `ReduceTrailing` across outer rows `A`. - **3-Segment Broadcasts & `Iota` (`internal/gobackend/broadcast.go`, `internal/gobackend/ops/iota.go`)**: Added `BroadcastRowColMatch`, `BroadcastColRowMatch`, `BroadcastMatchRowCol`, and `BroadcastMatchColRow` (`[A, 1, C]` op `[1, B, C]` and `[A, B, 1]` op `[A, 1, C]`), and optimized `Iota` with exponential doubling `copy()`. - **AVX-512 MatMul & Per-Arch `Where` Subpackages (`internal/gobackend/dot/matmul/avx512`, `internal/gobackend/ops/{avx2,avx512,neon}/where*`)**: Ported L2 row sub-chunking and cooperative `sharedRHSPanels` packing to `matmul/avx512`, added `avx512PackLHSKernelRows8{Float64,Float16,BFloat16}Asm` and `avx512SmallTransposedTile4x1*Asm`, added 512-bit AVX-512 `Where` assembly, and split `Where` into `ops/{avx2,avx512,neon}/`. ### 5. Lock-Free Atomic Treiber Stack `bufferPool` & Zero-Allocation `GetBuffer` - Replaced `sync.Pool` and `sync.Map` in `Backend.bufferPools` (`internal/gobackend/buffers.go`) with a direct-indexed `[128]atomic.Pointer[bufferPool]` array of lock-free LIFO Treiber stacks using a 64-bit ABA-tagged atomic head (`(tag << 32) | slotIdx`), retained `Buffer.Flat` slice headers across reuse, and eliminated variadic slice heap escape in `DType.SizeForDimensions` (`0 B/op, 0 allocs/op`, `13.85 ns/op`). --- ## Benchmark Summary (`mistralai/Timestral-28M` & `Timestral-123M`) | Platform / Architecture | Model | Before (`go`) | **After (`go`)** | `xla:cpu` Reference | Speedup vs Baseline | Speedup vs `xla:cpu` | | :--- | :--- | ---: | ---: | ---: | ---: | ---: | | **Apple M6 (ARM64 NEON + SME)** | `Timestral-28M` | 58.0 ms (`82.7 ms` NEON) | **39.1 ms** | — | **`1.48×` (`2.12×` vs NEON)** | — | | **Apple M6 (ARM64 NEON + SME)** | `Timestral-123M` | 179.4 ms (`293.4 ms` NEON) | **120.0 ms** | — | **`1.50×` (`2.45×` vs NEON)** | — | | **Intel Core i9-12900K (AMD64 AVX2)** | `Timestral-28M` | 134.4 ms | **60.3 ms** | 73.7 ms | **`2.23×`** | **`1.22× faster`** | | **AMD Ryzen 9 9950X3D (AMD64 AVX-512)** | `Timestral-28M` | 40.5 ms (`47.8 ms` AVX2) | **35.8 ms** | 51.8 ms | **`1.13×` (`1.34×` vs AVX2)** | **`1.45× faster` (`1.87×` less CPU)** | | **AMD Ryzen 9 9950X3D (AMD64 AVX-512)** | `Timestral-123M` | 155.6 ms (AVX2) | **124.8 ms** | 163.9 ms | **`1.25×` vs AVX2** | **`1.31× faster` (`1.35×` less CPU)** |
gobackend: Diffusion / FlowMatching U-Net training optimizations (AMD… …64 AVX2/AVX-512 & ARM64 NEON/SME) (#55) ## Summary This PR implements 6 end-to-end performance optimizations in `gobackend` identified by profiling the GoMLX `FlowMatching` U-Net diffusion training workload (`batch=32`, $64 \times 64 \times 3$ Oxford Flowers 102, 4-level U-Net with Sinusoidal Embedding, Residual Blocks, `swish`/`silu` activations, and spatial `LayerNorm(axes=[1, 2])`), across both **AMD64 (`AVX2` / `AVX-512`)** and **ARM64 (`NEON` / `NEON+SME`)**. Detailed profiling and step-by-step benchmark logs are documented in [`docs/DiffusionOptimization.md`](./docs/DiffusionOptimization.md). ### Key Optimizations 1. **`ReduceMiddle` SIMD Dispatch (`internal/gobackend/ops/reduce_simd.go`, `avx512/`, `neon/`)**: - Enabled `ReduceMiddle` (`[A, B, C] -> [A, 1, C]`, used in spatial LayerNorm backward passes) in `canExecuteReduceSIMD` and dispatched `ReduceSum`, `ReduceMax`, `ReduceMin`, and `ReduceProduct` via the `ReduceLeading` SIMD and architecture-specific assembly kernels (`reduceLeadingSumFloat32AVX512` and 8-accumulator `reduceLeadingSumFloat32NEON`), parallelized across `A`. 2. **`FusedDenseVJP` Parallel Cache-Tiled `transpose2D` & Buffer Pooling (`internal/gobackend/fusedops/dense/dense_vjp.go`)**: - Pooled temporary buffers (`dZ`, `xT`, `dZT`, `tmpF32_dX`, `tmpF32_dW`) via `backend.GetBuffer` / `backend.PutBuffer`, cutting total heap allocations per step by **51.4%** (`592 MB/step` $\to$ `288 MB/step`) and preventing mid-step GC cycles from evicting `Backend.bufferPools`. - Parallelized `transpose2D` across `backend.Scheduler.ParallelizeChunks` using $32 \times 32$ cache-tiled blocks and optimized `reduceBiasFloat`. 3. **Out-of-Place Forward Activations via `RegisterKernel` (`internal/gobackend/activations/`)**: - Converted AVX2 and AVX-512 `float32` forward activation kernels (`Silu`, `Relu`, `Sigmoid`, `HardSigmoid`, `LeakyRelu`, `Selu`, `HardSwish`, `Tanh`) from in-place `(inout []float32)` (`activations.Register`, which performed a redundant `copy(out, in)` when `x` was retained for VJP) to `(in, out []float32)` via `activations.RegisterKernel`. 4. **3-Segment Binary Broadcast (`BroadcastOuterInnerRHS/LHS`) & Register-Pinned `BroadcastMid` (`internal/gobackend/broadcast.go`, `ops/binaryops_simd.go`)**: - Added `BroadcastOuterInnerRHS` and `BroadcastOuterInnerLHS` (`[A, B, C]` op `[1, B, 1]`) so spatial LayerNorm `gamma`/`beta` scaling (`[B, H, W, C] * [1, H, W, 1]`) uses vectorized `vs`/`sv` slices instead of falling back to scalar `ZipIterator`. - Updated `BroadcastMidRHS` / `BroadcastMidLHS` to use register-pinned `v2v1`/`v1v2` loops across rows even when `B*C > 4096`, with unrolled `C = 64, 96, 128` fast paths. 5. **Middle-Axis `FusedLayerNorm` SIMD (`internal/gobackend/fusedops/layernorm_simd.go`)**: - Added `simdLayerNormMiddleAxesFloat32` to vectorize non-trailing `FusedLayerNorm` (e.g. `axes=[1, 2]` on `[B, H, W, C]`, where `dDim == 1 && cDim > 1`) across the contiguous channel dimension `cDim`. 6. **`ConvGeneral` Direct Zero-`im2col` 2D Microkernels & `(BatchSize, OutH)` `VJPKernel` Slicing (`internal/gobackend/ops/convgeneral_*.go`, `avx2/`, `avx512/`, `neon/`)**: - Added direct zero-`im2col` 2D forward/VJP-input assembly microkernels (`ConvAVX512Direct2DKernel8x32Float32Asm`, `ConvAVX512Direct2DKernel4x32Float32Asm`, `ConvAVX2Direct2DKernel4x16Float32Asm`, `ConvNEONDirect2DKernel4x16Float32Asm`) and direct-from-input SME LHS packing (`ConvPackLHS8StridedFloat32Asm`) + fast-cluster dual-worker pipelining (`SMEAcquireClusterWithLimit(2, 1)`), eliminating >95% of `ExtractPatches2DFloat32` `im2col` buffer traffic. - Added unified `(BatchSize, OutH)` reduction slicing and 2D `(Cin x Cout)` panel tiling for `Conv2DKernelGrad` across AVX-512, AVX2, NEON, and SME, plus new `Diffusion-LargeBatch` and `SmallBatch-LargeSpatial` compliance benchmarks in `support/backendtest/bench_convgeneral.go`. --- ## Benchmark Results (`FlowMatching` U-Net Training, Median Train Step Duration) | Platform / Mode | Baseline | Optimized | Reduction | Speedup | | :--- | ---: | ---: | ---: | ---: | | **AMD Ryzen 9 9950X3D (`AVX-512`, 16C/32T)** | `925.8 ms` | **`563.0 ms`** | `-362.8 ms/step` (`-39.2%`) | **1.64×** (beats `xla:cpu` at `~615 ms`) | | **Apple M6 (`NEON + SME`, 12C)** | `596.6 ms` | **`400.3 – 400.8 ms`** | `-196.3 ms/step` (`-32.9%`) | **1.49×** | | **Apple M6 (`NEON-only`, `GOMLX_GO_SIMD_SME=0`)** | `850.2 ms` | **`565.2 ms`** | `-285.0 ms/step` (`-33.5%`) | **1.50×** |
simdmath, gobackend: Work around Go simd/archsimd.Load*Part SIGSEGV a… …t page boundaries (#51) ## Summary - Adds stack-staged `simdmath.Load*Part` helpers (`support/simdmath/loadpart.go`) marked with `// Deprecated: temporary work around until release of fix in https://github.com/golang/go/issues/81692`. - Replaces `simd.Load*Part` with `simdmath.Load*Part` across `dtypes/{bfloat16,float16}`, `internal/gobackend/ops`, `internal/gobackend/fusedops`, and `internal/gobackend/activations`. - Removes unused scalar `bfloat16`/`float16` adapters from `support/simdmath/simdmath.go` so `dtypes/bfloat16` and `dtypes/float16` can directly reuse `simdmath.LoadUint16sPart`. - Adds a page-boundary regression test (`TestLoadPartPageBoundary` in `support/simdmath/loadpart_test.go`). ## Background In Go 1.27.1–1.27.3 (`GOEXPERIMENT=simd` on `amd64`), `simd/archsimd.Load*Part` lowers to an unmasked full-vector load (`VMOVDQU` / `VMOVDQU64`) followed by a register-to-register mask rather than a fault-suppressing hardware masked load (`VPMASKMOV*` / `VMOVDQU* K`), causing `SIGSEGV` when a tail slice (`len(s) < vLen`) lies within 64 bytes of an unmapped page boundary (golang/go#81692).
gobackend: Optimize SDPA 2D parallelization, multi-core unary ops, an… …d nosimd activations (#50) ## Summary - **`FusedScaledDotProductAttention` (`internal/gobackend/fusedops/sdpa.go`, `avx2/sdpa.go`, `avx512/sdpa.go`)**: - Added 2D multi-core work partitioning across `(batchSize * numKVHeads, qBlocks)` so models with small `(batch * numKVHeads)` (e.g. `batch=2, numKVHeads=6`) saturate all CPU cores across query sequence blocks instead of capping out at `batch * numKVHeads` workers. - Reduced per-worker `scores` scratch allocation from `groupSize * seqLen * kvLen` to a single `kvLen` row (`L1`-resident). - Inverted the value accumulation loop in `sdpaGeneric` (`for kvIdx { w := scores[kvIdx]; if w == 0 { continue }; for d ... }`) for contiguous cache-line access and zero-weight skipping. - Unrolled the scalar Q·K dot product in `sdpaGeneric` with 4 independent accumulators (`dot0..dot3`), tracked `activeKVLen` for boolean masks, and used `fastmath.Exp32` for `float32` softmax. - Fixed `exp512` in `avx512/sdpa.go` by removing the `+0.5` bias before `RoundScaled(0)` (round-to-nearest-even) so masked `-Inf` positions evaluate to exact `0` and trigger the `w == 0` fast skip. - **Multi-Core Unary Ops (`internal/gobackend/ops/unaryops.go`)**: - Parallelized `Logistic`, `Exp`, `Expm1`, `Log`, `Log1p`, `Rsqrt`, `Sqrt`, `Sin`, `Cos`, `Tanh`, `Erf`, `Sign`, `Abs`, `LogicalNot`, `Ceil`, `Floor`, `Round`, and `IsFinite` using `ParallelizeChunks`. - **Scalar (`nosimd`) Activations (`internal/gobackend/ops/unaryops.go`, `internal/gobackend/activations/nosimd.go`)**: - Accelerated `float32` `Logistic`, `Sigmoid`, `Silu`, `Selu`, and `SwiGLU` kernels using `fastmath.Exp32`.
PreviousNext