Skip to content

Tags

Tags

v0.1.30

Toggle v0.1.30's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
gobackend: ARM64 NEON/SME half-precision & transformer optimizations (#…

…82)

## Summary

This PR optimizes transformer / embedding model execution on ARM64 (NEON
& Apple M-series SME) across both native half-precision (`Float16` /
`BFloat16`) and `Float32` execution paths:

1. **ARM64 NEON Half-Precision Slice Converters
(`internal/gobackend/neon`, `dtypes/{float16,bfloat16}`)**:
- Registers hardware NEON `FCVTL`/`FCVTL2` and `FCVTN`/`FCVTN2`
(`Float16` $\leftrightarrow$ `Float32`) and `SHLL`/`SHLL2` +
round-to-nearest-even (`BFloat16` $\leftrightarrow$ `Float32`) slice
converters via `gobackend.RegisterHalfSliceConverters`.
- Raises `ParallelHalfConvThreshold` and `convertDTypeMinParallelElems`
to `262144` elements and skips `ParallelizeChunks` when the scheduler is
already busy with concurrent workers.

2. **ARM64 NEON/SME `MultiRHS` MatMul & Fused Epilogues
(`internal/gobackend/dot/matmul/neon`)**:
- Implements `neonLargeMulti{Float32,BFloat16,Float16,Float64}` and
`smeLargeMulti{Float32,BFloat16,Float16,Float64}` and registers them via
`matmul.RegisterMultiRHSImplementation`, enabling shared LHS panel
packing across Q/K/V in `FusedAttentionQKVProjection` and cache-resident
tile epilogues (bias + activation) in `FusedDense`.
- Adds ARM64 NEON `ApplyEpilogueTile` bias + GELU/ApproxGELU kernels
(`internal/gobackend/activations/neon`).
- Optimizes partial 16/32-row strip packing in `neonPackLHS` using
`clear(...)` + 8-row NEON sub-strip assembly
(`neonPackLHSKernelRows8Strided*Asm`), reducing scalar remainder copies
to at most 7 rows.
- Software-pipelines 2 K-steps in
`smeLargeKernel{Float16,BFloat16}Asm_32x32` and
`neonSubTile8x8Strided{Float16,BFloat16}Asm` to hide
`LD1H`/`FCVT`/`FCVTL`/`SHLL` latency.

3. **Half-Precision `FusedLayerNorm` & Graph Fusion
(`internal/gobackend/fusedops`, `internal/gobackend/executable.go`)**:
- Adds native `Float16` and `BFloat16` support to `FusedLayerNorm`
(accumulating in `float32` per row in L1 cache and supporting in-place
buffer reuse).
- Extends `fuseSingleConsumerResidualAddIntoLayerNorm()` to fuse `Add(a,
b) -> ConvertDType(Half -> Float32) -> FusedLayerNorm(Float32) ->
ConvertDType(Float32 -> Half)` into a single `FusedLayerNorm` node with
an optional residual input, eliminating 3 intermediate ops and buffer
allocations per LayerNorm block.
- Adds ARM64 NEON assembly for `FusedLayerNorm`
(`internal/gobackend/fusedops/neon`).

4. **ARM64 NEON Binary Broadcast Kernels (`internal/gobackend/ops/neon`,
`internal/gobackend/ops/binaryops_simd.go`)**:
- Adds ARM64 NEON fast paths for `Float16`, `BFloat16`, and `Float32`
binary ops (`Add`, `Sub`, `Mul`, `Div`, `Max`, `Min`) across all
broadcast patterns (`BroadcastNone`, `BroadcastLeading`, `BroadcastMid`,
`BroadcastTrailing`, scalar).

5. **ARM64 NEON Fused SDPA & Softmax
(`internal/gobackend/dot/matmul/neon`,
`internal/gobackend/fusedops/neon`)**:
- Adds NEON vectorization for `FusedScaledDotProductAttention`
(`sdpaNEONBlockFloat32`, `sdpaScaleMaskSoftmaxRowNEON`,
`sdpaSoftmaxRowNEON`, `sdpaValueAccumBlockNEON`) and `FusedSoftmax`.

6. **SME Cluster Lock & Scheduler Contention Reduction
(`internal/gobackend/simdarch`, `internal/gobackend/scheduler`)**:
- Caches `CurrentSMECluster()` (`ReadTPIDR_EL0`) and spins on a relaxed
L1-cached `.Load() < limit` check in `AcquireSMEWithSpin` before
attempting atomic `.Add(1)`.
- Caches `runtime.GOMAXPROCS(0)` in `Scheduler` so
`UnscaledParallelism()` avoids global `runtime.sched.lock` contention on
every op.

## Benchmark Results (Apple M6, `All-MiniLM-L6-v2-Embedding-GGUF`,
`f16`, 6 embedders, 10K passages)

| Mode | Warmup (Before $\rightarrow$ After) | Benchmark Wall Time |
Throughput (Passages/s) | Throughput (Tokens/s) |
| :--- | :--- | :--- | :--- | :--- |
| **Native F16 (`-force_f32=false`)** | `1.20s` $\rightarrow$
**`663.2ms` (1.81x)** | `19.2s` $\rightarrow$ **`16.2s`** | `521/s`
$\rightarrow$ **`616/s` (+18.2%)** | `40.2K/s` $\rightarrow$ **`47.7K/s`
(+18.7%)** |
| **Forced F32 (`-force_f32=true`)** | `1.00s` $\rightarrow$ **`610.6ms`
(1.64x)** | `15.6s` $\rightarrow$ **`14.2s`** | `641/s` $\rightarrow$
**`705/s` (+10.0%)** | `49.5K/s` $\rightarrow$ **`54.6K/s` (+10.3%)** |

v0.1.29

Toggle v0.1.29's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
gobackend, shapeinference: support dynamic RNGBitGenerator and intege…

…r ConvGeneral validation (#77)

## Summary
- **`gobackend` (`RNGBitGenerator`)**: Resolve dynamic output dimensions
at execution time via `f.DynamicAxisValue`, enabling dynamic-shape
random number generation (e.g. ONNX `RandomNormalLike`).
- **`support/backendtest`**: Add `DynamicRNGBitGenerator` compliance
test verifying multiple dynamic batch sizes (including zero-sized
outputs) on a single compiled executable.
- **`shapeinference` (`ConvGeneral`)**: Validate that `input` and
`kernel` share a matching numeric, non-packed `DType` (supporting
`Int32` convolutions for ONNX `ConvInteger`).

v0.1.28

Toggle v0.1.28's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
gobackend/dot/matmul: fix AVX-512 multi-worker useAccum copy-back til…

…ing and AVX2 half-precision/GoSIMD kernels (#67)

## Summary

Fixes a multi-threaded AVX-512 `LargeMatMul` regression introduced in
#63 (`v0.1.26`), along with two latent AVX2 `LargeMatMul` kernel bugs
uncovered while adding multi-dtype regression tests.

### 1. AVX-512 Multi-Worker `useAccum` Copy-Back Tiling Mismatch
(`internal/gobackend/dot/matmul/avx512`)
- **Bug**: When #63 ported L2 row sub-chunking (`maxMPanelsPerSubChunk`)
from `avx2/large.go` to `avx512/large.go`, `Loop 3`
(`avx512LargeMatrixSlice*`) was updated to advance `lhsPanelEnd` in
relative `lhsPanelRowIdx + Mc` increments when `len(cachedLHSPanels) ==
0`, while the `if useAccum` L2 accumulator copy-back loop still used
global `Mc`-aligned grid boundaries (`(rowPanelIdx + 1) * Mc`).
- **Impact**: Whenever multi-worker row splitting (`GOMAXPROCS >= 2`)
produced a worker slice starting at `rowStart % Mc != 0` with unaligned
tail rows `rowEnd % Mr != 0` and `numKPanels > 1` (e.g.,
`mistralai/Timestral-123M` `QuantileHead` `[36, 2560] @ [2560, 672]` at
`rowStart = 24, rowEnd = 36` on `H=576`, or `wO` `[810, 640] @ [640,
640]` at `rowStart = 712, rowEnd = 810` on `H=864`), `useAccum` either
skipped copy-back (`24..32` looked `Mr`-aligned) or read from the wrong
`accumBuffer[mIdx * panelSize]` slot, leaving uninitialized
pooled-buffer values in output rows from `rowStart` onward (forecast
steps `192..575`).
- **Fix**: Updated the `if useAccum` copy-back loop in `avx512/large.go`
(and regenerated `gen_large_{bf16,f16,f64}.go`) to compute `lhsPanelEnd`
identically to `Loop 3` and `avx2/large.go`, and ensured
`params.LHSL1KernelRows = 4` is set whenever `!AVX512UseAsm`.

### 2. AVX2 Go-SIMD Direct-Output BCE Check & Half-Precision ASM `Y13`
Clobber (`internal/gobackend/dot/matmul/avx2`)
- **Go-SIMD BCE bounds check (`avx2/large.go`, `avx512/large.go`)**:
Fixed `avx2LargeKernel*` checking
`packedOutput[lhsActiveRows*rhsPanelCols-1]`, which panicked with `index
out of range` under `GOMLX_GO_AVX2_ASM=false` when `canDirectOutput ==
true` wrote to a non-zero column tile with `rhsActiveCols <
rhsCrossSize`. Aligned both `avx2LargeKernel*` and `avx512LargeKernel*`
to check `(maxRow-1)*rhsPanelCols + maxCol - 1`.
- **AVX2 `BFloat16` / `Float16` ASM
(`large_amd64_{bfloat16,float16}.s`)**: Fixed `VPERMILPS $0xFF, X10,
X13` executing two instructions before Row 3's `VFMADD231PS Y8, Y13, Y6`
/ `VFMADD231PS Y9, Y13, Y7` in `k_pair_loop` and `k_last_pair`, which
clobbered `Y13` when `K >= 2`.

## Testing
- Added `TestAVX512LargeParallelUnalignedPanels`
(`internal/gobackend/dot/matmul/avx512/internal_test.go`) covering
`workers = 2, 3, 4, 8`, pre-poisoned backend buffer pools, `[36, 2560] @
[2560, 672]`, `[810, 640] @ [640, 640]`, batched/2D-unaligned shapes,
`Uncached` / `CachedRHS` / `CachedLHS` / `MultiRHS`, ASM + Go SIMD, and
`Float32` / `BFloat16` / `Float16` / `Float64`.
- Added `TestAVX2LargeParallelAndGoSIMDBounds`
(`internal/gobackend/dot/matmul/avx2/internal_test.go`) covering
multi-column-panel direct-output slices and `Float32` / `BFloat16` /
`Float16` / `Float64` under ASM and Go SIMD.
- Verified `GOEXPERIMENT=simd go test -race ./...` across `compute` and
end-to-end `mistralai/Timestral-123M` vs PyTorch across `GOMAXPROCS = 1,
2, 3, 4, 8, 32` (`Max Abs Diff = 1.5e-05` on all thread counts).

v0.1.27

Toggle v0.1.27's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Document, validate, and test negative Start/End margins in Pad and Dy…

…namicPad (#64)

In support of gomlx/onnx-gomlx#104.

### Summary
- **Documentation**: Document in `PadAxis`, `StandardOps.Pad`,
`DynamicPadAxis`, and `DynamicOps.DynamicPad` (`ops.go`,
`ops_dynamic.go`) that `Start`/`End` (and `StartValue`/`EndValue`)
accept negative values to truncate (crop) elements from the start or end
of an axis (applied after interior padding), while `Interior` (and
`InteriorValue`) must be non-negative (`>= 0`).
- **Shape Inference**:
- Update `shapeinference.Pad` and `shapeinference.DynamicPad` with full
docstrings, examples, operand dimension validation, and clearing static
axis names when static padding modifies the dimension size without a new
`TargetAxisName`.
- Add unit tests and testable examples (`TestPad`, `ExamplePad`,
`TestDynamicPad`, `ExampleDynamicPad`).
- **Compliance Tests**:
- Expand `Pad` and `DynamicPad` compliance tests in
`support/backendtest/special.go` to cover negative `Start`/`End`,
combined interior and negative edge padding, cropping all elements, 2D
negative margins, and partial `axesConfig`.

v0.1.26

Toggle v0.1.26's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
gobackend: ARM64 SME/NEON & AMD64 AVX2/AVX-512 optimizations for Time…

…stral, SDPA, SwiGLU, binary broadcasts, and bufferPool (#63)

## Summary

This PR optimizes the pure-Go backend (`internal/gobackend`) across
**ARM64 (NEON + SME)**, **AMD64 AVX2**, and **AMD64 AVX-512**, covering
both `DotGeneral` (`matmul`) scheduling/topology and non-MatMul
Transformer/Time-Series operations (`FusedScaledDotProductAttention`,
`FusedActivation` / `SwiGLU`, binary broadcasts, reductions, `Iota`,
`Where`, and backend buffer pooling).

Full profiling analysis and step-by-step benchmark logs are documented
in:
- `docs/OptimizeTimestral.md`
- `internal/gobackend/simdarch/Neon.md`
- `docs/GoBackendScheduler.md`

---

## Key Changes

### 1. Centralized `simdarch`, 1-Cycle `TPIDR_EL0` 2-L2 / 2-SME
Topology, 5-Tier Hybrid Scheduling & Two-Regime Scheduler
- **`internal/gobackend/simdarch`**: Centralized SIMD feature flags
(`IsNEONAllowed`, `IsSMEAllowed`, `IsAVX2Allowed`, `IsAVX512Allowed`,
`NEONUseAsm`, `AVX2UseAsm`, `AVX512UseAsm`) and 1-cycle `MRS TPIDR_EL0,
R0` SME cluster detection/locking (`AcquireSME`, `AcquireSMEWithLimit`,
`AcquireSMEWithSpin`, `ReleaseSME`, `CurrentSMECluster`).
- **5-Tier Heterogeneous Worker Model & Fused In-SME Small/Medium GEMM
(`internal/gobackend/dot/matmul/neon`)**:
- 3-zone tapered 2D work partitioning (`smeAppendHybridWorkItems`:
coarse $64 \times 128$, medium $32 \times 64$, fine $32 \times 32$
tiles) and 5-tier dual-ended claiming (`claimItem`).
- Split multi-worker execution into `//go:noinline func
neonLargeParallelWithParams*` + `neonScratchPool`, added hardware $32
\times 32$ `ZA0..ZA3` transpose (`smePackLHS32Float32Asm`), and fused
single-`SMSTART` strip-transpose + direct unpacked strided RHS
(`smeDirectNonTransposedFloat32Asm_32x32`), accelerating $64^3$–$256^3$
GEMMs by **1.92×–6.77×** and reaching **2,482–2,543 GFlops/s** on large
GEMMs (**1.05×–1.14× faster** than Apple `Accelerate` `cblas_sgemm`).
- **Two-Regime Inter-Call / Inter-Op Scheduling
(`internal/gobackend/scheduler`)**:
- Added `Scheduler.AcquireWholeChip()` / `ReleaseWholeChip()` (FIFO
1-slot channel gate for large ops $> 16\text{M}$ MACs vs. concurrent
per-cluster execution for small/medium ops $\le 16\text{M}$ MACs),
separated `interOpQueue` workers from `intraOpQueue` workers, and made
`RHSConstantData` cached panel count invariant across `maxWorkers`
transitions.

### 2. Multi-Row `FusedScaledDotProductAttention` (`SDPA`) Across ARM64
SME/NEON, AVX2, and AVX-512
- **ARM64 SME + 4-Row NEON
(`internal/gobackend/dot/matmul/neon/sdpa_arm64.go`)**:
- Unlocked `sdpaSMEBlockFloat32` ($4..32$ query rows) and
`sdpaNEONBlockFloat32` ($1..4$ query rows) with `booleanMask` and
`causal` masking, lazy once-per-KV-head `ZA` $K$ transpose
(`smePackLHS32Float32Asm`), fused `ZA` probability transpose + direct
strided $V$ `FMOPA` (`smeDirectStridedFloat32Asm_32x32`), per-phase SME
lock release during parallel NEON softmax, and trailing masked key
pruning (`sdpaActiveKVLenForBlock`) (**3.02×–3.26× faster** SDPA on
Apple M6).
- **AMD64 4-Row AVX2 & 4-Row AVX-512 Assembly
(`internal/gobackend/fusedops/{sdpa.go,avx2,avx512}`)**:
- Added 4-query-row register-blocked $Q K^T$ and $P V$ assembly kernels
(`sdpaQKT4RowsAVX2Asm`, `sdpaValueAccum4RowsAVX2Asm`,
`sdpaDot4RowsFloat32AVX512`, `sdpaWeightedAccum4RowsFloat32AVX512`) with
`headDim == 64` in-register accumulators, vectorized boolean mask
blending inside fused Estrin softmax, `isAllTrueBoolMask` fast path,
causal active-KV pruning, and dynamic `qBlockSize` targeting `4 *
numWorkers` tasks (**11.74× faster** SDPA on AVX2; **2.33× faster** SDPA
on AVX-512).

### 3. Handwritten `SwiGLU` (`FusedActivation`), `Sigmoid`, and `Tanh`
Assembly
- Split `SwiGLU` into dedicated `swiglu.go` and `swiglu*.s` files in
`internal/gobackend/activations/{neon,avx2,avx512}/`:
- **ARM64 NEON (`swigluFloat32NEONAsm`)**: 4× unrolled (16 floats/iter)
with hoisted `V19`–`V31` constants and overlapped `FDIV` +
`FRECPE`/`FRECPS` Newton-Raphson fast reciprocal (**4.3×–4.5× faster**).
- **AMD64 AVX2 (`swigluFloat32AVX2Asm`)**: 2× YMM (16 floats/iter) with
overlapped `VDIVPS` + `VRCPPS` (**4.51× faster**).
- **AMD64 AVX-512 (`swigluFloat32AVX512Asm`, `sigmoidFloat32AVX512Asm`,
`tanhFloat32AVX512Asm`)**: 2× ZMM (32 floats/iter) with hoisted
`Z16`–`Z29` constants and overlapped `VDIVPS` + `VRCP14PS` (**5.19×
faster**).

### 4. Binary Mid/Leading/Trailing Assembly, 3-Segment Broadcasts,
`Iota`, `Where`, and AVX-512 MatMul
- **Zero-Spill Binary Mid/Leading/Contiguous Assembly
(`internal/gobackend/ops/{avx2,avx512}/binary_mid_amd64.s`)**:
Eliminates Go 1.27 `simd` slice-header stack spills on `amd64` with
register-cached `C == 64` (and `C == 128` on AVX-512) fast paths; 4×
unrolled AVX-512 `binary_trailing_amd64.s`; parallelized
`tryBinaryTrailingArch` and `ReduceTrailing` across outer rows `A`.
- **3-Segment Broadcasts & `Iota` (`internal/gobackend/broadcast.go`,
`internal/gobackend/ops/iota.go`)**: Added `BroadcastRowColMatch`,
`BroadcastColRowMatch`, `BroadcastMatchRowCol`, and
`BroadcastMatchColRow` (`[A, 1, C]` op `[1, B, C]` and `[A, B, 1]` op
`[A, 1, C]`), and optimized `Iota` with exponential doubling `copy()`.
- **AVX-512 MatMul & Per-Arch `Where` Subpackages
(`internal/gobackend/dot/matmul/avx512`,
`internal/gobackend/ops/{avx2,avx512,neon}/where*`)**: Ported L2 row
sub-chunking and cooperative `sharedRHSPanels` packing to
`matmul/avx512`, added
`avx512PackLHSKernelRows8{Float64,Float16,BFloat16}Asm` and
`avx512SmallTransposedTile4x1*Asm`, added 512-bit AVX-512 `Where`
assembly, and split `Where` into `ops/{avx2,avx512,neon}/`.

### 5. Lock-Free Atomic Treiber Stack `bufferPool` & Zero-Allocation
`GetBuffer`
- Replaced `sync.Pool` and `sync.Map` in `Backend.bufferPools`
(`internal/gobackend/buffers.go`) with a direct-indexed
`[128]atomic.Pointer[bufferPool]` array of lock-free LIFO Treiber stacks
using a 64-bit ABA-tagged atomic head (`(tag << 32) | slotIdx`),
retained `Buffer.Flat` slice headers across reuse, and eliminated
variadic slice heap escape in `DType.SizeForDimensions` (`0 B/op, 0
allocs/op`, `13.85 ns/op`).

---

## Benchmark Summary (`mistralai/Timestral-28M` & `Timestral-123M`)

| Platform / Architecture | Model | Before (`go`) | **After (`go`)** |
`xla:cpu` Reference | Speedup vs Baseline | Speedup vs `xla:cpu` |
| :--- | :--- | ---: | ---: | ---: | ---: | ---: |
| **Apple M6 (ARM64 NEON + SME)** | `Timestral-28M` | 58.0 ms (`82.7 ms`
NEON) | **39.1 ms** | — | **`1.48×` (`2.12×` vs NEON)** | — |
| **Apple M6 (ARM64 NEON + SME)** | `Timestral-123M` | 179.4 ms (`293.4
ms` NEON) | **120.0 ms** | — | **`1.50×` (`2.45×` vs NEON)** | — |
| **Intel Core i9-12900K (AMD64 AVX2)** | `Timestral-28M` | 134.4 ms |
**60.3 ms** | 73.7 ms | **`2.23×`** | **`1.22× faster`** |
| **AMD Ryzen 9 9950X3D (AMD64 AVX-512)** | `Timestral-28M` | 40.5 ms
(`47.8 ms` AVX2) | **35.8 ms** | 51.8 ms | **`1.13×` (`1.34×` vs AVX2)**
| **`1.45× faster` (`1.87×` less CPU)** |
| **AMD Ryzen 9 9950X3D (AMD64 AVX-512)** | `Timestral-123M` | 155.6 ms
(AVX2) | **124.8 ms** | 163.9 ms | **`1.25×` vs AVX2** | **`1.31×
faster` (`1.35×` less CPU)** |

v0.1.25

Toggle v0.1.25's commit message
Updated missing entries in CHANGELOG.md.

v0.1.24

Toggle v0.1.24's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
gobackend: Diffusion / FlowMatching U-Net training optimizations (AMD…

…64 AVX2/AVX-512 & ARM64 NEON/SME) (#55)

## Summary

This PR implements 6 end-to-end performance optimizations in `gobackend`
identified by profiling the GoMLX `FlowMatching` U-Net diffusion
training workload (`batch=32`, $64 \times 64 \times 3$ Oxford Flowers
102, 4-level U-Net with Sinusoidal Embedding, Residual Blocks,
`swish`/`silu` activations, and spatial `LayerNorm(axes=[1, 2])`),
across both **AMD64 (`AVX2` / `AVX-512`)** and **ARM64 (`NEON` /
`NEON+SME`)**.

Detailed profiling and step-by-step benchmark logs are documented in
[`docs/DiffusionOptimization.md`](./docs/DiffusionOptimization.md).

### Key Optimizations

1. **`ReduceMiddle` SIMD Dispatch
(`internal/gobackend/ops/reduce_simd.go`, `avx512/`, `neon/`)**:
- Enabled `ReduceMiddle` (`[A, B, C] -> [A, 1, C]`, used in spatial
LayerNorm backward passes) in `canExecuteReduceSIMD` and dispatched
`ReduceSum`, `ReduceMax`, `ReduceMin`, and `ReduceProduct` via the
`ReduceLeading` SIMD and architecture-specific assembly kernels
(`reduceLeadingSumFloat32AVX512` and 8-accumulator
`reduceLeadingSumFloat32NEON`), parallelized across `A`.
2. **`FusedDenseVJP` Parallel Cache-Tiled `transpose2D` & Buffer Pooling
(`internal/gobackend/fusedops/dense/dense_vjp.go`)**:
- Pooled temporary buffers (`dZ`, `xT`, `dZT`, `tmpF32_dX`, `tmpF32_dW`)
via `backend.GetBuffer` / `backend.PutBuffer`, cutting total heap
allocations per step by **51.4%** (`592 MB/step` $\to$ `288 MB/step`)
and preventing mid-step GC cycles from evicting `Backend.bufferPools`.
- Parallelized `transpose2D` across
`backend.Scheduler.ParallelizeChunks` using $32 \times 32$ cache-tiled
blocks and optimized `reduceBiasFloat`.
3. **Out-of-Place Forward Activations via `RegisterKernel`
(`internal/gobackend/activations/`)**:
- Converted AVX2 and AVX-512 `float32` forward activation kernels
(`Silu`, `Relu`, `Sigmoid`, `HardSigmoid`, `LeakyRelu`, `Selu`,
`HardSwish`, `Tanh`) from in-place `(inout []float32)`
(`activations.Register`, which performed a redundant `copy(out, in)`
when `x` was retained for VJP) to `(in, out []float32)` via
`activations.RegisterKernel`.
4. **3-Segment Binary Broadcast (`BroadcastOuterInnerRHS/LHS`) &
Register-Pinned `BroadcastMid` (`internal/gobackend/broadcast.go`,
`ops/binaryops_simd.go`)**:
- Added `BroadcastOuterInnerRHS` and `BroadcastOuterInnerLHS` (`[A, B,
C]` op `[1, B, 1]`) so spatial LayerNorm `gamma`/`beta` scaling (`[B, H,
W, C] * [1, H, W, 1]`) uses vectorized `vs`/`sv` slices instead of
falling back to scalar `ZipIterator`.
- Updated `BroadcastMidRHS` / `BroadcastMidLHS` to use register-pinned
`v2v1`/`v1v2` loops across rows even when `B*C > 4096`, with unrolled `C
= 64, 96, 128` fast paths.
5. **Middle-Axis `FusedLayerNorm` SIMD
(`internal/gobackend/fusedops/layernorm_simd.go`)**:
- Added `simdLayerNormMiddleAxesFloat32` to vectorize non-trailing
`FusedLayerNorm` (e.g. `axes=[1, 2]` on `[B, H, W, C]`, where `dDim == 1
&& cDim > 1`) across the contiguous channel dimension `cDim`.
6. **`ConvGeneral` Direct Zero-`im2col` 2D Microkernels & `(BatchSize,
OutH)` `VJPKernel` Slicing (`internal/gobackend/ops/convgeneral_*.go`,
`avx2/`, `avx512/`, `neon/`)**:
- Added direct zero-`im2col` 2D forward/VJP-input assembly microkernels
(`ConvAVX512Direct2DKernel8x32Float32Asm`,
`ConvAVX512Direct2DKernel4x32Float32Asm`,
`ConvAVX2Direct2DKernel4x16Float32Asm`,
`ConvNEONDirect2DKernel4x16Float32Asm`) and direct-from-input SME LHS
packing (`ConvPackLHS8StridedFloat32Asm`) + fast-cluster dual-worker
pipelining (`SMEAcquireClusterWithLimit(2, 1)`), eliminating >95% of
`ExtractPatches2DFloat32` `im2col` buffer traffic.
- Added unified `(BatchSize, OutH)` reduction slicing and 2D `(Cin x
Cout)` panel tiling for `Conv2DKernelGrad` across AVX-512, AVX2, NEON,
and SME, plus new `Diffusion-LargeBatch` and `SmallBatch-LargeSpatial`
compliance benchmarks in `support/backendtest/bench_convgeneral.go`.

---

## Benchmark Results (`FlowMatching` U-Net Training, Median Train Step
Duration)

| Platform / Mode | Baseline | Optimized | Reduction | Speedup |
| :--- | ---: | ---: | ---: | ---: |
| **AMD Ryzen 9 9950X3D (`AVX-512`, 16C/32T)** | `925.8 ms` | **`563.0
ms`** | `-362.8 ms/step` (`-39.2%`) | **1.64×** (beats `xla:cpu` at
`~615 ms`) |
| **Apple M6 (`NEON + SME`, 12C)** | `596.6 ms` | **`400.3 – 400.8 ms`**
| `-196.3 ms/step` (`-32.9%`) | **1.49×** |
| **Apple M6 (`NEON-only`, `GOMLX_GO_SIMD_SME=0`)** | `850.2 ms` |
**`565.2 ms`** | `-285.0 ms/step` (`-33.5%`) | **1.50×** |

v0.1.23

Toggle v0.1.23's commit message
UPdated README.md.

v0.1.22

Toggle v0.1.22's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
simdmath, gobackend: Work around Go simd/archsimd.Load*Part SIGSEGV a…

…t page boundaries (#51)

## Summary

- Adds stack-staged `simdmath.Load*Part` helpers
(`support/simdmath/loadpart.go`) marked with `// Deprecated: temporary
work around until release of fix in
https://github.com/golang/go/issues/81692`.
- Replaces `simd.Load*Part` with `simdmath.Load*Part` across
`dtypes/{bfloat16,float16}`, `internal/gobackend/ops`,
`internal/gobackend/fusedops`, and `internal/gobackend/activations`.
- Removes unused scalar `bfloat16`/`float16` adapters from
`support/simdmath/simdmath.go` so `dtypes/bfloat16` and `dtypes/float16`
can directly reuse `simdmath.LoadUint16sPart`.
- Adds a page-boundary regression test (`TestLoadPartPageBoundary` in
`support/simdmath/loadpart_test.go`).

## Background

In Go 1.27.1–1.27.3 (`GOEXPERIMENT=simd` on `amd64`),
`simd/archsimd.Load*Part` lowers to an unmasked full-vector load
(`VMOVDQU` / `VMOVDQU64`) followed by a register-to-register mask rather
than a fault-suppressing hardware masked load (`VPMASKMOV*` / `VMOVDQU*
K`), causing `SIGSEGV` when a tail slice (`len(s) < vLen`) lies within
64 bytes of an unmapped page boundary
(golang/go#81692).

v0.1.21

Toggle v0.1.21's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
gobackend: Optimize SDPA 2D parallelization, multi-core unary ops, an…

…d nosimd activations (#50)

## Summary

- **`FusedScaledDotProductAttention`
(`internal/gobackend/fusedops/sdpa.go`, `avx2/sdpa.go`,
`avx512/sdpa.go`)**:
- Added 2D multi-core work partitioning across `(batchSize * numKVHeads,
qBlocks)` so models with small `(batch * numKVHeads)` (e.g. `batch=2,
numKVHeads=6`) saturate all CPU cores across query sequence blocks
instead of capping out at `batch * numKVHeads` workers.
- Reduced per-worker `scores` scratch allocation from `groupSize *
seqLen * kvLen` to a single `kvLen` row (`L1`-resident).
- Inverted the value accumulation loop in `sdpaGeneric` (`for kvIdx { w
:= scores[kvIdx]; if w == 0 { continue }; for d ... }`) for contiguous
cache-line access and zero-weight skipping.
- Unrolled the scalar Q·K dot product in `sdpaGeneric` with 4
independent accumulators (`dot0..dot3`), tracked `activeKVLen` for
boolean masks, and used `fastmath.Exp32` for `float32` softmax.
- Fixed `exp512` in `avx512/sdpa.go` by removing the `+0.5` bias before
`RoundScaled(0)` (round-to-nearest-even) so masked `-Inf` positions
evaluate to exact `0` and trigger the `w == 0` fast skip.
- **Multi-Core Unary Ops (`internal/gobackend/ops/unaryops.go`)**:
- Parallelized `Logistic`, `Exp`, `Expm1`, `Log`, `Log1p`, `Rsqrt`,
`Sqrt`, `Sin`, `Cos`, `Tanh`, `Erf`, `Sign`, `Abs`, `LogicalNot`,
`Ceil`, `Floor`, `Round`, and `IsFinite` using `ParallelizeChunks`.
- **Scalar (`nosimd`) Activations (`internal/gobackend/ops/unaryops.go`,
`internal/gobackend/activations/nosimd.go`)**:
- Accelerated `float32` `Logistic`, `Sigmoid`, `Silu`, `Selu`, and
`SwiGLU` kernels using `fastmath.Exp32`.