You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
On x86, repack_x86.rs::select() repacks only Q4K, Q6K and Q8_0, so Q5K weights take the generic k_quants::matmul path. aarch64 already packs Q5K (pack_to_q5kx8). In my #4003 measurements, Q5K prefill (2048x2048, m=128) ran at ~11.6k rows/s against ~31k rows/s for repacked Q4K on the same machine, and Q5_K_M models store most weights as Q5K.
Proposal: a TileQ5K mirroring TileQ4K (16 rows, unpacked scales/mins, nibbles plus the high bit merged into u8 0..31), reusing the existing dpbusd/maddubs loop with the same min correction, gated to m > 1 like Q6K so decode keeps vec_dot_q5k_q8k. Would you take a PR for this? @EricLBuehler
Activity