Skip to content

x86 repacked matmul: add a Q5K tile #4029

Open

Description

@bhaskargurram-ai

On x86, repack_x86.rs::select() repacks only Q4K, Q6K and Q8_0, so Q5K weights take the generic k_quants::matmul path. aarch64 already packs Q5K (pack_to_q5kx8). In my #4003 measurements, Q5K prefill (2048x2048, m=128) ran at ~11.6k rows/s against ~31k rows/s for repacked Q4K on the same machine, and Q5_K_M models store most weights as Q5K.

Proposal: a TileQ5K mirroring TileQ4K (16 rows, unpacked scales/mins, nibbles plus the high bit merged into u8 0..31), reusing the existing dpbusd/maddubs loop with the same min correction, gated to m > 1 like Q6K so decode keeps vec_dot_q5k_q8k. Would you take a PR for this? @EricLBuehler

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions