This is an example repo for CUDA MatMul implementation. The aim of this repo is to provide some insights in high-performance kernel design for CUDA beginners. Currently, I only provide some implementation examples in examples/matmul/this. Contributions for more kernels and other MatMul implementations are highly welcomed.

There is a detailed explanation about the different versions of MatMul kernels in examples/matmul/this.

• examples:

  • matmul: The MatMul implementations

    • this-sm90: The Hopper version Matmul
    • this-sm80: The MatMul implemented by this repo
    • cublas: Call CuBLAS for performance test
    • cutlass: Call CUTLASS for performance test
    • mlir-gen: The cuda code generated by MLIR
    • triton: Call Triton for performance test
    • tvm: Call Relay+CUTLASS/CuBLAS or TensorIR for performance test

  • atom: The usage of single intrinsic/instructions

  • reduction: Some reduction kernels for epilogue

The current version only achieves on average 70% performance of CuBLAS. I am still working on improving the performance.

Overall, the geometric mean speedup to Relay+CUTLASS is 1.73x, to TensorIR (1000 tuning trials using MetaSchedule per case) is 1.22x, to CuBLAS is 1.00x, to CUTLASS is 0.999x, to Triton is 1.07x. The 61 shapes are:

I also use MLIR to generate MatMul kernels. The generated ones are in examples/matmul/mlir-gen. The performance to handwritten ones (examples/matmul/this) is shown as belows. As MLIR generated ones only implement part of the optimizations used by handwritten ones, we call the MLIR generated ones partial and the handwritten ones full.

Overall, MLIR generated versions achieve 86% the performance of handwritten kernels.

I plan to implement kernels for other operators such as softmax in future.

There is a plan to use the CuTe interface of CUTLASS to implement high-performance kernels.