A high-throughput and memory-efficient inference and serving engine for LLMs

FB (Facebook) + GEMM (General Matrix-Matrix Multiplication) - https://code.fb.com/ml-applications/fbgemm/

Nano vLLM

Distributed RL System for LLM Reasoning

SGLang is a fast serving framework for large language models and vision language models.

Perplexity GPU Kernels

FlashInfer: Kernel Library for LLM Serving

Minimalistic 4D-parallelism distributed training framework for education purpose

📖A curated list of Awesome LLM/VLM Inference Papers with codes, such as FlashAttention, PagedAttention, Parallelism, etc. 🎉🎉

Tensors and Dynamic neural networks in Python with strong GPU acceleration

Fast and memory-efficient exact attention

Model components of the Llama Stack APIs

Pytorch domain library for recommendation systems

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper GPUs, to provide better performance with lower memory utilization in bot…

Hackable and optimized Transformers building blocks, supporting a composable construction.

PArametrized Recommendation and Ai Model benchmark is a repository for development of numerous uBenchmarks as well as end to end nets for evaluation of training and inference platforms.

CUDA Templates for Linear Algebra Subroutines

Pytorch domain library for recommendation systems

Effective Python: Second Edition — Source Code and Errata for the Book

Compiler for Neural Network hardware accelerators

PyTorch tutorials.

A natural language modeling framework based on PyTorch

Complete x86/x64 JIT and AOT Assembler for C++

Open deep learning compiler stack for cpu, gpu and specialized accelerators

BLISlab: A Sandbox for Optimizing GEMM