A retargetable MLIR-based machine learning compiler and runtime toolkit.

A lightweight inference engine supporting speculative speculative decoding (SSD).

IREE plugin repository for the AMD AIE accelerator

Hexagon-MLIR is a compiler toolchain for compiling and executing AI kernels and models on Qualcomm Hexagon Neural Processing Units (NPUs).

A unified library of state-of-the-art model optimization techniques like quantization, pruning, distillation, speculative decoding, etc. It compresses deep learning models for downstream deployment…

Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3.

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.

Quantized Attention achieves speedup of 2-5x and 3-11x compared to FlashAttention and xformers, without lossing end-to-end metrics across language, image, and video models.

Tensors and Dynamic neural networks in Python with strong GPU acceleration

Neural Network Compression Framework for enhanced OpenVINO™ inference

SOTA low-bit LLM quantization (INT8/FP8/INT4/FP4/NF4) & sparsity; leading model compression techniques on TensorFlow, PyTorch, and ONNX Runtime

Accessible large language models via k-bit quantization for PyTorch.

A high-throughput and memory-efficient inference and serving engine for LLMs

AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:

[ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

Codebase for Aria - an Open Multimodal Native MoE

Unofficial implementation of Titans, SOTA memory for transformers, in Pytorch

Cosmos is a world model development platform that consists of world foundation models, tokenizers and video processing pipeline to accelerate the development of Physical AI at Robotics & AV labs. C…

OpenStereo: A Comprehensive Benchmark for Stereo Matching and Strong Baseline

A generative world for general-purpose robotics & embodied AI learning.

Code for BLT research paper

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficie…

Model Compression Toolbox for Large Language Models and Diffusion Models

[ICLR2024 spotlight] OmniQuant is a simple and powerful quantization technique for LLMs.