Stateful API logic for agentic applications using vLLM

FlyDSL is the Python front‑end of the project: Flexible LaYout DSL.

From Minimal GEMM to Everything

A high-throughput and memory-efficient inference and serving engine for LLMs (Windows build & kernels)

amdgpu example code in hip/asm

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

A framework for efficient model inference with omni-modality models

A modern web interface for managing and interacting with vLLM servers (www.github.com/vllm-project/vllm). Supports both GPU and CPU modes, with special optimizations for macOS Apple Silicon and ent…

Tritonbench is a collection of PyTorch custom operators with example inputs to measure their performance.

Modular RDMA Interface

A "standard library" of Triton kernels.

No fortress, purely open ground. OpenManus is Coming.

Code for BLT research paper

QuickReduce is a performant all-reduce library designed for AMD ROCm that supports inline compression.

Submission for the SG Innovation Challenge

LLM KV cache compression made easy

Accessible large language models via k-bit quantization for PyTorch.

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

📰 Must-read papers and blogs on LLM based Long Context Modeling 🔥

Efficient LLM Inference over Long Sequences

Master programming by recreating your favorite technologies from scratch.

SGLang is a high-performance serving framework for large language models and multimodal models.

Open-source observability for your GenAI or LLM application, based on OpenTelemetry

LLM Serving Performance Evaluation Harness

Any model. Any hardware. Zero compromise. Built with @ziglang / @openxla / MLIR / @bazelbuild

A collection of LLM papers, blogs, and projects, with a focus on OpenAI o1 🍓 and reasoning techniques.

Efficient and easy multi-instance LLM serving

Artifact of OSDI '24 paper, ”Llumnix: Dynamic Scheduling for Large Language Model Serving“