LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

TokenSpeed is a speed-of-light LLM inference engine.

iTerm2 is a terminal emulator for Mac OS X that does amazing things.

☄🌌️ The minimal, blazing-fast, and infinitely customizable prompt for any shell!

Public repository for Agent Skills

Fast and memory-efficient exact attention

PyTensor allows you to define, optimize, and efficiently evaluate mathematical expressions involving multi-dimensional arrays.

PyTorch bindings for CUTLASS grouped GEMM.

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

Virtual whiteboard for sketching hand-drawn like diagrams

Development repository for the Triton language and compiler

Parse Python docstrings in various flavors.

A Easy-to-understand TensorOp Matmul Tutorial

Remote vanilla PDB (over TCP sockets).

Fast, Flexible and Portable Structured Generation

Enforce the output format (JSON Schema, Regex etc) of a language model

A guidance language for controlling large language models.

Structured Outputs

A debugging and profiling tool that can trace and visualize python code execution

Material for gpu-mode lectures

Demonstration of various hardware effects on CUDA GPUs.

Download M3U8 live streams to the local disk

Thread-safe asyncio-aware queue for Python

Flamegraph generator for cProfile stats

Generate stubs for python modules

FlashInfer: Kernel Library for LLM Serving

Examples in the MLX framework

Universal LLM Deployment Engine with ML Compilation

MLX: An array framework for Apple silicon

A high-throughput and memory-efficient inference and serving engine for LLMs