fp4

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

Model compression toolkit engineered for enhanced usability, comprehensiveness, and efficiency.

An innovative library for efficient LLM inference via low-bit quantization

From-scratch C++23/CUDA inference engine for the NVIDIA RTX 5090 (sm_120a). The best single-GPU backend for agentic AI: tool calling, long-context loops, reasoning and concurrent sub-agents. Decode beats llama.cpp b9976 by 42-48% on dense GGUF (measured 2026-07-12), at-or-ahead of vLLM on NVFP4. 100% written by Claude Code.

Hand-written NVFP4 W4A16 CUDA kernels and chain-MTP speculative serving — Qwen3.6-27B at up to 366 tok/s on four Tesla V100s, hardware with no FP4 support

An stress and benchmark utility for NVIDIA GPUs. Measures performance across various precisions (FP64, FP32, TF32, FP16, INT8) and monitors real-time vitals like power, temperature, and clock speeds.

Narrow precision floating point types

Out-of-tree vLLM plugin and open format spec for NVFP4-CB / FP8-CB product-codebook weights — 2-6 bit-per-weight LLM quantization served on native Blackwell tensor cores.

Base-13 FP4 limbs for Ozaki-scheme DGEMM emulation on FP4 tensor cores

Optimized vLLM setup for Gemma 4 31B NVFP4 with MTP on dual RTX PRO 6000 Blackwell using vllm and docker: native FP4 Tensor Cores, Multi-Token Prediction (96.5% acceptance rate), and prefix caching. Includes benchmark results and replication scripts.

LoRA fine-tune and serve NVFP4 models on one DGX Spark (GB10, 128 GB UMA): text backbones via generic-family onboarding, plus VLMs (vision tower, or LLM+tower jointly via --train-target both) validated end-to-end on Pixtral and Nemotron-Omni. Fused Triton dequant; runtime-LoRA and merge serving.

Python implementations for multi-precision quantization in computer vision and sensor fusion workloads, targeting the XR-NPE Mixed-Precision SIMD Neural Processing Engine. The code includes visual inertial odometry (VIO), object classification, and eye gaze extraction code in FP4, FP8, Posit4, Posit8, and BF16 formats.

The only deployed sparse FP4 GEMM on SM120: beats CUTLASS 80b on every shape, wins end-to-end request latency in 81 of 112 serving regimes vs dense NVFP4.

Three ComfyUI nodes that show the real runtime precision of MODEL, CLIP, and VAE objects.

Unlimited-OCR 的 OpenVINO 本地适配：显式 KV、R-SWA cache、12 层 sparse MoE decode、FP4/INT8 mixed experts

FP4 E2M1 Arithmetic Unit — Full RTL-to-GDSII tapeout on SKY130A PDK using OpenLane | 200 MHz timing closure | DRC/LVS clean | AI inference acceleration

Block-scaled FP8 / FP4 / INT4 tensor primitive with Triton scaled-matmul at FP32 parity on H100. NumPy / PyTorch / MLX / JAX backends.

llama.cpp super-fork: NVFP4 OMMA.SF.16864 tensor core path for consumer Blackwell (RTX 5070 Ti). NVFP4 KV cache lossless at 64k (KLD=0). MoE expert offloading. .den native format.