OCGQuant: Outlier-Companion Grouping for NVFP4 Quantization

This repository contains the PyTorch implementation of the EMNLP 2026 paper OCGQuant: Outlier-Companion Grouping for NVFP4 Quantization.

git clone --recurse-submodules https://github.com/Eshamont/OCGQuant.git
cd OCGQuant

Please make sure that CUDA 12.8 is available in your environment.

conda create -n ocgquant python=3.10 -y
conda activate ocgquant

sudo apt-get update
sudo apt-get install build-essential cmake python3-dev

conda install pybind11
pip install torch==2.9.0 --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt

Note:

• The CUDA kernels are currently configured for NVIDIA Blackwell SM120 GPUs.
• Non-Blackwell GPUs only support PyTorch fake-quantization experiments, but the real NVFP4 CUDA kernels and corresponding hardware acceleration are unavailable.

cd kernels
bash remake.sh
cd ..

Before evaluation, OCGQuant requires precomputed .pt files.

python reorder_indices.py --model /path/to/model --samples 128 --seqlen 2048 --dataset wikitext2 --method ocgquant

The generated files will be saved to ./saved/.

Run preprocessing before PPL or accuracy evaluation so the required files exist in ./saved/.

python main.py /path/to/model --dataset wikitext2 --method ocgquant --quant_type NVFP4 --eval_ppl

The example below evaluates common zero-shot tasks.

python main.py /path/to/model --dataset wikitext2 --method ocgquant --quant_type NVFP4 --tasks piqa,arc_challenge,boolq,hellaswag,lambada_openai,arc_easy --lm_eval_num_fewshot 0 --lm_eval_limit -1

This project builds on several excellent open-source efforts. We sincerely thank the community for their contributions:

• ARCQuant
• Atom
• QuaRot
• TensorRT-LLM
• FlashInfer
• CUTLASS
• lm-evaluation-harness

If you find this work useful in your research, please consider citing our paper:

@article{yao2026ocgquant,
  title={OCGQuant: Outlier-Companion Grouping for NVFP4 Quantization},
  author={Yao, Yishan and Li, Binjun and Yi, Hanling and Li, Pengyu and Liu, Xiaoqing and Yang, Zihan and Yu, Xiaotian and Yu, Zhiwen},
  journal={arXiv preprint arXiv:2609.00066},
  year={2026}
}