A high-throughput and memory-efficient inference and serving engine for LLMs

TROLL: Trust Region Optimization for Large Language models

verl: Volcano Engine Reinforcement Learning for LLMs

Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success

在verl上做reward的定制开发

Vocabulary Parallelism

Zero Bubble Pipeline Parallelism

Zero Bubble Pipeline Parallelism

Sequence-level 1F1B schedule for LLMs.

Sequence-level 1F1B schedule for LLMs.

PyTorch bindings for CUTLASS grouped GEMM.

Ongoing research training transformer language models at scale, including: BERT & GPT-2

[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

Best practice for training LLaMA models in Megatron-LM

Ongoing research training transformer language models at scale, including: BERT & GPT-2