cff-version: 1.2.0

title: "miniVERL: Auditable single-GPU alignment and distillation runtime"

message: "If you use miniVERL in your work, please cite it as below."

type: software

version: 0.6.3

date-released: 2026-08-06

license: Apache-2.0

repository-code: "https://github.com/DaoyuanLi2816/mini-verl"

url: "https://github.com/DaoyuanLi2816/mini-verl"

abstract: >-

miniVERL is a single-GPU runtime for comparing alignment and distillation

methods under conditions strict enough that the comparison means something.

Its core is multi-turn, tool-aware on-policy distillation: a student language

model samples its own tool-using trajectories against deterministic local

environments, a teacher scores exactly the states the student visited, and

the student is updated with token-level distributional supervision on its own

generated tokens only. It implements exact full-vocabulary forward KL,

reverse KL and Jensen-Shannon divergences as well as a compressed top-k plus

tail coarse-graining, enforces per-token provenance so that tool output can

never become a training label, and stores teacher targets in a versioned,

checksummed, pickle-free cache with policy-version enforcement.

Around that core it provides supervised fine-tuning, recorded-provenance

preference optimization and knowledge distillation as directly comparable

arms; shared-backbone execution that switches student, teacher and reference

roles across adapters on one set of base weights to fit low-memory hardware;

padded update batching; deterministic replay; transactional checkpoints and

cross-process run locks; and mechanism studies such as RecoveryBench that are

published with their negative and mixed results intact. A fail-closed

artifact bridge exchanges configs, LoRA adapters, tokenizers and Parquet

datasets with one pinned verl profile, verifying what it can and refusing to

imply the rest: it does not launch distributed jobs and does not establish

algorithmic parity with PPO. miniVERL is designed for one personal CUDA GPU,

automatically selects bf16 or fp16, and requires neither Ray nor a cluster.

Published performance is measured on one RTX 4080; other GPU models use the

same code path but remain unmeasured.

authors:

- family-names: Li

given-names: Daoyuan

email: lidaoyuan2816@gmail.com

keywords:

- on-policy distillation

- knowledge distillation

- llm alignment

- preference optimization

- large language models

- tool use

- agents

- PEFT

- QLoRA

- consumer GPU

- single GPU

- post-training

- reproducibility

references:

- type: article

title: "Distilling the Knowledge in a Neural Network"

authors:

- family-names: Hinton

given-names: Geoffrey

- family-names: Vinyals

given-names: Oriol

- family-names: Dean

given-names: Jeff

year: 2015

url: "https://arxiv.org/abs/1503.02531"

notes: "Source of the high-temperature forward-KL correction; reverse-KL and JSD use it only as an explicit heuristic."

- type: article

title: >-

On-Policy Distillation of Language Models: Learning from Self-Generated

Mistakes

authors:

- family-names: Agarwal

given-names: Rishabh

year: 2024

url: "https://arxiv.org/abs/2306.13649"

notes: "Generalized JSD and training on student-sampled sequences."

- type: article

title: "On-Policy Context Distillation for Language Models"

authors:

- family-names: Ye

given-names: Tianzhu

- family-names: Dong

given-names: Li

- family-names: Wu

given-names: Xun

- family-names: Huang

given-names: Shaohan

- family-names: Wei

given-names: Furu

year: 2026

url: "https://arxiv.org/abs/2602.12275"

notes: "The privileged-context teacher mode follows this formulation."

- type: article

title: "Entropy-Aware On-Policy Distillation of Language Models"

authors:

- family-names: Jin

given-names: Woogyeol

- family-names: Min

given-names: Taywon

- family-names: Yang

given-names: Yongjin

- family-names: Wei

given-names: Dennis

- family-names: Zhou

given-names: Yi

- family-names: Kadhe

given-names: Swanand Ravindra

- family-names: Baracaldo

given-names: Nathalie

- family-names: Lee

given-names: Kimin

year: 2026

url: "https://arxiv.org/abs/2603.07079"

notes: >-

Motivates recording per-token teacher entropy. Entropy-aware divergence

mixing is a roadmap item and is not implemented in miniVERL.