forked from DaoyuanLi2816/mini-verl
-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathCITATION.cff
More file actions
101 lines (101 loc) · 3.57 KB
/
Copy pathCITATION.cff
File metadata and controls
101 lines (101 loc) · 3.57 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
cff-version: 1.2.0
title: "miniVERL: On-policy distillation for tool-using agents on one GPU"
message: "If you use miniVERL in your work, please cite it as below."
type: software
version: 0.6.1
date-released: 2026-08-03
license: Apache-2.0
repository-code: "https://github.com/DaoyuanLi2816/mini-verl"
url: "https://github.com/DaoyuanLi2816/mini-verl"
abstract: >-
miniVERL is a single-GPU laboratory for multi-turn, tool-aware on-policy
distillation. A student language model samples its own tool-using
trajectories against deterministic local environments; a teacher scores
exactly the states the student visited; and the student is updated with
token-level distributional supervision on its own generated tokens only.
It implements exact full-vocabulary forward KL, reverse KL and
Jensen-Shannon divergences as well as a compressed top-k plus tail
coarse-graining, enforces per-token provenance so that tool output can never
become a training label, and stores teacher targets in a versioned,
checksummed, pickle-free cache with policy-version enforcement. It is
designed for one personal CUDA GPU, automatically selects bf16 or fp16, and
requires neither Ray nor a cluster. Published performance is measured on one
RTX 4080; other GPU models use the same code path but remain unmeasured.
authors:
- family-names: Li
given-names: Daoyuan
email: lidaoyuan2816@gmail.com
keywords:
- on-policy distillation
- knowledge distillation
- large language models
- tool use
- agents
- QLoRA
- consumer GPU
- single GPU
- post-training
- reproducibility
references:
- type: article
title: "Distilling the Knowledge in a Neural Network"
authors:
- family-names: Hinton
given-names: Geoffrey
- family-names: Vinyals
given-names: Oriol
- family-names: Dean
given-names: Jeff
year: 2015
url: "https://arxiv.org/abs/1503.02531"
notes: "Source of the high-temperature forward-KL correction; reverse-KL and JSD use it only as an explicit heuristic."
- type: article
title: >-
On-Policy Distillation of Language Models: Learning from Self-Generated
Mistakes
authors:
- family-names: Agarwal
given-names: Rishabh
year: 2024
url: "https://arxiv.org/abs/2306.13649"
notes: "Generalized JSD and training on student-sampled sequences."
- type: article
title: "On-Policy Context Distillation for Language Models"
authors:
- family-names: Ye
given-names: Tianzhu
- family-names: Dong
given-names: Li
- family-names: Wu
given-names: Xun
- family-names: Huang
given-names: Shaohan
- family-names: Wei
given-names: Furu
year: 2026
url: "https://arxiv.org/abs/2602.12275"
notes: "The privileged-context teacher mode follows this formulation."
- type: article
title: "Entropy-Aware On-Policy Distillation of Language Models"
authors:
- family-names: Jin
given-names: Woogyeol
- family-names: Min
given-names: Taywon
- family-names: Yang
given-names: Yongjin
- family-names: Wei
given-names: Dennis
- family-names: Zhou
given-names: Yi
- family-names: Kadhe
given-names: Swanand Ravindra
- family-names: Baracaldo
given-names: Nathalie
- family-names: Lee
given-names: Kimin
year: 2026
url: "https://arxiv.org/abs/2603.07079"
notes: >-
Motivates recording per-token teacher entropy. Entropy-aware divergence
mixing is a roadmap item and is not implemented in miniVERL v0.6.1.