trainer

verl Main Entrypoints

sft_trainer.py: SFT trainer based on model engine, support various backends: fsdp, megatron, veomni, torchtitan. Launched by torchrun and run in multi-controller mode.
[EXPERIMENTAL] sft_trainer_ray.py: SFT trainer based on model engine with single-controller mode. Launched by ray with a driver process coordinating multiple worker processes.

trainer	description	sync/async	trainer/rollout	partial rollout
main_ppo.py	rollout until a batch is completed, then train	synchronous	colocated	No
TBD	kimi-1.5 style trainer: streaming rollout with capped length partial rollout	asynchronous	colocated	Yes
TBD	Areal style trainer: fully decoupled trainer and rollout with staleness control	asynchronous	disaggregated	Yes

main_generation_server.py: Launch standalone servers and generate responses for a specified prompt dataset.
main_eval.py: Evaluate the performance of generated responses with reward function on a specified prompt dataset.