Open
Add REINFORCE RL training support (first step toward REINFORCE, DPO, GRPO, PPO)#400
Conversation
Copilot created this pull request from a session on behalf of
vince62s
July 8, 2026 07:25
View session
There was a problem hiding this comment.
Pull request overview
Adds initial on-policy RL fine-tuning support to the training stack by introducing a REINFORCE loss, an RL-specific trainer that plugs into the existing Trainer._train_step() extension point, and small generation-utility enhancements to streamline reward scoring.
Changes:
- Introduces
RLLossCompute+REINFORCELossComputeand an RL loss factory (build_rl_loss_compute). - Adds
RLTrainerthat performs rollout generation + reward scoring + differentiable log-prob recomputation + policy-gradient backprop. - Extends generation utilities to return detokenized references from
batch["tgt"]and auto-wire them into scoring whentexts_refis not provided.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| recipes/rl/reinforce.yaml | Adds an example REINFORCE training config (RL-specific training: fields + sampling params). |
| recipes/rl/README.md | Documents RL recipe status and explains the REINFORCE workflow and key config fields. |
| eole/utils/rl_loss.py | Implements REINFORCE policy-gradient loss (baseline + optional KL penalty) and a factory for RL loss selection. |
| eole/utils/generate_utils.py | Adds optional references output and auto-uses them as texts_ref in generate_and_score(). |
| eole/trainer.py | Dispatches to an RL trainer builder when training.rl_algorithm is set and wires RL-specific components. |
| eole/trainer_rl.py | New RLTrainer that overrides _train_step() to run rollout-based RL updates. |
| eole/tests/test_rl_loss.py | Adds unit tests for REINFORCE loss behavior and RL config validation. |
| eole/config/training.py | Adds RL-related config fields and validates supported algorithms/required RL settings. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…n _build_sampled_tgt, reinforce.yaml comment
vince62s
marked this pull request as ready for review
July 17, 2026 11:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Following the merged "RL readiness refactoring" (#389), the request was to add RL fine-tuning support for REINFORCE and DPO. Implementing both simultaneously wasn't practical: REINFORCE is on-policy and can directly reuse the generation/reward infra already added in #389, while DPO needs an offline chosen/rejected preference-pair data pipeline that doesn't exist yet. This PR implements REINFORCE first, with the abstractions designed so DPO/GRPO/PPO can follow incrementally.
RL loss
eole/utils/rl_loss.py:RLLossComputebase class +REINFORCELossCompute, a score-function policy-gradient loss with a batch-mean baseline and an optional KL penalty against a frozen reference model.RL trainer
eole/trainer_rl.py:RLTrainer(Trainer)overriding only_train_step()— samples rollouts, scores them with a configured scorer, recomputes differentiable log-probs of the sampled tokens, and backprops the policy-gradient loss. Reuses the outerTrainer.train()loop (validation, checkpointing, reporting) unchanged.Generation utilities
eole/utils/generate_utils.py:generate_from_batch()now also returnsreferencesrebuilt frombatch['tgt']when present;generate_and_score()uses them automatically astexts_refwhen not explicitly passed.Configuration
eole/config/training.py: newrl_algorithm,rl_reward_metric,rl_baseline,rl_kl_coef,rl_reference_model,rl_num_rollouts,rl_gen_*fields.dpo/grpo/ppoare reserved algorithm names that raiseNotImplementedErrorfor now, so future support doesn't require breaking config changes.eole/trainer.py:build_trainer()dispatches toRLTrainerwhenrl_algorithm="reinforce".Tests & examples
eole/tests/test_rl_loss.py: unit tests for the REINFORCE loss (baseline behavior, padding handling, KL penalty, gradient flow) and RL config validation.recipes/rl/reinforce.yaml+recipes/rl/README.md: example config and a status table tracking REINFORCE/DPO/GRPO/PPO.Follow-up (not in this PR)
RLTrainer/rl_losswith a multi-rollout group-relative baseline (rl_num_rolloutsis reserved for this, currently unused by REINFORCE).