MinRL

Learn reinforcement learning by reading the learning loop.

Small implementations of tabular and deep RL, with the mathematics kept close to the code. Start with a grid, learn a value function, then follow the same ideas into DQN and PPO. Each agent owns its training loop and update rule. No trainer framework or configuration file is needed to run the lessons; experiment tracking is an optional next step.

Q-learning on a 5×5 grid, seed 0, 1,500 episodes. Squares mark the start; the circle marks the final state. Traps terminate the episode—they are not walls. Reproduce this figure.

git clone https://github.com/10-OASIS-01/minrl.git
cd minrl
python3 -m venv .venv                  # Python 3.10 or newer
source .venv/bin/activate              # Windows: .venv\Scripts\activate
python -m pip install -e .
python -m examples.basic_navigation
python -m examples.q_learning_example

Or use the locked development environment: uv sync --python 3.11 --all-extras --frozen.

from minrl import GridWorld, QLearningAgent

env = GridWorld(size=3)
agent = QLearningAgent(env, seed=0)
rewards, lengths = agent.train(n_episodes=1000, max_steps=100)
policy = agent.get_optimal_policy()

The learning guide connects each equation to its implementation, explains the training/update flow, and includes small exercises with expected answers.

Neural agents accept a discrete observation (one-hot encoded) or a one-dimensional Box observation. They also accept a GridWorld directly through a thin Gymnasium adapter. Boundary actions are masked consistently during selection and learning. True terminations stop value bootstrapping; time-limit truncations do not.

Advanced options are kept out of the first lessons: prioritized replay, soft target updates, learning-rate decay, and evaluation-based early stopping are opt-in. See the experiment guide for their defaults and limitations.

python -m pip install -e '.[experiments]'
minrl train --config configs/cartpole_ppo.yaml --output runs/ppo-seed0
minrl evaluate --run runs/ppo-seed0 --episodes 100
minrl plot --run runs/ppo-seed0
tensorboard --logdir runs

# DQN, DQN + PER, PPO × five seeds × 100,000 environment steps
minrl benchmark --config configs/benchmark.yaml --output runs/cartpole

Every run saves its effective configuration, dependency versions, training CSV, evaluation JSON, TensorBoard events, and best/final checkpoints. Output folders must be new. Plotting reads saved results and exports SVG plus 300 DPI PNG without retraining.

CartPole-v1 · 100,000 environment steps per run · seeds 0–4 · CPU. Each final checkpoint is evaluated on 100 independent-seed episodes. Bands and ± values show one population standard deviation across five seed means, not confidence intervals.

All 15 runs completed; no seeds were dropped or tuned after inspection. PER did not improve the final mean in this small experiment, and intermediate scores were often non-monotonic. These are teaching results on one environment, not a general ranking of algorithms. See the full protocol and seed-level results.

src/minrl/
  environment/grid_world.py   # deterministic dynamics + thin Gymnasium adapter
  agents/                    # one algorithm per file; training and math together
    common.py                # encoding, masks, small neural-network helpers
    replay.py                # uniform replay and the optional PER extension
  utils/visualization.py     # plots, independent of learning
  experiments.py             # optional logging, evaluation and command-line runs
examples/                    # short, runnable lessons
tests/                       # hand-computed targets and end-to-end checks

Begin with train() and then read update() / train_step() in the same file. common.py does not implement any learning rule. experiments.py is not required reading until you want to compare multiple runs.

• Learning guide and exercises
• DQN: replay and target values
• Actor-critic: one-step advantages
• PPO: rollout, GAE and clipped updates
• Experiments and checkpoint limitations
• Migration from 0.1
• What was fixed

python -m pip install -e '.[dev,experiments]'
MPLBACKEND=Agg pytest --cov=minrl --cov-report=term-missing
ruff check src tests examples
python -m build

Prefer an explicit equation and a readable loop over an abstraction. Contributions should include a small correctness test, an example when useful, and documentation that describes the actual implementation. The CI workflow checks Python 3.10–3.12; local validation is performed on Python 3.11. Coverage numbers in reports are measured, not a promise of correctness.

This is a teaching library, not a general-purpose training framework. It currently uses single-environment CPU training, small MLPs, and discrete actions. It does not provide image policies, continuous control, vectorized collectors, exact checkpoint resumption, dynamic obstacles, multi-agent RL or curriculum learning. These are future directions, not implemented features. Monte Carlo estimates can be noisy and depend on visitation; excessively truncated episodes are reported and excluded, which can introduce selection bias.

• Mnih et al., Playing Atari with Deep Reinforcement Learning.
• Schaul et al., Prioritized Experience Replay.
• Schulman et al., Proximal Policy Optimization Algorithms.
• Gymnasium: handling time limits.
• Thanks to Professor Shiyu Zhao's Mathematical Foundations of Reinforcement Learning, and the educational work in CleanRL, simple_rl and RLCode.

Created by Yibin Liu · MIT license.