CS 5180 — Reinforcement Learning  |  Northeastern University
Authors: Shriman Raghav Srinivasan, Gautham Ramkumar
Report: AAAI-26 Format Paper  |  Poster

A systematic comparative study of six classical Reinforcement Learning agents and two LLM-based RLVR agents (GRPO-trained Qwen2.5-0.5B and 1.5B) on the puzzle game 2048. All classical deep agents share an identical 3-layer CNN backbone (330K parameters) to isolate algorithmic effects from architectural differences.

RLVR/
├── src/
│   ├── env/                     # 2048 Game Engine & Wrappers
│   │   ├── game_2048.py         # Core engine (pure NumPy, 4×4 board)
│   │   ├── gym_wrapper.py       # Gymnasium env (16-channel binary tensor, action masking)
│   │   └── text_wrapper.py      # ASCII text wrapper for LLM prompt formatting
│   │
│   ├── classical/               # Classical RL Agents
│   │   ├── dqn_agent.py         # Custom DQN (CNN, 100K replay buffer, target net, n_envs=1)
│   │   ├── ppo_agent.py         # MaskablePPO via SB3 (rollout=512, n_envs=8)
│   │   ├── a2c_agent.py         # A2C via SB3 (5-step returns, n_envs=8)
│   │   ├── qrdqn_agent.py       # QR-DQN via SB3-Contrib (50 quantiles, n_envs=8)
│   │   ├── sac_agent.py         # Discrete SAC (custom, twin Q-nets, 200K buffer, n_envs=8)
│   │   ├── lfa_agent.py         # Semi-gradient SARSA (30-dim features, 120 parameters)
│   │   ├── hunt_2048.py         # Hunt mode: unlimited attempts until tile 2048
│   │   ├── replay_gen.py        # Best-episode replay JSON generator
│   │   ├── export_all.py        # Batch export replays + manifest
│   │   ├── scaling_eval.py      # Evaluate milestone checkpoints → scaling curves
│   │   └── train.py             # Unified CLI: train / eval / replay
│   │
│   ├── llm/                     # LLM-RLVR Agents
│   │   ├── train_grpo.py        # GRPO training (Unsloth + TRL + Qwen2.5)
│   │   ├── reward.py            # Multi-component verifiable reward functions
│   │   ├── dataset.py           # Board-state dataset generator (2,000 prompts)
│   │   ├── predict.py           # Inference / evaluation for trained LLM
│   │   ├── prompt.py            # System prompt template (<think>/<answer>)
│   │   └── replay_gen.py        # LLM replay generator
│   │
│   ├── utils/
│   │   └── metrics.py           # CSV logger + matplotlib training curves
│   └── visualize.py             # Standalone visualization utilities
│
├── configs/                     # YAML hyperparameter configs
├── tests/                       # pytest suite (env + reward tests)
├── report/                      # AAAI-26 format LaTeX report + poster
│   ├── main.tex / main.pdf      # 10-page comparative study
│   ├── poster_a1.tex            # A1 research poster
│   ├── aaai26.sty               # AAAI style file
│   └── fig_*.png                # All figures (training curves, comparisons)
│
├── logs/                        # Training outputs (per-agent subdirectories)
│   ├── dqn_5m/ ppo_5m/ ...      # Classical agent logs, checkpoints, replays
│   ├── grpo_0.5b/ grpo_1.5b/    # GRPO adapter weights, merged models, train logs
│   └── manifest.json            # Agent registry for dashboard
│
├── index.html                   # Interactive web dashboard
├── serve.py                     # Dev HTTP server
└── requirements/                # Dependency files (base, classical, llm)

The 2048 game is modeled as a finite-horizon MDP:

• State space: 16-channel binary tensor x ∈ {0,1}^{16×4×4} (CNN agents) or labeled ASCII grid (LLM agents)
• Action space: {UP, RIGHT, DOWN, LEFT} — 4 discrete actions
• Transition: Deterministic tile-sliding + stochastic spawn (90% tile-2, 10% tile-4)
• Reward: Score-delta + milestone bonuses (+50 at 256, +100 at 512, +200 at 1024) + penalty for invalid moves (−1)
• Discount: γ = 0.99

All deep classical agents share an identical 3-layer CNN backbone:

Conv2D(16→128, 2×2) → ReLU → Conv2D(128→128, 2×2) → ReLU → Conv2D(128→128, 2×2) → ReLU → FC(128→256)

DQN — Custom PyTorch implementation with invalid Q-value masking inside the forward pass (SB3's DQN lacks this). Replay buffer 100K, batch 64, ε: 1.0→0.01 over 100K steps, target sync every 1K steps, lr=1e-4, gradient clipping 1.0.

SAC (Discrete) — Custom adaptation of Christodoulou (2019) with categorical policies, twin Q-networks, and automatic temperature tuning. Buffer 200K, batch 256, Polyak τ=0.005, target entropy 0.4·log|A|.

Semi-gradient SARSA — Sutton & Barto Algorithm 10.1 with 30-dimensional hand-crafted features:

Weight matrix: w ∈ ℝ^{4×30} = 120 parameters. Step-size α=1e-3, ε-decay 0.99995/step, ε_min=0.05.

PPO (MaskablePPO) — Rollout 512 steps × 8 envs, batch 128, 4 epochs, lr=2e-4, γ=0.995, clip ε=0.2, entropy coef=0.05.

A2C — 5-step returns, lr=7e-4, entropy coef=0.01, value coef=0.5.

QR-DQN — 50 quantiles (4×50=200 output values), buffer 100K, batch 64, lr=1e-4, target sync 1K steps.

Training Pipeline: Qwen2.5-Instruct models fine-tuned via GRPO (Group Relative Policy Optimization) using 4-bit NF4 quantization via Unsloth + LoRA (r=16, α=16).

Why Local GRPO, Not an LLM API? GRPO requires per-token log-probabilities and their gradients for the clipped policy update — commercial APIs don't expose model weights or gradient computation. Local deployment also ensures deterministic training under fixed seeds.

Reward Schedule (3-stage curriculum):

1. Steps 0–100: Format compliance only (XML tag structure, weight 0.5)
2. Steps 100–200: + Direction validity (weight 0.5)
3. Steps 200–500: + Game reward (Δscore/max_tile, weight 2.0)

Length and quality bonuses were removed after reward-hacking was detected (model generated filler text to maximize length rewards).

python -m venv venv && source venv/bin/activate
pip install -r requirements/base.txt

python -m src.env.game_2048
# Controls: W=Up, A=Left, S=Down, D=Right, Q=Quit

pip install -r requirements/classical.txt

# DQN (custom, ~2 hrs for 5M steps)
python -m src.classical.train train --agent dqn --steps 5000000 --mode 5m

# PPO (SB3, MaskablePPO)
python -m src.classical.train train --agent ppo --steps 5000000 --mode 5m

# Semi-gradient SARSA (NumPy, very fast)
python -m src.classical.train train --agent lfa --steps 5000000 --mode 5m

pip install -r requirements/llm.txt

# GRPO training — Qwen2.5-0.5B + QLoRA 4-bit
python -m src.llm.train_grpo \
  --dataset-size 2000 \
  --epochs 3 \
  --num-generations 4 \
  --lr 1e-6

# Run agents until tile 2048 or 3,000 attempts exhausted
python -m src.classical.hunt_2048
python -m src.classical.hunt_2048 --agents dqn lfa  # subset

python serve.py          # http://localhost:8080

Dashboard View	Description	Screenshot
Play 2048	Interactive game with undo, score tracking
Benchmark	Score convergence charts, comparative metrics
Agent Playback	Step-by-step replay with move-probability bars

python -m pytest tests/ -v

The research report follows the AAAI-26 conference format and covers:

1. Abstract — Problem statement and key results
2. Introduction — Why 2048 is a challenging benchmark; classical RL vs RLVR motivation
3. Background — MDP formulation, value-based methods, policy-gradient methods, linear FA, GRPO
4. Related Work — n-tuple TD networks (SOTA at tile 65536), deep RL for games, LLMs for game-playing, RLVR
5. Project Description — Environment, shared CNN backbone, all agent implementations with hyperparameters, GRPO pipeline
6. Experiments — 1M/5M/Hunt mode evaluation tiers, mechanism analysis of DQN dominance, on-policy degradation, GRPO scaling and spatial failure analysis
7. Conclusion — Four key findings with appropriate hedging for single-seed limitations

1. Guo et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Nature.
2. Shao et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.
3. Sutton & Barto (2018). Reinforcement Learning: An Introduction. 2nd ed. MIT Press.
4. Mnih et al. (2015). Human-Level Control Through Deep Reinforcement Learning. Nature.
5. Schulman et al. (2017). Proximal Policy Optimization Algorithms.
6. Haarnoja et al. (2018). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning.
7. Dabney et al. (2018). Distributional Reinforcement Learning with Quantile Regression.
8. Szubert & Jaśkowski (2014). Temporal Difference Learning of N-Tuple Networks for the Game 2048.
9. Hu et al. (2025). lmgame-Bench: How Good are LLMs at Playing Games?
10. Raffin et al. (2021). Stable-Baselines3: Reliable Reinforcement Learning Implementations.

This project was developed as part of CS 5180 (Reinforcement Learning) taught by Professor Robert Platt at Northeastern University.