Minesweeper RL — AI Decision-Making Under Uncertainty

A hybrid Minesweeper AI that chains an exact probabilistic solver, a CNN probability predictor, and a PPO reinforcement learning agent into a single end-to-end pipeline.

Minesweeper is a partially observable puzzle in which the player must select actions from incomplete information. Exact probabilistic solvers can play strongly, but their inference cost grows exponentially with frontier complexity. This project investigates whether a CNN can learn a fixed-cost approximation of the solver's posterior mine-probability maps, and whether a reinforcement learning agent trained on those learned maps can approach solver-level gameplay performance.

The three-stage pipeline is:

Exact Bayesian Solver  →  CNN Probability Predictor  →  PPO RL Agent
   (dense targets)           (fast approximation)         (action selection)

minesweeper_rl/
├── game_engine.py            # Minesweeper game engine (reveal, flood-fill, frontier)
├── solver.py                 # Exact probabilistic Minesweeper solver
├── solver_util.py            # Solver utilities (supercells, rule reduction)
├── predictor.py              # Unified probability interface (Bayesian or CNN)
├── rl.py                     # RL policy network architecture (actor-critic)
│
├── cnn_dataset_gen.py        # Generate solver-labelled training data
├── cnn_train.py              # Train the CNN probability predictor
├── cnn_compare_prediction.py # Evaluate & compare Bayesian vs CNN predictions
│
├── rl_train.py               # Train the RL agent with PPO
├── rl_play.py                # Evaluate the RL agent over N games
├── rl_compare_play.py        # Head-to-head: RL vs Bayesian gameplay
├── bayes_play.py             # Evaluate the Bayesian solver over N games
│
├── models/
│   ├── CNN/{easy,intermediate,hard}/model.keras
│   └── RL/{easy,intermediate,hard}/policy.keras
│
├── data/                     # CNN training datasets (generated locally)
│   └── {easy,intermediate,hard}/
│
└── logs/                     # Evaluation outputs, plots, JSON aggregates
    └── {easy,intermediate,hard}/

• Python 3.10+
• TensorFlow 2.x (GPU recommended)
• NumPy, tqdm, matplotlib

pip install tensorflow numpy tqdm matplotlib

git clone https://github.com/saminhasan/minesweeper_rl.git
cd minesweeper_rl

The pipeline has four ordered stages. Each stage can be run independently if the prior stage's outputs already exist.

Plays Minesweeper games using the exact Bayesian solver and records per-step board snapshots (visible state + posterior mine-probability map + covered mask) as binary training data.

python cnn_dataset_gen.py

Configure at the top of the file:

Output: data/<level>/<timestamp>/data.bin + metadata.json

Trains a fully-convolutional residual-dilated network to predict per-cell mine and safe probabilities from the four-channel board representation.

python cnn_train.py

Configure ACTIVE_LEVEL and LEVEL_CONFIGS at the bottom of the file.

Key hyperparameters (file top):

The model is saved to models/CNN/<level>/model.keras only when validation loss improves. Training curves are saved to models/CNN/<level>/training_curves/.

Trains a PPO actor-critic policy using the six-channel state:

[board_scaled | covered_mask | revealed_clue | frontier_mask | P̂(mine) | P̂(safe)]

python rl_train.py

Configure ACTIVE_LEVEL and LEVEL_CONFIGS at the bottom of the file.

Key hyperparameters:

The policy is saved to models/RL/<level>/policy.keras when win rate improves.

python bayes_play.py

python rl_play.py

Plays both agents on identical random seeds and produces a comparison plot and CSV.

python rl_compare_play.py

Evaluates Brier score, FPR, FNR, and ROC-AUC on held-out games.

python cnn_compare_prediction.py

All scripts write results to logs/<level>/ with timestamped subdirectories containing a CSV, aggregate.json, and plots.

Pre-trained models are included in the repository under models/:

models/CNN/easy/model.keras
models/CNN/intermediate/model.keras
models/CNN/hard/model.keras

models/RL/easy/policy.keras
models/RL/intermediate/policy.keras
models/RL/hard/policy.keras

To use the trained models directly, run any Stage 4 evaluation script — they load the model automatically from the models/ path.

The CNN tracked the exact Bayesian solver closely across all difficulty levels. Prediction quality degrades as board difficulty increases because harder boards contain larger ambiguous frontiers.

The CNN achieved lower FPR than the Bayesian baseline thresholded at 0.5 (more conservative about flagging safe cells as mines), consistent with the asymmetric training loss that penalizes mine underestimation. FNR — the safety-critical metric, since a false negative means treating a mine as safe — remained higher than the Bayesian teacher, but the gap was modest on easy and intermediate boards.

The RL agent was compared against the exact Bayesian baseline on matched seeds.

The RL policy matched or exceeded the Bayesian solver on easy boards and was competitive on intermediate. On hard boards the exact solver remained substantially stronger — shorter RL episode lengths on hard reflect earlier mine hits, not faster solutions.

• A learned CNN can faithfully approximate exact solver probability maps at constant inference cost regardless of frontier complexity.
• A PPO policy operating on uncertainty-aware state achieves near-Bayesian gameplay on easy and intermediate difficulty.
• Hard boards — where the frontier is large and clues under-constrain the hidden state — still benefit most from exact probabilistic reasoning.

• 4-channel input: [board_scaled, covered_mask, revealed_clue, frontier_mask]
• Stack of residual dilated convolutional blocks (dilation rates 1, 2, 4, 8)
• Final 2-channel softmax head: [P̂(mine), P̂(safe)]
• Asymmetric weighted loss: covered cells × 2, mine underestimation × 2

• 6-channel input: base 4 channels + [P̂(mine), P̂(safe)] from CNN
• Actor-critic with shared convolutional body and separate policy/value heads
• Action masking: only currently covered cells are selectable
• Reward: -1 on mine hit; Δuncovered/U + 0.5·Δfrontier/U + 1 on win

# 1. Generate data (easy level, ~260k games)
python cnn_dataset_gen.py

# 2. Train CNN
python cnn_train.py

# 3. Evaluate CNN predictions
python cnn_compare_prediction.py

# 4. Train RL
python rl_train.py

# 5. Compare RL vs Bayesian
python rl_compare_play.py

Change ACTIVE_LEVEL at the top (or bottom) of each script to "intermediate" or "hard" and repeat.