In this project, I have implemented the Deep Q Network presented by DeepMind researchers in the original paper and follow-up Nature paper. I have also implemented architectures that improve upon the original DQN: Double DQN and Dueling DQN.
| Episode 50 | Episode 150 | Episode 750 |
|---|---|---|
Keep in mind that the learning speed of DQN can be sensitive to the random seed assigned and that the models were only trained once. Additionally, the proposed improvements to DQN (Double, Dueling and PER) were assessed on the scores across on all Atari games and not on Pong in particular.
This project would not have been possible without all the brilliant resources available on the Internet. Among many, the following resources have helped me the most:
- DQN tutorial for
PyTorch
- Introductory tutorial to DQN that got me off the ground.
- OpenAI Baselines
- Provides implementations of most Deep Reinforcement Learning algorithms written in Tensorflow. However, I used Baselines ReplayBuffer to create ReplayBufferTorch and the wrappers available here.
- Pre-processing of
frames
- Goes through the pre-processing Deepmind used in their DQN paper in great detail.
- Guide to speeding up
DQN
- Excellent guide to speeding up the convergence of DQN, provides hyperparameters that converges faster.
Trained for ~800 episodes and performed an evaluation every 50 episodes that consisted of playing 5 episodes.
- Update frequency = 4 (number of steps in the environment before performing an optimization step),
- Batch size = 32*4 = 128,
- Gamma = 0.99,
- Epsilon linearly decreasing from 1 to 0.02 over 100 000 steps with an evaluation epsilon of 0.002,
- Target network updated every 1 000 steps,
- Replay buffer initialized with 10 000 random steps and maximum capacity of 100 000,
- Learning rate for ADAM optimizer = 0.0001,
- Prioritized Experience Replay:
- Alpha = 0.6,
- Beta linearly increasing from 0.4 to 1 over 20 000 000 steps.