DDA4230 Assignment 3 - Q learning 119010319.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "For this part of the assignment, we are still going to use the FrozenLake-v0 environment. We are going to implment the Q learning algorithm and evaluate it on the Frozenlake environment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Action Space: Discrete(4)\n",
      "Observation Space: Discrete(16)\n",
      "Max Episode Steps: 100\n",
      "Nondeterministic: False\n",
      "Reward Range: (0, 1)\n",
      "Reward Threshold: 0.7\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "import time\n",
    "import numpy\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def query_environment(name):\n",
    "  env = gym.make(name)\n",
    "  spec = gym.spec(name)\n",
    "  print(f\"Action Space: {env.action_space}\")\n",
    "  print(f\"Observation Space: {env.observation_space}\")\n",
    "  print(f\"Max Episode Steps: {spec.max_episode_steps}\")\n",
    "  print(f\"Nondeterministic: {spec.nondeterministic}\")\n",
    "  print(f\"Reward Range: {env.reward_range}\")\n",
    "  print(f\"Reward Threshold: {spec.reward_threshold}\")\n",
    "\n",
    "query_environment(\"FrozenLake-v1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q Learning Implmentation \n",
    "\n",
    "In it’s simplest implementation, Q-Learning is a table of values for every state (row) and action (column) possible in the environment. Within each cell of the table, we learn a value for how good it is to take a given action within a given state. In the case of the FrozenLake environment, we have 16 possible states (one for each block), and 4 possible actions (the four directions of movement), giving us a 16 x 4 table of Q-values. We start by initializing the table to be uniform (all zeros), and then as we observe the rewards we obtain for various actions, we update the table accordingly.\n",
    "\n",
    "The update rule is \n",
    "\n",
    "Q(s,a) = (1 - lr) Q(s,a) + lr(r(s,a) + γ Q(s,a))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[2.21259560e-01 1.45610496e-01 3.36594249e-01 1.91829395e-01]\n",
      " [3.97124083e-04 4.04258718e-01 1.41985974e-01 1.28571209e-01]\n",
      " [1.06394234e-01 1.31860384e-01 4.18685897e-01 1.00766430e-01]\n",
      " [5.86185081e-03 7.02078869e-04 2.15528740e-03 1.02640021e-01]\n",
      " [2.36056218e-01 5.34777879e-02 2.13405888e-01 4.88682200e-02]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [1.19179838e-02 1.34190002e-03 3.95423129e-01 1.29631716e-02]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [4.31979566e-02 2.13127519e-01 1.13460566e-02 2.57087778e-01]\n",
      " [7.35351643e-02 4.63529547e-01 2.09151476e-02 1.50600730e-01]\n",
      " [3.87618146e-01 2.90688772e-02 6.66281963e-01 2.99059250e-02]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]\n",
      " [5.24427208e-01 1.20008108e-01 3.83995183e-01 1.06174017e-01]\n",
      " [5.75087453e-01 6.51544723e-01 9.39173849e-01 6.50681432e-01]\n",
      " [0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "import gym\n",
    "env = gym.make('FrozenLake-v1')\n",
    "\n",
    "#Initialize table, with states as rows and actions (up, down, left, or right) as columns \n",
    "Q = np.zeros([env.observation_space.n, env.action_space.n])   # Q[nS][nA]\n",
    "#Set learning parameters\n",
    "lr = .8\n",
    "#Set discounted factor\n",
    "gamma = .95\n",
    "epsilon = 0.05\n",
    "num_episodes = 200000\n",
    "#create lists to contain total rewards and steps per episode\n",
    "# rewards = []\n",
    "avg_reward = 0\n",
    "\n",
    "for i in range(num_episodes):\n",
    "    #Reset environment and get first new observation\n",
    "    s = env.reset()     # start_state\n",
    "    #Total reward in one episode\n",
    "    reward_tot = 0\n",
    "    while True:\n",
    "        ###############################################################################\n",
    "        # TODO: Implement the Q-Table learning algorithm.                             #\n",
    "        # You will need to do the following:                                          #\n",
    "        # (1) Choose an action by greedily (with noise) picking from Q table given s  #\n",
    "        #     as input.                                                               #\n",
    "        # (2) Get new state s1, reward r and done d from environment                  #\n",
    "        # (3) Update Q-Table with new knowledge.                                      #\n",
    "        # (4) Cumulate the total reward                                           #\n",
    "        # (5) Update s                                                                #\n",
    "        # Note: You may use the gym interfaces env.action_space, env.step etc.        #\n",
    "        #       E.g. observation, reward, done, info = env.step(action)               #\n",
    "        #       Please refer to the docs for more information.                        #\n",
    "        #       For (1), consider adding noise as a mean of encouraging exploration.  #\n",
    "        #       For (3), calculate the new target Q-value using Bellman equation.     #\n",
    "        #       Instead of directly updating toward it, we take a small step in the   #\n",
    "        #       direction that will make the Q value closer to the target, i.e. use   #\n",
    "        #       learning rate that controls how much of the difference between        #\n",
    "        #       newly proposed Q-value and previous Q-value                           #\n",
    "        ###############################################################################\n",
    "        # greedy choice\n",
    "        if numpy.random.choice([1, 0], p=[1-env.action_space.n*epsilon, env.action_space.n*epsilon]):\n",
    "            action = numpy.argmax(Q[s])\n",
    "        # random choice from [0,...,nA-1]\n",
    "        else:\n",
    "            action = random.choice([i for i in range(env.action_space.n)])\n",
    "        next_state, reward, done, _ = env.step(action)\n",
    "        Q[s][action] = Q[s][action] + lr*(reward + gamma*numpy.max(Q[next_state]) - Q[s][action])\n",
    "        reward_tot += reward\n",
    "        s = next_state\n",
    "        ##############################################################################\n",
    "        #                             END OF YOUR CODE                               #\n",
    "        ##############################################################################\n",
    "\n",
    "        #end of one episode\n",
    "        if done == True:\n",
    "            break\n",
    "    avg_reward = (avg_reward*i+reward_tot)/(i+1)\n",
    "    # print(\"Score over time: \" +  str(avg_reward))\n",
    "print(Q)\n",
    "print(avg_reward)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "61dd0e9fca73d993dce8fd26eb066813c398e8547ba5df7c5876aeaa35e9886b"
  },
  "kernelspec": {
   "display_name": "Python 3.8.8 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}