UNIT-5 PART C
1)Explain the Q function and Q learning algorithm assuming
deterministic rewards and actions with example.
ans)
https://www.freecodecamp.org/news/an-introduction-to-q-learn
ing-reinforcement-learning-14ac0b4493cc/
2) Explain the K – nearest neighbor algorithm for
approximating a discrete– valued function f : Hn→ V with
pseudo code.
3)Compare unsupervised learning and reinforcement learning
with examples.
4)Develop a Q learning task for the recommendation system of
an online shopping website. What will be the environment of the
system? Write the cost function and value function for the
system
A Q-learning task has these components: Agent, Environment,
State, Reward Function, Value Function and Policy.
1)To simplify the problem, we assume a hypothetical user whose
experience on an online shopping store is pooled from all the
actual users.
2)Our recommender model is going to be the agent of the
system that processes products to this hypothetical user that
will buy/not-buy the recommendation.
3)The user behaves as the system’s environment, responding to
the system’s recommendation depending to the state of the
system.
4)User feedback determines our reward, that is, one score only
if the user buys.
5)Action of the agent is product recommended.
6)Our state is defined as the product features and
corresponding user reactions of past 5 steps, excluding the
current step.
7)Therefore, feedback and action together give us the next
state.
The goal of the agent is to learn a policy that maximizes
accumulated rewards.
5)Identify the suitable learning method for training a robotic
arm and explain it.
ans) Industrial robots deployed today across various industries
are mostly doing repetitive tasks. Basically, moving or putting
objects in predefined trajectories. But the reality is that the
ability of robots to handle different or complex environments is
really limited in today’s manufacturing.The main challenge that
we have to overcome is designing adaptable control algorithms
that can easily adapt to new environments.
Reinforcement learning (RL) is a type of Machine Learning where
we can teach an agent how to behave in an environment by
performing actions and seeing the results.
The concept of Reinforcement Learning has been around for a
while but the algorithm was not very adaptable and was
incapable of doing continuous tasks.
For RL, we use a framework called the Markov Decision Process
(MDP) which produces an easy framework for a really complex
problem. An agent (e.g. robotic arm) would first observe the
environment it’s in and take actions accordingly. Rewards are
given out according to the result.
For robotic control the state is measured by using sensors to
measure the joint angles, velocity, and the end-effector pose:
Policy
The main objective is to find a policy. A policy is something that
tells us how to act in a particular state. The objective is to find a
policy that makes the most rewarding decisions.
Now, you put the objective together. We want to find a sequence
of actions that maximize expected rewards or minimize cost
Q-Learning
Q-learning is a model-free reinforcement learning algorithm
which means that it does not require a model of the
environment. It’s especially effective because it can handle
problems with random transitions and rewards, without
requiring adaptations.The most common Q-learning method
consists of these steps:
1. Sample an action.
2. Observe the reward and the next state.
3. Take the action with the highest Q.
7 . How does Q function become able to learn with and without
complete knowledge of reward function and state transition
function.
Q-learning is a model-free reinforcement learning algorithm
which means that it does not require a model of the
environment. It’s especially effective because it can handle
problems with random transitions and rewards, without
requiring adaptations.
Q-learning is an off-policy learner. Means it learns the value of the
optimal policy independently of the agent’s actions. On the other
hand, an on-policy learner learns the value of the policy being carried
out by the agent, including the exploration steps and it will find a
policy that is optimal, taking into account the exploration inherent in
the policy.
8 . How does setting a Reinforcement Learning problem require
an understanding of the following parameters of the problem?
(a) Delayed reward
(b) Exploring unknown or exploiting already learned states and
actions.
(c) Number of old states should be considered to decide action
Ans. (a) Delayed reward:
In the general case of the reinforcement learning problem, the
agent's actions determine not only its immediate reward, but
also the next state of the environment. The agent must take into
account the next state as well as the immediate reward when it
decides which action to take. The model of long-run optimality
the agent is using determines exactly how it should take the
value of the future into account. The agent will have to be able
to learn from delayed reinforcement: it may take a long
sequence of actions, receiving insignificant reinforcement, then
finally arrive at a state with high reinforcement. The agent must
be able to learn which of its actions are desirable based on
reward that can take place arbitrarily far in the future.
(b)The agents have to explore in order to improve the state
which potentially yields higher rewards in the future or exploit
the state that yields the highest reward based on the existing
knowledge. Pure exploration degrades the agent’s learning but
increases the flexibility of the agent to adapt in a dynamic
environment. On the other hand pure exploitation drives the
agent’s learning process to locally optimal solutions.
© The state is the current board position, the actions are the
different places in which you can place an ‘X’ or ‘O’ in a game of
Tic Tac Toe, and the reward is +1 or -1 depending on whether you
win or lose the game. The “state space” is the total number of
possible states in a particular RL setup. Tic tac toe has a small
enough state space (one reasonable estimate being 593) that we
can actually remember a value for each individual state, using a
table. This is called a tabular method for this reason. For models
like playing chess we use value function approximation as the
total number of possibilities is around 1049
