•UNIT-4

Reinforcement Learning-I
• What is Reinforcement Learning?
• Reinforcement Learning is a feedback-based Machine learning
technique in which an agent learns to behave in an
environment by performing the actions and seeing the results
of actions. For each good action, the agent gets positive
feedback, and for each bad action, the agent gets negative
feedback or penalty.
• In Reinforcement Learning, the agent learns automatically
using feedbacks without any labeled data, unlike
supervised learning.
• Since there is no labeled data, so the agent is bound to learn
by its experience only.
• RL solves a specific type of problem where decision making is
sequential, and the goal is long-term, such as game-playing,
robotics, etc.
• Terms used in Reinforcement Learning
• Agent(): An entity that can perceive/explore the environment and act
upon it.
• Environment(): A situation in which an agent is present or surrounded
by. In RL, we assume the stochastic environment, which means it is
random in nature.
• Action(): Actions are the moves taken by an agent within the
environment.
• State(): State is a situation returned by the environment after each
action taken by the agent.
• Reward(): A feedback returned to the agent from the environment to
evaluate the action of the agent.
• Policy(): Policy is a strategy applied by the agent for the next action
based on the current state.
• Value(): It is expected long-term retuned with the discount factor and
opposite to the short-term reward.
• Q-value(): It is mostly similar to the value, but it takes one additional
parameter as a current action (a).
• The agent interacts with the environment and explores it by
itself. The primary goal of an agent in reinforcement learning is
to improve the performance by getting the maximum positive
rewards.
• The agent learns with the process of hit and trial, and based
on the experience, it learns to perform the task in a better
way. Hence, we can say that "Reinforcement learning is a
type of machine learning method where an intelligent
agent (computer program) interacts with the
environment and learns to act within that." How a
Robotic dog learns the movement of his arms is an example of
Reinforcement learning.
• It is a core part of Artificial intelligence, and all AI agent works
on the concept of reinforcement learning. Here we do not need
to pre-program the agent, as it learns from its own experience
without any human intervention.
• Example: Suppose there is an AI agent present within a maze
environment, and his goal is to find the diamond. The agent
interacts with the environment by performing some actions,
and based on those actions, the state of the agent gets
changed, and it also receives a reward or penalty as feedback.
• The agent continues doing these three things (take action,
change state/remain in the same state, and get
feedback), and by doing these actions, he learns and explores
the environment.
• The agent learns that what actions lead to positive feedback or
rewards and what actions lead to negative feedback penalty.
As a positive reward, the agent gets a positive point, and as a
penalty, it gets a negative point.
• Key Features of Reinforcement Learning
• In RL, the agent is not instructed about the environment and
what actions need to be taken.
• It is based on the hit and trial process.
• The agent takes the next action and changes states according
to the feedback of the previous action.
• The agent may get a delayed reward.
• The environment is stochastic, and the agent needs to explore
it to reach to get the maximum positive rewards.
• Elements of Reinforcement Learning
• There are four main elements of Reinforcement Learning,
which are given below:
1.Policy
2.Reward Signal
3.Value Function
4.Model of the environment
• 1) Policy: A policy can be defined as a way how an agent
behaves at a given time. It maps the perceived states of the
environment to the actions taken on those states. A policy is
the core element of the RL as it alone can define the behavior
of the agent. In some cases, it may be a simple function or a
lookup table, whereas, for other cases, it may involve general
computation as a search process. It could be deterministic or a
stochastic policy:
• 2) Reward Signal: The goal of reinforcement learning is
defined by the reward signal. At each state, the environment
sends an immediate signal to the learning agent, and this
signal is known as a reward signal. These rewards are given
according to the good and bad actions taken by the agent. The
agent's main objective is to maximize the total number of
rewards for good actions. The reward signal can change the
policy, such as if an action selected by the agent leads to low
reward, then the policy may change to select other actions in
the future.
• 3) Value Function: The value function gives information
about how good the situation and action are and how much
reward an agent can expect. A reward indicates
the immediate signal for each good and bad action,
whereas a value function specifies the good state and
action for the future. The value function depends on the
reward as, without reward, there could be no value. The goal
of estimating values is to achieve more rewards.
• 4) Model: The last element of reinforcement learning is the
model, which mimics the behavior of the environment. With
the help of the model, one can make inferences about how the
environment will behave. Such as, if a state and an action are
given, then a model can predict the next state and reward.

• The model is used for planning, which means it provides a way

to take a course of action by considering all future situations
before actually experiencing those situations. The approaches
for solving the RL problems with the help of the model are
termed as the model-based approach. Comparatively, an
approach without using a model is called a model-free
approach.
• How does Reinforcement Learning Work?
• To understand the working process of the RL, we need to
consider two main things:
• Environment: It can be anything such as a room, maze,
football ground, etc.
• Agent: An intelligent agent such as AI robot.
• 2. The Return in Reinforcement Learning
• The Return (G) is the total accumulated reward that the agent receives
starting from a specific state. It's calculated based on immediate and future
rewards. In the discounted return formula, future rewards are multiplied by a
discount factor γ\gammaγ (gamma) to prioritize immediate rewards over
future ones:
• 3. Making Decisions: Policies in Reinforcement Learning
• A Policy (π) defines the behavior of the agent at each state. It is either
deterministic or stochastic:
• Deterministic policy: Maps states to actions directly.
• Stochastic policy: Maps states to a probability distribution over actions.
• 4. Review of Key Concepts
• Exploration vs Exploitation: The balance between trying new actions
(exploration) and using known actions that yield high rewards (exploitation).
• Markov Decision Process (MDP): RL problems are often modeled as MDPs,
characterized by states, actions, rewards, and transition probabilities.
• Learning Rate: Determines how much new information overrides old
knowledge.
• Discount Factor (γ): A value between 0 and 1 that controls the importance of
future rewards.
• 6. State-action Value Function Example
• Example: In a simple grid-world environment, where an agent needs to
navigate from one point to another, the Q-function estimates the value of
moving in each direction (up, down, left, right) from any given point in the grid
based on potential rewards (e.g., reaching the goal).
• Advantages:
• No labeled data needed.
• Can adapt to changing environments.
• Suitable for long-term planning.
• Disadvantages:
• Requires many interactions with the environment (slow learning).
• High computational cost.
• Difficult to implement in continuous action spaces.
• Applications:
• Robotics, self-driving cars, personalized recommendations, gaming (e.g.,
AlphaGo), and financial trading.
• Real-time Use Case: In self-driving cars, RL can be used to make real-time
decisions about navigation, braking, and lane-changing by learning from
simulated driving environments.
• Future Scope:
• Expanding RL in fields like healthcare (e.g., treatment planning), industrial
automation, personalized learning systems, and robotics with greater
autonomy.
• The Return in Reinforcement Learning: Understanding the concept of return,
which represents the cumulative future reward“:

• In reinforcement learning (RL), return is a key concept that represents the

total reward an agent expects to receive from the current time step onward
into the future. To understand this fully, let’s break down the essential
elements involved in this concept

• 1. Reward in Reinforcement Learning

• In RL, an agent interacts with an environment and takes actions in various
states. For each action, the environment provides feedback in the form of a
reward, which could be either positive or negative, depending on how
beneficial the action was in relation to achieving a goal.
• 2. What is Return?
• The return is the cumulative sum of rewards that the agent collects over time.
When making decisions, the agent doesn’t just focus on immediate rewards;
instead, it aims to maximize the total future rewards it can accumulate. Return
helps formalize this concept, as it represents the expected future rewards
from a given state or state-action pair.
However, in many RL problems, we need to account for the fact that rewards in the distant future might be worth
less than immediate rewards.

This is handled using a discount factor γ\gammaγ, which helps discount future rewards.
• Conclusion
• In reinforcement learning, the return encapsulates the idea of cumulative
future rewards that guide the agent's learning process. By understanding and
maximizing the return, the agent learns to make better decisions that lead to
higher long-term rewards.

• The return is at the heart of the objective in RL, as agents aim to optimize
their behavior to achieve the highest possible return in their interactions with
the environment.
• 1. What is a Policy?
• A policy in reinforcement learning is a function or rule that defines the agent's
behavior. It tells the agent what action to take when it finds itself in a
particular state. Mathematically, a policy π\piπ is represented as:
• 3. Role of Policies in Reinforcement Learning
• The agent’s primary goal in RL is to find the optimal policy π∗\pi^*π∗ that
maximizes the expected return—the cumulative future reward—when
following that policy. The optimal policy leads to the highest long-term
rewards from any given state. This is typically done by interacting with the
environment and learning through trial and error.
• The agent must strike a balance between:
• Exploration: Trying new actions to discover their effects and potential long-
term rewards.
• Exploitation: Choosing actions that are known to provide high rewards based
on past experience.
• 4. Types of Policies
• There are two primary approaches to defining policies in RL:
• a. Implicit Policies: Value-Based Methods
• In value-based methods, such as Q-learning, the policy is not directly
represented. Instead, the agent maintains a value function (such as the Q-
value function) that estimates the expected return for each state-action pair.
The agent then implicitly follows a policy that chooses the action with the
highest value for each state:
• Applications of Policies in Real-World Problems
• Policies are the core of many real-world applications of reinforcement
learning:
• Robotics: A robot’s policy determines how it interacts with its environment,
moving and adjusting to achieve tasks like object manipulation or
autonomous navigation.
• Game AI: In games, policies guide AI agents to make decisions, such as in
AlphaGo or AlphaZero, where policies dictate moves based on game states.
• Autonomous Driving: Policies in self-driving cars decide how the car should
act in various traffic conditions.
• Healthcare: In medical decision-making, policies could guide treatment plans
based on patient states and expected outcomes.
• Conclusion
• In reinforcement learning, policies are fundamental because they define how
the agent behaves in any given state.

• Whether they are deterministic or stochastic, explicitly represented or learned

indirectly through value functions, policies guide the agent’s decision-making
process as it interacts with the environment.

• The ultimate goal in RL is to find an optimal policy that maximizes the

expected cumulative reward, ensuring that the agent makes the best possible
decisions over time.
• In reinforcement learning (RL), the state-action value function is a critical
concept that helps an agent evaluate the effectiveness of actions taken in
different states.

• This function, commonly referred to as the Q-function and denoted as

Q(s,a)Q(s, a)Q(s,a), represents the expected future reward that an agent can
obtain by taking a particular action in a given state and then following a
specific policy thereafter.
• 1. What is a State-Action Value Function?
• The state-action value function or Q-function measures the long-term value
of performing a certain action in a particular state under a specific policy. In
other words, it tells the agent what total rewards it can expect if it chooses an
action in the current state and follows the same strategy (policy) for future
actions.
• Extensions of the Q-Function
• In practice, the Q-function can be extended or approximated in different ways,
depending on the complexity of the problem:
• Deep Q-Learning (DQN): When the state or action space is large, the Q-
function can be approximated using deep neural networks. This leads to the
Deep Q-Network (DQN) algorithm, where a neural network is trained to
approximate Q(s,a)Q(s, a)Q(s,a) for all state-action pairs.
• Double Q-Learning: This variation addresses the problem of overestimation
bias in Q-learning by maintaining two sets of Q-values, which help reduce the
bias when updating the Q-function.
• Multi-Agent Q-Learning: In multi-agent environments, the Q-function is
adapted to account for the presence of other agents, leading to algorithms
like Independent Q-Learning or Joint Action Learners.
• Conclusion
• The state-action value function (Q-function) is a fundamental concept in
reinforcement learning, representing the expected future reward of taking an
action in a given state and following a particular policy thereafter.

• It serves as the foundation for various RL algorithms, including Q-learning and

Deep Q-Networks (DQN), and plays a vital role in enabling agents to make
informed decisions and discover optimal policies in their environments.
• The Bellman Equation is a fundamental concept in Reinforcement Learning
(RL), particularly in the context of Markov Decision Processes (MDPs). It
provides a recursive decomposition of the value of a decision problem,
breaking it into immediate reward plus the value of the subsequent states.

• This equation helps in determining the optimal policy (set of actions) to

maximize long-term rewards in decision-making scenarios.
• Key Components:
1.State (s): Represents the current situation or environment the agent is in.
2.Action (a): The decision or action the agent takes from the current state.
3.Reward (r): The immediate return or feedback the agent receives after taking
an action.
4.Policy (π): The strategy the agent follows to decide which action to take in
each state.
5.Value Function (V): Measures the long-term value or expected cumulative
reward of being in a state under a specific policy.
6.Discount Factor (γ): A factor between 0 and 1 that discounts future rewards,
indicating that rewards received sooner are preferred over those received
later.
• Bellman Equation for Value Function
• The Bellman equation for the value function V(s)V(s)V(s) expresses the value
of a state as the expected reward from taking an action and then following the
optimal policy from the resulting next state. It can be written as:
• Use in Reinforcement Learning:
• The Bellman equation is crucial for Dynamic Programming methods like Value
Iteration and Policy Iteration in RL. It allows the agent to update its estimates
of state values (or action values) by repeatedly applying the equation, leading
to convergence towards the optimal value function and policy.
• In summary, the Bellman equation provides a recursive way to solve for the
value of decisions in sequential decision-making problems, and it forms the
backbone of many algorithms in reinforcement learning.
• 1. Interactive Game Simulation:
• Wow Moment: Let students play a simple grid-based game where an agent
must find its way to a goal while avoiding traps and collecting rewards. The
students can see how the agent learns to optimize its path using the Bellman
Equation to evaluate states.
• How It Works: Start with a random policy, then show how applying the
Bellman equation helps the agent gradually improve its choices. As the policy
improves, students will realize how future rewards and immediate rewards
influence decisions.
• . Real-Life Decision-Making Example:
• Wow Moment: Introduce a relatable scenario like planning a trip with
multiple stops. The students must consider immediate benefits (fun at each
destination) vs. future costs (time, money). Demonstrate how the Bellman
equation models this real-life problem by balancing short-term and long-term
rewards.
• How It Works: Let students make their own decisions for each stop, calculate
the "value" of their plan, and compare it to the optimal plan generated by
applying the Bellman equation. They will be surprised to see how a
mathematical model can make better decisions!
• Robotics Simulation:
• Wow Moment: Show a robot navigating a maze with obstacles. At first, the
robot gets stuck or takes inefficient paths. After running an algorithm based
on the Bellman equation, the robot learns to navigate the maze quickly and
efficiently.
• How It Works: Use a simulation tool (or a physical robot if available) to
illustrate how reinforcement learning with the Bellman equation allows the
robot to learn from its actions. Students will be impressed by how the robot
"learns" over time.
• Financial Investment Scenario:
• Wow Moment: Frame a financial problem: "Should you invest in a stock now
or later?" Explain that this decision can be modeled with the Bellman
equation, where immediate rewards (current stock price) and future rewards
(potential future price) are taken into account.
• How It Works: Present a scenario with fluctuating stock prices over time and
ask students to use the Bellman equation to maximize their long-term
investment returns. Seeing the optimal decision in such a real-world context is
a powerful "aha!" moment.
• Traffic Light Optimization:
• Wow Moment: Present a scenario where students control traffic lights at an
intersection. At first, cars may get stuck in traffic. After applying the Bellman
equation, the traffic lights optimize flow to minimize wait time.
• How It Works: Use a simple traffic simulation to demonstrate how the
Bellman equation can model this problem. By balancing immediate rewards
(few cars at the intersection) and future states (incoming traffic), students can
see the efficiency of RL in real-world traffic management.
• AI-powered Virtual Assistant:
• Wow Moment: Create a story where an AI virtual assistant is trying to
schedule tasks for its user. The assistant must balance immediate rewards
(completing urgent tasks) with future benefits (scheduling breaks to avoid
burnout). Applying the Bellman equation optimizes the assistant's decisions.
• How It Works: Show how the assistant's actions evolve as it learns to prioritize
based on the Bellman equation. The wow factor comes when students realize
that everyday tools like virtual assistants can be powered by reinforcement
learning principles.
• Personalized Learning Path:
• Wow Moment: Tailor a personalized study path for each student based on
their current understanding of a subject (current state) and their long-term
goal (mastering the topic). The study plan can be optimized using the Bellman
equation to maximize their learning over time.
• How It Works: Let students input their current understanding, and use the
Bellman equation to generate an optimal path for study sessions. They will be
amazed to see how this could guide their learning trajectory.