Reinforcement Learning
o Reinforcement Learning is a feedback-based Machine learning technique in
which an agent learns to behave in an environment by performing the actions
and seeing the results of actions. For each good action, the agent gets positive
feedback, and for each bad action, the agent gets negative feedback or penalty.
o In Reinforcement Learning, the agent learns automatically using feedbacks
without any labelled data, unlike supervised learning.
o Since there is no labelled data, so the agent is bound to learn by its experience
only.
o RL solves a specific type of problem where decision making is sequential, and
the goal is long-term, such as game-playing, robotics, etc.
o The agent interacts with the environment and explores it by itself. The primary
goal of an agent in reinforcement learning is to improve the performance by
getting the maximum positive rewards.
o The agent learns with the process of hit and trial, and based on the experience, it
learns to perform the task in a better way. Hence, we can say
that "Reinforcement learning is a type of machine learning method where an
intelligent agent (computer program) interacts with the environment and
learns to act within that." How a Robotic dog learns the movement of his arms
is an example of Reinforcement learning.
o It is a core part of Artificial intelligence, and all AI agent works on the concept of
reinforcement learning. Here we do not need to pre-program the agent, as it
learns from its own experience without any human intervention.
o Example: Suppose there is an AI agent present within a maze environment, and
his goal is to find the diamond. The agent interacts with the environment by
performing some actions, and based on those actions, the state of the agent gets
changed, and it also receives a reward or penalty as feedback.
o The agent continues doing these three things (take action, change state/remain
in the same state, and get feedback), and by doing these actions, he learns and
explores the environment.
o The agent learns that what actions lead to positive feedback or rewards and
what actions lead to negative feedback penalty. As a positive reward, the agent
gets a positive point, and as a penalty, it gets a negative point.
Terms used in Reinforcement Learning
o Agent(): An entity that can perceive/explore the environment and act upon it.
o Environment(): A situation in which an agent is present or surrounded by. In RL,
we assume the stochastic environment, which means it is random in nature.
o Action(): Actions are the moves taken by an agent within the environment.
o State(): State is a situation returned by the environment after each action taken
by the agent.
o Reward(): A feedback returned to the agent from the environment to evaluate
the action of the agent.
o Policy(): Policy is a strategy applied by the agent for the next action based on
the current state.
o Value(): It is expected long-term retuned with the discount factor and opposite
to the short-term reward.
o Q-value(): It is mostly similar to the value, but it takes one additional parameter
as a current action (a).
Key Features of Reinforcement Learning
o In RL, the agent is not instructed about the environment and what actions need
to be taken.
o It is based on the hit and trial process.
o The agent takes the next action and changes states according to the feedback of
the previous action.
o The agent may get a delayed reward.
o The environment is stochastic, and the agent needs to explore it to reach to get
the maximum positive rewards.
Approaches to implement Reinforcement Learning
There are mainly three ways to implement reinforcement-learning in ML, which are:
1. Value-based:
The value-based approach is about to find the optimal value function, which is
the maximum value at a state under any policy. Therefore, the agent expects the
long-term return at any state(s) under policy π.
2. Policy-based:
Policy-based approach is to find the optimal policy for the maximum future
rewards without using the value function. In this approach, the agent tries to
apply such a policy that the action performed in each step helps to maximize
the future reward.
The policy-based approach has mainly two types of policy:
o Deterministic: The same action is produced by the policy (π) at any state.
o Stochastic: In this policy, probability determines the produced action.
3. Model-based: In the model-based approach, a virtual model is created for the
environment, and the agent explores that environment to learn it. There is no
particular solution or algorithm for this approach because the model
representation is different for each environment.
Types of Reinforcement learning
There are mainly two types of reinforcement learning, which are:
o Positive Reinforcement
o Negative Reinforcement
Positive Reinforcement:
The positive reinforcement learning means adding something to increase the tendency
that expected behaviour would occur again. It impacts positively on the behaviour of
the agent and increases the strength of the behaviour.
This type of reinforcement can sustain the changes for a long time, but too much
positive reinforcement may lead to an overload of states that can reduce the
consequences.
Negative Reinforcement:
The negative reinforcement learning is opposite to the positive reinforcement as it
increases the tendency that the specific behaviour will occur again by avoiding the
negative condition.
It can be more effective than the positive reinforcement depending on situation and
behaviour, but it provides reinforcement only to meet minimum behaviour.
How to represent the agent state?
We can represent the agent state using the Markov State that contains all the required
information from the history. The State St is Markov state if it follows the given
condition:
P[St+1 | St ] = P[St +1 | S1,......, St]
The Markov state follows the Markov property, which says that the future is
independent of the past and can only be defined with the present. The RL works on
fully observable environments, where the agent can observe the environment and act
for the new state. The complete process is known as Markov Decision process, which
is explained below:
Markov Decision Process
Markov Decision Process or MDP, is used to formalize the reinforcement learning
problems. If the environment is completely observable, then its dynamic can be
modeled as a Markov Process. In MDP, the agent constantly interacts with the
environment and performs actions; at each action, the environment responds and
generates a new state.
MDP is used to describe the environment for the RL, and almost all the RL problem can
be formalized using MDP.
MDP contains a tuple of four elements (S, A, P a, Ra):
o A set of finite States S
o A set of finite Actions A
o Rewards received after transitioning from state S to state S', due to action a.
o Probability Pa.
MDP uses Markov property, and to better understand the MDP, we need to learn about
it.
Markov Property:
It says that "If the agent is present in the current state S1, performs an action a1 and
move to the state s2, then the state transition from s1 to s2 only depends on the
current state and future action and states do not depend on past actions, rewards, or
states."
Or, in other words, as per Markov Property, the current state transition does not
depend on any past action or state. Hence, MDP is an RL problem that satisfies the
Markov property. Such as in a Chess game, the players only focus on the current state
and do not need to remember past actions or states.
Finite MDP:
A finite MDP is when there are finite states, finite rewards, and finite actions. In RL, we
consider only the finite MDP.
Markov Process:
Markov Process is a memoryless process with a sequence of random states S1, S2, .....,
St that uses the Markov Property. Markov process is also known as Markov chain, which
is a tuple (S, P) on state S and transition function P. These two components (S and P)
can define the dynamics of the system.
Reinforcement Learning Algorithms
Reinforcement learning algorithms are mainly used in AI applications and gaming
applications. The main used algorithms are:
o Q-Learning:
o Q-learning is an Off policy RL algorithm, which is used for the temporal
difference Learning. The temporal difference learning methods are the way of
comparing temporally successive predictions.
o It learns the value function Q (S, a), which means how good to take action "a" at
a particular state "s."
o The below flowchart explains the working of Q- learning:
Reinforcement Learning Applications
1. Robotics:
1. RL is used in Robot navigation, Robo-soccer, walking, juggling, etc.
2. Control:
1. RL can be used for adaptive control such as Factory processes,
admission control in telecommunication, and Helicopter pilot is an
example of reinforcement learning.
3. Game Playing:
1. RL can be used in Game playing such as tic-tac-toe, chess, etc.
4. Chemistry:
1. RL can be used for optimizing the chemical reactions.
5. Business:
1. RL is now used for business strategy planning.
6. Manufacturing:
1. In various automobile manufacturing companies, the robots use deep
reinforcement learning to pick goods and put them in some containers.
7. Finance Sector:
1. The RL is currently used in the finance sector for evaluating trading
strategies.
Temporal difference learning
● It is a combination of monte carlo and dynamic programming.
● It is a model-free learning. Hence does not require the model to be known
in advance.
● It can be applied for non episodic tasks.
(In an episodic environment, an agent's current action will not affect a future
action, whereas, in a non-episodic environment, an agent's current action will
affect a future action and is also called the sequential environment.)
Temporal difference learning uses an update rule for update the value of a state.
V(S’t)=V(St)+alpha[Rt+1+gammV(St+1)-V(St)]
V(St) – the value of the previous state
Alpha – learning rate
Gamma - discount factor
V(St+1 )- the value of the current sate
R - Reward
Monte carlo idea
Learns directly from raw experience i.e without a model. There is no predefined
model
Dynamic programming
It estimates based on part of learning rather than waiting for the final outcome.
Independent Component Analysis (ICA)
● is a machine learning technique to separate independent sources from
a mixed signal.
● Unlike principal component analysis which focuses on maximizing the
variance of the data points, the independent component analysis
focuses on independence, i.e. independent components.
Problem: To extract independent sources’ signals from a mixed signal
composed of the signals from those sources.
Given: Mixed signal from five different independent sources.
Aim: To decompose the mixed signal into independent sources:
● Source 1
● Source 2
● Source 3
● Source 4
● Source 5
Solution: Independent Component Analysis (ICA).
Consider Cocktail Party Problem or Blind Source Separation problem to
understand the problem which is solved by independent component analysis.
Here, There is a party going into a room full of people. There is ‘n’ number of
speakers in that room and they are speaking simultaneously at the party. In
the same room, there are also ‘n’ number of microphones placed at different
distances from the speakers which are recording ‘n’ speakers’ voice signals.
Hence, the number of speakers is equal to the number of microphones in the
room.
Now, using these microphones’ recordings, we want to separate all the ‘n’
speakers’ voice signals in the room given each microphone recorded the voice
signals coming from each speaker of different intensity due to the difference in
distances between them. Decomposing the mixed signal of each microphone’s
recording into an independent source’s speech signal can be done by using
the machine learning technique, independent component analysis.
[ X1, X2, ….., Xn ] => [ Y1, Y2, ….., Yn ]
where, X1, X2, …, Xn are the original signals present in the mixed signal and
Y1, Y2, …, Yn are the new features and are independent components which
are independent of each other.
Restrictions on ICA –
1. The independent components generated by the ICA are assumed to
be statistically independent of each other.
2. the number of independent components generated by the ICA is
equal to the number of observed mixtures.
Difference between PCA and ICA –
Independent Component
Principal Component Analysis Analysis
It reduces the dimensions to avoid It decomposes the mixed signal into its
the problem of overfitting. independent sources’ signals.
It deals with the Principal It deals with the Independent
Components. Components.
It focuses on maximizing the It doesn’t focus on the issue of
variance. variance among the data points.
It doesn’t focus on the mutual It focuses on the mutual independence
independence of the components. of the components.
