1) Introduction:

 Fuzzy Set :
 Fuzzy sets can be considered as an extension and gross oversimplification of classical sets.
 It can be best understood in the context of set membership. Basically it allows partial
membership which means that it contain elements that have varying degrees of membership
in the set.
 From this, we can understand the difference between classical set and fuzzy set. Classical set
contains elements that satisfy precise properties of membership while fuzzy set contains
elements that satisfy imprecise properties of membership.

 Definition (Membership function of fuzzy set):

In fuzzy sets, each elements is mapped to [0,1] by membership function. A : X [0, 1] where
[0,1] means real numbers between 0 and 1 (including 0 and 1).

1
 Fuzzy Rule:
A fuzzy rule can be represented by a fuzzy relation R = A → B

 R can be viewed as a fuzzy set with a two- dimensional membership function

• µR(x, y) = f(µA(x), µB(y)) where the function f, called the fuzzy implication function,
performs the task of transforming the membership degrees of x in A and y in B into those of (x,
y) in A × B.

• f is a min operator [Mamdani] and product operator [Larsen]

 Fuzzy logic Architecture:

2
2) Characteristics :

 Flexible and easy to implement machine learning technique.

 Helps you to mimic the logic of human thought.
 Logic may have two values which represent two possible solutions.
 Highly suitable method for uncertain or approximate reasoning.
 Fuzzy logic views inference as a process of propagating elastic constraints.
 Fuzzy logic allows you to build nonlinear functions of arbitrary complexity.
 Fuzzy logic should be built with the complete guidance of experts.

3) Advantages of Fuzzy Logic

 The structure of Fuzzy Logic is easy and understandable

 Fuzzy logic is widely used for commercial and practical purposes
 It helps you to control machines and consumer products
 It may not offer accurate reasoning, but the only acceptable reasoning
 It helps you to deal with the uncertainty in engineering
 Mostly robust as no precise inputs required
 It can be programmed to in the situation when feedback sensor stops working
 It can easily be modified to improve or alter system performance
 inexpensive sensors can be used which helps you to keep the overall system cost and
complexity low
 It provides a most effective solution to complex issues

4) Disadvantages of Fuzzy Logic

 Fuzzy logic is not always accurate, so the results are perceived based on assumption, so it
may not be widely accepted.

3
  Fuzzy systems don't have the capability of machine learning as-well-as neural network
type pattern recognition
 Validation and Verification of a fuzzy knowledge-based system needs extensive testing
with hardware
 Setting exact, fuzzy rules and, membership functions is a difficult task
 Some fuzzy time logic is confused with probability theory and the terms

5) Applications Of Fuzzy Logic:

Aerospace

In aerospace, fuzzy logic is used in the following areas −

 Altitude control of spacecraft

 Satellite altitude control
 Flow and mixture regulation in aircraft deicing vehicles

Business

In business, fuzzy logic is used in the following areas −

 Decision-making support systems

 Personnel evaluation in a large company

Electronics

In electronics, fuzzy logic is used in the following areas −

 Control of automatic exposure in video cameras

 Humidity in a clean room
 Air conditioning systems
 Washing machine timing
 Microwave ovens
 Vacuum cleaners

Finance

In the finance field, fuzzy logic is used in the following areas −

 Banknote transfer control

 Fund management
 Stock market predictions

4
 Industrial Sector

In industrial, fuzzy logic is used in following areas −

 Cement kiln controls heat exchanger control

 Activated sludge wastewater treatment process control
 Water purification plant control
 Quantitative pattern analysis for industrial quality assurance
 Control of constraint satisfaction problems in structural design
 Control of water purification plants

6) Problem:
Fuzzy Q-Learning (FQL) for any controllers with continues states:

Implementation:
Fuzzy Q-Learning is a fuzzy extension of Q-learning algorithm. For creating an FQL model you
first need to specify the input states and their corresponding fuzzy sets and then build your Fuzzy
Inference System (FIS) to integrate with the Q-Learning algorithm. In this code, two type of
fuzzy membership functions have been implemented:

1) Triangular

2) Trapezoidal

Fuzzy sets (as many as needed) can be created, initialized, and assigned to an input state variable
to build your FIS as follows:

x1 = StateVariable.InputStateVariable(FuzzySet.Trapeziums(-2.4, -2, -1, -0.5),

FuzzySet.Trapeziums(-1, -0.5, 0.5 , 1), FuzzySet.Trapeziums(0.5, 1, 2, 2.4) )
x2 = StateVariable.InputStateVariable(FuzzySet.Triangles(-2.4,-0.5,1), FuzzySet.Triangles(-
0.5,1,2.4))
fis = FIS.Build(x1,x2)

FQL model can built as follow:

model = FQL.Model(gamma = 0.9, alpha = 0.1 , ee_rate = 0.999, q_initial_value = 'zero',

action_set_length = 21, fis = fis)
7) SOURCE CODE:

5
statevariable.py

class InputStateVariable(object):

fuzzy_set_list = []

def __init__(self, *args):

self.fuzzy_set_list = args

def get_fuzzy_sets(self):

return self.fuzzy_set_list

FQL.py

import numpy as np

import FIS

import operator

import itertools

import functools

import random

import sys

import copy

class Model(object):

L = []

R = []

R_= []

M = []

Q=0

V=0

6
Error = 0

q_table = np.matrix([])

def __init__(self, gamma, alpha, ee_rate, q_initial_value, action_set_length,

fis = FIS.Build()):

self.gamma = gamma

self.alpha = alpha

self.ee_rate = ee_rate

self.q_initial_value = q_initial_value

self.action_set_length = action_set_length

self.fis = fis

if self.q_initial_value =='random':

self.q_table = np.random.random((self.fis.get_number_of_rules(), self.action_set_length))

if self.q_initial_value == 'zero':

self.q_table = np.zeros((self.fis.get_number_of_rules(), self.action_set_length))

def CalculateTruthValue(self,state_value):

self.R = []

self.L = []

input_variables = self.fis.list_of_input_variable

for index, variable in enumerate(input_variables):

X =[]

fuzzy_sets = variable.get_fuzzy_sets()

for set in fuzzy_sets:

membership_value = set.membership_value(state_value[index])

7
X.append(membership_value)

self.L.append(X)

for element in itertools.product(*self.L):

self.R.append(functools.reduce(operator.mul, element, 1))

def ActionSelection(self):

self.M = []

r = random.uniform(0, 1)

max = -sys.maxsize

for rull in self.q_table:

if r < self.ee_rate:

for index , action in enumerate(rull):

if action > max:

action_index = index

else:

action_index = random.randint(0, self.action_set_length -1)

self.M.append(action_index)

def InferredAction(self):

max = -sys.maxsize

for index , truth_value in enumerate(self.R):

if truth_value > max:

max = truth_value

action = self.M[index]

return action

def CalculateQValue(self):

8
self.Q = 0

for index, truth_value in enumerate(self.R):

self.Q = self.Q + truth_value * self.q_table[index,self.M[index]]

self.Q = self.Q / sum(self.R)

def CalculateStateValue(self):

self.V = 0

max = -sys.maxsize

for index, rull in enumerate(self.q_table):

for action in rull:

if action < max:

max = action

self.V = (self.R[index] * max) + self.V

if sum(self.R) == 0:

self.R[0] = 0.00001

self.V = self.V / sum(self.R)

def CalculateQualityVariation(self, reward):

self.Error = reward + ((self.gamma * self.V) - self.Q)

def UpdateqValue(self):

for index, truth_value in enumerate(self.R_):

delta_Q = self.alpha * (self.Error * truth_value)

self.q_table[index][self.M[index]] = self.q_table[index][self.M[index]] + delta_Q

def KeepStateHistory(self):

9
self.R_ = copy.copy(self.R)

def get_initial_action(self,state):

self.CalculateTruthValue(state)

self.ActionSelection()

action = self.InferredAction()

self.CalculateQValue()

self.KeepStateHistory()

return action

def run(self,state, reward):

self.CalculateTruthValue(state)

self.CalculateStateValue()

self.CalculateQualityVariation(reward)

self.UpdateqValue()

self.ActionSelection()

action = self.InferredAction()

self.CalculateQValue()

self.KeepStateHistory()

return action

FuzzySet.py

class Trapeziums(object):

def __init__(self, left, left_top, right_top, right):

self.left = left

10
self.right = right

self.left_top = left_top

self.right_top = right_top

def membership_value(self, input_value):

if (input_value >= self.left_top) and (input_value <= self.right_top):

membership_value = 1.0

elif (input_value <= self.left) or (input_value >= self.right_top):

membership_value = 0.0

elif input_value < self.left_top:

membership_value = (input_value - self.left) / (self.left_top - self.left)

elif input_value > self.right_top:

membership_value = (input_value - self.right) / (self.right_top - self.right)

else:

membership_value = 0.0

return membership_value

class Triangles(object):

def __init__(self, left, top, right):

self.left = left

self.right = right

self.top = top

11
def membership_value(self, input_value):

if input_value == self.top:

membership_value = 1.0

elif input_value <= self.left or input_value >= self.right:

membership_value = 0.0

elif input_value < self.top:

membership_value = (input_value - self.left) / (self.top - self.left)

elif input_value > self.top:

membership_value = (input_value - self.right) / (self.top - self.right)

return membership_value

FIS.py

class Build(object):

list_of_input_variable = []

def __init__(self,*args):

self.list_of_input_variable = args

def get_input(self):

return self.list_of_input_variable

def get_number_of_rules(self):

number_of_rules = 1

for input_variable in self.list_of_input_variable:

number_of_rules = (number_of_rules * self.get_number_of_fuzzy_sets(input_variable))

12
return number_of_rules

def get_number_of_fuzzy_sets(self, input_variable):

return len(input_variable.get_fuzzy_sets())

Environment.py

import math

class Environment(object):

action_set = []

state = []

def __init__(self):

# cart position in the horizontal direction x = state[0], and cart velocity denoted by x_ = state[1]

# vertical angle of the pole denoted by tetha = state[2], angular velocity of the pole tetha_ = state[3]

self.state = [0,0,0,0]

self.action_set = [-10,-9,-8,-7,-6,-5,-4,-3,-2,-1,0 ,1,2,3,4,5,6,7,8,9,10]

def apply_action(self,action):

u = self.action_set[action]

self.get_current_state(u)

reward = self.get_reward()

return reward, self.state

def get_state_variable(self,variable_name):

13
if variable_name == 'x':

return self.state[0]

elif variable_name == "x_":

return self.state[1]

elif variable_name == "tetha":

return self.state[2]

else:

return self.state[3]

def set_state_variable(self,variable_name,value):

if variable_name == 'x':

self.state[0] = value

elif variable_name == "x_":

self.state[1] = value

elif variable_name == "tetha":

self.state[2] = value

elif variable_name == "tetha_":

self.state[3] = value

def get_current_state(self,u):

mio_c = 0.000002

mio_p = 0.0005

m_big = 1

14
m = 0.1

l = 0.5

g = 9.8

# The dynamics of the cart-pole system

theta__ = ((g * math.sin(self.get_state_variable('tetha')) - math.cos(self.get_state_variable('tetha')))

* (u + (m * l * math.pow(self.get_state_variable('tetha_'),2) *
math.sin(self.get_state_variable('tetha'))) - (mio_c * math.copysign(self.get_state_variable('x_'),1)))
- mio_p * g * math.sin(self.get_state_variable('tetha')) - math.cos(self.get_state_variable('tetha')) *
(u + (m * l * math.pow(self.get_state_variable('tetha_'),2) *
math.sin(self.get_state_variable('tetha'))) - (mio_c * math.copysign(1,self.get_state_variable('x_'))))
- ((mio_p * self.get_state_variable('tetha_'))/(m*l))) / (l * ((4/3) - ((m *
math.pow(math.cos(self.get_state_variable('tetha')), 2))/ m_big + m)))

x__= (u + ((m * l) *
((math.pow(self.get_state_variable('tetha_'),2)*math.sin(self.get_state_variable('tetha'))) - (theta__
* math.cos(self.get_state_variable('tetha'))))) - (mio_c *
math.copysign(self.get_state_variable('x_'),1))) / (m_big + m)

self.set_state_variable('x', self.get_state_variable('x') + (self.get_state_variable('x_') * 0.02))

self.set_state_variable('x_', self.get_state_variable('x_') + (x__ * 0.02))

self.set_state_variable('tetha', self.get_state_variable('tetha') + (self.get_state_variable('tetha_') *

0.02))

self.set_state_variable('tetha_', self.get_state_variable('tetha_') + (theta__ * 0.02))

def get_reward(self):

if (math.copysign(self.get_state_variable('x'),1) > 2.4) or

(math.copysign(self.get_state_variable('tetha'),1) > 0.2094):

reward = -1

else:

reward = 0

return reward

CartPole.py

15
import FuzzySet

import StateVariable

import FQL

import FIS

from Environment import Environment

import matplotlib.pyplot as plt

# Create FIS

x1 = StateVariable.InputStateVariable(FuzzySet.Trapeziums(-2.4, -2, -1, -0.5),

FuzzySet.Trapeziums(-1, -0.5, 0.5 , 1), FuzzySet.Trapeziums(0.5, 1, 2, 2.4) )

x2 = StateVariable.InputStateVariable(FuzzySet.Triangles(-2.4,-0.5,1), FuzzySet.Triangles(-
0.5,1,2.4))

x3 = StateVariable.InputStateVariable(FuzzySet.Triangles(-3.14159, -1.5, 0), FuzzySet.Triangles(-

1.5, 0, 1.5), FuzzySet.Triangles(0, 1.5, 3.1459))

x4 = StateVariable.InputStateVariable(FuzzySet.Triangles(-3.14159, -1.5, 0), FuzzySet.Triangles(-

1.5, 0, 1.5), FuzzySet.Triangles(0, 1.5, 3.1459))

fis = FIS.Build(x1,x2,x3,x4)

# Create Model

angel_list = []

model = FQL.Model(gamma = 0.9, alpha = 0.1 , ee_rate = 0.999, q_initial_value = 'random',

action_set_length = 21, fis = fis)

env = Environment()

for iteration in range (0,5000):

16
if iteration % 100 == 0 or reward == -1:

env.__init__()

action = model.get_initial_action(env.state)

reward, state_value = env.apply_action(action)

action = model.run(state_value, reward)

reward, state_value = env.apply_action(action)

if reward != -1:

angel_list.append(state_value[2])

plt.figure(figsize=(14,3))

plt.plot(angel_list)

plt.ylabel('Pole Angel')

plt.show()

8) Output:

17
9) Conclusion:

18
19