42

CHAPTER 5

ADAPTIVE NEURO FUZZY MODELING AND

SOFTWARE IMPLEMENTATION

5.1 INTRODUCTION TO NEURO FUZZY SYSTEM

A neuro fuzzy system is a combination of neural network and

fuzzy systems in such a way that neural network or neural network
algorithms are used to determine the parameters of the fuzzy system. This
means that the main intention of neuro fuzzy approach is to create or
improve a fuzzy system automatically by means of neural network
methods. An even more important aspect is that the system should always
be interpretable in terms of fuzzy if-then rules, because it is based on a
fuzzy system reflecting vague knowledge.

The idea of a neuro fuzzy system is to find the parameters of a

fuzzy system by the means of learning methods obtained from neural
network. A common way to apply a learning algorithm to a fuzzy system
is to represent it in a special neural- network-like architecture. Then a
learning algorithm such as back propagation is used to train the system.
However, neural network learning algorithms are usually gradient descent
methods. This cannot be applied directly to a fuzzy system, because the
functions used to realize the inference process are usually not
differentiable. In order to realize the system, we need to replace the
functions used in the fuzzy system (like min and max) by differentiable
 43

functions or do not use a gradient-based neural learning algorithm but a

better-suited procedure.

Modern neuro fuzzy systems are often represented as

multilayer feed forward neural network. A neuro fuzzy system is a fuzzy
system that is trained by a learning algorithm (usually) derived from
neural network theory. The (heuristic) learning procedure operates on
local information, and causes only local modifications in the underlying
fuzzy system. The learning process is not knowledge-based, but data-
driven. Besides, a neural-fuzzy system can always be interpreted as a
system of fuzzy rules. It is possible both to create the system out of
training data from scratch, and to initialize it from prior knowledge in the
form of fuzzy-rules.

The learning procedure of a neural-fuzzy system takes the

semantical properties of the underlying fuzzy system into account. This
results in constraints on the possible modification of the system’s
parameters. It also approximates an n- dimensional (unknown) function
that is partially given by the training data. The fuzzy rules encoded within
the system represent vague samples, and can be viewed as vague
prototypes of the training data.

Generally, a neuro fuzzy system should not be seen as a kind of

(fuzzy) expert system, and it has nothing to do with fuzzy logic in the
narrow sense. It can be viewed as a special kind of feed forward neural
network. The units in this network use t-norms or t-conorms instead of the
activation functions normally used in neural networks. Fuzzy sets are
encoded as (fuzzy) connection weights.

The neuro fuzzy system is broadly classified into three systems

 44

namely; cooperative neuro fuzzy system, concurrent neuro fuzzy system

and hybrid neuro fuzzy system as shown in following Figure 5.1.

Figure 5.1 Types of Neuro Fuzzy System

The hybrid neuro fuzzy system is further classified into ten

networks namely Adaptive Neuro Fuzzy Inference System (ANFIS),
Fuzzy Adaptive Learning Control Network (FALCON), Neuronal Fuzzy
Controller (NEFCON), Fuzzy Net (FUN), Generalized Approximate
Reasoning based Intelligence Control (GARIC), Fuzzy Inference and Neural
Network in Fuzzy Inference Software (FINEST), Self Constructing Neural
Fuzzy Inference Network (SONFIN), Fuzzy Neural Network (FNN),
Evolving Fuzzy Neural Network (EFuNN) and Dynamic Evolving Fuzzy
Neural Network (dmEFuNN).

5.1.1 ANFIS Architecture

The Adaptive Network based Fuzzy Inference System (ANFIS)

(Jang, 1993) [29] shown in figure 5.2 implements a Takagi-Sugeno fuzzy
inference system and it has five layers. The first hidden layer is
responsible for the mapping of the input variable relatively to each
membership functions. The operator T-norm is applied in the second
hidden layer to calculate the antecedents of the rules. The third hidden
layer normalizes the rules strengths followed by the fourth hidden layer
where the consequents of the rules are determined. The output layer
 45

calculates the global output as the summation of all the signals that arrive
to this layer.

Figure 5.2 ANFIS Architecture

ANFIS uses back-propagation learning to determine the input

membership functions parameters and the least mean square method to
determine the consequents parameters. Each step of the iterative learning
algorithm has two parts. In the first part, the input patterns are propagated
and the parameters of the consequents are calculated using the iterative
minimum squared method algorithm, while the parameters of the premises
are considered fixed. In the second part, the input patterns are propagated
again and in each iteration, the learning algorithm back-propagation is
used to modify the parameters of the premises, while the consequents
remain fixed.

5.1.2 FALCON Architecture

The Fuzzy Adaptive Learning Control Network FALCON as

shown in figure 5.3 is an architecture of five layers. There are two
linguistics nodes for each output. One is for the patterns and the other is
for the real output of the FALCON. The first hidden layer is responsible
 46

for the mapping of the input variables relatively to each membership

functions.

Figure 5.3 FALCON Architecture

The second hidden layer defines the antecedents of the rules

followed by the consequents in the third hidden layer. FALCON uses an
hybrid learning algorithm composed by a unsupervised learning to define
the initial membership functions and initial rule base and it uses a
learning algorithm based on the gradient descent to optimise/adjust the
final parameters of the membership functions to produce the desired
output.

5.1.3 NEFCON Architecture

The Neural Fuzzy Controller (NEFCON) shown in figure 5.4

 47

implement a Mamdani type inference fuzzy system. The connections in

this architecture are weighted with fuzzy sets and rules using the same
antecedents (called shared weights), which are represented by the drawn
ellipses. They assure the integrity of the base of rules. The input units
assume the function of fuzzyfication interface, the logical interface is
represented by the propagation function and the output unit is responsible
for the defuzzyfication interface. The process of learning in architecture
NEFCON is based in a mixture of reinforcement learning with back-
propagation algorithm. This architecture can be used to learn the rule base
from the beginning, if there is no a prior knowledge of the system, or to
optimise an initial manually defined rule base. NEFCON has two variants
NEFPROX (for function approximation) and NEFCLASS (for
classification tasks).

Figure 5.4 NEFCON Architecture

 48

5.1.4 FUN Architecture

In Fuzzy Net (FUN) architecture shown in figure 5.5 the

neurons in the first hidden layer contain the membership functions and
this performs a fuzzification of the input values. In the second hidden
layer, the conjunctions are calculated. Membership functions of the output
variables are stored in the third hidden layer. Their activation function is a
fuzzy-OR. Finally the output neuron performs the defuzzification. The
network is initialized with a fuzzy rule base and the corresponding
membership functions and thereafter uses a stochastic learning technique
that randomly changes parameters of membership functions and
connections within the network structure. The learning process is driven
by a cost function, which is evaluated after the random modification. If
the modification resulted in an improved performance, then the
modification is kept, otherwise it is undone.
 49

Figure 5.5 FUN Architecture

5.1.5 GARIC Achitecture

The Generalized Approximate Reasoning based Intelligence

Control (GARIC) shown in figure 5.6 implements a neuro-fuzzy system
using two neural networks modules, ASN (Action Selection Network) and
AEN (Action State Evaluation Network). The AEN is an adaptive
evaluator of ASN actions. The ASN of the GARIC is an advanced
network of five layers. The connections between the layers are not
weighted. The first hidden layer stores the linguistics values of all input
 50

variables. Each input can only connect to the first layer, which represents
its associated linguistics values. The second hidden layer represents the
fuzzy rule nodes that determine the compatibility degree of each rule
using a softmin operator.

The third hidden layer represents the linguistics values of the

output variables. The conclusions of each rule are calculated depending
on the strength of the rules antecedents calculated in the rule nodes.
GARIC uses the mean of local mean of maximum method to calculate the
output of the rules. This method needs for a numerical value in the exit of
each rule. Thus, the conclusions should be transformed from fuzzy values
for numerical values before being accumulated in the final output value of
the system. GARIC uses a mixture of gradient descending and
reinforcement learning for a fine adjustment of its internal parameters.

Figure 5.6 GARIC Architecture

5.1.6 FINEST Architecture

The Fuzzy Inference and Neural Network in Fuzzy Inference

 51

Software (FINEST) architecture is capable of two kinds of tuning process,

the tuning of fuzzy predicates, combination functions and the tuning of an
implication function. The generalized modus ponens is improved in the
following four ways (1) Aggregation operators that have synergy and
cancellation nature (2) A parameterized implication function (3) A
combination function that can reduce fuzziness (4) Backward chaining
based on generalized modus ponens. FINEST make use of a back-
propagation algorithm for the fine-tuning of the parameters. Figure 5.7
shows the layered architecture of FINEST and the calculation process of
the fuzzy inference. FINEST provides a framework to tune any parameter,
which appears in the nodes of the network representing the calculation
process of the fuzzy data if the derivative function with respect to the
parameters is given.

Figure 5.7 FINEST Architecture

5.1.7 SONFIN Architecture

The Self Constructing Neural Fuzzy Inference Network

(SONFIN) architecture implements a modified Takagi-Sugeno FIS and is
illustrated in figure 5.8. In the structure identification of the precondition
part, the input space is partitioned in a flexible way according to an
 52

aligned clustering based algorithm. As to the structure identification of

the consequent part, only a singleton value selected by a clustering
method is assigned to each rule initially. Afterwards, some additional
significant terms (input variables) selected via a projection-based
correlation measure for each rule are added to the consequent part
incrementally as learning proceeds. For parameter identification, the
consequent parameters are tuned optimally by either least mean squares or
recursive least squares algorithms and the precondition parameters are
tuned by back propagation algorithm.

Figure 5.8 SONFIN Architecture

 53

5.1.8 EFuNN Architecture

In Evolving Neural Fuzzy Network EFuNN all nodes are created

during the learning phase. The first layer passes data to the second layer
that calculates the degrees of compatibility in relation to the predefined
membership functions. The third layer contains fuzzy rule nodes
representing prototypes of input- output data as an association of hyper-
spheres from the fuzzy input and fuzzy output spaces. Each rule node is
defined by two vectors of connection weights, which are adjusted through
a hybrid learning technique. The fourth layer calculates the degree to
which output membership functions are matched the input data and the
fifth layer carries out the defuzzyfication and calculates the numerical
value for the output variable.

Neural Fuzzy Network (dmEFuNN) shown in figure 5.9 is a

modified version of the EFuNN with the idea of not only the winning rule
node’s activation is propagated but a group of rule nodes that is dynamic
selected for every new input vector and their activation values are used to
calculate the dynamical parameters of the output function. While EFuNN
implements Mamdani type fuzzy rules, dmEFuNN implements Takagi-
Sugeno fuzzy rules.
 54

Figure 5.9 EFuNN Architecture

In this research work, Adaptive Neuro Fuzzy Inference System

(ANFIS) is taken into consideration for designing a neuro fuzzy controller
network to control the two-stage KY boost converter.

5.2 ADAPTIVE NEURO-FUZZY INFERENCE SYSTEM (ANFIS)

An Adaptive Neuro-Fuzzy Inference System (ANFIS) is a

combination of neural network and fuzzy systems in such a way that
neural network is used to determine the parameters of fuzzy system.
ANFIS largely removes the requirement for manual optimization of the
fuzzy system parameters. A neural network is used to automatically tune
 55

the system parameters, for example the membership functions bounds,

leading to improved performance without operator invention.

The neuro fuzzy system with the learning capability of neural

network and with the advantages of the rule-base fuzzy system can
improve the performance significantly and can provide a mechanism to
incorporate past observations into the classification process. In neural
network the training essentially builds the system. However, using a
neuro fuzzy scheme, the system is built by fuzzy logic definitions and is
then refined using neural network training algorithms.

Some advantages of ANFIS are:

 Refines fuzzy if-then rules to describe the behaviour of a complex

system.

 Does not require prior human expertise

 Uses membership functions with desired dataset to approximate

 Greater choice of membership functions to use.

 Very fast convergence time.

5.2.1 ANFIS ARCHITECTURE

The ANFIS is a fuzzy Sugeno model put in the framework of

adaptive systems to facilitate learning and adaptation (Jang, 1993 [29],
1995 [30]). Such framework makes the ANFIS modeling more systematic
and less reliant on expert knowledge. To present the ANFIS architecture,
two fuzzy if-then rules based on a first order Sugeno model are
 56

considered:

Rule 1: If (x is A1 ) and (y is B1 ) then (f = p1 x + q1 y + r1 )

Rule 2: If (x is A2 ) and (y is B2 ) then (f2 = p 2x + q 2 y + r2 )

where x and y are the inputs, Aj and Bj are the fuzzy sets, are the outputs
within the fuzzy region specified by the fuzzy rule, p, q and r are the
design parameters that are determined during the training process. Figure
5.10 illustrates the reasoning mechanism for this Sugeno model where it
is the basis of the ANFIS model.

Figure 5.10 A two-input first-order Sugeno fuzzy model with two rules

The ANFIS architecture to implement these two rules is shown

in Figure 5.11, in which a circle indicates a fixed node, whereas a square
indicates an adaptive node. Adaptive neuro fuzzy inference system
basically has 5 layer architectures and each of the function is explained in
detail later.
 57

Figure 5.11 ANFIS architecture

In the first layer, all the nodes are adaptive nodes. The outputs
of layer 1 are the fuzzy membership grade of the inputs, which are given
by equations 5.1 and 5.2;

(5.1)

(5.2)

where µ Ai (x), µBi(y) can adopt any fuzzy membership function. For
example, if the bell shaped membership function is employed, µA(x) is
given by equation 5.3;

(5.3)

where ai, b i and ci are the parameters of the membership function,

governing the bell shaped functions accordingly.
 58

In the second layer, the nodes are fixed nodes. They are labeled
with n, indicating that they perform as a simple multiplier. The outputs of
this layer can be represented as equation 5.4;

(5.4)

which are the so-called firing strengths of the rules.

In the third layer, the nodes are also fixed nodes labeled by N,
to indicate that they play a normalization role to the firing strengths from
the previous layer. The output of this layer can be represented as equation
5.5;

(5.5)

which are the so-called normalized firing strengths.

In the fourth layer, the nodes are adaptive. The output of each
node in this layer is simply the product of the normalized firing strength
and a first order polynomial (for a first order Sugeno model). Thus, the
output of this layer is given by equation 5.6;

(5.6)

In the fifth layer, there is only one single fixed node labeled
with £. This node performs the summation of all incoming signals. Hence,
the overall output of the model is given by equation 5.7;

(5.7)
 59

It can be observed that there are two adaptive layers in this

ANFIS architecture, namely the first and the fourth layers. In the first
layer, there are three modifiable parameters {a, b, c }, which are related to
the input membership functions. These parameters are the so-called
premise parameters. In the fourth layer, there are also three modifiable
parameters {p, q, r}, pertaining to the first order polynomial. These
parameters are the so-called consequent parameters (Jang, 1993 [29],
1995 [30]).

Figure 5.12 shows the variation in the Sugeno model that is

equivalent to a two-input first-order Sugeno fuzzy model with nine rules,
where each input is assumed to have three associated MFs. Figure 5.13
illustrates how the two dimensional input space is partitioned into nine
overlapping fuzzy regions, each of which is governed by a fuzzy if-then
rule. In other words, the premise part of a rule defines a fuzzy region,
while the consequent part specifies the output within the region.

Figure 5.12 Two-input first-order Sugeno fuzzy model with nine rules
 60

Figure 5.13 The input space that are partitioned into nine fuzzy regions

5.2.2 LEARNING ALGORITHM OF ANFIS

The task of the learning algorithm for this architecture is to

tune all the modifiable parameters, namely {aiy b, c} and {p, q, r}, to
make the ANFIS output match the training data. When the premise
parameters a, b and c of the membership function are fixed, the output of
the ANFIS model can be written as equation 5.8;

(5.8)

Substituting Equation 5.5 into Equation 5.8 yields equation 5.9:

(5.9)

Substituting the fuzzy if-then rules into Equation 5.9, it become as shown
in equation 5.10;
 61

(5.10)

After rearrangement, the output can be expressed as equation 5.11;

(5.11)

which is a linear combination of the modifiable consequent parameters p1,

q1, ri, p 2, q 2 and r2 . The least squares method can be used to identify the
optimal values of these parameters easily. When the premise parameters
are not fixed, the search space becomes larger and the convergence of the
training becomes slower. A hybrid algorithm combining the least squares
method and the gradient descent method is adopted to solve this problem.
The hybrid algorithm is composed of a forward and a backward pass. The
least squares method (forward pass) is used to optimize the consequent
parameters with the premise parameters fixed. Once the optimal
consequent parameters are found, the backward pass starts immediately.
The gradient descent method (backward pass) is used to adjust optimally
the premise parameters corresponding to the fuzzy sets in the input
domain. The output of the ANFIS is calculated by employing the
consequent parameters found in the forward pass. The following Table
5.1 summarizes the activities in each pass. The output error is used to
adapt the premise parameters by means of a standard backpropagation
algorithm. It has been proven that this hybrid algorithm is highly efficient
in training the ANFIS (Jang, 1993 [29], 1995 [30]).
 62

Table 5.1

Activities of forward and backward pass

Forward pass Backward pass

Premise parameters Fixed Gradient descent
Consequent Least-squares estimator Fixed
Signals Node outputs Error signals

5.2.3 ANFIS CLASSIFIER

Both neural network and fuzzy logic are universal estimators.

They can approximate any function to any prescribed accuracy, provided
that sufficient hidden neurons and fuzzy rules are available. Gradient
descent and Backpropagation algorithms are used to adjust the parameters
of membership functions (fuzzy sets) and the weights of defuzzification
(neural networks) for fuzzy neural networks. ANFIS applies two
techniques in updating parameters. The ANFIS is a FIS implemented in
the framework of an adaptive fuzzy neural network. It combines the
explicit knowledge representation of a FIS with the learning power of
ANNs. The objective of ANFIS is to integrate the best features of fuzzy
systems and neural network. The advantage of fuzzy is that prior
knowledge is represented into a set of constraints to reduce the
optimization research space. The adaptation of back propagation to
structured network so as to automate fuzzy control parametric tuning is
utilized from NN. For premise parameters that define membership
functions, ANFIS employs gradient descent algorithm to fine-tune them.
For consequent parameters that define the coefficients of each equation,
ANFIS uses the least- squares method to identify them. This approach is
thus called hybrid learning method since it combines gradient descent
 63

algorithm and least-squares method. To achieve good generalization of

unseen data, the size of the training data set should be at least as big as
the number of modifiable parameters in ANFIS. Functionally there are
almost no constrains on the node functions of an adaptive network except
for the requirement of piecewise differentiability. The neurons in ANFIS
have different structures.

• The Membership function is defined by parameterized soft

trapezoids (Generalized Bell Functions).

• The rules are differentiable T-norm usually product.

• The Normalization is by Sum and Arithmetic division.

• Functions are linear regressions and multiplication normalized

weights and Output (Algebraic Sum).

5.3 ANFIS EDITOR GUI

The ANFIS Editor GUI menu bar can be used to load a FIS
training initialization, save the trained FIS, open a new Sugeno system or
any of the other GUIs to interpret the trained FIS model. Any data set is
loaded into the ANFIS Editor GUI, (or that is applied to the command-
line function ANFIS) must be a matrix with the input data arranged as
vectors in all but the last column. The output data must be in the last
column. A sample of ANFIS Editor GUI with input is shown in Figure
5.14.
 64

Figure 5.14 ANFIS Editor GUI

5.4 FIS EDITOR

The FIS Editor displays general information about a fuzzy

inference system. There is a simple diagram at the top that shows the
names of each input variable on the left, and those of each output variable
on the right. The sample membership functions shown in the boxes are
just icons and do not depict the actual shapes of the membership
functions. This is shown in Figure 5.15.
 65

Figure 5.15 FIS Editor

5.5 MEMBERSHIP FUNCTIONS EDITOR

The Membership Function Editor shares some features with the

FIS Editor. In fact, all of the five basic GUI tools have similar menu
options, status lines, and Help and Close buttons. The Membership
Function Editor is the tool that lets you display and edits all of the
membership functions associated with all of the input and output
variables for the entire fuzzy inference system. Figure 5.16 shows the
membership function editor.
 66

Figure 5.16 Membership Function Editor

5.6 RULE EDITOR

The Rule Editor allows user to construct the rule statements

automatically, by clicking on and selecting one item in each input
variable box, one item in each output box, and one connection item as in
Figure 5.17. Choosing none as one of the variable qualities will exclude
that variable from a given rule. Choosing not under any variable name
will negate the associated quality. Rules may be changed, deleted, or
added, by clicking the appropriate button.
 67

Figure 5.17 Rule Editor

5.7 RULE VIEWER

The Rule Viewer as shown in Figure 5.18 allows users to

interpret the entire fuzzy inference process at once. The Rule Viewer also
shows how the shape of certain membership functions influences the
overall result. Since it plots every part of every rule, it can become
unwieldy for particularly large systems, but, for a relatively small number
of inputs and outputs, it performs well (depending on how much screen
space we devote to it) with up to 30 rules and as many as 6 or 7 variables.
The Rule Viewer shows one calculation at a time and in great detail. In
this sense, it presents a sort of micro view of the fuzzy inference system.
 68

Figure 5.18 Rule Viewer

5.8 ANFIS MODEL STRUCTURE

After the FIS is generated, the model structure can be viewed

by clicking the Structure button in the middle of the right side of the GUI.
A GUI can be seen in Figure 5.19. The branches in this graph are color
coded. Color coding of branches characterize the rules and indicate
whether or not and, not, or or are used in the rules. The input is
represented by the left-most node and the output by the right-most node.
The node represents a normalization factor for the rules. Clicking on the
nodes indicates information about the structure.
 69

Figure 5.19 ANFIS Model Structure

5.9 MODELING DATA THROUGH ANFIS

The modeling approach used by ANFIS is similar to many

system identification techniques. First, a parameterized model structure
(relating inputs to membership functions to rules to outputs to
membership functions, and so on) is hypothesized. Next, input/output data
is collected in a form that will be usable by ANFIS for training. ANFIS
can then be used to train the FIS model to emulate the training data
presented to it by modifying the membership function parameters
according to a chosen error criterion. Figures 5.20 and 5.21 shows the
example training error in ANFIS and output of the ANFIS.
 70

In general, this type of modeling works well if the training data

presented to ANFIS for training (estimating) membership function
parameters is fully representative of the features of the data that the
trained FIS intend to model. This is not always the case, however. In
some cases, data is collected using noisy measurements, and the training
data cannot be representative of all the features of the data that will be
presented to the model. This is where model validation comes into play.
[Jang, 1993 [29]].

Figure 5.20 Training Error

 71

Figure 5.21 Output of ANFIS

5.10 MODEL VALIDATION (CHECKING AND TESTING DATA)

Model validation is the process by which the input vectors from

input/output data sets on which the FIS was not trained, are presented to
the trained FIS model, to see how well the FIS model predicts the
corresponding data set output values. This is accomplished with the
ANFIS Editor GUI using the so-called testing data set, and its use is
described in a subsection that follows.

Another type of data set can also be used for model validation
in ANFIS. This type of validation data set is referred to as the checking
 72

data set and it set is used to control the potential for the model over fitting
the data. When the checking and training data are presented to ANFIS, the
FIS model having parameters associated with the minimum checking data
model error is then selected.

The problem with model validation for models constructed

using adaptive techniques is selecting a data set that is both representative
of the data the trained model intends to emulate, yet sufficiently distinct
from the training data set so as not to render the validation process trivial.
If a large amount of data has been collected, hopefully this data contains
all the necessary representative features, so the process of selecting a data
set for checking or testing purposes is made easier.

5.11 SUMMARY

In this chapter various architectures of neuro fuzzy system is

explained in detail and also the modeling and software implementation of
ANFIS architecture was explained in detail.