MODULE -1

Introduction to Soft Computing. Difference between Hard Computing & Soft Computing. Applications of
Soft Computing. Artificial Neurons Vs Biological Neurons. Basic models of artificial neural networks –
Connections, Learning, Activation Functions. McCulloch and Pitts Neuron. Hebb network.

Introduction
● There are main differences between soft computing and possibility. Possibility is used when we don't
have enough information to solve a problem but soft computing is used when we don't have enough
information about the problem itself.
● These kinds of problems originate in the human mind with all its doubts, subjectivity and emotions; an
example can be determining a suitable temperature for a room to make people feel comfortable.

SOFT COMPUTING
● Soft computing is a relatively new concept. The ultimate goal of soft computing is to be able to emulate
the human mind as closely as possible. Soft Computing involves partnership of several fields, the most
important being neural networks, GA and FL.
● Soft computing uses a combination of GAs, neural networks and FL. Neural networks are
Important for their ability to adapt and learn, FL for its exploitation of partial truth and imprecision, and
GAs for their application to optimization.
● In addition, it has to generate a fuzzy knowledge base, which has a linguistic representation and a very
low degree of computational complexity.
● An important thing about the constituents of soft computing is that they are complementary, not
competitive, offering their own advantages and techniques to partnerships to allow solutions to
otherwise unsolvable problems.

The two major problem solving technologies:

Soft computing
● Soft computing differs from conventional (hard) computing in that, unlike hard computing, it is
tolerant of imprecision, uncertainty, partial truth, and approximation. In effect, the role model for
soft computing is the human mind.
● Soft computing deals with approximate models and gives solution to complex problems.

Hard Computing
 ● Hard computing deals with precise models where accurate solutions are achieved quickly.
● Many analytical models are valid for ideal cases.
● Real world problems exist in a non-ideal environment.

Differentiate between Hard computing and Soft Computing

.NO Soft Computing Hard Computing

Soft Computing is liberal of inexactness, Hard computing needs a exactly state

1.
uncertainty, partial truth and approximation. analytic model.

Soft Computing relies on formal logic and Hard computing relies on binary logic and
2.
probabilistic reasoning. crisp system.

Soft computing has the features of Hard computing has the features of
3.
approximation and dispositionality. exactitude(precision) and categoricity.

4. Soft computing is stochastic in nature. Hard computing is deterministic in nature.

Soft computing works on ambiguous and noisy

5. Hard computing works on exact data.
data.

Soft computing can perform parallel Hard computing performs sequential

6.
computations. computations.

7. Soft computing produces approximate results. Hard computing produces precise results.

Hard computing requires programs to be

8. Soft computing will emerge its own programs.
written.

9. Soft computing incorporates randomness . Hard computing is settled.

10. Soft computing will use multivalued logic. Hard computing uses two-valued logic.
 NEURAL NETWORKS
● Neuron:
- The cell that performs information processing in the brain.
- Fundamental functional unit of all nervous system tissue.
● The computing world has a lot to gain from neural networks whose ability to learn by example makes
them very flexible and powerful.
● In case of neural networks, there is no need to devise an algorithm to perform a specific task, i.e., there
is no need to understand the internal mechanisms of that task.
● Neural networks are also well suited for real-time systems because of their fast response and
computational times, which are due to their parallel architecture.
● Neural networks also contribute to other areas of research such as neurology and psychology. They are
regularly used to model parts of living organisms and to investigate the internal mechanisms of the
brain.
● Neural networks are a new method of programming computers.
● They are exceptionally good at performing pattern recognition and other tasks that are very difficult to
program using conventional techniques.
● Programs that employ neural nets are also capable of learning on their own and adapting to changing
conditions.
● A neural network is a massively parallel distributed processor that has a natural propensity for storing
experiential knowledge and making it available for use. It resembles the brain in two respects: –
Knowledge is acquired by the network through a learning process. – Interneuron connection strengths
known as synaptic weights are used to store the knowledge
● A neural network is a circuit composed of a very large number of simple processing elements that are
neurally based. Each element operates only on local information. Furthermore each element operates
asynchronously; thus there is no overall system clock.
● In neuro science and computer science, synaptic weight refers to the strength or amplitude of a
connection between two nodes, corresponding in biology to the amount of influence the firing of one
neuron has on another. The term is typically used in artificial and biological neural network research.
● In information technology, a neural network is a system of hardware and/or software patterned after the
operation of neurons in the human brain. ... Commercial applications of these technologies generally
focus on solving complex signal processing or pattern recognition problems.
 NEURAL NETWORKS APPLICATIONS

● Photos and fingerprints could be recognized by imposing a fine grid over the photo. Each square of the
grid becomes an input to the neural network.
● River water levels could be predicted based on upstream reports, and rime and location of each report.
● Scheduling of buses, airplanes and elevators could be optimized by predicting demand.
● Staff scheduling requirements for restaurants, retail stores, police stations, banks, etc., could be
predicted based on the customer flow, day of week, paydays, holidays, weather, season, ere.
● Traffic flows could be predicted so that signal timing could be optimized. The neural network could
recognize "a weekday morning rush hour during a school holiday" or "a typical winter Sunday
morning."
● Fraud detection regarding credit cards, insurance or taxes could be automated using a neural network
analysis of past incidents.
● Handwriting and typewriting could be recognized by imposing a grid over the writing, and then each
square of the grid becomes an input to the neural network. This is called "Optical Character
Recognition.

BIOLOGICAL NEURAL NETWORK

● The human brain is composed of 100 billion nerve cells called neurons.

● They are connected to other thousand cells by Axons.

● Stimuli from external environment or inputs from sensory organs are accepted by dendrites.

● These inputs create electric impulses, which quickly travel through the neural network.

● A neuron can then send the message to other neuron to handle the issue or does not send it
forward.
 a. Soma or cell body : where the cell nucleus is located.
i. The neuron’s cell body (soma) processes the incoming activations
and converts them into output activations.
b. Synapse : Gap between neurons is called a synapse.
● Through synapse that e neuron introduces its signals to other
nearby neuron.
c. Dendrites : where the nerve is connected to the Soma or cell body.
d. Axons : Axons are fibers acting as transmission lines that send activation to
other neurons.

TERMINOLOGY RELATIONSHIPS BETWEEN BIOLOGICAL AND

ARTIFICIAL NEURONS:

Biological Neuron Artificial Neuron

Dendrites weights or interconnections
Cell body Neuron/Processor
Synaptic Link
Axon Output
Soma Net input

ARTIFICIAL NEURAL NETWORK

● An artificial neural network (ANN) is an efficient information processing system which resembles in
characteristics with a biological neural network.
● Information that flows through the network affects the structure of the ANN because a neural network
changes - or learns, in a sense - based on that input and output.
● ANNs' collective behavior is characterized by their ability to learn, recall and generalize training
patterns or data similar to that of a human brain. They have the capability to model networks of original
neurons as found in the brain. Thus, the ANN processing elements are called neurons or artificial
neurons.
● ANNs are composed of multiple nodes, which imitate biological neurons of human brain.
● The neurons are connected by links and they interact with each other.
● Each connection link is associated with weights which contain information about the input signal.
● The nodes can take input data and perform simple operations on the data.
● The result of these operations is passed to other neurons.
● The output at each node is called its activation or node value.
● A neuron can send only one signal at a time, which can be transmitted to several other neurons.

● Consider a set of neurons, say X1 and X2, transmitting signals to another neuron, Y.

● Here X1, and X2 are input neurons, which transmit signals, and Y is the output neuron, which receives
signals.

● Input neurons X1, and X2 are connected to the output neuron Y, over a weighted interconnection links
(W1 and W2) as shown in Figure.

● For the above simple neuron net architecture, the net input has to be calculated in the following way:

yin= +XIWI +X2W2+ +XnWn

Input Signals Weights Output Signals

x1 x2

Y
xn
● Where X1 and X2 are the activations of the input neurons X1 and X2, i.e., the output of input signals.
● The output y of the output neuron Y can be obtained by applying activations over the net input, i.e., the
function of the net input:
Y=f(yin)

Output= Function (net input calculated)

● The function to be applied over the net input is called activation function.

BIOLOGICAL NEURON AND ARTIFICIAL NEURON

(BRAIN VS COMPUTER)

Biological Neuron
● The human brain is composed of 100 billion nerve cells called neurons.
● The Brain is A massively parallel information processing system.
Artificial Neuron
● An artificial neuron is a mathematical function conceived as a model of biological neurons, a neural
network.
● Artificial neurons are elementary units in an artificial neural network.
● First model developed by McCulloch and Pitts in 1943

ANN possesses the following characteristics:

1. It is a neurally implemented mathematical model.
2. There exit a large no of highly interconnected processing elements called neuron in an ANN.
3. The interconnections with their weighted linkages hold the informative knowledge.
4. The input signals arrive at the processing elements through connections and connecting weights.
5. The processing elements of the ANN have the ability to learn, recall and generalize from the given
data by suitable assignment or adjustment of weights.
6. The computational power can be demonstrated only by the collective behavior of neurons, and it
should be noted that no single neuron carries specific information.
Terminologies Brain(Biological neuron) Computer(Artificial Neuron)

Speed Milliseconds Nanoseconds.

Artificial neuron modeled using a
computer is faster.
the biological neuron can perform The artificial neuron can also perform
Processing massive parallel several parallel operations
operations simultaneously simultaneously

Size 10 billion neurons and 60 the size and complexity of a an artificial

and complexity trillion synapses. Hence, complexity neuron is less than that Of
of the biological neuron
brain is comparatively higher.
Storage ● The biological neuron stores the ● It is stored in its contiguous
capacity information in its memory locations.
(memory) interconnections or in synapse ● In an artificial neuron, once the
strength. information is stored in its memory
● Sometimes its memory may fail locations, it can be retrieved.
to recollect the stored ● the adaptability is more toward an
information. artificial neuron

Tolerance ● The biological neuron assesses ● The artificial neuron has no fault
fault tolerant capability tolerance.
● The distributed nature of the ● The information gets corrupted if
biological neurons enables to the network interconnections are
store and retrieve information disconnected.
even when the interconnections in
them get disconnected.

Control ● no control unit for monitoring in ● Central Processing Unit

mechanism the brain ● the control mechanism of an
artificial neuron is very simple
compared to that of a biological
neuron.
 BASIC MODELS OF ARTIFICIAL NEURAL NETWORKS

The models of ANN are specified by the three basic entities namely:

1. The model's synaptic interconnections;

2. The training or learning rules adopted for updating and adjusting the connection weights;

3. Their activation functions.

I. CONNECTIONS

– An ANN consists of a set of highly interconnected processing elements (neurons) such that each
processing element output is found to be connected through weights to the other processing elements or
to itself.
– The arrangements of these processing elements and geometry of their interconnections are essential for
an ANN.
– The point where the connection originates and terminates should be noted, and the function of each
processing element in an ANN should be specified.
– The arrangement of neurons to form layers and the connection pattern formed within and between layers
is called the network architecture.
– A layer is formed by taking a processing element and combining it with other processing elements.
– Neural nets are classified into single-layer or multilayer neural nets.
FIVE TYPES OF NEURON CONNECTION ARCHITECTURES.
1. Single-layer feed-forward network
2. Multilayer feed-forward network
3. Single node with its own feedback
4. single-layer recurrent network
5. Multilayer recurrent network
Feed forward neural network:
● A feed forward neural network is an artificial neural network wherein connections between the units do not
form a cycle.
● The feed forward neural network was the first and simplest type of artificial neural network devised.
● The information always flows in a single direction (thus, unidirectional), which is from the input layer to the
output layer.
● In this network, the information moves in only one direction, forward, from the input nodes, through the
hidden nodes (if any) and to the output nodes.

● There are no cycles or loops in the network and no connection among neurons in the same layer.
1. Single-layer feed-forward network:
● In this single-layer feed forward neural network, the network’s inputs are directly connected to
the output layer neurons.
● The output neurons use activation functions to produce the outputs Y1 and Y2.
● The simplest kind of neural network is a single-layer perceptron network, which consists of a
single layer of output nodes; the inputs are fed directly to the outputs via a series of weights

2. Multilayer feed-forward network:

● A multilayer feed-forward network is formed by the interconnection of several layers. The input
layer is that which receives the input and this layer has no function except buffering the input
signal.
● Neurons are arranged in layers, with the first layer taking in inputs and the last layer producing
outputs.
● The middle layers have no connection with the external world, and hence are called hidden layers.
● It should be noted that there may be zero to several hidden layers in an ANN.
● More the number of the hidden layers, more is the complexity of network.
● Fully connected network: every output from one layer is connected to each and every node in the
next layer.
Feedback networks:
When outputs can be directed back as inputs to same or preceding layer nodes then it results in the
formation of feedback networks. .
Lateral feedback:

● If the feedback of the output of the processing elements is directed back as inputs to the processing
elements in the same layer then it is called lateral feedback.
● There exist couplings of neurons within one layer
● There is no essentially explicit feedback path amongst the different layers

Single node with its own feedback

Figure shows a simple recurrent neural network having a single neuron with feedback to itself.
Recurrent network:
● Recurrent networks are feedback networks with closed loop.
● In these networks, the outputs of the neurons are used as feedback inputs for other neurons.
● A recurrent neural network (RNN) is a class of artificial neural network where connections
between units form a directed cycle.
● The human brain is a recurrent neural network (RNN): a network of neurons with feedback
connections.
Single-layer recurrent network:
● Figure Shows a single· layer network with a feedback connection in which a processing element's output
can be directed back to the processing element itself or to the other processing element or to both.

Multilayer recurrent network:

● A processing element output can be directed back to the nodes in a preceding layer, forming a multilayer
recurrent network.
● Also, in these networks, a processing element output can be directed back to the processing element
itself and to other processing elements in the same layer.
 Lateral Inhibition Structure

● Another type of architecture with lateral feedback, which is called the on-center-off-surround or lateral
inhibition structure.
● In this structure, each processing neuron receives two different classes of inputs- "excitatory" input from
nearby processing elements and "inhibitory" inputs from more distantly located processing elements.
● In Figure, the connections with open circles are excitatory connections and the links with solid
connective circles are inhibitory connections.

II. Learning

● The main property of an ANN is its capability to learn.

● There are two kinds of learning in ANNs:

1. Parameter learning: updates the connecting weights in a neural net.

2. Structure learning: It focuses on the change in network structure (which includes the number of processing
elements as well as their connection types).

● The above two types of learning can be performed simultaneously or separately. Apart from these
two categories of learning, there are three broad types of learning:
 Supervised Learning (i.e. learning with a teacher)

o Supervised learning means learning with the help of a teacher

o It requires a training pair which consists of input vectors and a target vector associated with each input
vector.
o The input vector along with the target vector is called training pair

Unsupervised learning (i.e. learning with no help)

o Unsupervised learning means learning without the help of a teacher

o Unsupervised learning does not require any knowledge of the respective desired outputs.
o The system learns about the pattern from the data itself without a priori knowledge.

Reinforcement learning (i.e. learning with limited feedback)

● This learning process is similar to supervised learning.

● In the case of supervised learning, the correct target output values are known for each input
pattern.
● But, in some cases, less information might be available.
● For example, the network might be told chat its actual output is only "50% correct" or so. T
● hus, here only critic information is available, not the exact information.
● The learning based on this critic information is called reinforcement learning and the feedback
sent is called reinforcement signal
 III. Activation Functions

● To make work more efficient and for exact output, some force or activation is given.
● Like that, activation function is applied over the net input to calculate the output of an ANN.
● Information processing of processing elements has two major parts: input and output.
● An integration function (f) is associated with input of a processing element.
● The nonlinear activation function is used to ensure that a neuron’s response is bounded .
● When a signal is fed through a multilayer network with a linear activation function the output
obtained remains the same as that could be obtained using a single layer network.
● Due to this reason , nonlinear functions are widely used in multilayer networks compared to linear
function.
● Net input calculation is :
Yin= x1w1+x2w2
● Output is :
y = f(yin)
● Output= function (net input calculated)
● The function to be applied over the net input is called activation function

Different Activation Functions are:

A. Identity function:
It is a linear function which is defined as f(x) =x for all x
The output is the same as the input.
B. Binary step function:
it is defined as

where θ represents threshold value.

It is used in single layer nets to convert the net input to an output that is binary. ( 0 or 1).

C. Bipolar step function:

It is defined as

where θ represents the threshold value used in single layer nets to convert the net input to an output
that is bipolar (+1 or -1).
D. Sigmoid function
used in Backpropagation nets.
Two types:
a) binary sigmoid function
-logistic sigmoid function or unipolar sigmoid function.
-it is defined as
 where λ – steepness parameter.
The derivative of this function is
f ’(x) = λ f(x)[1-f(x)].
The range of sigmoid functions is 0 to 1.

b) Bipolar sigmoid function

where λ- steepness parameter and the sigmoid range is between -1 and +1.
The derivative of this function can be
F. Ramp function
 McCulloch-Pitts Neuron/ (M-P Neuron)

● M-P Neurons are connected by directed weighted paths.

● The activation of a McCulloch Pitts neuron is binary i.e neuron may or may not fire.
● The weights associated with the communication links may be
▪ excitatory (weight is positive)
▪ inhibitory (weight is negative).

● All excitatory connected weights entering a particular neuron will have the same weights.

ARCHITECTURE
● Threshold for each neuron is fixed
● If the net input to the neuron is greater than the threshold the neuron fires

● The M-P neuron has both excitatory and inhibitory connections.

● It is excitatory with weight (w > 0) or inhibitory with weight -p(p < 0).
● Inputs from Xi to Xn possess excitatory weighted connections and inputs from Xn+ 1 to Xn+m
possess inhibitory weighted interconnections.

Θ >nw-p
● Output will fire if it receives “k” or more excitatory input but no inhibitory input

kw>θ> (k-1)w

HEBB NETWORK

● According to the Hebb rule,

“ the weight vector is found to increase proportionately to the product of the input and the learning
signal.”
● Learning signal is equal to the neuron's output.
● In Hebb learning, if two interconnected neurons are 'on' simultaneously then the weights
associated with these neurons can be increased by the modification made in their synaptic gap
(strength).
● The weight update in Hebb rule is given by
wi(new) = wi(old) + xiy

● The Hebb rule is more suited for bipolar data than binary data.

Why not suitable for binary data ?

● If binary data is used, the weight updating formula cannot distinguish two conditions namely;
● A training pair in which an input unit is "on" and target value is "off."
● A training pair in which both the input unit and the target value are "off."
● Thus, there are limitations in Hebb rule application over binary data.
● Hence, the represention using bipolar data is advantageous.
●

Flowchart of Training Algorithm

● The training algorithm is used for the calculation and adjustment of weights.
● The flowchart for the training algorithm of Hebb network is given
▪ s- each training input
▪ t - target output pair.
● Till there exists a pair of training input and target output, the training process takes place; else,
It is stopped.
Training Algorithm
The training algorithm of Hebb network is given below:

Step 0: First initialize the weights. Basically in this network they may be set to zero, i.e., w = 0
for i= 1 to n where "n" may be the total number of input neurons. '
Step 1: Steps 2-4 have to be performed for each input training vector and target output pair, s: r
Step 2: Input units activations are set. Generally, the activation function of input layer is identity
function: xi=s; for i = 1 to n
Step 3: Output units activations are set: y = t
Step 4: Weight adjustments and bias adjustments are performed:
wi{new) = wi(old} + xiy
 b(new) = b(old) + y

Step 5 complete the algorithmic process.

In Step 4, the weight updating formula can also be given in vector form
as w(new)= w(old) +xy
Change in weight can be expressed as·
Δw = xy

w(new) = w(old) + Δw
● The Hebb rule can be used for pattern association, pattern categorization, pattern classification
and over a range of other areas.