Artificial Neural Network : Introduction

Credits:

Soft Computing Applications, Dr. Debasis Samanta, IIT Kharagpur

RC Chakraborty, www.myreaders.info
“Principles of Soft Computing” by S.N. Sivanandam & SN Deepa
Neural Networks, Fuzzy Logic and Genetic Algorithms by S. Rajasekaran and GAV Pai
Neural networks- a comprehensive foundation by Simon S Haykin
Biological nervous system

Biological nervous system is the most important part of many

living things, in particular, human beings.

brain is the center of human nervous system.

In fact, any biological nervous system consists of a large number

of interconnected processing units called neurons.

Each neuron is approximately 10µm long and they can operate in

parallel.

Typically, a human brain consists of approximately 1011 neurons

communicating with each other with the help of electrical
impulses.
Brain: Center of the nervous system
Computation
Neuron: Basic unit of nervous system

Basic structure of biological neuron

Neuron and its working
Neuron and its working
Figure shows a schematic of a biological neuron. There are different
parts in it : dendrite, soma, axon and synapse.

Dendrite : Extends away from the cell body and is the main input
to the neuron. It receives information from thousands of other
neurons. Receive information

Soma : It is also called a cell body, and just like as a nucleus of

cell. Process information
Axon : A long cylindrical fibre. Carries information to other neuron
Synapse : It is a junction where axon makes contact with the
dendrites of neighboring dendrites.
Nucleus: is located within the soma, contains genetic
information, directs protein synthesis, and supplies the energy
and the resources the neuron needs to function.
Neuron and its working
There is a chemical in each neuron called neurotransmitter.
A signal (also called sense) is transmitted across neurons by this
chemical.
That is, all inputs from other neuron arrive to a neurons through
dendrites.
These signals are accumulated at the synapse of the neuron and
then serve as the output to be transmitted through the neuron.
An action may produce an electrical impulse, which usually lasts
for about a millisecond.
Note that this pulse generated due to an incoming signal and all
signal may not produce pulses in axon unless it crosses a
threshold value.
Also, note that an action signal in axon of a neuron is commutative
signals arrive at dendrites which summed up at soma.
Definition of Neural Networks (NN)
According to the DARPA Neural Network Study (1988, AFCEA
International Press, p. 60):

• ... a neural network is a system composed of many simple

processing elements operating in parallel whose function is
determined by network structure, connection strengths, and the
processing performed at computing elements or nodes.

According to Haykin (1994), p. 2:

A neural network is a massively parallel distributed processor that has

a natural propensity for storing experiential knowledge and making it
available for use. It resembles the brain in two respects:
• Knowledge is acquired by the network through a learning
process.
• Interneuron connection strengths known as synaptic weights
are used to store the knowledge.
Artificial neural network

In fact, the human brain is a highly complex structure viewed as a

massive, highly interconnected network of simple processing
elements called neurons.

Artificial neural networks (ANNs) or simply we refer it as neural

network (NNs), which are simplified models (i.e. imitations) of the
biological nervous system, and obviously, therefore, have been
motivated by the kind of computing performed by the human brain.

The behavior of a biolgical neural network can be captured by a

simple model called artificial neural network.
Analogy between BNN and ANN

Dendrite
Soma
Axon
Synapse

Hippocampus: region of the brain that is associated primarily with memory.

Analogy between BNN and ANN

x1
w1

x2 w2

w3
x3
wn

xn
Artificial neural network

x1
w1

x2 w2

w3
x3
…..
wn

xn
Summation Threshold unit output
input weight
unit

Here, x1, x2, · · · ,xn are the n inputs to the artificial neuron.

w1, w2, · · · ,wn are weights attached to the input links.

Artificial neural network

Note that, a biological neuron receives all inputs through the

dendrites, sums them and produces an output if the sum is
greater than a threshold value.

The input signals are passed on to the cell body through the
synapse, which may accelerate or retard an arriving signal.

It is this acceleration or retardation of the input signals that is

modeled by the weights.

An effective synapse, which transmits a stronger signal will have a

correspondingly larger weights while a weak synapse will have
smaller weights.

Thus, weights here are multiplicative factors of the inputs to

account for the strength of the synapse.
Artificial neural network
Hence, the total input say I received by the soma of the artificial
neuron is

To generate the final output y , the sum is passed to a filter Ф

Called transfer function, which releases the output.
That is, y =Ф(I)

Summation Threshold unit output

input weight
unit
Artificial neural network

A very commonly known transfer function is the thresholding

function.

In this thresholding function, sum (i.e. I) is compared with a

threshold value θ.

If the value of I is greater than θ, then the output is 1 else it is 0

(this is just like a simple linear filter).
In other words,

where

Such a Φ is called step function(also known as Heaviside function).

Artificial neural network

Thresholding functions can be of two types: Binary (1, 0) or Bipolar [-1, 1].

output

+1.0

(I)
(I)

0 input -1.0

(a) Binary Threshold function (b) Bipolar Threshold function

Artificial neural network
Step vs Thresholding functions

Binary Step Binary Threshold

Bipolar Step Bipolar Threshold

Transformation functions

Binary transfer function : The output of the transfer function is

lies in the range of 0 to 1. It is generally used in perceptron
neuron.
In other words,
Binary Threshold

Bipolar transfer function : The output of the transfer function is

lies in the range of −1.0 to +1.0. It is also known Quantizer
function and it defined as

Bipolar Threshold
Transformation functions

Signum transfer function

In Bipolar Threshold function When , it is same as .

1 if weighted sum ( ) is positive.

0 if weighted sum ( ) is zero.

-1 if weighted sum ( ) is negative.

First use of the Activation functions

The first mathematical model of a biological neuron was presented

by McCulloch and Pitts. (1943).

Binary thresholding function were used in the model.

McCulloch-Pitts model does not exhibit any learning but just serves
as a basic building block for significant work in NN research.
 Question?

How to use Neural Network to learn pattern?

OSGN - OSPN Dr Dayal Kumar Behera, SCE, KIIT [22]

DU
 Simple Related Example
Let us predict the student X will pass the exam or not?

Output (y): Pass (1) Fail (0)

Inputs (x): features that influence or don’t influence the prediction

study hour (x1)

Name of the student (x2)
Previous Knowledge(x3)
Smartness (x4)

y = f(x) = (1) x1 + (0) x2 + (0.5) x3 + (0.2)x4

Given x1, x2, x3, x4 as input, what value shall we multiply, so that we can
get our desired output?

OSGN - OSPN Dr Dayal Kumar Behera, SCE, KIIT [23]

DU
 Simple Related Example

y = f(x) = (1) x1 + (0) x2 + (0.5) x3 + (0.2) x4

Weights

x1
1

x2 0
z y
f
x3 0.5

0.2
x3

Note# Weights give importance to the inputs.

OSGN - OSPN Dr Dayal Kumar Behera, SCE, KIIT [24]
DU
 Modelling the Brain

OSGN - OSPN Dr Dayal Kumar Behera, SCE, KIIT [25]

DU
Example
Input vector x = [1;2;5;8]

Weight vector w = [1; 1; -1; 2]

Find the output of the neuron if binary step function is used as

activation.

Ans: y = 1
Other transformation functions

Piecewise Linear: a connected sequence of line segments.

This is also called saturating linear function and can have
either binary or bipolar.
Other transformation functions

Piecewise Linear can have other forms.

Other transformation functions

Sigmoid transfer function : This function is a continuous

function that varies gradually between the asymptotic values 0
and 1 (called log-sigmoid) or -1 and +1 (called Tan-sigmoid)
function and is given by

Here, α is the coefficient of transfer function/ Slope parameter.

Other transformation functions

Unipolar Sigmoidal
Transfer functions in ANN

α=10
1

(I) α=0.3 1 α=10

α=1.0 α=0.5
(I) α=0.3
-1 1 -1 1
α=1.0 α=0.5
0

-1 -1

(a) Log-Sigmoid transfer function (b) Tan-Sigmoid transfer function

(Unipolar Sigmoidal) (Bipolar Sigmoidal)
0.2
Advantages of ANN

ANNs exhibits mapping capabilities, that is, they can map input
patterns to their associated output pattern.

The ANNs learn by examples. Thus, an ANN architecture can be

trained with known example of a problem before they are tested for
their inference capabilities on unknown instance of the problem. In
other words, they can identify new objects previously untrained.

The ANNs posses the capability to generalize. This is the power

to apply in application where exact mathematical model to
problem are not possible.
Advantages of ANN

The ANNs are robust system and fault tolerant. They can
therefore, recall full patterns from incomplete, partial or noisy
patterns.

The ANNS can process information in parallel, at high speed and

in a distributed manner. Thus a massively parallel distributed
processing system made up of highly interconnected (artificial)
neural computing elements having ability to learn and acquire
knowledge is possible.