Adaptive Networks
Presentation by:
C. Vinoth Kumar
SSN College of Engineering
Introduction
Neural network is the information processing paradigm
inspired by biological nervous systems.
It is the large number of highly interconnected
processing elements (neurons) working together.
Neural networks are configured for a specific application,
such as pattern recognition or data classification,
through a learning process (inductive learning).
In a biological system, learning involves adjustments to
the synaptic connections between neurons and it is
same for artificial neural networks (ANNs).
Adaptive Networks
An adaptive network is a network structure consisting of
a number of nodes connected through directional links.
Each node represents a process unit, and the links
between nodes specify the causal relationship between
the connected nodes.
The learning rule specifies how these parameters
should be updated to minimize the error.
The basic learning rule is the well-known steepest
descent method.
Adaptive Networks
An Adaptive Network is a network structure whose
overall input-output behaviour is determined by a
collection of modifiable parameters.
Each node performs a static node function on its
incoming signals to generate a single node output.
A node function is a parameterized function with
modifiable parameters.
Each link specifies the direction of flow from one node
to another.
The Neuron
Bias
b
x1 w1
Activation
Local function
Field
Output
Input
signal
x2 w2 ∑ v ϕ(−) y
M M Summing
function
xm wm
Synaptic
weights
A Feed Forward Adaptive Network
- layered representation
Adaptive Networks
Adaptive Node: If a node’s parameter set is not empty,
then its node function depends on parameter values;
Square is used to represent adaptive node.
Fixed Node: If a node has an empty parameter set, then
its function is fixed; Circle is used to represent fixed
node.
x
x f y f y
a
y=f(x,a) y=f(x,a)
Adaptive Networks
– Parameter sharing
x u
g x g u
a a
y v y v
h h
Adaptive Network
A linear single node adaptive network
x1
f3 x3 x3 = f3(x1, x2; a1, a2, a3)
= a1x1 + a2x2 + a3
x2
Perceptron network
x1 x3 = f3(x1, x2; a1, a2, a3)
= a1x1 + a2x2 + a3
f3 x f4 x4
3
x2 x4 = f4(x3)
Simple Single Unit Adaptive Networks
The network has 2 inputs, and one
output. All are binary. The output is
1 if W0I0 + W1I1 + Wb > 0
0 if W0I0 + W1I1 + Wb ≤ 0
We want it to learn simple OR:
output 1 if either I0 or I1 is 1
Change the weight by an amount
proportional to the difference
between the desired output and the
actual output
∆ Wi = α * (D-Y) * Ii
Adaptive Network
A Multilayer Perceptron
Adaptive Feed Forward Networks
Feed Forward Networks -
• Information flow is unidirectional
• Data is presented to Input layer
• Passed on to Hidden Layer
• Passed on to Output layer
• Information is Distributed
• Good at extracting patterns,
generalisation and prediction
• Parallel processing of data
• Training: Back propagation
Backpropagation for Feed Forward Networks
The procedure of finding the gradient vector(derivative of
an error) in a network structure is generally referred as
backpropagation because the gradient vector is
calculated in the direction opposite to the flow of the
output of each node.
Once the gradient is obtained, a number of derivative-
based optimization and regression techniques are
available for updating the parameters.
If the gradient vector uses simple steepest descent
method, the resulting learning procedure is known as
backpropagation learning rule.
Backpropagation for Feed Forward Networks
Backpropagation for Feed Forward Networks
To use steepest descent to minimize the error, first the
gradient vector is obtained.
The learning rate (α) has the effect in both outputs of
nodes, outputs of network and error.
Therefore, the derivative gradient vector is passed
starting from output layer and going backward layer by
layer until the input layer is reached
Backpropagation for Feed Forward Networks
Backpropagation for Feed Forward Networks
Backpropagation for Recurrent Networks
For recurrent adaptive networks, it is possible to derive
an extended version of the backpropagation procedure
that finds gradient vectors.
