Artificial Neural Networks

Part 2/3 – Perceptron

Slides modified from Neural Network Design

by Hagan, Demuth and Beale

Berrin Yanikoglu
 Perceptron
•  A single artificial neuron that computes its weighted input and
uses a threshold activation function.

•  It effectively separates the input space into two categories by

the hyperplane:
wTx + b = 0
Decision Boundary

The weight vector is orthogonal to the decision boundary

The weight vector points in the direction of the + class

(where an output of 1 is expected)
–  if w pointed in the opposite direction, the dot products of all input
vectors would have the opposite sign
–  would result in same classification but with opposite labels

The bias determines the position of the boundary

•  solve for wTp+b = 0 using one point on the decision boundary to
find b.
 Two-Input Case

+
a = hardlim(n) = [1 2]p + -2

w1 , 1 = 1 w1 , 2 = 2 -
Decision Boundary: all points p for which wTp + b =0
If we have the weights and not the bias, we can take a point on
the decision boundary, p=[2 0]T, and solving for [1 2]p + b = 0,
we see that b=-2.
p
w wT.p = ||w||||p||Cosθ
θ
proj. of p onto w
Decision Boundary
proj. of p onto w
T T = ||p||Cosθ
1w p + b = 0 1w p = -b
œb
= wT.p/||w||
• All points on the decision boundary have the same inner
product (= -b) with the weight vector
• Therefore they have the same projection onto the weight
vector; so they must lie on a line orthogonal to the weight
vector

ADVANCED
 An
Illustrative
Example
 Boolean OR

⎧ 0 , t = 0 ⎫ ⎧ 0 , t = 1 ⎫ ⎧ 1 ,t = ⎫ ⎧ 1 , t = 1 ⎫
p
⎨ 1 = 1 ⎬ p
⎨ 2 = 2 ⎬ ⎨ p3 = 3 1 ⎬ p
⎨ 4 = 4 ⎬
⎩ 0 ⎭ ⎩ 1 ⎭ ⎩ 0 ⎭ ⎩ 1 ⎭

Given the above input-output pairs (p,t), can you find (manually)
the weights of a perceptron to do the job?
 Boolean OR Solution

1) Pick an
admissable decision
boundary

2) Weight vector should be orthogonal to the decision boundary.

w = 0.5
1
0.5

3) Pick a point on the decision boundary to find the bias.

T 0 + b = 0.25 + b = 0
1 w p + b = 0.5 0.5 ⇒ b = œ0.25
0.5
Multiple-Neuron Perceptron: Matrix Form
weights of one neuron
in one row of W. w 1, 1 w 1, 2 … w 1, R
w w … w 2, R
W = 2, 1 2, 2
3x2
w S, 1 w S, 2 … w S, R

T
1w
w i, 1
i T
W = 2w w i, 2
iw =
2x1
T
Sw w i, R

ADVANCED
T
ai = har dlim(ni ) = hardlim( iw p + bi)
 Multiple-Neuron Perceptron

Each neuron will have its own decision boundary.

T
iw p + bi = 0

A single neuron can classify input vectors into

two categories.

An S-neuron perceptron can potentially

classify input vectors into 2S categories.
Perceptron Limitations
 Perceptron Limitations
•  A single layer perceptron can only learn linearly separable
problems.
–  Boolean AND function is linearly separable,
whereas Boolean XOR function is not.

Boolean AND Boolean XOR

Perceptron Limitations

Linear Decision Boundary

T
1w p + b = 0

Linearly Inseparable Problems

 AND Network

x1
W1=0.5

Σ
W2=0.5

x2 W0 = -0.8

X0=1
 Perceptron Limitations
For a linearly not-separable problem:
–  Would it help if we use more layers of neurons?
–  What could be the learning rule for each neuron?

Solution: Multilayer networks

and the backpropagation
learning algorithm
•  More than one layer of perceptrons (with a hardlimiting
activation function) can learn any Boolean function.

•  However, a learning algorithm for multi-layer perceptrons has

not been developed until much later
–  backpropagation algorithm
–  replacing the hardlimiter in the perceptron with a sigmoid
activation function
 Summary
•  So far we have seen how a single neuron with a threshold
activation function separates the input space into two.

•  We also talked about how more than one nodes may indicate
convex (open or closed) regions

•  The next slides = Backpropagation algorithm to learn the

weights automatically