ARTIFICIAL VIS BIOLOGICAL NEURONS

In this lecture we will understand how the idea of

neural networks is inspired from working of human brain
We will also talk about similarities bw a biological
neweon and an artificial neuron

Neuron is the basic unit of a neural network we will be

building a network which will do various tasks like

predicting language language modelling classifying images

Google Allo Alena all use deep learning which is based
on neural network

firstly lets understand how a neural network is related to

a biological neuron

Dendrites

it
Eg
ma
p you

Biologicalluron
In brain you hamefollection of neurons and each neuron

looks like this and you have dendrites which brings

input from other neurons to current neweon Lets
say
each dendrite is x1 v3 and 24
bringing input x2

This input goes to the processing unit called as Soma

here neuron does some processing of the information

In synaptic gap which basically modifies
b w there is

the signals signals are basically in the form of

electrical and chemical impulses Lets say the original
input was no but the
input which goes to Soma will
be wini i.e weighted input Soma will calculate sum
of
all inputs 2 win
Some neurons will fire and some won't
firing of
neurons depends on activation function when 2 Ewin
threshold value then we can
say neuron will fire
2 Wini
a
a
gets
a threshold fx which is
also called as activation fx

If 2 is greater than some threshold then neuron is

going to fire otherwise neuron will not fire In practice

g can be a sigmoid far but nowadays we use

very
interesting fx called as Relu Rectified Linear Unit
814 20
gas neuron fin
g _5 O
2
1
Neuron Doesn't five

we should add a bias term to 2 2 Wini b

want that should when the
If we a neuron
fire
value is 10 we will keep bias as 10

2 Wini b

Even if Wini is 10 still neuron will fire

activation fan

i g
9 86
activation fun

Artificial Neuron

Wi is the weight that we will learn

using algo like
Gradient Descent This is very similar to Logistic Regression

if your activation fa is a sigmoid fun

and Neurons
Resemblance bw Biological Artificial
synaptic Gap weights
Axon output
Soma Activation
 HOW DOES AN ARTIFICIAL NEURON LEARN

In this
lecture we will see what is a Perceptron
Perceptron is a single layer neural network We will
about the
be learning loss function and later on we
will derive weight update rule

w no win when

if
a
mme w

Activation
fan

I
Ilnsin

oca.tt
OpisFany no bw o and I

let us say we are

doing a classification problems in
to
which want predict if given age is of dog or cat

g
we

features pixel values

Go 1
Fff o or 1
if lies in range 0 and 1

win wine W 00mn00 2 o a o or 1

If I 0 8 1 Round off to 1 if tells which class
a given input belongs
j 0 3 0 Round off to 0
to

Perceptron will act as a linear classifier It can do Binary

classification

Basic Neural Network

with a mode
only

Goal of Learning Algorithm is to learn parameters w if

you get data of type 1 then output should be close to

and if you get data of type 0 then output should be

close to 0

If input is very large lets say 2 100 then o 2 I

2 200 then 2 0

Activation fr is compressing your number line in the range

0 to 1

2 ID
for
012

2 tells the probability a given input belongs to which class

Lets say 2 0.7 you are 70 confident that it

belongs to class 1 you are

307 confident that it belongs to
class 0

Ply 1 z

P y O 1 012
Z W x

This is how make

we can predictions using a simple
perceptron

How to train a perceptron

In every ML Algo we do the following

Model single Node of Perceptron when we combine

multiple such nodes a neural network

we
get
which is called as multi layer perceptron
Loss

Algo predicts probability

with what probability it
belongs to class Dog
Ply 1 J 0.7 0.8 0.8 0.4

with 0.3
probability it
belongs to class
Dog it belongs
to class cat with
probability 0.7
y 1 1 1 0 0
 MSE j yes
w

Problem with this kind

of
y o z o W x
loss function is it is a

non convex function

j is a
function of w

JCW

feetIf you start from this

L

point and use gradient

descent then you will be
stuck in local minima
There are multiple local
minimas and you will
stuck inside it

thisfficulty
To overcome we use another
kind of loss called as
log loss

Log Loss Binary Cross Entropy

Proof for this is already covered in Maximum

Likelihood Estimation for Logistic Regression

y ng j r
y log r j
It is a

offkey local and global minima

 JCW
if y o

then loss
for log r
j
if y
i
note freedrichtonstheane
we add
yn
then loss
fan log log of misclassification
i
if y O and j O

then loss for 0 that means actual values matches

when
with predicted value then you are
adding 0 to loss

if y I and j
i
I
then loss 0
fx

Weight update Rule

II iv
Start with some random w and w as
update
we
Gradient update Rule

JCW y log j r y log i j

 T.IE
j
Tiina J
w is

is
a

a
fan of Jci
fan of wTx i
 GRADIENT DESCENT UPDATE
In the last lecture we about loss function to
talked
train a perceptron and how perceptron acts as a
binary
classifier In this video our aim is to derive weight
update rule that will help us to find optimal set of
parameters w for the perceptron and we will use gradient
descent update rule to update the parameters

w Em y log j 1 y log i
g's
FPunction

goaeistolearmw

FEItndn.tt mÉtthEann

at some
f whd
Ñdom

up optimal
This can be done by using
Gradient update rule which
repeatedly decreases our loss in the w w
direction of reducing gradient

W wn
wo W wa n no
of features

C1
chain rule
Eg
for
amke f FF send east rs age 5
notifier
Ti.IE c
 j 06 2

E E 5 012

j o

8 z r o z

j o z i 2

j j r j

2 Wtx
z No Wise win Whan

ni
i

f JE
Ey t LED a

f f
 j y xi

feature for
ith
win
Offit y ni
given a example

wj Wj y É j yes af
7
update the jth
gradient

Single Perceptron sigmoid Logistic Regression

PERCEPTRON IMPLEMENTATION
In this video we will implement a perceptron which is a
network It linear
single layer neural acts as a

classifier We will use binary cross entropy as a loss

function and gradient descent optimizer

Learning Goals

How to implement Perceptron

We won't use any for loop we will see how
vectorization in
python works

what happens when you use non linear dataset and a

linear classifier like perceptron

CODE

import numpy as up
import matplotlib pyplot as felt
from sklearn datasets import make_blobs

Generating Data
x Y make_blobs
I returns gaussian blobs

X Y make_blobs n samples 500 centers 2 n features 2

random state 10

print x shape 4 Shape

2500 2 500
pet style use seaborn

pet scatter 0 1 C Y
chap pet.cm Accent
pet show

it is linearly
separable data
wÉ Goal of perception

v learning algo
1
is to figure out
boundary which
separates data
into 2 classes

Model and Helper functions

def sigmoid z

return 1.0 it upexp z

2
up array 1,2 3 4,5
sigmoid 2
7
you will get an array
This kind of functionality is called broadcasting

sigmoid fun is now applied on

every element
of array

1
It is possible only in numpy array
Numpy does it b.cz ofa technique called
as broadcasting

Implement Perceptron Learning Algorithm

Learn the weights
Reduce the loss

Make the predictions w wow

why

def predict x weights

2
up dot
predictions
x weights

sigmoid z
g I
return
Effort
predictions
 mx nti
matrix

Wnx vector

def loss x Y weights

log
Binary Cross Entropy
Y
i yii ogi y
predict x weights
cost up mean C Y uplog x 1 7
up log 1 Y
D
return cost

def update X Y weights learning rate

Perform weight updates for 1 epoch

Wj Wj 7
Y predict x weights j
a y
np.at y
y.p
g
m Shape 0
weights weights learning rate du float m

return weights

EEE I
MXI

y y

E
ji_
µ11
nxm mxl
 j y x

def train x Y
learning rate 0.5 max Epochs 100

Modify input to handle bias term x x2 n

x
ones
up ones shape 0 s
Jmx
up hstack Cones

b
F Init weights
f
o 9,1494m

weights up zeros x shape I

2 n

Iterate over all epochs and make

xp
updates

for epoch in range max Epochs mx nm

weights update x Y weights If seiwi

learning rate

if epoch7.10 0
after every
10 epochs l loss x Y weights
the
print
progress print Epoch 7d loss To 4f 7 epoch e

return weights

train x Y
 is Reducing
Joss
weights learnt by your
i classifier for n features
we will have n I
T T T weights
weights train x Y max Epochs 500

do oh
ÉdwhÉg

Loss is close to 0
VISUALISING DECISION SURFACE
We are
building a simple neural network by training
one perceptron We will see how to make predictions
all data and also visualise the
for points hypothesis
generated by our
algo We will run this perceptron over
a non linear dataset and will see the results

Perceptron Implementation Part 11

Make Predictions
visualise Decision Surface
Linear vs Non linear Classification

def getPredictions X Test weights labels True

it test data
if x Test Shape 1 weights shape o doesnot have
column of 1s
ones up ones x Test Shape 0 1 appended then
we are going
x Test up hstack ones X Test to add 0

Test weights mxn net

probs predict x

wo.co won
if not labels
return frobs W

we Mds shake
Pires

output of algo is
return labels to be a
going

fffisifY
All the places where I
 a up zeros 55

a 2 3 10

print a

a a 0 20

print a

weights

feet Scatter X 0 I C Y
map feet cm Accent
flt show
 No Ws wz
000

two
fed
see

If we plot 22 V S
this
x2 using
formula we will
get the line

8 2,10
x1
up linspace
x2 weights 0 weights I xD weights 2

pet Scatter X 0 1 C Y
map pet ons Accent
pet plot x1 x2 C red

pet show C
 We are able to
separate the data

Lets
try with different value of random state

X Y make_blobs n samples 500 centers 2 n features 2

random state 1
 find the accuracy
Y getPredictions X weights labels True

print Y

comparison

fruit
 500

7
Training Accuracy is 1007
All predictions are correct

what if we use non linear data

Now not able to

Wyway
learn a
very good
boundary bcz data

my yummy
some other hypothesis
which should be
able to separate
data like this
This kind of boundary
can be learnt by
MLP Multi Layer
Perceptron

now
accuracy is only 86.8
Some points are not
classified
correctly