UNIT-4

Q1. What is Gradient Descent Problem?Explain its types, also explain delta rule
and related algorithms.

Gradient Descent is an optimization algorithm used to minimize a function (often a loss/cost

function in machine learning) by iteratively moving in the direction of the steepest descent
(negative gradient).

The Gradient Descent problem generally refers to challenges or issues encountered when using
gradient descent to find the minimum of a function, such as:

 Getting stuck in local minima instead of the global minimum

 Slow convergence or taking too long to reach minimum
 Choosing a suitable learning rate
 Handling large datasets efficiently

Types of Gradient Descent

There are mainly three types of gradient descent methods, differing in how much data they use to
compute the gradient at each step:

1. Batch Gradient Descent

 Uses all training examples to compute the gradient at each step.

 Updates parameters after calculating the error for the entire dataset.
 Pros: Stable convergence because the gradient is exact.
 Cons: Slow and computationally expensive for large datasets.

2. Stochastic Gradient Descent (SGD)

 Uses only one training example at a time to update parameters.

 Parameters are updated more frequently, leading to noisy but faster updates.
 Pros: Faster, can escape local minima due to noise.
 Cons: Noisy updates can make convergence less stable.

3. Mini-batch Gradient Descent

 Uses a small batch (subset) of training examples (e.g., 32, 64) to compute gradients.
 Balances between batch and stochastic gradient descent.
 Pros: Faster than batch GD and less noisy than SGD, widely used in practice.
 Cons: Requires choosing a suitable batch size.
Delta Rule (Widrow-Hoff Rule)

The Delta Rule is a learning rule used for updating weights in simple neural networks like the
perceptron or single-layer networks. It’s a special case of gradient descent applied to minimize
the mean squared error (MSE) between predicted and actual outputs.

Formula:

Δwi = η(d−y)xi

Where:

 Δwi = change in weight i

 η = learning rate (step size)
 d = desired output
 y= actual output from the neuron
 xi = input corresponding to weight wi

Explanation:

 Weights are adjusted proportionally to the error (d−y) and the input value xi.
 The goal is to reduce the difference between actual and desired output iteratively.

Related Algorithms

1. Perceptron Learning Algorithm

o Updates weights only when the output is incorrect.
o Uses a step activation function (hard threshold).
o Simple but limited to linearly separable data.
2. Widrow-Hoff (Least Mean Squares, LMS) Algorithm
o Uses the Delta Rule for weight updates.
o Works with linear neurons and continuous outputs.
o Minimizes mean squared error.
3. Backpropagation Algorithm
o Generalizes the delta rule to multi-layer neural networks.
o Uses gradient descent to update weights in all layers by propagating the error
backward through the network.

Q2. Describe CNN Architecture. Specify the function of each layer.

A Convolutional Neural Network (CNN) is a type of deep learning model designed

specifically for processing data with a grid-like structure, such as images.
Key Characteristics of CNN:

 It uses convolutional layers to automatically extract spatial features like edges, textures,
and patterns from input data.
 CNNs are particularly effective in image recognition, classification, and computer
vision tasks.
 They reduce the number of parameters compared to fully connected networks by sharing
weights through filters (kernels).

CNN Architecture and Layer Functions

A typical CNN is made up of several types of layers stacked together to process and learn from
images (or similar grid-like data). The main layers are:

1. Input Layer

 Function: Takes the raw input image (e.g., 32x32x3 for a color image).
 It simply passes the image data to the next layer.
2. Convolutional Layer (Conv Layer)

 Function: Applies multiple filters (kernels) to the input image to detect various features
like edges, textures, and shapes.
 Each filter slides over the input image, performs element-wise multiplication and sums it
up, creating a feature map.
 It helps the network learn spatial hierarchies of features.

3. Activation Layer (e.g., ReLU)

 Function: Applies a non-linear activation function (most commonly ReLU: Rectified

Linear Unit) to introduce non-linearity into the network, allowing it to learn complex
patterns.
 It replaces negative values with zero, keeping positive values unchanged.

4. Pooling Layer (e.g., Max Pooling)

 Function: Reduces the spatial size of the feature maps, decreasing the number of
parameters and computation.
 Helps make the features more robust to small translations and distortions.
 Max pooling, for example, takes the maximum value from a small window (e.g., 2x2).

5. Fully Connected (FC) Layer

 Function: After several convolutional and pooling layers, the high-level features are
flattened into a 1D vector and fed into one or more fully connected layers.
 These layers work like a traditional neural network to combine features and classify the
input into output categories.

6. Output Layer

 Function: Produces the final prediction, often using a softmax activation for
classification tasks to give probabilities for each class.

Q3. Discuss Feed-Forward and Feed-Backward Neural Networks.

1. Feed-Forward Neural Networks (FFNN)

  Structure: Information flows in one direction — from input layer, through hidden
layers, to output layer.
 No cycles or loops in the network connections.
 Operation:
o Input data is passed forward through the layers.
o Each neuron applies weights, bias, and an activation function to produce output.
o The final layer produces the prediction or output.
 Training: Uses algorithms like backpropagation to adjust weights.
 Use cases: Classification, regression, image recognition, simple pattern recognition.

2. Feed-Backward Neural Networks (More accurately called Feedback or

Recurrent Neural Networks, RNNs)

 Structure: Contains loops or cycles — outputs from neurons can feed back into previous
layers or themselves.
 Operation:
o The network can use information from previous steps (memory) to influence
current outputs.
o This feedback enables the network to maintain a state or context over time.
 Training: Often trained using specialized algorithms like Backpropagation Through
Time (BPTT).
 Use cases: Sequence data like speech recognition, language modeling, time series
prediction.
Key Differences:

Aspect Feed-Forward NN Feed-Backward (Recurrent) NN

Data flow One-way, no cycles Has loops, cyclic connections
Memory No memory of past inputs Has memory, can use past information
Use cases Static data (images, tabular) Sequential/time-dependent data
Complexity Simpler More complex to train and analyze

Q4. Write notes on Virtual Assistant, Recommendation System, and

Personalized Health Care.

Virtual Assistant

 Definition: A software agent that can perform tasks or services for an individual based
on commands or questions.
 Examples: Siri, Alexa, Google Assistant, Cortana.
 Functions: Voice recognition, natural language processing, task automation, answering
queries, setting reminders, controlling smart devices.
 Benefits: Hands-free operation, improves productivity, provides quick access to
information, enhances user experience.
 Challenges: Understanding context, privacy concerns, handling complex commands.

Recommendation System

 Definition: A system that suggests products, services, or information to users based on

their preferences and behavior.
 Types:
o Content-based filtering: Recommends items similar to those the user liked
before.
o Collaborative filtering: Uses preferences of similar users to make
recommendations.
o Hybrid methods: Combine both approaches for better accuracy.
 Applications: E-commerce (Amazon), streaming services (Netflix), social media
(YouTube), online shopping, news feeds.
 Benefits: Personalizes user experience, increases user engagement, drives sales.
 Challenges: Cold start problem (new users/items), scalability, diversity in
recommendations.
Personalized Health Care

 Definition: Tailoring medical treatment and health care practices to individual patient
characteristics such as genetics, lifestyle, and environment.
 Techniques: Use of big data, genomics, AI, wearable sensors, and health records.
 Benefits: More accurate diagnoses, effective treatments, reduced side effects, preventive
care.
 Applications: Precision medicine, chronic disease management, fitness tracking, remote
monitoring.
 Challenges: Data privacy, integration of diverse data sources, cost, ethical concerns.

Q5. What is Neural Network? Give details on the different layers used in Neural
Network.

A Neural Network is a computational model inspired by the human brain’s network of neurons.
It is used in machine learning to recognize patterns, classify data, and make predictions.

 It consists of interconnected nodes called neurons arranged in layers.

 Each neuron receives inputs, applies weights and biases, processes the input with an
activation function, and passes the output to the next layer.
 Neural networks can learn complex relationships in data by adjusting weights during
training.

Different Layers in a Neural Network

1. Input Layer
o Receives the raw data/features.
o The number of neurons equals the number of input features.
 o Passes input values to the next layer without processing.
2. Hidden Layers
o Intermediate layers between input and output.
o Perform computations and extract features/patterns.
o Each neuron applies weights, bias, and activation function to inputs.
o Can be one or multiple layers (deep neural networks have many hidden layers).
o Activation functions introduce non-linearity (e.g., ReLU, Sigmoid, Tanh).
3. Output Layer
o Produces the final result (e.g., classification label, regression value).
o Number of neurons depends on the task:
 For classification, usually one neuron per class with softmax activation.
 For regression, usually one neuron with linear activation.

Q6. Explain the concept of Perceptron along with its working with a suitable
example.

What is a Perceptron?

 The Perceptron is the simplest type of artificial neural network, used for binary
classification.
 It was introduced by Frank Rosenblatt in 1958.
 It’s essentially a single neuron model that takes multiple inputs, applies weights, sums
them up, and passes the result through an activation function to produce an output (0 or
1).

Working of a Perceptron

1. Inputs: The perceptron receives multiple inputs (X1,X2,...,xn).

2. Weights: Each input is multiplied by a corresponding weight (w1,w2,...,wn).
3. Summation: The weighted inputs are summed, often adding a bias term b: