RNN
Recurrent Neural Network
What are RNNs?
RNNs are a type of neural network designed to process sequential data.
Unlike traditional feedforward networks, RNNs have loops that allow them to maintain
a memory of previous inputs.
This makes them ideal for problems where order and context matter.
Real-Life Examples of Sequential Data:
Application Input Sequence Task
Language Modeling "I am going to the..." Predict next word
Time Series Daily temperatures Forecast future values
Speech Recognition Audio waveform Convert to text
Text Generation Seed text Generate new sentence
2. Feedforward vs Recurrent Neural Networks
Feedforward Neural Networks:
Input flows in one direction only.
Each input is treated independently.
Not suitable for sequential data.
Recurrent Neural Networks:
Have a loop within the architecture.
Output at time t depends on input at time t and the hidden state from t-1.
Can remember information for short durations.
RNN Architecture and Workflow:
Key Components:
Input vector xt : current element in the sequence
Hidden state ht : memory of the network
Output yt : predicted output at current step
Applications of RNNs:
Natural Language Processing:
Sentiment analysis
Language modeling
Machine translation
Named Entity Recognition
Time Series Prediction:
Forecasting stock prices
Weather prediction
Power consumption prediction
Music & Audio:
Speech recognition
Music generation
Voice cloning
Recurrent Neural Networks (RNN) – Core Concepts & Terms
1. Sequential Data
Definition:
Data where the order of elements matters, and current values often depend on previous ones.
Real-life Examples:
Sentences in a paragraph
Stock market prices over time
Heartbeat signals (ECG)
Audio waves in speech
Recurrent Neural Network (RNN)
Definition:
A type of neural network that processes sequences of data by maintaining a memory (hidden
state) of previous inputs.
Real-life Example:
Reading a sentence: you understand the current word based on the context of the previous ones.
Hidden State
Definition:
An internal memory of the network that stores information from previous time steps in a
sequence.
Real-life Example:
When listening to a song, your brain remembers the tune that just played to anticipate the next
part.
Unrolling an RNN
Definition:
Breaking the loop structure of an RNN into a series of steps over time for visualization or
training.
Real-life Example:
If you think of a TV series, each episode (time step) follows the storyline (memory) of the
previous ones.
Weight Sharing
Definition:
The same set of weights is reused across all time steps in an RNN, making it efficient for
sequences.
Real-life Example:
Like using the same grammar rules repeatedly while constructing different sentences.
Vanishing Gradient Problem
Definition:
A challenge during training where gradients become very small and stop the network from
learning long-term dependencies.
Real-life Example:
Trying to remember what you had for lunch two weeks ago—it fades away because it’s too far
back.
Exploding Gradient Problem
Definition:
An issue where gradients become extremely large during training, causing unstable updates.
Real-life Example:
An overreaction in memory: misremembering a small event as a big trauma because the signal
amplified too much.
Short-Term Memory
Definition:
RNNs can recall only recent inputs effectively; distant past inputs are often forgotten.
Real-life Example:
Recalling the last few words in a sentence you just read, but forgetting the first ones.
Time Step
Definition:
Each position in a sequence that the RNN processes, step-by-step.
Real-life Example:
Each word spoken in a conversation is a time step in an audio signal.
Sequence-to-Sequence (Seq2Seq)
Definition:
An RNN model where input and output are both sequences, possibly of different lengths.
Real-life Example:
Language translation: input = English sentence, output = French sentence.
Sequence-to-One
Definition:
An RNN where a whole input sequence maps to one output.
Real-life Example:
Sentiment analysis: input = product review sentence, output = positive/negative sentiment.
Sequence-to-Many
Definition:
A single input is used to generate a sequence of outputs.
Real-life Example:
Text generation: input = topic or seed text, output = entire paragraph.
Tanh Activation Function
Definition:
A function that squashes input values to between -1 and 1, helping stabilize RNN computations.
Real-life Example:
Helps "moderate" the flow of information like a volume control knob.
Embedding (in NLP)
Definition:
A way to represent words or characters as dense numerical vectors to make them
understandable by neural networks.
Real-life Example:
Translating each word in a sentence into a format a computer can understand and process.
Context / Memory
Definition:
The accumulated information that helps the RNN understand the current input better by
referencing previous steps.
Real-life Example:
In a novel, each chapter builds on the previous ones—you can’t understand the plot without the
earlier context.
Applications of RNNs
Area Real-life Application
NLP Text prediction, chatbots, translation
Time Series Weather forecasting, sales prediction
Speech Voice recognition, virtual assistants
Healthcare ECG pattern analysis, symptom prediction
Music Melody generation, music recommendation
Summary Table of Core Terms
Term Definition Real-life Example
RNN Neural network for sequences Understanding a spoken sentence
Hidden State Internal memory Remembering earlier conversation
Time Step One point in a sequence A word in a sentence
Vanishing Gradient Memory fades during learning Forgetting past events
Exploding Gradient Memory overload Overreacting to a small event
Sequence-to-Sequence Input & output are sequences English → French translation
Sequence-to-One Input sequence → single output Emotion classification
Embedding Word to vector representation Translating language to numbers
Tanh Smoothing function Moderating data flow
Unrolling Viewing RNN over time Watching TV episodes in order
1. Batch Definition:
A batch is a subset of the training dataset used to train the model in one forward and backward
pass.
Why It Matters:
It’s inefficient to train the model on the entire dataset all at once, especially when it’s
large.
So the data is split into smaller groups (batches) for efficiency and faster computation.
Types:
Batch Gradient Descent: uses the whole dataset at once (slow, rarely used).
Mini-Batch Gradient Descent: uses small chunks of the data (most common).
Stochastic Gradient Descent (SGD): uses 1 sample per update (noisy but can work).
Real-Life Example:
Imagine learning from a textbook. Instead of reading the whole book in one go, you study
it chapter by chapter (batches).
Epoch
Definition:
An epoch is one full pass through the entire training dataset.
Why It Matters:
The model doesn’t learn everything in one pass.
You need multiple epochs so the model can repeatedly adjust and improve its
predictions.
Real-Life Example:
Practicing a speech multiple times: each practice round is like one epoch.
With each round, you remember more, correct mistakes, and improve delivery.
Loss Function (Cost Function)
Definition:
A loss function measures how far off the model's predictions are from the actual values.\
Why It Matters:
It gives the model a goal to minimize.
The lower the loss, the better the model is performing.
Common Loss Functions:
Problem Type Loss Function
Regression Mean Squared Error (MSE)
Classification Cross Entropy Loss
Binary Output Binary Cross Entropy
Real-Life Example:
A loss function is like exam results: the higher the error (wrong answers), the lower your
score. Your goal is to reduce your mistakes.
Optimizer
Definition:
An optimizer is an algorithm that adjusts the model’s weights to minimize the loss.
Why It Matters:
The optimizer uses gradients (slope of the loss curve) to update the model.
The goal is to move the model's predictions closer to the actual answers with each step.
Common Optimizers:
Optimizer Description
SGD (Stochastic Gradient Descent) Basic, uses a learning rate to update weights
Adam (Adaptive Moment Estimation) Most used, adapts learning rate automatically
RMSprop Good for noisy problems like RNNs
Real-Life Example:
Optimizer is like a GPS recalculating the route as you drive toward your destination
(minimum loss).
Learning Rate
Definition:
A hyperparameter that determines how big a step the optimizer takes during weight updates.
Why It Matters:
Too high → model overshoots the best answer.
Too low → model takes forever to learn.
Real-Life Example:
Like adjusting the speed of your car:
o Too fast → you might miss a turn.
o Too slow → you’ll take forever to reach.
Forward Pass and Backward Pass
Forward Pass:
Input is passed through the network to generate output.
Loss is computed by comparing output with the correct label.
Backward Pass (Backpropagation):
The network calculates gradients of the loss with respect to weights.
These gradients help the optimizer adjust the weights.
Real-Life Example:
Forward pass is like taking a test.
Backward pass is like getting feedback on your mistakes and improving.
Overfitting and Underfitting
Overfitting:
Model memorizes training data but performs poorly on new data.
Happens when training too long or with too complex a model.
Underfitting:
Model is too simple or hasn’t trained enough to learn the pattern.
Real-Life Example:
Overfitting: Memorizing answers to past papers but failing a new test.
Underfitting: Not studying enough to even do the basics.
Summary Table of Terms
Term Definition Real-Life Analogy
Batch A chunk of training data used at once Reading a chapter of a book
Epoch One full pass over all data Practicing a speech once
Loss Function Measures error Exam score
Optimizer Adjusts weights to reduce loss GPS finding the best route
Learning Rate Step size of optimizer Driving speed
Forward Pass Prediction step Taking a test
Backward Learning from mistake Feedback session
Pass
Overfitting Learns too much detail Memorizing without understanding
Underfitting Learns too little Not preparing enough
