Program 13: Write a python code for Tokenization & One-Hot Encoding using

CountVectorizer

Description:
Tokenization
Before representing text numerically, it must be broken down into smaller units
called tokens.
In the code, tokenizer=lambda x: x.split() splits each sentence by whitespace.
Example:
"apple and banana" → ["apple", "and", "banana"]
The vectorizer builds a vocabulary of all unique tokens in the corpus.
One-Hot Encoding (Binary Representation)
This method represents whether each token from the vocabulary is present
(1) or absent (0) in a sentence.
• binary=True in CountVectorizer ensures that we use a binary vector, not raw
counts.
• Each vector’s length = size of the vocabulary.
• Each sentence becomes a vector indicating the presence of each word.
Example Corpus:
1. "apple and banana"
2. "banana and orange"
3. "grape apple banana"
Vocabulary: ['and', 'apple', 'banana', 'grape', 'orange'] Binary
Matrix Output:
• [1, 1, 1, 0, 0] # "apple and banana"
• [1, 0, 1, 0, 1] # "banana and orange"
• [0, 1, 1, 1, 0] # "grape apple banana"

This matrix numerically represents the text and is suitable for downstream ML tasks.

Code:

22261A6634 29
 from sklearn.preprocessing import OneHotEncoder
from sklearn.feature_extraction.text import CountVectorizer

corpus = [
"apple and banana",
"banana and orange",
"grape apple banana"
]

# Tokenize and extract unique words

vectorizer = CountVectorizer(tokenizer=lambda x: x.split(),
binary=True)
X = vectorizer.fit_transform(corpus).toarray()

print("Vocabulary:", vectorizer.get_feature_names_out())
print("One-Hot Encoded Matrix:\n", X)

Output:
Vocabulary: ['and' 'apple' 'banana' 'grape' 'orange']
One-Hot Encoded Matrix:
[[1 1 1 0 0]
[1 0 1 0 1]
[0 1 1 1 0]]

Program 14: Write a python code for demonstrating Count Vectorization, also
known as the Bag-of-Words (BoW) model — a foundational text representation
technique in NLP.

Description:
This code demonstrates Count Vectorization, also known as the Bag-of-Words
(BoW) model — a foundational text representation technique in NLP.

What is Count Vectorization?

• CountVectorizer converts a collection of text documents into a matrix of token
counts.

22261A6634 30
 • It breaks each sentence into tokens (typically words), builds a vocabulary of all
unique tokens, and creates vectors indicating the frequency of each word in a
sentence.
How it works:
1. Tokenization:
The text is split into words (tokens) using default rules (like splitting by spaces
and removing punctuation).
2. Vocabulary Creation:
A set of all unique words across the corpus is created.
3. Vectorization:
Each sentence is converted into a numerical vector where:
o Each dimension corresponds to a word in the vocabulary.
o The value represents how many times that word appears in the sentence.

Example Corpus:
1. "Natural language processing is a field of artificial intelligence and language
processing."
2. "Machine learning and deep learning are parts of AI."
3. "Natural language techniques are used in chatbots and translation."
4. "The future of AI depends on advances in NLP."

22261A6634 31
 • A vocabulary is extracted: All unique words from all sentences.
• The Count Vector Matrix shows how often each vocabulary word appears in each
sentence.

This Bag-of-Words model captures word frequency information but ignores grammar,
word order, and semantics. It is commonly used for:
• Text classification
• Spam detection
• Sentiment analysis
• Document similarity tasks Code:
from sklearn.feature_extraction.text import CountVectorizer
# Sample corpus (can be sentences, paragraphs, documents)
corpus = [
"Natural language processing is a field of artificial
intelligence and language processing.",
"Machine learning and deep learning are parts of AI.",
"Natural language techniques are used in chatbots and
translation.",
"The future of AI depends on advances in NLP."
]

# Initialize the vectorizer vectorizer

= CountVectorizer()

# Fit and transform the corpus

X = vectorizer.fit_transform(corpus)

# Convert result to array and get feature names

count_matrix = X.toarray()
vocab = vectorizer.get_feature_names_out()

# Print vocabulary and count matrix

print("📚 Vocabulary:") print(vocab)
print("\n🧮 Count Vector Matrix:") for
i, row in enumerate(count_matrix):
print(f"Sentence {i+1}: {row}")

22261A6634 32
 Output:
📚 Vocabulary:
['advances' 'ai' 'and' 'are' 'artificial' 'chatbots' 'deep' 'depends'
'field' 'future' 'in' 'intelligence' 'is' 'language' 'learning' 'machine'
'natural' 'nlp' 'of' 'on' 'parts' 'processing' 'techniques' 'the'
'translation' 'used']

🧮 Count Vector Matrix:

Sentence 1: [0 0 1 0 1 0 0 0 1 0 0 1 1 2 0 0 1 0 1 0 0 2 0 0 0 0]
Sentence 2: [0 1 1 1 0 0 1 0 0 0 0 0 0 0 2 1 0 0 1 0 1 0 0 0 0 0]
Sentence 3: [0 0 1 1 0 1 0 0 0 0 1 0 0 1 0 0 1 0 0 0 0 0 1 0 1 1]
Sentence 4: [1 1 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 1 1 1 0 0 0 1 0 0]

22261A6634 33
 Program 14 A: Write a python code for TF-IDF (Term Frequency-Inverse Document
Frequency) to understand which words are important, not just frequent.

Description:
1. Count Vectorizer (Bag of Words)
• Converts each document into a vector based on word frequency.
• Ignores grammar and word order; only counts occurrences of words.
• Commonly used for basic text classification and NLP tasks.
Limitation:
Frequent but less meaningful words (like "data" or "systems") may dominate the
representation, even if they don’t carry much information.

2. TF-IDF (Term Frequency–Inverse Document Frequency)

• Goes beyond raw word counts by evaluating how important a word is in a
document relative to the entire corpus.
How it works:
• TF (Term Frequency): Measures how often a word appears in a single document.
• IDF (Inverse Document Frequency): Downweights words that appear in many
documents, as they are less informative.
ð Final TF-IDF score = TF × IDF
• High TF-IDF → word is important in that document but rare in others.
• Low TF-IDF → word is common across documents (less informative).

In this example:
• Corpus: A small collection of sentences related to AI.
• CountVectorizer: Creates a matrix with raw word counts (excluding stop words).
• TfidfVectorizer: Creates a matrix of weighted scores based on word importance.
This helps in tasks like:
• Document classification
• Keyword extraction
• Search relevance ranking

22261A6634 34
 Code:
from sklearn.feature_extraction.text import CountVectorizer,
TfidfVectorizer import pandas as pd

corpus = [
"Artificial Intelligence is transforming industries and daily
life through automation and smart systems.",
"Machine Learning, as a subset of AI, enables systems to
learn from data without being explicitly programmed.",
"Deep Learning techniques use neural networks with many
layers to model complex patterns in data such as images and
speech.",
"Applications of AI include self-driving cars, medical
diagnosis, financial forecasting, and personalized recommendations.",
"Natural Language Processing helps computers understand,
interpret, and generate human language using linguistic and
statistical techniques.",
"With rapid advancements in computing power and data
availability, the future of AI continues to grow exponentially."
]

# Count Vectorizer
count_vectorizer = CountVectorizer(stop_words='english')
count_matrix = count_vectorizer.fit_transform(corpus) count_df
= pd.DataFrame(count_matrix.toarray(),
columns=count_vectorizer.get_feature_names_out())

# TF-IDF Vectorizer
tfidf_vectorizer = TfidfVectorizer(stop_words='english')
tfidf_matrix = tfidf_vectorizer.fit_transform(corpus) tfidf_df
= pd.DataFrame(tfidf_matrix.toarray(),
columns=tfidf_vectorizer.get_feature_names_out())

# Display
print("📊 Count Vector (Bag of Words):") print(count_df)

print("\n🌟 TF-IDF Vector:") print(tfidf_df)

22261A6634 35
 Output:
📊 Count Vector (Bag of Words):
advancements ai applications artificial automation availability cars \ 0
0 0 0 1 1 0 0
1 0 1 0 0 0 0 0
2 0 0 0 0 0 0 0
3 0 1 1 0 0 0 1
4 0 0 0 0 0 0 0
5 1 1 0 0 0 1 0

complex computers computing ... smart speech statistical subset \ 0

0 0 0 ... 1 0 0 0
1 0 0 0 ... 0 0 0 1
2 1 0 0 ... 0 1 0 0
3 0 0 0 ... 0 0 0 0
4 0 1 0 ... 0 0 1 0
5 0 0 1 ... 0 0 0 0

systems techniques transforming understand use using 0

1 0 1 0 0 0
1 1 0 0 0 0 0
2 0 1 0 0 1 0
3 0 0 0 0 0 0
4 0 1 0 1 0 1
5 0 0 0 0 0 0

[6 rows x 61 columns]

🌟 TF-IDF Vector:
advancements ai applications artificial automation availability \ 0
0.000000 0.000000 0.00000 0.33957 0.33957 0.000000
1 0.000000 0.240255 0.00000 0.00000 0.00000 0.000000
2 0.000000 0.000000 0.00000 0.00000 0.00000 0.000000
3 0.000000 0.204336 0.29515 0.00000 0.00000 0.000000
4 0.000000 0.000000 0.00000 0.00000 0.00000 0.000000
5 0.316885 0.219383 0.00000 0.00000 0.00000 0.316885

cars complex computers computing ... smart speech \ 0

0.00000 0.000000 0.000000 0.000000 ... 0.33957 0.000000
1 0.00000 0.000000 0.000000 0.000000 ... 0.00000 0.000000
2 0.00000 0.290814 0.000000 0.000000 ... 0.00000 0.290814
3 0.29515 0.000000 0.000000 0.000000 ... 0.00000 0.000000
4 0.00000 0.000000 0.252599 0.000000 ... 0.00000 0.000000
5 0.00000 0.000000 0.000000 0.316885 ... 0.00000 0.000000

statistical subset systems techniques transforming understand \ 0

0.000000 0.000000 0.278453 0.000000 0.33957 0.000000
1 0.000000 0.347033 0.284572 0.000000 0.00000 0.000000
2 0.000000 0.000000 0.000000 0.238472 0.00000 0.000000
3 0.000000 0.000000 0.000000 0.000000 0.00000 0.000000
4 0.252599 0.000000 0.000000 0.207135 0.00000 0.252599
5 0.000000 0.000000 0.000000 0.000000 0.00000 0.000000

use using 0
0.000000 0.000000
1 0.000000 0.000000
2 0.290814 0.000000
3 0.000000 0.000000
4 0.000000 0.252599
5 0.000000 0.000000
[6 rows x 61 columns]

22261A6634 36
 Program 15: Write a python code to implement word2vec word-embedding
technique

Description: Word2vec represents a word as a high-dimension vector of numbers

which capture relationships between words. In particular, words which appear in similar
contexts are mapped to vectors which are nearby as measured by cosine similarity. This
indicates the level of semantic similarity between the words, so for example the vectors
for walk and ran are nearby, as are those for "but" and "however", and "Berlin" and
"Germany".

Code:
import gensim
import pandas as pd

df = pd.read_json("Sports_and_Outdoors_5.json", lines=True)
review_text =
df.reviewText.apply(gensim.utils.simple_preprocess)
df.reviewText.loc[0] model =
gensim.models.Word2Vec( window=10, min_count=2,
workers=4,
)
model.build_vocab(review_text, progress_per=1000) model.train(review_text,
total_examples=model.corpus_count, epochs=model.epochs)
model.save("./word2vec-outdoor-reviews-short.model")
print(model.wv.most_similar("awful"))

Output:
[('terrible', 0.7352169156074524),
('horrible', 0.6891771554946899),
('overwhelming', 0.6227911710739136),
('impossibility', 0.5835400819778442),

22261A6634 37
 ('horrendous', 0.5827057957649231),
('enormous', 0.5721909999847412),
('ugly', 0.567825436592102),
('unusual', 0.566750705242157),
('isolated', 0.5588798522949219),
('unfortunate', 0.5560564994812012)]

• model.wv.similarity(w1="good", w2="great") ð
output: 0.7870506

• model.wv.similarity(w1="slow", w2="steady")

ð output: 0.3472042

Program 16 A: Write a python program to create a sample list for at least 5 words with

ambiguous sense Description:

A word sense is a specific meaning of a word, especially when the word has multiple
meanings depending on the context. This concept is central to understanding and
processing natural language correctly.

Example of Word Senses

Take the word "bat" — it has several senses:

1. Bat (animal): A flying nocturnal mammal.

"The bat flew out of the cave."
2. Bat (sports equipment): A stick used to hit a ball in games like cricket or
baseball.
"He swung the bat and hit a home run."

Each of these is a different word sense of the word bat.

22261A6634 38
 Word Sense in Lexical Databases

Tools like WordNet organize word senses systematically. For example:

• Word: "bank" o bank.n.01: financial institution o bank.n.02: sloping land beside a

river

Each sense has:

• A definition (gloss)
• Examples
• Synonyms (called synsets)
• Relations (hypernyms, hyponyms, etc.)
Code:

import nltk from nltk.corpus import

wordnet as wn

# Download WordNet data if you don't have it

nltk.download('wordnet')

# Define a function to resolve word sense based on context

def wsd(word, context):
# Look for different senses (meanings) of the word using WordNet
senses = wn.synsets(word) # Print the senses of the word
print(f"Senses of the word '{word}':") for i, sense in
enumerate(senses):
print(f"{i+1}. {sense.name()}: {sense.definition()}")

# A simple approach to disambiguate: Check if any context words match with the word's
definition for i, sense in enumerate(senses): for example in sense.examples():

22261A6634 39
 if any(context_word in example for context_word in context):
print(f"\nContext matched with sense {i+1}: {sense.name()}")
print(f"Example: {example}") return sense.name() # Return
the sense name based on context

return "No match found"

# Sample sentences (context for ambiguous words) contexts

={
'bank': ["money", "deposit", "river", "side", "finance"],
'bark': ["tree", "rough", "dog", "loud"],
'bat': ["fly", "animal", "hit", "game"],
'bow': ["tie", "gift", "respect", "gesture"],
'lead': ["guide", "direct", "metal", "pipes"]
}

# Run WSD on each ambiguous word with its context for

word in contexts:
print(f"\nWSD for the word: {word}")
word_context = contexts[word] sense
= wsd(word, word_context)
print(f"Disambiguated sense of '{word}': {sense}")

Output:
WSD for the word: bank
Senses of the word 'bank':
1. bank.n.01: sloping land (especially the slope beside a body of water) 2.
depository_financial_institution.n.01: a financial institution that accepts
deposits and channels the money into lending activities
3. bank.n.03: a long ridge or pile
4. bank.n.04: an arrangement of similar objects in a row or in tiers

22261A6634 40
 5. bank.n.05: a supply or stock held in reserve for future use (especially in
emergencies)
6. bank.n.06: the funds held by a gambling house or the dealer in some
gambling games
7. bank.n.07: a slope in the turn of a road or track; the outside is higher
than the inside in order to reduce the effects of centrifugal force 8.
savings_bank.n.02: a container (usually with a slot in the top) for
keeping money at home
9. bank.n.09: a building in which the business of banking transacted 10.
bank.n.10: a flight maneuver; aircraft tips laterally about its
longitudinal axis (especially in turning)
11. bank.v.01: tip laterally
12. bank.v.02: enclose with a bank
13. bank.v.03: do business with a bank or keep an account at a bank
14. bank.v.04: act as the banker in a game or in gambling
15. bank.v.05: be in the banking business
16. deposit.v.02: put into a bank account
17. bank.v.07: cover with ashes so to control the rate of burning
18. trust.v.01: have confidence or faith in

Context matched with sense 1: bank.n.01

Example: he sat on the bank of the river and watched the currents
Disambiguated sense of 'bank': bank.n.01

WSD for the word: bark

Senses of the word 'bark':
1. bark.n.01: tough protective covering of the woody stems and roots of trees
and other woody plants
2. bark.n.02: a noise resembling the bark of a dog
3. bark.n.03: a sailing ship with 3 (or more) masts
4. bark.n.04: the sound made by a dog
5. bark.v.01: speak in an unfriendly tone
6. bark.v.02: cover with bark
7. bark.v.03: remove the bark of a tree
8. bark.v.04: make barking sounds
9. bark.v.05: tan (a skin) with bark tannins

Context matched with sense 8: bark.v.04

Example: The dogs barked at the stranger
Disambiguated sense of 'bark': bark.v.04

WSD for the word: bat

Senses of the word 'bat':
1. bat.n.01: nocturnal mouselike mammal with forelimbs modified to form
membranous wings and anatomical adaptations for echolocation by which
they navigate
2. bat.n.02: (baseball) a turn trying to get a hit
3. squash_racket.n.01: a small racket with a long handle used for playing
squash
4. cricket_bat.n.01: the club used in playing cricket
5. bat.n.05: a club used for hitting a ball in various games
6. bat.v.01: strike with, or as if with a baseball bat
7. bat.v.02: wink briefly
8. bat.v.03: have a turn at bat

22261A6634 41
 9. bat.v.04: use a bat
10. cream.v.02: beat thoroughly and conclusively in a competition or fight

Context matched with sense 2: bat.n.02

Example: he got four hits in four at-bats
Disambiguated sense of 'bat': bat.n.02

WSD for the word: bow

Senses of the word 'bow':
1. bow.n.01: a knot with two loops and loose ends; used to tie shoelaces 2.
bow.n.02: a slightly curved piece of resilient wood with taut horsehair
strands; used in playing certain stringed instruments
3. bow.n.03: front part of a vessel or aircraft
4. bow.n.04: a weapon for shooting arrows, composed of a curved piece of
resilient wood with a taut cord to propel the arrow
5. bow.n.05: something curved in shape
6. bow.n.06: bending the head or body or knee as a sign of reverence or
submission or shame or greeting
7. bow.n.07: an appearance by actors or performers at the end of the concert
or play in order to acknowledge the applause of the audience
8. bow.n.08: a decorative interlacing of ribbons
9. bow.n.09: a stroke with a curved piece of wood with taut horsehair
strands that is used in playing stringed instruments
10. bow.v.01: bend one's knee or body, or lower one's head
11. submit.v.06: yield to another's wish or opinion
12. bow.v.03: bend the head or the upper part of the body in a gesture of
respect or greeting
13. crouch.v.01: bend one's back forward from the waist on down
14. bow.v.05: play on a string instrument with a bow
Disambiguated sense of 'bow': No match found

WSD for the word: lead

Senses of the word 'lead':
1. lead.n.01: an advantage held by a competitor in a race
2. lead.n.02: a soft heavy toxic malleable metallic element; bluish white
when freshly cut but tarnishes readily to dull grey 3. lead.n.03:
evidence pointing to a possible solution
4. lead.n.04: a position of leadership (especially in the phrase `take the
lead')
5. lead.n.05: the angle between the direction a gun is aimed and the
position of a moving target (correcting for the flight time of the
missile)
6. lead.n.06: the introductory section of a story
7. lead.n.07: (sports) the score by which a team or individual is winning
8. star.n.04: an actor who plays a principal role
9. lead.n.09: (baseball) the position taken by a base runner preparing to
advance to the next base
10. tip.n.03: an indication of potential opportunity
11. lead.n.11: a news story of major importance
12. spark_advance.n.01: the timing of ignition relative to the position of
the piston in an internal-combustion engine
13. leash.n.01: restraint consisting of a rope (or light chain) used to
restrain an animal

22261A6634 42
 14. lead.n.14: thin strip of metal used to separate lines of type in printing
15. lead.n.15: mixture of graphite with clay in different degrees of
hardness; the marking substance in a pencil
16. jumper_cable.n.01: a jumper that consists of a short piece of wire
17. lead.n.17: the playing of a card to start a trick in bridge
18. lead.v.01: take somebody somewhere
19. leave.v.07: have as a result or residue
20. lead.v.03: tend to or result in
21. lead.v.04: travel in front of; go in advance of others
22. lead.v.05: cause to undertake a certain action
23. run.v.03: stretch out over a distance, space, time, or scope; run or
extend between two points or beyond a certain point
24. head.v.02: be in charge of
25. lead.v.08: be ahead of others; be the first
26. contribute.v.03: be conducive to
27. conduct.v.02: lead, as in the performance of a composition
28. go.v.25: lead, extend, or afford access
29. precede.v.04: move ahead (of others) in time or space
30. run.v.23: cause something to pass or lead somewhere
31. moderate.v.01: preside over
Disambiguated sense of 'lead': No match found
[nltk_data] Downloading package wordnet to /root/nltk_data... [nltk_data]
Package wordnet is already up-to-date!

Program 16 B: Write a python program to implement Lesk’s algorithm for word sense
disambiguity.

Description:

Word Sense Disambiguation (WSD) is the process of identifying the correct meaning
(sense) of a word based on its context, especially when the word has multiple
meanings.

Lesk’s algorithm is a knowledge-based method for WSD that disambiguates a word

by comparing the dictionary definitions (glosses) of its possible senses with the context
in which the word appears.

The correct sense of an ambiguous word is the one whose definition overlaps the
most with the definitions of the surrounding words in the sentence.

Working of algorithm:

• Identify all possible senses of the ambiguous word using a dictionary like
WordNet.

22261A6634 43
 • For each sense, take its definition (gloss).
• Compare it with the glosses or words in the context (surrounding words).
• Count overlapping words between glosses and context.
• Choose the sense with the maximum overlap.

Code:
import nltk
from nltk.corpus import wordnet as wn from
nltk.tokenize import word_tokenize def
lesk_algorithm(word, context):

# Get all the senses of the word from WordNet

senses = wn.synsets(word)

if not senses:
return None
max_overlap = 0
best_sense = None

# Tokenize the context context_tokens =

set(word_tokenize(' '.join(context).lower()))

for sense in senses:

# Get the definition and examples of the sense
definition = set(word_tokenize(sense.definition().lower()))
examples = set()

for example in sense.examples():

examples.update(word_tokenize(example.lower()))

# Compute the overlap between the context and the sense definition + examples
overlap = len(context_tokens.intersection(definition.union(examples)))

22261A6634 44
 # Keep track of the sense with the maximum overlap
if overlap > max_overlap: max_overlap = overlap
best_sense = sense

return best_sense

# Example usage:
context = ["The bark of the tree is rough and textured."] word
= "bark"

best_sense = lesk_algorithm(word, context)

if best_sense:
print(f"Disambiguated sense for '{word}': {best_sense.name()}")
print(f"Definition: {best_sense.definition()}") else: print(f"No
sense found for '{word}'")

context = ["The dog started to bark loudly in the yard."] word

= "bark"

best_sense = lesk_algorithm(word, context)

if best_sense:
print(f"Disambiguated sense for '{word}': {best_sense.name()}")
print(f"Definition: {best_sense.definition()}") else: print(f"No
sense found for '{word}'")

Output:
Disambiguated sense for 'bark': bark.v.03

22261A6634 45
 Definition: remove the bark of a tree
Disambiguated sense for 'bark': bark.n.02
Definition: a noise resembling the bark of a dog

Program 17: Write a python program using NLTK package to convert audio file to text

and text to audio files Description:

NLTK it does not handle audio files directly.We will use speech_recognition and gTTS
libraries for audio, and use NLTK to process the text in between.

1. Speech_recognition:
speech_recognition is a Python library that helps you convert spoken audio
into written text using speech recognition engines.

2. gTTS libraries (Convert text file to audio)

gTTS stands for Google Text-to-Speech — it's a Python library and CLI tool that
lets you convert text into spoken audio using the Google Text-to-Speech API.
It takes in text (string), sends it to Google's Text-to-Speech engine, and returns a
spoken audio file in MP3 format.

Install the following packages:

1. !pip install SpeechRecognition pydub

2. !pip install gTTS

Code:

22261A6634 46
 import speech_recognition as sr from
gtts import gTTS def
audio_to_text(audio_file_path):
recognizer = sr.Recognizer() with
sr.AudioFile(audio_file_path) as source:
audio_data = recognizer.record(source)
try:
text = recognizer.recognize_google(audio_data)
print("Recognized Text:\n", text)
return text except sr.UnknownValueError:
print("Speech Recognition could not understand the audio")
except sr.RequestError:
print("Could not request results from Google Speech Recognition service")

def text_to_audio(text, output_audio_path):

tts = gTTS(text)
tts.save(output_audio_path)
print(f"Saved audio to {output_audio_path}")

audio_path = "Sports.wav"
output_audio_path = "output.mp3"

# Convert audio to text

text = audio_to_text(audio_path)

# Optionally use NLTK for processing import

nltk
from nltk.tokenize import word_tokenize

if text:
tokens = word_tokenize(text)
print("Tokenized Text:\n", tokens)

# Convert text back to audio

text1="hello welcome to class"
text_to_audio(text1, output_audio_path)

Output:

Recognized Text: good evening ladies and gentlemen we like to welcome

you to play the new videos Broadcast

22261A6634 47
 Tokenized Text:
['good', 'evening', 'ladies', 'and', 'gentlemen', 'we', 'like', 'to',
'welcome', 'you', 'to', 'play', 'the', 'new', 'videos', 'Broadcast']

Saved audio to output.mp3

Program 18: Write a python program using NLTK package to convert audio file to text
and text to audio files

Description:

FrameNet is a linguistic database that organizes words based on the situations (called
frames) they describe, showing how words are connected to roles and events in
realworld experiences.

1. Helps computers understand not just words, but meanings and relationships.

2. Useful in AI, language learning, chatbots, machine translation, and semantic

search.

The key elements of FrameNet are Frames, Frame Elements (FEs), and Lexical
Units (LUs).

Frames:
These are scripts or conceptual structures that describe specific types of situations,
events, or objects. For example, a "Cooking" frame would describe the situation of
preparing food, including the roles of a cook, the food, and the heating instrument.

Frame Elements (FEs):

These are the semantic roles or participants within a frame. In the "Cooking" frame, FEs
might include "Cook," "Food," and "Heating_instrument".

Lexical Units (LUs):

22261A6634 48
 A lexical unit is a word (or phrase) in a specific sense, linked to a particular frame. For
example, "bake" in its sense of preparing food would be an LU associated with the
"Cooking" frame.

Code:

Program to print di-erent framenet import nltk from

nltk.corpus import framenet as fn # Download
FrameNet data (only needed once)
nltk.download('framenet_v17')
# Get all frames frames = fn.frames() # Print some frame names
print("Total number of frames:", len(frames)) print("\nSample frames
in FrameNet:\n") for i, frame in enumerate(frames[:20]): # Change
20 to show more print(f"{i+1}. {frame.name}")

Output:
[nltk_data] Downloading package framenet_v17 to /root/nltk_data...
[nltk_data] Unzipping corpora/framenet_v17.zip.
Total number of frames: 1221

Sample frames in FrameNet:

1. Abandonment
2. Abounding_with
3. Absorb_heat
4. Abundance
5. Abusing
6. Access_scenario
7. Accompaniment
8. Accomplishment
9. Accoutrements
10. Accuracy
11. Achieving_first
12. Active_substance
13. Activity
14. Activity_abandoned_state
15. Activity_done_state
16. Activity_finish
17. Activity_ongoing

22261A6634 49
 18. Activity_pause
19. Activity_paused_state
20. Activity_prepare

Program to print detail of a particular given frame

import nltk
from nltk.corpus import framenet as fn

# Download if not already done

nltk.download('framenet_v17')

# Choose a frame by name, e.g., "Commerce_buy" frame =

fn.frame('Awareness')

# Print basic info

print("📌 Frame Name:", frame.name)
print("📝 Definition:", frame.definition)

# Print Lexical Units (words that evoke this frame)

print("\n🔤 Lexical Units (LUs):") for lu in
frame.lexUnit: print(f"- {lu}")

# Print Frame Elements (semantic roles)

print("\n🧩 Frame Elements (FEs):") for
fe_name, fe in frame.FE.items():
print(f"- {fe_name}: {fe.coreType} — {fe.definition}")

Output:
[nltk_data] Downloading package framenet_v17 to /root/nltk_data...
📌 Frame Name: Awareness
📝 Definition: A Cognizer has a piece of Content in their model of the world.
The Content is not necessarily present due to immediate perception, but
usually, rather, due to deduction from perceivables. In some cases, the
deduction of the Content is implicitly based on confidence in sources of
information (believe), in some cases based on logic (think), and in other
cases the source of the deduction is deprofiled (know). 'Your boss is aware
of your commitment.' '' Note that this frame is undergoing some degree of
reconsideration. Many of the targets will be moved to the Opinion frame.
That frame indicates that the Cognizer considers something as true, but the
Opinion (compare to Content) is not presupposed to be true; rather it is
something that is considered a potential point of difference, as in the
following: 'I think that you are awesome.' In the uses that will remain
in the Awareness frame, however, the Content is presupposed. '' This frame
is also distinct from the Certainty frame, in that it does not profile the
relationship of the Cognizer to the Content, but rather presupposes it. In

22261A6634 50
 Certainty, the Degree of confidence or certainty is expressible as a separate
frame element, as in the following: 'She absolutely knew that he would be
there .'
🔤 Lexical Units (LUs):
- aware.a
- awareness.n
- believe.v
- comprehend.v
- comprehension.n
- conceive.v
- conception.n
- conscious.a
- hunch.n
- imagine.v
- know.v
- knowledge.n
- knowledgeable.a
- presume.v
- presumption.n
- reckon.v
- supposition.n
- suspect.v
- suspicion.n - think.v
- thought.n
- understand.v
- understanding.n
- ignorance.n
- consciousness.n
- cognizant.a
- unknown.a
- idea.n

🧩 Frame Elements (FEs):

- Cognizer: Core — The Cognizer is the person whose awareness of phenomena is
at question. With a target verb or adjective the Cognizer is generally
expressed as an External Argument with the Content expressed as an Object
or Complement. 'Your boss is aware of your commitment.' 'The students do
not know the answer.' Examples like the following belong in Opinion: 'Pat
believes that things will change for the better.'
- Content: Core — The Content is the object of the Cognizer's awareness.
Content can be expressed as a direct object or in a PP Complement. 'Your
boss is aware of your commitment.' 'The students do not know of your
commitment.' 'The students do not know how committed you are.' 'The
students did not know that you were so committed.' Examples like the
following belong in Opinion: 'Pat believes that things will change for the
better.' - Evidence: Peripheral — The source of awareness or knowledge
which can be expressed in a PP Complement: 'The sailors knew from the look
of the sky that a storm was coming.' 'I knew from experience that Jo would
be late.' - Topic: Core — Some verbs in this frame allow a Topic to be
expressed in about-PPs. 'Kim knows about first aid.' However, a number of
nouns and adjectives in this frame which cannot take about-phrases allow
Topic to be expressed as an adjectival or adverbial modifier. 'Kim is
politically aware.
' ' Environmental consciousness is increasing.'

22261A6634 51
 - Degree: Peripheral — This FE identifies the Degree to which an event
occurs.
- Manner: Peripheral — This FE identifies the Manner in which the Cognizer
knows or thinks something.
- Expressor: Core — Expressor is the body part that reveals the Cognizer's
state to the observer. 'Bob's eyes were overly aware'
- Role: Peripheral — Role is the category within which an element of the
Content is considered. 'He understood her remark as an insult.'
- Paradigm: Extra-Thematic — This frame element identifies the Paradigm
which serves as the basis for the Cognizer's awareness. 'The formation of
black holes should be understood in astrophysic terms.'
- Time: Peripheral — The time interval during which the Cognizer is aware of
the Content. 'Yet there is no evidence that Mr. Parrish was cognizant at
the time of the signing of the notes that the clauses in issue were
present.'
- Explanation: Extra-Thematic — The reason why or how it came to be that the
Cognizer has awareness of the Topic or Content.
[nltk_data] Package framenet_v17 is already up-to-date!

Program to invoke a particular Frame based on Lexical unit in the given sentence

import nltk import

spacy
from nltk.corpus import framenet as fn

# Download required resources nltk.download('framenet_v17')

nlp = spacy.load("en_core_web_sm")

def find_frames_for_sentence(sentence): doc

= nlp(sentence)
frames_found = {}

for token in doc: if token.pos_ in {"VERB", "NOUN"}: #

Likely to evoke frames lemma = token.lemma_
frames = fn.frames_by_lemma(lemma)
if frames:
frames_found[token.text] = [frame.name for frame in frames]

return frames_found

# Example sentence
sentence = "We believe it is a fair and generous price." frames_invoked =
find_frames_for_sentence(sentence)

# Display result

22261A6634 52
 print(f"Sentence: {sentence}") print("\
nInvoked FrameNet Frames:") for word,
frames in frames_invoked.items(): print(f"-
{word}: {', '.join(frames)}")

Output:

[nltk_data] Downloading package framenet_v17 to /root/nltk_data...

[nltk_data] Package framenet_v17 is already up-to-date!

Sentence: We believe it is a fair and generous price.

Invoked FrameNet Frames:

- believe: Awareness, Certainty, Opinion, Religious_belief, Taking_sides,
Trust
- price: Commerce_scenario, Expensiveness

22261A6634 53
 Program 19: Write a python program using NLTK package to convert audio file to text
and text to audio files

Description:WordNet is a lexical database of the English language, widely used in

Natural Language Processing (NLP) tasks. It groups words into sets of synonyms
(synsets) and organizes them into a network based on their semantic relationships
like hyponyms (more specific), hypernyms (more general), meronyms (part-whole),
and holonyms (whole-part).

Key Features of WordNet:

• Synsets: Groups of synonymous words (e.g., "car" and "automobile" are

synonyms in the same synset).
• Hyponyms: WordNet organizes words by their hypernyms (general terms)
Generalized concepts (e.g., "vehicle" for "car").
• Hypernyms: Specific terms. Specific instances (e.g., "sports car" for "car").

Code:
1. Finding Synonyms of a given word:

from nltk.corpus import wordnet as wn

# Get synsets for the word 'car'

synsets = wn.synsets('car') for
synset in synsets:
print(f"Synset: {synset.name()}")
print(f"Definition: {synset.definition()}")
print(f"Examples: {synset.examples()}") print()

Output:
Synset: car.n.01

22261A6634 54
 Definition: a motor vehicle with four wheels; usually
propelled by an internal combustion engine Examples:
['he needs a car to get to work']

Synset: car.n.02
Definition: a wheeled vehicle adapted to the rails of
railroad
Examples: ['three cars had jumped the rails']

Synset: car.n.03
Definition: the compartment that is suspended from an
airship and that carries personnel and the cargo and the
power plant Examples: []

Synset: car.n.04
Definition: where passengers ride up and down
Examples: ['the car was on the top floor']

Synset: cable_car.n.01
Definition: a conveyance for passengers or freight on a
cable railway
Examples: ['they took a cable car to the top of the
mountain']

2. Finding Hypernyms (More General

Terms)

# Get the hypernyms (more general terms) for 'car' car_synset

= wn.synset('car.n.03') # First synset of 'car' hypernyms =
car_synset.hypernyms()

for hypernym in hypernyms:

22261A6634 55
 print(f"Hypernym: {hypernym.name()}")

OUTPUT: Hypernym: compartment.n.02

3. Finding Hyponyms (More Specific Terms)

# Get the hyponyms (more specific terms) for
'car' hyponyms = car_synset.hyponyms()

for hyponym in hyponyms:

print(f"Hyponym: {hyponym.name()}")

4. Finding Antonyms synsets =

wn.synsets('happy')

for synset in synsets:

for lemma in synset.lemmas(): if
lemma.antonyms(): print(f"Antonym:
{lemma.antonyms()[0].name()}") Output: Antonym:
unhappy

5. Word Similarity word1 =

wn.synset('dog.n.01') word2 =
wn.synset('cat.n.01')

similarity = word1.path_similarity(word2) print(f"Similarity

between 'dog' and 'cat': {similarity}")
Output: Similarity between 'dog' and 'cat': 0.2 Program 20:

Write a Python code to generate n-grams using NLTK n-gram Library Description:

An n-gram is a continuous sequence of n items (typically words or characters)

from a given text or speech. It's a fundamental concept in Natural Language Processing

22261A6634 56
 (NLP) and is used in many tasks, including language modeling, text analysis, speech
recognition, and machine translation. n represents the number of items (usually
words) in the sequence.

Types of N-Grams:

1. Unigram (1-gram):
o A unigram is simply a single word (or character) from the text.
o Example: "I love NLP"
§ Unigrams: ['I', 'love', 'NLP']
2. Bigram (2-gram):
o A bigram is a sequence of two consecutive words.
o Example: "I love NLP"
§ Bigrams: [('I', 'love'), ('love', 'NLP')]
3. Trigram (3-gram):
o A trigram is a sequence of three consecutive words.
o Example: "I love NLP"
§ Trigrams: [('I', 'love', 'NLP')]
4. Tetragram (4-gram):
o A tetragram is a sequence of four consecutive words.
o Example: "I love NLP very much"
§ Tetragrams: [('I', 'love', 'NLP', 'very'), ('love', 'NLP', 'very', 'much')]
5. Higher-order n-grams (n > 3):
o You can generate n-grams for any n. For instance, you can create
5grams, 6-grams, etc., depending on how much context you want to
capture.

Use of N-Grams in NLP:

• Language Modeling: In speech recognition or text generation, n-grams are used

to predict the likelihood of a word given its previous words. For example, in a
bigram model, the probability of the word "processing" might depend on whether
the previous word was "language".

22261A6634 57
 o Example:

§ Unigram Model: P(“language”) = count(“language”) / total words

§ Bigram Model: P(“language” | “natural”) = count(“natural language”)
/ count(“natural”)
• Text Classification: N-grams can be used as features for tasks like sentiment
analysis or spam detection.
• Text Analysis: N-grams help identify common phrases or collocations in the
text.
• Machine Translation: N-grams are used to predict translations by matching
ngrams in the source language with those in the target language.

Code:
import collections
import nltk
from nltk.util import ngrams
from nltk.tokenize import word_tokenize

def get_ngrams(text, n):

"""Generate n-grams from the given text."""
tokens = word_tokenize(text.lower())
n_grams = ngrams(tokens, n)
return list(n_grams)

# Example text text1 = "this is a sample text with several words this is another sample
text with some different words"
text = "Sample list of words"

# Generate Uni-grams
print("List of Unigram")
ngrams_list = get_ngrams(text,
1) for ngram in ngrams_list:
print(ngram)

# Generate bi-grams
print("List of Bigrams")
ngrams_list = get_ngrams(text, 2) for
ngram in ngrams_list:
print(ngram)

22261A6634 58
 # Generate tri-grams print("List of
Trigrams") ngrams_list =
get_ngrams(text, 3) for ngram in
ngrams_list:
print(ngram)

Output:
List of Unigram
('sample',)
('list',)
('of',)
('words',)

List of Bigrams
('sample', 'list')
('list', 'of')
('of', 'words')

List of Trigrams
('sample', 'list', 'of')
('list', 'of', 'words')

Program 21: Write a python program to train bi-gram model for a given corpus of
text to predict the next probable word given the previous two words of a
sentence.

Code:

import nltk from nltk.util import

ngrams from collections import
defaultdict
# Step 1: Prepare the corpus (tokenize sentences) corpus = [
"The weather is beautiful today",
"I am learning natural language processing",
"I am studing ML",
"Machine learning is fascinating",
"Language models are powerful tools",
"I love IITM”
"I am a Coder"

22261A6634 59
 "My passion is developing real world problem solving applications"
]

# Tokenize the corpus (split sentences into words) tokenized_corpus =

[nltk.word_tokenize(sentence.lower()) for sentence in corpus]

# Step 2: Train the bigram model def train_bigram_model(corpus):

model = defaultdict(lambda: defaultdict(int)) # Bigram model
for sentence in corpus:
# Generate bigrams for each sentence
bigrams = ngrams(sentence, 2) for w1,
w2 in bigrams:
model[w1][w2] += 1 # Increment the count for each bigram
return model

# Train the bigram model bigram_model =

train_bigram_model(tokenized_corpus)

# Step 3: Calculate word probabilities (bigram probabilities) def

calculate_word_probabilities(model):
probabilities = defaultdict(lambda: defaultdict(float))
for w1 in model: total_count =
float(sum(model[w1].values())) for w2 in
model[w1]:
probabilities[w1][w2] = model[w1][w2] / total_count # Probability of w2 after w1
return probabilities

# Calculate word probabilities word_probabilities =

calculate_word_probabilities(bigram_model)

# Step 4: Predict the next word given a context (bigram prediction) def
predict_next_word(model, context): if context[-1] in model:
next_word_probs = model[context[-1]] next_word =
max(next_word_probs, key=next_word_probs.get) return
next_word else:
return None

# Take user input for context user_input = input("Enter a sentence or context

(e.g., 'The bank'): ")

# Process the user input (split into words) context =

user_input.split()

22261A6634 60
 # Predict the next word based on the user input predicted_word =
predict_next_word(bigram_model, context)

# Print results print("\nPredicted next word given context '{}':".format("

".join(context)), predicted_word)

Output:
[nltk_data] Downloading package punkt_tab to /root/nltk_data... [nltk_data]
Package punkt_tab is already up-to-date!
Enter a sentence or context (e.g., 'The bank'): I love

Predicted next word given context 'I love': iitm

Program 22: Write a python program to train bi-gram model for a given corpus of

text to predict the next probable word given the previous two words of a

sentence. Aim: To write a python program to implement n-grams smoothing technique

Description: Incorporating smoothing into a bi-gram model helps to handle cases

where some bi-grams may not appear in the training corpus. Simple Additive Smoothing
(also known as Laplace Smoothing) method. This program will build n-grams from a
given text and apply smoothing to handle zero probabilities.

Implementation of bi-gram model with Laplace smoothing:

Steps to Incorporate Smoothing
• Count the Bi-grams and Unigrams:
o Count occurrences of each bi-gram.
o Count occurrences of each unigram (individual word).

• Calculate Probabilities with Laplace Smoothing:

o For each bi-gram, apply the formula:

22261A6634 61
 Steps are

1. get_ngrams: Tokenizes the input text and generates n-grams.

2. count_ngrams: Counts the occurrences of each n-gram.
3. laplace_smoothing: Applies Laplace smoothing to the n-gram counts. It adds 1
to each count and normalizes by the total count plus the vocabulary size to avoid
zero probabilities.
4. build_vocabulary: Builds a set of unique tokens (vocabulary) from the input
text..

Code:
import collections import
nltk
from nltk.util import ngrams
from nltk.tokenize import word_tokenize

def get_ngrams(text, n):

"""Generate n-grams from the given text."""
tokens = word_tokenize(text.lower())
n_grams = ngrams(tokens, n) return
list(n_grams)

def count_ngrams(ngrams):
"""Count the occurrences of each n-gram."""
ngram_counts = collections.Counter(ngrams) return
ngram_counts

def laplace_smoothing(ngram_counts, unigram_counts, vocab_size, n):

"""Apply Laplace smoothing to the n-gram counts."""
smoothed_probs = {}

for ngram in ngram_counts:

22261A6634 62
 context = ngram[:-1] context_count = unigram_counts[context] if n > 1 else
sum(unigram_counts.values()) smoothed_probs[ngram] = (ngram_counts[ngram] + 1)
/ (context_count + vocab_size)

return smoothed_probs

def build_vocabulary(text):
"""Build a vocabulary from the given text."""
tokens = word_tokenize(text.lower()) vocab
= set(tokens)
return vocab

# Example text
text = "this is a sample text with several words this is another sample text with some different
words"

# Set n for n-grams n = 2 #

For example, bi-grams #
Generate n-grams ngrams_list
= get_ngrams(text, n)
unigrams_list = get_ngrams(text, 1)

# Count n-grams and unigrams ngram_counts

= count_ngrams(ngrams_list)
unigram_counts = count_ngrams(unigrams_list)

# Build vocabulary vocab =

build_vocabulary(text)
vocab_size = len(vocab)

# Apply Laplace smoothing

smoothed_probs = laplace_smoothing(ngram_counts, unigram_counts, vocab_size, n)

# Output results print("N-grams with their

smoothed probabilities:") for ngram, prob in
smoothed_probs.items(): print(f"{ngram}:
{prob:.6f}")

Output:

N-grams with their smoothed probabilities:

('this', 'is'): 0.230769

22261A6634 63
 ('is', 'a'): 0.153846
('a', 'sample'): 0.166667
('sample', 'text'): 0.230769
('text', 'with'): 0.230769
('with', 'several'): 0.153846
('several', 'words'): 0.166667
('words', 'this'): 0.153846
('is', 'another'): 0.153846
('another', 'sample'): 0.166667
('with', 'some'): 0.153846
('some', 'different'): 0.166667
('different', 'words'): 0.166667

22261A6634 64