Application of NLP:Design of

Chatbot for New Research

Scholars Savinay E
U2008017
17th November 2023

ANSEL KANJIRAPPALLIL THOMAS

U2008017
17th November 2023
OUTLINE:
● INTRODUCTION
● MOTIVATION
● OBJECTIVE
● DATASET USED
● THEORETICAL BACKGROUND
● MODEL BUILDING PROCESS
● CHATBOT FRAMEWORK
● FINDINGS
● CONCLUSION
● LIMITATIONS AND FUTURE WORK
● REFERENCES
2
INTRODUCTION:
● Artificial Intelligence (AI) refers to technologies that enable machines to perform tasks
that typically require human intelligence.
● AI uses various techniques like machine learning, deep learning, and natural language
processing.
● Conversational agents, also known as chatbots, are AI applications that can engage in
conversations with users.
● Chatbots can be divided in two different ways:
○ Text-based vs. Speech-based
○ Retrieval-based vs. Generative

3
● Retrieval based bot uses a database of predefined responses that is used to answer the
user queries. In this type, the bot is trained to rank the best response from the finite set of
responses .This type of bot seldom makes mistakes and has higher accuracy as it is
completely based on predefined responses.
● Generative chatbot does not use any kind of predefined repository. This kind of bot
generates a response by itself by using natural language generation techniques. It is
trained using a database of past conversations, based upon which responses to the user
are generated. Generative chatbots can be created using RNN and attention mechanisms
techniques .
● RNN is Recurrent Neural Network which is a type of neural network where output of the
previous step is given as input of the current step.

4
MOTIVATION:
● The motivation for researching and developing chatbots for new scholars lies in
addressing the common challenges they face when entering the world of research.
● Scholars often struggle with where and how to begin, spending substantial time
searching for fundamental information on research processes, publications, and
resources.
● By creating an intelligent chatbot tailored to their needs, the research aims to streamline
this initial learning phase, providing prompt and accurate responses to scholars' queries,
ultimately saving valuable time and fostering more efficient research endeavors

5
OBJECTIVE:
● The aim of this research is to develop a chatbot leveraging simple neural network to assist
new research scholars.
● This innovative chatbot aims to address the specific information needs of research
scholars by providing prompt and satisfactory answers to common queries related to
research, publications, funding agencies, and more.
● Ultimately, the objective is to enhance the efficiency of research scholars by saving their
time and guiding them through the initial stages of their research journey.

6
DATASET USED:
The research uses question answer database that is generated by collecting data from web as
well as through personal interactions with new research scholars. The data is stored in JSON
format in which conversational intents are defined. Each conversational intent contains tag,
pattern, responses and context information. The dataset contains data about more than two
hundred research questions.

7
THEORETICAL BACKGROUND
● Neural networks, also known as artificial neural networks (ANNs) are a subset of machine
learning and are at the heart of deep learning algorithms.
● ANN consists of node layers, containing an input layer, one or more hidden layers, and an
output layer. Each layer consists of a number of artificial neurons called as nodes that are
connected to the nodes in next layer. The connections have associated weight.
● Net input to the neuron is computed using the following formula.

Where, wi is the weight of the connection and xi is the input.

8
MODEL BUILDING PROCESS
1.BUILDING VOCABULARY:

● Intent patterns are processed to build a vocabulary to be used in a model. Python

WordNetLemmatizer is used to collapse distinct word forms which are then translated into
bag of words.
● Intent patterns likely refer to predefined phrases or expressions that the model aims to
understand or respond to. These patterns can be statements or questions related to a
specific domain.
● The processed intent patterns are used to build a vocabulary, which is essentially a
collection of unique words present in the patterns.
● The WordNetLemmatizer is a tool in the Natural Language Toolkit (NLTK) library in Python.
It is used to lemmatize words, meaning it reduces them to their base or root form. This is
then converted to bag of words.
9
2.MODEL SPECIFICATION:

● Keras deep learning library is used to build a classification model. Keras sequential model is built that
consists of three layers, input, hidden and output layer.
● Popular ReLu(Rectified Linear Unit) activation is used for input and hidden layer whereas output layer
uses softmax activation function. The problem is about intent classification where classification output is
a multiclass array, which would help identify encoded intent.
● Adagrad optimizer with learning rate of 0.05 is used after experimenting with other optimizers.
● Categorical_crossentropy function is used to measure the loss.

10
3.EXPERIMENT WITH DIFFERENT OPTIMIZERS AT DIFFERENT LEARNING RATES:

● Optimizer and learning rate are the hyperparameters of the sequential model. The choice of
optimizer
and learning rate significantly affect the performance of the model.
● In the present work we attempted five optimizers namely Stochastic Gradient Descent (SGD),
Adagrad, Adadelta, Adam and RMSprop at different learning rates in order to optimize the
accuracy of the model.
● The model is implemented with these five optimizers and the accuracy of the model is
measured at five learning rate values of 0.001, 0.005, 0.01, 0.05 and 1.

11
● Stochastic Gradient Descent (SGD): A basic optimization algorithm that updates model parameters by moving
in the direction opposite to the gradient of the loss function, scaled by a learning rate, aiming to minimize the
loss iteratively.
● Adagrad: An adaptive optimization algorithm that adjusts the learning rates individually for each parameter
based on the historical gradients, allowing for larger updates for infrequently occurring features and smaller
updates for frequently occurring ones.
● Adadelta: A variant of Adagrad that seeks to overcome its diminishing learning rates by using a moving
average of squared parameter updates, enabling more stable convergence and eliminating the need for
manually setting a global learning rate.
● RMSprop: A variant of Adagrad that addresses its aggressive and monotonically decreasing learning rates by
applying a moving average to the squared gradients, enabling the algorithm to adapt to varying feature
scales and converge more reliably.
● Adam: An adaptive optimization algorithm that combines elements of momentum and RMSprop,
incorporating moving averages of both gradients and squared gradients, and using bias correction to provide
effective and efficient updates for a wide range of machine learning tasks.

12
ACCURACY OF MODEL FOR DIFFERENT OPTIMIZERS AT DIFFERENT LEARNING RATES

Adagrad optimizer with 0.05 learning rate gives the highest accuracy of
0.9908. This is why this optimizer at 0.05 learning rate is selected .
13
CHATBOT FRAMEWORK
● The developed chatbot is a Retrieval Based text chatbot. The chatbot has a repository of
predefined responses that is used to answer the user queries.
● The bot does the tasks of intent classification, entity identification and response selection.
● Based on the user query, an appropriate response will be selected from the repository and
delivered to the user.

14
1.CHATBOT GUI MODULE:

● This module creates a simple graphical interface for users to interact with a chatbot. Users input
their questions in a text box, but sometimes typos or spelling mistakes can cause issues.
● To tackle this, the module includes a spell check feature using the Python enchant library, which
automatically corrects spelling errors in user input. Before sending the user's question to the
main processing module, the input is checked for possible mistakes and corrected if needed.
● The chatbot then displays both the user's question and its response in a chat window. This spell
check feature improves the chatbot’s performance by making it more effective in understanding
and responding to user queries.

15
2.clean_up MODULE:

● The clean_up module is like a virtual janitor for the chatbot. When users type in their questions, the input might

have mistakes or unnecessary details. The clean_up module's job is to tidy up this input.

● It gets rid of any messiness in the sentence and figures out the important words called "tokens." Then, it smartly

finds the base form of these words, considering the context in which they are used.

● This process helps the chatbot better understand what the user is asking, making it easier for the bot to give a

helpful response. The clean_up module makes sure the input is neat and gets to the core of what the user is
saying.

16
3.bag_of_words MODULE:

● The chatbot cannot understand words directly, it need to be converted to numbers . This module helps the

chatbot convert the user's sentences into a numerical format.

● It does this by creating something called a "bag of words." It's like having a checklist of all the words the chatbot

knows.

● For each word in the sentence, it puts a "1" if the word is there and a "0" if it's not. So, you end up with a list of 0s

and 1s that represents the words in the sentence. This list of numbers is then sent to another module called
"classify" for further computer-like processing.

17
4.CLASSIFY MODULE:

● The classify module is like a detective for the chatbot. It gets a list of numbers (the bag of words) that represents
the words in the user's question.
● Then, it tries to figure out the probability of the question belonging to different categories or "intents." It's like the
detective guessing what the user is asking about.
● The module returns a list of intents that have a high probability of being the right one, based on a set accuracy
level. This list is then passed on to the response module, which decides how the chatbot should reply.

18
5.RESPONSE MODULE:

● The classify module is like a detective for the chatbot. It gets a list of numbers (the bag of words) that represents
the words in the user's question.
● Then, it tries to figure out the probability of the question belonging to different categories or "intents." It's like the
detective guessing what the user is asking about.
● The module returns a list of intents that have a high probability of being the right one, based on a set accuracy
level. This list is then passed on to the response module, which decides how the chatbot should reply.

19
5.RESPONSE MODULE:

The response module has two important jobs:

1.CONTEXT MAINTAINING:

● In regular human conversations, it's pretty natural to remember what was talked about earlier and keep
the discussion going smoothly. However, making chatbots do the same is quite challenging for
developers.
● The response module tackles this difficulty by using a dictionary, a kind of storage, to keep track of the
ongoing conversation. Each user gets a special ID, like a name tag, which helps the chatbot remember the
conversation details for different people at the same time.
● As soon as the intent and context are identified , the chatbot enters into a loop. The loop contains related
questions of the same subject , this makes the chatbot focused on the user queries.

20
2. TAG MATCHING

● The response module works like a librarian searching for the right book on a specific topic. It checks if the words
in your question match any tags stored in its database of intents.
● When it finds a match, it randomly picks a response from the list of possible answers related to that topic. If
there's no match with any tags, it means the chatbot doesn't have information on that particular topic, so it
returns an empty response.
● The response module helps the chatbot provide relevant answers based on the recognized topic tags, ensuring
meaningful and contextually appropriate replies.

21
FINDINGS
● Accuracy of Adam and RMSprop optimizers is inversely proportional to the learning rate.
● Accuracy of Adadelta is directly proportional to the learning rate.
● SGD and Adagrad optimizers have higher accuracy at medium values of learning rate.

22
CONCLUSION:
● In this study, a Keras Sequential model was implemented with five different optimizers, each at various
learning rates, to evaluate their impact on model performance. The results indicated that the Adagrad
optimizer outperformed the other four in terms of accuracy.
● The experiments also highlighted a relationship between learning rate and optimizer accuracy, showing
that the accuracy of Adam and RMSprop optimizers is inversely proportional to the learning rate, while
the accuracy of the Adadelta optimizer is directly proportional.
● Additionally, the introduction of a spell check feature significantly enhanced the bot's performance.
● The model demonstrated success in maintaining context during conversations, leading to a smoother
flow and increased user satisfaction.
● Despite its simplicity and computational efficiency, the model effectively addresses basic queries from
new researchers, saving time and effort. The inclusion of features like spell check and context
maintenance enhances the bot's responsiveness and utility for users.

23
LIMITATIONS AND FUTURE WORK:
The current model has some limitations, but there are potential areas for improvement and expansion in future
research:

● Real-time Spell Suggestions:

While the current model corrects spelling mistakes, enhancing it to provide spell suggestions as the user types
could further improve user experience and communication accuracy.
● Generative Chatbot Development:
The existing chatbot follows a retrieval-based approach, relying on predefined responses. Future work could
explore the development of a generative chatbot that has the ability to generate responses dynamically based
on the context and user input, providing a more flexible and dynamic interaction.
● Speech-based Interaction:
Considering that people often prefer audio communication, transforming the text-based chatbot into a
speech-based one could broaden its utility and cater to users who find spoken communication more
convenient.
● Mobile Application Deployment:
To align with user preferences for mobile apps and messengers, future work could focus on deploying the
chatbot as a mobile application or integrating it with popular messaging platforms like WhatsApp. This approach
aims to enhance user accessibility and encourage more widespread usage of the chatbot. 24
REFERENCES:
1. M. Dahiya (2017). “A Tool of Conversation: Chatbot,” International Journal of Computer Sciences and
Engineering,(IJCSE), 5(5), 2347-2693
2. S. Ghose, J. J. Barua (2013). “Toward the Implementation of a Topic Specific Dialogue Based Natural
Language Chatbot as an Undergraduate Advisor,” International Conference on Informatics, Electronics &
Vision, 1-5
3. R. P. Schumaker, H. Chen (2007). “Leveraging Question Answer Technology to Address Terrorism Inquiry,”
Decision Support Systems, 4(3), 1419-1430.
4. Anjana Tiha (2018). “Intelligent Chatbot using Deep Learning”, M.S. thesis, Dept. Comp. Sci.,Memphis
Univ.,Memphis.
5. Claus Mobus(2006). “Web communication with OpenSource chatbots, virtual fairs, rich media content,”
Springer- Verlag Berlin Heidelberg. https://doi.org/10.1007/3-540-29093-1.
6. Allison, DeeAnn (2011). "Chatbots in the Library: is it time?" Faculty Publications, UNL Libraries. 280. Available:
https://digitalcommons.unl.edu/libraryscience/280.
7. Vinothini Kasinatha(2020). “FLOW: INTELLIGENT FLOOD AID CHATBOT AS VIRTUAL FLOOD ASSISTANT,”
International Journal of Management, 11(10) 1749-1756.

25
8. Deshpande Aditya (2017). “A Survey of Various Chatbot Implementation Techniques,” International
Journal of Computer Engineering and Applications, 11.
9. Ramakrishna Kumar, Maha Mahmoud Ali (2020). “A Review on Chatbot Design and Implementation
Techniques,”International Research Journal of Engineering and Technology (IRJET), 7(2), 2791-2800.
10. Bertsimas Jeton Arifi, Markus Ebner, Martin Ebner (2019). “Potentials of Chatbots for Spell Check
among Youngsters,” iJAI, 1(1), 77-88.
11. B. P. Kiptonui (2013). “Chatbot Technology: A Possible Means of Unlocking Student Potential to
Learn How to Learn,” Educational Research. Journal of Computer and System Sciences, 74(3), 394–403.
12. J. Jia (2003). “The Study of the Application of a Keywords-based Chatbot System on the Teaching of
Foreign Languages”, Report of University of Augsburg, Augsburg, 1-36.
13. Garvit Bajpai, Rakesh Kumar Kannaujiya (2018). “CHATBOT IN PYTHON”, B Tech. thesis, Dept. IT, APJ
AKT Univ., Luckhnow, UP.
14. Ward, D (2005). “Why Users Choose Chat: A Survey of Behavior and Motivations”, Internet Reference
Services Quarterly, 10(1), 29-46.

26
THANK YOU

27