Proceedings of the 15th International Multi-Conference on Complexity, Informatics and Cybernetics (IMCIC 2024)

A Customer Service Chatbot Using Python, Machine

Learning, and Artificial Intelligence
Ty EBSEN
University of Arkansas Little Rock
Little Rock, AR 72204 USA

Richard S. SEGALL
Arkansas State University
State University, AR 72467 USA

Hyacinthe ABOUDJA
Oklahoma City University
Oklahoma City, OK 73106 USA

Daniel BERLEANT
University of Arkansas Little Rock
Little Rock, AR72204 USA

ABSTRACT The documentation is also available for public download

on SPPs website.
This report shows that with the most recent advancements
in Artificial Intelligence (AI) and Natural Language This purpose of this project is to build a customer service
Processing (NLP) using generative-pretrained question answering system that members can use to ask
transformers, we can develop robust AI applications to questions regarding SPPs APIs. The ultimate goal is to
assist customer service departments with question answer allow customer service representatives to increase their
systems. This paper addresses the question answering task efficiency by avoiding the need to answer the same
using an OpenAI Application Programming Interface questions repeatedly.
(API). This report examines how to create an AI question
answering application from documents that generated This project was done using an OpenAI API. We
correct answers to questions about those documents. We implemented the task in two different ways to support a
used two different approaches to create the question comparison. We used the OpenAI API directly and also
answering system. One was to use just the OpenAI API. used the LangChain framework. The LangChain
The other was to use the LangChain framework and framework used very little code to accomplish this task.
libraries. Both applications did answer questions correctly. However, a lot of the process was hidden in the
LangChain used less code with a higher learning curve. background. The OpenAI approach used more code, but
The OpenAI API used more code and provided more the answers were noticeably more detailed.
detailed answers.
Quality Assurance (QA) is any systematic process of
Keywords: Artificial Intelligence, Chatbot, Machine determining whether a product or service meets specified
Learning, Natural Language Processing, Python requirements. This report next looks at some of the related
work in the QA task area, then discusses the data collection
process, followed by the approaches taken to implement
1. BACKGROUND the QA system and a comparison between the two
approaches.
Southwest Power Pool (SPP) is a regional transmission
organization with 112 members and a footprint of 2. RELATED WORK
552,000-square-miles in 17 states It operates more than
70,000 miles of high-voltage transmission lines in the
A step-by-step tutorial on how to build your own AI
Eastern Interconnection. SPP also operates a Day-Ahead
energy market with 324 market participants. SPP hosts Chatbot with the ChatGPT API was published by Sha
(2023a). That article lists a few things to keep in mind
several APIs which require two-factor authentication to
when creating a chatbot. These include that (i) a chatbot
access. SPP’s customer service department receives
can be built on any platform; and therefore (ii) a powerful
numerous requests (many asking the same questions) from
computer is not needed to create a chatbot. Sha (2023a)
members requesting assistance with the APIs. Many of the
questions are in the documentation that SPP supplies used Python, OpenAI, and Gradio to build a chatbot.
members when they participate in SPPs energy market. Gradio is notable as a convenient way to demo machine
learning projects with a web interface. Chatbots, in
ISBN: 978-1-950492-78-7
ISSN: 2771-5914
148 https://doi.org/10.54808/IMCIC2024.01.148
 Proceedings of the 15th International Multi-Conference on Complexity, Informatics and Cybernetics (IMCIC 2024)

particular, can help convert a larger body of information works well with other popular languages such as French,
into short, focused, customized extracts (Berleant and Spanish, German, etc.
Berghel, 1994).
Step 4: As in the earlier guide, the author walks through
Sha (2023a) provided the following steps to set up the the process of downloading and setting up the
development environment for chatbot development. Steps development environment with Python, PIP, and Gradio.
1-3 provide instructions for downloading and installing Some new libraries are used in this guide. These include
Python on your local computer, adding Python.exe to the GPT Index, PyPDF2, PyCryptodone, and LangChain.
PATH environment variable and checking that the
installation was successful. The remaining steps are GPTIndex (also known as LlamaIndex) allows connecting
summarized individually next. external data to the LLM (Large Language Model). The
example in this article uses an older version of GPTIndex.
Step 4 is to upgrade PIP (Package Installer for Python).
PIP is installed as part of the normal installation. PIP is the PyPDF2 and PyCryptodome are Python libraries that will
Python package manager that allows the user to install allow using PDF documents as input to the project.
Python libraries. It is a good idea to upgrade PIP once the
installation is complete.
The Gradio library allows interaction with a Chatbot using
a web interface.
Step 5 is to install the OpenAI and Gradio libraries using
PIP. OpenAI provides the libraries that are used to access
Step 5: Downloading and installation of the desired code
OpenAI’s API. The Gradio libraries are used to create a
editor.
web interface.
Step 6: Walk-through of the process of obtaining an API
Step 6 is to download and install a code editor. The guide
key from OpenAI.
uses Notepad++, a simple and general purpose editor.
Step 7: Training and creating the chatbot. This is where
Step 7 is to obtain an OpenAI API (Application the chatbot developer begins using their own documents
Programming Interface) Key. To use the OpenAI API the
for the knowledge base. The “gpt-3.5-turbo” model was
developer must have an API key that is provided by
used because it is cheaper and faster than many other
OpenAI. There is more than one way to get the API. One
models.
can check the OpenAI web site for free offerings or,
alternatively, purchase credits. Sha walks developers
through the process of getting an API key from OpenAI. The instructions are to create a local folder called docs
which will store all the pdf files that will be used. Note that
text files and CSV files can also be stored. There is also a
Step 8 is where Sha (2023a) provides code showing how
way to add SQL database files and that process is
to get a chatbot up and running, using “gpt-3.5-turbo” for
explained in another article
the model. The developer can save this code and run it with (https://twitter.com/LangChainAI/status/1635304794335
Python. The instructions provided give the location of the 363072).
URL for the required page.
The larger the corpus of documents, the longer it takes to
Building the chatbot. Sha (2023b) follows up with further
process them. The amount of time also depends on the
instructions. These give a step-by-step guide to building an CPU (Central Processing Unit) and GPU (Graphics
AI chatbot using ChatGPT and constitute a good example Processing Unit) resources devoted to the task.
to how to consume documents that are used to create an AI
chatbot with a custom knowledge base.
3. PROJECT BASICS
Step 1: As in Sha (2023a), the author states that a chatbot
developer can build a chatbot on any platform and that the The Python code for this project is this:
guide is meant for such users.
from gpt_index import
Step 2: The author points out that since an AI is being SimpleDirectoryReader,
trained as part of the process, it is recommended to take GPTListIndex,
advantage of a good GPU due to the computational GPTSimpleVectorIndex,
demands of the training. LLMPredictor, PromptHelper
from langchain.chat_models import
ChatOpenAI
Step 3: The guide recommends that the data set be in the
import gradio as gr
English language for the example used. However, OpenAI

149
 Proceedings of the 15th International Multi-Conference on Complexity, Informatics and Cybernetics (IMCIC 2024)

import sys This code first produces a file called index.json using the
import os user defined construct_index function with all of the files
in the docs folder/directory.
os.environ["OPENAI_API_KEY"]
= 'Your API Key' PromptHelper is a GPT_index class that deals with Large
Language Model (LLM) context window token
def construct_index limitations. PromptHelper calculates the available context
(directory_path): size using the context window size of a LLM, reserve
max_input_size = 4096
token space for the prompt template, and the output.
num_outputs = 512
max_chunk_overlap = 20
chunk_size_limit = 600 A Large Language Model (LLM) is a type of Artificial
Intelligence (AI) algorithm that uses deep learning
prompt_helper = techniques and potentially massively large data sets to
PromptHelper(max_input_size, understand, summarize, generate and predict new content
num_outputs, (Kerner, 2023).
max_chunk_overlap,
chunk_size_limit The Chatbot function is used to take the question or the
=chunk_size_limit) input text and search the json.index for the answer. The
function returns the response. The iFace object is used to
llm_predictor = create the Gradio web interface that will accept a prompt
LLMPredictor and return an answer.
(llm=ChatOpenAI
(temperature=0.7, Obtaining an answer to a question. Shapiro (2022) notes
model_name="gpt-3.5-turbo", that “There is a need for answering questions from
max_tokens=num_outputs)) arbitrary volumes of data.” This is a general problem
statement for which the solution does not need to leverage
documents = knowledge about the type of document (Ding et al. 2006)
SimpleDirectoryReader or understanding a specific domain (Ding et al. 2005).
(directory_path).load_data() Shapiro refers to this as multi-document answering and
uses the U.S. Supreme Court opinion on Dobbs vs. Jackson
index = GPTSimpleVectorIndex which is about 450,000 characters long as the data source.
(documents,
OpenAI did have an answering API but it has been
llm_predictor=llm_predictor,
deprecated due to a lack of use. Shapiro’s Multi Document
prompt_helper=prompt_helper)
Answering code is at
https://github.com/daveshap/MultiDocumentAnswering.
index.save_to_disk('index.json')
return index
The build_index.py script creates an index.json file from
def chatbot(input_text): an input.txt file. The input.txt file contains the Dobbs vs
index = Jackson opinion. The build_index.py script splits the data
GPTSimpleVectorIndex.load_from_disk into 4000-word chunks. For each chunk an embedding is
('index.json') created using the OpenAI 'text-similarity-ada-001' engine.
response = index.query One way to think of this is as a vector with the text
(input_text, followed by the language representation. All chunks are
response_mode="compact") combined with its embedding and written out to the
return response.response index.json file.

iface = gr.Interface The answer_questions.py script is where the answers to

(fn=chatbot, our questions are generated. The script first does a vector
inputs=gr.components.Textbox search for the given question and searches for the answer
(lines=7, in the index.json file. A similarity score is calculated
label="Enter your text"), which is the dot product of the question vector and the
outputs="text", vectors that are searched in the index.json file. The results
title are sorted by similarity score in descending order and the
="Custom-trained AI Chatbot") top 20 are returned.

index = construct_index("docs")
iface.launch(share=True)

150
 Proceedings of the 15th International Multi-Conference on Complexity, Informatics and Cybernetics (IMCIC 2024)

The top 20 results from the index search are then sent to We researched different ways of developing this QA
the OpenAI completion API with the same question. The application, experimenting with using both the OpenAI
results are then summarized into a final answer. APIs and the LangChain libraries. We developed both
applications using Python. We developed the LangChain
Attention. Vaswani (2017) notes that “The self-attention application using Google Colaboratory notebook. We
mechanism is key to the transformer. The attention developed the OpenAI API application locally. It was
mechanism allows the model to weigh the importance of necessary to obtain an OpenAI API key to use in both
different elements (or tokens) in the input sequence when applications.
generating representations. This enables the model to
focus on relevant parts of the sequence, capturing long- The document “two-factor authentication technical
range dependencies and improving performance on tasks specifications v1.3 20170908.pdf” was used for this
that require understanding of context.” (Vaswani, 2017) project. It is possible to use multiple documents by
copying them into the “docs” folder of the project.
Using HuggingFace. HuggingFace (2023) discussed
question answering and provides open-source machine 5.1 LangChain
learning libraries. The question answering course is a step-
by-step guide to using the libraries to fine-tune a model. LangChain is a framework that is used to build AI
The guide uses the Stanford Question Answering Dataset applications. Although the code is much more condensed
(SQuAD) dataset released by Stanford University. The than the OpenAI version, the LangChain learning curve
SQuAD dataset is based on questions about Wikipedia was much steeper. The results using LangChain were just
articles. This is an extractive question-answer guide where as good as the OpenAI approach. This application was
the answer to the question is extracted from a given developed in a Google Colaboratory notebook. There are
context. several Python libraries that must be install first:
langchain, openai, pydf, tiktoken, and chromadb. One of
The dataset is loaded using the HuggingFace dataset the features of LangChain that helped greatly is that it is
library. A function is created to tokenize the questions and not necessary to do any conversion of the pdf files. The
context, generate the sequence_ids, and locate the start and chatbot developer can simply set the location of the pdf
end of the context. The function is then applied to the train files in the script and it reads each file in the directory,
split of the dataset. The model, train_dataset, and splits the files, and generates the embeddings.
validation dataset are used as inputs to the Trainer class in
the Hugging Face transformers library. The fine-tuned Here is listing of code with comments noted using #:
model was then pushed to Hugging Face and then used in
a pipeline to answer the questions. This process produced #import the OS library used for
answers that were nearly all reasonable. #reading the API key from a local
#file and setting the required
4. DATA COLLECTION #environment variable
#OPENAI_API_KEY.
PDF (Portable Document Format) files that contain import os
information related to making API calls using two-factor
authentication to Southwest Power Pools APIs were #Chroma is a vector data store that
downloaded from spp.org. These documents are available #allows us to store our embeddings.
publicly on spp.org. The data could potentially be from langchain.vectorstores
collected from the SPP customer service system, but that import Chroma
data is not publicly available and permission would have
to be obtained to use it outside of SPP. #OpenAIEmbeddings is used to create
#the embedding for the documents as
#well as the question.
5. METHODOLOGY
from langchain.embeddings.openai
import OpenAIEmbeddings
This project was developed using OpenAI APIs. OpenAI
is an Artificial Intelligence company that is part non-profit #TextSplitter allows us to split
and part for profit that was founded in 2015. In 2020 #our input data into chunks. We
OpenAI introduced GPT-3 (Generative Pre-trained #have to do this because the OpenAI
Transformer) a large language model that was trained on #models have limits to the number
large datasets from the internet that is aimed at question #of tokens that can be passed to
answering. #the API. For exampe the ada-001
#model the max number of tokens is
#2,049.

151
 Proceedings of the 15th International Multi-Conference on Complexity, Informatics and Cybernetics (IMCIC 2024)

from langchain.text_splitter Chroma.from_documents

import (texts, embeddings)
RecursiveCharacterTextSplitter qa = RetrievalQA.from_chain_type
(llm=OpenAI(),
#This is the LLM that is used from chain_type="stuff",
#OpenAI. retriever
from langchain.llms import OpenAI =docsearch.as_retriever())

#This library is used for query = "What is required for SPP\

#retrieving the answer two-factor authentication?”
from langchain.chains qa.run(query)
import RetrievalQA
5.2 OpenAI
#These two libraries are used to
#read a directory and load all PDF
#files. The OpenAI code it more involved than the LangChain
from langchain.document_loaders approach but it is easier to follow and understand what is
import PyPDFLoader happening.
from langchain.document_loaders
import DirectoryLoader pdf_2_txt.py: This Python script uses the PyPDF2 and OS
libraries to read all the pdf files in a directory, convert them
#open file function used to open to text, and save them to a file call input_docs.txt. The
#the file that contains the input_docs.txt will be the input to the create_index.py
#OpenAI API KEY script.
def open_file(filepath):
with open(filepath, 'r', import PyPDF2
encoding='utf-8') import os
as infile: pdf_path =
return infile.read() '/Users/tyebsen/projects/docs/'
for filename in os.listdir(pdf_path):
if filename.endswith('.pdf'):
#Sets the environment variable reader = PyPDF2.PdfReader
#OPENAI_API_KEY (os.path.join(pdf_path,
os.environ['OPENAI_API_KEY'] filename))
= open_file('openaiapikey.txt') for i in range(len(reader.pages)):
page = reader.pages[i]
#This code loads all pdf files that page = page.extract_text()
#are in the docs directory page = " ".join(page.split())
loader = print(page)
DirectoryLoader file1 =
open(r"input_docs.txt","a")
('./docs/', file1.writelines(page)
glob="./*.pdf", file1.close()
loader_cls=PyPDFLoader)
documents = loader.load()
Create_index.py: This Python script reads the
#splitting the text into 1000 byte input_docs.txt file, and breaks the data into 3,000-byte
#chunks. chunks. For each chunk, it generates the embedding,
text_splitter = saving the text and its embedding in a json formatted file
RecursiveCharacterTextSplitter called index.json. This file is used by the qa.py Python
(chunk_size=1000, script to answer questions about the pdf files. We use
chunk_overlap=200) 3,000-byte chunks because it is safely under the model
texts = prompt limit of 4,000 bytes.
text_splitter.split_documents
(documents) Import openai
import json
#creates the embeddings and creates import textwrap
#a Chroma docsearch object that
#will be used to retrieve the def open_file(filepath):
#answer to the question. with open(filepath, 'r',
embeddings = OpenAIEmbeddings() encoding='utf-8')
docsearch = as infile: return infile.read()

152
 Proceedings of the 15th International Multi-Conference on Complexity, Informatics and Cybernetics (IMCIC 2024)

openai.api_key = openai.api_key =
open_file('openaiapikey.txt') open_file('openaiapikey.txt')

def gpt3_embedding def gpt3_embedding

(content, (content,
Engine engine=
='text-embedding-ada-002'): 'text-embedding-ada-002'):
response = content = content.encode
openai.Embedding.create (encoding='ASCII',
(input=content,engine=engine) errors='ignore'
vector = # this is a normal list ).decode()
response['data'][0]['embedding'] response =
return vector openai.Embedding.create
(input=content,engine=engine)
if __name__ == '__main__': vector = #this is a normal list
alltext = response['data'][0]['embedding']
open_file('input_docs.txt') return vector
chunks =
textwrap.wrap(alltext, 3000) def similarity(v1, v2):
result = list() # return cosine similarity
for chunk in chunks: return
embedding=gpt3_embedding np.dot(v1,v2)/norm(v1)*norm(v2)
(chunk.encode
(encoding= def search_index(text, data,
'ASCII', count=20):
errors='ignore' vector = gpt3_embedding(text)
).decode() scores = list()
) for i in data:
info = score = similarity(vector,
{'content': chunk, i['vector'])
'vector': embedding} scores.append
print(info, '\n\n\n') ({'content': i['content'],
result.append(info) 'score': score})
with open('index.json', 'w') ordered = sorted
as outfile: (scores,
json.dump(result, outfile, key=lambda d: d['score'],
indent=2) reverse=True)
return ordered[0:count]
qa.py: This Python script is heavily based on Shapiro’s
multi-document project that scans all the documents with def gpt3_completion
the same question and returns the top 20 results. Shapiro (prompt,
used a dot product to determine the score. We updated the engine='text-davinci-002',
script to use cosine similarity for the score. Cosine temp=0.6, top_p=1.0,
similarity is what OpenAI recommends. The final answer tokens=2000, freq_pen=0.25,
is generated from a detailed summary of the 20 results. pres_pen=0.0, stop=['<<END>>']):
max_retry = 5
retry = 0
import openai prompt =
import json prompt.encode
import numpy as np (encoding='ASCII',
import textwrap errors='ignore'
import re ).decode()
from time import time,sleep while True:
from numpy.linalg import norm try:
response =
def open_file(filepath): openai.Completion.create
with open(filepath, 'r', (engine=engine,
encoding='utf-8') prompt=prompt,
as infile: return infile.read() temperature=temp,

153
 Proceedings of the 15th International Multi-Conference on Complexity, Informatics and Cybernetics (IMCIC 2024)

max_tokens=tokens, all_answers =
top_p=top_p, '\n\n'.join(answers)
frequency_penalty= chunks = textwrap.wrap
freq_pen, (all_answers, 10000)
presence_penalty= final = list()
pres_pen, for chunk in chunks:
stop=stop prompt =
) f"Write a detailed summary of the \
text = response following: {chunk}"
['choices'][0]['text'] summary =
.strip() gpt3_completion(prompt)
text = final.append(summary)
re.sub('\s+', ' ', text) print('\n\n=========\n\n',
filename = '\n\n'.join(final))
'%s_gpt3.txt' % time()
with open( 6. RESULTS
'gpt3_logs/%s'
% filename,
'w') Both approaches generated reasonable answers. However,
as outfile: the OpenAI approach generated much more detailed
outfile.write('PROMPT:\n\n' answers. The OpenAI approach required much more code,
+ prompt + but we believe it provides a better understanding of what
'\n\n==========\n\nRESPONSE:\n\n' is actually happening with OpenAI. In comparison, the
+ text) LangChain framework hides a lot of what is going on
return text behind the scenes.
except Exception as oops:
retry += 1 We also scored each predicted answer with the actual
if retry >= max_retry: answer from the documents using the cosine similarity
return "GPT3 error: %s" calculation. The LangChain answers scored better against
% oops the actual answers than the OpenAI method. Table 1 shows
print the LangChain results.
('Error communicating \
with OpenAI:', 7. CONCLUSIONS
oops)
sleep(1)
This project shows that, using the latest advances in
Artificial Intelligence (AI) and Natural Language
if __name__ == '__main__':
Processing (NLP), a chatbot developer can produce a
with open('index.json', 'r')
robust QA (Quality Assurance) system that can be used to
as infile:
assist customers. OpenAI was selected for use because is
data = json.load(infile)
while True: currently leading in the AI Natural Language Processing
query = input("Enter your \ (NLP) field with generative-pretrained transformer
question here: ") models. OpenAI does charge a small fee for the use of their
results = Application Programming Interfaces (APIs), but it was
search_index(query, data) negligible for this project.
answers = list()
#answer the same question We also chose the LangChain framework because of its
#for all returned chunks ability to consume multiple documents. Although a lot of
for result in results: the processing is done behind the scenes, it is a powerful
prompt=f"Use the following \ set of libraries that can be used to build AI applications
passage to give a detailed answer \ albeit with a bigger learning curve than the OpenAI API.
to the question:\n\QUESTION: \
{query}\n\nPASSAGE: \ The OpenAI version needs some performance
{result['content']}\n\nDETAILED \ enhancements for a production system, as it otherwise can
ANSWER:" take too long to produce an answer. One thing to look at is
answer = lowering the number of results returned from the
gpt3_completion(prompt) search_index function from 20. This could however
answers.append(answer) possibly reduce the accuracy of the results. Further testing
#summarize the would need to be done. Further research could also be done
#answers together

154
 Proceedings of the 15th International Multi-Conference on Complexity, Informatics and Cybernetics (IMCIC 2024)

using vector stores like Chroma rather than a JSON 1946391. The content reflects the views of the authors and
(JavaScript Object Notation) formatted file. The not necessarily the NSF.
LangChain version performs much better.
10. REFERENCES
For a production system, we would probably recommend
the LangChain version. The code would be easier to Berleant, D. and Berghel, H. (1994). Customizing
maintain, the performance better, and using multiple pdf Information: Part 1, Getting what We Need, when We
files (or any other type of files) is easier since it is only Need It. Computer, vol. 27, no. 9, pp. 96–98,
necessary to drop them into a folder without having to do https://ieeexplore.ieee.org/document/312053.
format conversion or other manipulation. Cheng, R. (2021). Question Answering with Pretrained
Transformers Using PyTorch. Toward Data Science.
We conclude that Southwest Power Pool (SPP) and other [Online] January 19, 2021.
organizations could benefit from an AI question answering https://towardsdatascience.com/.
chatbot system to assist users with their API questions. Cyca, M. (2022). Chatbot Analytics 101: Essential
Additional performance, load, and accuracy testing would Metrics to Track. https://blog.hootsuite.com/chatbot-
need to be done prior to production. User testing would analytics/. [Online] September 21, 2022.
also need to be done. Security would also be a Ding, J., Hughes, L.M., Berleant, D., Fulmer, A.W.,
consideration in a project like this, as hackers should not Wurtele, E.S. (2006). PubMed Assistant: A Biologist-
be able to use such a service to gain access to internal Friendly Interface for Enhanced PubMed Search,
organization systems. Bioinformatics, vol. 22, issue 3, pp. 378–380,
https://doi.org/10.1093/bioinformatics/bti821.
It is important to note that SPP is a non-profit organization. Ding, J., Viswanathan, K., Berleant, D., Hughes, L.,
A project that costs less than a certain limit would not Wurtele, E.S., Ashlock, D., Dickerson, J.A., Fulmer,
normally be subject to a cost-based analysis. Also, these A., Schnable, P.S. (2005). Using the Biological
types of projects would best be discussed in multiple Taxonomy to Access Biological Literature with
working groups made up of SPP members and staff to PathBinderH, Bioinformatics, vol. 21, issue 10, pp.
determine the viability of the project, as could be 2560–2562,
recommended for other organizations as well. https://doi.org/10.1093/bioinformatics/bti381.
HuggingFace (2023). Question Answering. Hugging
To calculate the “people cost,” the rate of $41.00 was used Face. [Online] 2023. https://huggingface.com.
Kerner, S.M. (2023). Large Language Model. Retrieved
as it is published in the Department of Labor “Employer
August 6, 2023 from
Costs for Employee Compensation – March 2023” report.
https://www.techtarget.com/whatis/definition/large-
The Open AI cost was priced at 12,000 transactions at
language-model-LLM.
approximately $0.00700 per transaction (https://gptforwo
Khanna, C. (2021). Question Answering with a fine-
rk.com/tools/openai-chatgpt-api-pricing-calculator).
tuned BERT. Towards Data Science. [Online] May
15, 2021. https://towardsdatascience.com/.
8. FUTURE DIRECTIONS Sha, A. (2023a). How to Build Your Own AI Chatbot
With ChatGPT API: A Step-by-Step Tutorial.
Additional research could be done in providing metrics for beebom.com. [Online] June 19, 2023.
a production QA (Quality Assurance) system. These https://beebom.com/how-build-own-ai-chatbot-with-
metrics would be used to determine how the QA system is chatgpt-api/.
performing based on metrics such as conversation length, Sha, A. (2023b). How to Train an AI Chatbot With
number of conversations, number of unique users, human Custom Knowledge Base Using ChatGPT API.
takeover rate, and others. Michelle Cyca describes these beebom.com. [Online] June 14, 2023.
metrics in the article “Chatbot Analytics 101: Essential https://beebom.com/how-train-ai-chatbot-custom-
Metrics to Track” (Cyca, 2022). knowledge-base-chatgpt-api/.
Shapiro, D. (2022). Answer complex questions from an
More research could also be done with different OpenAI arbitrarily large set of documents with vector search
models. This project used the text-embedding-ada-002 and GPT-3. youtune.com. [Online] June 25, 2022.
engine for the embeddings and text-divinci-002 for the https://www.youtube.com/watch?v=es8e4SEuvV0.
completions. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J.,
Jones, L.. Gomez, A.N., Kaiser, L., Polosukhin, I.
9. ACKNOWLEDGMENT (2017). Attention Is All You Need. In: NIPS'17:
Proceedings of 31st International Conference on
Neural Information Processing Systems. Dec.2021.
Publication of this work was supported in part by the
https://proceedings.neurips.cc/paper_files/paper/2017/
National Science Foundation under Award No. OIA-
file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.

155
 Proceedings of the 15th International Multi-Conference on Complexity, Informatics and Cybernetics (IMCIC 2024)

Table 1: LangChain Results

Ques on Answer LangChain Answer COSINE Similarity Score

1) What is required In order to authenticate, A valid x509 client 0.849
for SPP two-factor users or applications Cer ficate Authority (CA)
authen ca on? accessing the User that is trusted by SPP and a
Interfaces (UIs) and APIs login ID (Iden fica on) and
must present a valid x509 password.
certificate and present a
second authentication factor.
2) What is the In order to authenticate, The integrated marketplace 0.933
integrated users or applications two-factor authen ca on
marketplace two- accessing the UIs and APIs requires users to present a
factor must present a valid x509 valid x509 cer ficate and a
authen ca on? certificate and present a second authen ca on
second authentication factor. factor, such as a login ID
and password.
3) What is the format The concatenated token, The request header consists 0.954
of the request made up of the Request of the Request Created
header? Created Timestamp, Timestamp, Request Nonce,
Request Nonce, and and HMAC Sha512 Hash,
Hash-based message concatenated together and
authentication code(HMAC) separated by hyphens.
Sha512 Hash, concatenated
together and separated by
hyphens.

4) What is the format requestCreated + “-“ + The token is a Nonce and 0.823
of the token? requestNonce + “-“ HMAC Sha512 Hash,
concatenated together and
separated by hyphens.
5) What is the format yyyy-Mdd’T’HH:mm:ss’Z’ The format of the 0.880
of the mestamp? where ‘T’ is the literal mestamp is yyyy-MM-
character T and ‘Z’ is the dd'T'HH:mm:ss'Z', with yyyy
literal character Z. being the 4 digit year, MM
being the 2 digit month (01-
12), dd being the 2 digit day
(01-31), HH being the 2
digit hour (00-23), mm
being the 2 digit minute
(00-59), and ss being the 2
digit second (00-50).

156