Large language models (LLMs) such as Llama2 have been shown effective for question-answering (Touvron et al., 2023), however, they are often limited by their knowledge in certain domains. A common technique here is to augment LLM's knowledge with documents that are relevant to the question. In this assignment, you will develop a retrieval augmented generation system (RAG) (Lewis et al., 2021) that's capable of answering questions about Pittsburgh and CMU, including history, culture, trivia, and upcoming events.

Q: Who is Pittsburgh named after?
A: William Pitt

Q: What famous machine learning venue had its first conference in Pittsburgh in 1980?
A: ICML

Q: What musical artist is performing at PPG Arena on October 13?
A: Billie Eilish

So far in your machine learning classes, you may have experimented with standardized tasks and datasets that were easily accessible. However, in the real world, NLP practitioners often have to solve a problem from scratch (like this one!). This includes gathering and cleaning data, annotating your data, choosing a model, iterating on the model, and possibly going back to change your data. In this assignment, you'll get to experience this full process.

Please note that you'll be building your own system end-to-end for this assignment, and there is no starter code. You must collect your own data and develop a model of your choice on the data. We will be releasing the inputs for the test set a few days before the assignment deadline, and you will run your already-constructed system over this data and submit the results. We also ask you to follow several experimental best practices, and describe the result in your report.

The key checkpoints for this assignment are,

• Understand the task specification
• Prepare your raw data
• Annotate data for model development
• Develop a retrieval augmented generation system
• Generating results
• Write a report
• Submit your work

All deliverables are due by Tuesday, October 22nd. This is a group assignment, see the assignment policies for this class.¹

You'll be working on the task of factual question-answering (QA). We will focus specifically on questions about various facts concerning Pittsburgh and CMU. Since existing QA systems might not have the necessary knowledge in this domain, you will need to augment each question with relevant documents. Given an input question, your system will first retrieve documents and use those documents to generate an answer.

Input (questions.txt): A text file containing one question per line.

Output (system_output.txt): A text file containing system generated answers. Each line contains a single answer string generated by your system for the corresponding question from questions.txt.

Reference (reference_answers.txt): A text file containing reference answers. Each line contains one or more reference answer strings for the corresponding question from questions.txt.

Read our model and data policy for this assignment.

For your test set and the RAG systems, you will first need to compile a knowledge resource of relevant documents. You are free to use any publicly available resource, but we highly recommend including the websites below. Note that we can also ask you questions from relevant subpages (e.g. "about", "schedule", "history", "upcoming events", "vendors", etc.) from these websites:

• General Info and History of Pittsburgh/CMU
  • Wikipedia pages (Pittsburgh, History of Pittsburgh).
  • City of Pittsburgh webpage
  • Encyclopedia Brittanica page
  • Visit Pittsburgh webpage: This website also contains subpages that would be useful for other topics (see below), like events, sports, music, food, etc.
  • City of Pittsburgh Tax Regulations: See the links under the "Regulations" column of the table
  • City of Pittsburgh 2024 Operating Budget
  • About CMU & CMU History
• Events in Pittsburgh and CMU (We will only ask about annual/recurring events and events happening after October 27.)
  • Pittsburgh events calendar: Navigate to month-specific pages for easier scraping
  • Downtown Pittsburgh events calendar
  • Pittsburgh City Paper events
  • CMU events calendar and campus events page
• Music and Culture (Note that many of these pages also contain upcoming events, also see Wikipedia pages for each.)
  • Pittsburgh Symphony, Opera, and Cultural Trust
  • Pittsburgh Museums (Carnegie Museums, Heinz History Center), The Frick, and more)
  • Food-related events
    • Food Festivals
    • Picklesburgh
    • Pittsburgh Taco Fest
    • Pittsburgh Restaurant Week
    • Little Italy Days
    • Banana Split Fest
• Sports (Note that many of these pages also contain upcoming events, also see Wikipedia pages for each. Don't worry about scraping news/scores/recent stats from these sites.)
  • General info (Visit Pittsburgh)
  • Pittsburgh Pirates, Steelers, and Penguins

Your knowledge resource might include a mix of HTML pages, PDFs, and plain text documents. You will need to clean this data and convert it into a file format that suites your model development. Here are some tools that you could use:

• To process HTML pages, you can use beautifulsoup4.
• To parse PDF documents into plain text, you can use pypdf or pdfplumber.

By the end of this step, you will have a collection of documents that will serve as the knowledge resource for your RAG system.

Next, you will want to annotate question-answer pairs for two purposes: testing/analysis and training. Use the documents you compiled in the previous step to identify candidate questions for annotation. You will then use the same set of documents to identify answers for your questions.

The testing (and analysis) data will be the data that you use to make sure that your system is working properly. In order to do so, you will want to annotate enough data so that you can get an accurate estimate of how your system is doing, and if any improvements to your system are having a positive impact. Some guidelines on this,

• Domain Relevance: Your test data should be similar to the data that you will finally be tested on (questions about Pittsburgh and CMU). Use the knowledge resources mentioned above to curate your test set.
• Diversity: Your test data should cover a wide range of questions Pittsburgh and CMU.
• Size: Your test data should be large enough to distinguish between good and bad models. If you want some guidelines about this, see the lecture on experimental design and human annotation.²
• Quality: Your test data should be of high quality. We recommend that you annotate it yourself and validate your annotations within your team.

To help you get started, here are some example questions,

• Questions that could be answered by just prompting a LLM
  • When was Carnegie Mellon University founded?
• Questions that can be better answered by augmenting LLM with relevant documents
  • What is the name of the annual pickle festival held in Pittsburgh?
• Questions that are likely answered only through augmentation
  • When was the Pittsburgh Soul Food Festival established?
• Questions that are sensitive to temporal signals
  • Who is performing at X venue on Y date?

See Vu et al., 2023 for ideas about questions to prompt LLMs. For questions with multiple valid answers, you can include multiple reference answers per line in reference_answers.txt (separated by a semicolon ;). As long as your system generates one of the valid answers, it will be considered correct.

This test set will constitute data/test/questions.txt and data/test/reference_answers.txt in your submission.

The choice of training data is a bit more flexible, and depends on your implementation. If you are fine-tuning a model, you could possibly:

• Annotate it yourself manually through the same method as the test set.
• Do some sort of automatic annotation and/or data augmentation.
• Use existing datasets for transfer learning.

If you are using a LLM in a few-shot learning setting, you could possibly:

• Annotate examples for the task using the same method as the test set.
• Use existing datasets to identify examples for in-context learning.

This training set will constitute data/train/questions.txt and data/train/reference_answers.txt in your submission.

An important component of every data annotation effort is to estimate its quality. A standard approach is to measure inter-annotator agreement (IAA). To measure this, at least two members of your team should annotate a random subset of your test set. Compute IAA on this subset and report your findings.

Unlike assignment 1, there is no starter code for this assignment. You are free to use any open-source model and library, just make sure you provide due credit in your report. See our model policy.

For your RAG system, you will need the following three components, 

1. Document & query embedder
2. Document retriever
3. Document reader (aka. question-answering system)

To get started, you can try langchain's RAG stack that utilizes GPT4All, Chroma and Llama2, as well as LlamaIndex.

Some additional resources that could be useful,

• 11711 lecture notes
• ACL 2023 tutorial on retrieval-augmented LMs
• llama-recipes for an example RAG chatbot with Llama2.
• Ollama or llama.cpp to run LLMs locally on your machine.

All the code for your data preprocessing, model development and evaluation will be a part of your GitHub repository (see submission for details).

Finally, you will run your systems on our test set (questions only) and submit your results to us. This test set will be released the day before the assignment is due (Monday, October 21st).

This test set will be curated by the course staff and will evaluate your system's ability to respond to a variety of questions about Pittsburgh and CMU. Because the goal of this assignment is not to perform hyperparameter optimization on this private test set, we ask you to not overfit to this test set. You are allowed to submit up to three output files (system_outputs/system_output_{1,2,3}.txt). We will use the best performing file for grading.

Your submissions will be evaluated on standard metrics, answer recall, exact match and F1. See section 6.1 of the original SQuAD paper for details. These metrics are token-based and measure the overlap between your system answer and the reference answer(s). Therefore, we recommend keeping your system generated responses as concise as possible.

We ask you to write a report detailing various aspects about your end-to-end system development (see the grading criteria below).

There will be a 7 page limit for the report, and there is no required template. However, we encourage you to use the ACL template.

Submit all deliverables on Canvas. Your submission checklist is below,

• Your report.
• A link to your GitHub repository containing your code.³
• A file listing contributions of each team member,
  • data annotation contributions from each team member (e.g. teammate A: instances 1-X; teammate B: instances X-Y, teammate C: instances Y-Z).
  • data collection (scraping, processing) and modeling contributions from each team member (e.g. teammate A: writing scripts to ..., implementing ...; teammate B:...; teammate C:...;)
• Testing and training data you annotated for this assignment.
• Your system outputs on our test set.

Your submission should be a zip file with the following structure (assuming the lowercase Andrew ID is ANDREWID). Make one submission per team.

ANDREWID/
├── report.pdf
├── github_url.txt
├── contributions.md
├── data/
│   ├── test/
│   │   ├── questions.txt
│   │   ├── reference_answers.txt
│   ├── train/
│   │   ├── questions.txt
│   │   ├── reference_answers.txt
├── system_outputs/
│   ├── system_output_1.txt
│   ├── system_output_2.txt (optional)
│   ├── system_output_3.txt (optional)
└── README.md

The following points (max. 100 points) are derived from the results and your report. See course grading policy.⁴

• Submit data (15 points): submit testing/training data of your creation.
• Submit code (15 points): submit your code for preprocessing and model development in the form of a GitHub repo. We may not necessarily run your code, but we will look at it. So please ensure that it contains up-to-date code with a README file outlining the steps to run it. Your repo
• Results (30 points): points based on your system's performance on our private test set. 10 points for non-trivial performance,⁵ plus up to 20 points based on level of performance relative to other submissions from the class.
• Report: below points are awarded based on your report.
  • Data creation (10 points): clearly describe how you created your data. Please include the following details,
    • How did you compile your knowledge resource, and how did you decide which documents to include?
    • How did you extract raw data? What tools did you use?
    • What data was annotated for testing and training (what kind and how much)?
    • How did you decide what kind and how much data to annotate?
    • What sort of annotation interface did you use?
    • How did you estimate the quality of your annotations? (IAA)
    • For training data that you did not annotate, did you use any extra data and in what way?
  • Model details (10 points): clearly describe your model(s). Please include the following details,
    • What kind of methods (including baselines) did you try? Explain at least two variations (more is welcome). This can include which model you used, which data it was trained on, training strategy, etc.
    • What was your justification for trying these methods?
  • Results (10 points): report raw numbers from your experiments. Please include the following details,
    • What was the result of each model that you tried on the testing data that you created?
    • Are the results statistically significant?
  • Analysis (10 points): perform quantitative/qualitative analysis and present your findings,
    • Perform a comparison of the outputs on a more fine-grained level than just holistic accuracy numbers, and report the results. For instance, how did your models perform across various types of questions?
    • Perform an analysis that evaluates the effectiveness of retrieve-and-augment strategy vs closed-book use of your models.
    • Show examples of outputs from at least two of the systems you created. Ideally, these examples could be representative of the quantitative differences that you found above.

To make the assignment accessible to everyone,

• You are only allowed to use models that are also accessible through HuggingFace. This means you may not use closed models like OpenAI models, but you can opt to use a hosting service for an open model (such as the Hugging Face or Together APIs).
• You are only allowed to include publicly available data in your knowledge resource, test data and training data.
• You are welcome to use any open-source library to assist your data annotation and model development. Make sure you check the license and provide due credit.

If you have any questions about whether a model or data is allowed, please ask on Piazza.

This assignment was based on the the Spring 2024 version of this assignment.

• Lewis et al., 2021. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.
• Touvron et al., 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models.
• Vu et al., 2023. FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation.