• Lectures: DC2568, Friday: 1:30 PM – 4:20 PM

Hong Zhang

Email: honzhang at uwaterloo dot ca (to submit presentation slides and paper summaries)

Office: DC3530 (office hours by appointment)

Generative AI (GenAI) applications are revolutionizing the world. The latest GenAI models such as GPT-4 have achieved unprecedented performance in various tasks such as code generation, text classification, and problem reasoning. However, serving GenAI applications, i.e., deploying trained GenAI models on a compute cluster and conducting model inference for incoming user requests, presents challenges in systems design.

This seminar-based course will introduce you to the key concepts and the state-of-the-art in model serving systems for emerging Generative AI (GenAI) and encourage you to think about either building new tools or applying an existing one in your own research. The course will cover various important topics for serving systems for GenAI, including efficient batching, memory/cache management, request scheduling and load balancing, and compound AI systems such as Retrieval-Augmented Generation. Note that this course is NOT focused on AI methods. Instead, we will focus on how to build efficient serving systems for existing AI methods.

Students are expected to have strong programming skills and have completed at least one undergraduate-level systems-related course, such as Operating Systems, Databases, Distributed Systems, or Computer Networks. While an undergraduate course in Machine Learning or Artificial Intelligence would be beneficial, it is not a requirement.

This course has no textbooks or exams. We will read recent papers to understand trends and important topics in serving systems for GenAI.

This is an evolving list and the schedule is subject to changes.

Before Each Lecture: Each lecture will have three required readings that everyone must read and will be presented in the class. For some lectures, there will be some background readings providing necessary knowledge for the corresponding topic. There are also some optional related reading(s) under the theme. They are optional for the class.

During Lectures: Active participation is crucial for both your own understanding and to improve the overall quality of the course. You are expected to attend all lectures, and more importantly, participate in class discussions. Not everyone must have add something every day, but it is expected that everyone has something to share over the semester.

After Lectures: Participation also involves contributing to discussions on Piazza. The group responsible for the summary should initiate the (remaining) discussion, and the rest of the members are encouraged to participate.

The course will be structured as a seminar. Each student will be assigned at least one paper to present over the semester. Only the students assigned will present in each class. Since there are three required readings, there will be three student presenters assigned each day. Presentations for each paper should last around 35-45 minutes without interruption. However, presenters should expect questions and interruptions throughout. The presenters are free to come up with separate presentations or work together to merge their presentations. However, the goal of your presentation must be the following:

• Provide necessary background and motivate the problem. Note that in principle, each lecture has a “theme” such that the papers are connected in some way. For instance, perhaps they are trying to solve the same problem using different approaches, or maybe one is building on top of the other. Your presentation should try to make this connection.
• Clearly describe the problem the paper solves and the corresponding challenges.
• Present the high-level idea, approach, and/or insight (using examples, whenever appropriate) in the required reading.
• Discuss technical details so that one can understand key details without carefully reading.
• Explain the differences between related works.
• Identify strengths and weaknesses of the required reading and propose directions of future research.

**Note: ** The slides for a presentation must be emailed to the instructor at least 24 hours before the corresponding class (in *.pptx format). Do not just re-use slides provided by the paper authors. You may borrow, with attribution, figures, and animations, but your slides should be created independently.

To foster a deeper understanding of the papers and to encourage critical thinking, each lecture (from Lecture 3) will be followed by a panel discussion. This discussion will feature three distinct roles played by different student groups, simulating an interactive and dynamic scholarly exchange.

1. The Authors

• The students who present will play the role of the authors.
• Responsibility: As authors, you are expected to defend your paper against critiques, answer questions, and discuss how you might improve or extend your research in the future, akin to writing a rebuttal during the peer-review process.

2. The Reviewers

• Additional students will be assigned to be the “reviewers” for the required papers in each class.
• Responsibility: Each reviewer will write a detailed review for the paper assigned. The review must be 3-4 pages long. Reviewers critically assess the paper, posing challenging questions and highlighting potential weaknesses or areas for further investigation. Your goal is to engage in a constructive critique of the paper, simulating a peer review scenario.
• You must use this template for your review. You should submit your reviews before the lecture. Late submissions will not be entertained.

3. Rest of the Class

• Responsibility:
  • During the panel discussions, feel free to actively ask questions and engage in the dialogue.

You will have to complete substantive work on an instructor-approved problem and have original contributions. Surveys are not permitted as projects; instead, each project must contain a survey of background and related work.

You must meet the following milestones (unless otherwise specified in future announcements) to ensure a high-quality project at the end of the semester:

• Form a group of 2-4 members by Oct 1. After this date, we will form groups from the remaining students.
• Turn in a 2-page draft proposal (including references) by Oct 23. Remember to include the names and email addresses of the group members.
• Each group must present the proposal during class hours on Oct 25.
• Each group must turn in a 6- to 8-page final report via email on or before Dec 9. The report must be submitted as a PDF file, with formatting similar to that of the papers you've read in the class.

• The course is heavily inspired by CSE 585: Advanced Scalable Systems for Generative AI (UMich), CS 598: Systems for Generative AI (UIUC), CS8803: Systems for AI - LLMs(Gatech), and 294-162 Machine Learning Systems (UC Berkeley).