About
At Microsoft Azure Research, I work on systems for AI efficiency, including agentic AI…
Activity
2K followers
Experience
Education
-
University of Illinois at Urbana-Champaign
-
Efficient and Robust Online Learning for Cloud Systems Design and Optimization
-
-
-
-
-
Licenses & Certifications
Volunteer Experience
Publications
-
SIMPPO: A Scalable and Adaptive Online Learning Framework for Serverless Resource Management
SoCC 2022
-
FIRM: An Intelligent Fine-Grained Resource Management Framework for SLO-Oriented Microservices
OSDI 2020
-
A Geography-Based P2P Overlay Network for Fast and Robust Blockchain Systems
IEEE TSC 2022
-
A Mean-Field Game Approach to Cloud Resource Management with Function Approximation
NeurIPS 2022
-
Delay Sensitivity-driven Congestion Mitigation for HPC Systems
ICS 2021
-
Evaluating Hardware Memory Disaggregation Under Delay and Contention
CompSys 2022
-
Is Serverless Function-as-a-Service a Good Fit for Latency-critical Services?
WoSC 2021
-
OWL: Understanding and Detecting Concurrency Attacks
DSN 2018
-
PLOVER: Fast, Multi-core Scalable Virtual Machine Fault-tolerance
NSDI 2018
-
Reinforcement Learning for Resource Management in Multi-tenant Serverless Platforms
EuroMLSys 2022
Courses
-
Advanced Algorithms
CS 577
-
Advanced Distributed Systems
CS 525
-
Advanced Operating Systems
CS 523
-
Artificial Intelligence
COMP 3270
-
Compiler Techniques
CS 3235
-
Computer Networks
CS 640
-
Computer Organization
COMP 2120
-
Data Science and Analytics
CS 498
-
Data Structure and Algorithms
COMP 2119
-
Database Systems
COMP 3278
-
Dependable AI Systems
ECE 598
-
Discrete Mathematics
COMP 2121
-
Distributed Systems
CS 425
-
Fault-tolerant System Design
CS 536
-
Functional Programming
CS 3258
-
High-speed and Programmable Networks
ECE 598
-
ML for Software Engineering
CS 598
-
Machine Learning
CS 446
-
Object-oriented Programming
COMP 2396
-
Operating Systems
COMP 3230
-
Principles of Programming Languages
COMP 3259
-
Reinforcement Learning
CS 498
-
Software Engineering
COMP 3297
-
System Architecture and Distributed Computing
COMP 3412
-
Trustworthy Machine Learning
CS 442
Projects
-
Power-aware Deep Learning Model Serving
-
With the increasing popularity of large deep-learning model serving workloads like LLMs, there is a pressing need to reduce the energy consumption of a model-serving cluster while maintaining satisfied throughput or model-serving latency requirements. Model multiplexing approaches such as model parallelism, model placement, replication, and batching aim to optimize the model-serving performance. However, they fall short of leveraging the GPU frequency scaling opportunity for power saving. In…
With the increasing popularity of large deep-learning model serving workloads like LLMs, there is a pressing need to reduce the energy consumption of a model-serving cluster while maintaining satisfied throughput or model-serving latency requirements. Model multiplexing approaches such as model parallelism, model placement, replication, and batching aim to optimize the model-serving performance. However, they fall short of leveraging the GPU frequency scaling opportunity for power saving. In this project, we demonstrate (1) the benefits of GPU frequency scaling in power saving for model serving; and (2) the necessity for co-design and optimization of fine-grained model multiplexing and GPU frequency scaling. We explore the co-design space and present a novel power-aware model-serving system, µ-Serve. µ-Serve is a model-serving framework that optimizes the power consumption and model serving latency/throughput of serving multiple ML models efficiently in a homogeneous GPU cluster. Evaluation results on production workloads show that µ-Serve achieves 1.2–2.6x power saving by dynamic GPU frequency scaling (up to 61% reduction) without SLO attainment violations.
Published at ATC 2024: https://haoran-qiu.com/publication/atc-2024/ -
AWARE: Automate Workload Autoscaling with RL in Production Cloud Systems
-
AWARE (ATC 2023, NeurIPS 2023) is an extensible framework for deploying and managing RL-based agents in production systems. AWARE leverages meta-learning and bootstrapping to (a) automatically and quickly adapt to different workloads and (b) provide safe and robust RL exploration. AWARE provides a common OpenAI Gym-like RL interface to agent developers for easy integration with different systems tasks. We demonstrated the use of AWARE in the case of multidimensional workload…
AWARE (ATC 2023, NeurIPS 2023) is an extensible framework for deploying and managing RL-based agents in production systems. AWARE leverages meta-learning and bootstrapping to (a) automatically and quickly adapt to different workloads and (b) provide safe and robust RL exploration. AWARE provides a common OpenAI Gym-like RL interface to agent developers for easy integration with different systems tasks. We demonstrated the use of AWARE in the case of multidimensional workload autoscaling.
Published at ATC 2023: https://www.usenix.org/conference/atc23/presentation/qiu-haoran -
Scalable Serverless Computing Resource Management Framework that Support Multi-tenancy
-
SIMPPO (SoCC 2022, NeurIPS 2022): SIMPPO is a scalable multi-agent RL framework that provides systems support for RL agents to coexist in a dynamic multi-tenant cloud environment. With SIMPPO, each RL agent is able to converge efficiently during training and achieve online policy-serving performance comparable to the agent trained in an isolated environment. We demonstrated the use of SIMPPO in the task of serverless resource management.
https://dl.acm.org/doi/abs/10.1145/3542929.3563475 -
CoCo: Coordinated Container Scheduling with Last-Level Cache and Memory Bandwidth Partitioning
-
See projectLast-level cache (LLC) and memory bandwidth partitioning are commonly used in existing work to meet QoS requirements of all co-scheduled latency-critical applications consolidated on a physical server. With the increasing popularity of cloud microservices and the Function-as-a-Service paradigm, the number of containers consolidated together increases significantly. However, due to the limitation of hardware features, existing work fails to support such a number of applications.
To bridge…Last-level cache (LLC) and memory bandwidth partitioning are commonly used in existing work to meet QoS requirements of all co-scheduled latency-critical applications consolidated on a physical server. With the increasing popularity of cloud microservices and the Function-as-a-Service paradigm, the number of containers consolidated together increases significantly. However, due to the limitation of hardware features, existing work fails to support such a number of applications.
To bridge this gap, this project proposes CoCo, coordinated container scheduling with LLC, and Memory bandwidth partitioning. Our quantitative evaluation shows that CoCo outperforms no-partitioning and baseline approaches by up to 920% and 9.4% respectively. Our main contributions are:
- We present a comprehensive characterization on the impact of LLC and memory bandwidth partitioning and the sensitivity to shared resource contention for popular services widely used in cloud microservices.
- We propose CoCo, coordinated container scheduling with LLC, and memory bandwidth partitioning. CoCo dynamically profiles consolidated workloads and provides a time-sharing algorithm (multi-level queue-based weighted round-robin algorithm) to maximize the overall performance and resource utilization.
- We implemented a prototype of CoCo as a user-level runtime system on Linux, which requires no modification to the underlying OS or the consolidated applications. -
FIRM: Intelligent Fine-grained Resource Orchestration for Cloud Microservices
-
FIRM (OSDI 2020): FIRM is an intelligent fine-grained resource management framework for sharing resources across microservices to provide predictable performance and drive up overall utilization. FIRM leverages online telemetry data and ML/RL methods to adaptively (a) detect/localize microservices that cause SLO violations, (b) identify low-level resources in contention, and (c) take actions to mitigate SLO violations via dynamic…
FIRM (OSDI 2020): FIRM is an intelligent fine-grained resource management framework for sharing resources across microservices to provide predictable performance and drive up overall utilization. FIRM leverages online telemetry data and ML/RL methods to adaptively (a) detect/localize microservices that cause SLO violations, (b) identify low-level resources in contention, and (c) take actions to mitigate SLO violations via dynamic reprovisioning.
https://www.usenix.org/conference/osdi20/presentation/qiu
Honors & Awards
-
ML and Systems Rising Stars 2023
MLCommons
-
Mavis Future Faculty Fellowship
Grainger College of Engineering, University of Illinois at Urbana-Champaign
-
Yunni & Maxine Pao Memorial Fellowship
Grainger College of Engineering, University of Illinois, Urbana-Champaign
-
Conference Presentation Award
Graduate College, University of Illinois, Urbana-Champaign
-
Dean's Honor List
Faculty of Engineering
-
First Runner-up of Final Year Project Competition
Department of Computer Science, HKU
-
Dean's Honor List
Faculty of Engineering
-
Exchange & Visiting International Student Academic Excellence Award
University of Wisconsin-Madison
-
Lee Shau Kee Scholarships for Student Enrichment
The University of Hong Kong
-
Honorable Mention, Mathematical Contest of Modelling
COMAP
-
Dean's Honor List
Faculty of Engineering
-
HKU Foundation Entrance Scholarship
The University of Hong Kong
Languages
-
English
Professional working proficiency
-
Chinese
Native or bilingual proficiency
-
Cantonese
Elementary proficiency
Other similar profiles
Explore top content on LinkedIn
Find curated posts and insights for relevant topics all in one place.
View top content