Distributed Systems for Cost-Efficient Inference Serving of AI Models at Scale 13 views

AI inference has become the dominant workload behind many real-world applications. However, the growing size of AI models and the proliferation of AI-powered applications demand expensive, power-hungry GPUs, driving up datacenter operating costs. To improve cost-efficiency, this dissertation proposes novel methodologies across the AI inference serving stack, spanning model instance provisioning and GPU placement, request routing, and instance-level scheduling. The first part of the dissertation improves GPU resource utilization through spatial multiplexing. While co-locating multiple model instances on a single GPU can improve utilization, existing space-sharing techniques suffer from inter-model interference that prevents high computation and memory utilization from being achieved simultaneously. This chapter introduces an interference-aware system comprising three components: (1) a GPU kernel-based estimator that efficiently profiles per-model resource requirements, (2) a heuristic-based scheduler that jointly determines batch sizes, replication degrees, and placements to minimize cost under latency SLO constraints, and (3) an operator graph merging technique that consolidates models to reduce cache-level interference. The second part focuses on request routing. Existing schedulers emphasize load balancing for memory management but overlook the compute layout (the arrangement of tokens across iterations), which critically affects latency for LLM workloads with heterogeneous request characteristics. This chapter proposes a workload-adaptive routing scheduler that minimizes latency by optimizing compute layouts through a multi-step strategy: classifying requests by prefill and decode lengths, balancing load, and selectively distributing execution across instances. The third part targets instance-level batching policies. Through experiments on production traces from a large-scale cloud provider, we find that existing batching techniques incur high latency for agentic workloads, where requests involve multiple tool-call rounds with varying token lengths and arrival times. This chapter introduces a scheduling system that leverages the distinctive structure of agentic workloads through dynamic switching between batching policies and interleaving of different token types. Together, the proposed works present complementary techniques that maximize GPU utilization and minimize latency to achieve cost-efficient distributed inference serving of AI models.

PHD (Doctor of Philosophy)

Distributed Systems; Inference Serving; AI

English

All rights reserved by the author (no additional license for public reuse)

2026-08-03