Abstract
Deep Learning (DL) has driven significant advancements across numerous domains, but its training and inference jobs are resource-intensive and often bound by strict deadlines. Distributed DL training divides datasets into smaller tasks processed by multiple workers. However, a major challenge in these environments is managing stragglers slower-performing tasks caused by factors such as resource contention, hardware heterogeneity, or insufficient resources. Stragglers delay training iterations and waste resources, as faster-performing workers remain idle while waiting, which is especially problematic in dynamic, shared, or heterogeneous environments like production clusters or spot markets (e.g., AWS EC2). Traditional DL training systems often assume static resource availability and rely on reactive methods, such as task migration or job re-launching, which fail to address the underlying causes and lead to inefficiencies. While proactive methods like spawning duplicate tasks or adjusting straggler data sizes exist, they either increase resource consumption or risk compromising model accuracy.
This dissertation addresses the above issues by proposing novel solutions to minimize the impact of stragglers in DL training, thereby reducing training times and improving resource efficiency. It introduces a Straggler-Avoiding DL Training Job Scheduling (SAS) system, which leverages machine learning to predict resource availability for each node and generates a resource matrix for clustering nodes with similar availability and close proximity. Tasks are then assigned to node groups that optimize training job completion time while ensuring high probabilities of adequate resource provisioning. Additionally, a deep reinforcement learning-based approach, SAS-RL, is proposed to accelerate decision-making in SAS. To further tackle stragglers, this dissertation presents a Straggler Mitigation System (SMS), which predicts potential stragglers by leveraging predictable computation patterns in DL tasks. SMS employs a novel batch-size reduction technique to dynamically adjust workloads, removing similar data from clusters within the batch to maintain accuracy. SMS also integrates a gradient transfer control mechanism to bypass redundant gradient transfers from stragglers. Furthermore, this dissertation presents STRET, a straggler-aware time and resource-efficient distributed DL Training system designed to address persistent straggler problems in both homogeneous and heterogeneous GPU-based clusters, particularly those caused by communication bottlenecks. STRET introduces a hybrid architecture that pairs stragglers with non-stragglers that have high-bandwidth links to reduce communication latency. If this strategy does not resolve the bottleneck, STRET applies two additional techniques: selective gradient reporting based on accuracy increment thresholds and one-time batch size tuning to reduce iteration time.
