Scheduling Techniques in Different Architectures of Data-parallel Clusters for High Performance

Over the past decades, to deal with the rapidly growing data volumes in this big data era, data-parallel clusters were designed to process the data-parallel jobs, each of which runs on many machines in parallel by dividing the entire job into individual tasks that process different pieces of data. Many large organizations such as Facebook, Google and Yahoo! have deployed their data-parallel clusters to process thousands of jobs every day.

Many previous studies show that current production clusters process increasingly diverse jobs with various job characteristics (e.g., input data size, shuffle data size, and output data size). First, previous works have shown that there are a large number of shuffle-heavy jobs in current production workloads, which may result in network bottleneck in the clusters and hence greatly degrades the performance of the clusters. Second, the small jobs (i.e., jobs with small data to process) often dominate the workloads in production. However, current architectures and schedulers of data-parallel clusters were originally built to process data-intensive jobs that have large input datasets. The mismatch between the actual workloads and design objectives results in poor performance of the jobs in the clusters.

The key contribution of this dissertation is designing job schedulers in different architectures of data-parallel clusters to handle the diverse workloads. First, we design a Network-Aware job Scheduler (NAS) for data-parallel frameworks in traditional datacenter networks that can schedule tasks carefully to avoid and reduce the network congestion caused by the large amount of shuffle-heavy jobs.

Second, to build high-capacity, low-latency datacenter networks, researchers have proposed hybrid electrical/optical datacenter network (Hybrid-DCN) architectures, which augment the traditional electrical-packet switching (EPS) datacenter network with an on-demand rack-to-rack network using the optical circuit switch (OCS). In order to utilize Hybrid-DCN efficiently, job schedulers for data-parallel frameworks must keep pace to meet the needs of such hybrid networks. Thus, we design a job scheduler called JobPacker that can efficiently exploit OCS in Hybrid-DCN to improve the job performance by finding the optimal tradeoff between the parallelism and traffic aggregation.

Third, recent works advocate hybrid scale-up/scale-out clusters (in short Hybrid clusters) to handle the workloads that consist of a majority of jobs with small input data sizes and a small number of jobs with large input data sizes. However, previous works did not solve the challenges for job placement and data placement in designing such a Hybrid cluster. We design the job placement and data placement strategies in Hybrid cluster to address the challenges, which can significantly improve the performance of workloads with a large amount of small jobs.

Finally, we demonstrate through trace-driven simulation and real cluster evaluation that our proposed schedulers improve the performance of diverse workloads in the data-parallel clusters in terms of throughput and job completion time.