Traffic Engineering in Packet/Circuit Hybrid Networks

Scientific computing applications, used in fields such as high-energy physics, climate science, genomics, etc., generate large (tera- to peta-byte sized) data sets. To move these heavy-hitter datasets fast, supercomputing sites invest in high-end clusters that can sustain high-speed transfers. Such large-sized, high-speed transfers are called alpha flows. These flows can have adverse effects on packet delays of real-time flows as alpha flows being bursty in nature can cause router buffer buildups.

To enable the support of both alpha flows and real-time flows on the same network infrastructure while meeting quality-of-service (QoS) requirements of both types of flows, we propose a Hybrid Network Traffic Engineering System (HNTES). HNTES performs two tasks, 1) automatic identification of alpha flows at a core provider network's ingress routers, with no requirement of modifying end-user applications, 2) redirects these flows to traffic-engineered QoS-controlled virtual circuits.

This dissertation describes two versions of the HNTES design. The first design explored the possibility of implementing an online mechanism that identifies alpha flows from live traffic. But the design proved to be cost prohibitive. Our second design uses an offline approach by analyzing NetFlow reports of completed flows to determine the IP addresses of source-destination pairs that move large datasets at high rates. These extracted IP addresses are used to set firewall filters in ingress routers to capture future alpha flows for redirection. This solution is based on a hypothesis that alpha flows are repeatedly created by the same source-destination hosts. NetFlow data for a 7-month period was obtained from an ESnet router to test the hypothesis, which proved to be true. It showed that had HNTES been deployed at the start of this period, over 91\% of data that appeared in bursts from alpha flows would have been redirected. The final contribution of this thesis resulted from an experimental study of different scheduling and policing mechanisms implemented in routers. The objective was to find a suitable combination of mechanisms that achieved dual goals: reducing delay and jitter of real-time delay-sensitive flows, while at the same time allowing alpha flows to achieve high throughput. The best combination was found to be a no-policing, two-queue solution with priority and weighted fair queueing forms of packet scheduling. This result influenced the configuration of routers of a US backbone network provider.