Characterization, Management, and Online Traffic Engineering of Heavy-Hitter Flows in Software Defined Networks

This dissertation describes advances made in the field of network management and high-performance networking. For network management, we designed and implemented an algorithm to reconstruct flows from NetFlow records collected at IP routers. We executed this system in a large Research and Education Network (REN), Energy Sciences Network (ESnet). We found that scientists move 100 GB to TB sized datasets at rates of 1 to 2.5 Gbps, and seldom use the network for transfers more than 10 hours. Our findings are useful for network planning and traffic engineering, and in improving user experience.

For high-performance networking we designed, implemented, and evaluated a high-speed Cheetah Flow Identification Network Function (CFINF). Two key features of the CFINF design are: (i) the ability to scale easily to higher levels of traffic utilization, and (ii) the flexibility for execution on general-purpose hardware. The system is designed with efficient data structures that are optimized to detect cheetah flows on a 10 Gb/s link that causes 1 M flows/min. With 10 CPU cores, CFINF can handle a 1-min 10-Gbps real Center for Applied Internet Data Analysis (CAIDA) traffic trace that contained 1.5M flows and 38M packets without loss. To improve efficiency, we ran CFINF in an 8-core configuration. However, with this configuration there were packet drops (max. rate of 0.036\%).

To determine optimal values for CFINF parameters, we designed and implemented the Cheetah Flow Traffic Engineering System (CFTES). For an acceptable packet drop rate on a congested link, a high value of the rate threshold will result in few flows being redirected. A novel two-queue traffic redirection solution is presented that addresses the problem of packet reordering in TCP when a flow is redirected. We quantify a metric called burstiness and show that packet drop rate increases with increasing burstiness, even when the average background rate is constant. The packet drop rate is also higher for high-RTT cheetah flows.

Finally, we propose a network service to diagnose throughput performance for large data transfers. Analysis of data-transfer logs, which were created by running experiments across Internet2, a US-wide REN, offer insights into the causes of poor performance.