Guohui Wang
ABSTRACT
Recent advances of software defined networking and optical switching technology make it possible to program the network stack all the way from physical topology to flow level traffic control. In this paper, we leverage the combination of SDN controller with optical switching to explore the tight integration of application and network control. We particularly study the run-time network configuration for big data applications to jointly optimize application performance and network utilization. We use Hadoop as an example to discuss the integrated network control architecture, job scheduling, topology and routing configuration mechanisms for Hadoop jobs. Our analysis suggests that such an integrated control has great potential to improve application performance with relatively small configuration overhead. We believe our study shows early promise of achieving the long-term goal of tight network and application integration using SDN.
Supplemental Material
- Apache Hadoop, http://hadoop.apache.org.Google Scholar
- Apache HBase, http://hbase.apache.org.Google Scholar
- Floodlight openflow controller. http://floodlight.openflowhub.org/.Google Scholar
- Open vswitch. http://openvswitch.org/.Google Scholar
- H. Abu-Libdeh, P. Costa, A. Rowstron, G. O'Shea, and A. Donnelly. Symbiotic routing in future data centers. In SIGCOMM'10, August 2010. Google ScholarDigital Library
- M. Al-Fares, S. Radhakrishnan, B. Raghavan, N. Huang, and A. Vahdat. Hedera: Dynamic flow scheduling for data center networks. In USENIX NSDI'10, April 2010. Google ScholarDigital Library
- G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, and E. Harris. Reining in the outliers in map-reduce clusters using mantri. In USENIX OSDI'10, December 2010. Google ScholarDigital Library
- T. Benson, A. Akella, A. Shaikh, and S. Sahu. Cloudnaas: A cloud networking platform for enterprise applications. In ACM SOCC'11, October 2011. Google ScholarDigital Library
- T. Benson, A. Anand, A. Akella, and M. Zhang. Microte: The case for fine-grained traffic engineering in data centers. In ACM CoNEXT'11, December 2011. Google ScholarDigital Library
- P. Chandra, A. Fisher, C. Kosak, T. S. E. Ng, P. Steenkiste, E. Takahashi, and H. Zhang. Darwin: Resource management for value-added customizable network service. In IEEE ICNP'98, October 1998.Google ScholarCross Ref
- P. Costa, A. Donnelly, A. Rowstron, and G. O'Shea. Camdoop: Exploiting in-network aggregation for big data applications. In USENIX NSDI'12, April 2012. Google ScholarDigital Library
- S. Das, Y. Yiakoumis, G. Parulkar, P. Singh, D. Getachew, P. D. Desai, and N. McKeown. Application-aware aggregation and traffic engineering in a converged packet-circuit network. In OFC'11, March 2011.Google ScholarCross Ref
- B. Hindman et al. Mesos: A platform for fine-grained resource sharing in the data center. In USENIX NSDI'11, March 2011. Google ScholarDigital Library
- H. Bazzaz et al. Switching the optial divide: Fundamental challenges for hybrid electrical/optical data center networks. In ACM SOCC'11, October 2011. Google ScholarDigital Library
- K. Chen et al. OSA: An optical switching architecture for data center networks with unprecedented flexibility. In NSDI'12, April 2012. Google ScholarDigital Library
- L. Schares et al. A reconfigurable interconnect fabric with optical circuit switch and software optimizer for stream computing systems. In OFC'09, March 2009.Google ScholarCross Ref
- Y. Chen et al. Energy efficiency for large-scale mapreduce workloads with significant interactive analysis. In ACM EuroSys'12, April 2012. Google ScholarDigital Library
- N. Farrington, G. Porter, S. Radhakrishnan, H. Bazzaz, V. Subramanya, Y. Fainman, G. Papen, and A. Vahdat. Helios: A hybrid electrical/optical switch architecture for modular data centers. In ACM SIGCOMM, August 2010. Google ScholarDigital Library
- C. Guo, G. Lu, D. Li, H. Wu, X. Zhang, Y. Shi, C. Tian, Y. Zhang, and S. Lu. Bcube: A high performance, server-centric network architecture for modular data centers. In ACM SIGCOMM'09, August 2009. Google ScholarDigital Library
- M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data-parallel programs from sequential building blocks. In ACM EurySys'07, March 2007. Google ScholarDigital Library
- S. Kavulya, J. Tan, R. Gandhi, and P. Narasimhan. An analysis of traces from a production mapreduce cluster. In Carnegie Mellon University Technical Report, December 2009.Google Scholar
- M. Reitblatt, N. Foster, J. Rexford, C. Schlesinger, and D. Walker. Abstractions for network update. In ACM SIGCOMM'12, August 2012. Google ScholarDigital Library
- J. Seedorf and E. Burger. Application-layer traffic optimization (alto) problem statement. In RFC-5693, 2009.Google Scholar
- D. L. Tennenhouse, J. M. Smith, W. Sincoskie, D. Wetherall, and G. Minden. A survey of active network research. In IEEE Communications Magazine, January 1997. Google ScholarDigital Library
- A. Vahdat, H. Liu, X. Zhao, and C. Johnson. The emerging optical data center. In OFC'11, March 2011.Google ScholarCross Ref
- G. Wang, D. Andersen, M. Kaminsky, K. Papagiannaki, T. S. E. Ng, M. Kozuch, and M. Ryan. c-Through: Part-time optics in data centers. In ACM SIGCOMM, August 2010. Google ScholarDigital Library
- K. Webb, A. Snoeren, and K. Yocum. Topology switching for data center networks. In USENIX Hot-ICE'11, March 2011. Google ScholarDigital Library
- E. Weigle and W. Feng. A comparison of tcp automatic tuning techniques for distributed computing. In IEEE HPCS'02, July 2002. Google ScholarDigital Library
- M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. Spark: Cluster computing with working sets. In USENIX HotCloud'10, June 2010. Google ScholarDigital Library
- M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica. Improving mapreduce performance in heterogeneous environments. In USENIX OSDI'08, December 2008. Google ScholarDigital Library
Index Terms
- Programming your network at run-time for big data applications
Recommendations
Performance evaluation of a hybrid IP/SDN network in data centre network architectures
The current Internet is facing many challenges to accommodate the growing demands of cloud computing. Software‐defined networking (SDN) is an emerging paradigm, proposed to be used in this context in order to investigate these issues. SDN virtualises ...
Auto-Configuration of SDN Switches in SDN/Non-SDN Hybrid Network
AINTEC '15: Proceedings of the 11th Asian Internet Engineering ConferenceThis paper proposes an auto-configuration mechanism for a newly attached SDN (Software-defined Networking) switch and intermediate switches in an SDN/non-SDN hybrid network. Automation of initial configuration of SDN switches brings the benefit of ...
Big Data Processing Technology Research and Application Prospects
IMCCC '14: Proceedings of the 2014 Fourth International Conference on Instrumentation and Measurement, Computer, Communication and ControlWith the rapid development of cloud computing, Internet of Things, Mobile Internet and other related technologies, data is growing at an unprecedented rate in both scales and types. Nowadays, data has been a kind of enormous business resources in the ...
Comments