Sushant Jain
Urs Hölzle
ABSTRACT
We present the design, implementation, and evaluation of B4, a private WAN connecting Google's data centers across the planet. B4 has a number of unique characteristics: i) massive bandwidth requirements deployed to a modest number of sites, ii) elastic traffic demand that seeks to maximize average bandwidth, and iii) full control over the edge servers and network, which enables rate limiting and demand measurement at the edge.
These characteristics led to a Software Defined Networking architecture using OpenFlow to control relatively simple switches built from merchant silicon. B4's centralized traffic engineering service drives links to near 100% utilization, while splitting application flows among multiple paths to balance capacity against application priority/demands. We describe experience with three years of B4 production deployment, lessons learned, and areas for future work.
- Al-Fares, M., Loukissas, A., and Vahdat, A. A Scalable, Commodity Data Center Network Architecture. In Proc. SIGCOMM (New York, NY, USA, 2008), ACM. Google Scholar
Digital Library
- Allalouf, M., and Shavitt, Y. Centralized and Distributed Algorithms for Routing and Weighted Max-Min Fair Bandwidth Allocation. IEEE/ACM Trans. Networking 16, 5 (2008), 1015--1024. Google ScholarDigital Library
- Aukia, P., Kodialam, M., Koppol, P. V., Lakshman, T. V., Sarin, H., and Suter, B. RATES: A Server for MPLS Traffic Engineering. IEEE Network Magazine 14, 2 (March 2000), 34--41. Google ScholarDigital Library
- Awduche, D., Berger, L., Gan, D., Li, T., Srinivasan, V., and Swallow, G. RSVP-TE: Extensions to RSVP for LSP Tunnels. RFC 3209, IETF, United States, 2001. Google ScholarDigital Library
- Awduche, D., Malcolm, J., Agogbua, J., O'Dell, M., and McManus, J. Requirements for Traffic Engineering Over MPLS. RFC 2702, IETF, 1999. Google ScholarDigital Library
- Caesar, M., Caldwell, D., Feamster, N., Rexford, J., Shaikh, A., and van der Merwe, K. Design and Implementation of a Routing Control Platform. In Proc. of NSDI (April 2005). Google ScholarDigital Library
- Casado, M., Freedman, M. J., Pettit, J., Luo, J., McKeown, N., and Shenker, S. Ethane: Taking Control of the Enterprise. In Proc. SIGCOMM (August 2007). Google ScholarDigital Library
- Casado, M., Garfinkel, T., Akella, A., Freedman, M. J., Boneh, D., McKeown, N., and Shenker, S. SANE: A Protection Architecture for Enterprise Networks. In Proc. of Usenix Security (August 2006). Google ScholarDigital Library
- Chandra, T. D., Griesemer, R., and Redstone, J. Paxos Made Live: an Engineering Perspective. In Proc. of the ACM Symposium on Principles of Distributed Computing (New York, NY, USA, 2007), ACM, pp. 398--407. Google ScholarDigital Library
- Choi, T., Yoon, S., Chung, H., Kim, C., Park, J., Lee, B., and Jeong, T. Design and Implementation of Traffic Engineering Server for a Large-Scale MPLS-Based IP Network. In Revised Papers from the International Conference on Information Networking, Wireless Communications Technologies and Network Applications-Part I (London, UK, UK, 2002), ICOIN '02, Springer-Verlag, pp. 699--711. Google ScholarDigital Library
- Curtis, A. R., Mogul, J. C., Tourrilhes, J., Yalagandula, P., Sharma, P., and Banerjee, S. DevoFlow: Scaling Flow Management for High-Performance Networks. In Proc. SIGCOMM (2011), pp. 254--265. Google ScholarDigital Library
- Danna, E., Hassidim, A., Kaplan, H., Kumar, A., Mansour, Y., Raz, D., and Segalov, M. Upward Max Min Fairness. In INFOCOM (2012), pp. 837--845.Google Scholar
- Danna, E., Mandal, S., and Singh, A. A Practical Algorithm for Balancing the Max-min Fairness and Throughput Objectives in Traffic Engineering. In Proc. INFOCOM (March 2012), pp. 846--854.Google Scholar
- Elwalid, A., Jin, C., Low, S., and Widjaja, I. MATE: MPLS Adaptive Traffic Engineering. In Proc. IEEE INFOCOM (2001), pp. 1300--1309.Google Scholar
- Farrington, N., Rubow, E., and Vahdat, A. Data Center Switch Architecture in the Age of Merchant Silicon. In Proc. Hot Interconnects (August 2009), IEEE, pp. 93--102. Google ScholarDigital Library
- Fortz, B., Rexford, J., and Thorup, M. Traffic Engineering with Traditional IP Routing Protocols. IEEE Communications Magazine 40 (2002), 118--124. Google ScholarDigital Library
- Fortz, B., and Thorup, M. Increasing Internet Capacity Using Local Search. Comput. Optim. Appl. 29, 1 (October 2004), 13--48. Google ScholarDigital Library
- Greenberg, A., Hamilton, J. R., Jain, N., Kandula, S., Kim, C., Lahiri, P., Maltz, D. A., Patel, P., and Sengupta, S. VL2: A Scalable and Flexible Data Center Network. In Proc. SIGCOMM (August 2009). Google ScholarDigital Library
- Greenberg, A., Hjalmtysson, G., Maltz, D. A., Myers, A., Rexford, J., Xie, G., Yan, H., Zhan, J., and Zhang, H. A Clean Slate 4D Approach to Network Control and Management. SIGCOMM CCR 35, 5 (2005), 41--54. Google ScholarDigital Library
- Greenberg, A., Lahiri, P., Maltz, D. A., Patel, P., and Sengupta, S. Towards a Next Generation Data Center Architecture: Scalability and Commoditization. In Proc. ACM workshop on Programmable Routers for Extensible Services of Tomorrow (2008), pp. 57--62. Google ScholarDigital Library
- Gude, N., Koponen, T., Pettit, J., Pfaff, B., Casado, M., McKeown, N., and Shenker, S. NOX: Towards an Operating System for Networks. In SIGCOMM CCR (July 2008). Google ScholarDigital Library
- He, J., and Rexford, J. Toward Internet-wide Multipath Routing. IEEE Network Magazine 22, 2 (March 2008), 16--21. Google ScholarDigital Library
- Hong, C.-Y., Kandula, S., Mahajan, R., Zhang, M., Gill, V., Nanduri, M., and Wattenhofer, R. Have Your Network and Use It Fully Too: Achieving High Utilization in Inter-Datacenter WANs. In Proc. SIGCOMM (August 2013). Google ScholarDigital Library
- Kandula, S., Katabi, D., Davie, B., and Charny, A. Walking the Tightrope: Responsive Yet Stable Traffic Engineering. In Proc. SIGCOMM (August 2005). Google ScholarDigital Library
- Kipp, S. Bandwidth Growth and the Next Speed of Ethernet. Proc. North American Network Operators Group (October 2012).Google Scholar
- Koponen, T., Casado, M., Gude, N., Stribling, J., Poutievski, L., Zhu, M., Ramanathan, R., Iwata, Y., Inoue, H., Hama, T., and Shenker, S. Onix: a Distributed Control Platform for Large-scale Production Networks. In Proc. OSDI (2010), pp. 1--6. Google ScholarDigital Library
- Lakshman, T., Nandagopal, T., Ramjee, R., Sabnani, K., and Woo, T. The Softrouter Architecture. In Proc. HotNets (November 2004).Google Scholar
- McKeown, N., Anderson, T., Balakrishnan, H., Parulkar, G., Peterson, L., Rexford, J., Shenker, S., and Turner, J. OpenFlow: Enabling Innovation in Campus Networks. SIGCOMM CCR 38, 2 (2008), 69--74. Google ScholarDigital Library
- Medina, A., Taft, N., Salamatian, K., Bhattacharyya, S., and Diot, C. Traffic Matrix Estimation: Existing Techniques and New Directions. In Proc. SIGCOMM (New York, NY, USA, 2002), ACM, pp. 161--174. Google ScholarDigital Library
- Nascimento, M. R., Rothenberg, C. E., Salvador, M. R., and Magalh\ aes, M. F. QuagFlow: Partnering Quagga with OpenFlow (Poster). In Proc. SIGCOMM (2010), pp. 441--442. Google ScholarDigital Library
- OpenFlow Specification. http://www.openflow.org/wp/documents/.Google Scholar
- Rothenberg, C. E., Nascimento, M. R., Salvador, M. R., Corrêa, C. N. A., Cunha de Lucena, S., and Raszuk, R. Revisiting Routing Control Platforms with the Eyes and Muscles of Software-defined Networking. In Proc. HotSDN (2012), pp. 13--18. Google ScholarDigital Library
- Roughan, M., Thorup, M., and Zhang, Y. Traffic Engineering with Estimated Traffic Matrices. In Proc. IMC (2003), pp. 248--258. Google ScholarDigital Library
- Scoglio, C., Anjali, T., de Oliveira, J. C., Akyildiz, I. F., and UhI, G. TEAM: A Traffic Engineering Automated Manager for DiffServ-based MPLS Networks. Comm. Mag. 42, 10 (October 2004), 134--145. Google ScholarDigital Library
- Sherwood, R., Gibb, G., Yap, K.-K., Appenzeller, G., Casado, M., McKeown, N., and Parulkar, G. FlowVisor: A Network Virtualization Layer. Tech. Rep. OPENFLOW-TR-2009--1, OpenFlow, October 2009.Google Scholar
- Suchara, M., Xu, D., Doverspike, R., Johnson, D., and Rexford, J. Network Architecture for Joint Failure Recovery and Traffic Engineering. In Proc. ACM SIGMETRICS (2011), pp. 97--108. Google ScholarDigital Library
- Thaler, D. Multipath Issues in Unicast and Multicast Next-Hop Selection. RFC 2991, IETF, 2000. Google ScholarDigital Library
- Wang, H., Xie, H., Qiu, L., Yang, Y. R., Zhang, Y., and Greenberg, A. COPE: Traffic Engineering in Dynamic Networks. In Proc. SIGCOMM (2006), pp. 99--110. Google ScholarDigital Library
- Xu, D., Chiang, M., and Rexford, J. Link-state Routing with Hop-by-hop Forwarding Can Achieve Optimal Traffic Engineering. IEEE/ACM Trans. Netw. 19, 6 (December 2011), 1717--1730. Google ScholarDigital Library
- Yu, M., Rexford, J., Freedman, M. J., and Wang, J. Scalable flow-based networking with DIFANE. In Proc. SIGCOMM (2010), pp. 351--362. Google ScholarDigital Library
Index Terms
- B4: experience with a globally-deployed software defined wan
Recommendations
B4: experience with a globally-deployed software defined wan
We present the design, implementation, and evaluation of B4, a private WAN connecting Google's data centers across the planet. B4 has a number of unique characteristics: i) massive bandwidth requirements deployed to a modest number of sites, ii) elastic ...
A roadmap for traffic engineering in SDN-OpenFlow networks
Software Defined Networking (SDN) is an emerging networking paradigm that separates the network control plane from the data forwarding plane with the promise to dramatically improve network resource utilization, simplify network management, reduce ...
OpenWiMesh: A Framework for Software Defined Wireless Mesh Networks
SBRC '14: Proceedings of the 2014 Brazilian Symposium on Computer Networks and Distributed SystemsThe adoption of Software-Defined Networks (SDN) in the context of wireless mesh networks opens up room for new functionalities such as traffic engineering, flow based traffic forwarding and, more generally, "programmability". If on one hand the new ...
Comments