Mohammad Alizadeh
Albert Greenberg
Abstract
Cloud data centers host diverse applications, mixing workloads that require small predictable latency with others requiring large sustained throughput. In this environment, today's state-of-the-art TCP protocol falls short. We present measurements of a 6000 server production cluster and reveal impairments that lead to high application latencies, rooted in TCP's demands on the limited buffer space available in data center switches. For example, bandwidth hungry "background" flows build up queues at the switches, and thus impact the performance of latency sensitive "foreground" traffic.
To address these problems, we propose DCTCP, a TCP-like protocol for data center networks. DCTCP leverages Explicit Congestion Notification (ECN) in the network to provide multi-bit feedback to the end hosts. We evaluate DCTCP at 1 and 10Gbps speeds using commodity, shallow buffered switches. We find DCTCP delivers the same or better throughput than TCP, while using 90% less buffer space. Unlike TCP, DCTCP also provides high burst tolerance and low latency for short flows. In handling workloads derived from operational measurements, we found DCTCP enables the applications to handle 10X the current background traffic, without impacting foreground traffic. Further, a 10X increase in foreground traffic does not cause any timeouts, thus largely eliminating incast problems.
- P. Agarwal, B. Kwan, and L. Ashvin. Flexible buffer allocation entities for traffic aggregate containment. US Patent 20090207848, August 2009.Google Scholar
- M. Al-Fares, A. Loukissas, and A. Vahdat. A scalable, commodity data center network architecture. In SIGCOMM, 2008. Google ScholarDigital Library
- M. Alizadeh et al. Data Center TCP (DCTCP). Technical report.Google Scholar
- G. Appenzeller, I. Keslassy, and N. McKeown. Sizing router buffers. In SIGCOMM, 2004. Google ScholarDigital Library
- L. Brakmo, S. O'Malley, and L. Peterson. TCP Vegas: New techniques for congestion detection and avoidance. In SIGCOMM, 1994. Google ScholarDigital Library
- N. Dukkipati, M. Kobayashi, R. Zhang-Shen, and N. McKeown. Processor sharing flows in the internet. In IWQOS, 2006. Google ScholarDigital Library
- S. Floyd. RED: Discussions of setting parameters. http://www.icir.org/floyd/REDparameters.txt.Google Scholar
- S. Floyd. RFC 3649: HighSpeed TCP for large congestion windows. Google ScholarDigital Library
- S. Floyd, R. Gummadi, and S. Shenker. Adaptive RED: An algorithm for increasing the robustness of RED's active queue management. Technical report, ACIRI, 2001.Google Scholar
- S. Floyd and V. Jacobson. Random early detection gateways for congestion avoidance. IEEE/ACM ToN, 1993. Google ScholarDigital Library
- S. Floyd and V. Jacobson. The synchronization of periodic routing messages. IEEE/ACM ToN, 1994. Google ScholarDigital Library
- A. Greenberg et al. VL2: A scalable and flexible data center network. In SIGCOMM, 2009. Google ScholarDigital Library
- R. Griffith, Y. Chen, J. Liu, A. Joseph, and R. Katz. Understanding TCP incast throughput collapse in datacenter networks. In WREN Workshop, 2009. Google ScholarDigital Library
- Y. Gu, D. Towsley, C. Hollot, and H. Zhang. Congestion control for small buffer high bandwidth networks. In INFOCOM, 2007.Google Scholar
- C. Guo et al. Bcube: High performance, server-centric network architecture for data centers. In SIGCOMM, 2009. Google ScholarDigital Library
- J. Hamilton. On designing and deploying Internet-scale services. In USENIX LISA, 2007. Google ScholarDigital Library
- C. V. Hollot, V. Misra, D. Towsley, and W.-B. Gong. On designing improved controllers for AQM routers supporting TCP flows. In INFOCOM, April 2001. Google ScholarDigital Library
- A. Kabbani and B. Prabhakar. In defense of TCP. In The Future of TCP: Train-wreck or Evolution, 2008.Google Scholar
- S. Kandula, S. Sengupta, A. Greenberg, P. Patel, and R. Chaiken. The nature of datacenter traffic: Measurements and analysis. In IMC, 2009. Google ScholarDigital Library
- F. Kelly, G. Raina, and T. Voice. Stability and Fairness of Explicit Congestion Control with Small Buffers. CCR, July 2008. Google ScholarDigital Library
- R. Kohavi et al. Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO. KDD, 2007. Google ScholarDigital Library
- D. Leith, R. Shorten, and G. McCullagh. Experimental evaluation of cubic-TCP. In Proc. Protocols for Fast Long Distance Networks 2007, 2007.Google Scholar
- Y.-T. Li, D. Leith, and R. N. Shorten. Experimental evaluation of TCP protocols for high-speed networks. IEEE/ACM Trans. Netw., 15(5):1109--1122, 2007. Google ScholarDigital Library
- R. Pan, B. Prabhakar, and A. Laxmikantha. QCN: Quantized congestion notification an overview. http://www.ieee802.org/1/files/public/docs2007/au_ prabhakar_qcn_overview_geneva.pdf.Google Scholar
- I. A. Qazi, L. Andrew, and T. Znati. Congestion control using efficient explicit feedback. In INFOCOM, 2009.Google ScholarCross Ref
- G. Raina, D. Towsley, and D. Wischik. Part II: Control theory for buffer sizing. CCR, July 2005. Google ScholarDigital Library
- K. Ramakrishnan, S. Floyd, and D. Black. RFC 3168: the addition of explicit congestion notification (ECN) to IP. Google ScholarDigital Library
- K. K. Ramakrishnan and R. Jain. A binary feedback scheme for congestion avoidance in computer networks. ACM Trans. Comp. Systems, 1990. Google ScholarDigital Library
- J. Rothschild. High performance at massive scale: Lessons learned at facebook. mms://video-jsoe.ucsd.edu/calit2/JeffRothschildFacebook.wmv.Google Scholar
- I. R. Sangtae Ha and L. Xu. Cubic: A new TCP-friendly high-speed TCP variant. SIGOPS-OSR, July 2008. Google ScholarDigital Library
- K. Tan, J. Song, Q. Zhang, and M. Sridharan. A Compound TCP Approach for High-speed and Long Distance Networks. In INFOCOM, 2006.Google ScholarCross Ref
- V. Vasudevan et al. Safe and effective fine-grained TCP retransmissions for datacenter communication. In SIGCOMM, 2009. Google ScholarDigital Library
- D. X. Wei, C. Jin, S. H. Low, and S. Hegde. FAST TCP: motivation, architecture, algorithms, performance. IEEE/ACM ToN, Dec. 2006. Google ScholarDigital Library
- Y. Xia, L. Subramanian, I. Stoica, and S. Kalyanaraman. One more bit is enough. In SIGCOMM, 2005. Google ScholarDigital Library
Index Terms
- Data center TCP (DCTCP)
Recommendations
Data center TCP (DCTCP)
SIGCOMM '10: Proceedings of the ACM SIGCOMM 2010 conferenceCloud data centers host diverse applications, mixing workloads that require small predictable latency with others requiring large sustained throughput. In this environment, today's state-of-the-art TCP protocol falls short. We present measurements of a ...
Analysis of DCTCP: stability, convergence, and fairness
Performance evaluation reviewCloud computing, social networking and information networks (for search, news feeds, etc) are driving interest in the deployment of large data centers. TCP is the dominant Layer 3 transport protocol in these networks. However, the operating conditions---...
Comments