ABSTRACT
For large-scale networks, high-radix switches reduce hop and switch count, which decreases latency and power. The ITRS projections for signal-pin count and per-pin bandwidth are nearly flat over the next decade, so increased radix in electronic switches will come at the cost of less per-port bandwidth. Silicon nanophotonic technology provides a long-term solution to this problem. We first compare the use of photonic I/O against an all-electrical, Cray YARC inspired baseline. We compare the power and performance of switches of radix 64, 100, and 144 in the 45, 32, and 22 nm technology steps. In addition with the greater off-chip bandwidth enabled by photonics, the high power of electrical components inside the switch becomes a problem beyond radix 64.
We propose an optical switch architecture that exploits highspeed optical interconnects to build a flat crossbar with multiplewriter, single-reader links. Unlike YARC, which uses small buffers at various stages, the proposed design buffers only at input and output ports. This simplifies the design and enables large buffers, capable of handling ethernet-size packets. To mitigate head-of-line blocking and maximize switch throughput, we use an arbitration scheme that allows each port to make eight requests and use two grants. The bandwidth of the optical crossbar is also doubled to to provide a 2x internal speedup. Since optical interconnects have high static power, we show that it is critical to balance the use of optical and electrical components to get the best energy efficiency. Overall, the adoption of photonic I/O allows 100,000 port networks to be constructed with less than one third the power of equivalent all-electronic networks. A further 50% reduction in power can be achieved by using photonics within the switch components. Our best optical design performs similarly to YARC for small packets while consuming less than half the power, and handles 80% more load for large message traffic.
Supplemental Material
- 2010. Mike Parker, personal communication.Google Scholar
- J. Ahn, N. Binkert, A. Davis, M. McLaren, and R. S. Schreiber. HyperX: Topology, Routing, and Packaging of Efficient Large-Scale Networks. Supercomputing, Nov. 2009. Google ScholarDigital Library
- J. Ahn, M. Fiorentino, R. Beausoleil, N. Binkert, A. Davis, D. Fattal, N. Jouppi, M. McLaren, C. Santori, R. Schreiber, S. Spillane, D. Vantrease, and Q. Xu. Devices and architectures for photonic chip-scale integration. Applied Physics A: Materials Science & Processing, 95(4):989--997, 2009.Google ScholarCross Ref
- B. Analui, D. Guckenberger, D. Kucharski, and A. Narasimha. A Fully Integrated 20-Gb/s Optoelectronic Transceiver Implemented in a Standard 0.13 micron CMOS SOI Technology. IEEE Journal of Solid-State Circuits, 41(25):2945--2955, Dec 2006.Google ScholarCross Ref
- N. L. Binkert, R. G. Dreslinski, L. R. Hsu, K. T. Lim, A. G. Saidi, and S. K. Reinhardt. The M5 Simulator: Modeling Networked Systems. IEEE Micro, 26(4):52--60, Jul/Aug 2006. Google ScholarDigital Library
- Broadcom. BCM56840 Series High Capacity StrataXGS® Ethernet Switch Series. http://www.broadcom.com/products/Switching/Data-Center/BCM56840-Series.Google Scholar
- L. Chen, K. Preston, S. Manipatruni, and M. Lipson. Integrated GHz silicon photonic interconnect with micrometer-scale modulators and detectors. Optical Express, 17(17):15248--15256, 2009.Google ScholarCross Ref
- U. Cummings. FocalPoint: A Low-Latency, High-Bandwidth Ethernet Switch Chip. In Hot Chips 18, Aug 2006.Google Scholar
- G. Dimitrakopoulos and K. Galanopoulos. Fast Arbiters for On-Chip Network Switches. In International Conference on Computer Design, pages 664--670, Oct 2008.Google Scholar
- K. Fukuda, H. Yamashita, G. Ono, R. Nemoto, E. Suzuki, T. Takemoto, F. Yuki, and T. Saito. A 12.3mW 12.5Gb/s complete transceiver in 65nm CMOS. In ISSCC, pages 368--369, Feb 2010.Google ScholarCross Ref
- S. J. Hewlett, J. D. Love, and V. V. Steblina. Analysis and design of highly broad-band, planar evanescent couplers. Optical and Quantum Electronics, 28:71--81, 1996. 10.1007/BF00578552.Google ScholarCross Ref
- R. Ho. On-Chip Wires: Scaling and Efficiency. PhD thesis, Stanford University, August 2003.Google Scholar
- M. Karol, M. Hluchyj, and S. Morgan. Input versus output queueing on a space-division packet switch. Communications, IEEE Transactions on, 35(12):1347--1356, Dec. 1987.Google ScholarCross Ref
- J. Kim, W. J. Dally, and D. Abts. Adaptive Routing in High-Radix Clos Network. In SC'06, Nov 2006. Google ScholarDigital Library
- J. Kim, W. J. Dally, and D. Abts. Flattened Butterfly: a Cost-efficient Topology for High-Radix Networks. In ISCA, Jun 2007. Google ScholarDigital Library
- J. Kim, W. J. Dally, S. Scott, and D. Abts. Technology-Driven, Highly-Scalable Dragonfly Topology. In ISCA, Jun 2008. Google ScholarDigital Library
- J. Kim, W. J. Dally, B. Towles, and A. K. Gupta. Microarchitecture of a High-Radix Router. In ISCA, Jun 2005. Google ScholarDigital Library
- N. Kirman, M. Kirman, R. K. Dokania, J. F. Martinez, A. B. Apsel, M. A. Watkins, and D. H. Albonesi. Leveraging Optical Technology in Future Bus-based Chip Multiprocessors. In MICRO, pages 492--503, 2006. Google ScholarDigital Library
- B. R. Koch, A. W. Fang, O. Cohen, and J. E. Bowers. Mode-locked silicon evanescent lasers. Optics Express, 15(18):11225, Sep 2007.Google ScholarCross Ref
- P. M. Kogge (editor). Exascale computing study: Technology challenges in achieving exascale systems. Technical Report TR-2008-13, University of Notre Dame, 2008.Google Scholar
- A. Krishnamoorthy, R. Ho, X. Zheng, H. Schwetman, J. Lexau, P. Koka, G. Li, I. Shubin, and J. Cunningham. The integration of silicon photonics and vlsi electronics for computing systems. In Photonics in Switching, 2009. PS '09. International Conference on, pages 1--4, 2009.Google ScholarCross Ref
- M. Lipson. Guiding, Modulating, and Emitting Light on Silicon-Challenges and Opportunities. Journal of Lightwave Technology, 23(12):4222--4238, Dec 2005.Google ScholarCross Ref
- G. Mora, J. Flich, J. Duato, P. López, E. Baydal, and O. Lysne. Towards an efficient switch architecture for high-radix switches. In Proceedings of the 2006 ACM/IEEE symposium on Architecture for Networking and Communications Systems, pages 11--20, 2006. Google ScholarDigital Library
- N. Muralimanohar, R. Balasubramonian, and N. Jouppi. Optimizing NUCA Organizations and Wiring Alternatives for Large Caches with CACTI 6.0. In MICRO, Dec 2007. Google ScholarDigital Library
- R. Palmer, J. Poulton, W. J. Dally, J. Eyles, A. M. Fuller, T. Greer, M. Horowitz, M. Kellam, F. Quan, and F. Zarkeshvarl. A 14mW 6.25Gb/s Transceiver in 90nm CMOS for Serial Chip-to-Chip Communications. In ISSCC, Feb 2007.Google ScholarCross Ref
- S. Scott, D. Abts, J. Kim, and W. J. Dally. The Black Widow High-Radix Clos Network. In ISCA, Jun 2006. Google ScholarDigital Library
- Semiconductor Industries Association. International Technology Roadmap for Semiconductors. http://www.itrs.net, 2009 Edition.Google Scholar
- A. Shacham, K. Bergman, and L. P. Carloni. On the Design of a Photonic Network-on-Chip. In NOCS, pages 53--64, 2007. Google ScholarDigital Library
- D. Vantrease, N. Binkert, R. S. Schreiber, and M. H. Lipasti. Light Speed Arbitration and Flow Control for Nanophotonic Interconnects. In MICRO, Dec 2009. Google ScholarDigital Library
- M. R. Watts, W. A. Zortman, D. C. Trotter, G. N. Nielson, D. L. Luck, and R. W. Young. Adiabatic Resonant Microrings (ARMs) with Directly Integrated Thermal Microphotonics. 2009.Google Scholar
- Q. Xu, B. Schmidt, S. Pradhan, and M. Lipson. Micrometre-Scale Silicon Electro-Optic Modulator. Nature, 435:325--327, May 2005.Google Scholar
Index Terms
- The role of optics in future high radix switch design
Recommendations
The role of optics in future high radix switch design
ISCA '11For large-scale networks, high-radix switches reduce hop and switch count, which decreases latency and power. The ITRS projections for signal-pin count and per-pin bandwidth are nearly flat over the next decade, so increased radix in electronic switches ...
Achieving High Throughput in High-Radix Switch
TRUSTCOM '11: Proceedings of the 2011IEEE 10th International Conference on Trust, Security and Privacy in Computing and CommunicationsCurrent research and trends reveals that high-radix switches can remarkably lower down the network diameter and switch count of interconnection network in high performance computers, and it is much valuable in saving packet latency, cost and power ...
Scalable high-radix router microarchitecture using a network switch organization
As the system size of supercomputers and datacenters increases, cost-efficient networks become critical in achieving good scalability on those systems. High-radix routers reduce network cost by lowering the network diameter while providing a high ...
Comments