Kinshuk Govil
Mendel Rosenblum
Abstract
Despite the fact that large-scale shared-memory multiprocessors have been commercially available for several years, system software that fully utilizes all their features is still not available, mostly due to the complexity and cost of making the required changes to the operating system. A recently proposed approach, called Disco, substantially reduces this development cost by using a virtual machine monitor that laverages the existing operating system technology. In this paper we present a system called Cellular Disco that extends the Disco work to provide all the advantages of the hardware partitioning and scalable operating system approaches. We argue that Cellular Disco can achieve these benefits at only a small fraction of the development cost of modifying the operating system. Cellular Disco effectively turns a large-scale shared-memory multiprocessor into a virtual cluster that supports fault containment and heterogeneity, while avoiding operating system scalability bottlenecks. Yet at the same time, Cellular Disco preserves the benefits of a shared-memory multiprocessor by implementing dynamic, fine-grained resource sharing, and by allowing users to overcommit resources such as processors and memory. This hybrid approach requires a scalable resource manager that makes local decisions with limited information while still providing good global performance and fault containment. In this paper we describe our experience with a Cellular Disco prototype on a 32-processor SGI Origin 2000 system. We show that the execution time penalty for this approach is low, typically within 10% of the best available commercial operating system formost workloads, and that it can manage the CPU and memory resources of the machine significantly better than the hardware partitioning approach.
- BARTON,J.M.AND BITAR, N. 1995. A scalable multi-discipline, multiple-processor schedul-ing framework for IRIX. Lecture Notes in Computer Science 949, 45-69. Google Scholar
- BUGNION, E., DEVINE, S., GOVIL, K., AND ROSENBLUM, M. 1997. Disco: Running commodity operating systems on scalable multiprocessors. ACM Transactions on Computer Systems 15, 4 (November), 412-447. Google Scholar
- CHAPIN, J., ROSENBLUM, M., DEVINE, S., LAHIRI, T., TEODOSIU, D., AND GUPTA, A. 1995. Hive: Fault containment for shared-memory multiprocessors. In Proceedings of the 15th ACM Symposium on Operating Systems Principles (SOSP) (December 1995), pp. 12-25. Google Scholar
- COMPAQ. 2000. OpenVMS Galaxy. Available: http://www.openvms.compaq.com/availability/ galaxy.html.Google Scholar
- CREASY, R. 1981. The origin of the VM/370 time-sharing system. IBM J. Res. Develop 25,5, 483-490.Google Scholar
- FEELEY,M.J.,MORGAN,W.E.,PIGHIN,F.H.,KARLIN,A.R.,LEVY,H.M.,AND THEKKATH,C. 1995. Implementing global memory management in a workstation cluster. In Proceedings of the 15th ACM Symposium on Operating Systems Principles (SOSP) (December 1995), pp. 201-212. Google Scholar
- GALLES,M.AND WILLIAMS, E. 1994. Performance optimizations, implementation, and verifi-cation of the SGI Challenge multiprocessor. In T. N. Mudge and B. D. Shriver Eds., Proceedings of the 27th Hawaii International Conference on System Sciences. Volume 1: Architecture (Los Alamitos, CA, USA, January 1994), pp. 134-143. IEEE Computer Society Press.Google Scholar
- GAMSA, B., KRIEGER, O., APPAVOO, J., AND STUMM, M. 1999. Tornado: Maximizing locality and concurrency in a shared memory multiprocessor operating system. In Proceedings of the 3rd Symposium on Operating Systems Design and Implementation (OSDI) (February 1999), pp. 87-100. Google Scholar
- GOLDBERG, R. P. 1974. Survey of virtual machine research. IEEE Computer Magazine 7,4 (June), 34-45.Google Scholar
- IBM. 2000. The K42 Project. Available: http://www.research.ibm.com/K42/index.html.Google Scholar
- KANE,G.AND HEINRICH, J. 1992. MIPS RISC Architecture. Prentice Hall. Google Scholar
- KARGER,P.A.,ZURKO,M.E.,BONIN,D.W.,MASON,A.H.,AND KAHN, C. E. 1991. A retrospective on the VAX VMM security kernel. IEEE Transactions on Software Engineering 17, 11 (November), 1147-1165. Special Section on Security and Privacy. Google Scholar
- KUSKIN, J., OFELT, D., HEINRICH, M., HEINLEIN, J., SIMONI, R., GHARACHORLOO, K., CHAPIN, J., NAKAHIRA, D., BAXTER, J., HOROWITZ, M., GUPTA, A., ROSENBLUM, M., AND HENNESSY,J. 1994. The Stanford FLASH Multiprocessor. In Proceedings of the 21st International Symposium on Computer Architecture (ISCA) (April 1994), pp. 302-313. Google Scholar
- LAUDON,J.AND LENOSKI, D. 1997. The SGI Origin: A ccNUMA highly scalable server. In Proceedings of the 24th Annual International Symposium on Computer Architecture (ISCA) (June 1997), pp. 241-251. Google Scholar
- LEVY,H.M.AND LIPMAN, P. H. 1982. Virtual memory management in the VAX/VMS operating system. Computer 15, 3 (March), 35-41.Google Scholar
- LYNCH, N. A. 1996. Distributed Algorithms. Morgan Kaufmann series in data management systems. Morgan Kaufmann Publishers, Los Altos, CA. Google Scholar
- MILOJICIC,D.S.,DOUGLIS, F., PAINDAVEINE, Y., WHEELER, R., AND ZHOU, S. 1996. Process migration. Technical report (December), TOG Research Institute.Google Scholar
- RASHID, R., TEVANIAN, A., YOUNG, M., GOLUB, D., BARON, R., BALCK, D., BOLOSKY,W.J.,AND CHEW, J. 1988. Machine-independent virtual memory management for paged uniproces-sor and multiprocessor architectures. IEEE Transactions on Computer 37, 8 (August), 896-908. Google Scholar
- ROSENBLUM, M., BUGNION, E., HERROD,S.A.,AND DEVINE, S. 1997. Using the SimOS machine simulator to study complex computer systems. ACM Transactions on Modeling and Computer Simulation 7, 1 (January), 78-103. Google Scholar
- SEAWRIGHT,L.H.AND MACKINNON, R. A. 1979. VM/370: a study of multiplicity and useful-ness. IBM Systems Journal 18, 1, 4-17.Google Scholar
- SEQUENT COMPUTER SYSTEMS. 2000. Application Region Manager. Available: http://www. sequent.com/dcsolutions/agile_wp1.html.Google Scholar
- SGI. 2000. IRIX6.5. Available: http://www.sgi.com/software/irix6.5.Google Scholar
- SPEC. 2000. SPECweb96 Benchmark. Available: http://www.spec.org/osg/web96.Google Scholar
- SUN MICROSYSTEMS. 2000. Sun Enterprise 10000: Dynamic System Domains. Available: http://www.sun.com/servers/highend/10000/Tour/domains.html.Google Scholar
- TEODOSIU, D., BAXTER, J., GOVIL, K., CHAPIN, J., ROSENBLUM, M., AND HOROWITZ, M. 1997. Hardware fault containment in scalable shared-memory multiprocessors. In Proceedings of the 24th Annual International Symposium on Computer Architecture (ISCA), Volume 25, 2 of Computer Architecture News (New York, June 1997), pp. 73-84. ACM Press. Google Scholar
- TRANSACTION PROCESSING PERFORMANCE COUNCIL. 1997. TPC Benchmark D (Decision Sup-port) Standard Specification. TPC, San Jose, CA.Google Scholar
- UNISYS. 2000. Cellular MultiProcessing Architecture. Available: http://www.unisys.com/ marketplace/ent/cmp.html.Google Scholar
- VERGHESE, B., DEVINE, S., GUPTA, A., AND ROSENBLUM, M. 1996. Operating system support for improving data locality on CC-NUMA computer servers. In Proceedings of the Seventh International Conference on Architectural Support for Programming Languages and Operat-ing Systems (ASPLOS) (October 1996), pp. 279-289. Google Scholar
- VMWARE. 2000. VMware Virtual Platform. Available: http://www.vmware.com/products/ virtualplatform.html.Google Scholar
- WOO,S.C.,OHARA, M., TORRIE, E., SHINGH,J.P.,AND GUPTA, A. 1995. The SPLASH-2 programs: Characterization and methodological considerations. In Proceedings of the 22nd Annual International Symposium on Computer Architecture (Santa Margherita Ligure, Italy, June 1995), pp. 24-36. ACM SIGARCH and IEEE Computer Society TCCA. Google Scholar
Index Terms
- Cellular disco: resource management using virtual clusters on shared-memory multiprocessors
Recommendations
Cellular Disco: resource management using virtual clusters on shared-memory multiprocessors
SOSP '99: Proceedings of the seventeenth ACM symposium on Operating systems principlesDespite the fact that large-scale shared-memory multiprocessors have been commercially available for several years, system software that fully utilizes all their features is still not available, mostly due to the complexity and cost of making the ...
Disco: running commodity operating systems on scalable multiprocessors
In this article we examine the problem of extending modern operating systems to run efficiently on large-scale shared-memory multiprocessors without a large implementation effort. Our approach brings back an idea popular in the 1970s: virtual machine ...
Cellular Disco: resource management using virtual clusters on shared-memory multiprocessors
Despite the fact that large-scale shared-memory multiprocessors have been commercially available for several years, system software that fully utilizes all their features is still not available, mostly due to the complexity and cost of making the ...
Reviews
Andrew Robert Huber
Disco [1] is a virtual machine monitor that turns shared memory multiprocessors into multiple virtual machines, each running an unmodified commercial operating system. Disco can take advantage of technology such as non-uniform memory access (NUMA) that the commodity systems do not support. But without support for hardware partitioning in Disco, any hardware fault causes total system failure. Cellular Disco adds support for fault containment and scalable resource management to address these drawbacks in Disco. Cellular Disco is structured as a virtual cluster of semi- independent cells. These cells must be divisible into one or more fault containment units with fail-stop behavior. This limits Cellular Disco’s generality, since few shared-memory systems exhibit fail- stop behavior. Fault recovery is further simplified by assuming the correctness of Disco’s code, allowing cells to trust each other. The authors justify this assumption by the small amount of code (50K lines). Good fault containment requires many cells, while good resource management needs few cells with many resources. CPU and memory management support was added to address this tradeoff, including process migration for load balancing and memory borrowing between cells. Global shared memory was also added to support large applications split across virtual machines. Experiments on a 32- processor SGI Origin system using database, I/O, CPU, and kernel intensive workloads showed the worst-case virtualization costs of Cellular Disco to be 9% in the uniprocessor case, and 20% for a 32- processor system executing a single virtual machine. Comparing 1-cell to 8-cell systems showed that the cost of fault containment was only 1%. The paper includes valuable sections discussing other approaches and related work. The work is interesting, well reported, and recommended to all with an interest in operating systems or multi- processors. [1] Bugnion, E., Devine, S., Govil, K., and Rosenblum, M. 1997. Disco: Running commodity operating systems on scalable multi-processors. ACM Transactions on Computer Systems, 15, 4 (November), 412-447. Online Computing Reviews Service
Access critical reviews of Computing literature here
Become a reviewer for Computing Reviews.
Comments