Weitere Artikel dieser Ausgabe durch Wischen aufrufen
Processing a workflow (or a job) created by a user, who can be a researcher from a scientific laboratory or an analysis from a commercial organization, is the main functionality that a data center or a high-performance computing center is generally expected to provide. It can be accomplished with a single core processor and rather small amount of memory if the problem is adequately small while it may require thousands of nodes to solve a complicated problem and peta-bytes of storage for its output. Also specific applications on various platforms are required in general by users for resolving the problems appropriately. In this aspect, a data center should operate non-homogeneous systems for resource management, so-called batch system, in which it results in inefficient resource utilization due to stochastic behavior of user activity. Implementation of virtualization for resource management, e.g. Cloud Computing, is one of promising solutions recently arising, however, it results in the increase of complexity of the system itself as well as the system administration because it naturally implies the intervention of virtualization stack, e.g. hypervisor, between Operating System and applications for resource management. In this paper, we propose a new conceptual design to be implemented as a pre-scheduler capable to insert user submitted jobs dedicated to a specific batch system into available resources managed by other kind of batch systems. The proposed design features transparency in between clients and batch systems, accuracy in terms of monitoring and prediction on the available resources, and scalability for additional batch systems. We suggest the implementation example of the conceptual design based on the scenario established from our experience of operating a data center.
Bitte loggen Sie sich ein, um Zugang zu diesem Inhalt zu erhalten
Sie möchten Zugang zu diesem Inhalt erhalten? Dann informieren Sie sich jetzt über unsere Produkte:
Bandyopadhyay, D., & Sen, J. (2011). Internet of things: Applications and challenges in technology and standardization. Wireless Personal Communications, 58(1), 49–69. CrossRef
Park, S.-T., Kim, Y.-R., Jeong, S.-P., Hong, C.-I., & Kang, T.-G. (2016). A case study on effective technique of distributed data storage for big data processing in the wireless internet environment. Wireless Personal Communications, 86(1), 239–253.
Marias, G. F., Prigouris, N., Papazafeiropoulos, G., Hadjiefthymiades, S., & Merakos, L. (2004). Brokering positioning data from heterogeneous infrastructures. Wireless Personal Communications, 30(2–4), 233–245. CrossRef
Hager, M., Finke, T., Seitz, J., & Waas, T. (2014). Software-based management for ethernet networks. Wireless Personal Communications, 74(3), 1021–1032. CrossRef
Lamanna, M. (2004). The LHC computing grid project at CERN. Nuclear Instruments & Methods in Physics Research, Section A: Accelerators, Spectrometers, Detectors, and Associated Equipment, 534(1–2), 1–6. CrossRef
Evans, L., & Bryant, P. (2008). LHC machine. Journal of Instrumentation, 3, S08001. CrossRef
Gagliardi, F. (2004). The EGEE European grid infrastructure project. In Proceedings of High Performance Computing for Computational Science, pp. 194–203.
Pordes, R., Petravick, D., Kramer, B., Olson, D., Livny, M., Roy, A., et al. (2007). The open science grid. Journal of Physics: Conference Series, 78(1), 012057.
Ahn, S. U., Yeo, I. Y., & Park, S. O. (2014). Secure and efficient high-performance PROOF-based cluster system for high-energy physics. Journal of Supercomputing, 70(1), 166–176. CrossRef
Tera-scale Open-source Resource and QUEue manager (TORQUE), class. http://www.clusterresources.com/pages/products/torque-resource-manager.php.
Litzkow, M., & Livny, M. (1990). Experience with the condor distributed batch system. In Proceedings of IEEE Workshop on Experimental Distributed Systems, pp. 97–101.
Henderson, R. L. (1995). Job scheduling under the portable batch system. In Proceedings of Workshop on Job Scheduling Strategies for Parallel Processing, pp. 279–294.
Nitzberg, B., Schopf, J. M., & Jones, J. P. (2004). PBS Pro: Grid computing and scheduling attributes. Grid Resource Management, 64, 183–190. CrossRef
Gentzsch, W. (2001). Sun grid engine: Towards creating a compute power grid. In Proceedings of First IEEE/ACM International Symposium on Cluster Computing and the Grid, pp. 35–36.
Univa Grid Engine. http://www.univa.com/products/grid-engine.php.
Zhou, S. (1992). LSF: Load sharing in large heterogeneous distributed systems. In Proceedings of Workshop on Cluster Computing.
Prenneis Jr., A. (1996). Loadleveler: Workload management for parallel and distributed computing environments. In Proceedings of Supercomputing Europe, pp. 176.
Yoo, A. B., Jette, M. A., & Grondona, M. (2003). Slurm: Simple linux utility for resource management. In Proceedings of Workshop on Job Scheduling Strategies for Parallel Processing, pp. 44–60.
Yan, Y., & Chapman, B. (2008). Comparative study of distributed resource management systems-SGE, LSF, PBS Pro, and LoadLeveler”, Technical Report-Citeseerx.
Feitelson, D. G., Rudolph, L., & Schwiegelshohn, U. (2005). Parallel job scheduling—A status report. In Proceedings of Job Strategies for Parallel Processing, pp. 1–16.
Foster, I., & Kesselman, C. (2003). The Grid 2: Blueprint for a new computing infrastructure. Philadelphia: Elsevier.
Garzoglio, G., Levshina, T., Mhashilkar, P., & Timm, S. (2009). ReSS: A resource selection service for the open science grid. In S. C. Lin & E. Yen (Eds.), Grid Computing (pp. 89–98). Boston, MA: Springer. CrossRef
Kim, C. W., Yoon, H., Jin, D., & Park, S. O. (2015). Integrated management system for a large computing resources in a scientific data center. Journal of Supercomputing,. doi: 10.1007/s11227-015-1480-2.
Yoon, H., Yeo, I. Y., & Kim, J. H. (2014). Updating the trusted connection of re-organized computing resource under the automated system management platform. Journal of Supercomputing, 70(1), 200–210. CrossRef
Zhang, Q., Cheng, L., & Boutaba, R. (2010). Cloud computing: state-of-the-art and research challenges. Journal of Internet Services and Applications, 1(1), 7–18. CrossRef
Jackson, D., Snell, Q., & Clement, M. (2001). Core algorithms of the Maui scheduler. In Proceedings of Workshop on Job Scheduling Strategies for Parallel Processing, pp. 87–102.
- A Conceptual Design of Job Pre-processing Flow for Heterogeneous Batch Systems in Data Center
Sang Un Ahn
- Springer US
Neuer Inhalt/© Filograph | Getty Images | iStock