Botong Huang
Abstract
We present a system called Cümülön-D for matrix-based data analysis in a spot market of a public cloud. Prices in such markets fluctuate over time: while users can acquire machines usually at a very low bid price, the cloud can terminate these machines as soon as the market price exceeds their bid price. The distinguishing features of Cümülön-D include its continuous, proactive adaptation to the changing market, and its ability to quantify and control the monetary risk involved in paying for a workflow execution. We solve the dynamic optimization problem in a principled manner with a Markov decision process, and account for practical details that are often ignored previously but nonetheless important to performance. We evaluate Cümülön-D's effectiveness and advantages over previous approaches with experiments on Amazon EC2.
- O. Agmon Ben-Yehuda, M. Ben-Yehuda, A. Schuster, and D. Tsafrir. Deconstructing Amazon EC2 spot instance pricing. CloudCom 2011. Google Scholar
Digital Library
- A. Andrzejak, D. Kondo, and S. Yi. Decision model for cloud computing under SLA constraints. MASCOTS 2010. Google ScholarDigital Library
- A. B. Downey. A model for speedup of parallel programs. Technical report, University of California, Berkeley, 1997. Google ScholarDigital Library
- B. Huang, S. Babu, and J. Yang. Cumulon: Optimizing statistical data analysis in the cloud. SIGMOD 2013. Google ScholarDigital Library
- B. Huang, N. W. Jarrett, S. Babu, S. Mukherjee, and J. Yang. Cumülön: Matrix-based data analytics in the cloud with spot instances. PVLDB, 9(3):156--167, Sept. 2015. Detailed technical report version: http://db.cs.duke.edu/papers/pvldb15-HuangJarrettEtAl-cumulon-spot.pdf. Google ScholarDigital Library
- B. Huang and J. Yang. Cumülön-D: Data analytics in a dynamic spot market. Technical report, Duke University, 2016. http://db.cs.duke.edu/papers/2016-HuangYang-cumulon_dynamic_spot.pdf.Google Scholar
- H. Huang, L. Wang, B. Tak, L. Wang, and C. Tang. CAP3: a cloud autoprovisioning framework for parallel processing using on-demand and spot instances. Cloud 2013. Google ScholarDigital Library
- B. Javadi, R. K. Thulasiram, and R. Buyya. Statistical modeling of spot instance prices in public cloud environments. Utility and Cloud Computing 2011. Google ScholarDigital Library
- S. Khatua and N. Mukherjee. Application-centric resource provisioning for Amazon EC2 spot instances. EuroPar 2013. Google ScholarDigital Library
- D. D. Lee and H. S. Seung. Algorithms for non-negative matrix factorization. NIPS 2000. Google ScholarDigital Library
- V. Rokhlin, A. Szlam, and M. Tygert. A randomized algorithm for principal component analysis. SIAM Journal on Matrix Analysis and Applications, 31(3):1100--1124, Aug. 2009.Google ScholarDigital Library
- Y. Song, M. Zafer, and K.-W. Lee. Optimal bidding in spot instance market. INFOCOM 2012.Google ScholarCross Ref
- M. Taifi, J. Y. Shi, and A. Khreishah. SpotMPI: A framework for auction-based HPC computing using amazon spot instances. ICA3PP 2011. Google ScholarDigital Library
- S. Tang, J. Yuan, and X.-Y. Li. Towards optimal bidding strategy for Amazon EC2 cloud spot instance. Cloud 2012. Google ScholarDigital Library
- W. Voorsluys and R. Buyya. Reliable provisioning of spot instances for computeintensive applications. Advanced Information Networking and Applications 2012. Google ScholarDigital Library
- P. Wang, Y. Qi, D. Hui, L. Rao, and X. Liu. Present or future: Optimal pricing for spot instances. ICDCS 2013. Google ScholarDigital Library
- S. Yi, A. Andrzejak, and D. Kondo. Monetary cost-aware checkpointing and migration on amazon cloud spot instances. IEEE Trans. Services Computing, 5(4):512--524, 2012. Google ScholarDigital Library
- M. Zafer, Y. Song, and K. Lee. Optimal bids for spot VMs in a cloud for deadline constrained jobs. Cloud 2012. Google ScholarDigital Library
- Q. Zhang, Q. Zhu, and R. Boutaba. Dynamic resource allocation for spot markets in cloud computing environments. Utility and Cloud Computing 2011. Google ScholarDigital Library
- L. Zheng, C. Joe-Wong, C. Tan, M. Chiang, and X. Wang. How to bid the cloud. SIGCOMM 2015. Google ScholarDigital Library
- A. C. Zhou, B. He, and C. Liu. Monetary cost optimizations for hosting workflow-as-a-service in IaaS clouds. IEEE Trans. Cloud Computing, 4(1):34--48, 2016. Google ScholarDigital Library
Recommendations
Cümülön: matrix-based data analytics in the cloud with spot instances
We describe Cümülön, a system aimed at helping users develop and deploy matrix-based data analysis programs in a public cloud. A key feature of Cümülön is its end-to-end support for the so-called spot instances---machines whose market price fluctuates ...
An equilibrium analysis of trading across multiple double auction marketplaces using fictitious play
We investigate traders' market selection and bidding strategies in multiple double auctions.We model the setting as a Bayesian game with continuous private values.We consider different trading environments with independent, substitutable and ...
Implications of a Reserve Price in an Agent-Based Common-Value Auction
Auction sellers can use a reserve price to require a minimum bid before items are sold. Theoretical and experimental research has tested the influence of a reserve price in an independent private values auction, but little focus has been given to the ...
Comments