Ravi Jampani
Peter J. Haas
ABSTRACT
To deal with data uncertainty, existing probabilistic database systems augment tuples with attribute-level or tuple-level probability values, which are loaded into the database along with the data itself. This approach can severely limit the system's ability to gracefully handle complex or unforeseen types of uncertainty, and does not permit the uncertainty model to be dynamically parameterized according to the current state of the database. We introduce MCDB, a system for managing uncertain data that is based on a Monte Carlo approach. MCDB represents uncertainty via "VG functions," which are used to pseudorandomly generate realized values for uncertain attributes. VG functions can be parameterized on the results of SQL queries over "parameter tables" that are stored in the database, facilitating what-if analyses. By storing parameters, and not probabilities, and by estimating, rather than exactly computing, the probability distribution over possible query answers, MCDB avoids many of the limitations of prior systems. For example, MCDB can easily handle arbitrary joint probability distributions over discrete or continuous attributes, arbitrarily complex SQL queries, and arbitrary functionals of the query-result distribution such as means, variances, and quantiles. To achieve good performance, MCDB uses novel query processing techniques, executing a query plan exactly once, but over "tuple bundles" instead of ordinary tuples. Experiments indicate that our enhanced functionality can be obtained with acceptable overheads relative to traditional systems.
- P. Agrawal, O. Benjelloun, A. D. Sarma, C. Hayworth, S. U. Nabar, T. Sugihara, and J. Widom. Trio: A system for data, uncertainty, and lineage. In VLDB, 2006. Google Scholar
Digital Library
- P. Andritsos, A. Fuxman, and R. J. Miller. Clean answers over dirty databases: A probabilistic approach. In ICDE, page 30, 2006. Google ScholarDigital Library
- L. Antova, C. Koch, and D. Olteanu. 10106
worlds and beyond: Efficient representation and processing of incomplete information. In ICDE, pages 606--615, 2007.Google Scholar
- L. Antova, C. Koch, and D. Olteanu. MayBMS: Managing incomplete information with probabilistic world-set decompositions. In ICDE, pages 1479--1480, 2007.Google ScholarCross Ref
- B. Biller and B. L. Nelson. Modeling and generating multivariate time-series input processes using a vector autoregressive technique. ACM Trans. Modeling Comput. Simulation, 13(3):211--237, 2003. Google ScholarDigital Library
- D. Burdick, A. Doan, R. Ramakrishnan, and S. Vaithyanathan. OLAP over imprecise data with domain constraints. In VLDB, pages 39--50, 2007. Google ScholarDigital Library
- R. Cheng, D. V. Kalashnikov, and S. Prabhakar. Evaluation of probabilistic queries over imprecise data in constantly-evolving environments. Inf. Syst., 32(1):104--130, 2007. Google ScholarDigital Library
- R. Cheng, S. Singh, and S. Prabhakar. U-DBMS: A database system for managing constantly-evolving data. In VLDB, pages 1271--1274, 2005. Google ScholarDigital Library
- R. Cheng, S. Singh, S. Prabhakar, R. Shah, J. S. Vitter, and Y. Xia. Efficient join processing over uncertain data. In CIKM, pages 738--747, 2006. Google ScholarDigital Library
- D. Chu, A. Deshpande, J. M. Hellerstein, and W. Hong. Approximate data collection in sensor networks using probabilistic models. In ICDE, page 48, 2006. Google ScholarDigital Library
- N. N. Dalvi and D. Suciu. Efficient query evaluation on probabilistic databases. VLDB J., 16(4):523--544, 2007. Google ScholarDigital Library
- A. Deshpande and S. Madden. MauveDB: supporting model-based user views in database systems. In SIGMOD, pages 73--84, 2006. Google ScholarDigital Library
- L. Devroye. Non-Uniform Random Variate Generation. Springer, 1986.Google ScholarCross Ref
- L. Devroye and G. Lugosi. Combinatorial Methods in Density Estimation. Springer, 2001.Google ScholarCross Ref
- G. Fishman. Monte Carlo: Concepts, Algorithms, and Applications. Springer, 1996.Google Scholar
- N. Fuhr and T. Rölleke. A probabilistic relational algebra for the integration of information retrieval and database systems. ACM Trans. Inf. Syst., 15(1):32--66, 1997. Google ScholarDigital Library
- J. E. Gentle. Random Number Generation and Monte Carlo Methods. Springer, second edition, 2003.Google Scholar
- L. Getoor and B. Taskar, editors. Introduction to Statistical Relational Learning. MIT Press, 2007. Google ScholarDigital Library
- R. Gupta and S. Sarawagi. Creating probabilistic databases from information extraction models. In VLDB, pages 965--976, 2006. Google ScholarDigital Library
- J. M. Hellerstein, P. J. Haas, and H. J. Wang. Online aggregation. In SIGMOD, pages 171--182, 1997. Google ScholarDigital Library
- S. G. Henderson and B. L. Nelson, editors. Simulation. North-Holland, 2006.Google Scholar
- C. M. Jermaine, S. Arumugam, A. Pol, and A. Dobra. Scalable approximate query processing with the DBO engine. In SIGMOD, pages 725--736, 2007. Google ScholarDigital Library
- B. Kimelfeld and Y. Sagiv. Matching twigs in probabilistic XML. In VLDB, pages 27--38, 2007. Google ScholarDigital Library
- B. Kimelfeld and Y. Sagiv. Maximally joining probabilistic data. In PODS, pages 303--312, 2007. Google ScholarDigital Library
- R. Murthy and J. Widom. Making aggregation work in uncertain and probabilistic databases. In Proc. 1st Int. VLDB Work. Mgmt. Uncertain Data (MUD), pages 76--90, 2007.Google Scholar
- A. Nadas. An extension of a theorem by Chow and Robbins on sequential confidence intervals for the mean. Ann. Math. Statist., 40(2):667--671, 1969.Google ScholarCross Ref
- R. B. Nelsen. An Introduction to Copulas. Springer, second edition, 2006. Google ScholarDigital Library
- A. O'Hagan and J. J. Forster. Bayesian Inference. Volume 2B of Kendall's Advanced Theory of Statistics. Arnold, second edition, 2004.Google Scholar
- J. Pei, B. Jiang, X. Lin, and Y. Yuan. Probabilistic skylines on uncertain data. In VLDB, pages 15--26, 2007. Google ScholarDigital Library
- C. Re, N. N. Dalvi, and D. Suciu. Query evaluation on probabilistic databases. IEEE Data Eng. Bull., 29(1):25--31, 2006.Google Scholar
- C. Re, N. N. Dalvi, and D. Suciu. Efficient top-k query evaluation on probabilistic data. In ICDE, pages 886--895, 2007.Google ScholarCross Ref
- C. Re and D. Suciu. Materialized views in probabilistic databases for information exchange and query optimization. In VLDB, pages 51--62, 2007. Google ScholarDigital Library
- P. Sen and A. Deshpande. Representing and querying correlated tuples in probabilistic databases. In ICDE, pages 596--605, 2007.Google ScholarCross Ref
- R. J. Serfling. Approximation Theorems of Mathematical Statistics. Wiley, 1980.Google Scholar
- S. Singh, C. Mayfield, S. Prabhakar, R. Shah, and S. E. Hambrusch. Indexing uncertain categorical data. In ICDE, pages 616--625, 2007.Google ScholarCross Ref
- J. Xie, J. Yang, Y. Chen, H. Wang, and P. Yu. A sampling-based approach to information recovery. In ICDE, 2008. To appear. Google ScholarDigital Library
Index Terms
- MCDB: a monte carlo approach to managing uncertain data
Recommendations
The monte carlo database system: Stochastic analysis close to the data
The application of stochastic models and analysis techniques to large datasets is now commonplace. Unfortunately, in practice this usually means extracting data from a database system into an external tool (such as SAS, R, Arena, or Matlab), and then ...
MCDB-R: risk analysis in the database
Enterprises often need to assess and manage the risk arising from uncertainty in their data. Such uncertainty is typically modeled as a probability distribution over the uncertain data values, specified by means of a complex (often predictive) ...
Evaluation of probabilistic threshold queries in MCDB
SIGMOD '10: Proceedings of the 2010 ACM SIGMOD International Conference on Management of dataMCDB is a prototype database system for managing stochastic models for uncertain data. In this paper, we study the problem of how to use MCDB to answer statistical queries that search for database objects which satisfy some filter condition with greater ...
Comments