Sharad Goel
Daniel M. Reeves
David M. Pennock
ABSTRACT
Decision makers can benefit from the subjective judgment of experts. For example, estimates of disease prevalence are quite valuable, yet can be difficult to measure objectively. Useful features of mechanisms for aggregating expert opinions include the ability to: (1) incentivize participants to be truthful; (2) adjust for the fact that some experts are better informed than others; and (3) circumvent the need for objective, "ground truth" observations. Subsets of these properties are attainable by previous elicitation methods, including proper scoring rules, prediction markets, and the Bayesian truth serum. Our mechanism of collective revelation, however, is the first to simultaneously achieve all three. Furthermore, we introduce a general technique for constructing budget-balanced mechanisms-where no net payments are made to participants--that applies both to collective revelation and to past peer-prediction methods.
- G. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78:1--3, 1950.Google Scholar
Cross Ref
- Yiling Chen, Stanko Dimitrov, Rahul Sami, Daniel M. Reeves, David M. Pennock, Robin D. Hanson, Lance Fortnow, and Rica Gonen. Gaming prediction markets: Equilibrium strategies with a market maker. Algorithmica, to appear. Google ScholarDigital Library
- Norman Dalkey and Olaf Helmer. An experimental application of the Delphi method to the use of experts. Management Science, 9, 1963.Google Scholar
- Paul H. Garthwaite, Joseph B. Kadane, and Anthony O'Hagan. Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470):680--700, 2005.Google ScholarCross Ref
- Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. Bayesian Data Analysis. Chapman&Hall, 2nd edition, 2004.Google Scholar
- Allan Gibbard. Manipulation of voting schemes: A general result. Econometrica, 41:587--601, 1973.Google ScholarCross Ref
- I.J. Good. Rational decisions. JRSS-B, 1952.Google ScholarCross Ref
- Jerry R. Green and Jean-Jacques Laffont. Incentives in Public Decision-Making. North-Holland Publishing Company, 1979.Google Scholar
- D.J. Johnstone. The parimutuel Kelly probability scoring rule. Decision Analysis, 4(2):66, 2007. Google ScholarDigital Library
- Radu Jurca and Boi Faltings. Incentives for expressing opinions in online polls. In ACM Conference on Electronic Commerce, 2008. Google ScholarDigital Library
- D.M. Kilgour and Y. Gerchak. Elicitation of probabilities using competitive scoring rules. Decision Analysis, 1(2):108--113, 2004. Google ScholarDigital Library
- Nicolas Lambert, John Langford, Jennifer Wortman, Yiling Chen, Daniel Reeves, Yoav Shoham, and David Pennock. Self-financed wagering mechanisms for forecasting. In ACM Conference on Electronic Commerce, pages 170--179, 2008. Google ScholarDigital Library
- Nicolas Lambert and Yoav Shoham. Truthful surveys. Lecture Notes in Computer Science, 5385:154--165, 2008. Google ScholarDigital Library
- Andrew Mas-Colell, Michael D. Whinston, and Jerry R. Green. Microeconomics Theory. Oxford University Press, New York, NY, 1995.Google Scholar
- James E. Matheson and Robert L. Winkler. Scoring rules for continuous probability distributions. Management Science, 22(10):1087--1096, 1976.Google ScholarDigital Library
- Paul Milgrom and Nancy L. Stokey. Information, trade and common knowledge. Journal of Economic Theory, 26(1):17--27, 1982.Google ScholarCross Ref
- Nolan Miller, Paul Resnick, and Richard Zeckhauser. Eliciting informative feedback: The peer-prediction method. Management Science, 51:1359--1373, 2005. Google ScholarDigital Library
- A. Murphy and R. Winkler. Probability forecasting in meteorology. Journal of the American Statistical Association, 79(387):489--500, 1984.Google Scholar
- R. Nelson and D. Bessler. Subjective probabilities and scoring rules: Experimental evidence. Journal of Agricultural Economics, 71(2):363--369, 1989.Google ScholarCross Ref
- F. O'Carroll. Subjective probabilities and short-term economic forecasts: An empirical investigation. Applied Statistics, 26(3):269--278, 1977.Google ScholarCross Ref
- Drazen Prelec. A Bayesian truth serum for subjective data. Science, 306:462--466, 2004.Google ScholarCross Ref
- Leonard J. Savage. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336):783--801, 1971.Google ScholarCross Ref
- D. Spiegelhalter. Probabilistic prediction in patient management and clinical trials. Stat. Med., 5:421--433, 1986.Google ScholarCross Ref
- James Surowiecki. The Wisdom of Crowds. Doubleday, 2004. Google ScholarDigital Library
- Robert L. Winkler. The assessment of prior distributions in Bayesian analysis. Journal of the American Statistical Association, 62(319):776--800, 1967.Google ScholarCross Ref
- Robert L. Winkler. Probabilistic prediction: Some experimental results. Journal of the American Statistical Association, 66(336):675--685, 1971.Google ScholarCross Ref
- Robert L. Winkler, J. Muñoz, J.L. Cervera, J.M. Bernardo, G. Blattenberger, J.B. Kadane, D.V. Lindley, A.H. Murphy, R.M. Oliver, and D. Ríos-Insua. Scoring rules and the evaluation of probabilities. TEST/Springer, 5(1):1--60, 1996.Google Scholar
Index Terms
- Collective revelation: a mechanism for self-verified, weighted, and truthful predictions
Recommendations
Decision rules and decision markets
AAMAS '10: Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1 - Volume 1We explore settings where a principal must make a decision about which action to take to achieve a desired outcome. The principal elicits the probability of achieving the outcome by following each action from a self-interested (but decision-agnostic) ...
Correcting vindictive bidding behaviors in sponsored search auctions
In this study, we aim to develop a pricing mechanism that reduces the effects resulted by vindictive advertisers who bid on sponsored search auctions run by search engine providers. In particular, we aim to ensure payment fairness and price stability in ...
Competing sellers in online markets: reserve prices, shill bidding, and auction fees
AAMAS '06: Proceedings of the fifth international joint conference on Autonomous agents and multiagent systemsIn this paper, we consider competition between sellers offering similar items in concurrent online auctions, where each seller must set its individual auction parameters (such as the reserve price) in such a way as to attract buyers. We show that there ...
Comments