Shailesh Kochhar
ABSTRACT
In this paper we describe Rabj, an engine designed to simplify collecting human input. We have used Rabj to collect over 2.3 million human judgments to augment data mining, data entry, and curation tasks at Freebase over the course of a year. We illustrate several successful applications that have used Rabj to collect human judgment. We describe how the architecture and design decisions of Rabj are affected by the constraints of content agnosticity, data freshness, latency and visibility. We present work aimed at increasing the yield and reliability of human computation efforts. Finally, we discuss empirical observations and lessons learned in the course of a year of operating the service.
- K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor. Freebase: a collaboratively created graph database for structuring human knowledge. In SIGMOD '08: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247--1250, New York, NY, USA, 2008. ACM. Google Scholar
Digital Library
- D. Farber. Google's Marissa Mayer: Speed wins. CNET Between the Lines. http://blogs.zdnet.com/BTL/?p=3925, 2006.Google Scholar
- D. F. Galletta, R. Henry, S. Mccoy, and P. Polak. Web site delays: How tolerant are users? Journal of the Association for Information Systems, 5:1--28, 2004.Google ScholarCross Ref
- A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with Mechanical Turk. In CHI '08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pages 453--456, New York, NY, USA, 2008. ACM. Google ScholarDigital Library
- V. S. Sheng, F. Provost, and P. G. Ipeirotis. Get another label? Improving data quality and data mining using multiple, noisy labelers. In KDD '08: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 614--622, New York, NY, USA, 2008. ACM. Google ScholarDigital Library
- K. Siorpaes and M. Hepp. Games with a purpose for the semantic web. Intelligent Systems, IEEE, 23:50--60, 2008. Google ScholarDigital Library
- R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast -- but is it good? Evaluating non-expert annotations for natural language tasks. In EMNLP, pages 254--263. ACL, 2008. Google ScholarDigital Library
- L. von Ahn. Games with a purpose. IEE Computer Magazine, 39:92--94, 2006. Google ScholarDigital Library
- L. von Ahn and L. Dabbish. Designing games with a purpose. Communications of the ACM, 51(8):58--67, 2008. Google ScholarDigital Library
- L. von Ahn, M. Kedia, and M. Blum. Verbosity: a game for collecting common-sense facts. In Proceedings of ACM CHI 2006 Conference on Human Factors in Computing Systems, volume 1 of Games, pages 75--78. ACM Press, 2006. Google ScholarDigital Library
Index Terms
- The anatomy of a large-scale human computation engine
Recommendations
The anatomy of a large-scale social search engine
WWW '10: Proceedings of the 19th international conference on World wide webWe present Aardvark, a social search engine. With Aardvark, users ask a question, either by instant message, email, web input, text message, or voice. Aardvark then routes the question to the person in the user's extended social network most likely to ...
Virtual Human Anatomy
To learn human anatomy, medical students must practice on cadavers, as must physicians when they want to brush up on their anatomy knowledge. However, cadavers are in short supply in medical schools worldwide. Virtual anatomy and surgery can potentially ...
Comments