Simone Stumpf
Erin Sullivan
ABSTRACT
The potential for machine learning systems to improve via a mutually beneficial exchange of information with users has yet to be explored in much detail. Previously, we found that users were willing to provide a generous amount of rich feedback to machine learning systems, and that the types of some of this rich feedback seem promising for assimilation by machine learning algorithms. Following up on those findings, we ran an experiment to assess the viability of incorporating real-time keyword-based feedback in initial training phases when data is limited. We found that rich feedback improved accuracy but an initial unstable period often caused large fluctuations in classifier behavior. Participants were able to give feedback by relying heavily on system communication in order to respond to changes. The results show that in order to benefit from the user's knowledge, machine learning systems must be able to absorb keyword-based rich feedback in a graceful manner and provide clear explanations of their predictions.
- Altendorf, E., Restificar, E., Dietterich, T. Learning from sparse data by exploiting monotonicity constraints. Proc. UAI (2005).Google Scholar
- Beckwith, L. Kissinger, C., Burnett, M., Wiedenbeck, S., Lawrance, J., Blackwell, A., Cook, C. Tinkering and gender in end-user programmers' debugging, Proc. CHI 2006, ACM Press (2006), 231--240. Google ScholarDigital Library
- Billsus, D., Hilbert, D., Maynes-Aminzade, D. Improving proactive information systems. Proc. IUI (2005), 159--166. Google ScholarDigital Library
- Blythe, J. Task learning by instruction in Tailor. Proc. IUI (2005), 191--198. Google ScholarDigital Library
- Blum, A., Mitchell, T. Combining labeled and unlabeled data with co-training. Proc. COLT (1998). Google ScholarDigital Library
- Brutlag, J., Meek, C. Challenges of the email domain for text classification. Proc. ICML (2000), 103--110. Google ScholarDigital Library
- Byrnes, J. P., Miller, D. C., Schafer W. D. Gender differences in risk taking: A meta-analysis. Psychological Bulletin 125 (1999), 367--383.Google ScholarCross Ref
- Chapelle, O., Scholkopf, B., Zien, A. Semi-Supervised Learning. MIT Press, Cambridge, MA, 2006. Google ScholarDigital Library
- Cohen, W. Learning rules that classify e-mail. Proc. AAAI Spring Symp. Information Access (1996).Google Scholar
- Crawford, E., Kay, J., McCreath, E. IEMS -- The Intelligent Email Sorter. Proc. ICML (2002), 83--90. Google ScholarDigital Library
- Culotta, A. Kristjansson, T. McCallum, A., Viola, P. Corrective Feedback and Persistent Learning for Information Extraction. Artificial Intelligence 170, (2006), 1101--1122. Google ScholarDigital Library
- Cypher, A. (ed.) Watch What I Do: Programming by Demonstration, MIT Press, Cambridge, MA, 1993. Google ScholarDigital Library
- Facione, P. A. The California Critical Thinking Skills Test: College Level Technical Report #2. California Academic Press, Millbrae CA. 1990 (ERIC Document Reproduction Service No. ED 327 550).Google Scholar
- Fails, J. A., Olsen, D. R. Interactive machine learning. Proc. IUI (2003), 39--45. Google ScholarDigital Library
- Fung, G., Mangasarian, O., Shavlik, J. Knowledge-based support vector machine classifiers. Proc. NIPS (2002).Google Scholar
- Hart, S., Staveland, L. Development of a NASA-TLX (Task load index): Results of empirical and theoretical research, Human Mental Workload (1988), Hancock, P. and Meshkati, N. (eds.), 139--183.Google Scholar
- Herlocker, J., Konstan, J., Riedl, J. Explaining collaborative filtering recommendations. Proc. CSCW (2000), 241--250. Google ScholarDigital Library
- Holtzblatt, K., Beyer, H. Making customer-centered design work for teams. Comms ACM 36, 10 (1993), 92--103. Google ScholarDigital Library
- Huang, Y., Mitchell, T. M. Text clustering with extended user feedback. Proc. SIGIR (2006), 413--420. Google ScholarDigital Library
- Lieberman, H., (ed.) Your Wish is My Command: Programming By Example. 2001.Google Scholar
- Liu, B. Li, X. Lee, W., Yu, P. Text Classification by Labeling Words. Proc. AAAI (2004). Google ScholarDigital Library
- McCarthy, K., Reilly, J., McGinty, L., Smyth, B. Experiments in dynamic critiquing. Proc. IUI (2005), 175--182. Google ScholarDigital Library
- McDaniel, R. G. and Myers, B. A. Getting more out of programming-by-demonstration. Proc. CHI (1999), 442--449. Google ScholarDigital Library
- Oblinger, D., Castelli, V., Bergman, L. Augmentation-based learning. Proc. IUI (2006), 202--209. Google ScholarDigital Library
- Phalgune, A., Kissinger, C., Burnett, M., Cook, C., Beckwith, L. Ruthruff, J. Garbage in, garbage out? An empirical look at oracle mistakes by end-user programmers. Proc. VL/HCC (2005), 45--52. Google ScholarDigital Library
- Shen, J., Li, L., Dietterich, T. Herlocker, J. A hybrid learning system for recognizing user tasks from desk activities and email messages. Proc. IUI (2006), 86--92. Google ScholarDigital Library
- Stumpf S, Rajaram V, Li L, Burnett M, Dietterich T, Sullivan E, Drummond R, Herlocker J. Toward Harnessing User Feedback For Machine Learning. Proc. IUI (2007). Google ScholarDigital Library
- Stumpf S, Rajaram V, Li L, Wong W-K, Burnett M, Dietterich T, Sullivan E, Herlocker J. Interacting Meaningfully with Machine Learning Systems: Three Experiments. EECS OSU Technical Report 2007-46, http://eecs.oregonstate.edu/library.Google Scholar
- Ware, M., Frank, E., Holmes, G., Hall, M., Witten, I. H. Interactive machine learning: letting users build classifiers. IJHCS 55 (2001), 281--292. Google ScholarDigital Library
Index Terms
- Integrating rich user feedback into intelligent user interfaces
Recommendations
Toward harnessing user feedback for machine learning
IUI '07: Proceedings of the 12th international conference on Intelligent user interfacesThere has been little research into how end users might be able to communicate advice to machine learning systems. If this resource--the users themselves--could somehow work hand-in-hand with machine learning systems, the accuracy of learning systems ...
Efficiently incorporating user feedback into information extraction and integration programs
SIGMOD '09: Proceedings of the 2009 ACM SIGMOD International Conference on Management of dataMany applications increasingly employ information extraction and integration (IE/II) programs to infer structures from unstructured data. Automatic IE/II are inherently imprecise. Hence such programs often make many IE/II mistakes, and thus can ...
A biologically inspired approach to learning multimodal commands and feedback for human-robot interaction
CHI EA '09: CHI '09 Extended Abstracts on Human Factors in Computing SystemsIn this paper we describe a method to enable a robot to learn how a user gives commands and feedback to it by speech, prosody and touch. We propose a biologically inspired approach based on human associative learning. In the first stage, which ...
Comments