John J. Dudley
Per Ola Kristensson
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Interactive Machine Learning (IML) seeks to complement human perception and intelligence by tightly integrating these strengths with the computational power and speed of computers. The interactive process is designed to involve input from the user but does not require the background knowledge or experience that might be necessary to work with more traditional machine learning techniques. Under the IML process, non-experts can apply their domain knowledge and insight over otherwise unwieldy datasets to find patterns of interest or develop complex data-driven applications. This process is co-adaptive in nature and relies on careful management of the interaction between human and machine. User interface design is fundamental to the success of this approach, yet there is a lack of consolidated principles on how such an interface should be implemented. This article presents a detailed review and characterisation of Interactive Machine Learning from an interactive systems perspective. We propose and describe a structural and behavioural model of a generalised IML system and identify solution principles for building effective interfaces for IML. Where possible, these emergent solution principles are contextualised by reference to the broader human-computer interaction literature. Finally, we identify strands of user interface research key to unlocking more efficient and productive non-expert interactive machine learning applications.
Supplemental Material
- Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. AI Mag. 35, 4 (2014), 105--120.Google Scholar
Digital Library
- Saleema Amershi, Max Chickering, Steven M. Drucker, Bongshin Lee, Patrice Simard, and Jina Suh. 2015. ModelTracker: Redesigning performance analysis tools for machine learning. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI’15). ACM, New York, NY, 337--346. Google ScholarDigital Library
- Saleema Amershi, James Fogarty, Ashish Kapoor, and Desney Tan. 2010. Examining multiple potential models in end-user interactive concept learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’10). ACM, New York, NY, 1357--1360. Google ScholarDigital Library
- Saleema Amershi, James Fogarty, Ashish Kapoor, and Desney Tan. 2011a. Effective end-user interaction with machine learning. In Proceedings of the 25th AAAI Conference on Artificial Intelligence. AAAI Press, 1529--1532. Google ScholarDigital Library
- Saleema Amershi, James Fogarty, and Daniel Weld. 2012. ReGroup: Interactive machine learning for on-demand group creation in social networks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’12). ACM, New York, NY, 21--30. Google ScholarDigital Library
- Saleema Amershi, Bongshin Lee, Ashish Kapoor, Ratul Mahajan, and Blaine Christian. 2011c. CueT: Human-guided fast and accurate network alarm triage. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’11). ACM, New York, NY, 157--166. Google ScholarDigital Library
- Saleema Amershi, Bongshin Lee, Ashish Kapoor, Ratul Mahajan, and Blaine Christian. 2011b. Human-guided machine learning for fast and accurate network alarm triage. In Proceedings of the 22nd International Joint Conference on Artificial Intelligence (IJCAI’11). AAAI Press, 2564--2569. Google ScholarDigital Library
- Oisin Mac Aodha, Vassilios Stathopoulos, Gabriel J. Brostow, Michael Terry, Mark Girolami, and Kate E. Jones. 2014. Putting the scientist in the loop -- Accelerating scientific progress with interactive machine learning. In Proceedings of the 2014 22nd International Conference on Pattern Recognition. 9--17. Google ScholarDigital Library
- Michel Beaudouin-Lafon. 2004. Designing interaction, not interfaces. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI’04). ACM, New York, NY, 15--22. Google ScholarDigital Library
- Harry Brenton, Andrea Kleinsmith, and Marco Gillies. 2014. Embodied design of dance visualisations. In Proceedings of the 2014 International Workshop on Movement and Computing (MOCO’14). ACM, New York, NY, 124--129. Google ScholarDigital Library
- Michael Brooks, Saleema Amershi, Bongshin Lee, Steven M. Drucker, Ashish Kapoor, and Patrice Simard. 2015. FeatureInsight: Visual support for error-driven feature ideation in text classification. In Proceedings of the 2015 IEEE Conference on Visual Analytics Science and Technology (VAST’15). 105--112. Google ScholarCross Ref
- E. T. Brown, J. Liu, C. E. Brodley, and R. Chang. 2012. Dis-function: Learning distance functions interactively. In Proceedings of the 2012 IEEE Conference on Visual Analytics Science and Technology (VAST’12). 83--92. Google ScholarDigital Library
- Nicholas J. Bryan, Gautham J. Mysore, and Ge Wang. 2014. ISSE: An interactive source separation editor. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’14). ACM, New York, NY, 257--266. Google ScholarDigital Library
- Maya Cakmak, Crystal Chao, and Andrea L. Thomaz. 2010. Designing interactions for robot active learners. IEEE Trans. Auton. Mental Dev. 2, 2 (Jun. 2010), 108--118. Google ScholarDigital Library
- Maya Cakmak and Andrea L. Thomaz. 2014. Eliciting good teaching from humans for machine learners. Artif. Intell. 217 (Dec. 2014), 198--215. Google ScholarDigital Library
- Nadya A. Calderon, Brian Fisher, Jeff Hemsley, Billy Ceskavich, Greg Jansen, Richard Marciano, and Victoria L. Lemieux. 2015. Mixed-initiative social media analytics at the World Bank: Observations of citizen sentiment in Twitter data to explore “trust” of political actors and state institutions and its relationship to social protest. In Proceedings of the 2015 IEEE International Conference on Big Data (Big Data’15). 1678--1687. Google ScholarDigital Library
- Stuart K. Card, Jock D. Mackinlay, and Ben Shneiderman. 1999. Readings in Information Visualization: Using Vision to Think. Morgan Kaufmann Publishers, San Francisco, CA. Google ScholarDigital Library
- Sheelagh Carpendale. 2008. Evaluating information visualizations. In Information Visualization: Human-Centered Issues and Perspectives, Andreas Kerren, John T. Stasko, Jean-Daniel Fekete, and Chris North (Eds.). Springer, Berlin, 19--45. Google ScholarDigital Library
- Shuo Chang, Peng Dai, Lichan Hong, Cheng Sheng, Tianjiao Zhang, and Ed H. Chi. 2016. AppGrouper: Knowledge-based interactive clustering tool for app search results. In Proceedings of the 21st International Conference on Intelligent User Interfaces (IUI’16). ACM, New York, NY, 348--358. Google ScholarDigital Library
- Justin Cheng and Michael S. Bernstein. 2015. Flock: Hybrid crowd-machine learning classifiers. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW’15). ACM, New York, NY, 600--611. Google ScholarDigital Library
- E. H. Chi and John Riedl. 1998. An operator interaction framework for visualization systems. In Proceedings of the 1998 IEEE Symposium on Information Visualization (INFOVIS’98). IEEE Computer Society, Washington, DC, 63--70. http://dl.acm.org/citation.cfm?id=647341.721078 Google ScholarDigital Library
- Eric C.-P. Chua, Kunjan Patel, Mary Fitzsimons, and Chris J. Bleakley. 2011. Improved patient specific seizure detection during pre-surgical evaluation. Clin. Neurophysiol. 122, 4 (2011), 672--679. Google ScholarCross Ref
- William Curran, Travis Moore, Todd Kulesza, Weng-Keen Wong, Sinisa Todorovic, Simone Stumpf, Rachel White, and Margaret Burnett. 2012. Towards recognizing “cool”: Can end users help computer vision recognize subjective attributes of objects in images? In Proceedings of the 17th International Conference on Intelligent User Interfaces (IUI’12). ACM, New York, NY, 285--288. Google ScholarDigital Library
- Anind K. Dey, Raffay Hamid, Chris Beckmann, Ian Li, and Daniel Hsu. 2004. A CAPpella: Programming by demonstration of context-aware applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’04). ACM, New York, NY, 33--40. Google ScholarDigital Library
- Kirstin Early, Stephen E. Fienberg, and Jennifer Mankoff. 2016. Test time feature ordering with FOCUS: Interactive predictions with minimal user burden. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp’16). ACM, New York, NY, 992--1003. Google ScholarDigital Library
- Alex Endert, Patrick Fiaux, and Chris North. 2012. Semantic interaction for visual text analytics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’12). ACM, New York, NY, 473--482. Google ScholarDigital Library
- Jerry Alan Fails and Dan R. Olsen, Jr. 2003. Interactive machine learning. In Proceedings of the 8th International Conference on Intelligent User Interfaces (IUI’03). ACM, New York, NY, 39--45. Google ScholarDigital Library
- Rebecca Fiebrink, Perry R. Cook, and Dan Trueman. 2011. Human model evaluation in interactive supervised learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’11). ACM, New York, NY, 147--156. Google ScholarDigital Library
- Rebecca Anne Fiebrink. 2011. Real-time Human Interaction with Supervised Learning Algorithms for Music Composition and Performance. Ph.D. thesis. Princeton University, Princeton, NJ.Google ScholarDigital Library
- James Fogarty, Desney Tan, Ashish Kapoor, and Simon Winder. 2008. CueFlik: Interactive concept learning in image search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’08). ACM, New York, NY, 29--38. Google ScholarDigital Library
- B. J. Fogg and Hsiang Tseng. 1999. The elements of computer credibility. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’99). ACM, New York, NY, 80--87. Google ScholarDigital Library
- Jules Françoise, Frédéric Bevilacqua, and Thecla Schiphorst. 2016. GaussBox: Prototyping movement interaction with interactive visualizations of machine learning. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA’16). ACM, New York, NY, 3667--3670. Google ScholarDigital Library
- Zoubin Ghahramani. 2015. Probabilistic machine learning and artificial intelligence. Nature 521 (2015), 452--459. Google ScholarCross Ref
- Rayid Ghani and Mohit Kumar. 2011. Interactive learning for efficiently detecting errors in insurance claims. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’11). ACM, New York, NY, 325--333. Google ScholarDigital Library
- Marco Gillies, Harry Brenton, and Andrea Kleinsmith. 2015. Embodied design of full bodied interaction with virtual humans. In Proceedings of the 2nd International Workshop on Movement and Computing (MOCO’15). ACM, New York, NY, 1--8. Google ScholarDigital Library
- Marco Gillies, Rebecca Fiebrink, Atau Tanaka, Jérémie Garcia, Frédéric Bevilacqua, Alexis Heloir, Fabrizio Nunnari, Wendy Mackay, Saleema Amershi, Bongshin Lee, Nicolas d’Alessandro, Joëlle Tilmanne, Todd Kulesza, and Baptiste Caramiaux. 2016. Human-centred machine learning. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA’16). ACM, New York, NY, 3558--3565. Google ScholarDigital Library
- Marco Gillies, Andrea Kleinsmith, and Harry Brenton. 2015. Applying the CASSM framework to improving end user debugging of interactive machine learning. In Proceedings of the 20th International Conference on Intelligent User Interfaces (IUI’15). ACM, New York, NY, 181--185. Google ScholarDigital Library
- Dominic Girardi, Josef Kueng, and Andreas Holzinger. 2015. A domain-expert centered process model for knowledge discovery in medical research: Putting the expert-in-the-loop. In Proceedings of the International Conference on Brain Informatics and Health. Springer, 389--398. Google ScholarCross Ref
- D. Gopinath, S. Jain, and B. D. Argall. 2017. Human-in-the-loop optimization of shared autonomy in assistive robotics. IEEE Robot. Automat. Lett. 2, 1 (Jan 2017), 247--254. Google ScholarCross Ref
- A. Groce, T. Kulesza, C. Zhang, S. Shamasunder, M. Burnett, W. K. Wong, S. Stumpf, S. Das, A. Shinsel, F. Bice, and K. McIntosh. 2014. You are the only possible oracle: Effective test selection for end users of interactive machine learning systems. IEEE Trans. Softw. Eng. 40, 3 (Mar. 2014), 307--323. Google ScholarDigital Library
- Xuan Guo, Qi Yu, Rui Li, Cecilia Ovesdotter Alm, Cara Calvelli, Pengcheng Shi, and Anne Haake. 2016. An expert-in-the-loop paradigm for learning medical image grouping. In Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, 477--488. Google ScholarCross Ref
- Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. ACM SIGKDD Explor. Newslett. 11, 1 (Nov. 2009), 10--18. Google ScholarDigital Library
- Björn Hartmann, Leith Abdulla, Manas Mittal, and Scott R. Klemmer. 2007. Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’07). ACM, New York, NY, 145--154. Google ScholarDigital Library
- Neal Harvey and Reid Porter. 2016. User-driven sampling strategies in image exploitation. Inf. Vis. 15, 1 (2016), 64--74. Google ScholarCross Ref
- Kyle Hipke, Michael Toomim, Rebecca Fiebrink, and James Fogarty. 2014. BeatBox: End-user interactive definition and training of recognizers for percussive vocalizations. In Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces (AVI’14). ACM, New York, NY, 121--124. Google ScholarDigital Library
- Andreas Holzinger. 2013. Human-computer interaction and knowledge discovery (HCI-KDD): What is the benefit of bringing those two fields to work together? In Availability, Reliability, and Security in Information Systems and HCI, Alfredo Cuzzocrea, Christian Kittl, Dimitris E. Simos, Edgar Weippl, and Lida Xu (Eds.). Springer, Berlin, 319--328. Google ScholarCross Ref
- Andreas Holzinger and Igor Jurisica. 2014. The future is in integrative, interactive machine learning solutions. In Knowledge Discovery and Data Mining in Biomedical Informatics, Andreas Holzinger and Igor Jurisica (Eds.). Springer, Berlin, Heidelberg, Chapter 1, 1--18. Google ScholarCross Ref
- Andreas Holzinger, Markus Plass, Katharina Holzinger, Gloria Cerasela Crişan, Camelia-M. Pintea, and Vasile Palade. 2016. Towards interactive machine learning (iML): Applying ant colony algorithms to solve the traveling salesman problem with the human-in-the-loop approach. In Availability, Reliability, and Security in Information Systems, Francesco Buccafurri, Andreas Holzinger, Peter Kieseberg, A. Min Tjoa, and Edgar Weippl (Eds.). Springer International Publishing, Cham, 81--95. Google ScholarCross Ref
- Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’99). ACM, New York, NY, 159--166. Google ScholarDigital Library
- Shih-Wen Huang, Pei-Fen Tu, Wai-Tat Fu, and Mohammad Amanzadeh. 2013. Leveraging the crowd to improve feature-sentiment analysis of user reviews. In Proceedings of the 18th International Conference on Intelligent User Interfaces (IUI’13). ACM, New York, NY, 3--14. Google ScholarDigital Library
- Shervin Javdani, James Andrew Bagnell, and Siddhartha S. Srinivasa. 2016. Minimizing user cost for shared autonomy. In Proceedings of the 11th ACM/IEEE International Conference on Human Robot Interaction (HRI’16). IEEE Press, Piscataway, NJ, 621--622. http://dl.acm.org/citation.cfm?i=2906831.2907011 Google ScholarDigital Library
- Mayank Kabra, Alice A. Robie, Marta Rivera-Alba, Steven Branson, and Kristin Branson. 2013. JAABA: Interactive machine learning for automatic annotation of animal behavior. Nat. Methods 10, 1 (2013), 64--67. Google ScholarCross Ref
- Ashish Kapoor, Bongshin Lee, Desney Tan, and Eric Horvitz. 2010. Interactive optimization for steering machine classification. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’10). ACM, New York, NY, 1343--1352. Google ScholarDigital Library
- Simon Katan, Mick Grierson, and Rebecca Fiebrink. 2015. Using interactive machine learning to support interface development through workshops with disabled people. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI’15). ACM, New York, NY, 251--254. Google ScholarDigital Library
- Been Kim, Elena Glassman, Brittney Johnson, and Julie Shah. 2015a. iBCM: Interactive Bayesian Case Model Empowering Humans via Intuitive Interaction. Technical Report. MIT Computer Science and Artificial Intelligence Laboratory.Google Scholar
- Been Kim, Kayur Patel, Afshin Rostamizadeh, and Julie Shah. 2015b. Scalable and interpretable data representation for high-dimensional, complex data. In Proceedings of the 29th AAAI Conference on Artificial Intelligence (AAAI’15). AAAI Press, 1763--1769. http://dl.acm.org/citation.cfm?id=2886521.2886565. Google ScholarDigital Library
- Andrea Kleinsmith and Marco Gillies. 2013. Customizing by doing for responsive video game characters. Int. J. Hum.-Comput. Stud. 71, 7--8 (2013), 775--784. Google ScholarDigital Library
- W. Bradley Knox and Peter Stone. 2015. Framing reinforcement learning from human reward: Reward positivity, temporal discounting, episodicity, and performance. Artif. Intell. 225 (2015), 24--50. Google ScholarDigital Library
- Ilker Kose, Mehmet Gokturk, and Kemal Kilic. 2015. An interactive machine-learning-based electronic fraud and abuse detection system in healthcare insurance. Appl. Soft Comput. 36 (Nov. 2015), 283--299. Google ScholarDigital Library
- Nataliya Kosmyna, Franck Tarpin-Bernard, and Bertrand Rivet. 2015. Adding human learning in brain--computer interfaces (BCIs): Towards a practical control modality. ACM Trans. Comput.-Hum. Interact. 22, 3, Article 12 (May 2015), 37 pages. Google ScholarDigital Library
- Anna Kreshuk, Christoph N. Straehle, Christoph Sommer, Ullrich Koethe, Marco Cantoni, Graham Knott, and Fred A. Hamprecht. 2011. Automated detection and segmentation of synaptic contacts in nearly isotropic serial electron microscopy images. PLoS ONE 6, 10 (10 2011), 1--8. Google ScholarCross Ref
- Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, and Denis Charles. 2014. Structured labeling for facilitating concept evolution in machine learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’14). ACM, New York, NY, 3075--3084. Google ScholarDigital Library
- Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. Principles of explanatory debugging to personalize interactive machine learning. In Proceedings of the 20th International Conference on Intelligent User Interfaces (IUI’15). ACM, New York, NY, 126--137. Google ScholarDigital Library
- Todd Kulesza, Simone Stumpf, Margaret Burnett, Sherry Yang, Irwin Kwan, and Weng-Keen Wong. 2013. Too much, too little, or just right? Ways explanations impact end users’ mental models. In Proceedings of the 2013 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC’13). 3--10. Google ScholarCross Ref
- James Robert Lloyd, David Duvenaud, Roger Grosse, Joshua B. Tenenbaum, and Zoubin Ghahramani. 2014. Automatic construction and natural-language description of nonparametric regression models. In Proceedings of the 28th AAAI Conference on Artificial Intelligence (AAAI’14). AAAI Press, 1242--1250. Google ScholarDigital Library
- V. Losing, B. Hammer, and H. Wersing. 2015. Interactive online learning for obstacle classification on a mobile robot. In Proceedings of the 2015 International Joint Conference on Neural Networks (IJCNN’15). 1--8. Google ScholarCross Ref
- Hao Lü, James A. Fogarty, and Yang Li. 2014. Gesture script: Recognizing gestures and their structure using rendering scripts and interactively trained parts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’14). ACM, New York, NY, 1685--1694. Google ScholarDigital Library
- David Meignan, Sigrid Knust, Jean-Marc Frayret, Gilles Pesant, and Nicolas Gaud. 2015. A review and taxonomy of interactive optimization methods in operations research. ACM Trans. Interact. Intell. Syst. 5, 3, Article 17 (Sept. 2015), 43 pages. Google ScholarDigital Library
- Luana Micallef, Pierre Dragicevic, and Jean-Daniel Fekete. 2012. Assessing the effect of visualizations on Bayesian reasoning through crowdsourcing. IEEE Trans. Vis. Comput. Graph. 18, 12 (2012), 2536--2545. Google ScholarDigital Library
- Bonnie M. Muir and Neville Moray. 1996. Trust in automation. Part II. experimental studies of trust and human intervention in a process control simulation. Ergonomics 39, 3 (1996), 429--460. Google ScholarCross Ref
- Jakob Nielsen. 1995. Heuristic evaluation. In Usability Inspection Methods, J. Nielsen and R. L. Mack (Eds.). Wiley & Sons, Chapter 2, 25--62. Google ScholarDigital Library
- Donald A. Norman. 2014. Some observations on mental models. In Mental Models, Dedre Gentner and Albert L. Stevens (Eds.). Psychology Press, New York, NY, Chapter 1, 7--14.Google Scholar
- Alex T. Pang, Craig M. Wittenbrink, and Suresh K. Lodha. 1997. Approaches to uncertainty visualization. Vis. Comput. 13, 8 (1997), 370--390. Google ScholarCross Ref
- Kayur Patel, James Fogarty, James A. Landay, and Beverly Harrison. 2008. Investigating statistical machine learning as a tool for software development. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’08). ACM, New York, NY, 667--676. Google ScholarDigital Library
- Stephen J. Payne and Andrew Howes. 2013. Adaptive interaction: A utility maximization approach to understanding human interaction with technology. Synth. Lect. Hum.-Centered Inf. 6, 1 (2013), 1--111. Google ScholarDigital Library
- Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine learning in python. J. Mach. Learn. Res. 12 (Nov. 2011), 2825--2830. Google ScholarDigital Library
- Peter Pirolli and Stuart Card. 1995. Information foraging in information access environments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’95). ACM Press/Addison-Wesley, New York, NY, 51--58. Google ScholarDigital Library
- Reid Porter, Christy Ruggiero, Don Hush, Neal Harvey, Patrick Kelly, Wayne Scoggins, and Lav Tandon. 2011. Interactive image quantification tools in nuclear material forensics. Proc. Soc. Photo-Opt. Instrum. Eng. 7877 (2011), 787708-1--787708-9. Google ScholarCross Ref
- Reid Porter, James Theiler, and Don Hush. 2013. Interactive machine learning in data exploitation. Comput. Sci. Eng. 15, 5 (Sept 2013), 12--20. Google ScholarDigital Library
- Hema Raghavan, Omid Madani, and Rosie Jones. 2006. Active learning with feedback on features and instances. J. Mach. Learn. Res. 7 (2006), 1655--1686. Google ScholarDigital Library
- M. Ribeiro, K. Grolinger, and M. A. M. Capretz. 2015. MLaaS: Machine learning as a service. In Proceedings of the 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA’15). 896--902. Google ScholarCross Ref
- Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. “Why should I trust you?”: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’16). ACM, New York, NY, 1135--1144. Google ScholarDigital Library
- Alan Ritter and Sumit Basu. 2009. Learning to generalize for complex selection tasks. In Proceedings of the 14th International Conference on Intelligent User Interfaces (IUI’09). ACM, New York, NY, 167--176. Google ScholarDigital Library
- Stephanie L. Rosenthal and Anind K. Dey. 2010. Towards maximizing the accuracy of human-labeled sensor data. In Proceedings of the 15th International Conference on Intelligent User Interfaces (IUI’10). ACM, New York, NY, 259--268. Google ScholarDigital Library
- Dominik Sacha, Hansi Senaratne, Bum Chul Kwon, Geoffrey Ellis, and Daniel A. Keim. 2016. The role of uncertainty, awareness, and trust in visual analytics. IEEE Trans. Vis. Comput. Graph. 22, 1 (2016), 240--249. Google ScholarDigital Library
- Isaias Sanchez-Cortina, Nicolás Serrano, Alberto Sanchis, and Alfons Juan. 2012. A prototype for interactive speech transcription balancing error and supervision effort. In Proceedings of the 17th International Conference on Intelligent User Interfaces (IUI’12). ACM, New York, NY, 325--326. Google ScholarDigital Library
- Alvaro Sarasua, Baptiste Caramiaux, and Atau Tanaka. 2016. Machine learning of personal gesture variation in music conducting. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI’16). ACM, New York, NY, 3428--3432. Google ScholarDigital Library
- Advait Sarkar. 2015. Confidence, command, complexity: Metamodels for structured interaction with machine intelligence. In Proceedings of the 26th Annual Conference of the Psychology of Programming Interest Group (PPIG’15). 23--36.Google Scholar
- Advait Sarkar, Mateja Jamnik, Alan F. Blackwell, and Martin Spott. 2015. Interactive visual machine learning in spreadsheets. In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC’15). 159--163. Google ScholarCross Ref
- Advait Sarkar, Cecily Morrison, Jonas F. Dorn, Rishi Bedi, Saskia Steinheimer, Jacques Boisvert, Jessica Burggraaff, Marcus D’Souza, Peter Kontschieder, Samuel Rota Bulò, Lorcan Walsh, Christian P. Kamm, Yordan Zaykov, Abigail Sellen, and Siân Lindley. 2016. Setwise comparison: Consistent, scalable, continuum labels for computer vision. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI’16). ACM, New York, NY, 261--271. Google ScholarDigital Library
- Jaromır Šavelka, Gaurav Trivedi, and Kevin D. Ashley. 2015. Applying an interactive machine learning approach to statutory analysis. In Proceedings of the 28th Annual Conference on Legal Knowledge and Information Systems (JURIX’15). IOS Press, 101--110.Google Scholar
- Jessica Zeitz Self, Radha Krishnan Vinayagam, J. T. Fry, and Chris North. 2016. Bridging the gap between user intention and model parameters for human-in-the-loop data analytics. In Proceedings of the Workshop on Human-In-the-Loop Data Analytics (HILDA’16). ACM, New York, NY, Article 3, 6 pages. Google ScholarDigital Library
- Burr Settles. 2010. Active Learning Literature Survey. Technical Report 1648. University of Wisconsin-Madison.Google Scholar
- Michael Shilman, Desney S. Tan, and Patrice Simard. 2006. CueTIP: A mixed-initiative interface for correcting handwriting errors. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (UIST’06). ACM, New York, NY, 323--332. Google ScholarDigital Library
- Ben Shneiderman. 1982. The future of interactive systems and the emergence of direct manipulation. Behav. Inf. Technol. 1, 3 (1982), 237--256. Google ScholarCross Ref
- Ben Shneiderman. 1996. The eyes have it: A task by data type taxonomy for information visualizations. In Proceedings of the IEEE Symposium on Visual Languages. IEEE, 336--343. Google ScholarDigital Library
- Charles D. Stolper, Adam Perer, and David Gotz. 2014. Progressive visual analytics: User-driven visual exploration of in-progress analytics. IEEE Trans. Vis. Comput. Graph. 20, 12 (Dec. 2014), 1653--1662. Google ScholarCross Ref
- Simone Stumpf, Vidya Rajaram, Lida Li, Weng-Keen Wong, Margaret Burnett, Thomas Dietterich, Erin Sullivan, and Jonathan Herlocker. 2009. Interacting meaningfully with machine learning systems: Three experiments. Int. J. Hum.-Comput. Stud. 67, 8 (2009), 639--662. Google ScholarDigital Library
- Qiang Sun and Gerald DeJong. 2005. Explanation-augmented SVM: An approach to incorporating domain knowledge into SVM learning. In Proceedings of the 22nd International Conference on Machine Learning (ICML’05). ACM, New York, NY, 864--871. Google ScholarDigital Library
- Justin Talbot, Bongshin Lee, Ashish Kapoor, and Desney S. Tan. 2009. EnsembleMatrix: Interactive visualization to support machine learning with multiple classifiers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’09). ACM, New York, NY, 1283--1292. Google ScholarDigital Library
- Theophanis Tsandilas, Catherine Letondal, and Wendy E. Mackay. 2009. Musink: Composing music through augmented drawing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’09). ACM, New York, NY, 819--828. Google ScholarDigital Library
- Fujio Tsutsumi and Yutaka Tateda. 2009. A method to recognize and count leaves on the surface of a river using user’s knowledge about color of leaves. In New Frontiers in Applied Data Mining, Sanjay Chawla, Takashi Washio, Shin-ichi Minato, Shusaku Tsumoto, Takashi Onoda, Seiji Yamada, and Akihiro Inokuchi (Eds.). Springer, Berlin, 203--212. Google ScholarDigital Library
- Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9 (2008), 2579--2605.Google Scholar
- Cristina Versino and Paolo Lombardi. 2011. Filtering Surveillance Image Streams by Interactive Machine Learning. Springer, Berlin, 289--325. Google ScholarCross Ref
- Keith Vertanen and Per Ola Kristensson. 2008. On the benefits of confidence visualization in speech recognition. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’08). ACM, New York, NY, 1497--1500. Google ScholarDigital Library
- Vedrana Vidulin, Marko Bohanec, and Matjaž Gams. 2014. Combining human analysis and machine data mining to obtain credible data relations. Inf. Sci. 288 (2014), 254--278. Google ScholarDigital Library
- Byron C. Wallace, Kevin Small, Carla E. Brodley, Joseph Lau, and Thomas A. Trikalinos. 2012. Deploying an interactive machine learning system in an evidence-based practice center: Abstrackr. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium (IHI’12). ACM, New York, NY, 819--824. Google ScholarDigital Library
- Malcolm Ware, Eibe Frank, Geoffrey Holmes, Mark Hall, and Ian H. Witten. 2001. Interactive machine learning: Letting users build classifiers. Int. J. Hum.-Comput. Stud. 55, 3 (2001), 281--292. Google ScholarDigital Library
- Weng-Keen Wong, Ian Oberst, Shubhomoy Das, Travis Moore, Simone Stumpf, Kevin McIntosh, and Margaret Burnett. 2011. End-user feature labeling: A locally-weighted regression approach. In Proceedings of the 16th International Conference on Intelligent User Interfaces (IUI’11). ACM, New York, NY, 115--124. Google ScholarDigital Library
- Seid Muhie Yimam, Chris Biemann, Ljiljana Majnaric, Šefket Šabanović, and Andreas Holzinger. 2015. Interactive and iterative annotation for biomedical entity recognition. In Brain Informatics and Health, Yike Guo, Karl Friston, Faisal Aldo, Sean Hill, and Hanchuan Peng (Eds.). Springer International Publishing, Cham, 347--357. Google ScholarCross Ref
Index Terms
- A Review of User Interface Design for Interactive Machine Learning
Recommendations
Recent research advances on interactive machine learning
Interactive machine learning (IML) is an iterative learning process that tightly couples a human with a machine learner, which is widely used by researchers and practitioners to effectively solve a wide variety of real-world application problems. ...
A Pen-Based Prosodic User Interface for Schoolchildren
A prosodic user interface is defined as an user interface that can deal with not only what is entered by the user but also how it is entered. A pen-based user interface provides more prosodic information than a mouse-based graphical user interface. The ...
A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges
DIS '20: Proceedings of the 2020 ACM Designing Interactive Systems ConferenceInteractive reinforcement learning (RL) has been successfully used in various applications in different fields, which has also motivated HCI researchers to contribute in this area. In this paper, we survey interactive RL to empower human-computer ...
Comments