Diane Kelly
Jaime Arguello
ABSTRACT
One of the most challenging aspects of designing interactive information retrieval (IIR) experiments with users is the development of search tasks. We describe an evaluation of 20 search tasks that were designed for use in IIR experiments and developed using a cognitive complexity framework from educational theory. The search tasks represent five levels of cognitive complexity and four topical domains. The tasks were evaluated in the context of a laboratory IIR experiment with 48 participants. Behavioral and self-report data were used to characterize and understand differences among tasks. Results showed more cognitively complex tasks required significantly more search activity from participants (e.g., more queries, clicks, and time to complete). However, participants did not evaluate more cognitively complex tasks as more difficult and were equally satisfied with their performances across tasks. Our work makes four contributions: (1) it adds to what is known about the relationship among task, search behaviors and user experience; (2) it presents a framework for task creation and evaluation; (3) it provides tasks and questionnaires that can be reused by others and (4) it raises questions about findings and assumptions of many recent studies that only use behavioral signals from search logs as evidence for task difficulty and searcher satisfaction, as many of our results directly contradict these findings.
- Ageev, M., Guo, Q., Lagun, D. & Agichtein, E. (2011). Find it if you can: A game for modeling different types of web search success using interaction data. Proc. of SIGIR, 345--354. Google Scholar
Digital Library
- Allan, J., Croft, B., Moffat, A. & Sanderson, M. (Eds). (2012). Frontiers, challenges and opportunities for Information retrieval: Report from SWIRL 2012. SIGIR Forum, 46(1), 2--32. Google ScholarDigital Library
- Anderson, L. W. & Krathwohl, D. A. (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom's taxonomy of educational objectives. New York: Longman.Google Scholar
- Arguello, J. (2014). Predicting search task difficulty. Proc. ECIR, 88--99.Google ScholarDigital Library
- Arguello, J., Wu, W.C., Kelly, D., & Edwards, A. (2012). Task complexity, vertical display and user interaction in aggregated search. Proc. of SIGIR, 435--444. Google ScholarDigital Library
- Aula, A., Khan, R. M. & Guan, Z. (2010). How does search behavior change as search becomes more difficult? Proc. of SIGCHI Conference, 35--44. Google ScholarDigital Library
- Bailey, P., Moffat, A., Scholer, F., & Thomas, P. (2015). User Variability and IR System Evaluation. Proc. of SIGIR. Google ScholarDigital Library
- Bell, D. J. & Ruthven, I. (2004). Searcher's assessments of task complexity for web searching. Proc. of ECIR, 57--71.Google ScholarCross Ref
- Belkin, N. J., Dumais, S. Kando, N. & Sanderson, M. (2012, October). Whole Session Evaluation of Interactive Information Retrieval Systems. National Institute of Informatics Shonan Meeting, Shonan Village Center, Japan.Google Scholar
- Borlund, P. (2003). The IIR evaluation model: A framework for the evaluation of interactive information retrieval systems. Information Research, 8(3), paper 152.Google Scholar
- Brennan, K., Kelly, D., & Arguello, J. (2014). The effect of cognitive abilities on information search for tasks of varying levels of complexity. Proc. of IIiX, 165--174. Google ScholarDigital Library
- Broder, A. (2002). A taxonomy of web search. SIGIR Forum, 36(2), 3--10. Google ScholarDigital Library
- Byström, K. & Hansen, P. (2005). Conceptual framework for tasks in information studies. JASIST, 56(10), 1050--1061. Google ScholarCross Ref
- Byström, K. & Järvelin, K. (1995). Task complexity affects information seeking and use. IP&M, 31, 191--213. Google ScholarDigital Library
- Campbell, D. J. (1988). Task complexity: A review and analysis. Academy of Management Review, 13, 40--52.Google ScholarCross Ref
- Crescenzi, A., Capra, R. & Arguello, J. (2013). Time Pressure, User Satisfaction and Task Difficulty. Proc. of ASIST Conference. Google ScholarDigital Library
- Diriye, A., White, R. W., Buscher, G., & Dumais, S. T. (2012). Leaving so soon? Understanding and predicting web search abandonment rationales. Proc. of CIKM, 1025--1034. Google ScholarDigital Library
- Dumais, S. T., & Belkin, N. J. (2005). The TREC Interactive Tracks: Putting the user into search. In E. M. Voorhees & D. K. Harman (Eds.) TREC: Experiment and Evaluation in Information Retrieval (pp. 123--153), Cambridge, MA: MIT Press.Google Scholar
- Gwizdka, J. & Spence, I. (2006). What can searching behavior tell us about the difficulty of information tasks? A study of Web navigation. Proc. of ASIST, 1--22.Google ScholarCross Ref
- Hassan, A., White, R. W., Dumais, S. T., & Wang, Y. M. (2014). Struggling or exploring?: Disambiguating long search sessions. Proc. of WSDM, 53--62. Google ScholarDigital Library
- Jansen, B. J., Booth, D. & Smith, B. (2009). Using the taxonomy of cognitive learning to model online searching. IP&M, 45, 643--663. Google ScholarDigital Library
- Jiang, J., He, D. & Allan, J. (2014). Searching, browsing and clicking in a search session: Changes in user behavior by task and over time. Proc. of SIGIR, 607--616. Google ScholarDigital Library
- Kelly, D. (2009). Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval, 3(1-2). Google ScholarDigital Library
- Kelly, D., Arguello, J., & Capra, R. (2013). NSF workshop on task-based information search systems. SIGIR Forum, 47(2). Google ScholarDigital Library
- Kelly, D., Dumais, S., & Pedersen, J. (2009). Evaluation challenges and directions for information seeking support systems. IEEE Computer, 42(3), 60--66. Google ScholarDigital Library
- Kim, J. (2006). Task difficulty as a predictor and indicator of web searching interaction. Proc. of CHI (Extended Abstracts), 959--964. Google ScholarDigital Library
- Li, Y. & Belkin, N. J. (2008). A faceted approach to conceptualizing tasks in information seeking. IP&M, 44, 1822--1837. Google ScholarDigital Library
- Li, Y. & Belkin, N. J. (2010). An exploration of the relationship between work task and interactive information search behavior. JASIST, 61(9), 1771--1789. Google ScholarDigital Library
- Liu, J., Liu, C., Cole, M., Belkin, N. J., & Zhang, X. (2012). Exploring and predicting search task difficulty. Proc. of CIKM, 1313--1322. Google ScholarDigital Library
- Liu, J., Kim, C. S. & Creel, C. (2013). Why do users feel search task difficult? Proc. of ASIST. Google ScholarDigital Library
- Liu, Q., Agichtein, E., Dror, G., Maarek, Y. & Szpektor, I. (2012). When web search fails, searchers become askers: Understanding the transition. Proc. of SIGIR, 801--810. Google ScholarDigital Library
- Moffat, A., Thomas, P., & Scholer, F. (2013). Users versus models: What observation tells us about effectiveness metrics. Proc. of CIKM, 659--668. Google ScholarDigital Library
- Robertson, S. (2008). On the history of evaluation in IR. Journal of Documentation, 34(4), 439--456. Google ScholarDigital Library
- Singer, G., Norbisrath, U. & Lewandowski, D. (2012). Ordinary search engine users assessing difficulty, effort, and outcome for simple and complex search tasks. Proc. of IIiX, 110--119. Google ScholarDigital Library
- Smith, C. L. (2008). Searcher adaptation: A response to topic difficulty. Proc. of the ASIST Conference.Google ScholarCross Ref
- Toms, E. (2011). Task-based information searching and retrieval. In Ruthven, I., & Kelly, D. (Eds.) Interactive Information-seeking, Behaviour and Retrieval (pp. 43--59).Google Scholar
- Toms, E., O'Brien, H. L., MacKenzie, T., Jordan, C., Freund, L., Toze, S., Dawe, E., & MacNutt, A. (2007). Task effects on interactive search: The query factor. Proc. of INEX, 359--372.Google Scholar
- Vakkari, P. (2003). Task-based information searching. ARIST, 37, 413--464.Google Scholar
- White, R.W. & Dumais, S. T. (2009). Characterizing and predicting search engine switching behavior. Proc. of CIKM, 87--96. Google ScholarDigital Library
- White, R. W. & Roth, R.A. (2009). Exploratory search: Beyond the query-response paradigm. Morgan & Claypool. Google ScholarDigital Library
- Whittaker, S., Terveen, L., & Nardi, B. (2000). Let's stop pushing the envelope and start addressing it: A reference task agenda for HCI. Human Computer Interaction, 15, 75--106. Google ScholarDigital Library
- Wildemuth, B. W., Freund, L. & Toms, E. G. (2014). Untangling search task complexity and difficulty in the context of interactive information retrieval studies. Journal of Documentation, 70(6), 1118--1140.Google ScholarCross Ref
- Wu, W.C., Kelly, D., Edwards, A., & Arguello, J. (2012). Grannies, tanning beds, tattoos and NASCAR: Evaluation of search tasks with varying levels of cognitive complexity. Proc. of IIiX, 254--257. Google ScholarDigital Library
Index Terms
- Development and Evaluation of Search Tasks for IIR Experiments using a Cognitive Complexity Framework
Recommendations
Grannies, tanning beds, tattoos and NASCAR: evaluation of search tasks with varying levels of cognitive complexity
IIIX '12: Proceedings of the 4th Information Interaction in Context SymposiumOne of the most challenging aspects of designing an interactive information retrieval (IIR) study is the development of search tasks. In this paper, we present preliminary results of a study designed to evaluate a set of search tasks that were developed ...
An examination of multisession web tasks
Today, people perform many types of tasks on the web, including those that require multiple web sessions. In this article, we build on research about web tasks and present an in-depth evaluation of the types of tasks people perform on the web over ...
An exploration of the relationships between work task and interactive information search behavior
This study explores the relationships between work task and interactive information search behavior. Work task was conceptualized based on a faceted classification of task. An experiment was conducted with six work-task types and simulated work-task ...
Comments