Maria Gäde
ABSTRACT
This perspective paper on resource re-use intends to draw the attention of the interactive information retrieval (IIR) community to the challenges of research documentation and archiving for future use. Resources are understood as encompassing research designs, research data and research infrastructures. It proposes eight principles for improving the re-use of resources in the IIR community and presents concrete steps on how to achieve them. A five-level system for data archiving and documentation envisions increasingly open and stable documentation and access infrastructures.
- ACM. 2020. Artifact Review and Badging. https://www.acm.org/publications/policies/artifact-review-badging.Google Scholar
- Alice Allen, Cecilia Aragon, Christoph Becker, Jeffrey Carver, Andrei Chis, Benoit Combemale, Mike Croucher, Kevin Crowston, Daniel Garijo, Ashish Gehani, Carole Goble, Robert Haines, Robert Hirschfeld, James Howison, Kathryn Huff, Caroline Jay, Daniel S. Katz, Claude Kirchner, Katie Kuksenok, Ralf Lämmel, Oscar Nierstrasz, Matt Turk, Rob van Nieuwpoort, Matthew Vaughn, and Jurgen J. Vinju. 2017. Engineering Academic Software (Dagstuhl Perspectives Workshop 16252). Dagstuhl Manifestos 6, 1 (2017), 1--20. https://doi.org/10.4230/DagMan.6.1.1Google Scholar
- J. Andreoni, W. Harbaugh, and L. Vesterlund. 2003. The Carrot or the Stick: Rewards, Punishments, and Cooperation. American Economic Review 93, 3 (2003), 893--902.Google ScholarCross Ref
- Toine Bogers, Samuel Dodson, Luanne Freund, Maria Gäde, Mark Hall, Marijn Koolen, Vivien Petras, Nils Pharo, and Mette Skov. 2019. Overview of the CHIIR 2019 Workshop on Barriers to InteractiveIR Resources Re-use (BIIRRR 2019). In CEUR Workshop Proceedings, Vol. 2337. CEUR-WS.org, Aachen, Germany, 1--6. http://ceur-ws.org/Vol-2337/overview.pdfGoogle Scholar
- Toine Bogers, Maria Gäde, Mark M. Hall, Vivien Petras, and Mette Skov. 2017. Lessons Learned from the CHiC and SBS Interactive Tracks: A Wishlist for Interactive IR Evaluation. In Proceedings of the Second Workshop on Supporting Complex Search Tasks co-located with the ACM SIGIR Conference on Human Information Interaction & Retrieval (CHIIR 2017), Oslo, Norway, March 11, 2017 (CEUR Workshop Proceedings, Vol. 1798), Marijn Koolen, Jaap Kamps, Toine Bogers, Nicholas J. Belkin, Diane Kelly, and Emine Yilmaz (Eds.). CEUR-WS.org, Aachen, Germany, 11--14. http://ceur-ws.org/Vol-1798/paper2.pdfGoogle Scholar
- Toine Bogers, Maria Gäde, Mark Michael, Luanne Freund, Marijn Koolen, Vivien Petras, and Mette Skov. 2018. Report on the workshop on barriers to interactive IR resources re-use (BIIRRR 2018). ACM SIGIR Forum 52, 1 (2018), 119--128.Google ScholarDigital Library
- Christine L Borgman. 2012. The Conundrum of Sharing Research Data. Journal of the American Society for Information Science and Technology 63, 6 (2012), 1059-- 1078.Google ScholarDigital Library
- Christine L Borgman. 2015. Big data, little data, no data: Scholarship in the networked world. MIT press, Cambridge, MA.Google ScholarDigital Library
- Pia Borlund. 2003. The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information research 8, 3 (2003), 8--3.Google Scholar
- Julianne Cheek. 2008. Research design. In The SAGE encyclopedia of qualitative research methods. SAGE Publications, Inc., Thousand Oaks, CA, 762--764.Google Scholar
- Henry Chesbrough and Jim Spohrer. 2006. A Research Manifesto for Services Science. Commun. ACM 49, 7 (July 2006), 35--40.Google ScholarDigital Library
- Jon F. Claerbout and Martin Karrenbach. 1992. Electronic Documents give Reproducible Research a New Meaning. In SEG Technical Program Expanded Abstracts 1992. Society of Exploration Geophysicists, Tulsa, OK, 601--604.Google Scholar
- Rosaria Conte, Nigel Gilbert, Giulia Bonelli, Claudio Cioffi-Revilla, Guillaume Deffuant, Janos Kertesz, Vittorio Loreto, Suzy Moat, J-P Nadal, Anxo Sanchez, et al. 2012. Manifesto of computational social science. The European Physical Journal Special Topics 214, 1 (2012), 325--346.Google ScholarCross Ref
- Joenio Costa, Paulo Meirelles, and Christina Chavez. 2018. On the Sustainability of Academic Software: The Case of Static Analysis Tools. In Proceedings of the XXXII Brazilian Symposium on Software Engineering (Sao Carlos, Brazil) (SBES '18). Association for Computing Machinery, New York, NY, USA, 202--207. https: //doi.org/10.1145/3266237.3266243Google ScholarDigital Library
- John W. Creswell and J. David Creswell. 2017. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Sage Publications, Thousand Oaks, CA.Google Scholar
- Giorgio De Michelis, Eric Dubois, Matthias Jarke, Florian Matthes, John Mylopoulos, Mike Papazoglou, Klaus Pohl, Joachim Schmidt, Carson Woo, and Eric Yu. 1997. Cooperative information systems: a manifesto. Accademic-Press, Cambridge, MA.Google Scholar
- Taylor & Francis. 2020. Open and FAIR data sharing policy. https://authorservices.taylorandfrancis.com/data-sharing-policies/openand-fair/.Google Scholar
- Luanne Freund and Barbara M. Wildemuth. 2014. Documenting and Studying the Use of Assigned Search Tasks: RepAST. Proceedings of the American Society for Information Science and Technology 51, 1 (2014), 1--4.Google ScholarCross Ref
- Maria Gäde, Mark Michael Hall, Hugo C. Huurdeman, Jaap Kamps, Marijn Koolen, Mette Skov, Toine Bogers, and David Walsh. 2016. Overview of the SBS 2016 Interactive Track. In Working Notes of the CLEF 2016 Conference (CEUR Workshop Proceedings, Vol. 1609), Krisztian Balog, Linda Cappellato, Nicola Ferro, and Craig Macdonald (Eds.). CEUR-WS.org, Aachen, Germany, 1024--1038.Google Scholar
- Maria Gäde, Mark Michael Hall, Hugo C. Huurdeman, Jaap Kamps, Marijn Koolen, Mette Skov, Elaine Toms, and David Walsh. 2015. Overview of the SBS 2015 Interactive Track. In Working Notes of the CLEF 2015 Conference (CEUR Workshop Proceedings, Vol. 1391), Linda Cappellato, Nicola Ferro, Gareth J. F. Jones, and Eric SanJuan (Eds.). CEUR-WS.org, Aachen, Germany, 14.Google Scholar
- Teresa Gomez-Diaz and Tomas Recio. 2019. On the evaluation of research software: the CDUR procedure. F1000Research 8 (2019). https://doi.org/10.12688/ f1000research.19994.2Google Scholar
- Mark M Hall. 2019. To re-use is to re-write: experiences with re-using IIR experiment software. In CEUR Workshop Proceedings, Vol. 2337. CEUR-WS.org, Aachen, Germany, 19--23.Google Scholar
- Mark Michael Hall and Toine Bogers. 2020. A Standardised Format for Exchanging User Study Instruments. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval. ACM, New York City, NY, 457--461.Google ScholarDigital Library
- Mark Michael Hall, Hugo C. Huurdeman, Marijn Koolen, Mette Skov, and David Walsh. 2014. Overview of the INEX 2014 Interactive Social Book Search Track. In Working Notes of the CLEF 2014 Conference (CEUR Workshop Proceedings, Vol. 1180), Linda Cappellato, Nicola Ferro, Martin Halvey, and Wessel Kraaij (Eds.). CEUR-WS.org, Aachen, Germany, 480--493.Google Scholar
- Mark M Hall, Spyros Katsaris, and Elaine Toms. 2013. A Pluggable Interactive IR Evaluation Work-bench. In European Workshop on Human-Computer Interaction and Information Retrieval. Springer, Heidelberg, Germany, 35--38.Google Scholar
- Diana Hicks, Paul Wouters, Ludo Waltman, Sarah De Rijcke, and Ismael Rafols. 2015. Bibliometrics: the Leiden Manifesto for research metrics. Nature 520, 7548 (2015), 429--431.Google Scholar
- Stephen Hilgartner and Sherry I. Brandt-Rauf. 1994. Data Access, Ownership, and Control: Toward Empirical Studies of Access Practices. Knowledge 15, 4 (1994), 355--372. https://doi.org/10.1177/107554709401500401 arXiv:https://doi.org/10.1177/107554709401500401Google Scholar
- Timothy Holborn. 2014. What is 5 Star Linked Data? https://www.w3.org/community/webize/2014/01/17/what-is-5-star-linkeddata/.Google Scholar
- Hugo C. Huurdeman, Jaap Kamps, and Max L. Wilson. 2019. The Multi-Stage Experience: the Simulated Work Task Approach to Studying Information Seeking Stages. In Proceedings of the CHIIR 2019 Workshop on Barriers to Interactive IR Resources Re-use co-located with the ACM SIGIR Conference on Human Information Interaction and Retrieval, BIIRRR@CHIIR 2019, Glasgow, UK, March 14, 2019 (CEUR Workshop Proceedings, Vol. 2337), Toine Bogers, Samuel Dodson, Maria Gäde, Luanne Freund, Mark M. Hall, Marijn Koolen, Vivien Petras, Nils Pharo, and Mette Skov (Eds.). CEUR-WS.org, Aachen, Germany, 7--13. http://ceur-ws.org/Vol2337/paper1.pdfGoogle Scholar
- Diane Kelly. 2009. Methods for Evaluating Interactive Information Retrieval Systems with Users. Foundations and Trends in Information Retrieval 3, 1--2 (2009), 1--224.Google ScholarDigital Library
- Florian G Kern and Kristian Skrede Gleditsch. 2017. Exploring pre-registration and pre-analysis plans for qualitative inference. Preprint ahead of publication (2017), 1--15.Google Scholar
- Youngseek Kim and Melissa Adler. 2015. Social scientists? data sharing behaviors: Investigating the roles of individual motivations, institutional pressures, and data repositories. International journal of information management 35, 4 (2015), 408--418.Google ScholarDigital Library
- Effie Lai-Chong Law, Arnold POS Vermeeren, Marc Hassenzahl, and Mark Blythe. 2007. Towards a UX manifesto. In Proceedings of HCI 2007 The 21st British HCI Group Annual Conference University of Lancaster, UK 21. BCS, The Chartered Institute for IT, Swindon, UK, 1--2.Google Scholar
- Sara Mannheimer, Amy Pienta, Dessislava Kirilova, Colin Elman, and Amber Wutich. 2019. Qualitative data sharing: Data repositories and academic libraries as key partners in addressing challenges. American Behavioral Scientist 63, 5 (2019), 643--664.Google ScholarCross Ref
- Edward Miguel, Colin Camerer, Katherine Casey, Joshua Cohen, Kevin M Esterling, Alan Gerber, Rachel Glennerster, Don P Green, Macartan Humphreys, Guido Imbens, et al. 2014. Promoting transparency in social science research. Science 343, 6166 (2014), 30--31.Google Scholar
- Geoffrey Miller. 2012. The Smartphone Psychology Manifesto. Perspectives on Psychological Science 7, 3 (2012), 221--237. https://doi.org/10.1177/1745691612441215 arXiv:https://doi.org/10.1177/1745691612441215 PMID: 26168460.Google ScholarCross Ref
- Marcus R. Munafò, Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button, Christopher D. Chambers, Nathalie Percie Du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J. Ware, and John P. A. Ioannidis. 2017. A Manifesto for Reproducible Science. Nature Human Behaviour 1, 1 (2017), 1--9.Google ScholarCross Ref
- Ragnar Nordlie and Nils Pharo. 2012. Seven years of INEX interactive retrieval experiments--lessons and challenges. In International Conference of the CrossLanguage Evaluation Forum for European Languages. Springer, Heidelberg, Germany, 13--23.Google ScholarDigital Library
- Paul Over. 2001. The TREC Interactive Track: An Annotated Bibliography. Information Processing & Management 37, 3 (2001), 369--381.Google ScholarDigital Library
- Heinz Pampel, Paul Vierkant, Frank Scholze, Roland Bertelmann, Maxi Kindling, Jens Klump, Hans-Jürgen Goebelbecker, Jens Gundlach, Peter Schirmbacher, and Uwe Dierolf. 2013. Making research data repositories visible: the re3data. org registry. PloS one 8, 11 (2013), e78080.Google ScholarCross Ref
- Irene V. Pasquetto, Christine L. Borgman, and Morgan F. Wofford. 2019. Uses and Reuses of Scientific Data: The Data Creators? Advantage. Harvard Data Science Review 1, 2 (15 11 2019). https://doi.org/10.1162/99608f92.fc14bf2d https://hdsr.mitpress.mit.edu/pub/jduhd7og.Google ScholarCross Ref
- Irene V Pasquetto, Bernadette M Randles, and Christine L Borgman. 2017. On the reuse of scientific data. Data Science Journal 16 (2017), 8. https://doi.org/10. 5334/dsj-2017-008Google ScholarCross Ref
- Ray Pawson. 2013. The science of evaluation: a realist manifesto. SAGE Publications, Los Angeles, CA.Google Scholar
- Vivien Petras, Toine Bogers, and Maria Gaede. 2019. Elements of IIR Studies: A Review of the 2006--2018 IIiX and CHIIR Conferences. In Proceedings of the CHIIR 2019 Workshop on Barriers to Interactive IR Resources Re-use (BIIRRR 2019). CEUR-WS.org, Aachen, Germany, 37--41.Google Scholar
- Vivien Petras, Toine Bogers, Elaine G. Toms, Mark M. Hall, Jacques Savoy, Piotr Malak, Adam Pawlowski, Nicola Ferro, and Ivano Masiero. 2013. Cultural Heritage in CLEF (CHiC 2013). In CLEF '13: Proceedings of the 4th International Conference of the CLEF Initiative (Lecture Notes in Computer Science, Vol. 8138), Pamela Forner, Henning Müller, Roberto Paredes, Paolo Rosso, and Benno Stein (Eds.). Springer, Heidelberg, Germany, 192--211.Google ScholarDigital Library
- Vivien Petras, Marijn Koolen, Maria Gaede, and Toine Bogers. 2019. Experiences with the 2013--2016 CLEF Interactive Information Retrieval Tracks. In Proceedings of the CHIIR 2019 Workshop on Barriers to Interactive IR Resources Re-use (BIIRRR 2019). CEUR-WS.org, Aachen, Germany, 29--36.Google Scholar
- Nils Pharo, Thomas Beckers, Ragnar Nordlie, and Norbert Fuhr. 2011. Overview of the INEX 2010 Interactive Track. In INEX '10: Proceedings of the Ninth International Workshop of the Initiative for the Evaluation of XML Retrieval, Shlomo Geva, Jaap Kamps, Ralf Schenkel, and Andrew Trotman (Eds.). Springer, Berlin, Heidelberg, 227--235.Google Scholar
- Rosalind W Picard, Seymour Papert, Walter Bender, Bruce Blumberg, Cynthia Breazeal, David Cavallo, Tod Machover, Mitchel Resnick, Deb Roy, and Carol Strohecker. 2004. Affective learning?a manifesto. BT technology journal 22, 4 (2004), 253--269. https://doi.org/10.1023/B:BTTJ.0000047603.37042.33Google Scholar
- Graham Pryor. 2012. Managing Research Data. Facet Publishing, London, UK.Google Scholar
- Karsten Boye Rasmussen and Grant Blank. 2007. The data documentation initiative: a preservation standard for research. Archival Science 7, 1 (2007), 55--71.Google ScholarCross Ref
- Gareth Renaud and Leif Azzopardi. 2012. SCAMP: A Tool for Conducting Interactive Information Retrieval Experiments. In IIiX '12: Proceedings of the 4th Information Interaction in Context Symposium. ACM, New York City, NY, 286--289.Google Scholar
- Christof Schöch. 2017. Wiederholende Forschung in den digitalen Geisteswissenschaften. In DHd2017: Digital Nachhaltigkeit (DHd2017), Bern, Switzerland, 13--18 February 2017. Zenodo, Genève, Swizterland, 207--212. https://doi.org/10. 5281/zenodo.277113Google Scholar
- Nicholas Smale, Kathryn Unsworth, Gareth Denyer, and Daniel Barr. 2018. The History, Advocacy and Efficacy of Data Management Plans. bioRxiv (2018), 30. https://doi.org/10.1101/443499 arXiv:https://www.biorxiv.org/content/early/2018/10/17/443499.full.pdfGoogle Scholar
- Diana Soltani, Matthew Mitsui, and Chirag Shah. 2019. Coagmento v3. 0: Rapid prototyping of web search experiments. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval. ACM, New York City, NY, 367--371.Google ScholarDigital Library
- Susan E Swogger. 2013. PsycTESTS. Journal of the Medical Library Association: JMLA 101, 3 (2013), 234.Google ScholarCross Ref
- Anastasios Tombros, Birger Larsen, and Saadia Malik. 2005. The Interactive Track at INEX 2004. In INEX '04: Proceedings of the Third International Workshop of the Initiative for the Evaluation of XML Retrieval, Norbert Fuhr, Mounia Lalmas, Saadia Malik, and Zoltán Szlávik (Eds.). Springer, Berlin, Heidelberg, 410--423.Google ScholarDigital Library
- Elaine G Toms, Luanne Freund, and Cara Li. 2004. WiIRE: the Web interactive information retrieval experimentation system prototype. Information Processing & Management 40, 4 (2004), 655--675.Google ScholarDigital Library
- Elaine G. Toms and Mark M. Hall. 2013. The CHiC Interactive Task (CHiCi) at CLEF 2013. In Working Notes of the CLEF 2013 Conference (CEUR Workshop Proceedings, Vol. 1179), Pamela Forner, Roberto Navigli, Dan Tufis, and Nicola Ferro (Eds.). CEUR-WS.org, Aachen, Germany, 12.Google Scholar
- U.S. National Science Board. 2005. Long-lived digital data collections: Enabling research and education in the 21st century (No. US NSF-NSB-05--40). https://www.nsf.gov/pubs/2005/nsb0540/.Google Scholar
- Stephanie van de Sandt, Artemis Lavasa, Sünje Dallmeier-Tiessen, and Vivien Petras. 2019. The Definition of Reuse. Data Science Journal 18 (2019), 22.Google ScholarCross Ref
- Anna Elisabeth van?t Veer and Roger Giner-Sorolla. 2016. Pre-registration in social psychology?A discussion and suggested template. Journal of Experimental Social Psychology 67 (2016), 2--12.Google ScholarCross Ref
- Katrin Weller and Katharina E. Kinder-Kurlanda. 2016. A Manifesto for Data Sharing in Social Media Research. In Proceedings of the 8th ACM Conference on Web Science (Hannover, Germany) (WebSci '16). Association for Computing Machinery, New York, NY, USA, 166--172. https://doi.org/10.1145/2908131.2908172Google Scholar
- Mark Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gaby Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Olavo Bonino da Silva Santos, Philip Bourne, Jildau Bouwman, Anthony Brookes, Tim Clark, Merce Crosas, Ingrid Dillo, Olivier Dumon, Scott Edmunds, Chris Evelo, Richard Finkers, and Barend Mons. 2016. The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data 3 (03 2016). https://doi.org/10.1038/sdata.2016.18Google Scholar
- Paul Willis and Mats Trondman. 2000. Manifesto for Ethnography. Ethnography 1, 1 (2000), 5--16. https://doi.org/10.1177/14661380022230679Google ScholarCross Ref
Index Terms
- A Manifesto on Resource Re-Use in Interactive Information Retrieval
Recommendations
Third Workshop on Building towards Information Interaction and Retrieval Resources Re-use (BIIRRR 2022)
CHIIR '22: Proceedings of the 2022 Conference on Human Information Interaction and RetrievalWorkshop on Barriers to Interactive IR Resources Re-use (BIIRRR 2019)
CHIIR '19: Proceedings of the 2019 Conference on Human Information Interaction and RetrievalHow we Work, Share, and Re-use at CHIIR
CHIIR '23: Proceedings of the 2023 Conference on Human Information Interaction and RetrievalIn this paper, we present the results of an initial study of the research, sharing, and re-use practices at the CHIIR conference through a systematic analysis of all CHIIR papers published from 2016 to 2022. We find that CHIIR is a conference ...
Comments