Abstract
Empirical evaluations are needed to determine which users are helped or hindered by user-adapted interaction in user modeling systems. A review of past UMUAI articles reveals insufficient empirical evaluations, but an encouraging upward trend. Rules of thumb for experimental design, useful tests for covariates, and common threats to experimental validity are presented. Reporting standards including effect size and power are proposed.
Article PDF
Similar content being viewed by others
References
Albrecht, D. W., Zukerman, I. and Nicholson, A. E.: 1998, Bayesian models for keyhole plan recognition in an adventure game. User Modeling and User-Adapted Interaction 8(1–2), 5-47.
Alspector, J., Kołcz, A. and Karunanithi, N.: 1997, Feature-based and clique-based user models for movie selection: a comparative study. User Modeling and User-Adapted Interaction 7(4), 279-304.
Balabanović, M.: 1998, Exploring versus exploiting when learning user models for text recommendation. User Modeling and User-Adapted Interaction 8(1–2), 71-102.
Boyle, C. and Encarnacion, A. O.: 1994, MetaDoc: An adaptive hypertext reading system. User Modeling and User-Adapted Interaction 4(1), 1-19.
Brusilovsky, P.: 1996, Methods and techniques of adaptive hypermedia. User Modeling and User-Adapted Interaction 6(2–3), 87-129.
Brusilovsky, P.: 1998, Methods and techniques of adaptive hypermedia. In: P. Brusilovsky, A. Kobsa and J. Vassileva (eds.), Adaptive Hypertext and Hypermedia. Kluwer Academic Publishers, Dordrecht, pp. 1-43.
Brusilovsky, P.: 2001, Adaptive hypermedia. User Modeling and User-Adapted Interaction 11(1–2), 87-110 (this issue).
Brusilovsky, P. and Eklund, J.: 1998, A study of user-model based link annotation in educational hypermedia. Journal of Universal Computer Science 4(4), 429-448.
Calistri-Yeh, R. J.: 1991, Utilizing user models to handle ambiguity and misconceptions in robust plan recognition. User Modeling and User-Adapted Interaction 1(4), 289-322.
Campbell, D. T. and Stanley, J. C.: 1963, Experimental and quasi-experimental designs for research. In: N. L. Gage (ed.), Handbook of Research on Teaching. Rand McNally and Co., Chicago.
Carberry, S.: 2001, Techniques for Plan Recognition. User Modeling and User-Adapted Interaction 11(1–2), 31-48 (this issue).
Carbonaro, A., Maniezzo, V., Roccetti, M. and Salomoni, P.: 1995, Modelling the student in Pitagora 2.0. User Modeling and User-Adapted Interaction 4(4), 233-251.
Chiu, B. C. and Webb, G. I.: 1998, Using decision trees for agent modeling: Improving prediction performance. User Modeling and User-Adapted Interaction 8(1–2), 131-152.
Chu-Carroll, J. and Brown, M. K.: 1998, An evidential model for tracking initiative in collaborative dialogue interactions. User Modeling and User-Adapted Interaction 8(3–4), 215-253.
Corbett, A.T., Anderson, J. R. and O'Brien, A. T.: 1993, The predictive validity of student modeling in the ACT programming tutor. In: P. Brna, S. Ohlsson and H. Pain (eds.), Artificial Intelligence and Education, 1993: The Proceedings of AI-ED 93. Charlottesville, VA: AACE.
Corbett, A. T. and Anderson, J. R.: 1995, Knowledge tracing: modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4(4), 253-278.
Debevc, M., Meyer, B., Donlagic, D. and Svecko, R.: 1996, Design and evaluation of an adaptive icon toolbar. User Modeling and User-Adapted Interaction 6(1), 1-21.
Fischer, G.: 2001, User modeling in human-computer interaction. User Modeling and User-Adapted Interaction 11(1–2), 65-86 (this issue).
Gatti, G.G. and Harwell, M.: 1998, Advantages of computer programs over power charts for the estimation of power. Journal of Statistics Education 6(3). Also available at www.amstat.org/publications/jse/v6n3/gatti.html.
Gmytrasiewicz, P. J. and Noh, S. and Kellogg, T.: 1998, Bayesian update of recursive agent models. User Modeling and User-Adapted Interaction 8(1–2), 49-69.
Green, N. and Carberry, S.: 1999, A computational mechanism for initiative in answer generation. User Modeling and User-Adapted Interaction 9(1–2), 93-132.
Guinn, C. I.: 1998, An analysis of initiative selection in collaborative task-oriented discourse. User Modeling and User-Adapted Interaction 8(3–4), 255-314.
Hirashima, T., Kashihara, A. and Toyoda, J.: 1997, Information filtering using user's context on browsing in hypertext. User Modeling and User-Adapted Interaction 7(4), 239-256.
Huck, S., Cormier, W. and Bounds, W.: 1974, Reading Statistics and Research. Harper and Row, New York.
Ishizaki, M., Crocker, M. and Mellish, C.: 1999, Mixed-initiative dialogue using computer dialogue simulation. User Modeling and User-Adapted Interaction 9(1–2), 79-91.
Illeényi, A. and Korpaãssy, P.: 1981, Correlation between loudness and quality of stereophonic loudspeakers. Acustica 49(4), 334-336.
Jennings, A. and Higuchi, H.: 1993, A user model neural network for a personal news service. User Modeling and User-Adapted Interaction 3(1), 1-25.
Kaplan, C., Fenwick, J. and Chen, J.: 1993, Adaptive hypertext navigation based on user goals and context. User Modeling and User-Adapted Interaction 3(3), 193-220.
Kashihara, A., Hirashima, T. and Toyoda, J.: 1995, A cognitive load application in tutoring. User Modeling and User-Adapted Interaction 4(4), 279-303.
Keppel, G.: 1991, Design and Analysis: A Researcher's Handbook 3rd Edn. Prentice-Hall, Englewood Cliffs, NJ.
Krause, J., Hirschmann, A. and Mittermaier, E.: 1993, The intelligent help system COM-FOHELP: Towards a solution of the practicability problem for user modeling and adaptive systems. User Modeling and User-Adapted Interaction 3(3), 249-282.
Lipschitz, S. P. and Van der Kooy, J.: 1981, The great debate: Subjective evaluation. Journal of the Audio Engineering Society 29(7/8), 482-491.
London, R.: 1992, Student modeling to support multiple instructional approaches. User Modeling and User-Adapted Interaction, 2(1–2), 117-154.
Marshall, C. and Rossman, G. B.: 1999, Designing Qualitative Research 3rd Edn. Alta Mira Press, Walnut Creek, CA.
Matheé, N. and Chen, J. R.: 1996, User-centered indexing for adaptive information access. User Modeling and User-Adapted Interaction 6(2–3), 225-261.
Miles, M. B. and Huberman, A. H.: 1994, Qualitative Data Analysis: An Expanded Sourcebook 2nd Edn. Sage Publications, London.
Milne, S., Shiu, E. and Cook, J.: 1996, Development of a model of user attributes and its implementation within an adaptive tutoring system. User Modeling and User-Adapted Interaction 6(4), 303-335.
Mitrovic, A., Djordjevic-Kajan, S. and Stoimenov, L.: 1996, INSTRUCT: Modeling students by asking questions. User Modeling and User-Adapted Interaction 6(4), 273-302.
Neter, J., Wasserman, W., and Kutner, M. H.: 1985, Applied Linear Statistical Models 2nd Edn. Richard D. Irvin, Homewood, IL.
Newell, S.: 1997, User models and ¢ltering agents for improved internet information retrieval. User Modeling and User-Adapted Interaction 7(4), 223-237.
Nwana, H. S.: 1991, User modelling and user adapted interaction in an intelligent tutoring system. User Modeling and User-Adapted Interaction 1(1), 1-32.
Raskutti, B., Beitz, A. and Ward, B.: 1997, A feature-based approach to recommending selections based on past preferences. User Modeling and User-Adapted Interaction 7(3), 179-218.
Shanefield, D.: 1980, The great ego crunchers: Equalized, double-blind tests. High Fidelity, March, 57-61.
Shute, V. J.: 1995, SMART: Student modeling approach for responsive tutoring. User Modeling and User-Adapted Interaction 5(1), 1-44.
Silverman, D.: 1993, Interpreting qualitative data: Methods for analysing talk, text and interaction. Sage Publications, Thousand Oaks, CA.
Sison, R. C., Numao, M. and Shimura, M.: 1998, Discovering error classes from discrepancies in novice behaviors via multistrategy conceptual clustering. User Modeling and User-Adapted Interaction 8(1–2), 103-129.
Specht, M. and Kobsa, A.: 1999, Interaction of domain expertise and interface design in adaptive educational hypermedia. In Workshops on Adaptive Systems and User Modeling on the World Wide Web at WWW-8, Toronto, Canada and UM99, Banff, Canada.
Stevens, J.: 1992, Applied Multivariate Statistics for the Social Sciences 2nd Edn. Lawrence Erlbaum, Hillsdale, NJ.
Tattersall, C.: 1992, Generating help for users of application software. User Modeling and User-Adapted Interaction, 2(3), 211-248.
Vassileva, J.: 1996, A task-centered approach for user modeling in a hypermedia office documentation system. User Modeling and User-Adapted Interaction 6(2–3), 185-224.
Virvou, M. and du Bulay, B.: 1999, Human plausible reasoning for intelligent help. User Modeling and User-Adapted Interaction 9(4), 323-377.
Webb, G. I. and Kuzmycz, M.: 1996, Feature based modelling: a methodology for producing coherent, consistent, dynamically changing models of agent's competencies. User Modeling and User-Adapted Interaction 5(2), 117-150.
Webb, G. I., Pazzani, M. J. and Billsus, D.: 2000, Machine learning for user modeling. User Modeling and User-Adapted Interaction 11(1–2), 19-29 (this issue).
Weber, R. P.: 1990, Basic Content Analysis 2nd Edn. Sage Publications, Beverly Hills.
Yin, R. K.: 1994, Case Study Research: Design and Methods 2nd Edn. Sage Publications, London.
Zukerman, I. and Albrecht, D. W.: 2000, Predictive statistical user models for user modeling. User Modeling and User-Adapted Interaction 11(1–2), 5-18 (this issue).
Zukerman, I. and Litman, D.: 2000, Natural language processing and user modeling: Synergies and limitations. UserModeling and User-Adapted Interaction 11(1–2), 129-158 (this issue).
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Chin, D.N. Empirical Evaluation of User Models and User-Adapted Systems. User Modeling and User-Adapted Interaction 11, 181–194 (2001). https://doi.org/10.1023/A:1011127315884
Issue Date:
DOI: https://doi.org/10.1023/A:1011127315884