Skip to main content

2017 | OriginalPaper | Buchkapitel

Challenges in Research Generalizability: The Need for Standardization of Performance Metrics and Methodology

verfasst von : Kathryn A. Feltman, Kyle A. Bernhardt, Amanda M. Kelley

Erschienen in: HCI in Business, Government and Organizations. Interacting with Information Systems

Verlag: Springer International Publishing

Aktivieren Sie unsere intelligente Suche, um passende Fachinhalte oder Patente zu finden.

search-config
loading …

Abstract

The proliferation of psychological and psychophysiological metrics, data collection techniques, and data analysis strategies used throughout psychological research of operator performance presents cross-study synthesis complications. Currently, the lack of defined and established standardizations in psychological and psychophysiological research continues to present challenges to researchers studying an array of interrelated constructs. Without standardizations, differences in measurement implementation, data reduction techniques, and the interpretation of results make it difficult to directly compare studies and reach unequivocal conclusions while synthesizing literature and transfer laboratory findings to field-ready applications.

Sie haben noch keine Lizenz? Dann Informieren Sie sich jetzt über unsere Produkte:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

  • über 102.000 Bücher
  • über 537 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Maschinenbau + Werkstoffe
  • Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

Springer Professional "Technik"

Online-Abonnement

Mit Springer Professional "Technik" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 390 Zeitschriften

aus folgenden Fachgebieten:

  • Automobil + Motoren
  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Elektrotechnik + Elektronik
  • Energie + Nachhaltigkeit
  • Maschinenbau + Werkstoffe




 

Jetzt Wissensvorsprung sichern!

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

  • über 67.000 Bücher
  • über 340 Zeitschriften

aus folgenden Fachgebieten:

  • Bauwesen + Immobilien
  • Business IT + Informatik
  • Finance + Banking
  • Management + Führung
  • Marketing + Vertrieb
  • Versicherung + Risiko




Jetzt Wissensvorsprung sichern!

Literatur
2.
Zurück zum Zitat Button, K.S., Ioannidis, J.P.A., Mokrysz, C., Nosek, B.A., Flint, J., Robinson, E.S.J., Munafò, M.R.: Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013)CrossRef Button, K.S., Ioannidis, J.P.A., Mokrysz, C., Nosek, B.A., Flint, J., Robinson, E.S.J., Munafò, M.R.: Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013)CrossRef
3.
Zurück zum Zitat Larson, M.J.: Commitment to cutting-edge research with rigor and replication in psychophysiological science. Int. J. Psychophysiol. 102, ix–x (2016) Larson, M.J.: Commitment to cutting-edge research with rigor and replication in psychophysiological science. Int. J. Psychophysiol. 102, ix–x (2016)
4.
Zurück zum Zitat Pendlebury, S.T., Mariz, J., Bull, L., Mehta, Z., Rothwell, P.M.: Impact of different operational definitions on mild cognitive impairment rate and MMSE and MoCA performance in transient ischaemic attack and stroke. Cerebrovasc. Dis. 36(5–6), 355–362 (2013)CrossRef Pendlebury, S.T., Mariz, J., Bull, L., Mehta, Z., Rothwell, P.M.: Impact of different operational definitions on mild cognitive impairment rate and MMSE and MoCA performance in transient ischaemic attack and stroke. Cerebrovasc. Dis. 36(5–6), 355–362 (2013)CrossRef
5.
Zurück zum Zitat Kristman, V.L., Borg, J., Godbolt, A.K., Salmi, L.R., Cancelliere, C., Carroll, L.J., Holm, L.W., Nygren-de Boussard, C., Hartvigsen, J., Abara, U., Donovan, J., Cassidy, J.D. Methodological issues and research recommendations for prognosis after mild traumatic brain injury: results of the international collaboration on mild traumatic brain injury prognosis. Arch. Phys. Med. Rehab. 95(3), S265–S277 (2014) Kristman, V.L., Borg, J., Godbolt, A.K., Salmi, L.R., Cancelliere, C., Carroll, L.J., Holm, L.W., Nygren-de Boussard, C., Hartvigsen, J., Abara, U., Donovan, J., Cassidy, J.D. Methodological issues and research recommendations for prognosis after mild traumatic brain injury: results of the international collaboration on mild traumatic brain injury prognosis. Arch. Phys. Med. Rehab. 95(3), S265–S277 (2014)
6.
Zurück zum Zitat Berntson, G.G., Bigger, J.T., Eckberg, D.L., Grossman, P., Kaufmann, P.G., Malik, M., Nagaraja, H.N., Porges, S.W., Saul, J.P., Stone, P.H., van der Molen, M.W.: Heart rate variability: origins, methods, and interpretive caveats. Psychophysiology 34, 623–648 (1997)CrossRef Berntson, G.G., Bigger, J.T., Eckberg, D.L., Grossman, P., Kaufmann, P.G., Malik, M., Nagaraja, H.N., Porges, S.W., Saul, J.P., Stone, P.H., van der Molen, M.W.: Heart rate variability: origins, methods, and interpretive caveats. Psychophysiology 34, 623–648 (1997)CrossRef
7.
Zurück zum Zitat Pivik, R.T., Broughton, R.J., Coppola, R., Davidson, R.J., Fox, N., Nuwer, M.R.: Guidelines for the recording and quantitative analysis of electroencephalographic activity in research contexts. Psychophysiology 30, 547–558 (1993)CrossRef Pivik, R.T., Broughton, R.J., Coppola, R., Davidson, R.J., Fox, N., Nuwer, M.R.: Guidelines for the recording and quantitative analysis of electroencephalographic activity in research contexts. Psychophysiology 30, 547–558 (1993)CrossRef
8.
Zurück zum Zitat Ritz, T., Dahme, B., Dubois, A.B., Folgering, H., Fritz, G.K., Harver, A., Kotses, H., Lehrer, P.M., Ring, C., Steptoe, A., Van de Woestijne, K.P.: Guidelines for mechanical lung function measures in psychophysiology. Psychophysiology 39, 546–567 (2002) Ritz, T., Dahme, B., Dubois, A.B., Folgering, H., Fritz, G.K., Harver, A., Kotses, H., Lehrer, P.M., Ring, C., Steptoe, A., Van de Woestijne, K.P.: Guidelines for mechanical lung function measures in psychophysiology. Psychophysiology 39, 546–567 (2002)
9.
Zurück zum Zitat Ryu, K., Myung, R.: Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. Int. J. Ind. Ergon. 35, 991–1009 (2005)CrossRef Ryu, K., Myung, R.: Evaluation of mental workload with a combined measure based on physiological indices during a dual task of tracking and mental arithmetic. Int. J. Ind. Ergon. 35, 991–1009 (2005)CrossRef
10.
Zurück zum Zitat Bonner, M.A., Wilson, G.F.: Heart rate measures of flight test and evaluation. Int. J. Aviat. Psychol. 12(1), 63–77 (2002)CrossRef Bonner, M.A., Wilson, G.F.: Heart rate measures of flight test and evaluation. Int. J. Aviat. Psychol. 12(1), 63–77 (2002)CrossRef
11.
Zurück zum Zitat Durantin, G., Gagnon, J.F., Tremblay, S., Dehais, F.: Using near infrared spectroscopy and heart rate variability to detect mental overload. Behav. Brain Res. 259, 16–23 (2014)CrossRef Durantin, G., Gagnon, J.F., Tremblay, S., Dehais, F.: Using near infrared spectroscopy and heart rate variability to detect mental overload. Behav. Brain Res. 259, 16–23 (2014)CrossRef
12.
Zurück zum Zitat Hsu, B.W., Wang, M.J.J., Chen, C.Y.: Effective indices for monitoring mental workload while performing multiple tasks. Percept. Mot. Skills 121(1), 94–117 (2015)CrossRef Hsu, B.W., Wang, M.J.J., Chen, C.Y.: Effective indices for monitoring mental workload while performing multiple tasks. Percept. Mot. Skills 121(1), 94–117 (2015)CrossRef
13.
Zurück zum Zitat Berntson, G.G., Quigley, K.S., Lozano, D.: Cardiovascular psychophysiology. In: Cacioppo, J.T., Tassinary, L.G., Berntson, G.G. (eds.) Handbook of Psychophysiology, pp. 182–210. Cambridge University Press, New York (2007)CrossRef Berntson, G.G., Quigley, K.S., Lozano, D.: Cardiovascular psychophysiology. In: Cacioppo, J.T., Tassinary, L.G., Berntson, G.G. (eds.) Handbook of Psychophysiology, pp. 182–210. Cambridge University Press, New York (2007)CrossRef
14.
Zurück zum Zitat Kligfield, P., Gettes, L.S., Bailey, J.J., Childers, R., Deal, B.J., Hancock, W., van Herpen, G., Kors, J., Macfarlane, P., Mirvis, D., Pahlm, O., Rautaharju, P., Wagner, G.S.: Recommendations for the standardization and interpretation of the electrocardiogram. Circulation 115, 1306–1324 (2007) Kligfield, P., Gettes, L.S., Bailey, J.J., Childers, R., Deal, B.J., Hancock, W., van Herpen, G., Kors, J., Macfarlane, P., Mirvis, D., Pahlm, O., Rautaharju, P., Wagner, G.S.: Recommendations for the standardization and interpretation of the electrocardiogram. Circulation 115, 1306–1324 (2007)
15.
Zurück zum Zitat Strube, M.J., Newman, L.C.: Psychometrics. In: Cacioppo, J.T., Tassinary, L.G., Berntson, G.G. (eds.) Handbook of Psychophysiology, pp. 789–811. Cambridge University Press, New York (2007) Strube, M.J., Newman, L.C.: Psychometrics. In: Cacioppo, J.T., Tassinary, L.G., Berntson, G.G. (eds.) Handbook of Psychophysiology, pp. 789–811. Cambridge University Press, New York (2007)
16.
Zurück zum Zitat Weippert, M., Kumar, M., Kreuzfeld, S., Arndt, D., Rieger, A., Stoll, R.: Comparison of three mobile devices for measuring R-R intervals and heart rate variability: Polar S810i, Suunto t6 and an ambulatory ECG system. Eur. J. Appl. Physiol. 109(4), 779–786 (2010)CrossRef Weippert, M., Kumar, M., Kreuzfeld, S., Arndt, D., Rieger, A., Stoll, R.: Comparison of three mobile devices for measuring R-R intervals and heart rate variability: Polar S810i, Suunto t6 and an ambulatory ECG system. Eur. J. Appl. Physiol. 109(4), 779–786 (2010)CrossRef
17.
Zurück zum Zitat Di Stasi, L.L., Diaz-Piedra, C., Suárez, J., McCamy, M.B., Martinez-Conde, S., Roca-Dorda, J., Catena, A.: Task complexity modulates pilot electroencephalographic activity during real flights. Psychophysiology 52(7), 951–956 (2015)CrossRef Di Stasi, L.L., Diaz-Piedra, C., Suárez, J., McCamy, M.B., Martinez-Conde, S., Roca-Dorda, J., Catena, A.: Task complexity modulates pilot electroencephalographic activity during real flights. Psychophysiology 52(7), 951–956 (2015)CrossRef
18.
Zurück zum Zitat Fairclough, S.H., Venables, L., Tattersall, A.: The influence of task demand and learning on the psychophysiological response. Int. J. Psychophysiol. 56, 171–184 (2005)CrossRef Fairclough, S.H., Venables, L., Tattersall, A.: The influence of task demand and learning on the psychophysiological response. Int. J. Psychophysiol. 56, 171–184 (2005)CrossRef
19.
Zurück zum Zitat Cohen, M.X.: Rigor and replication in time-frequency analyses of cognitive electrophysiology data. Int. J. Psychophysiol. 111, 80–87 (2017)CrossRef Cohen, M.X.: Rigor and replication in time-frequency analyses of cognitive electrophysiology data. Int. J. Psychophysiol. 111, 80–87 (2017)CrossRef
20.
Zurück zum Zitat Cacioppo, J.T., Tassinary, L.G.: Inferring psychological significance from physiological signals. Am. Psychol. 45(1), 16–28 (1990)CrossRef Cacioppo, J.T., Tassinary, L.G.: Inferring psychological significance from physiological signals. Am. Psychol. 45(1), 16–28 (1990)CrossRef
21.
Zurück zum Zitat Pizzagalli, D.A.: Electroencephalography and high-density electrophysiological source localization. In: Cacioppo, J.T., Tassinary, L.G., Berntson, G.G. (eds.) Handbook of Psychophysiology, pp. 56–84. Cambridge University Press, New York (2007)CrossRef Pizzagalli, D.A.: Electroencephalography and high-density electrophysiological source localization. In: Cacioppo, J.T., Tassinary, L.G., Berntson, G.G. (eds.) Handbook of Psychophysiology, pp. 56–84. Cambridge University Press, New York (2007)CrossRef
22.
Zurück zum Zitat Ekman, P., Levenson, R.W., Friesen, W.V.: Autonomic nervous system activity distinguishes among emotions. Science 221, 1208–1210 (1983)CrossRef Ekman, P., Levenson, R.W., Friesen, W.V.: Autonomic nervous system activity distinguishes among emotions. Science 221, 1208–1210 (1983)CrossRef
23.
Zurück zum Zitat Pattyn, N., Neyt, X., Henderickx, D., Soetens, E.: Psychophysiological investigation of vigilance decrement: Boredom or cognitive fatigue? Physiol. Behav. 93, 369–378 (2008)CrossRef Pattyn, N., Neyt, X., Henderickx, D., Soetens, E.: Psychophysiological investigation of vigilance decrement: Boredom or cognitive fatigue? Physiol. Behav. 93, 369–378 (2008)CrossRef
24.
Zurück zum Zitat Stern, R.M., Ray, W.J., Quigley, K.S.: Psychophysiological Recording. University Press Inc., New York (2001) Stern, R.M., Ray, W.J., Quigley, K.S.: Psychophysiological Recording. University Press Inc., New York (2001)
25.
Zurück zum Zitat Pollak, M.H.: Heart rate reactivity to laboratory tasks and ambulatory heart rate in daily life. Psychosom. Med. 53, 25–35 (1991)CrossRef Pollak, M.H.: Heart rate reactivity to laboratory tasks and ambulatory heart rate in daily life. Psychosom. Med. 53, 25–35 (1991)CrossRef
26.
Zurück zum Zitat Jennings, J.R., Kamarck, T., Steward, C., Eddy, M., Johnson, P.: Alternate cardiovascular baseline assessment techniques: vanilla or resting baseline. Psychophysiology 29(6), 742–750 (1992)CrossRef Jennings, J.R., Kamarck, T., Steward, C., Eddy, M., Johnson, P.: Alternate cardiovascular baseline assessment techniques: vanilla or resting baseline. Psychophysiology 29(6), 742–750 (1992)CrossRef
27.
Zurück zum Zitat Fishel, S.R., Muth, E.R., Hoover, A.W.: Establishing appropriate physiological baseline procedures for real-time physiological measurement. J. Cogn. Eng. Decis. Making 1, 286–308 (2007). doi:10.1518/15553407X255636 CrossRef Fishel, S.R., Muth, E.R., Hoover, A.W.: Establishing appropriate physiological baseline procedures for real-time physiological measurement. J. Cogn. Eng. Decis. Making 1, 286–308 (2007). doi:10.​1518/​15553407X255636 CrossRef
28.
29.
Zurück zum Zitat Davidson, R.J., Marshall, J.R., Tomarken, A.J., Henriques, J.B.: While a phobic waits: regional brain electrical and autonomic activity in social phobics during anticipation of public speaking. Biol. Psychol. 47(2), 85–95 (2000). doi:10.1016/S0006-3223(99)00222-X Davidson, R.J., Marshall, J.R., Tomarken, A.J., Henriques, J.B.: While a phobic waits: regional brain electrical and autonomic activity in social phobics during anticipation of public speaking. Biol. Psychol. 47(2), 85–95 (2000). doi:10.​1016/​S0006-3223(99)00222-X
30.
Zurück zum Zitat Hart, S.G., Bortolussi, M.R.: Pilot errors as a source of workload. Hum. Factors 26(5), 545–556 (1984) Hart, S.G., Bortolussi, M.R.: Pilot errors as a source of workload. Hum. Factors 26(5), 545–556 (1984)
34.
Zurück zum Zitat Keil, A., Debener, S., Gratton, G., Junghofer, M., Kappenman, E.S., Luck, S.J., Luu, P., Miller, G.A., Yee, C.M.: Committee report: publication guidelines and recommendations for studies using electroencephalography and magnetoencephalograph. Psychophysiology 51(1), 1–21 (2014)CrossRef Keil, A., Debener, S., Gratton, G., Junghofer, M., Kappenman, E.S., Luck, S.J., Luu, P., Miller, G.A., Yee, C.M.: Committee report: publication guidelines and recommendations for studies using electroencephalography and magnetoencephalograph. Psychophysiology 51(1), 1–21 (2014)CrossRef
35.
Zurück zum Zitat Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., Bablioni, F.: Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue, and drowsiness. Neurosci. Biobehav. Rev. 44, 58–75 (2014). doi:10.1016/jneubiorev.2012.10.003 CrossRef Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., Bablioni, F.: Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue, and drowsiness. Neurosci. Biobehav. Rev. 44, 58–75 (2014). doi:10.​1016/​jneubiorev.​2012.​10.​003 CrossRef
36.
Zurück zum Zitat Mehler, B., Reimer, B., Coughlin, J.F., Dusek, J.A.: Impact of incremental increases in cognitive workload on physiological arousal and performance in young adult drivers. Transp. Res. Rec. 2138, 6–12 (2009)CrossRef Mehler, B., Reimer, B., Coughlin, J.F., Dusek, J.A.: Impact of incremental increases in cognitive workload on physiological arousal and performance in young adult drivers. Transp. Res. Rec. 2138, 6–12 (2009)CrossRef
37.
Zurück zum Zitat Riniolo, T., Porges, S.W.: Inferential and descriptive influences on measures of respiratory sinus arrthymia: sampling rate, R-wave trigger accuracy, and variance estimates. Psychophysiology 34, 613–621 (1997)CrossRef Riniolo, T., Porges, S.W.: Inferential and descriptive influences on measures of respiratory sinus arrthymia: sampling rate, R-wave trigger accuracy, and variance estimates. Psychophysiology 34, 613–621 (1997)CrossRef
38.
Zurück zum Zitat Nyquist, H.: Certain topics in telegraph transmission theory. IEEE Trans. Commun. 47, 617–644 (1928) Nyquist, H.: Certain topics in telegraph transmission theory. IEEE Trans. Commun. 47, 617–644 (1928)
39.
Zurück zum Zitat Bolek, J.E.: Digital sampling, bits, and psychophysiological data: A primer, with cautions. Appl. Psychophys. Biofeedback 38(4), 303–308 (2013)CrossRef Bolek, J.E.: Digital sampling, bits, and psychophysiological data: A primer, with cautions. Appl. Psychophys. Biofeedback 38(4), 303–308 (2013)CrossRef
40.
Zurück zum Zitat Weiergräber, M., Papazoglou, A., Broich, K., Muller, R.: Sampling rate, signal bandwidth and related pitfalls in EEG analysis. J. Neurosci. Methods 268, 53–55 (2016)CrossRef Weiergräber, M., Papazoglou, A., Broich, K., Muller, R.: Sampling rate, signal bandwidth and related pitfalls in EEG analysis. J. Neurosci. Methods 268, 53–55 (2016)CrossRef
41.
Zurück zum Zitat Parasuraman, R.: Neuroergonomic perspectives on human systems integration: mental workload, vigilance, adaptive automation, and training. In: Boehm-Davis, D.A., Durso, F.T., Lee, J.D. (eds.) APA Handbook of Human Systems Integration, pp. 163–176. American Psychological Association, Washington, DC (2015)CrossRef Parasuraman, R.: Neuroergonomic perspectives on human systems integration: mental workload, vigilance, adaptive automation, and training. In: Boehm-Davis, D.A., Durso, F.T., Lee, J.D. (eds.) APA Handbook of Human Systems Integration, pp. 163–176. American Psychological Association, Washington, DC (2015)CrossRef
42.
Zurück zum Zitat Warm, J.S., Parasuraman, R., Matthews, G.: Vigilance requires hard mental work and is stressful. Hum. Factors 50(3), 433–441 (2008)CrossRef Warm, J.S., Parasuraman, R., Matthews, G.: Vigilance requires hard mental work and is stressful. Hum. Factors 50(3), 433–441 (2008)CrossRef
43.
Zurück zum Zitat RTO human factors, medicine panel task group: operator functional state assessment. Technical report, Research and Technology Organization (2004) RTO human factors, medicine panel task group: operator functional state assessment. Technical report, Research and Technology Organization (2004)
44.
Zurück zum Zitat Borghini, G., Aricò, P., Graziani, I., Salinari, S., Sun, Y., Taya, F., Bezerianos, A., Thakor, N.V., Babiloni, F.: Quantitative assessment of the training improvement in a motor-cognitive task by using EEG, ECG, and EOG signals. Brain Topogr. 29, 149–161 (2016)CrossRef Borghini, G., Aricò, P., Graziani, I., Salinari, S., Sun, Y., Taya, F., Bezerianos, A., Thakor, N.V., Babiloni, F.: Quantitative assessment of the training improvement in a motor-cognitive task by using EEG, ECG, and EOG signals. Brain Topogr. 29, 149–161 (2016)CrossRef
45.
Zurück zum Zitat Stuiver, A., Brookhuis, K.A., de Waard, D., Mulder, B.: Short-term cardiovascular measures for driver support: increasing sensitivity for detecting changes in mental workload. Int. J. Psychophysiol. 92, 35–41 (2014)CrossRef Stuiver, A., Brookhuis, K.A., de Waard, D., Mulder, B.: Short-term cardiovascular measures for driver support: increasing sensitivity for detecting changes in mental workload. Int. J. Psychophysiol. 92, 35–41 (2014)CrossRef
Metadaten
Titel
Challenges in Research Generalizability: The Need for Standardization of Performance Metrics and Methodology
verfasst von
Kathryn A. Feltman
Kyle A. Bernhardt
Amanda M. Kelley
Copyright-Jahr
2017
DOI
https://doi.org/10.1007/978-3-319-58481-2_7

Premium Partner