Top

Quality of Life Research

Published in:

20-04-2018 | Review

A systematic review of the quality of reporting of simulation studies about methods for the analysis of complex longitudinal patient-reported outcomes data

Authors: Aynslie M. Hinds, Tolulope T. Sajobi, Véronique Sebille, Richard Sawatzky, Lisa M. Lix

Published in: Quality of Life Research | Issue 10/2018

Activate our intelligent search to find suitable subject content or patents.

search-config

AI-assisted search

Off

Abstract

Purpose

This study describes the characteristics and quality of reporting for published computer simulation studies about statistical methods to analyze complex longitudinal (i.e., repeated measures) patient-reported outcomes (PROs); we included methods for longitudinal latent variable measurement and growth models and response shift.

Methods

Scopus, PsycINFO, PubMed, EMBASE, and Social Science Citation Index were searched for English-language studies published between 1999 and 2016 using selected keywords. Extracted information included characteristics of the study purpose/objectives, simulation design, software, execution, performance, and results. The quality of reporting was evaluated using published best-practice guidelines.

Synthesis

A total of 1470 articles were reviewed and 42 articles met the inclusion criteria. The majority of the included studies (73.8%) investigated an existing statistical method, primarily a latent variable model (95.2%). Most studies specified the population model, including variable distributions, mean parameters, and correlation/covariances. The number of time points and sample size(s) were reported by all studies, but justification for the selected values was rarely provided. The majority of the studies (52.4%) did not report on model non-convergence. Bias, accuracy, and model fit were commonly reported performance metrics. All studies reported results descriptively, and 26.2% also used an inferential method.

Conclusions

While methodological research on statistical analyses of complex longitudinal PRO data is informed by computer simulation studies, current reporting practices of these studies have not been consistent with best-practice guidelines. Comprehensive reporting of simulation methods and results ensures that the strengths and limitations of the investigated statistical methods are thoroughly explored.

previous article Correction to: Health-related quality of life in coronary heart disease: a systematic review and meta-analysis mapped against the International Classification of Functioning, Disability and Health

next article Weak social networks and restless sleep interrelate through depressed mood among elderly

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Online-Abonnement

Mit Springer Professional "Wirtschaft+Technik" erhalten Sie Zugriff auf:

über 102.000 Bücher
über 537 Zeitschriften

aus folgenden Fachgebieten:

Automobil + Motoren
Bauwesen + Immobilien
Business IT + Informatik
Elektrotechnik + Elektronik
Energie + Nachhaltigkeit
Finance + Banking
Management + Führung
Marketing + Vertrieb
Maschinenbau + Werkstoffe
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Springer Professional "Wirtschaft"

Online-Abonnement

Mit Springer Professional "Wirtschaft" erhalten Sie Zugriff auf:

über 67.000 Bücher
über 340 Zeitschriften

aus folgenden Fachgebieten:

Bauwesen + Immobilien
Business IT + Informatik
Finance + Banking
Management + Führung
Marketing + Vertrieb
Versicherung + Risiko

Jetzt Wissensvorsprung sichern!

inform now

Available only for authorised users

Ahmed, S., & Ring, L. (2008). Influence of response shift on evaluations of change in patient-reported outcomes. Expert Review of Pharmacoeconomics and Outcomes Research, 8(5), 479–489. https://doi.org/10.1586/14737167.8.5.479.CrossRefPubMed

Fayers, P. M., & Machin, D. (2016). Quality of life: The assessment, analysis, and reporting of patient-reported outcomes (3rd ed.). Chichester: Wiley.

Sawatzky, R., Chan, E. K. H., Zumbo, B. D., Ahmed, S., Bartlett, S. J., Bingham, C. O., … Lix, L. M. (2016). Modern perspectives of measurement validation emphasize justification of inferences based on patient-reported outcome scores: Seventh paper in a series on patient reported outcomes. Journal of Clinical Epidemiology. https://doi.org/10.1016/j.jclinepi.2016.12.002.CrossRefPubMed

van Rijn, T. (2009). A physiatrist’s view of response shift. Journal of Clinical Epidemiology, 62, 1191–1195. https://doi.org/10.1016/j.jclinepi.2009.01.023.CrossRefPubMed

Schwartz, C. E., & Sprangers, M. A. (1999). Methodological approaches for assessing response shift in longitudinal health-related quality-of-life research. Social Science & Medicine, 48, 1531–1548.CrossRef

Ring, L., Hofer, S., Heuston, F., Harris, D., & O’Boyle, C. A. (2005). Response shift masks the treatment impact on patient reported outcomes (PROs): The example of individual quality of life in edentulous patients. Health and Quality of Life Outcomes, 3, 55. https://doi.org/10.1186/1477-7525-3-55.CrossRefPubMedPubMedCentral

Millsap, R. E. (2010). Testing measurement invariance using item response theory in longitudinal data: An introduction. Child Development Perspectives, 4(1), 5–9.CrossRef

Sawatzky, R., Sajobi, T. T., Brahmbhatt, R., Chan, E. K. H., Lix, L. M., & Zumbo, B. D. (2017). Longitudinal change in response processes: A response shift perspective. In B. D. Zumbo & A. M. Hubley (Eds.), Understanding and investigating response processes in validation research (1st ed., pp. 251–276). Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-319-56129-5_14.CrossRef

Ahmed, S., Sawatzky, R., Levesque, J. F., Ehrmann-Feldman, D., & Schwartz, C. E. (2014). Minimal evidence of response shift in the absence of a catalyst. Quality of Life Research, 23(9), 2421–2430. https://doi.org/10.1007/s11136-014-0699-3.CrossRefPubMed

10.

Guilleux, A., Blanchin, M., Vanier, A., Guillemin, F., Falissard, B., Schwartz, C. E., … Sébille, V. (2015). RespOnse Shift ALgorithm in item response theory (ROSALI) for response shift detection with missing data in longitudinal patient-reported outcome studies. Quality of Life Research, 24(3), 553–564. https://doi.org/10.1007/s11136-014-0876-4.CrossRefPubMed

11.

Mayo, N. E., Scott, S. C., Dendukuri, N., Ahmed, S., & Wood-Dauphinee, S. (2008). Identifying response shift statistically at the individual level. Quality of Life Research, 17(4), 627–639. https://doi.org/10.1007/s11136-008-9329-2.CrossRefPubMed

12.

Lowy, A., & Bernhard, J. (2004). Quantitative assessment of changes in patients’ constructs of quality of life: An application of multilevel models. Quality of Life Research, 13(7), 1177–1185. https://doi.org/10.1023/B:QURE.0000037510.17893.d2.CrossRefPubMed

13.

Lix, L. M., Sajobi, T. T., Sawatzky, R., Liu, J., Mayo, N. E., Huang, Y., … Bernstein, C. N. (2013). Relative importance measures for reprioritization response shift. Quality of Life Research, 22(4), 695–703. https://doi.org/10.1007/s11136-012-0198-3.CrossRefPubMed

14.

Blanchin, M., Hardouin, J. B., Le Neel, T., Kubis, G., Blanchard, C., Mirallie, E., & Sebille, V. (2011). Comparison of CTT and Rasch-based approaches for the analysis of longitudinal patient reported outcomes. Statistics in Medicine, 30(8), 825–838. https://doi.org/10.1002/sim.4153.CrossRefPubMed

15.

Hallgren, K. A. (2013). Conducting simulation studies in the R programming environment. Tutorials in Quantitative Methods for Psychology, 9(2), 43–60. https://doi.org/10.20982/tqmp.09.2.p043.CrossRefPubMedPubMedCentral

16.

Sainani, K. (2015). What is computer simulation? PM&R: The Journal of Injury, Function & Rehabilitation, 7(12), 1290–1293. https://doi.org/10.1016/j.pmrj.2015.10.010.CrossRef

17.

Boomsma, A. (2013). Reporting Monte Carlo studies in structural equation modeling. Structural Equation Modeling: A Multidisciplinary Journal, 20(3), 518–540. https://doi.org/10.1080/10705511.2013.797839.CrossRef

18.

Gorter, R., Fox, J.-P., & Twisk, J. W. R. (2015). Why item response theory should be used for longitudinal questionnaire data analysis in medical research. BMC Medical Research Methodology, 15(55), 1–12. https://doi.org/10.1186/s12874-015-0050-x.CrossRef

19.

de Bock, E., Hardouin, J., Blanchin, M., Le Neel, T., Kubis, G., Bonnaud-Antignac, A., … Sebille, V. (2013). Rasch-family models are more valuable than score-based approaches for analysing longitudinal patient-reported outcomes with missing data. Statistical Methods in Medical Research, 16, 1–21. https://doi.org/10.1177/0962280213515570.CrossRef

20.

de Bock, E., Hardouin, J. B., Blanchin, M., Le Neel, T., Kubis, G., & Sebille, V. (2015). Assessment of score- and Rasch-based methods for group comparison of longitudinal patient-reported outcomes with intermittent missing data (informative and non-informative). Quality of Life Research, 24(1), 19–29. https://doi.org/10.1007/s11136-014-0648-1.CrossRefPubMed

21.

Burton, A., Altman, D. G., Royston, P., & Holder, R. L. (2006). The design of simulation studies in medical statistics. Statistics in Medicine, 25(24), 4279–4292. https://doi.org/10.1002/sim.2673.CrossRefPubMed

22.

Sajobi, T. T., Lix, L. M., Singh, G., Lowerison, M., Engbers, J., & Mayo, N. E. (2015). Identifying reprioritization response shift in a stroke caregiver population: A comparison of missing data methods. Quality of Life Research, 24(3), 529–540. https://doi.org/10.1007/s11136-014-0824-3.CrossRefPubMed

23.

Vanier, A., Sebille, V., Blanchin, M., Guilleux, A., & Hardouin, J. B. (2015). Overall performance of Oort’s procedure for response shift detection at item level: A pilot simulation study. Quality of Life Research, 24(8), 1799–1807. https://doi.org/10.1007/s11136-015-0938-2.CrossRefPubMed

24.

Paxton, P., Curran, P. J., Bollen, K. A., Kirby, J., & Chen, F. (2001). Monte Carlo experiments: Design and implementation. Structural Equation Modeling, 8(2), 287–312. https://doi.org/10.1207/S15328007SEM0802.CrossRef

25.

Skrondal, A. (2000). Design and analysis of Monte Carlo experiments: Attacking the conventional wisdom. Multivariate Behavioral Research, 35(2), 137–167. https://doi.org/10.1207/S15327906MBR3502.CrossRefPubMed

26.

Feinberg, R. A., & Rubright, J. D. (2016). Conducting simulation studies in psychometrics. Educational Measurement: Issues and Practice, 35(2), 36–49. https://doi.org/10.1111/emip.12111.CrossRef

27.

Waltemath, D., Adams, R., Beard, D. A., Bergmann, F. T., Bhalla, U. S., Britten, R., … Le Novère, N. (2011). Minimum information about a simulation experiment (MIASE). PLoS Computational Biology, 7(4), 5–8. https://doi.org/10.1371/journal.pcbi.1001122.CrossRef

28.

Elsevier. (2017). Scopus. Retrieved July 31, 2017, from https://www.elsevier.com/solutions/scopus.

29.

Smith, M. K., & Marshall, A. (2011). Importance of protocols for simulation studies in clinical drug development. Statistical Methods in Medical Research, 20(6), 613–622. https://doi.org/10.1177/0962280210378949.CrossRefPubMed

30.

Kuribayashi, K. (2014). Design and analysis of clinical trial simulations. In K. van Monfort, J. Oud & W. Ghidey (Eds.), Developments in statistical evaluation of clinical trials (pp. 139–151). Rotterdam: Springer. https://doi.org/10.1007/978-3-642-55345-5.CrossRef

31.

Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Journal of Clinical Epidemiology, 62(10), 1006–1012. https://doi.org/10.1016/j.jclinepi.2009.06.005.CrossRefPubMed

32.

Verhagen, J., & Fox, J. P. (2013). Longitudinal measurement in health-related surveys. A Bayesian joint growth model for multivariate ordinal responses. Statistics in Medicine, 32(17), 2988–3005. https://doi.org/10.1002/sim.5692.CrossRefPubMed

33.

Huang, H.-Y. (2015). A multilevel higher order item response theory model for measuring latent growth in longitudinal data. Applied Psychological Measurement, 39(5), 362–372. https://doi.org/10.1177/0146621614568112.CrossRefPubMedPubMedCentral

34.

Kohli, N., Harring, J. R., & Zopluoglu, C. (2016). A finite mixture of nonlinear random coefficient models for continuous repeated measures data. Psychometrika, 81(3), 851–880. https://doi.org/10.1007/s11336-015-9462-0.CrossRefPubMed

35.

Rouquette, A., Côté, S. M., Hardouin, J.-B., & Falissard, B. (2016). Rasch modelling to deal with changes in the questionnaires used during long-term follow-up of cohort studies: A simulation study. BMC Medical Research Methodology, 16(105), 1–9. https://doi.org/10.1186/s12874-016-0211-6.CrossRef

36.

Geiser, C., Keller, B. T., & Lockhart, G. (2013). First- versus second-order latent growth curve models: Some insights from latent state-trait theory. Structural Equation Modeling: A Multidisciplinary Journal, 20(3), 479–503. https://doi.org/10.1080/10705511.2013.797832.CrossRef

37.

Geiser, C., Keller, B. T., Lockhart, G., Eid, M., Cole, D., & Koch, T. (2015). Distinguishing state variability from trait change in longitudinal data: The role of measurement (non)invariance in latent state-trait analyses. Behavior Research Methods, 47(1), 172–203. https://doi.org/10.3758/s13428-014-0457-z.CrossRefPubMedPubMedCentral

38.

Kim, E. S., & Willson, V. L. (2014). Testing measurement invariance across groups in longitudinal data: Multigroup second-order latent growth model. Structural Equation Modeling: A Multidisciplinary Journal, 21(4), 566–575. https://doi.org/10.1080/10705511.2014.919821.CrossRef

39.

Kim, E. S., & Willson, V. L. (2014). Measurement invariance across groups in latent growth modeling. Structural Equation Modeling: A Multidisciplinary Journal, 21(3), 408–424. https://doi.org/10.1080/10705511.2014.915374.CrossRef

40.

Koch, T., Schultze, M., Eid, M., & Geiser, C. (2014). A longitudinal multilevel CFA-MTMM model for interchangeable and structurally different methods. Frontiers in Psychology, 5(APR), 1–19. https://doi.org/10.3389/fpsyg.2014.00311.CrossRef

41.

Rhemtulla, M., Jia, F., Wu, W., & Little, T. D. (2014). Planned missing designs to optimize the efficiency of latent growth parameter estimates. International Journal of Behavioral Development, 38(5), 423–434. https://doi.org/10.1177/0165025413514324.CrossRef

42.

Sterba, S. K. (2017). Pattern mixture models for quantifying missing data uncertainty in longitudinal invariance testing. Structural Equation Modeling: A Multidisciplinary Journal, 24(2), 283–300. https://doi.org/10.1080/10705511.2016.1250635.CrossRef

43.

Cole, D. A., Martin, N. C., & Steiger, J. H. (2005). Empirical and conceptual problems with longitudinal trait- state models: Introducing a trait- state-occasion model. Psychological Medicine, 10(1), 3–20. https://doi.org/10.1037/1082-989X.10.1.3.CrossRef

44.

Diallo, T. M. O., Morin, A. J. S., & Parker, P. D. (2013). Statistical power of latent growth curve models to detect quadratic growth. Behavior Research Methods, 46(2), 357–371. https://doi.org/10.3758/s13428-013-0395-1.CrossRef

45.

Fan, X., & Fan, X. (2005). Power of latent growth modeling for detecting linear growth: Number of measurements and comparison with other analytic approaches. The Journal of Experimental Education, 73(2), 121–139. https://doi.org/10.3200/JEXE.73.2.121-139.CrossRef

46.

Holgado-Tello, F. P., Chacon-Moscoso, S., Sanduvete-Chaves, S., & Perez-Gil, J. A. (2016). A simulation study of threats to validity in quasi-experimental designs: Interrelationship between design, measurement, and analysis. Frontiers in Psychology, 7, 897. https://doi.org/10.3389/fpsyg.2016.00897.CrossRefPubMedPubMedCentral

47.

Estabrook, R. (2015). Evaluating measurement of dynamic constructs: Defining a measurement model of derivatives. Psychological Methods, 20(1), 117–141. https://doi.org/10.1037/a0034523.Evaluating.CrossRefPubMed

48.

Fan, X. (2003). Power of latent growth modeling for detecting group differences in linear growth trajectory parameters. Structural Equation Modeling: A Multidisciplinary Journal, 10(3), 380–400. https://doi.org/10.1207/S15328007SEM1003.CrossRef

49.

Kwok, O.-M., Luo, W., & West, S. G. (2010). Using modification indexes to detect turning points in longitudinal data: A Monte Carlo study. Structural Equation Modeling: A Multidisciplinary Journal, 17(2), 216–240. https://doi.org/10.1080/10705511003659359.CrossRef

50.

Wilson, M., Zheng, X., & McGuire, L. (2012). Formulating latent growth using an explanatory item response model approach. Journal of Applied Measurement, 13(1), 1–22.PubMed

51.

Bacci, S. (2012). Longitudinal data: Different approaches in the context of item-response theory models. Journal of Applied Statistics, 39(9), 2047–2065. https://doi.org/10.1080/02664763.2012.700451.CrossRef

52.

Koch, T., Schultze, M., Eid, M., & Geiser, C. (2014). A longitudinal multilevel CFA-MTMM model for interchangeable and structurally different methods. Frontiers in Psychology, 5, 311. https://doi.org/10.3389/fpsyg.2014.00311.CrossRefPubMedPubMedCentral

53.

Nestler, S. (2014). How the 2SLS/IV estimator can handle equality constraints in structural equation models: A system-of-equations approach. British Journal of Mathematical and Statistical Psychology, 67(2), 353–369. https://doi.org/10.1111/bmsp.12023.CrossRefPubMed

54.

Feddag, M. L., Blanchin, M., Hardouin, J. B., & Sebille, V. (2014). Power analysis on the time effect for longitudinal Rasch model. Journal of Applied Measurement, 15(3), 292–301.PubMed

55.

Tavares, H. R., & Andrade, D. F. (2006). Item response theory for longitudinal data: Item and population ability parameters estimation. Sociedad de Estadistica e Investigacion Operativa, 15(1), 97–123.

56.

Blanchin, M., Hardouin, J.-B., Le Neel, T., Kubis, G., & Sebille, V. (2011). Analysis of longitudinal patient-reported outcomes with informative and non-informative dropout: Comparison of CTT and Rasch-based methods. International Journal of Applied Mathematics & Statistics, 24(SI-11A), 107–124.

57.

Jeon, M., & Rabe-Hesketh, S. (2016). An autoregressive growth model for longitudinal item analysis. Psychometrika, 81(3), 830–850. https://doi.org/10.1007/s11336-015-9489-2.CrossRefPubMed

58.

Luo, S., Ma, J., & Kieburtz, K. D. (2013). Robust Bayesian inference for multivariate longitudinal data by using normal/independent distributions. Statistics in Medicine, 32(22), 3812–3828.CrossRef

59.

Hamon, A., & Iovleff, S. (2003). Computation three stochastic versions of the EM Algorithm for estimating longitudinal Rasch model. Communications in Statistics - S, 32(2), 275–295. https://doi.org/10.1081/SAC-120017492.CrossRef

60.

Andrade, D. F., & Tavares, H. R. (2005). Item response theory for longitudinal data: Population parameter estimation. Journal of Multivariate Analysis, 95, 1–22. https://doi.org/10.1016/j.jmva.2004.07.005.CrossRef

61.

Feddag, M.-L., Grama, I., & Mesbah, M. (2003). Generalized estimating equations (GEE) for mixed logistic models. Communications in Statistics - Theory and Methods, 32(4), 851–874. https://doi.org/10.1081/STA-120018833.CrossRef

62.

Kim, S.-Y. (2014). Determining the number of latent classes in single- and multi-phase growth mixture models. Structural Equation Modeling: A Multidisciplinary Journal, 21(2), 263–279. https://doi.org/10.1080/10705511.2014.882690.CrossRef

63.

Bang, J. W., Schumacker, R. E., & Schlieve, P. L. (1998). Random-number generator validity in simulation studies: An investigation of normality. Educational and Psychological Measurement, 58(3), 430–450.CrossRef

64.

Gadermann, A. M., Sawatzky, R., Palepu, A., Hubley, A. M., Zumbo, B. D., Aubry, T., … Hwang, S. W. (2017). Minimal impact of response shift for SF-12 mental and physical health status in homeless and vulnerably housed individuals: An item-level multi-group analysis. Quality of Life Research, 26(6), 1463–1472. https://doi.org/10.1007/s11136-016-1464-6.CrossRefPubMed

65.

Meijer, R. R., & Sijtsma, K. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25(2), 107–135.CrossRef

Title: A systematic review of the quality of reporting of simulation studies about methods for the analysis of complex longitudinal patient-reported outcomes data
Authors: Aynslie M. Hinds
Tolulope T. Sajobi
Véronique Sebille
Richard Sawatzky
Lisa M. Lix
Publication date: 20-04-2018
Publisher: Springer International Publishing
Published in: Quality of Life Research / Issue 10/2018
Print ISSN: 0962-9343
Electronic ISSN: 1573-2649
DOI: https://doi.org/10.1007/s11136-018-1861-0

Springer Professional

Abstract

Purpose

Methods

Synthesis

Conclusions

Please log in to get access to your license.

Dont have a licence yet? Then find out more about our products and how to get one now:

Springer Professional "Wirtschaft+Technik"

Springer Professional "Wirtschaft"

Other articles of this Issue 10/2018

Relationship between low handgrip strength and quality of life in Korean men and women

Quality of life and anorexia/cachexia in lung cancer: validation of the Spanish version of the FAACT instrument

Third molar removal and its impact on quality of life: systematic review and meta-analysis

Quality of life in Sardinian patients with transfusion-dependent Thalassemia: a cross-sectional study

A population-based analysis of life satisfaction and social support among children of diverse backgrounds in British Columbia, Canada

Fatigue in the general population: German normative values of the EORTC QLQ-FA12