Second-order synthesis of meta-analytic studies in applied linguistics (1998–2021)

The present article reports on a study which examined research synthetic studies over a period of 24 years from 1998 to 2021 collected from 30 representative applied linguistic journals. The study aimed to investigate the adequacy of reporting and adherence to standards of rigor and transparency in all steps of the meta-analyses. Following a synthetic approach, we synthesized 90 meta-analyses in applied linguistics (AL) journals. The results revealed strengths in all steps and a number of weaknesses in some steps of the process. AL meta-analysts did fairly well in locating the candidate studies by using different databases and supplementary search strategies. In their data analysis phase, they used random effects models more than a combination of random and fixed effects models. To interpret the effect sizes, almost half of the studies used Cohen’s benchmark and almost one third of them used Plonsky and Oswald’s field-specific benchmark. The areas which need to be addressed in future synthetic studies were identified as: more precise identification of moderator variables; more reliable data collection by paying adequate attention to coding sheet piloting and coder qualifications and training; and more reliance on measures other than Q-statistics. The implications of these results are discussed.