SOEP Brown Bag Seminar

Comparisons of educational achievement across cohorts are frequently used to examine changes in educational performance and to evaluate the impact of societal or educational events, such as the COVID-19 pandemic. However, drawing causal inferences from observational data without strong experimental control remains challenging, as often no single clear methodological approach is universally accepted for addressing issues such as matching, missing data treatment, or competence scaling.  Different defensible analytical choices may substantially influence estimates of the focal effect. A multiverse approach can therefore help make uncertainty in effect estimation transparent. The present study demonstrates this approach using data from the German National Educational Panel Study (NEPS) that tracks multiple cohorts of children, adolescents, and adults across their life course. Scientific literacy assessed in Grades 1 and 2 were compared across two longitudinal cohorts (total N = 6,365) to investigate the impact of pandemic-related school closures on changes in science achievement. One cohort was unaffected by the pandemic, whereas the other experienced pandemic-related disruptions during primary school. By systematically varying analytical decisions, the study illustrates how conclusions regarding the same research question may differ across equally defensible analytical approaches. Finally, methodological challenges involved in combining the diverse effect estimates generated across the multiverse of analyses are discussed.