1,354,134 research outputs found
Emslander, Bardach, Dåvang, & Scherer (2025). Seeing what is (not) there: A systematic review of meta-meta-analyses in education [Presentation]
No full textReference
Emslander, V., Bardach, L., Dåvang, E.S., & Scherer, R. (2025, August 25 – 29). Seeing what is (not) there—A systematic review of meta-meta-analyses in education. In Emslander, V. (Ed.) (2025, August 25 – 29). Advancing Research on Learning and Instruction through Meta-Meta-Analyses: Insights and Innovations. Symposium at the 21st Biennial EARLI Conference, Graz, Austria. https://doi.org/10.17605/OSF.IO/4TK7Y
Abstract
As research information and evidence continue to grow, policymakers and researchers increasingly need systematic syntheses of primary studies. This approach provides them with a comprehensive overview of specific aspects of education, such as the effectiveness of various interventions. Addressing this need, educational research has produced a plethora of meta-analyses, oftentimes on similar educational issues. Meta-meta-analyses—sometimes referred to as “second-order meta-analyses” (SOMAs)—are vehicles to consolidate findings across such similar meta-analyses, offering a broader synthesis of evidence. In this preregistered systematic review, we describe and illustrate the principles and practices of SOMAs with a focus on research in learning and instruction. Based on 69 SOMAs in psychology and education, our systematic review of SOMAs examined (a) the methodological quality and adherence to open science practices, (b) the quantitative approaches to synthesize meta-analytic findings, (c) the extent to which heterogeneity was explored, (d) the procedures of dealing with primary study overlap, and (e) the risk-of-bias assessment. This study aims to provide a comprehensive overview of SOMAs, best practices, and needs for further methodological development in learning and instruction. Our findings reveal variability in the methodological quality of the reviewed SOMAs, with most employing standard random-effects models for synthesis but with several notable exceptions that will be discussed. Although most SOMAs uniformly appreciated the overlap of primary studies as a problem, we found diverse approaches to address this issue. Based on our analysis, we propose strategies to enhance the rigor and consistency of SOMAs in psychological and educational research.
Theoretical background
As the body of research and evidence grows, there is an increasing need to systematically synthesize primary studies. This gives researchers and policymakers a comprehensive view of specific aspects of education, like intervention effectiveness. To meet this need, educational research has produced numerous meta-analyses, often covering similar topics. Meta-meta-analyses, also called “second-order meta-analyses” (SOMAs), go a step further by combining findings from similar meta-analyses, providing an even broader perspective on the evidence (e.g., Emslander et al., 2025; Jansen et al., 2022; Schneider & Preckel, 2017). Aims In this paper, we systematically review and critically discuss the applied SOMA methods and highlight best practice examples with a focus on learning and instruction research. Based on 69 SOMAs in psychology and education, our systematic review of SOMAs examined (a) the methodological quality and adherence to open science practices, (b) the quantitative approaches to synthesize meta-analytic findings, (c) the extent to which heterogeneity was explored, (d) the procedures of dealing with primary study overlap, and (e) the risk-of-bias assessment. This study aims to provide a comprehensive overview of SOMAs, best practices, and needs to further improve knowledge generation and dissemination in research on learning and instruction.
Methodology
To review these five common issues, we analyzed 69 SOMAs from 1996 to 2024 including approximately 2400 meta-analyses with over 134,000 primary studies. After preregistration, we conducted a systematic literature search for eligible meta-analyses. During several screening rounds, we identified 69 appropriate SOMAs. From these SOMAs, we extracted data on the publication, the included samples, the search, screening, and coding processes, the overlap check, the quality assessment, the analysis, reporting, discussion, and open science adherence. We summarized the results of our five research aims and developed a narrative overview.
Findings
Looking at (a) the quality of reporting and the adherence to open science, the SOMAs showed a trend toward better quality and more open science in more recent years. (b) Most SOMAs applied standard random-effects models as recommended by Cooper and Koenka (2012), which does not separately account for the variation between effect sizes and between meta-analyses. (c) Most SOMAs quantified heterogeneity by common heterogeneity indices and then explored it further with a wide array of moderator analyses. Student and teacher variables such as their backgrounds or age were common moderators in SOMAs on learning and instruction. (d) Common approaches in educational SOMAs to deal with primary study overlap were (1) excluding meta-analyses overlapping more than 25%, (2) testing the sensitivity of results depending on whether overlap was accounted for or not, and (3) using a uniqueness score to weigh included meta-analyses. (e) Finally, risk-of-bias assessment has gained importance in SOMAs over the past three decades and an increasing number of visual and statistical methods have been used in more recent years. Similarly, SOMA authors widely discuss the importance of the search and inclusion of gray literature to mitigate publication bias.
Theoretical and educational significance of the research
Since SOMAs provide a meta-view on synthesized evidence, their potential for educational research lies in systematically mapping evidence and approaches in a field and, ultimately, the building and testing of theories. Given the variety and shortcomings of approaches educational researchers have taken to deal with core issues in a SOMA (e.g., study overlap, quality), we propose strategies to enhance the rigor and consistency of SOMA methodologies in educational research.
References
Cooper, H., & Koenka, A. C. (2012). The overview of reviews: Unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. American Psychologist, 67(6), Article 6. https://doi.org/10.1037/a0027119
Emslander, V., Holzberger, D., Ofstad, S. B., Fischbach, A., & Scherer, R. (2025). Teacher–student relationships and student outcomes: A systematic second-order meta-analytic review. Psychological Bulletin, 151(3), 365–397. https://doi.org/10.1037/bul0000461
Jansen, T., Meyer, J., Wigfield, A., & Möller, J. (2022). Which student and instructional variables are most strongly related to academic motivation in K-12 education? A systematic review of meta-analyses. Psychological Bulletin, 148(1–2), Article 1–2. https://doi.org/10.1037/bul0000354
Schneider, M., & Preckel, F. (2017). Variables associated with achievement in higher education: A systematic review of meta-analyses. Psychological Bulletin, 143(6), Article 6. https://doi.org/10.1037/bul000009
Fine coding dataset and manual for: Emotional Intelligence Trainings for University Students: a Systematic Review on Models, Content, and Measures
No full textThis supplementary material contains fine coding dataset and coding manual for: Asiedu, A., Emslander, V., Müller-Kreiner, C., Seidel., T., Holzberger, D. Emotional Intelligence Trainings for University Students: a Systematic Review on Models, Content, and Measuresunknow
Figures for: Emotional Intelligence Trainings for University Students: a Systematic Review of Models, Content, and Measures
No full textThis supplementary material contains additional figures for: Asiedu, A., Emslander, V., Müller-Kreiner, C., Seidel, T., Holzberger, D. Emotional Intelligence Trainings for University Students: a Systematic Review of Models, Content, and Measuresunknow
List of References of included studies for: Emotional Intelligence Trainings for University Students: a Systematic Review of Models, Content, and Measures
No full textThis supplementary material contains a references of the included studies for: Asiedu, A., Emslander, V., Müller-Kreiner, C., Seidel, T., Holzberger, D. Emotional Intelligence Trainings for University Students: a Systematic Review of Models, Content, and Measuresunknownunknow
What Luxembourg's Primary Schools are Doing Right: A Value-Added Comparison in the Luxembourgish School Context
No full textKurz-Abstract (120 Wörter)
Luxemburgs Bildungssystem ist geprägt von multi-kulturellen und vielsprachigen Schüler:innen und einem zweimaligen Wechsel der Instruktionssprache. Dies führt zu sehr unterschiedlichen Voraussetzungen für die Schullaufbahn der Schüler:innen.
Das Ziel des vorliegenden SIVA-Projekts (Systematic Identification of High Value-Added in Educational Contexts) ist herauszufinden, welche pädagogischen Strategien Schulen mit hohen Value-Added (VA)-Werten für Schuleffektivität anwenden und was andere Schulen von ihnen lernen können, um diese Ungleichheiten abzubauen.
Zuerst ermittelten wir 16 Schulen, die stabil hohe, mittlere oder niedrige VA-Werte aufwiesen. Danach sammelten wir Daten anhand von Fragebögen und Unterrichtsbeobachtungen über pädagogische Strategien und das Schulklima und glichen sie mit repräsentativen Schulmonitoringergebnissen ab.
Wir werden das SIVA-Projekt, seine Ziele und die Datenerhebung diskutieren, die zu unserem reichhaltigen Datensatz aus sechs Perspektiven führte.
Zusammenfassung (480 Wörter)
In einem multi-kulturellen und vielsprachigen Land wie Luxemburg können leicht Bildungsungleichheiten entstehen. Unterschiedliche zu Hause gesprochene Sprachen, Migrationshintergründe oder der sozioökonomische Status einer Familie können zu ungleichen Erfolgschancen in der Schule werden. Gepaart mit einem Schulsystem, in dem zweimal die Instruktionssprache gewechselt wird, führt diese Vielfalt zu unterschiedlichen Voraussetzungen für das Erlernen von Mathematik und Sprachen und prägt somit die Schullaufbahn der Schüler:innen (Hadjar & Backes, 2021). Diese Gemengelage ist einerseits herausfordernd für Schüler:innen, Lehrkräfte und Schulen, zeigt aber andererseits, dass es gelingende soziale und pädagogische Praktiken geben muss, diese Herausforderungen zu meistern, da die Schulen weiterhin effektiv arbeiten.
In den USA wurde Schuleffektivität häufig mit Value-Added-Werten (VA) quantifiziert, welche durch ihre Instabilität zu ungerechtfertigten Finanzierungs- und Personalentscheidungen führten (Emslander, Levy, Scherer, et al., 2022). Ziel des Projekts Systematic Identification of High Value-Added in Educational Contexts (SIVA; Emslander, Levy, & Fischbach, 2022) ist es, dieses repressiv genutzte Instrument der VA-Werte konstruktiv anzuwenden. VA ist ein statistisches Regressionsverfahren, um die Effektivität von Schulen unter Berücksichtigung unterschiedlicher Schüler:innenhintergründe gerecht zu schätzen. Wir untersuchten, (1) was hocheffektive Schulen "richtig" machen und (2) was andere Schulen von ihnen lernen können, um Ungleichheiten abzubauen. In Zusammenarbeit mit der Section Qualité Scolaire des Observatoire National de l’Enfance, de la Jeunesse et de la Qualité Scolaire, untersuchten wir die Unterschiede zwischen Schulen mit stabil hohen, mittleren oder niedrigen VA-Werten aus verschiedenen Perspektiven.
Zunächst haben wir 16 Schulen ermittelt, die über zwei Jahre hinweg stabile hohe, mittlere oder niedrige VA-Werte aufwiesen. Als Zweites sammelten wir Fragebogen- und Unterrichtsbeobachtungsdaten über ihre pädagogischen Strategien, den Hintergrund der Schüler:innen und das Schulklima. Als Drittes glichen wir unsere Daten mit den Ergebnissen des luxemburgischen Schulmonitorings ÉpStan (LUCET, 2021) ab. Wir haben die Variablen auf der Grundlage von Lernmodellen ausgewählt, die sich auf Aspekte wie die Schulorganisation oder das Klassenmanagement konzentrieren (z.B. Hattie, 2008; Klieme et al., 2001). Darüber hinaus untersuchten wir die Besonderheiten des luxemburgischen Schulsystems, die in internationalen schulischen Lernmodellen nicht vertreten sind (z. B. die Einteilung in zweijährige Lernzyklen, die mehrsprachige Schulumgebung und die vielfältige Schülerschaft).
Wir werden das SIVA-Projekt, seine Ziele und Besonderheiten diskutieren, die zu Daten aus 49 Klassenzimmerbeobachtungen und Fragebögen mit über 500 Zweitklässler:innen, ihren Eltern, 200 Lehrkräften sowie Schulleiter:innen und Schulaufsichtsbehörden führte.
Literature
Emslander, V., Levy, J., & Fischbach, A. (2022). Systematic Identification of High “Value-Added” in Educational Contexts (SIVA). https://doi.org/10.17605/OSF.IO/X3C48
Emslander, V., Levy, J., Scherer, R., & Fischbach, A. (2022). Value-added scores show limited stability over time in primary school. PLOS ONE, 17(12), e0279255. https://doi.org/10.1371/journal.pone.0279255
Hadjar, A., & Backes, S. (2021). Bildungsungleichheiten am Übergang in die Sekundarschule in Luxemburg. https://doi.org/10.48746/BB2021LU-DE-21A
Hattie, J. (2008). Visible Learning: A synthesis of over 800 meta-analyses relating to achievement (0 ed.). Routledge. https://doi.org/10.4324/9780203887332
Klieme, E., Schümer, G., & Knoll, S. (2001). Mathematikunterricht in der Sekundarstufe I: “Aufgabenkultur” und Unterrichtsgestaltung. TIMSS - Impulse für Schule und Unterricht, 43–57.
LUCET. (2021). Épreuves Standardisées (ÉpStan). https://epstan.luSIV
Going Beyond Counting First Authors in Author Co-citation Analysis
No full textThe present study examines one of the fundamental aspects of author co-citation analysis (ACA) - the way co-citation
counts are defined. Co-citation counting provides the data on which all subsequent statistical analyses and mappings
are based, and we compare ACA results based on two different types of co-citation counting - the traditional type that
only counts the first one among a cited work's authors on the one hand and a non-traditional type that takes into
account the first 5 authors of a cited work on the other hand. Results indicate that the picture produced through this non-traditional author co-citation counting contains more coherent author groups and is therefore considerably clearer. However, this picture represents fewer specialties in the research field being studied than that produced through the traditional first-author co-citation counting when the same number of top-ranked authors is selected and analyzed. Reasons for these effects are discussed
Variations on the Author
No full text“Variations on the Author” discusses two of Eduardo Coutinho’s recent films (Um Dia na Vida, from 2010, and Últimas Conversas, posthumously released in 2015) and their contribution to the general question of documentary authorship. The director’s filmography is characterized by a consistent yet self-effacing form of authorial self-inscription: Coutinho often features as an interviewer that rather than express opinions propels discourses; an interviewer that is good at listening. This mode of self-inscription characterizes him as an author who is not expressive but who is nonetheless markedly present on the screen. In Um Dia na Vida, however, Coutinho is completely absent form the image, while Últimas Conversas, on the contrary, includes a confessional prologue that moves the director from the margins to the center of his films. This article examines the ways in which these works stand out in the filmography of a director who offers new insights into the notion of cinematic authorship
Appropriate Similarity Measures for Author Cocitation Analysis
No full textWe provide a number of new insights into the methodological discussion about author cocitation analysis. We first argue that the use of the Pearson correlation for measuring the similarity between authors’ cocitation profiles is not very satisfactory. We then discuss what kind of similarity measures may be used as an alternative to the Pearson correlation. We consider three similarity measures in particular. One is the well-known cosine. The other two similarity measures have not been used before in the bibliometric literature. Finally, we show by means of an example that our findings have a high practical relevance.information science;Pearson correlation;cosine;similarity measure;author cocitation analysis
Dispelling the Myths Behind First-author Citation Counts
No full textWe conducted a full-scale evaluative citation analysis study of scholars in the XML research field to explore just how different from each other author rankings resulting from different citation counting methods actually are, and to demonstrate the capability of emerging data and tools on the Web in supporting more realistic citation counting methods. Our results contest some common arguments for the continued
use of first-author citation counts in the evaluation of scholars, such as high correlations between author rankings by first-author citation counts and other citation
counting methods, and high costs of using more realistic citation counting methods that are not well-supported by the ISI databases. It is argued that increasingly available digital full text research papers make it possible for citation analysis studies to go beyond what the ISI databases have directly supported and to employ more
sophisticated methods
Author, publisher and bookseller : a tripartite synergy in Nigerian book industry
No full textThis work is about the roles of Author, Publisher and Bookseller in Book development in
Nigeria. The paper started by delving into the history of Book Publishing in Nigeria after
which it proceeded by defining who an author, a publisher, and a bookseller is and
expatiated on the indispensable roles of these key actors in Nigerian Book Industry and in
the emerging Information Society. Furthermore, the various constraints to book
development were identified while the paper advised on how the Book Industry can be
further promoted in Nigeria. However, the paper concluded and made recommendations
on how the Book sector can help in enhancing scholarship in the country
- …
