Unveiling the Impact of STEM Education: A Meta-Analysis Overview
Introduction
STEM education, a multidisciplinary approach integrating science, technology, engineering, and mathematics, has garnered significant attention in both industry and academia since its emergence in 1986. The primary objective of STEM education is to cultivate students' innovation, creativity, and practical skills. While traditional learning outcomes focused on cognitive domains like memory and understanding, modern educational theory has expanded to encompass affective and motor skills. This article delves into a meta-analysis examining the effects of STEM education on student learning outcomes, exploring the influence of various moderating variables.
The Evolving Landscape of Learning Outcomes
Traditional learning outcomes have centred on ‘knowledge mastery’ in the cognitive domain (e.g., memory, understanding), but modern educational theory has expanded its objectives to include the affective domain and motor skills domain. Constructivism, for instance, identifies problem-solving and creativity as higher-order learning outcomes. Social cognitive theory incorporates beliefs and attitudes related to ‘self-efficacy’ into learning objectives and emphasises its driving role in cognitive behaviour. Cognitive Load Theory posits that students’ ability to produce innovative results is an important indicator of the impact of STEM education on student learning outcomes. This multidimensional approach aims to capture the multifaceted impact of STEM education.
Conflicting Findings: A Need for Meta-Analysis
The impact of STEM education on student learning outcomes has been a subject of debate. Some studies suggest a positive impact, with STEM education enhancing science and maths achievement. Others confirm that STEM education only enhances some learning abilities. However, some studies comprehensively reject the impact of STEM education on student learning outcomes. The lack of a consistent conclusion underscores the need for a more rigorous and objective assessment.
Meta-analysis, a quantitative research method that analyses the results of multiple experiments on the same topic, offers a solution. It synthesises existing studies, providing a more accurate and objective assessment of their corresponding metrics. Unlike traditional literature reviews, meta-analysis focuses on comparing the results of different studies and providing an overall effect size through the same criteria. Meta-analysis is able to synthesise the commonalities between individual studies with inconsistent findings on the same research topic, and ultimately develop consistent, generalised, and more precise findings by integrating individual studies.
Methodology: A Comprehensive Approach
This study adopts a broad definition of learning outcomes to avoid publication bias caused by indicator limitations. Integrating multi-dimensional indicators better reflects the overall effect of the intervention. This study references Linnenbrink-Garcia et al.’s (2010) “three-dimensional learning outcomes model,” defining learning outcomes as a combination of cognitive acquisition, motivational development, and skill transfer. Based on this, a meta-analysis was conducted on 66 experimental and quasi-experimental studies on the effects of STEM education on student learning outcomes. The study further explores how student learning outcomes are affected by five moderating variables: sample size, academic level, subjects, experimental period and teaching method. These variables were identified as moderators in previous studies and formed the basis for the use of these moderating variables in this study.
Read also: International Journal of Science and Mathematics Education: An overview.
Defining Learning Outcomes
The learning outcomes in this study are not a single concept but follow the classic framework of ‘cognitive-affective-motor skills’ in educational measurement (Bloom et al., 1956), divided into cognitive ability, non-cognitive ability, and skill performance. Cognitive ability include academic performance and knowledge retention rates, which are measured using standardised tools. Non-cognitive ability encompass psychological traits such as self-efficacy and learning interest. Skill performance includes observable behavioural manifestations such as problem-solving ability and collaborative ability.
Literature Search and Selection
In this study, the keywords of STEM education and learning outcomes were used to conduct the literature search in the scientific databases Scopus, Web of Science, and Google Scholar in strict accordance with the guidelines of PRISMA, which has detailed process standards that can be applied to most other literature review types of studies. The keywords for STEM education are ‘STEM’, ‘STEM Education’, ‘STEM Teaching’, ‘STEM Learning’. The keywords for learning outcomes are ‘Learning Result’, ‘Study Outcomes’, ‘Learning Effect’, ‘Study Performance’, ‘Learning Effect’ and ‘Study Performance’. Use search formulas to combine keywords from each category with the Boolean operator OR, and then further connect each combination of keywords with the operator AND. For example, (STEM OR STEM Education OR …) AND (Learning Outcomes OR Learning Results OR …). A search of journal literature from the above mentioned scientific databases for the period 2000-May 2024 was conducted, and the selected literature were all from international refereed journals listed in SSCI, AHCI or SCI, and a total of 2,568 literature were found to meet the requirements, of which 568 literature were from Scopus, 763 literature from Web of Science, and 1,237 literature from Google Scholar.
The 2,568 literature data searched were imported into Endnote, and 795 literature were left after deleting duplicates, and 795 literature were left after completing the initial screening. Then the literature was double screened by two experts in related fields based on topics and abstracts, and Cohen’s Kappa coefficients were 0.89, 0.88, 0.91, and 0.89, respectively, which were all greater than 0.8 (almost perfect agreement), which proves that the screening results have good reliability, and 795 literature were screened to obtain 59 literature. The study applied the snowball method, which tracks citations forward and backward on top of this literature to find other relevant literature. Seven literatures were added using the ‘snowball’ method, resulting in 66 literatures as shown in Figure 1. The snowball method was chosen because it involves references cited in the selected literature. This approach not only benefits from checking the initial list of references only, but also complements it by checking the references cited in the literature. PRISMA process for literature screening.
Coding of Moderator Variables
The impact of STEM education on student learning outcomes may be influenced by moderating variables such as sample size, academic level, subjects, experimental period and teaching method. This study is based on the meta-analysis framework proposed by Cooper et al. (2019) and employs a multi-dimensional hierarchical coding model to categorise moderator variables into three layers: contextual layer (sample size, academic level), intervention layer (subjects, experimental period, teaching method), and outcome layer (learning outcomes).
In the sample size category, class sizes are categorised as small (1-50), medium (51-100), and large (>100) based on the recommendations proposed by Chingos (2013). In the academic level category, participants are categorised as primary school, secondary school, and university students based on Piaget’s stages of cognitive development (Diamond, 2013). In the subjects category, according to the framework of the National Research Council (2014), STEM corresponds to science (research focused on natural sciences), technology (research involving digital tools or robotics), engineering (research centred on design or manufacturing), and mathematics (research emphasising quantitative reasoning). In the teaching method category, based on the three-dimensional teaching model of STEM education, and corresponding to the knowledge integration model proposed by Bransford et al. (2013), it is divided into problem-oriented, project-oriented, and inquiry-oriented. Therefore, in addition to coding the overall learning outcomes and groups, it is also necessary to code the above moderator variables.
Read also: Engaging Science Education
Two researchers with a major in education were selected for this study to content analyse 66 pieces of literature. First, training was provided on the meaning of the coding system and the coding methodology. Then, 10 randomly selected literature were pre-coded and inconsistent codes were explained and analysed so that a consistent understanding of the coding system could be reached. Finally, two researchers independently coded all the literature. After coding, the coding results of the two researchers were checked again and the reliability of the codes was calculated. For the inconsistent coding, the results of the discussion between the two researchers were selected. In this study, Cohen’s Kappa coefficient was used to calculate the consistency of the coding results, and the consistency coefficient was 0.88, which indicated that the coding results had good reliability.
Data Analysis
In order to comprehensively explore the impact of STEM education on student learning outcomes, this study followed the analytical steps of Cooper et al. (2019) and used the Comprehensive Meta-Analysis (CMA) Version 3 software developed by Biostat to process and deeply profile the data. The meta-analysis study mainly used the fixed effects model and random effects model proposed by Borenstein et al. (2021). This study found that the relationship between STEM education and learning outcomes may be influenced by complex factors such as sample size, academic level, subjects, experimental period and teaching method. In this study, the effect size calculation method proposed by Cohen (1988) was used as the combined effect value to assess the extent of the impact of STEM education on students’ learning outcomes.
Key Findings
The study reveals several important insights:
- Overall Effect: Overall, STEM education has a moderate effect on students’ learning outcomes, but the overall moderate effect size masks these key differences.
- Moderating Variables: Subgroup analysis showed that STEM education had the most significant impact on cognitive outcomes in high school (d = 0.58) and reduced heterogeneity (I2 = 62.1%), while the overall effect size was exploratory due to construct diversity.
These findings highlight the need to tailor STEM interventions to outcome type and academic level, strengthening the integration of theory and practice in STEM education.
Addressing Publication Bias
Bias refers to the deviation of a study’s results or inferred values from their true values. In the field of social science research, research reporting bias is prevalent, and the test for publication bias is indispensable because only when the degree of publication bias is correctly evaluated can its impact on the results of meta-analysis be minimised. Commonly used testing methods include the funnel plot method, Egger’s test, Begg’s test and loss of safety factor.
Read also: Planning Education Research
The funnel plot is characterised as being more intuitive, allowing the researcher to visually determine whether there is bias in the findings. The funnel plot centres on effect size (x-axis) and uses standard error (y-axis) as a measure of precision, illustrating the distribution characteristics of 66 studies. Ideally, the precision of effect size estimates improves with increasing sample size. Studies with small sample sizes, which have larger standard errors, are distributed at the bottom of the plot, while studies with large sample sizes are concentrated at the top, forming a symmetrical funnel shape. If publication bias exists, the funnel plot will exhibit asymmetry. As shown in Figure 2, all effect points are distributed within the 95% confidence interval and are symmetrically distributed around the pooled effect value (0.5). The ratio of points on the left side (effect size < 0.5) to those on the right side (effect size > 0.5) is approximately 1:1, no obvious clustering or absence of points on either side is observed. The range of effect sizes in the small-sample studies at the bottom (−0.3 to 1.2) does not exceed the expected random error interval, and no clustering of points on the right side due to ‘positive result bias’ is observed. Funnel scatter graphic.
In this study, publication bias was also calculated using the Classic fail-safe N proposed by Rosenthal (1979), which assesses how many published studies are necessary for the total effect size of published studies to reach the level of non-significance. The measure is 5n + 10 (n refers to the number of papers included in the meta-analysis), and if the fail-safe N is much larger th# Unveiling the Impact of STEM Education on Student Learning Outcomes: A Meta-Analysis
Introduction
STEM education, an innovative educational approach emphasizing Science, Technology, Engineering, and Mathematics, has garnered significant attention from both industry and academia since its inception in 1986. The core objective of STEM education is to foster students' innovation, creativity, and practical skills. While traditional education focused on cognitive domains like memory and understanding, modern educational theories have broadened their scope to include affective and psychomotor domains. This article delves into the impact of STEM education on student learning outcomes, exploring the nuances and complexities revealed through a comprehensive meta-analysis.
The Multifaceted Nature of Learning Outcomes in STEM Education
Modern educational theory emphasizes higher-order learning outcomes, encompassing not only knowledge mastery but also creativity, problem-solving, self-efficacy, and innovation. Constructivism, for instance, places "creation" at the pinnacle of cognitive achievement, underscoring the importance of problem-solving and creativity. Social cognitive theory integrates beliefs and attitudes related to self-efficacy, recognizing its crucial role in driving cognitive behavior. Cognitive Load Theory highlights the ability to produce innovative results as a key indicator of STEM education's effectiveness.
This multidimensional perspective on learning outcomes necessitates rigorous experimentation to validate the impact of STEM education, moving beyond subjective judgments or anecdotal experiences. The learning outcomes in this study are aligned with the classic framework of ‘cognitive-affective-motor skills’ in educational measurement, divided into cognitive ability, non-cognitive ability, and skill performance. Cognitive abilities include academic performance and knowledge retention rates, which are measured using standardized tools. Non-cognitive abilities encompass psychological traits such as self-efficacy and learning interest. Skill performance includes observable behavioral manifestations such as problem-solving ability and collaborative ability.
Contradictory Findings: A Need for Meta-Analysis
Existing research presents conflicting views on the impact of STEM education on student learning outcomes. Some studies suggest a positive influence on science and mathematics achievement, academic performance, programming skills, physics knowledge, and self-efficacy in modeling. Other studies indicate that STEM education enhances specific skills like learning interest and hands-on abilities, without significantly affecting creativity. Conversely, some research finds no statistically significant difference in academic performance between students engaged in T-STEM education and those in non-T-STEM education.
These inconsistent findings underscore the need for a more systematic and objective approach to evaluate the overall effect of STEM education. Meta-analysis, a quantitative research method that synthesizes the results of multiple studies on the same topic, offers a powerful tool for achieving this goal. By comparing the results of different studies and providing an overall effect size, meta-analysis can reveal consistent, generalized, and precise findings.
Methodology: A Rigorous Approach to Data Synthesis
This study adopted a meta-analysis approach to analyze 66 experimental and quasi-experimental studies on the effects of STEM education on student learning outcomes. The research aimed to explore the overall impact of STEM education and to identify moderating variables that influence this impact. The study followed the analytical steps of Cooper et al. (2019) and used the Comprehensive Meta-Analysis (CMA) Version 3 software developed by Biostat to process and deeply profile the data. The meta-analysis study mainly used the fixed effects model and random effects model proposed by Borenstein et al. (2021).
Literature Search and Selection
A comprehensive literature search was conducted in Scopus, Web of Science, and Google Scholar using keywords related to STEM education and learning outcomes, adhering to PRISMA guidelines. The search covered the period from 2000 to May 2024, focusing on international refereed journals listed in SSCI, AHCI, or SCI. After removing duplicates and screening based on topics and abstracts, 59 articles were selected. The "snowball" method was then applied to identify additional relevant literature, resulting in a final sample of 66 studies.
Coding of Moderator Variables
The impact of STEM education on student learning outcomes can be influenced by several moderating variables, including sample size, academic level, subjects, experimental period, and teaching method. This study categorized these variables into three layers: the contextual layer (sample size, academic level), the intervention layer (subjects, experimental period, teaching method), and the outcome layer (learning outcomes).
- Sample Size: Class sizes were categorized as small (1-50), medium (51-100), and large (>100).
- Academic Level: Participants were categorized as primary school, secondary school, and university students.
- Subjects: STEM subjects were categorized as science, technology, engineering, and mathematics.
- Teaching Method: Teaching methods were categorized as problem-oriented, project-oriented, and inquiry-oriented.
Two researchers with expertise in education independently coded the literature, and the consistency of the coding results was assessed using Cohen’s Kappa coefficient, which indicated good reliability.
Key Findings: Unveiling the Nuances of STEM Education's Impact
The meta-analysis revealed several important findings regarding the impact of STEM education on student learning outcomes.
Overall Effect
Overall, STEM education has a moderate effect on students’ learning outcomes, but the overall moderate effect size masks these key differences.
Impact on Cognitive Outcomes
Subgroup analysis showed that STEM education had the most significant impact on cognitive outcomes in high school (d = 0.58) and reduced heterogeneity (I2 = 62.1%), while the overall effect size was exploratory due to construct diversity.
These findings highlight the need to tailor STEM interventions to outcome type and academic level, strengthening the integration of theory and practice in STEM education.
Addressing Publication Bias
Publication bias, the tendency for studies with statistically significant results to be more likely to be published, can potentially distort the findings of a meta-analysis. To assess publication bias, this study employed several methods, including funnel plots and the Classic fail-safe N.
The funnel plot visually examined the distribution of effect sizes across the 66 studies. The symmetrical distribution of points around the pooled effect value suggested no significant publication bias. Furthermore, the Classic fail-safe N indicated that a substantial number of unpublished studies would be required to negate the overall effect of STEM education, further supporting the robustness of the findings.
Implications and Future Directions
The findings of this meta-analysis have important implications for educators, policymakers, and researchers. The results underscore the potential benefits of STEM education in enhancing student learning outcomes, particularly cognitive abilities. However, the study also highlights the need for carefully designed and targeted STEM interventions that consider the specific academic level and subject area.
Future research should focus on further exploring the moderating effects of various factors, such as teacher training, curriculum design, and instructional strategies. Additionally, longitudinal studies are needed to examine the long-term impact of STEM education on student success in college and careers.
tags: #journal #of #stem #education #overview
