Reproducible Learning in Higher Education: Cultivating a Culture of Open Scholarship

The scientific community has increasingly emphasized the importance of research credibility, robustness, and reproducibility. This has led to a surge in interest in open and transparent research practices. However, there is a significant gap in how these principles are integrated into undergraduate and postgraduate research training. This article explores the integration of open scholarship principles into higher education to ensure long-term improvements in research quality and culture.

The Credibility Revolution and the Rise of Open Scholarship

Revelations about the lack of reproducibility in scientific research have led to a "credibility revolution." This has resulted in the scientific community taking steps to improve research practices. These steps include:

Higher standards of scientific evidence
Preregistration of studies
Registered reports
Direct replications
Transparency and openness regarding research materials
Research integrity

In addition, introductions, manifestos, and guidelines for conducting open and reproducible research have been proposed across various fields and disciplines. These reforms align with the following pillars for advancing social sciences:

Opening scientific communication
Restructuring incentives and practices
Collaborative and crowdsourced science

The Fourth Pillar: Integrating Open Scholarship into Higher Education

The integration of open scholarship principles into higher education is a crucial, often overlooked, pillar. Open scholarship emphasizes that knowledge of all kinds should be openly shared, transparent, rigorously researched, accumulative, and inclusive. Developing educational resources is essential to facilitate engagement with, adherence to, and learning of research transparency, replicability, openness, and reproducibility. Integrating open scholarship into teaching and mentoring is necessary for research integrity and to make the positive changes of recent reforms sustainable. This change should not be seen as an additional layer to existing proposals for reform. Instead, it should unite them to advance research transparency, reproducibility, rigor, and ethics through pedagogical reform.

FORRT: A Framework for Open and Reproducible Research Training

The Framework for Open and Reproducible Research Training (FORRT) is a community of over 350 scholars across various disciplines, career stages, and identities. FORRT aims to embed reproducibility and research integrity into higher education to ensure long-term improvements in research quality and culture.

Read also: Understanding PLCs

FORRT's Goals

FORRT has three main goals:

To build a pathway for educators toward the incremental adoption of open scholarship practices into higher education.
To generate a conversation about the ethics and social impact of a higher-education pedagogy that emphasizes openness, epistemic uncertainty, and research credibility.
To promote reflection about the perceived importance of different academic activities and advocate for greater recognition of educational resources.

FORRT's Achievements

The FORRT community has already developed pedagogical resources, including:

A bank of ready-to-use lesson plans and activities
A consensus-based glossary of over 250 open scholarship terms
A systematic review of the impact of open scholarship on students’ outcomes
A pedagogically-oriented database of replications and reversals
A collection of over 200 summaries of open and reproducible research literature, as well as diversity, equity, accessibility, and inclusion works
Over 700 curated FAIR searchable resources
A comprehensive but straightforward, evolving, and accessible didactic framework to learn, teach, and mentor open scholarship

FORRT's Vision

FORRT envisions advancing research transparency, reproducibility, rigor, and ethics through pedagogical reform and meta-scientific research. FORRT provides a pedagogical infrastructure and didactic resources designed to recognize and support the teaching and mentoring of open and reproducible science. FORRT strives to raise awareness of the pedagogical implications of open and reproducible science and its associated challenges and advocates for the opening and formal recognition of teaching and mentoring materials.

Open Educational Resources and Metascientific Works

FORRT creates, collects, and catalogs examples of successful pedagogical approaches in teaching and mentoring, as well as the openness of these resources. Interested parties can use FORRT’s pedagogies as a template toward the creation of their own teaching strategies. FORRT pedagogies strive to support teachers and mentors to integrate field-specific education with open and reproducible science tenets.

Educational Nexus

The educational NEXUS is an e-learning platform for educators to find, access, interoperate, and reuse teaching and mentoring resources and materials. The pedagogical content of all resources is maximized to facilitate adaptation and reuse. FORRT strongly advocates for these resources being used, cited, and recognized.

Read also: Learning Resources Near You

Publications, Manuscripts, and Policy

FORRT has published meta-scientific works and has manuscripts advocating and explaining why open and reproducible tenets should be incorporated in the teaching and mentoring of prototypical subject matters. In line with principles enshrined in FORRT, all manuscripts and ongoing projects are open contributorship projects. The work of all contributors are valued and formally recognized, hoping to expand the scope and diversity of FORRT’s mission and its team.

Evidence for Embedding Open and Reproducible Scholarship

Recent literature has explored the need for open scholarship in teaching and learning contexts. A critical review of the literature investigates how integrating open and reproducible science may influence student outcomes.

Key Findings

The review highlighted how embedding open and reproducible scholarship appears to be associated with:

Students' scientific literacies (i.e., students’ understanding of open research, consumption of science, and the development of transferable skills)
Student engagement (i.e., motivation and engagement with learning, collaboration, and engagement in open research)
Students' attitudes toward science (i.e., trust in science and confidence in research findings)

However, the review also identified a need for more robust and rigorous methods within pedagogical research, including more interventional and experimental evaluations of teaching practice.

Defining Open and Reproducible Scholarship

Open and reproducible scholarship is defined as a set of practices that endeavor to make scientific research, knowledge, and empirical data widely accessible, rigorous, and available to professionals and citizens. Scientific progress becomes feasible, and the scientific process becomes more rigorous, transparent, and reproducible.

Read also: Learning Civil Procedure

Methodology of the Review

The review synthesized the empirical evidence that investigates whether open scholarship influences students' scientific literacy, engagement, and attitudes toward science. It considered a diverse range of practices, such as teaching students about replicability, open scholarship, preregistration, and other approaches, and generally implementing such topics in the curriculum. It also reviewed the impact of allowing students to experience firsthand open scholarship through, for example, using open resources, open data, and hands-on research experience.

Team Science Approach

This project used a large, interdisciplinary Team Science approach to conduct the review and synthesize the evidence. Diverse perspectives were included due to contributors’ varied academic and cultural backgrounds. Contributors stemmed from academic institutions in 17 countries on four continents and represented a wide range of disciplines.

Eleven Strategies for Making Reproducible Research and Open Science Training the Norm

Reproducible research and open science practices have the potential to accelerate scientific progress by allowing others to reuse research outputs and by promoting rigorous research that is more likely to yield trustworthy results. However, these practices are uncommon in many fields, so there is a clear need for training that helps and encourages researchers to integrate reproducible research and open science practices into their daily work. Eleven strategies for making training in these practices the norm at research institutions are outlined below.

Adapting Research Assessment Criteria and Program Requirements

Integrate reproducible research and open science practices into required curriculum: Required courses reach more students than elective courses; hence, integrating courses into a required curriculum is an important step toward making reproducibility and open science training the norm.
Require reproducible research and open science practices in theses: Degree programs may require reproducible research and open science practices in undergraduate or graduate theses.
Incorporate reproducible and open science practices in hiring and evaluation processes: Traditional assessment criteria for hiring and evaluation of individual researchers still focus on third-party funding and the number of publications. A growing number of coalitions and initiatives are underway to reform the way we assess research(ers).

Training

Offer standalone training sessions on reproducible research and open science practices: This was the most common activity that event participants engaged in. Formats included single lectures, webinar series, workshops, summer schools, and courses.
Integrate reproducible research and open science skills into existing courses: Even when reproducible and open research skills are not part of the official curricula, instructors who teach required courses on other topics can integrate reproducible research and open science skills.
Provide hands-on training: Traditional courses and workshops often cover many practices in a short time; hence, participants need to decide which practices to implement, and how to implement them, after returning to their research group. In contrast, participants in hands-on courses implement practices in their own research during training.
Train entire research teams together: Implementing reproducible research and open science practices often requires collaboration among members of a research team.
Incorporate meta-research or replication studies into courses: Rather than teaching reproducible research or open science skills that researchers can use in their project, this approach trains participants to conduct meta-research (science of science) or replication studies. As the class collaborates on one project, participants also build skills for collaborative team science and gain experience leading small teams.

Building Communities

Organize regular meetings to discuss reproducible research and open science practices:
Establish local networks:
Connect with others working on related strategies:

Institutional Support

Institutions can support those working on the eleven strategies by allocating resources and monitoring impact.

The Reproducibility Challenge at the University of Michigan

Ensuring that data science research results can be reliably reproduced is particularly challenging. Universities play a critical role in supporting, incentivizing, and regulating research at their institutions, and are crucial in ensuring responsible conduct of research, implementing safety protocols, managing conflicts of interest, and so on. However, universities have not taken systematic approaches to do so.

Defining Reproducibility

A 2019 report from the National Academies of Sciences, Engineering, and Medicine (NASEM) defined reproducibility to mean computational reproducibility - obtaining consistent computational results using the same input data, computational steps, methods, and code, and conditions of analysis; and replicability to mean obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data.

In practice, the boundary between reproducibility and replicability is often blurred. Researchers’ effort to make their findings reproducible is often part of the larger effort to make their research findings more robust and reliable.

The Interdisciplinary Nature of Data Science

Reproducing a data science project often includes both specific considerations within each discipline and general considerations across disciplines. Researchers’ efforts to create tools and processes are often limited to reproducing results for their own projects, with little consideration for whether these tools can be adopted by others outside their immediate research field, or even by others in the same field.

Gigantum: A Data Science Platform for Reproducibility and Sharing

Gigantum captures the entire state of a data-science experiment and records a complete history of all changes to code, data, and environment. It is an automation tool that integrates Docker for software configuration, Git for versioning and storage, and JupyterLab or RStudio as a user environment for data science and machine learning.

Benefits of Using Gigantum

Allows instructors and TAs to build programming projects that include example source code, input data, and a software container
Students can start programming without installing software, launching instances on the cloud, or building virtual machines
Lectures can be presented as interactive Jupyter notebooks
Students can reproduce all previous work exactly
Automated versioning manages changing content

tags: #reproducible #learning #in #higher #education