Measuring and Activating iSTEM Key Principles among Student Teachers in STEM

Abstract

Graduates with a STEM profile are in great demand, yet the outflow from these fields of study is highly insufficient. This is partly due to the fragmented way STEM learning content is taught in secondary education. Although the problem can be mitigated with the use of integrated STEM education (i.e., iSTEM), teachers are often unfamiliar with this type of education. To support teachers in implementing high-quality iSTEM education, a digital collaborative learning environment called “CODEM for iSTEM” was created. This study examined to what extent student teachers were immersed in six key principles of iSTEM education through cooperative design of iSTEM learning tools in multidisciplinary teams, namely “problem-centered learning”, “integration of different STEM disciplines”, “modeling”, “inquiry-based learning”, “design-based learning”, and “cooperative learning”.

1. Introduction

In order to face the environmental, economic, and societal challenges in our globalized economy, the need for more science, technology, engineering, and mathematics (STEM) professionals is widely recognized among (inter)national organizations, governments, companies, and actors in the educational field [1,2,3]. Recently, the call to educate STEM-literate citizens who are able to understand and function in our information and communication society that relies more and more on technology is becoming louder [4]. Multiple sources [5,6,7,8] have proposed a definition of STEM literacy, but they all share the common idea that STEM literacy enables an individual to integrate content or skills from separate STEM disciplines to tackle problems in their everyday or professional lives even if the individual does not pursue a STEM study or career [4].

Integrated STEM (iSTEM) education has been proposed as a vehicle to increase both STEM literacy and STEM specialization for elementary, middle, and high school students. iSTEM refers to an educational approach in which boundaries between traditional scientific, technological, and mathematical school subjects are removed. The level of integration can vary from multidisciplinarity (in which skills and contents are learned separately for each discipline but are related to a common theme) to interdisciplinarity (in which students learn concepts and skills from two or more disciplines that are closely related) to transdisciplinarity (in which real-world problems are solved by applying concepts and skills from two or more STEM disciplines) [9].

A systematic literature review provided a framework for instructional practices in integrated STEM secondary education containing five key principles [10,11]: (1) the integration of content and skills between the different STEM disciplines (i.e., INT); (2) problem-centered learning by posing a real-world challenge that is motivating and engaging (i.e., PCL); inquiry-based learning by questioning, examining, gathering information, and interpreting results (i.e., IBL); design-based learning by using a technological or engineering design (i.e., DBL); and cooperative learning through collaboration among the team members (i.e., COO). Another key principle that was underexposed in the systematic literature review is modeling (i.e., MOD), which refers to the use of a scientific model in order to better understand a phenomenon. It is an important part of the STEM framework of the Flemish ministry of education in Belgium [12].

Previous research has shown that integrated STEM education has the potential to increase pupils’ learning outcomes [2,3,13] as well as their interests in and motivations for STEM (study) careers [14,15,16].

Despite iSTEM’s apparent benefits, designing qualitative iSTEM projects and implementing these in the classroom are not at all straightforward for high-school teachers [2,8,10,17]. Among other reasons, this is because most them have had training in only one or two STEM disciplines [18], and class periods are typically organized separately for each STEM subject. Shernoff et al. [19] interviewed 22 teachers and four administrators to identify challenges and needs for support to aid iSTEM implementation. The teachers indicated that they had difficulties envisioning what teaching iSTEM looks like and that they needed to experience good examples of iSTEM education from the perspective of a student. Concerning pre-service teacher programs, the teachers expressed the need for coursework on learning standards in all STEM subjects and STEM pedagogical practices such as cooperative (i.e., the iSTEM key principle COO) and project-based learning (i.e., PCL). Furthermore, in-service professional development (PD) should focus on having teachers themselves experience problem solving (i.e., PCL) or the engineering design process (i.e., DBL) first-hand. Using classroom implementation data and interviews, Dare et al. [17] identified three common challenges faced by teachers: (1) integration of STEM learning content (i.e., INT); (2) an apparent dichotomy between incorporating engineering design and science content (i.e., IBL and DBL); and (3) providing a realistic and authentic yet feasible design challenge to elicit and maintain student engagement and motivation (i.e., PCL).

Pre-service teacher training and in-service continued PD should prepare (prospective) teachers to tackle these challenges. Research has shown that characteristics of effective PD are: (1) a focus on subject matter content and how students learned that content; (2) a focus on pedagogical knowledge; (3) coherence of PD learning objectives with government and school policy, research evidence, and teachers’ own knowledge and beliefs; (4) accommodation ofteachers’ needs and interests (ownership); (5) the use of active and inquiry-based teacher learning methods; (6) the use of cooperative or collaborative teacher learning methods; (7) extended and intensive activities; (8) applicability in the daily teaching context; and (9) trainer knowledge and skills [20,21]. The iSTEM key principles relate to several of these characteristics as shown in

Table 1. Preliminary analysis of relation between characteristics of effective professional development and iSTEM key principles.

iSTEM Key Principle → Effective PD Characteristic ↓	INT	PCL	IBL	DBL	COO
Content-focused	x
Coherence	x		x	x
Active and inquiry-based learning		x	x	x
Cooperative learning					x

, which suggests that PD focused on letting teachers gain active experience with the iSTEM key principles inherently incorporates several characteristics of effective PD. This is interesting because it is known that to prepare teachers to implement new instructional principles, it is a good practice to immerse them in these instructional principles themselves [22].

In Flanders, which is the Dutch-speaking region of Belgium, a large research project called STEM@School investigated the effectiveness of iSTEM implementation on students’ learning outcomes and their interests and motivations regarding STEM between 2014 and 2018 [2,16]. In the scope of this project, iSTEM learning materials were developed [1,23], as well as an iSTEM design methodology called “CODEM for iSTEM” (i.e., Collaborative Online Design of Educational Materials for integrated STEM) [24]. Since 2019, KU Leuven has opted to include a mandatory course on iSTEM project design and a corresponding internship that are based on the “CODEM for iSTEM” methodology in its “Master of Teaching in Science and Technology” program. During this course, student teachers (i.e., pre-service teachers) are grouped in multidisciplinary teams, in which they cooperatively design iSTEM learning units that correspond to the iSTEM key principles. An online environment guides the student groups through subsequent evidence-based design phases. The approach using the “CODEM for iSTEM” methodology (provided via the online platform) and the organization in multidisciplinary teams were chosen for the following five reasons. First, through the design of iSTEM packages in multidisciplinary teams, student teachers themselves should experience all iSTEM key principles [22] and the approach incorporates many characteristics of effective professional development [1,20,21,25], including: (i) INT/content-focused: student teachers experience integration, and the task of designing an iSTEM learning unit adhering to the iSTEM key principles is inherently content-focused; (ii) PCL and DBL/active learning: student teachers experience problem-centered and design-based learning because their challenge is to design an iSTEM learning unit that adheres to the iSTEM key principles; (iii) IBL/evidence-based learning: by incorporating findings from the relevant (educational) literature, student teachers learn based on inquiry; and (iv) COO/cooperative learning: as student teachers design the iSTEM materials in multidisciplinary teams, they are required to collaborate. Second, experiencing these principles can boost the teachers’ self-confidence regarding the implementation of the principles in the classroom [26]. Third, an integrated, multidisciplinary approach with more hands-on experience could improve the ability to teach an integrated STEM course [19]. Fourth, involving teachers in the design process of the curriculum is also beneficial to the realization of that curriculum in the classroom [27]. Finally, the iSTEM design using the online platform can be seen as an authentic learning experience that supports reflective learning and allows teachers to gain comfort in using digital tools [28].

In order to assess the effectiveness of this teacher training course, this study aimed to investigate two research questions:

• “To what extent does the digital collaborative learning environment immerse student teachers in six key principles of iSTEM education?” (i.e., RQ1);
• “How does the activation of these key principles progress throughout the development process?” (i.e., RQ2).

We hypothesized that the learning environment would sufficiently activate the iSTEM key principles in student teachers but that integration of learning contents would require more support than the other five principles because it is the key principle that student teachers would have the least experience with and thus the lowest starting position. We further hypothesized that student teachers would show increasing activation of the key principles during the development process due to the accumulation of iSTEM experience. However, the growth pattern of the key principles might show some deviation from a perfect linear trajectory because the development process comprises different phases, and certain phases might require or result in a stronger activation of a specific key principle.

In the following sections, first the materials and methods will be described (Section 2), starting with an introduction of the course on interdisciplinary education, continuing with a detailed description of the “CODEM for iSTEM” methodology, further situating this research in a commonly used PD evaluation framework, and ending with a description of the methods and measures used for the PD evaluation. In Section 3, the developed measurement instrument as well as the results are described. Finally, in Section 4, the selected research approach and results are discussed in relation to the relevant literature.

2. Materials and Methods

2.1. KU Leuven Course on “Pedagogies of Interdisciplinary STEM Education”

In Flanders, two teacher education programs exist: one at the bachelor’s level (180 ECTS), which prepares students immediately after high school to teach lower and middle secondary education; and one at the master’s level (120 ECTS), which prepares students to teach middle and higher secondary education. Student teachers enter the Master of Teaching in Science and Technology program either after completing a domain-specific (e.g., science, mathematics, engineering, etc.) bachelor’s degree or a domain-specific master’s degree. Students that already hold a master’s degree can follow a shorter track of 60 ECTS. Due to the promising yet demanding nature of iSTEM education and in an effort to optimally prepare prospective teachers, the Master of Teaching in Science and Technology at KU Leuven contains two mandatory courses on interdisciplinary education: (1) “Pedagogies of Interdisciplinary Education”; and (2) “Internship in Interdisciplinary Education”. Each student who envisions teaching a subject either in science (biology, chemistry, earth sciences, or physics), mathematics, or technology (engineering, ICT, etc.) is automatically enrolled in the interdisciplinary education courses as well.

In the interdisciplinary education courses, student teachers design and implement an iSTEM learning unit in multidisciplinary teams. For the 2021–2022 academic year, exactly 100 students who were studying at nine different campuses across Flanders subscribed to the iSTEM design course. Although all of the students held at least an academic bachelor’s degree in mathematics, science, or engineering, it was a very diverse group that consisted of both full-time and part-time students who were combining the master’s program with a teaching job, a non-teaching job, or a family. These students were divided over 19 multidisciplinary teams based on their geographical location and chosen subjects. From these teams, a random sample of eight teams was selected for observation. Each observed team consisted of four to five members.

The multidisciplinary teams gathered at least once a week for two hours over the course of approximately 10 weeks between the second half of October and the end of December 2021. These weekly team meetings were recorded. The final goal of each team was to develop an iSTEM learning unit with the backbone written as a “script” containing a central, authentic, real-world challenge for pupils to solve; intended learning objectives; and learning activities. Each team had a team coach who was an experienced teacher that clarified the design process if necessary, aided the team with practical questions concerning the design or internship, provided formative feedback, resolved conflicts if they occurred, and eventually performed the summative evaluation of the iSTEM learning unit and internship implementation. In additon to the coach, an online learning environment guided the teams through the design process based on the “CODEM for iSTEM” methodology [24], which is described in the next subsection.

2.2. “CODEM for iSTEM” Methodology

To support (student) teachers in implementing high-quality iSTEM education, a digital collaborative learning environment called “CODEM for iSTEM” was created [24]. The “CODEM for iSTEM” methodology was developed based on a multiple-case study that identified crucial, counterproductive, and missing steps in the design process of four multidisciplinary iSTEM design teams. The iSTEM design process consists of five phases with each consisting of one or more stages (see [24] for detailed information):

• Context-analysis phase: identification of the target group for which the iSTEM learning unit will be developed, the target groups’ prior knowledge, and scanning of (Flemish) curriculum guidelines to select learning contents that could be integrated.
• Theme-selection phase: discussion of possible themes comprising the selected learning contents from the different STEM curricula.
• Content/challenge brainstorm phase:

  a.

  Defining the learning objectives for each theme;

  b.

  Identifying competencies to be linked + linking;

  c.

  Discipline-specific educational literature review;

  d.

  Definition of central challenge;

  e.

  Division of central challenge into subproblems;

  f.

  Study of requirements and feasibility to solve challenges;

  g.

  Selection and formulation of concrete learning objectives;

  h.

  Design of leaning activities and instructional strategies.

• Reporting phase: presentation of preliminary script and materials and exchange of feedback among peer design teams.
• Development phase: finalization of script and construction of student syllabus.

Phases 1–3 of the “CODEM for iSTEM” methodology were integrated in an online collaborative learning environment to support student teachers in their design process and for use in a blended course setup due to the multicampus model. The online learning environment was centered around flowcharts representing the five phases and their stages. Each flowchart block provided textual information for each stage (key issues to consider, evidence-based good practices and pitfalls, and scaffolding questions), and reflective questions to assess the design at critical points (Figure 1).

This study aimed to investigate the effect of the “CODEM for iSTEM” methodology provided via the online learning environment on prospective teachers’ preparedness for iSTEM classroom implementation. The next subsections situate the approach in a conceptual framework for professional development evaluation and discuss the measures used.

2.3. Evaluation of the Effectiveness of the “CODEM for iSTEM” Methodology Provided via an Online Learning Environment

Desimone proposed a conceptual framework for studying the effects of professional development on teachers and students [21] that consists of four components: (1) the core features of professional development, which (could) result in (2) a change in teachers’ knowledge, skills, attitudes, and beliefs that (could) elicit (3) a change in instruction, eventually leading to (4) improved student learning.

This study aimed to evaluate the effectiveness of the “CODEM for iSTEM” methodology provided via an online learning environment by focusing on the first two components of Desimone’s framework and with a specific interest in the active experience of iSTEM key principles by student teachers.

Merchie et al. provided an overview of measurement methods that can be used to assess the components of Desimone’s [21] and Merchie’s extended framework [25]. In this study, a qualitative measure was selected to provide answers to the research questions as recommended by Desimone [21]. Qualitative research methods provided detailed insights into the complex interactions that took place in the multidisciplinary design teams. More specifically, the video recordings of the eight randomly selected iSTEM teams’ meetings were analyzed using a scoring rubric (i.e., CiSTEM²-TTR). Informed consent to use these video data was obtained from all of the student teachers who participated in the study (Institutional Review Board approval number: G-2021-3888-R2). The intensity of the data gathering was limited because the iSTEM design teams usually met online due to the blended nature of the “Pedagogies of Interdisciplinary Education” course. Still, the scoring and analysis of these recordings was very time-intensive. However, no other measures such as interviews or self-report questionnaires could provide similar detailed, fine-grained, and time-specific answers to the research questions.

2.4. Scoring Rubric for iSTEM Key Principles

Given the novelty of the concept under investigation, validated evaluation instruments to assess the activation of the iSTEM key principles during the iSTEM design process were unlikely to be available. To the best of our knowledge, no such instrument exists in the scientific literature. Consequently, a scoring instrument for iSTEM’s key principles had to be created, which led to the development of the CiSTEM²-Teacher Training Rubric (i.e., CiSTEM²-TTR) (see Appendix A) as part of the CiSTEM² project (i.e., Cooperative interdisciplinary Student Teacher Education Model for Coaching integrated STEM). A video-observation analysis generally requires the video to be partitioned in segments of equal lengths of time. However, the iSTEM design process as described in Section 2.2 contained different phases and stages that varied in duration between the teams. Moreover, the teams can collectively ignore the work for a phase or stage or perform an additional phase or stage iteration. Consequently, comparing fixed video segments of equal size between the teams was impossible. Therefore the CiSTEM²-TTR scored a stage in its entirety.

The construction of the CiSTEM²-TTR was the result of an multistep process that consisted of a literature review, expert refinement, and a pilot study. The systematic literature review by Thibaut et al. functioned as the backbone of the rubric [10]. The findings of the review; i.e., the iSTEM key principles, that made up the items of the rubric were supplemented with additional items that were suggested by experienced iSTEM experts as crucial to activate the iSTEM key principles. These experts were former and current iSTEM coaches of the “Pedagogies of Interdisciplinary Education” course at KU Leuven. Finally, we tested the rubric in a pilot study in which two observers scored a team during the first half of the development process. This pilot study resulted in additional adjustments that increased the inter-rater reliability of the rubric.

For each item, the rubric contains four scoring criteria: insufficient, sufficient, strong, and very strong. Achieving a higher scoring criterion always assumes that the criteria of the lower scores are also fulfilled. For example, the table in Appendix A shows that when a team achieves a “strong” score on the item “Explicitly formulating expected objectives/results” of the PCL key principle, this indicates that the team not only fulfill the criterion of the “strong” score (“Showed awareness of the underlying reason/usefulness why these results should be achieved”, but automatically also the criterion of the “sufficient” score (“Explicitly formulated which actions should be taken and what the result should be”). Logically, in order for a team to be aware of the results’ usefulness, it first needs to formulate what the results should be.

According to the literature review, the iSTEM key principle of PCL is seen as the conception of an authentic real-world challenge that is motivating and engaging [10]. The iSTEM experts regarded this interpretation of PCL to be somewhat narrow and suggested additional items, namely formulating expected objectives and results, identifying preconditions, splitting a problem into smaller (sub)problems, and determining priorities. Afterward, these items were tested in the pilot study. The item criteria for achieving sufficient vs. strong vs. very strong activation were based on expert opinions and later confirmed in the pilot study based on their frequency of occurrence. The initial main PCL item of posing an authentic real-world challenge showed absolutely no variation during the pilot study: all teams reached the maximum score due to the fact that the assignment they received explicitly demanded a real-world iSTEM challenge that was of interest to the target group before a team could continue to the next phase. Therefore, this item had to be removed from the CiSTEM²-TTR.

The iSTEM key principle of INT was operationalized by Thibaut et al. as the integration of content and skills over the different STEM disciplines. This was captured in the item “Achieving a high level of integration”. The scoring of this item was based on the aforementioned levels of integration, with sufficient INT activation requiring the linking of different disciplines to the central challenge but not directly to each other (i.e., multidisciplinarity). Strong scores were an indication of several links across disciplines while still allowing the use of one’s own discipline-specific terminology because the two disciplines were closely related (i.e., interdisciplinarity). Very strong scores were achieved after linking across disciplines while using concepts, principles, or analogies of different disciplines to solve the central challenge (i.e., transdisciplinarity). The pilot study showed that the search for integration was often performed individually instead of collaboratively. Consequently, an additional item was created to control for this behavior.

The iSTEM key principle of IBL is defined as learning by questioning, examining, gathering information, and interpreting results [10]. These constitute the components of scientific research and are captured by the item that examines the extensiveness of inquiry. The scoring criteria increase with each component of the inquiry process. Sufficient IBL activation is achieved by performing research when prior knowledge is lacking. Doing this, but also questioning the how and why of a (sub)phenomenon, would result in a strong score on this item. Additionally, very strong scores also require a team to reflect upon the inquiry process. Expert refinement resulted in the addition of a second item that took the quality of the examined sources into account. The pilot study showed that student teachers often reached for sources explicitly recommended by the learning environment such as feedback from the coach or governmental curriculum guidelines objectives. Nevertheless, these sources still needed to be processed correctly by the student teachers. As such, these type of sources constituted a sufficient level of IBL activation. Strong and very strong scores were achieved by referring to external sources found by the team members themselves (i.e., non-academic and academic sources, respectively).

The iSTEM key principle of DBL demands technological/engineering design. More specifically, the literature review first emphasized the importance of considering alternative solutions and justifying design choices. We combined these aspects into the item “Generating design ideas”. Student teachers were deemed to have achieved sufficient DBL activation when they generated sufficient design ideas, which adhered to the consideration of alternative solutions. Strong scores were achieved when the advantages and disadvantages of those different ideas were regularly articulated but without further ado. In contrast, very strong scores required the design choices to be based on the consideration of the advantages and disadvantages of different design ideas. Second, the literature review emphasized the iterative nature of the design process: the engineering problem needs to be defined and then a solution determined and ultimately tested and evaluated, after which an iteration follows. Expert refinement suggested to differentiate the conception of the engineering problem into (a) scientific/technical requirements (e.g., calculations, physical principles, and results of an inquiry process), (b) practical conditions (e.g., available time, available material/space, and safety), (c) the level and interest of the target group (e.g., cognitive ability, attitudes, and diversity), and (d) the (learning) objectives for the target group (e.g., self-formulated objectives, curricula, and targets). The scoring on these items followed a stepwise approach: sufficient levels merely demanded awareness of the requirements, conditions, level, interests, and objectives. Strong scores required not only awareness but also a solution. Moreover, very strong scores included the testing of solutions.

The literature review regards the iSTEM key principle of COO as cooperative learning through collaboration and interdependency among team members. This was captured by the item that investigated the collaboration intensity. Since this was an iSTEM rubric, a decision was made to tie the strong and very strong scores on this item to the integration between STEM learning contents or skills identified by team members from different disciplines. Expert refinement added three more items: using effective tools, participating actively, and providing feedback. Strong activation of these items required an efficient use of appointments/tools/methods to optimize the teamwork, active participation of all team members, and constructive peer feedback followed by constructive responses to that feedback, respectively. Very strong activation was achieved when teams reflected on their collaboration tools, all team members participated actively and to a large extent, and teams reflected on their entire collaboration process itself, respectively.

MOD was added to the five original iSTEM key principles and therefore only consisted of one item: the act of modeling itself. This is defined as using a scientific model to understand and communicate about a phenomenon [12]. A sufficient scoring on this item demanded the mapping of the relationships between the concepts, component, parameters, or variables playing a role in the challenge created by a team. Strong levels of activation required awareness of the model assumptions. Lastly, very strong scores demanded testing of the validity of the model at regular intervals so necessary adjustments could be made early.

No observer-specific bias was detected. We examined this by having a second researcher observe one of the eight teams. The inter-rater reliability was κ = 0.82, 95% CI [0.59, 1.06].

3. Results

Our findings showed that not all teams went through all the stages and phases of the platform collaboratively. As

Table 2. Phases of the design process and the number of teams that performed each phase.

Phase		#Teams
(P1)	Context-analysis phase (i.e., determining the target group)	8
(P2)	Theme-selection phase	8
	Defining the learning objectives for each theme	4
(P3)	Identification of competencies to be learned + linking	8
	Discipline-specific educational literature review	2
	Definition of central challenge	4
(P4)	Division of central challenge into subproblems	8
	Study of requirements and feasibility to solve challenge	1
	Iteration of central challenge and subproblems	3
	Second iteration of central challenge and subproblems (after feedback coach)	4
	Iteration of learning objectives	2
	Second iteration of learning objectives (after feedback coach)	1
(P5)	Design of learning activities and instructional strategies	8
	Iteration of learning activities and instructional strategies (after feedback coach)	4

illustrates, only five stages (from here on labeled P1 to P5) were completed by all the teams. The other stages and phases were mostly visited individually outside the team meetings and in some cases were even ignored. In addition, the online learning environment contained flowcharts and information texts that encouraged iterations of stages and phases in order to result in a better-designed learning unit. Some teams relied on the script as a guideline instead of the online learning environment. As a consequence, the information texts were not granted proper attention.

Although the activation of the iSTEM key principles varies between the teams, on average, the key principle of “integration of different STEM disciplines” seemed to be insufficiently activated among the student teachers, whereas the other key principles were sufficiently but not strongly activated during the training via the digital collaborative learning environment. These observations confirmed our first hypothesis.

Scoring via the CiSTEM²-TTR rubric indicated that the insufficient levels of integration seemed to be due to a great need for more activation in making in-depth, cross-disciplinary linkages and searching for such linkages collaboratively. Concerning IBL, it was established that student teachers mainly built on their prior knowledge. PCL did not achieve a strong level due to the lack of insight, reflection, and depth. The extreme lack of testing was the reason for the lower-than-expected DBL activation. Only one team did once test one of the conditions of the design. For MOD, the reason lay in remaining unaware of the assumptions or validity of a model. Finally, it was observed that cooperative learning did not reach a higher level due to the low “integration of different STEM disciplines”, which did not allow for building upon the knowledge of the team members. The lack of reflection and the fact that often at least one team member participated to a lesser degree also played a role in the scoring of COO.

Regarding the second research question (i.e., RQ2), the iSTEM key principle activation levels varied across the stages of the design process. PCL, DBL, IBL, and MOD showed a modest positive linear growth curve (see Figure 2). IBL and PCL deviated somewhat from a linear trajectory. IBL showed a deviation at the theme-selection stage (P2) and PCL during the context-analysis stage (P1) and the theme-selection stage (P2). During the theme-selection stage (P2) IBL was relatively highly activated because the teams needed to refresh the curriculum guidelines and research unknown aspects of potential themes. PCL was also important during the first two stages. The observed positive linear growth with some deviations due to certain stages demanding more activation of specific iSTEM key principles confirmed our second hypothesis and indicated that the online learning environment had the ability to activate these iSTEM key principles, albeit modestly. In contrast, INT did not exhibit positive growth, as it seemed to increase slightly when participating in the stage in which linking between STEM disciplines took centerstage and decreased afterward during the division of the central challenge into subproblems and the design of the learning activities and instructional strategies. The last key principle (COO) showed a flat trajectory without any growth.

4. Discussion

4.1. Summary and Interpretation of Results

To the best of our knowledge, this study was the first to investigate the time- and stage-dependent activation of inquiry- and design-based cooperative and problem-centered learning and modeling and integrative practices of student teachers during the process of collaboratively designing an iSTEM learning unit using an online learning environment. An intensive qualitative analysis of the weekly team meeting recordings was used to gain detailed and fine-grained insights into the interactions within the multidisciplinary iSTEM design teams as a first step to explore pre-service teacher competence development using the online learning environment based on the “CODEM for iSTEM” methodology. To this end, a scoring rubric was developed based on a literature review, expert input, and adjustments based on pilot study results.

The results indicated that the “CODEM for iSTEM” methodology and learning environment immerses the student teachers to a sufficient, though not large, extent in most iSTEM key principles, except for “integration of different STEM disciplines”; i.e., INT. The activation of the other iSTEM key principles was sufficiently present during training via the online clearning environment, but here too no strong scores were achieved. Based on low scores on certain items in the scoring rubric, working points were identified that were mentioned in the Results section. The lack of strong scores on the scoring rubric was in line with previous research that assessed general (albeit elementary in-service) teachers’ iSTEM competences after professional development during classroom observations. The assessment with a scoring rubric consisting of five domains and four levels, showed that teachers scored “approaching proficiency” (rubric level 2) for all five domains [29].

Our research findings suggest that improvements to the online learning environment are needed to boost student teachers’ activation of the iSTEM key principles to higher levels of proficiency. This is in line with previous research which examined teachers’ PD needs and ideas with respect to iSTEM educational design and implementation after professional development via self-report questionnaires [30,31]. Regarding the six iSTEM key principles [10], teachers explicitly mentioned needing support related to the principles of INT, DBL, and COO. Additionally, studies that investigated teachers’ changes in iSTEM education conceptions after PD initiatives reported on changed perceptions related to the iSTEM key principles, although it remains to be investigated whether the used approaches are also useful for developing pre-service teachers’ competences. After taking an iSTEM education course that allowed pre-service teachers to experience STEM education from a pupil’s perspective and critically reflect on this experience, they reported an improved perceived understanding of iSTEM education [18]. Radloff et al. also reported improved perceptions of iSTEM education after pre-service teachers’ video analysis and critical reflection on iSTEM instructional practices. Both before and after interviews, the pre-service teachers stressed the importance of “seamless” and purposeful integration of learning contents from several STEM disciplines (i.e., INT); working on “real-world scenarios” and hands-on applying of knowledge (i.e., PCL); failing, redesign, and the usage of the engineering design process (i.e., DBL); and student-centered approaches highlighted by group work in which the teacher acts as facilitator (i.e., COO) [32].

The literature related to the iSTEM key principles INT, DBL, and COO will be discussed in more detail. We focused on these three principles because INT exhibited the lowest scores, which also hampered COO activation due to the need for integrated collaboration. DBL in turn showed extremely low frequencies of very strong scores across all the teams, which begs the question whether certain research effects might be to blame.

4.1.1. Integration and Cooperative Learning

The iSTEM key principle of INT consistently showed average activation levels that were below sufficient throughout the design process. This observation is in line with the existing literature and seems to be caused by a great need for better support in making in-depth cross-disciplinary links. Berlin et al. [33] described a five-quarter teacher-preparation program with three courses that were focused on integrated content and three courses that were focused on integrated pedagogy followed by action research and examination. A quantitative analysis that used a semantic differential instrument to probe attitudes and perceptions related to the value and difficulty of iSTEM integration showed that pre-service teachers valued integration equally high before and after the intervention but perceived STEM integration as significantly more difficult after the intervention. After the intervention, pre-service teachers showed a more realistic, practical, and cautious approach to integration. This was confirmed by using a qualitative analysis of the answers to open-ended questions that probed what STEM integration meant to the participants. Complementary, Singer et al. observed that upon exposure to a curriculum bearing (albeit non-explicit) chances to build interdisciplinary links, student teachers did not make these connections on their own [34].

A practice that may support student teachers in building in-depth cross-disciplinary links is collaborative concept mapping. A concept map is a diagram that contains concepts in boxes and arrows that visualize the links between concepts and provide a textual explanation of how the concepts are linked [35]. Concept mapping is a promising technique for meaningful learning by identifying and tackling key learning concepts and actively constructing knowledge [36,37] as well as for authentic assessment [38]. Previous research into collaborative concept mapping showed promising results. Chen et al. reported significant increases in pre-service teachers’ self-efficacy after participation in a mandatory course with a design task to create a technology-integrated interdisciplinary thematic unit (not limited to STEM) for middle school students [39]. The qualitative data once more confirmed that student teachers found making connections challenging yet judged the concept-mapping exercise to be helpful in identifying connections. Collaborative concept mapping is said to be a “messy and challenging” non-linear process that requires communication, sharing ideas, and providing feedback [39]. Cavlazoglu et al. [36] compared the quality of concept maps constructed by teachers individually versus in a group before and after a STEM workshop. Prior to the workshop, no significant differences in quality between the individually and collaboratively constructed concept maps were found; however, after the workshop, the collaboratively constructed concept maps were of significantly higher quality than the individually constructed concept maps.

Concept mapping thus seems to be a promising tool that may enhance both the iSTEM key principles of INT and COO. However, pre-service teachers experienced difficulties during concept map construction: they found the process labor-intensive and time-consuming [40]. Research findings indicated that in our online learning environment, even more emphasis should be put on the importance and potential of collaborative concept mapping to achieve meaningful integration and on the support of student teacher design teams in the concept-mapping process.

Our findings showed a relatively stable sufficient-to-strong activation of the iSTEM key principle of COO throughout the stages of the development process. The flat trajectory of the COO principle could be explained by a high starting position compared to the other iSTEM key principles followed by low growth. The high starting position could be the result of collaboration and interdependency being a constant factor during the development process right from the start (the student teachers would receive a team score at the end of the Interdisciplinary STEM courses). Both growth in COO activation and very strong COO activation levels require purposeful collaborate integration, which was something no team has seemed able to accomplish.

4.1.2. Design-Based Learning

As teaching can be viewed as a design science with the teacher as the designer [41,42], the iSTEM design process is a complex and creative process of analysis, creation, prototyping, feedback gathering, and redesign [43] in which student teachers should activate the iSTEM key principle of DBL.

While an increase in the average DBL activation was observed, again the student teachers did not exceed strong activation levels. The rubric indicates that very strong DBL activation levels correspond with a testing of the design. Testing must be interpreted in the broadest sense by incorporating actual testing but also involving reflecting, gathering evidence, etc. Due to the practical organization of the training course “Pedagogies of Interdisciplinary STEM Education”, not all of the design aspects could be tested by the student teachers; e.g., because their target group was not yet known at the time of the challenge conception, making it impossible to assess the interest of the target group in the designed challenge. Furthermore, the rubric expected that student teachers would prototype and test their hands-on learning activities in the development phase of the “CODEM for iSTEM” methodology, which was out of the scope of the training course. Other research has shown that teachers who experience hands-on activities during professional development are encouraged to implement hands-on activities in their classroom practices [44].

Despite the explanations for the relatively low DBL activation levels, further DBL support in the online environment and course guidance may be warranted. Wu et al. [42] compared static (automatic) and adaptive (human-aided) scaffolding during an iSTEM collaborative design process with the assistance of an online platform by analyzing coded group chat data using an epistemic network analysis. Static scaffolding led to the development of routine expertise, mix-and-match strategies of formulating design solutions, and suppressed divergent thinking. Adaptive scaffolding; e.g., a human tutor pointing out incongruent views, led to revised solutions and deeper reflections. Group members also asked for clarification and confirmation of design solutions more often. Adaptive and static scaffolding thus play different roles and should complement each other. Wu et al. concluded “This kind of just-in-time support from a human tutor is critical, especially for novice designers”[42]. The value of coaching support to provide expert content and pedagogical knowledge was confirmed in other studies [45]. Our online learning environment contained static scaffolding by means of reflective questions in the flowchart and supporting questions included in the textual information for each substage. The team coach complemented this static scaffolding by providing demand-driven clarification and feedback (adaptive scaffolding). Whether this support was sufficient should be critically reviewed in further research.

4.2. Limitations and Future Work

4.2.1. Limitations

This study aimed to investigate the activation of iSTEM key principles during the design process of iSTEM multidisciplinary teams. Video recordings of weekly team meetings were scored using a newly created scoring rubric: the CiSTEM²-TTR. Although this rubric was developed based on a literature review, expert refinement, and a pilot study, its validity should be further investigated in other contexts because the literature base used to construct the rubric was limited.

Although approximately 200 h of video recordings were analyzed, the sample size was still limited to only eight teams. In the future, we intend to analyze the video recordings of the remaining 11 teams to expand the dataset and corroborate the research findings.

4.2.2. Future Work

Within the scope of the Erasmus+ project CiSTEM², additional data were collected from student teachers in the 2021–2022 academic year. In addition to recordings of the weekly team meetings, student teachers also filled in a questionnaire before and after the development process that probed their attitudes toward iSTEM education and they answered an open-ended question that asked what approach they would take in designing an iSTEM project concerning a specific theme. These data will provide insights into the learning gains and attitude shifts of student teachers after taking the courses “Pedagogies” and “Internship in Interdisciplinary STEMEducation”. The results should be triangulated with the qualitative data of the observed team meetings (this study). Furthermore, when video recordings of all teams have been scored using the CiSTEM²-TTR rubric, these scores and the metrics detailed above should be compared to the summative scores received by each team at the end of both courses to assess the relationship between a team’s process and the quality of their final product.

As part of the CiSTEM² project, the “CODEM for iSTEM” learning environment is currently being improved in line with findings of this study. The video observations resulted in multiple suggestions. Firstly, the student teachers often neglected to read the textual information. Therefore, instructional and explainer videos will be added for every substage in the flowcharts. Secondly, the activation of the iSTEM principle of INT was too low for all teams. Therefore, the principle of INT will receive a more central role in the optimized learning environment in the form of concept mapping, and an instructional video demonstrating the iterative process of concept mapping in an iSTEM context will be added. Student teachers’ difficulties regarding concept mapping will receive special attention by providing good and bad examples. Thirdly, for the other iSTEM key principles (MOD, PCL, IBL, DBL, and COO), merely sufficient activation levels were reached. Therefore, explainer videos in the new learning environment will place more emphasis on insight, reflection, research, assumptions, and testing. In each substage, the iSTEM key principles that should be activated will be explicitly mentioned in the explainer video. During the academic year 2022–2023, student teachers will use the new learning environment, while the same measures of iSTEM key principle activation (questionnaire, open-ended question, and video recordings of weekly meetings) will be collected in order to study the effects of using the new environment as a training tool.

5. Conclusions

Integrated STEM has the potential to increase students’ interest in STEM education. However, before students can be immersed in the key principles of iSTEM, their teachers first need to be trained in these key principles themselves. For this reason, the online collaborative learning environment “CODEM for iSTEM” was created. Flemish teams of student teachers designed learning materials via the support of the online learning environment. This study investigated to what extent the online learning environment activates student teachers throughout their design process in the six key principles of iSTEM education. Video recordings of the student teachers’ team meetings were observed and analyzed with the newly developed scoring rubric CiSTEM²-TTR. The results indicated that the online learning environment has immersed the student teachers to a sufficient (though not high) degree in the iSTEM key principles of “problem-centered learning”, “modeling”, “inquiry-based learning”, “design-based learning”, and “cooperative learning”. Except for “cooperative learning”, these principles showed modest growth throughout the student teachers’ use of the online learning environment. However, the activation of “integration of different STEM disciplines” remained insufficient. Based on these findings, improvements to the online learning environment are currently being implemented as part of the CiSTEM² project. The assessment instrument and method developed in this study provide new ways for future teacher training programs to analyze (student) teachers’ endeavors in designing integrated STEM education.