Why Do Students Attend STEM Clubs, What Do They Get Out of It, and Where Are They Heading?
Round 1
Reviewer 1 Report
Introduction: The introduction as it is presented is very informative and uses literature appropriately to support claims. However, having read the entire article, the introduction does not flow well with the findings and discussion sections. For coherence, literature focused on intrinsic, utility, attainment, and cost values that impact STEM programs. Consider incorporating text around Ladson-Billings- education debt rather than achievement gap and relying on standardized test scores for marginalized students. In reference to identities (before figure 1), consider Nasir- negotiating identities in regarding identity shifts.
Good rationale for research and literature support throughout for claims made. However, there was not a clear path to the research questions. Culturally relevant (Ladson-Billings, 1999) and sustaining (Paris & Alim, 2014) pedagogies was not well cited or incorporated into the justification for teaching strategies at the STEM camps. Additionally, the mentioning of the pedagogical frame was not used to guide the research questions, data analysis, or findings. More coherent coherents connections of the introduction, methods, and results are needed. For example, motivation is used throughout the paper but research is not well connected with the discussion and implications. Another example is training and resources of club leaders for culturally sustaining and pedagogies is mentioned but not well connected to the research questions or findings and is therefore not contributing strongly to the overall manuscript. Overall, more literature around intersectional and marginalized communities in STEM in the introduction would strengthen the findings significantly.
Research Questions: As a mixed methods study, the research questions did not lend themselves to qualitative or quantitative data. Along those lines, there were many small research questions without 1-2 overarching, broad questions that can be answered through mixed methods.
Materials and Methods: Very clear and detailed research design. Lines 168-169 mention demographics of the schools differed and how those demographics related to one another is important to this research. Epstein's Framework (line 186) is not clearly connected to the introduction, research design, or findings. Add a justification into the methods about the use of this framework. Very detailed and thorough analysis with the exploratory and confirmation analysis and survey instrument validation.
Results: Reiterating the research questions alongside the data that addresses the research question would aid in clarity and detail of the presentation of the findings as well as ensure all research questions are answered adequately. Intersectional, racialized, and gendered experiences would be appropriate for analysis in this research. Additionally, demographics of the instructors would be clarifying for data connected to teacher background and experience teaching. The section on "STEM club involvement" and determining that data was unclear and the importance of the data point was also unclear.
Formatting: The article is very well organized with clear headings and appropriate formatting overall. Lines 47-49 and 50-52 read as repetitive language and information. The first paragraph after "3.Results" seems to be text from a template/placeholder. Consider a different representation for figure 3 as it is difficult to interpret the participants with STEM and non-STEM future career goals.
Author Response
Responses to reviewer 1
Thank you for your helpful feedback on the manuscript. All of our additions and major changes to our resubmitted manuscript are in red font.
Introduction: The introduction as it is presented is very informative and uses literature appropriately to support claims. However, having read the entire article, the introduction does not flow well with the findings and discussion sections. For coherence, literature focused on intrinsic, utility, attainment, and cost values that impact STEM programs. Consider incorporating text around Ladson-Billings- education debt rather than achievement gap and relying on standardized test scores for marginalized students. In reference to identities (before figure 1), consider Nasir- negotiating identities in regarding identity shifts.
We have added more detail on the SEVT aspects and added text related to Ladson-Billings and Nasir. In section 1.2 Cultural Responsiveness and 1.5 Theoretical Framework.
Good rationale for research and literature support throughout for claims made. However, there was not a clear path to the research questions. Culturally relevant (Ladson-Billings, 1999) and sustaining (Paris & Alim, 2014) pedagogies was not well cited or incorporated into the justification for teaching strategies at the STEM camps. Additionally, the mentioning of the pedagogical frame was not used to guide the research questions, data analysis, or findings. More coherent coherents connections of the introduction, methods, and results are needed. For example, motivation is used throughout the paper but research is not well connected with the discussion and implications. Another example is training and resources of club leaders for culturally sustaining and pedagogies is mentioned but not well connected to the research questions or findings and is therefore not contributing strongly to the overall manuscript. Overall, more literature around intersectional and marginalized communities in STEM in the introduction would strengthen the findings significantly.
We addressed your concerns by greatly modifying and reorganizing the literature review and making better connect the motivational aspects of SEVT throughout the paper, including expanding the explanation of SEVT constructs and in the revised organization and discussion section and adding additional references. Also, under 2.1 Research Design we revised our methods use.
Research Questions: As a mixed methods study, the research questions did not lend themselves to qualitative or quantitative data. Along those lines, there were many small research questions without 1-2 overarching, broad questions that can be answered through mixed methods.
We addressed this concern by adding an overarching research question and slightly revising the wording of some of the research questions, including reducing the RQs to 3, and adding parts a, b, and c under Research question 1 (see 1.7 Research Questions).
Materials and Methods: Very clear and detailed research design. Lines 168-169 mention demographics of the schools differed and how those demographics related to one another is important to this research. Epstein's Framework (line 186) is not clearly connected to the introduction, research design, or findings. Add a justification into the methods about the use of this framework. Very detailed and thorough analysis with the exploratory and confirmation analysis and survey instrument validation.
We have added a section in the literature review on Epstein and the focus on connecting school to home in section 1.1. We also added a description of the development of the STEM Club survey in section 2.4.2.
Results: Reiterating the research questions alongside the data that addresses the research question would aid in clarity and detail of the presentation of the findings as well as ensure all research questions are answered adequately. Intersectional, racialized, and gendered experiences would be appropriate for analysis in this research. Additionally, demographics of the instructors would be clarifying for data connected to teacher background and experience teaching. The section on "STEM club involvement" and determining that data was unclear and the importance of the data point was also unclear.
We clearly restated the research questions at the beginning of the findings sections (in red font). We also added a section on the teacher coaches from the 4 middle schools who led the STEM clubs in section 2.3.2. Under Table 4, the text was modified to better explain the purpose of the analyses and clarity on how they were determined.
Formatting: The article is very well organized with clear headings and appropriate formatting overall. Lines 47-49 and 50-52 read as repetitive language and information. The first paragraph after "3.Results" seems to be text from a template/placeholder. Consider a different representation for figure 3 as it is difficult to interpret the participants with STEM and non-STEM future career goals.
We have removed the repeated language and modified Figure 3 to include totals for the non-STEM and STEM career interest, to make the differences clearer.
Author Response File: 
Author Response.docx
Reviewer 2 Report
Thank you for allowing me to review the paper "Why do students attend STEM clubs, what do they get out of it, and where are heading?": A three-year, mixed-methods study.
This manuscript covers an interesting topic. The aim is to study the motivation for choosing to attend STEM clubs. Very impressive perspective from the afterschool STEM clubs.
The theory is well-reviewed and the framework is clear. But, I have several concerns with the methodological aspect, which I detail below:
- I would ask the authors to clarify the methodological part instead. Why did the authors use a mixed method? What are the potentials? The limitations I would probably put in the final part of the paper.
- The STEM Club Survey should be clarified and detailed. Which instrument(s) were used? Provide examples of items for each scale. Provide the range of the response scale.
- The authors state that an open-response item was included at the end of the survey that asked students what they thought was the purpose of the STEM Clubs. Is it possible to quote the question formulated?
- The authors propose the validation of an instrument from the STEM Club Survey. What instrument is this? Is it a translation of another instrument? Is it an ad-hoc created instrument? How were the items created? From a data analysis point of view, the authors compute EFA and CFA with the same sample. They could have computed an EFA with a random subsample of the participants (~50%) and a CFA with the remaining part of the sample. Please see Anderson, J. C., & Gerbing, D. W. (1988). Structural Equation Modeling in Practice: A Review and Recommended Two-Step Approach. Psychological Bulletin, 103(3), 411-423. https://doi.org/10.1037/0033-2909.103.3.411
- The results of the CFA should be reported in a table (with the respective betas onto the latent factors).
- The qualitative results are interesting. I would ask the authors to clarify the practical implications of the study in terms of educational policy.
Author Response
This is a response to Reviewer 2
Thank you for your review of this article to help us improve the manuscript. All of the responses are in red font, below your comments.
Comments and Suggestions for Authors
Thank you for allowing me to review the paper "Why do students attend STEM clubs, what do they get out of it, and where are heading?": A three-year, mixed-methods study.
We appreciate your time and efforts!
This manuscript covers an interesting topic. The aim is to study the motivation for choosing to attend STEM clubs. Very impressive perspective from the afterschool STEM clubs.
Thank you.
The theory is well-reviewed and the framework is clear. But, I have several concerns with the methodological aspect, which I detail below:
- I would ask the authors to clarify the methodological part instead. Why did the authors use a mixed method? What are the potentials? The limitations I would probably put in the final part of the paper.
We have added text to explain our methods in several ways. We added an overarching research question and grouped 3 of the questions under a revised research question 1 in section 1.7. Under section 2.1 we added more about the choice of the mixed-methods design. We moved the limitations to the end of the paper.
- The STEM Club Survey should be clarified and detailed. Which instrument(s) were used? Provide examples of items for each scale. Provide the range of the response scale.
We have a added a lot of information about the development of the items and the validation of the scale in section 2.4.2 STEM Club Survey. We also added a new Table 1 that gives factor loadings for the items with CFA.
- The authors state that an open-response item was included at the end of the survey that asked students what they thought was the purpose of the STEM Clubs. Is it possible to quote the question formulated?
We included the exact wording of the question in section 2.4.2 , “What do you believe is the actual purpose of the STEM Club/ Program?”
- The authors propose the validation of an instrument from the STEM Club Survey. What instrument is this? Is it a translation of another instrument? Is it an ad-hoc created instrument? How were the items created? From a data analysis point of view, the authors compute EFA and CFA with the same sample. They could have computed an EFA with a random subsample of the participants (~50%) and a CFA with the remaining part of the sample. Please see Anderson, J. C., & Gerbing, D. W. (1988). Structural Equation Modeling in Practice: A Review and Recommended Two-Step Approach. Psychological Bulletin, 103(3), 411-423. https://doi.org/10.1037/0033-2909.103.3.411
We have a added a lot of information about the development of the items and the validation of the scale in section 2.4.2 STEM Club Survey. We also added a new Table 1 that gives factor loadings for the items with CFA.
- The results of the CFA should be reported in a table (with the respective betas onto the latent factors).
This has been done in the new Table 1.
- The qualitative results are interesting. I would ask the authors to clarify the practical implications of the study in terms of educational policy.
We have added a paragraph in the recommendations, 5.1 about policy implications.
Round 2
Reviewer 1 Report
Overall, the additions to the paper incorporated the feedback from the from the 1st review. The introduction presents a coherent argument supported in literature with context. The additions to this section are strong!The addition of the paragraph before the "1.2 cultural responsiveness" section is good but a transition and ending to the section is needed. The paragraph To further support the argument and purpose of the paper, define/operationalize equity pedagogies. What is it and why is it necessary What is inequitable? A critical lens here would greatly support overall framing of the paper and the findings as well. The research questions were edited appropriately. Good change to the expanded mixed method design sentences. Table 1 and explaining paragraphs strengthen the overall research design. Good additions to the results sections with the research question. Overall strong addition to the discussion and conclusion.
Overall coherence modifications needed. Points are expanded below.:
Restructuring the 1.3 Future Goals section as part of the implications/discussion sections would support the overall flow of the paper.
Connecting the focus of the study to the previous literature presented would support the argument's flow
Representation is mentioned in section 1.5 but not expanded on. Why and how is representation important? How does cultural matching (Eddy, C. M., & Easton-Brooks, D. (2011). Ethnic matching, school placement, and mathematics achievement of African American students from kindergarten through fifth grade. Urban Education, 46(6), 1280-1299.) impact motivation? Consider expanding this section.
A critical lens applied to the subjective task values would greatly strengthen the arguments presented. How do the values shift for historically marginalized and minoritized youth? Additionally, is there literature that relates rural and urban youth motivation?
Along those lines, the section about the demographics of students and teacher coaches is not intersectional in regards to representation and cultural relevance. Are there more Black men represented, Asian women, etc. How do the demographics of the instructors compare to the students in various grade levels and each year? In what ways do the coaches impact motivation and values? Perhaps a note for future studies.
pg 8- MFA and CFA are not defined at first mention. The abbreviations are expanded later and need to be moved up further.
Figure 1 could be condensed to only be of the subjective task value since the rest of the figure is not expanded on or the focus of this study.
In section 4 (pg 20), female is used as a noun and should reference female participants or students.
pg 21 has a placeholder for a Eccles citation and needs to be updated with the correct reference number.
Author Response
Author's Reply to the Review Report (Reviewer 1)
Please provide a point-by-point response to the reviewer’s comments and either enter it in the box below or upload it as a Word/PDF file. Please write down "Please see the attachment." in the box if you only upload an attachment. An example can be found here.
Thank you for these comments. Please see below for our responses to the recommendations of Reviewer 1.
* Author's Notes to Reviewer
Review Report Form
Open Review
( ) I would not like to sign my review report
(x) I would like to sign my review report
Quality of English Language
( ) English very difficult to understand/incomprehensible
( ) Extensive editing of English language and style required
( ) Moderate English changes required
(x) English language and style are fine/minor spell check required
( ) I am not qualified to assess the quality of English in this paper
|
|
Yes |
Can be improved |
Must be improved |
Not applicable |
|
Is the content succinctly described and contextualized with respect to previous and present theoretical background and empirical research (if applicable) on the topic? |
( ) |
(x) |
( ) |
( ) |
|
Are all the cited references relevant to the research? |
(x) |
( ) |
( ) |
( ) |
|
Are the research design, questions, hypotheses and methods clearly stated? |
(x) |
( ) |
( ) |
( ) |
|
Are the arguments and discussion of findings coherent, balanced and compelling? |
( ) |
(x) |
( ) |
( ) |
|
For empirical research, are the results clearly presented? |
(x) |
( ) |
( ) |
( ) |
|
Is the article adequately referenced? |
(x) |
( ) |
( ) |
( ) |
|
Are the conclusions thoroughly supported by the results presented in the article or referenced in secondary literature? |
(x) |
( ) |
( ) |
( ) |
Comments and Suggestions for Authors
Overall, the additions to the paper incorporated the feedback from the from the 1st review. The introduction presents a coherent argument supported in literature with context. The additions to this section are strong! The addition of the paragraph before the "1.2 cultural responsiveness" section is good but a transition and ending to the section is needed. The paragraph To further support the argument and purpose of the paper, define/operationalize equity pedagogies. What is it and why is it necessary What is inequitable? A critical lens here would greatly support overall framing of the paper and the findings as well. The research questions were edited appropriately. Good change to the expanded mixed method design sentences. Table 1 and explaining paragraphs strengthen the overall research design. Good additions to the results sections with the research question. Overall strong addition to the discussion and conclusion.
Thank you for your careful read of our revised manuscript. We feel that following your suggestions has strengthened our paper. All of our most recent additions in the manuscript are in red.
Overall coherence modifications needed. Points are expanded below.:
Restructuring the 1.3 Future Goals section as part of the implications/discussion sections would support the overall flow of the paper.
We took your advice and added some of the references in this section to connect to our discussion in section 4. Thank you for this suggestion.
We were not sure if you were suggesting eliminating this section in the literature review, but decided that was not what you meant. Because we have research questions related to future goals, and STEM career explorations and STEM professionals were included in virtually every STEM club, we felt we needed a section that explicitly addressed this aspect of the project.
Connecting the focus of the study to the previous literature presented would support the argument's flow
In section 1.4 we have added a summary statement that takes into account the salient aspects of the literature review (which we modified based on your feedback, below).
Representation is mentioned in section 1.5 but not expanded on. Why and how is representation important? How does cultural matching (Eddy, C. M., & Easton-Brooks, D. (2011). Ethnic matching, school placement, and mathematics achievement of African American students from kindergarten through fifth grade. Urban Education, 46(6), 1280-1299.) impact motivation? Consider expanding this section.
This feedback led to quite a few changes in the literature review and some additions connecting to those in 1.5. We changed the header of 1.2 to be Equity Pedagogies and inserted information about cultural assets, and a whole paragraph focused on role models who are similar in race and gender, with new citations (including Eddy & Easton-Brooks, 2011).
A critical lens applied to the subjective task values would greatly strengthen the arguments presented. How do the values shift for historically marginalized and minoritized youth? Additionally, is there literature that relates rural and urban youth motivation?
We addressed this suggestion by making several changes. In the intro paragraph in section 1.5, we added information about the use of SEVT as a qualitative tool to capture or operationalize the views of marginalized or minoritized youth. We also cited from some Author publications that have used this approach with similar students. In addition, some examples from previous work with these populations were inserted in a couple of places to make stronger connections to how these values might be expressed by similar students (section 1.5.1 & 1.5.2). The few motivation papers that we found were quantitative in nature and mostly tied to representation in role models and focused on urban settings, which we included in section 1.2.
Along those lines, the section about the demographics of students and teacher coaches is not intersectional in regards to representation and cultural relevance. Are there more Black men represented, Asian women, etc. How do the demographics of the instructors compare to the students in various grade levels and each year? In what ways do the coaches impact motivation and values? Perhaps a note for future studies.
We addressed this suggestion in several ways. In section 2.2, we added more information explaining how the students would interact with other students and teacher coaches during many aspects of the club meetings and our purposeful recruitment of STEM professional speakers who were from underrepresented minority groups and ideally, from rural communities. In addition, we added a new Table 1 in a new section 2.3.3 that compares the demographics of the teacher coaches and students at the four middle schools, overall. The teacher coaches were fairly stable and the racial and gender percentages of the students were also similar from year to year at each school. We added a description of our perceptions of likely ethnic matching of the students to some of their teacher coaches. We also explained that our STEM professionals and the career videos students watched were recruited/selected based on providing models of underrepresented groups in STEM.
We added a note in the recommendations about collecting demographic information on students and club leaders, and providing relevant role models.
pg 8- MFA and CFA are not defined at first mention. The abbreviations are expanded later and need to be moved up further.
This change has been made.
Figure 1 could be condensed to only be of the subjective task value since the rest of the figure is not expanded on or the focus of this study.
In this study we analyzed data on identity, subjective task values, and goals, as well as choice to attend the STEM clubs. So we don’t feel we should remove those from the framework. We did not explicitly address expectation of success, because it was mostly tied into the other aspects that students expressed. We included a slightly revised model for Figure 1.
In section 4 (pg 20), female is used as a noun and should reference female participants or students.
This has been corrected.
pg 21 has a placeholder for a Eccles citation and needs to be updated with the correct reference number.
This has been corrected.
Submission Date
30 January 2023
Date of this review
01 Apr 2023 21:12:56
Reviewer 2 Report
Thanks to the authors for taking my comments into account when revising the paper. In my opinion the paper is ready for publication.
Author Response
Thank you! We submitted quite a few changes to address recommendations from reviewer 1. They are in red in the revised manuscript.
