Design and Validation of a Classroom Observation Instrument to Evaluate the Quality of Mathematical Activity from a Gender Perspective

Round 1

Reviewer 1 Report

This paper presents the design and validation of a classroom observational tool to measure quality of math activity from a gender perspective. The details the development of this instrument and reports the psychometric statistics related to interrater, reliability, and other dimensions of the instrument. While the paper is generally, very strong, there are some key elements that could enhance the effectiveness of this manuscript for publication.

Below are my specific comments on each area of the paper. 

Introduction:

Within the introduction, the authors should clarify their definition of interactions. Within the literature, there are many dimensions of interactions, and within the first two paragraphs, it is unclear the specific types of interactions. The authors are referencing. More clarity would provide a stronger introduction for the constructs associated with this study.

Theoretical Framework:

In general, the theoretical framework is well written, well organized, and provides a wide array of references across an international perspective. In terms of strengthening this area, authors should look at the most recent publications in this area of research from the past five years. Though the citations they provide are strong and quality, some are significantly outdated at this point. Providing additional citations from the last five years with strengthen the theoretical framework significantly.

Method 

In the first section of the method portion of the paper, the authors describe the development procedures for this instrument. In the section, the authors reference that the instrument was developed based on observations of videos of more than 100 classes, observed and videotaped at multiple centers. This section could be strengthened by providing additional detail on how these observations evolved into the current constructs represented in the instrument. For example, was there a qualitative coding method used to identify salient constructs? The constructs themselves are presented in table 2, but the method to arrive at these indicators is not well articulated. In addition, the authors claim that the instrument is also grounded in significant theory. This paper with benefit from a table in the methods section that summarizes the theoretical connections to specific citations and theories for each of the constructs represented in the instrument. 

Results

The results section is a strength of this paper as the psychometric statistics are well presented and align with the constructs of the instrument. The statistics selected to study the content validity of the instrument are appropriate and well articulated.

Discussion 

In terms of the discussion and conclusion, I do believe it is an overstatement to claim that this study represents a validation of the proposed instrument. There are many steps that will have to occur to fully validate an instrument of this caliber. I believe the authors would be better, served to adjust their language and acknowledge that this piece is focusing only on one element of validation, content validity. In order to truly validate this instrument, the authors will have to collect data multiple times, and conduct measures of internal consistency, construct validity, etc. I believe it’s the authors just acknowledge the limitations of the current study. They can highlight the work they have done and situated in an ongoing project towards the future validation of this observational tool.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

 

I have read “Design and validation of a classroom observation instrument to evaluate the quality of mathematical activity from a gender perspective.” The authors have engaged in a thorough and thoughtful creation of an observation instrument, this manuscript reporting the development and validation of the instrument is not ready for publication.

First, while I agree that “very few results based on the observation evidence in the classroom, have examined and evaluated both analytically and critically the nature of interactions and mathematical activity from a gender perspective,” I felt the work of Shah, Reinholz and others on the EQUIP rubrics should be reviewed by the authors: 

Shah, N., Reinholz, D., Guzman, L., Bradfield, K., Beaudine, G., & Low, S. (2016). Equitable Participation in a Mathematics Classroom from a Quantitative Perspective. North American Chapter of the International Group for the Psychology of Mathematics Education.

Reinholz, D. L., & Shah, N. (2018). Equity analytics: A methodological approach for quantifying participation patterns in mathematics classroom discourse. Journal for Research in Mathematics Education, 49(2), 140-177.

Second, likely due to language translation, there are a number of confusing and awkwardly constructed sentences. If the manuscript is to be published in English, please do a thorough review of sentence construction. Perhaps related, several sentences did not seem connect or did not make sense For example:

Page 1: More connection is needed between these sentences. How is the “research” in sentence 2 and 3 related to the literature in sentence 1? Is there a connection between sentence 2 and 3? Currently, they seem very disconnected: “Literature regarding interactions is extensive and focuses particularly on observations in the classroom. Research reveals the fundamental role of interactions and how they impact students' learning processes [5, 6]. Interactions have become a primary dimension of research related to mathematics education [4].”

Page 3: This sentences needs to be clarified. Should the word “fewer” be replaced with “more”? “In this sense we could infer female students learning more mathematics is nothing more than a consequence of the fact that, for gender reasons, they have fewer opportunities to be involved substantively in mathematical study processes than they are experiencing”

Page 3: Paragraph lines 121-131: This paragraph is confusing. Some sentences have awkward construction (“Several key implicit ideas in this brief description of the process of building on learning mathematics” and “The context and the interconnections give meaning to the problem must be accurate and significant.”). Some sentences need to be clarified: (1) What do you mean by “allow them to be solved?” in “The third idea is that no one becomes a good solver and student of mathematical problems 127 without working steadily on the procedures that allow them to be solved.” (2) Do you mean project it to students who will learn later, or project it to mathematical topics the students will learn later, in “The last idea is related to the fact that teachers must have a relationship between built in 129 processes based on mathematical knowledge already learned by the students previously, 130 and project it to those who will learn later”?

Third, while the dimensions around gender and bias were well-described and well-designed, the observation tool did not seem to focus on mathematical discourse and high-quality math teaching as described in the manuscript. I wondered why the authors did not apply Dimension 2 and 3 to an existing instrument (such as the authors’ own instrument, MQI, etc.)? Using a tool already validated for high quality instruction, and adding the gender/bias dimensions, might be quite valuable. The strength of the authors’ work in the manuscript is Dimensions 2 and 3. Also, based on the descriptions, in Dimension 1, #6 and 7 seem to align better with Dimension 2.

Fourth, phase 1 and 2 of the methodology are under-described, particularly since this is the focus of the manuscript. In phase 1, what themes/dimensions/indicators were identified by the researchers? How were those themes narrowed down or selected to be indicators? What level of agreement was necessary between researchers? In phase 2, what stood out from using the tool on the videos? Were the dimensions able to be scored reliably within the development team?  What value did the tool add to previous analyses of the lessons?  

Finally, regarding phase 3, recent discussions of validity contend that the instrument should be valid in the specific context/study in which it being used. I wonder if the expert review should have been phase 2, and then researchers use the revised tool on classroom videos as phase 3.

I suggest that for a revision of this manuscript, the authors use results from the expert review to revise the tool and then rate videos previously scored for instructional quality, and see what that gender-perspective rubrics add to the field’s understanding of opportunities to learn mathematics.  

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

I applaud the authors for this work – measures are often created without care for considering the content validity with such depth. And the work that goes into creating measures and gathering evidence for many types of claims (content, reliability, generalizability, etc.) is often under-appreciated in the field.  Given the aim of promoting equity as well, this is important work.

I have a few suggestions for the authors. I will first explain a suggestion I have regarding the perspective on validity and validation. I will then describe specific questions or suggestions for particular sections of the manuscript.

My perspective on validity and validation may be U.S.-centric, in that the standards I am going to discuss are those developed by organizations in the United States. However, I feel that it is something for the authors to consider. The American Psychological Association, National Council on Measurement in Education, and American Educational Research Association have a particular position on validity that does not align with the way validation in framed in this paper.  The research standards these organizations have published (APA, NCME, AERA, 2014) state that measurement should focus on providing evidence for validity of claims based on the intended interpretation and use of an instrument. So rather than say that this research validates an instrument based on the construct of content validity, these standards would say the research provides evidence for a claim that the instrument is sufficient, coherent, relevant, and clear.  I hope you can see how this is quite a significant shift from validating an instrument by measuring content validity.  There is a considerable press in North America and perhaps beyond to approach the idea of validity from the perspective of Interpretation and Use Arguments (see the work of M. Kane; Carney et al., 2019). I have seen some manuscripts receive rejections for failing to address reviewer concerns regarding framing their validity evidence in terms of Interpretation and Use Arguments.    This is a suggestion-- given that the journal Education Sciences is an international, something to consider.

Introduction:

The structure of the introduction is very good.  My suggestions are

a)       to consider whether your analysis of sufficiency, coherence, relevance, and clarity actual answers the research questions posed on page 2 lines 72-77

b)      elaborate how the theoretical perspectives contributed to the content of the measure (I didn’t follow how the theoretical ideas are manifested in the instrument)

c)       ensure that the claims made in each section are supported by the material that comes before. For example, the sections end with phrases such as “Based on all these elements” (pg 2 line 71), “In this way,” (pg 3 line 132), but I did not follow the connection between the information provided in the previous paragraphs and the claim made in that summary paragraph.

Methodology:

Again, I think it is important work and I appreciate attention to the content of an instrument.

Suggestions:

a)       Please elaborate on the development process. I am interested in how you used inductive processes, and then paired that with a literature review. Please provide more detail about what you did. When and in what ways were the theoretical frames included? This would be valuable for other instrument developers.

b)      Please clarify how your analysis answers the research questions listed in the introduction.

c)       Can you clarify what the Kendall’s W offers that is not provided by the Fleiss Kappa?

Results

I appreciate the explanation of how the statistical information contributed to modifications to the instrument.

Suggestions:

a)       Why was the Fleiss’ Kappa and the Kendall’s W run after the modifications? Why is this an acceptable point in the process to assess agreement?

Discussion

My suggestions for the discussion are described previously in the section on validity.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

I have read the revision of this manuscript, and I commend the authors for the thorough ways in which they incorporated feedback. The manuscript read very well, and the authors resolved the earlier issues of disconnectedness.  The enhanced explanation of the argumentation process and how it was proceduralized in the validation of the observation instrument (p. 6-7) was also very valuable.  I also liked how the final paragraph of the conclusion tied back to this process as a way to situate the current work and frame future work. A few minor editing suggestions are below.

P.3, lines 107-109:  Do you mean “greater” here? It didn’t make sense to me that greater proficiency was the result of limited opportunities: “It can be deduced that the greater mathematical proficiency of female students is just the result of their limited opportunities for meaningful participation in mathematical learning processes, due to gender-related reasons.”

P.5, line 217: should be ‘that the authors used’: “The process is an adaptation that used the authors in a previous work (Authors 2013).”

P. 7, line 255: should “studying” be “studied”: “that has been studying are present”

p. 12: The first paragraph under the Discussion section needs to be revised. First, tis phrase does not make sense: “were adjusted the structure of the measurement instrument” (line 398). Next, it says that 17 items were revised, and there is the beginning of a numbered list, but not of 17 items. What are the numbered items? Also the numbering does not work within the text of the paragraph. Maybe make a numbered list? Or move to a table or figure?  Then the second paragraph repeats that “17 of 58 items of the instrument were adjusted to increase clarity and improve their writing.” Again, maybe a table of all 17 and in the text, a few examples of how they were revised?

p. 13 Conclusion: “and providing content related evidence” -- Not sure this phrase works in the first sentence (verb tense, should be ‘provide’), and then it is also repeated in the second and third paragraphs. In the last paragraph, “provide content related evidence,” is should be ‘providing’

 

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Thank you for the careful attention to suggestions and thoughtful revisions. I appreciate your time in conducting this research.

Author Response

Thank you for the careful attention to suggestions and thoughtful revisions. I appreciate your time in conducting this research.

Thank you so much.