Social and Emotional Learning and Early Literacy Skills: A Quasi-Experimental Study of RULER

Round 1

Reviewer 1 Report

Thank you for the opportunity to review the manuscript entitled, “Social and Emotional Learning and Early Literacy Skills: A Quasi-Experimental Study of RULER”. This manuscript uses a quasi-experimental approach to whether preschool level access to RULER was associated with growth in early literacy in racially and ethnically diverse preschool-aged children. The researchers found that children in preschools with access to RULER did in fact have more growth in Vocabulary /Oral language and Phonological Awareness than children in preschools without access to RULER.

This study does provide deeper insight into RULER’s relationship to literacy for young children, I have some concern about whether, as currently written, this study adds much to the literature that is already available. My comments are below:

1.     Introduction

a.     The introduction makes a case that universal SEL programming may impact not only social-emotional development but also literacy acquisition. The authors then go on to provide evidence of this from prior studies including studies of RULER. This leads me to question what this study is adding to the literature? For example, were the previous studies of older children? Did they focus broadly on literacy while this study looks at literacy skills separately?

b.     The hypothesis presented on page 3, line 109-110 is about teacher-child interactions. This seems to come out of nowhere since teacher-child interactions has not been a major part of the discussion up to this point in the paper.

c.     On page 5, line 218-219 the authors state that the effects of RULER on early literacy skills have not yet been explored. This seems in contract to earlier findings that were presented for RULER. If this is the case, please provide clarity in the RULER section that the results listed there are for RULER for older students and not for preschool-aged students to clarify this.

d.     Under section 1.4 (pg. 5), the authors propose conceptual links between 1) emotional intelligence and early literacy, 2) teacher-child interactions and early literacy skills, and 3) dialogic reading and other practices and early literacy skills. These links were not studied as part of the paper. The paper examined whether RULER was associated with increased literacy skills. Thus, I suggest removing this from the paper as it sets the stage for a mediation analysis that examines RULER’s effect on EI, teacher-child relationships, and dialogic reading and then these outcomes on early literacy skills.

2.     Methods

a.     The methods section was a strength in this paper. The use of propensity scores strengthens the authors use of secondary data that has its limitations. I am not an expert in this methodology, however, and so I suggest having a methodologist review this section for accuracy.

3.     Results/Discussion

a.     The discussion on children’s literacy growth in Vocabulary/Oral language and Phonological Awareness could be strengthened by discussing how literacy typically develops. Emergent readers often are at the stage of phonological awareness and phonemic awareness which may be why this was where the growth happened. Also, how does the fact that there may be English language learners and bilingual students affect the results?

Author Response

Thank you for your insightful comments and suggestions. Upon addressing your feedback, we feel our manuscript is clearer and stronger. Responses to your specific questions and concerns are below:

• What is this study adding to the literature?
  • Although we originally noted on page 2 that “The purpose of this study is to investigate whether RULER is associated with growth in early literacy,” we added the following to provide greater context about what has been studied previously and with whom: “Previous studies have investigated RULER’s impact on teacher-student interactions from kindergarten through grade 12 (Brackett et al., 2012; Castillo et al., 2013; Hagelskamp et al., 2013; Rivers et al., 2013) as well as 5^th and 6^th grade student English Language Arts grades, engagement, attention problems, learning problems, leadership, study skills, and social skills (Brackett et al., 2012; NoVo Foundation & Education First, 2018; Reyes et al., 2012). Grade 9 students in schools and classrooms using RULER have also shown improvements in emotion understanding and grade point averages, However, RULER’s impact on preschool-aged children’s early literacy has yet to be explored. The purpose of this study is to investigate whether RULER is associated with growth in early literacy.”
• What is the relevance of teacher-child interactions (e.g., the hypothesis on page 3)?
  • We made note early in the introduction on page 2 that RULER tools and practices are embedded into teacher-child interactions and routines. We also noted on page 2 that prior research has shown teacher-child interactions have been improved in schools using RULER.
• Contradictory evidence from previous studies about RULER and early literacy.
  • Our revision to page 2 addressed this confusion by clarifying what has been studied previously. Specifically, RULER has been shown to impact 5th and 6th grade student English Language Arts grades, but RULER’s impact on preschool-aged children’s early literacy has yet to be explored.
• The theoretical foundation and conceptual links mentioned in Section 1.4 result in confusion.
  • The reviewer is correct that the paper examined whether RULER was associated with increased literacy skills. We therefore removed mention in Section 1.4 of the conceptual links. Now, the section focuses on the one effect we did explore and now reads: “We argue that children attending preschools with access to RULER may score higher on assessments of early literacy skills than their peers in preschools without access to RULER. We anticipate a small- to moderate-sized effect based on previous research on RULER and other SEL programs (Blewitt et al., 2018; Brackett et al., 2012; Durlak et al., 2011; Hagelskamp, Brackett, Rivers, & Salovey, 2013; Reyes et al., 2012; Rivers, Brackett, et al., 2013).”
• Strength of methods.
  • Thank you for your comment about our use of methods to address limitations with secondary data.
• The discussion could be strengthened by adding in how literacy typically develops.
  • On page 15, we added that “Lonigan and colleagues (2000) found significant variation in young children’s phonological sensitivity (i.e., sensitivity and manipulation of oral language) despite it being highly stable in early elementary school and predictive of other early literacy components. The authors suggested that preschool-aged children may especially benefit from practices that formally and informally enhance phonological sensitivity, which is supported by meta-analyses showing that phonemic awareness and phonics interventions work well for young children (Bus & van Ijzendoorn, 1999; Ehri, Nunes, Stahl, et al., 2001). Given the variability in typically-developing children, RULER practices may support teachers in meeting children where they are as emergent readers in their phonological and phonemic awareness, in addition to their vocabulary and oral language skills (Lonigan et al., 2008).”
• How might English language learners and bilingual students affect the results?
  • Multilanguage learner status may have contributed to unobserved selection bias. We modified our Methods section on page 6 to say that “Home language, multilingual exposure and proficiency, immigration status, and other aspects of children’s language context and skills were also not collected by the district.”
  • We also updated our acknowledgement of these factors in our limitations by saying that “Myriad unobserved variables provide irrefutable alternative hypotheses in this study. Although we found that the Access to RULER and No Access to RULER groups were equivalent at baseline according to What Works Clearinghouse standards, threats to causal inference may be on selection bias due to observed and unobserved confounding variables (e.g., home language, multilingual exposure and proficiency, and immigration status). Complex associations were not possible in this study—it could be that children in RULER preschools differentiate in their initial emotional intelligence, which drove the differential growth we attributed to RULER (Curby et al., 2015; Rhoades, Warren, Domitrovich, & Greenberg, 2011). With inverse probability weighting, we are confident in the observed covariate-adjusted balance between the groups of preschools at baseline. However, despite our analysis that unobserved covariates (e.g., home language, multilingual exposure and proficiency, and immigration status) must be at least as strong in association with the PELI as the association with the version of the PELI used (i.e., > r = .43), it is still possible that our results were influenced by unobserved confounding variables resulting in unobserved selection bias.”

Reviewer 2 Report

The study's goal was to examine the role of a social and emotional learning program (RULER) in determining the developmental trajectories of emergent literacy skills in preschool children.  

This is an important study with the potential to expand our understanding of the role of SEL interventions. It is a well-written paper that fits in with the goals of Education Sciences.

The introduction is well written as are the linkages between SEL and literacy. I found it a bit complex to follow the Analytic Strategy section. I think that although the info is necessary, this section is too long and restricts the flow of the paper reading, thus I suggest streamlining this section. A lot of information can be reported as additional material, so the paper reading would benefit from it.  

I have one main concern about participants and controlled variables. I very much appreciated the use of covariates such as (age, gender and SES), I wondered if there was also information about the amount of cumulative L2 exposure. From what we read, the group of participants is very heterogeneous in terms of ethnic background. Are these children all born at the research location? Are these children bilingual? Has their exposure to the context language been controlled for?  

Emotions vary from culture to culture and from one speech community to another. Since this variation in emotions is expected from the flexible nature of emotional experiences and concepts in different language contexts, I would like the authors to provide, if available, information about children's language exposure and whether this may be related to the effects of the program on literacy development.

Overall, I think the work is well done, well-reported and worthy of publication in this journal 

Author Response

Thank you for your insightful comments and suggestions. Upon addressing your feedback, we feel our manuscript is clearer and stronger. Responses to your specific questions and concerns are below:

• The Analytic Strategy section is too long and restricts the flow of the paper.
  • We appreciation this comment and the suggestion to streamline this section. Many of the supplementary analyses and results are now in footnotes leaving the broader points in text. We feel this condenses the section to the most important information for the reader.
• Was there information about the amount of cumulative L2 exposure, including multilingual and immigration status?
  • We agree that children’s language context and skills may have contributed to unobserved selection bias. We added in the Methods section on page 6 that “Home language, multilingual exposure and proficiency, immigration status, and other aspects of children’s language context and skills were also not collected by the district.”
  • We also updated our acknowledgement of these factors in our limitations by saying that “Myriad unobserved variables provide irrefutable alternative hypotheses in this study. Although we found that the Access to RULER and No Access to RULER groups were equivalent at baseline according to What Works Clearinghouse standards, threats to causal inference may be on selection bias due to observed and unobserved confounding variables (e.g., home language, multilingual exposure and proficiency, and immigration status). Complex associations were not possible in this study—it could be that children in RULER preschools differentiate in their initial emotional intelligence, which drove the differential growth we attributed to RULER (Curby et al., 2015; Rhoades, Warren, Domitrovich, & Greenberg, 2011). With inverse probability weighting, we are confident in the observed covariate-adjusted balance between the groups of preschools at baseline. However, despite our analysis that unobserved covariates (e.g., home language, multilingual exposure and proficiency, and immigration status) must be at least as strong in association with the PELI as the association with the version of the PELI used (i.e., > r = .43), it is still possible that our results were influenced by unobserved confounding variables resulting in unobserved selection bias.”
• Emotions vary from culture to culture and from one speech community to another. Since this variation in emotions is expected from the flexible nature of emotional experiences and concepts in different language contexts, I would like the authors to provide, if available, information about children's language exposure and whether this may be related to the effects of the program on literacy development.
  • Although the study did not explore the factors that contribute to early literacy development (e.g., children’s language exposure), it is reasonable to assume that the teachers using RULER modified the language to which children were exposed as this is part of RULER’s theory of change. We noted in the discussion section on page 15 that “…findings are consistent with how RULER works and likely influences children’s early literacy skills through (1) enhancing educator skills and teacher-child interactions and (2) providing classroom curricula focused on expanding language, integrating and improving literacy instruction, and helping children better understand personal and social narratives. These mechanisms include an emphasis on building and enhancing children’s emotion vocabulary using teacher-child interactions and social narratives as well as storybook read-alouds. Given these mechanisms and what has been reported in the literature (Howes et al., 2008; Wasik & Hindman, 2011), we expected RULER-associated growth due to children’s Vocabulary/Oral Language as the subscale taps into naming objects from a story and explaining as much as they can about those objects. This finding in particular is consistent with existing frameworks describing the relationship between language acquisition and early literacy (Rhyner et al., 2009).”
  • However, we also acknowledge in the limitation section on page 17 that we did not explore these mechanisms in the paper: “The analyses in this study thus focused on RULER access and not RULER use. Certainly, RULER’s impact on young children’s early literacy skills could have been due to several mechanisms that make up RULER’s theory of change (Brackett et al., 2019), which is likely given meta-analyses that point to variation in SEL impact due to how programs are implemented (Durlak et al., 2011).”

Reviewer 3 Report

Dear Authors

I appreciated the detailed robustness of the methodology and the succinctness of your overall report. This was well-written and presented complex findings with excellent clarity. Limitations were addressed well.

Author Response

Thank you for your comments and support.

Round 2

Reviewer 1 Report

The authors have improved the manuscript by clarifying the novelty of this work to the current literature on RULER. They have also addressed the limitations to the study that I presented in the first review. I believe in it's current state this paper will be a contribution to the literature.