Sampling and Generalizability in Lx Research: A Second-Order Synthesis

Abstract

As in many other social sciences, second/additional language (Lx) researchers are often interested in generalizing their findings beyond the samples they collect data from. However, very little is known about the range of learner backgrounds and settings found in Lx research. Moreover, the few papers that have addressed the range of settings and demographics sampled in Lx research paint a disappointing picture). The current study examines the extent to which concerns expressed over this issue are merited and worthy of further attention. Toward that end, sample-related features such as L1, Lx/target language, age, proficiency, and educational setting (or lack thereof) were extracted from a sample of 308 systematic reviews of Lx research. The data from this “meta-synthetic” sample are then used to estimate the extent to which Lx research has sampled—and might or might not be able to generalize to—different populations and contexts including those pertinent to migrant populations, the focus of this special issue. The results reveal an incredibly disproportionate interest in participants with English as a first or target language and as well as university students in a narrow range of countries. The findings are used to call out the applied linguistics community on this gross oversight while also seeking to inform future research and contribute to the ongoing methodological reform movement in applied linguistics.

1. Introduction

Applied linguistics as a discipline has made significant substantive and methodological progress since its inception in the second half of the 20th century (see historical treatments in Ortega 2009, chap. 1; Pica 1997). However, until recently, relatively little scrutiny has been paid to the state or development of the field’s quantitative methods, an unfortunate circumstance given the empirical rigor needed to reliably and accurately advance second language (L2) theory and practice (see Gass et al. 2021).

The introduction of research synthetic and meta-analytic techniques has raised researchers’ awareness of methodological issues as examined within and across different domains of L2 research. “Methodological syntheses” have addressed topics such as the psychometric properties of our instruments, transparency in data reporting, replication research, the ethics of data handling and reporting, and the appropriacy of statistical techniques being employed, among other issues (e.g., Crowther et al. 2021; Gass et al. 2021; Isbell et al. 2022; Marsden et al. 2018; Plonsky 2013; Sudina 2021). Stemming from the literature on synthetic methods in this and related fields (e.g., Aguinis et al. 2023; Ioannidis et al. 2015; Plonsky and Gass 2011), this line of inquiry systematically examines different facets of study quality and other methodological features as a means to provide empirically grounded recommendations meant to lead to higher validity evidence (see Plonsky 2022).

The present study seeks to further this agenda by considering sampling practices in a subset of L2 research. The focus here is not on statistical power, defined as the probability of a study detecting a statistically significant result given an expected effect and sample size (Cohen 1988), which can affect the internal validity of a study’s findings (see recent discussion in Nicklin and Vitta 2021). Rather than consider the quantity of samples in this field, the present study focuses on their qualities and, more specifically, the extent to which research findings at the field-wide level might hold beyond the participants and contexts sampled (i.e., generalizability or external validity).

Specifically, this brief report seeks to examine the extent to which L2 research has drawn samples from learners of different ages, educational levels, nationalities, first languages, and so forth as reported in a sample of meta-analyses and other types of synthetic research in applied linguistics.

2. Literature Review

To date, there is very little empirical evidence to support the notion that the field has—or has not—conducted research across the broad range of contexts and learner demographics that it seeks to generalize to worldwide. Taking a similar approach to that of the present study, Andringa and Godfroid (2020) examined several sampling characteristics in a selection of 17 meta-analyses of L2 research. Among other findings, the authors showed that L2 research samples consist overwhelmingly of adults in university contexts. In their own words, “A staggering 88% of all adult samples are university student samples” (p. 138). Staggering indeed.

Collins and Muñoz (2016) coded 97 primary studies of classroom-based “foreign language” learning published in the Modern Language Journal in the 21st century for demographics and other features. Almost half (45%) of the studies were carried out in the US, followed by 10% in Japan and 5% in Canada. Also represented were 21 additional countries, mostly in western or central Europe or East Asia. A total of 75% of the samples were comprised of adults, most of whom spoke as their L1 either English (52%) or a small handful of other languages with origins, again, mostly in Europe or East Asia. The target languages were similarly limited (e.g., English = 35%, Spanish = 20%, French = 14%, German = 13%).

Guo et al. (2021) conducted a similar review of target languages as found in the journal System. The authors were particularly interested in the extent to which primary studies had focused on languages other than English (LOTE). Their examination of 1974 articles found that only 10.5% had addressed the learning and teaching of LOTEs, most commonly Spanish, Chinese, French, German, Japanese, and Arabic.

The findings of the three reviews described here align well with some of the more anecdotal evidence and associated calls for greater sampling in AL research beyond the so-called WEIRD demographics (i.e., Western, Educated, Industrialized, Rich, Democratic; Henrich et al. 2010). Indeed, a growing chorus of scholars have criticized the demographic limitations in L2 research, noting especially a lack of research with naturalistic or non-instructed learners, children, heritage learners, and adults that vary in socioeconomic status, educational level, and literacy, and with LOTEs (e.g., Blum 2017; Cox 2019; Gironzetti and Belpoliti 2018; Mackey and Sachs 2012; Mathews-Aydinli 2008; Ortega 2009, 2019; Plonsky 2017; Vinogradov 2016; Young-Scholten 2013). If verified empirically, the natural consequence of such limited sampling is a lack of generalizability (Tarone and Bigelow 2007); for pedagogy, the consequence is that practice cannot be accurately informed and thus these populations of learners cannot be best served (Ortega 2005).

The concern here actually goes beyond generalizability and practicality. A number of widely believed findings may or may not hold up in different settings and with different populations, thus pushing the field to reconsider the theories and models that led to those results. Huettig and Mishra (2014), for example, present compelling evidence that one’s literacy may affect the way they perceive the world and the way they learn in ways that are not limited to the linguistic domain. Such findings are particularly relevant to the positions and findings presented in the current issue, focused on migrant populations. The present study seeks to build on the evidence provided here in previous reviews—both anecdotal and empirical—to help us understand the extent to which different populations and demographics have been examined in Lx research to date.

This issue is not unlike the observation often made about psychological research that oversamples English-speaking university students yet often seeks to generalize to much broader populations (e.g., Shen et al. 2011). Similar biases toward certain demographics can also be found in biomedical research, where Oh et al. (2015) warn that practitioners “are informed by research extrapolated from a largely homogeneous population, usually white and male” (p. 1).

There is little consolation in knowing that our field is not unique in this respect. Although the vast majority of language learning occurs outside of tertiary institutions in North America, much of the published L2 research appears to be conducted within a small range of institutional settings. Additionally, despite the status of English as likely one of, if not the most, commonly learned (additional) language in the world, there are, of course, multitudes of multilingual communities where English is not a main additional language (e.g., Sridhar 1994). The lack of attention to these contexts and learners, if found, introduces a serious threat to the generalizability of L2 research (see Bachman 2006). To be sure, a comprehensive theory for language learning and teaching needs to be able to account for learning that takes place in a variety of settings and contexts and with learners of many different backgrounds. This goal cannot be met with such limited sampling in the empirical domain.

The present study seeks to build on the existing evidence of limited sampling practices in applied linguistics. The study is also motivated by the potential ramifications that such restricted sampling may have for theory, research, and for the field’s ability to make a meaningful contribution to society at large (Ortega 2019). In response to these concerns, a second-order synthesis was conducted to address the following research question: To what extent does applied linguistics research sample from different proficiency levels, ages, educational levels, learning settings, countries, L1s, and target languages?

3. Materials and Methods

3.1. Sample

The sample for the present study consists of 308 reports of synthetic applied linguistics research. The 308 syntheses comprise 15,019 primary studies and 2,519,933 individual participants1. This sample was drawn from an ongoing collection of synthetic research in applied linguistics. Various forms of synthesis are included in the sample including meta-analyses, scoping reviews, and methodological syntheses (for a breakdown of different types of synthetic research, see Chong and Plonsky, forthcoming). However, all reports in the sample share core principles of research synthesis such as systematicity both in collecting primary studies as well as in coding them for substantive and/or methodological features of interest (Plonsky et al., forthcoming). Although the sample covers a wide range of topics within applied linguistics, it may not be representative of the field as a whole due to the fact that some topics and methods are more immediately amenable to synthetic techniques than others. A number of syntheses have been published since the coding was performed on this sample. Therefore, the present study should not be seen as exhaustive but it is believed to be representative of the domains that have been subject to systematic reviews in applied linguistics. It is also worth noting that some of the syntheses addressed a similar scope and included some of the same individual/primary studies, which may thus be slightly over-represented in the current sample.

3.2. Data Collection

A coding scheme was developed following best practices in research synthesis. In addition to basic bibliographic information (e.g., author, publication outlet), the items in the coding scheme pertained to sampling and demographic features reported in the sample of systematic reviews. These included: (a) sample size (number of primary studies and total number of participants), (b) proficiency levels, (c) participant ages, (d) types of educational institutions, (e) the country where the research was conducted, (f) research setting, (g) participants’ L1, and (h) target language.

Extracted from each report in the sample was the percentage of primary studies in their samples that pertained to each of the demographic/sample-related items. As we might expect, the reporting of this information throughout the sample was irregular and inconsistent. All coding was carried out by two doctoral students with extensive training in research synthesis and meta-analysis.

3.3. Analysis

The research question driving this study asked about the extent to which applied linguistics research has sampled across different proficiency levels, ages, educational levels, learning settings, countries, L1s, and target languages. To address this question, the percentages of each feature were simply averaged across all syntheses that had reported them. Only those averages based on three or more syntheses are reported here. In some of the results presented below, the number of secondary studies contributing to the different percentages is relatively small compared to the entire sample of 308. This is because the results are, by nature, limited to what secondary authors reported. Additionally, some syntheses’ inclusion criteria necessarily limited their samples to learners of English (e.g., Hall et al. 2017). In such cases, data pertaining to the percentages of different target languages were excluded from the aggregated results.

4. Results

4.1. Proficiency Levels

Looking across the sample of secondary research, as shown in Figure 1, intermediate learners comprise the largest group at 32%. Almost 23% of the participants in synthesized applied linguistics studies have been labeled as beginners and only 14% as possessing advanced proficiency. An insufficient number of syntheses focused on or reported the presence of heritage learners which were, therefore, not included here.

4.2. Age

Another important demographic variable to consider is age. The rate of inclusion of adults (18–55; labeled “Adult-1” in Figure 2) is nearly twice that of children (0–12). Very few secondary studies in the sample included teens in their sample and only 1% focused on older adults (>55; “Adult-2”).

4.3. Type of Educational Institution

As displayed in Figure 3, the proportion of samples drawn from university settings is approximately twice that of elementary and middle schools and four times that of high schools. Data collection with participants in language institutes is rare (2%) and even less frequent than samples drawn outside of educational settings (6%).

4.4. Setting

Primary studies included in research syntheses in applied linguistics are carried out fairly evenly in classroom and laboratory settings (see Figure 4). Approximately one in five studies are carried out in both classrooms and labs. Of note as well is the fact that no syntheses in the sample separately reported the percentage of primary studies that focused on migrant populations.

4.5. Country Where Research Was Conducted

The present sample of syntheses shows that primary applied linguistics research is carried out in many different regions (see Figure 5). There are a number of particularly strong concentrations that are worth noting, however. These ‘hubs’ include Central and Western Europe (e.g., Spain), the Middle East (Saudi Arabia, Iran), and East Asia (e.g., Japan, Taiwan). The US is arguably an outlier here and even more so if we consider it along with Canada as comprising a single North American region. Together they represent the setting for approximately 60% of all primary studies in the sample of research syntheses.

4.6. First Language

A range of first languages is represented in the primary studies that made up the current sample of research syntheses (see Figure 6). Somewhat reflective of the findings for the countries where applied linguistics research has been carried out, there is a substantial presence of Asian and European languages and especially English (>40%). There is little to no evidence of sampling among L1 speakers of African, Indigenous/First Peoples, or Aboriginal languages, for example.

4.7. Target Language

The final category that was coded for in this study is the target language of the participants in the primary studies. The range of target languages is actually somewhat smaller than that of the first languages (see Figure 7). Furthermore, the dominance of English as the most commonly studied target language in applied linguistics research is even more pronounced.

5. Discussion

The present study was carried out in response to concerns expressed over the years—and a growing body of empirical evidence—regarding the potentially limited sampling practices within applied linguistics (e.g., Andringa and Godfroid 2020; Bigelow and Watson 2012; Mackey and Sachs 2012; Ortega 2005; Plonsky 2017). More specifically, a number of scholars have warned that we may not be sampling in a manner that allows us to serve and generalize findings to the diverse populations of interest to the field. With these concerns in mind, a second-order synthesis was conducted (see Plonsky and Ziegler 2016) in which a sample of 308 syntheses was coded for a set of sample-related and demographic variables found in their samples of primary studies such as age, research setting, proficiency, L1, and target language.

The results presented above illustrate a fairly clear profile of the typical participant in applied linguistics who is a younger adult with intermediate2 proficiency in a classroom-based study of English at a North American university. This demographic is certainly important and can contribute to many very useful insights into the nature of language development, language teaching, language usage, language assessment, and so forth. At the same time, the results also confirm what many have long since suspected about severely limited sampling practices found in applied linguistics. They also confirm to some extent DeKeyser et al.’s (2010) claim that “…almost every sample has been one of convenience” (p. 416).

Our colleagues in the psychological sciences are often criticized for relying very heavily on participant pools comprised of undergraduate university students who differ in many respects (e.g., cognitively, socially, educationally, racially, and economically) from the population of adult humans that psychologists often intend to generalize to. (A set of statistical techniques have even been developed by psychologists to account for the attenuation in observed effects that occurs when, as is so common in psychometric studies including in applied linguistics, researchers are working with a restricted range of values in a given scale (see McKay and Plonsky 2021).) We are no better and we have no empirical, intellectual, or moral high ground to stand on.

The problem here, of course, goes well beyond lacking a moral or intellectual high ground compared to our colleagues in other social sciences. The consequences of such limited samples are felt at both theoretical and practical levels. Theoretically, a lack of research involving a more diverse range of demographics and populations necessarily yields a restricted base of knowledge. We as applied linguists are interested in advancing our understanding of all language (learning, teaching, usage, etc.) among humans, not solely among the severely truncated populations we tend to sample from. From a practical standpoint, we are simply unable to speak to and inform the many matters of an applied nature that pertain to understudied contexts and demographics when our research does not sample them. As Ortega (2019) put it, “If the goal of linguistic–cognitive SLA communities is to contribute to knowledge building about the human capacity for language—a goal that rests on ideas of post-positivist logic, quantitative rigor, and the golden rule of generalizability—then clearly part of the disciplinary mandate is to produce knowledge about all types, all shades and grades, of multilingualism” (p. 34). In my experience as a researcher, editor, reviewer, teacher trainer, and so forth, it is this brand of applied linguistics that Ortega refers to, however, that is perhaps least likely to seek out samples in less studied settings.

The findings of this study demonstrate that the deficits in our knowledge are particularly grave among children, older adults, non-institutional/classroom learners, and individuals with L1s and target languages other than those typically spoken and studied outside a narrow range of countries generally found in North America, Europe, and East Asia. If anyone was looking for yet another instance of systematic bias and disfavor toward already-marginalized or indigenous populations or the global south, look no further (see Pennycook and Makoni 2020). All this is to say nothing of the many forms of bias and racism experienced by scholars in less privileged settings in applied linguistics. That we as a field do not sample in a wider range of contexts and with a wider range of populations, though clearly detrimental, is perhaps too simple and comfortable of an explanation. Whether for financial, practical, and/or cultural reasons, researchers in applied linguistics tend to focus on populations in their vicinity (e.g., Grenoble and Osipov 2023; Lloyd-Smith and Kupisch 2023). Although outside the scope of the present study, some might argue that researchers’ preferences also reflect epistemological racism and a bias in favor of white Eurocentric hegemonic knowledge (Flores and Rosa, forthcoming; Kubota 2020).

Recommendations for Future Research

The findings of the present study carry with them several immediate implications for future research that include but also go beyond the need for broader sampling. Yes, of course, sampling in a wider range of settings and with a wider range of demographics and languages is needed. The need is perhaps most acute among non-instructed learners, non-literate and low-literate learners, children, older adults, learners of very low and very high proficiency, and L1–TL combinations that do not include English.

One perhaps-obvious step to take, which aligns with ongoing movements in the field toward open science and synthetic-mindedness (Marsden and Plonsky 2018; Norris and Ortega 2006), is to engage in replication research with samples that differ from that of the initial study. I hasten to add at this point, though, that simply altering the participant demographic is not in and of itself necessarily sufficient justification for a replication (see Porte and McManus 2019). One logistically attractive approach to broader our sampling may come through samples obtained virtually (i.e., through the internet; see relevant discussion and examples in Shepperd 2022). However, even with the use of such technologies, samples that are comparable (e.g., literacy-wise) may not be possible. In such cases, researchers might aim for conceptual (not “close”) replication by employing alternative measures and procedures (see Ranta and Zavialova 2022, this issue, for further recommendations and considerations).

There is also an urgent need for research that is informed by and that subsequently seeks to inform on the practical issues pertinent to marginalized settings, countries, and regions. To state the obvious, working in such contexts can introduce logistical, cultural, administrative, and technological hurdles. As I see it, however, this work is our ethical duty and responsibility to each other, to the field, and to the communities who stand to benefit from participating in such research. This is not a call for researchers to ride in on the proverbial white horse of (our own brand of) empiricism. This is a call to develop sincere and hopefully, long-lasting professional relationships wherein all parties may benefit intellectually, practically, and professionally. (As an example, see Rhonda Oliver’s work over the last 20+ years in Australian Aboriginal communities such as Oliver and Forrest 2021, and Shay et al. 2022; in the context of migrant language learners, see work by Bigelow, Tarone, and colleagues.) The next steps needed for the field are not as simple as engaging with such populations in research, however. There may also be a need for the field to consider alternative or novel theoretical models and to consider a range of logistical, cultural, and/or ethical issues (see further guidance on these matters in Hamel-Michaud et al. this issue).

At a more conceptual level, this is also a call for epistemological open-mindedness. If we want to move away from the structural biases that are baked into the paths by which knowledge advances in applied linguistics, we have to put aside some of the conventions we have been trained to adhere to regarding the ’right’ ways of knowing. I personally do not know how to do this or exactly what this looks like, but I want to learn. One point of departure on the path toward epistemological open-mindedness might be to simply listen by, for example, attending talks by scholars from traditions that differ from our own without assuming their standards for knowledge construction will be the same as ours. When doing so, it can be helpful to then question our own ways of arriving at new knowledge. A third step toward epistemological open-mindedness would be to collaborate with scholars from a tradition other than our own and, in doing so, to try to embrace their approach.

One potential threat to the validity of the present study concerns whether the syntheses in my sample conducted adequately comprehensive searches. In other words, it may be that primary applied linguistics research does actually sample from a wider range of settings and demographics but that the synthesists that carried out the studies in my sample did not search enough to locate those studies. Plonsky and Brown (2015), in fact, found the searches in L2 meta-analyses to be far from comprehensive. Whether or not this is the case, I urge synthesists to engage in comprehensive searches and, in particular, to seek to identify primary studies that are unpublished or that are not necessarily published in journals hosted by large, western-based, revenue-seeking publishers.

6. Conclusions

It would be fairly obvious to mention to a colleague the overreliance in applied linguistics on university learners, on WEIRD samples, and on younger so-called college-aged adults. Nevertheless, this study allows us to go beyond such casual—if also largely accurate—observations, to gauge the extent of the problem at hand, which is quite substantial. It is a problem not only on both theoretical and practical grounds; it is also an ethical beach in my view that we as applied linguists have failed so grossly at the field-wide level to contribute to the vast majority of the language-related settings and populations in the world. The near-exclusive representation in mainstream journals both of learners in wealthy countries and of L1 or TL users of English is particularly concerning. However, we can improve and expand. We will improve, I believe, and I hope that the evidence and recommendations provided here take us one step closer to doing so.