The Citizen Science Paradox

Abstract

Citizen science (CS) is now very popular in ecology. The number of scientific publications referencing CS has increased steadily over the past 15 years, with more than 1150 publications today. However, the multiplicity of research involved suggests that this number is highly underestimated. Based on this paradox, a literature review on CS shows that while its formalization in 2009 facilitated its referencing, about 70% of the publications are not referenced using CS as keyword. To understand this under-representation, an analysis of 149 publications related to the famous Christmas Bird Count program shows that this underestimation is not mainly related to the diversity of keywords used to describe CS, but rather to the fact that CS is mainly considered as a method (four publications out of five). The results also show that taking into account the whole text of a publication would represent a substantial improvement for the analysis of scientific databases, whatever the field of research.

1. Introduction

The production of scientific knowledge based on data collected by volunteers (amateurs or not) is not recent. Many collections in natural history museums owe much to amateur naturalists of past centuries. Research areas such as astrology and biology have strongly involved the effort of volunteers since the end of the 19th century [1]. In 1874, the British government founded the project “the transit of Venus” to measure the distance from the earth to the sun with the help of sailors and astronomers (Ratcliff, 2008). In ecology, historically, sciences involving volunteers have played an important role, particularly in ornithology [2]. A first mention dating from 1749 is referenced for the counting of migratory birds by gathering amateurs in Finland [2]. In 1879, the United States Geological Survey requested the involvement of volunteers, resulting in the well-known Breeding Bird Survey [3], officially launched in 1966 in North America. In the UK, the British Trust for Ornithology was founded in 1932 with the aim of using ornithologists’ observations for science and species conservation [4]. One of the most documented examples is the Christmas Bird Count in the United States, initiated in 1900 [5]. This project brings together tens of thousands of observers for a total of 63 million birds counted [6]. Currently, on a global scale, the majority of CS projects in ecology are animal-focused (83%) and 20% are multi-taxonomic. Of the projects focusing on a single taxonomic group, 24% are insects (non-lepidopterans), 19% birds, 17% plants, 12% lepidopterans and 9% mammals [7]. Experiments carried out with the help of non-scientific volunteers contribute to 55% of the registrations in the Global Biodiversity Information Facility database [7], which includes 2.1 billion of occurrence data (February, 2022). Approximately 1.3 million volunteers participate in scientific programs around the world, an estimated in-kind contribution of $2.5 billion [8].

In science, this recognition of the involvement of volunteers in research programs was formalized by the term “citizen science” (CS) in the 90s [9,10]. The first conceptualization carried out by Irwin (1995), a science policy analyst, differs from Bonney (1996), a member of the Cornell Ornithology Laboratory [11]. In his book, Irwin (1995) explains that CS must take into account the bidirectionality of the relationship between scientists and citizens. CS is a science that must serve the interests of citizens (“Science for people”, [12]) but in which citizens must be involved (“Science by the people”). What Irwin (1995) said reflects more a discussion of the ideals and limits of voluntary participation during the 70s than on current scientific knowledge production practices. Bonney (1996) defines CS as a scientific project in which volunteers participate (by providing observational data) but also as a tool to make volunteers aware of science (“a two-way street”). However, these two paradigms converge towards three formalized trends [13], which are the production of new scientific knowledge, the disclosure of science and the transformation of the relationship between science and society. A new and relatively consensual paradigm therefore makes it possible to define CS as a scientific project involving a partnership with volunteers, both novices and experts, in the generation of new knowledge [14,15,16,17,18,19]. In the literature, these volunteers are considered unpaid contributors who are not professional scientists and do not claim to have a level of scientific expertise. In order to avoid confusion related to the different terms used implying the participation of volunteers in the scientific literature (such as “public participation in scientific research”, “volunteer-based monitoring”, “citizen science”, “participatory science”), the use of the term CS was recommended [20]. The formalization of CS in scientific projects has also made it possible to better define the role of volunteers [18,20], and thus to encourage their commitment [21].

The growing success of CS in ecology [22] can be explained because CS makes it possible to investigate research questions on spatial and temporal scales unattainable with scientists only [1,23,24]. Programs involving volunteers such as the Breeding Bird Survey [3], the Christmas Bird Count [5] or Vigie Nature in France [25,26] have made it possible to acquire biotic and abiotic data at national and continental scales [7]. These scales make it possible to highlight ecological phenomena linked in particular to global changes such as global warming, invasive species, species conservation, environmental restoration, population or community ecology, and many other environmental concerns [7,16,17,27,28]. For scientists, one of the major interests of CS is therefore the access to many types of data on ecosystems, allowing the quantification of its ecological status (e.g., through population monitoring [29]), impact studies (e.g., pollution [30]) or the analysis of adaptive management (e.g., [31]). More specifically, CS allows for the characterization of the composition of an ecosystem (e.g., monitoring of indicator species, species with a special conservation status, invasive species, etc. [32,33,34]), its structure (e.g., monitoring of communities, keystone species, prey–predator relationships, etc. [35,36,37]), its conservation status (e.g., monitoring of species with special conservation status, invasive species, etc., [38]) and its management. CS also allows for the study of specific ecological processes (e.g., species–environment relationships, nutrient cycling, etc. [29,39,40]). Another interest of CS is the sampling of diverse environments. Their democratization includes agricultural environments (e.g., [41]), alpine environments (e.g., [42]), polar environments (e.g., [30]), desert environments (e.g., [43]), freshwater environments (e.g., [44]), marine environments (e.g., [45]), aerial environments (e.g., [46]), or the ground (e.g., [47]). Some environments are even inaccessible without the help of volunteers, such as private spaces in urban areas (e.g., [26]). Indeed, urban ecology has been developing for several years [48], and many green spaces supporting biodiversity are privately owned, especially in residential areas [1]. CS is therefore an interesting alternative for scientists who want to access a large amount of data on areas that are not usually accessible [49,50]. Although CS allows for the study of phenomena on large spatial and temporal scales, in some cases, the time series are short. One of the strengths of CS is that it can rely on the principle of replacing time with space [51,52,53]. Indeed, it is possible to identify a number of sites that collectively represent the range of variability in environmental conditions that can be found over a long period of time for a single site [54,55]. This principle allows for the study of long ecological processes on a short time scale [51].

CS also presents a pedagogical and decisional interest. CS allows for the construction of a common knowledge through the involvement of volunteers in a process of learning new knowledge. It also promotes the acquisition of new skills and improves the expertise capacities of participants [16]. This learning process is universal and applies to different kinds of people (e.g., novices [56], stakeholders, enlightened or expert volunteers [57,58]) and in different institutional structures (e.g., associations, schools, etc. [59,60,61,62]). However, as in all research, the knowledge and information generated by CS is constructed in particular socio-economic and political contexts. It therefore depends on the nature of the questions asked, the methods used, as well as how and why the scientists developed the protocols [63,64]. The success of CS therefore lies in informing and sensitizing volunteers, but also in integrating them as CS actors to encourage them to think about the approaches used and the results [65]. CS programs engaged in monitoring biodiversity at different taxonomic levels and/or abiotic variables at different spatiotemporal scales, such as essential biodiversity variables [66], are considered essential to facilitate the transcription of academic results to field actors [67,68].

The involvement of volunteers and the amount of scientific data generated by CS are nevertheless facing a paradox. When searching for the term CS in international databases such as the ISI Web of Science database, until 2010, very few publications are referenced compared to the large number of existing CS programs [7]. This result is counter-intuitive, as several hundred scientific publications are derived from data collected through programs such as the Christmas Bird Counts or the Breeding Bird Survey prior to 2010 [3,16,27]. Evidently, the proportion of CS data in published research results is much higher than is recognized [69] (Figure 1).

To elucidate this paradox and based on a literature review, an analysis of the representativeness of CS was conducted. In the literature, CS is not seen as a science in itself but rather as a new way of approaching science. It is therefore useful to analyze whether the term CS is used in publications (especially since 2009), and if not, which keywords have been used instead. Secondly, to explain why CS is underrepresented in the scientific literature, two non-exclusive hypotheses were tested (Figure 1) by reviewing 149 scientific publications related to the famous Christmas Bird Count CS program. The first hypothesis (H1) lies in the terminology used before the formalization of the CS made by [17]. Indeed, there is an abundance of keywords that does not make it easy to identify the publications related to CS. The second hypothesis (H2) lies in the fact that CS could be seen more as a method or a tool [7,69]. Based on these two hypotheses, it was checked (1) whether or not the authors used specific keywords related to CS and (2) in which sections of the publication these keywords were used, if they were used.

2. Materials and Methods

A first approach based on CS-related keywords was carried out in a literature search to analyze the number of citations in the ISI Web of Science database, before and after 2009, and using different combinations of CS-related keywords. This first part of the analysis allows for quantifying the number of citations on the subject in the scientific literature. The second approach focuses on a bibliographic search for CS-related keywords based on Audubon’s Christmas Bird Count program to understand why CS is underrepresented in the scientific literature.

2.1. Literature Review Based on CS Keywords

Before the formalization of the CS by [17,20], it is therefore necessary to list a set of terms that can be used to account for the inclusion of publications incorporating CS. Based on scientific publications [13,70] and personal expertise, fifteen keywords were used to identify publications related to CS in the Web of Science Core collection database. These keywords are: “public”, “amateur”, “citizen”, “observer”, “volunteer”, “engagement”, “involve”, “participation”, “participatory”, “monitoring”, “community”, “scheme”, “program”, “action research” and “science”. Many words used alone do not refer specifically to CS. This is the case for “science” or “citizen”, for example. After several tests in the Web of Science Core collection database, combinations of these 15 keywords were used to develop a relevant literature search based on 29 combinations (

Table 1. Number of publications associated with the searches performed in the Web of Science Core collection database on 24 March 2022 (period: 1900–2022).

Search Terms on 24 March 2022 (Period: 1900–2022)	Number of Publications (without/with the Condition: And “Ecology”)	Percentage of Publications Published after 2009	Denomination Used in the Histogram Legend in Figure 1
“public * engagement” OR “amateur * engagement” OR “citizen * engagement” OR “observer * engagement” OR “volunteer * engagement” OR “public* involve *” OR “amateur * involve *” OR “citizen* involve *” OR “observer * involve *” OR “volunteer * involve *” OR “public * participation” OR “amateur * participation” OR “citizen * participation” OR “observer* participation” OR “volunteer * participation” OR “public * monitoring” OR “amateur * monitoring” OR “citizen* monitoring” OR “observer * monitoring” OR “volunteer * monitoring” OR “volunteer * based monitoring” OR “participatory monitoring” OR “community monitoring” OR “monitoring scheme *” OR “monitoring program *” OR “participat * action research *” OR “participatory science*” OR “community science *” OR “citizen science *”	65,428/4408	84.14% of 4408 publications	All (4408)
“public * engagement” OR “amateur* engagement” OR “citizen * engagement” OR “observer * engagement” OR “volunteer * engagement” OR “public* involv*” OR “amateur* involv*” OR “citizen * involve *” OR “observer * involve *” OR “volunteer * involve *” OR “public * participation” OR “amateur* participation” OR “citizen * participation” OR “observer * participation” OR “volunteer * participation” OR “public * monitoring” OR “amateur * monitoring” OR “citizen * monitoring” OR “observer * monitoring” OR “volunteer * monitoring” OR “volunteer * based monitoring” OR “participatory monitoring” OR “community monitoring” OR “monitoring scheme *” OR “monitoring program *” OR “participat * action research *” OR “participatory science *” OR “community science *”	60,030/3461	80.26% of 3461 publications	All without CS (3461)
“citizen science *”	6624/1161	98.45% of 1161 publications	CS only (1161)

Asterisk (*) has been used to truncate and find the spelling variation of a specific word (e.g., plural, noun, verb). Quotation marks (“”) have been used to find exact phrases. The use of the operator “OR” retrieves publications that contain at least one of the terms in the title, keywords or abstract. CS: citizen science.

). Finally, an association of 29 keyword combinations with the term “ecology” was made in order to focus the results. The search was completed on 24 March 2022.

2.2. Audubon’s Christmas Bird Count Review

Audubon’s Christmas Bird Count (CBC) program provides an exceptional dataset for unravelling the spatiotemporal drivers of bird occurrence because data have been systematically collected by volunteers on hundreds of species throughout North America since 1900. As the CBC is a CS program, by definition, all scientific publications mentioning this program should be related, at least partly, to one or more CS keywords. Analyzing publications based on CBC allows us to understand how CS is perceived scientifically in this emblematic subsample.

A search in the Web of Science core collection database was made with Christmas Bird Count keywords. The search yielded 171 results on 24 March 2022. Books (3) and chapters (2) were excluded. The relevance of the 166 remaining items was assessed by taking into consideration whether CBC data were used for analyses, and 149 publications were kept (Supplementary Table S1). For each publication, it was checked whether the 15 predefined keywords and the 29 associated combinations had a direct and unequivocal relationship with CS. The word “people” was also checked. This word was not previously included because it is a generic term leading to a high number of selected publications (1,283,534 results when “people” was added to the 29 keyword combinations; 19,710 when added to the 29 keyword combinations and “ecology”).

Then, the section of each keyword was referenced (e.g., title, abstract, keyword, introduction, materials and methods, results, discussion, including conclusion where appropriate, and acknowledgements). To check whether CS was seen more as a method or a tool, the additional keywords “tool”, “database”, “dataset” and “data set” were also screened.

3. Results

The use of the 29 keyword combinations associated with the term “ecology” in the Web of Science core database returns 3461 publications (Table 1). The addition of the “citizen science” keyword increases the number of publications to 4408 (84.14% are published after 2009). Of the 947 additional publications, 214 publications have at least one of the 29 keyword combinations and “citizen science”. The use of the keyword “citizen science” alone and associated with the term “ecology” in the Web of Science core database returns 1161 publications. A total of 98.45% of these publications were published after 2009 (Table 1).

Focusing on the titles, abstracts and keywords of the 149 publications using CBC data, 101 publications had none of the 15 predefined keywords and the additional keywords “people”, “dataset, data set and database”. This represents 67.78% of the publications. If all parts of the publications are taken into consideration (i.e., title, keywords, abstract, introduction, materials and methods, results, discussion and acknowledgements), 93.23% have at least one of the keywords directly related to CS. A total of 10 publications do not present any of these keywords (6.71%).

More specifically, the three main keywords are “observer”, “database, dataset and data set” and “volunteer”, which are found in 59.73%, 53.02% and 49.66% of the publications, respectively (

Table 2. Occurrences of the 15 keywords related to citizen science and the 5 additional keywords (people, dataset, data set, database and program) in the 149 publications using Christmas Bird Count data. Numbers indicate the number of publications concerned by each of the keywords according to the section (T: Title, A: Abstract, Kw: Keyword, I: Introduction, M: Materials and Methods, R: Results, D: Discussion/Conclusion, Ak: Acknowledgements). The numbers in bold indicate the number of publications concerned for a given keyword (last column) or a given section (last row).

Keywords	T	A	Kw	I	M	R	D	Ak	Number of Publications
Observer	1	10		15	71	24	30	7	89
Database/dataset/data set		9		26	66	22	28	6	79
Volunteer	1	7	3	11	35	5	10	48	74
Monitoring/monitor	4	9	5	22	22	7	22	2	49
Participant		4		8	27	9	14	13	48
Citizen	7	16	15	19	16	6	14	10	36
People		1		6	12	5	14	4	36
Program	1	8		16	16	7	10	3	30
Participation/participate/participatory		1		7	14	4	12	4	27
Involvement/involve/involving		1		6	9		5	3	19
Public		3		9	7	1	5	1	15
Engagement/engage		2		7	2	2	5		10
Amateur		1		3	4	1	3		9
Tool		2		2	2		5		7
Scheme		2		2	3	2	3		5
Community science		1	2	1	1		2	1	3
Action research									0
Number of publications	11	41	17	64	116	59	81	72

). More than one paper out of two contains at least one of these keywords, directly related to the sampling method (“observer”/“volunteer”) or as a dataset (“dataset”/“data set”/“dataset”).

The sections with the most keywords are the materials and methods, the discussion and the acknowledgements (Table 2). By searching the 15 predefined keywords and the additional keywords “people, dataset, data set and database”, 116 (78.85%), 81 (54.36%) and 72 (48.32%) publications were found through an analysis of the materials and methods, the discussion and the acknowledgements, respectively (Figure 2).

4. Discussion

Today, CS in the scientific community faces a paradox. On the one hand, programs involving volunteers such as the Breeding Bird Survey [3], the Christmas Bird Count [5] or Vigie Nature in France [25,26] have made it possible to acquire biotic and abiotic data on national and continental scales to highlight ecological phenomena related to global changes (e.g., global warming, invasive species, species conservation, environmental restoration and many other environmental concerns [7,16,17,27,28]). One of the main interests of CS is therefore the access to many types of data on ecosystems, allowing for quantification of their ecological status (e.g., through population monitoring [29]), impact studies (e.g., pollution [30]) or analyses of adaptive management [31]. More specifically, CS makes it possible to characterize the composition of an ecosystem (e.g., monitoring indicator species, species with special conservation status, invasive species, etc., [32,33,34,38,71]), its structure (e.g., community monitoring, keystone species, prey–predator relationships, etc., [26,37]) or ecological processes (e.g., species–environment relationships, nutrient cycling, etc., [29,39,40]). On the other hand, there are not many publications on CS in ecology (4408) compared to the research areas involved. More specifically, the term CS is rarely cited before 2009 (98.45% after 2009), and only 19.74% of papers related to CS using other keywords are found before 2009. The formalization of CS [17] seems to be effective and has facilitated the indexing of scientific citations since 2010. However, by using the term CS only, just over a quarter (26.33%) of the publications are found. The keyword CS alone is thus not sufficient to report on scientific publications using CS. While it is logical to find more publications after 2009 because of the formalization of the CS, more paradoxical is the fact that few publications referring to CS programs (but using CS-related keywords) emerged from literature searches before 2009 [72].

In order to understand why CS seems to be under-cited in the scientific literature, the flagship of the Audubon’s Christmas Bird Count (CBC) CS program was used. As the CBC is a CS program, by definition, all scientific publications mentioning this program should be related, at least partly, to one or more CS keyword. Using 15 keywords and 29 predefined combinations, 67.78% of the publications (101/149) do not refer to keywords related to CS. It is therefore not by multiplying the number of keywords or keyword combinations in a bibliographic search that the majority of publications related to CBC can be obtained. In other words, based only on keywords related to CS, approximately 70% of the publications related to CBC will not be found. The hypothesis that the multitude of keywords would explain the under-representation in terms of citations of CS-related publications [69] is therefore not valid.

The main explanation of such a result comes from the method of referencing publications in bibliographic databases such as the Web of Science core collection itself. Indeed, this database refers to keywords belonging to titles, abstracts and keywords. A rigorous analysis of the whole text (including title, abstract, keywords, introduction, materials and methods, results, discussion and acknowledgements) shows that when considering all parts, 93.29% (139/149) of the publications refer to one or more CS-related keyword. More precisely, 77.85% of the publications refer to a keyword relating to CS in the materials and methods section (i.e., almost four publications out of five). Following our second hypothesis and as suggested by some authors [7,69], CS is thus mainly considered a tool for scientists. It should be noted that 10 publications do not refer to any CS-related keywords (not even in the acknowledgements section). For these publications, without knowledge of the CBC, an uninformed reader would not know that the data used came from a CS program.

A potentially explanatory but untestable factor may also explain the limited number of scientific publications on CS. Beyond scientific referencing, the evolution of the roles of participants in CS [18] allows for the integration of Community Based Monitoring (CBM) in CS. CBM is described as “community science” or “participatory action research” [73]. CBM is supported directly by stakeholders and/or volunteers, not scientists [74]. They use their own resources to monitor, record and respond to specific environmental issues (e.g., [74,75,76]). For stakeholders, CS through the CBM has made it possible to initiate projects that scientists would not have initiated by themselves. This is mainly due to the type of question asked, but also due to the geographical location and the scale concerned. Some stakeholders (e.g., watershed managers, park and reserve managers, green spaces, etc.) need to have a holistic approach for understanding their own ecosystems and associated ecological functions in order to provide adequate expertise and adaptive management actions [21,77,78]. However, CS developed on large scales are often unsuitable for these stakeholders [79]. Many local-scale projects have therefore been initiated by volunteers, both professionals and non-professionals, to address their environmental concerns [79]. These projects can focus on the cause of an environmental problem, such as pollution, “suspicious” animal deaths, the spread of invasive species or the conservation of species of interest for their heritage [80]. As an example, the “Save our streams” program, initiated in 1969 to monitor, protect and restore streams in the state of Maryland (USA), served as a model for launching the national scientific program Izaak Walton League of America. This new type of approach no longer reflects the traditional perception of scientists using volunteers to collect data, but rather refers to a two-way process that goes beyond the mere production of information and data [14,81]. Many CBM results are available through reports (“grey” literature) or dedicated websites and are thus not published in scientific journals [79,82].

In this review, the ISI Web of Science database was used. There is another widely used scientific database, Scopus. ISI Web of Science and Scopus are the most widespread databases on different scientific fields which are frequently used for searching the literature. There are still debates about the use of Web of Science or Scopus, depending on the field and the research objectives [83]. Scopus includes scientific journals, books and conference proceedings, covering research topics across all scientific and technical disciplines, ranging from medicine and social sciences to arts and humanities. Scopus is more inclusive and contains more types of material, and therefore, it is less selective. ISI Web of Science is the oldest citation database, having strong coverage with citation data and bibliographic data which goes back to 1900. As the CBC appeared in 1900, that is why this review used ISI Web of Science rather than Scopus. It is nevertheless important to consider that the use of Scopus could have led to different results, mainly for the last few years, with potentially complementary publications related to CS.

5. Conclusions

The CS paradox highlights the limits of formalizing a recent scientific trend by keywords and referencing them in scientific databases in order to conduct an exhaustive bibliographic search (Question 1). Even with multiple keywords (Hypothesis 1), it is the referencing of specific parts of publications themselves in search engines (e.g., title, abstract and keywords) that generates a potential under-representation of publications. This review shows that while formalizing CS improves its referencing, CS is mainly considered as a method (Hypothesis 2). There are, however, two major elements that may limit the conclusions of this study. The first is the choice of bibliographic database (ISI Web of Science), which, although justified by its historical aspect, can be complemented by other databases (e.g., Scopus). The second is the question of the integration of the “grey” bibliography (CBM) and the way in which this existing literature considers CS. Despite the limitations of this study, the results show that basing searches only on titles, abstracts and keywords greatly underestimates the number of scientific publications related to CS. One way of avoiding a bias such as the CS paradox would be to improve the consideration of the whole text in bibliographic searches, in particular by allowing for the selection of specific parts (e.g., materials and methods or discussion). By taking this aspect into account, bibliographic database managers would prevent the under-representation of specific research areas solely because of the referencing system. It should be taken into account that many CS results are also published in the grey literature or are not published, as is the case for many CBM results. The referencing of CS should therefore also be completed through more general databases, covering both academic and non-academic databases.