Next Article in Journal
Roles and Responsibilities for Peer Reviewers of International Journals
Previous Article in Journal
Panorama of Undergraduate Research in Brazil: Profile, Scientific Production, and Perceptions
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Publications
Volume 11
Issue 2
10.3390/publications11020031
publications-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Measuring the Impact and Influence of Scientific Activity in the Humanities and Social Sciences

by
Daniela De Filippo
1,*,
Fernanda Morillo
1 and
Borja González-Albo
2
1
Institute of Philosophy (IFS), Spanish National Research Council (CSIC), 26-28 Albasanz Street, 28037 Madrid, Spain
2
Centre for Human and Social Sciences (CCHS), Spanish National Research Council (CSIC), 26-28 Albasanz Street, 28037 Madrid, Spain
*
Author to whom correspondence should be addressed.
Publications 2023, 11(2), 31; https://doi.org/10.3390/publications11020031
Submission received: 31 March 2023 / Revised: 11 May 2023 / Accepted: 22 May 2023 / Published: 1 June 2023
Browse Figures
Versions Notes

Abstract

:
Scientific activity in the Humanities and Social Sciences (HSS) presents special characteristics that require the use of various sources and methodologies to adequately assess its impact and influence on both academic and non-academic audiences. This study aims to explore the validity of traditional and alternative information sources for the analysis of the characteristics of HSS research and its academic impact and influence (considering social, media, informative and political influence). It is also intended to highlight the differences between Humanities (H) and Social Sciences (SS) and analyse the variables that determine the different types of impact and influence of research in each of them. The following sources of information are used: Web of Science, conCIENCIA (institutional database), Google Scholar, Unpaywall, Altmetric.com and Overton, focused on the study of the Spanish National Research Council (CSIC). The results obtained show that institutional sources make local research visible, which has high percentages of open access. The usefulness of alternative sources to measure social, media, informative and political influence is verified, since HSS publications have an important number of mentions. Significant differences are observed between H and SS in terms of publication coverage (higher in H in the institutional database), language (more Spanish in H), open access (higher percentages in SS) and impact measured through conCIENCIA (the greatest number of documents with a high impact is found in H). In addition, the influence on non-academic audiences is increased by the international orientation of research, the greater academic impact, the participation of SS centres and the immediacy of publications. This study is a starting point for future research, as it explores several tools and data sources to analyse the influence of HSS research on different audiences. A comprehensive analysis will also facilitate the proposal of new metrics applied to the HSS assessment, highlighting its importance for society as a whole.

1. Introduction

Since Eugene Garfield coined the term “impact” associated with citation as an indicator of the influence of a research study [1], its definition, measurement and scope have been one of the most discussed aspects in the field of bibliometrics [2]. In fact, as Priem mentions [3], the evaluation of the impact of academic publications has become a key research area for bibliometric specialists, scientific institutions, academic journals and countries. In this sense, as noted by Yang et al. [4], a classic interpretation of research impact by Research Councils UK states that it involves two categories: academic impact (research contribution to scientific advancement) and economic and societal impact. Indeed, Bornmann [5] confirms that scientific evaluation increasingly covers more fields, and that new approaches to social problems should be stimulated, provoking public debate and informing policy management.
In recent years, together with the new ways of producing and disseminating knowledge, we have witnessed a growing discussion about the need for new metrics for evaluation. Declarations such as those of DORA [6], ENRESSH COST Action [7], Leiden [8] or the Metric Tide Report [9] plead not to base measurements solely on the impact of publication journals, but also to recognise the importance of the specific characteristics of the different scientific fields’ output when considering their impacts.
In the case of the Humanities and Social Sciences (HSS), the need to find more suitable metrics has been especially discussed due to their already known particularities. Their practices differ from the patterns of production and diffusion in other research fields: the scope of the objects of study, generally local or national, the use of the vernacular language and the dissemination in books or book chapters [10,11,12], although the publication in international journals is increasing. HSS require alternative methodologies adapted to them to measure their activity and impact. It is also noted that HSS researchers often write not only for academic readers, but also for lay audiences [10], sometimes seeking broader public benefits through their work [13], so it is very valuable to also collect and analyse their influence on other audiences. In fact, as Giménez Toledo [14] points out, although scientific publications on HSS continue to be the most concrete result of research, their impact and influence on society must also be taken into account, considering various dimensions and data sources (e.g., research management systems—CRISs). In this sense, according to Sīle et al. [15], relative to commercial databases, national bibliographic databases (or CRISs) are often more comprehensive and therefore more suitable for bibliometric analyses.
This is why the scientometric community has made great efforts to improve and complement the methodologies for the HSS evaluation [11,16,17], proposing alternative metrics to traditional citation analyses. Since the arrival of the Web 2.0, there has been an important change in the way of generating and disseminating knowledge and, with it, in the way of interaction between academics and society. In this sense, Van Noorden [18] pointed out that scientific communication has changed in an unpredictable way, since some academics want to share their knowledge and experiences “openly”, as witnessed by the use of multiple digital identity tools and social networks. This set of tools is reinforced by open access (OA) to science and online publications and repositories and, as Mohammadi and Thelwall [19] mentioned, it becomes clear that this proliferation of informal communication channels constitutes a new challenge for the analysis of scientific activity and for the measurement of its impact. In this line, Priem [3] also stated that the web has provided new data to measure academic performance, and that bibliometrics has been transformed from the mere monitoring of citations.
In the case of HSS, altmetric indicators have found especially fertile ground, since they have shown to have a close relationship with these fields [4,20,21]. There are several studies that have detected a greater presence of altmetrics in HSS disciplines in relation to the natural sciences [22,23]. For this reason, they suggest that these metrics may be used as a complement to citations in their scientific activity assessment, providing new perspectives in relation to the study of their impact and influence. One of the main advantages of altmetric indicators is that the importance of each publication can be measured without considering the quality or visibility of the journal that publishes it [24], which is very favourable for HSS. In addition, given the extremely large heterogeneity of the research carried out in these fields, the analysis of mentions in social media offers new opportunities to study their influence on various audiences.
With the new ways of communication, the number of sources and indicators to analyse scientific activity has also increased, and the discussion on measuring the research impact has taken new directions. In this sense, it has become evident that traditional studies based on bibliometrics can be complemented with new altmetric indicators that make it possible to measure the interest that research arouses in society and that have had an important incidence since its appearance in 2010 [25]. However, although there is a certain consensus on the value that altmetric indicators offer to study the diffusion or influence of research, and in particular in HSS, it is important to consider what these indicators really measure. Some researchers have analysed the meaning of these new metrics as well as the need to build a theoretical framework for their study [26,27,28]. The scope of the different platforms and indicators has also been examined, and the providers have been compared [29,30,31,32,33,34]. Other researchers have focused on the characteristics of documents that can affect their societal impact [13,26,35,36] or in the relationship between the impact measured by bibliometric and altmetric indicators [19,22,37,38,39,40]. Lastly, the advantages and limitations of altmetric indicators have also been widely described in the literature [41,42,43].
On the other hand, it is interesting to recover the reflection of Sugimoto [44] when she argues that the impact and interest that a scientific publication arouses are not synonymous. In general, altmetric indicators measure the attention that academic output receives. For this reason, some authors consider that mentions from platforms such as Google+, Twitter and Facebook could represent social attention or the influence exerted on a general public. For its part, Mendeley readership could be related to some extent to academic attention, while blogs and news would reflect media attention or journalistic interest, and Wikipedia could explain the informative influence [4,45] or the transfer of knowledge to society [46]. Another interesting source of information are policy documents that show how scientific research impacts areas related to decision making. In addition to the mentions collected on platforms such as Altmetric.com, the recent creation of Overton [47] allow us to delve into this new area of research impact [48]. Analysing these different types of attention or influence, researchers such as Torres-Salinas and Romero [46] consider four dimensions: social, political, media and educational. Based on these proposals, this paper also studies four types of influence according to the origin of the altmetric mentions: social (measured through Twitter), media (journalistic interest), informative (considering Wikipedia) and political (seen through Overton).
The results obtained in these previous studies have revealed the importance of using additional alternative metrics to collect information not only on the impact, but also on aspects related to the influence or interest that the research arouses both in the academic and non-academic public, especially in HSS. Although these new metrics are not without controversy [49], and certain limitations continue to be attributed to them [28,35], there are several investigations that show that the criticism received does not undermine their ability to offer an insight on the perception and consumption of scientific literature in non-academic contexts, and therein lies their enormous potential. Alternative metrics provide a unique view of science, especially when combined with other methodologies or applied to specific case studies [50].
In order to study the scientific activity in HSS, as well as their academic impact and their influence on other audiences, the research performance of the Spanish National Research Council (CSIC) is analysed. This institution is particularly suitable since its main activity is scientific research, and it has institutes working in almost all fields of knowledge. In addition, the CSIC’s institutes are structured according to their main field of research, although their publications can also be assigned to other areas, which is more appropriate, considering that centres are one of the objects of scientific evaluation and monitoring of compliance with goals. Therefore, this is a good starting point to analyse which sources and methodologies are the most suitable for the assessment of the research performance of HSS, which correspond with one of the eight scientific-technical areas in which CSIC is divided, Area 1, and with one of the three global areas in which this institution is structured. This area is made up of 16 institutes that work in the fields of Economics, Geography, Sociology, Political Science, Anthropology, Documentation, History, Philosophy, Archaeology and Literature (see the list in Appendix A).
In this context, the present study aims to obtain information from traditional and alternative sources to study the scientific activity in HSS, taking as an example the output of the centres of CSIC’s Area 1. We consider that HSS research is of special interest to the non-academic public, so using alternative sources, it is possible to find out about this interest and its scope and influence. In this line our starting hypotheses are as follows:
H1. 
Practices of knowledge production and dissemination of the Humanities (H) and Social Sciences (SS) are quite different so their impact and influence are expected to be different as well.
H2. 
Detecting variables related to the visibility of publications will help to increase their impact and influence on other audiences.
With the purpose of validating or refuting these hypotheses, the following objectives are proposed:
  • Explore the validity of traditional and alternative sources and their relationship with academic impact and influence of research.
  • Analyse the patterns of activity and impact and influence, comparing Humanities vs. Social Sciences.
  • Study the existence of possible relationships and interactions between variables, which may shape the impact and influence of HSS research.

2. Sources and Methodology

2.1. Sources

The following sources of information were used in this study:
  • Web of Science Core Collection (WoS). This source includes high-quality international scientific publications and offers indicators of publication impact, visibility, funding and use. Although the HSS output is studied, the SCI database was also considered, since it includes a significant volume of the publications analysed (around 25% of the production of the CSIC’s HSS centres).
  • conCIENCIA. This is an information system for the registration, maintenance and validation of information on the CSIC scientific contribution [51], which was implemented in 2010. It includes all the research results of the CSIC staff, with the objective of assessing the activity of its institutes to award economic incentives for meeting objectives. The conCIENCIA is used in this study to retrieve documental typologies not considered by traditional bibliometric databases.
  • Altmetric.com (https://www.altmetric.com, accessed on 12 May 2022). This is a tool that offers information on the effect and influence of scientific publications through alternative sources, covering mentions received by publications on social networks (such as Twitter), in the media, in Wikipedia, etc. These mentions are analysed to learn about the consumption of scientific production by a broader audience.
  • Overton Database (https://www.overton.io, accessed on 12 May 2022). Since 2019 Overton has been storing information extracted from the set of documents coming from bodies such as national and regional governments, international organisations, think tanks and NGDOs. Overton refers to these items as “policy documents”. This tool was chosen because it offers broader, more consolidated information than Altmetric.com’s “mentions in policy documents” field.
  • Unpaywall (https://unpaywall.org, accessed on 12 May 2022). This is a database that collects information on the open availability of documents in more than 50,000 scientific journals and open-access repositories, gathering data from legal sources.
  • Google Scholar. The academic text search engine was used to find the number of citations of the documents retrieved both in WoS and in conCIENCIA since it is a source that allows quantification of citations for all types of documents.

2.2. Methodology

This study was performed in three stages. In the description of each stage, the general information of the HSS area is presented, and then the disaggregated data are shown: Humanities vs. Social Science (Table 1). The procedure followed in each stage is explained below.

2.2.1. Bibliometric Analysis

WoS bibliographic records were obtained to select the HSS area CSIC centres from the total set of Spanish documents from 2017 to 2020. The data were structured in a relational database containing all the bibliographic data including information on authors and centres, so that the main bibliometric indicators could be obtained. The working addresses of the downloaded documents were subjected to an automatic coding process [52]; subsequently, the addresses that could not be assigned coding and the CSIC-affiliated addresses were reviewed. Bibliographic data were downloaded in March 2022. Although all the publications’ documental typologies were identified, this study focused on the analysis of citable items (articles and reviews, referred to in the text as articles, WoS-art).
Searches were carried out for each of the institutes in the CSIC’s HSS area (see the list in Appendix A) to extract the records from conCIENCIA. From the various files, downloaded in “csv” format, a database was built identifying each of the documental typologies included in the resource (articles, books, book chapters, conference presentations, scientific dissemination documents, intellectual property documents and doctoral theses). Downloaded data were cleaned and treated to remove erroneous, missing and duplicate DOIs (the same DOI may exist for different book chapters or for the electronic and paper versions of a document). The analysis focused on articles (C-art) and books and book chapters (C-book), because they are the only documental typologies with DOIs. In addition, items already collected in WoS were excluded.
The information on the open-access documents (both in WoS and in conCIENCIA) was obtained from Unpaywall queries using each publication’s DOI.
For the analysis of the academic impact, both the citations collected by Google Scholar and the CSIC’s classification methodology for conCIENCIA were used. These classifications have been applied to the three documental typologies used (WoS-art, C-art, C-book). In the methodology used in conCIENCIA for the assessment of institutional production, documents are classified as having LOW, MEDIUM or HIGH impact. This classification is based on the equivalences with the quartiles that other sources (WoS, Scopus and CIRC: www.clasificacioncirc.es) give to scientific journals. In the case of books and chapters, the classification of conCIENCIA follows the classification proposed in Scholarly Publishers Indicators [53]. Google Scholar citations were obtained using the document DOIs. The results showed a few documents with a disproportionate number of citations, which led to a manual review to correct erroneous data 1.
The results are shown at two levels: CSIC’s HSS area as a whole and Humanities vs. Social Sciences. Documents were assigned to one or the other based on the institute with which the authors were affiliated. The percentage of scientific production included in WoS from 2017 to 2020 was analysed to determine whether its activity focused mainly on one field or the other.
Chi-square tests have been applied to check if there are statistically significant differences in the results of Humanities versus Social Sciences.

2.2.2. Altmetric Analysis

After the identification and recovery of documents from CSIC’s HSS area, those publications with DOIs were selected.
The main indicators provided by the Altmetric.com and Overton tools were obtained. The second one is used because it has a broader coverage of political documents. The information they offer makes it possible to learn the influence of each of the scientific publications in the media and on social networks. The analysis considers four influence dimensions: social (measured through Twitter), media (journalistic interest), informative (considering Wikipedia) and political (seen through Overton), based on the classifications proposed in previous studies [4,45,46].
Chi-square tests have been applied to check if there are statistically significant differences in the results of Humanities versus Social Sciences.

2.2.3. Relationships among Variables

Several logistic regression models were applied with all the variables of interest, finding both the main effects and the interactions between variables. In addition, to assess model quality, the Wald test (p < 0.05) and Nagelkerke’s R2 were applied, and a crosstab was created with the predicted versus observed values.
Logistic regression models were built with the presence/absence of influence (1/0), as a dependent variable, according to the four dimensions used in this study (social influence, media influence, informative influence, political influence) and several predictors or independent variables. Variables included were scientific activity (considering the prevalence of the documental typologies WoS-art -0-, C-art -1- and C-book -2- and the English language -1/0-), thematic specialisation (in Social Sciences centres vs. Humanities centres -1/0-), international collaboration (1/0), accessibility (OA -1/0-), impact by Google Scholar (highest quartile) and publication year (most recent year, i.e., the closest to 2020).
In order to allow comparisons considering thematic specialisation, the impact by Google Scholar was proportionally distributed (except for the group without citations) across four quartile ranks: (1) no citations, 0 for both (H and SS); (2) low citations, 1–2 for H and 1–4 for SS; (3) medium citations, 3–8 for H and 5–13 for SS; (4) high citations, ≥9 for H and ≥14 for SS. The threshold was automatically assigned by a statistical process.

3. Results

The results of the analysis of the scientific production of the CSIC’s HSS area, in the period 2017–2020, as well as its impact and influence are presented below.

3.1. Bibliometric Analysis

A total of 48,062 CSIC documents were identified in WoS in the period under study; these account for 14% of the Spanish output, the reason why CSIC is positioned as the institution with the highest number of publications in the country [54].
In the HSS area, 1397 publications (1035 articles) were retrieved in WoS, while a total of 1935 documents were identified in the conCIENCIA database. Of these, 569 (29% of the conCIENCIA total) were journal articles not included in WoS, while another 889 2 (46%) were articles also included in WoS, and 477 (26%) were books and book chapters. Since documents had to have a DOI to enable the consultation and retrieval of bibliometric and altmetric data, only documents with a DOI were included in this study. As seen in Figure 1, documents with a DOI make up 95% of the WoS articles, 100% of the articles in conCIENCIA and 32% of the books and book chapters.
The non-overlapping information between information sources is presented below. The data are grouped into WoS-art, C-art and C-book.
Table 2 presents the main characteristics of these documents. The predominant language was English, in proportions ranging from 86% in the case of WoS-art to 53% of C-art. Spanish was the second dominant language, exhibiting especially high proportions in C-art. Although there were documents in other languages (Portuguese, Italian, German, French, Catalan, Galician and Arabic), the proportions were minimal 3. International collaboration rose above 50% only in WoS-art, while the lowest percentage was observed in C-art (very similar to that of C-book). The OA percentage was high in C-art (72%) 4 and WoS (64%), while it hit its lowest figure (37%) in C-book.
Following the CSIC’s Low, Medium and High impact classification methodology, made by its institutional database conCIENCIA (and its adaptation for WoS), both WoS-art and C-book reached mostly a high impact, while the distribution was usually more homogeneous in the case of C-art. WoS-art had the greatest impact in terms of citations in Google Scholar, since 84% obtained some citation there, while the percentages dropped to 62% for C-book and 52% for C-art (Table 2). In order to detect the level of congruence between impact classification criteria (conCIENCIA vs. Google Scholar), the results were compared for all types of publications. The results show equivalence between both types of impact classifications (see Figure A1 in Appendix B).
After the general analysis of the HSS area, the information was disaggregated into Humanities and Social Sciences.
Interestingly, it was verified that while in the case of WoS-art, the distribution between Humanities and Social Sciences was roughly even, the conCIENCIA documents were mostly from the Humanities. This affects the dynamics of article production in the latter, e.g., the proportion of English-language documents in C-art is less than 45%, whereas it surpasses 70% in the other types and in WoS. The percentages of documents in OA and in international collaboration also differ, with lower figures in the Humanities in both sources and all documental typologies. Books and book chapters in the Humanities are an exception, with 40% available in open access compared to 25% in the Social Sciences (Figure 2).
With respect to impact, in the case of the conCIENCIA classification (Low, Medium, High), no notable differences between Humanities and Social Sciences were observed for each data source, with some minor exceptions. However, considering Google Scholar, the percentages of cited documents in Social Sciences are higher than those in Humanities (Figure 3).
In most of the indicators there is an association between variables, that is, the differences in the values obtained in the Humanities are statistically significant in relation to those of the Social Sciences, regardless of the data source. There are only two exceptions: documents in Spanish language and documents in international collaboration, the proportion of which does not seem to depend on each of them alone but rather on the data sources.

3.2. Altmetric Analysis

The analysis of the repercussion that HSS documents have had on other audiences reveals that social influence has the greatest presence. On Twitter, 59% of the WoS-art have received mentions, representing 23% in the case of C-art and 31% of C-book. Political influence is also relevant, especially in the case of WoS-art, since 14% of the publications received mentions in policy documents (Table 3).
Figure 4 shows that the percentage of mentions received varies between Humanities and Social Sciences. Thus, it can be seen that the social influence of publications is much greater in the case of the Social Sciences than in the Humanities, especially for C-art. The media influence shows, however, that WoS-art in the Humanities have received more mentions in the media than those in the Social Sciences. In turn, the informative influence of Humanities publications is also greater than that of Social Sciences, with the exception of C-book.
The proportions differed even more severely in the case of mentions in policy documents. Analysis of this source showed that the Social Sciences percentages of mentions were between 11% (C-book) and 24% (WoS-art), while the Humanities’ figures ranged from 2% (C-art and C-book) to around 4% (WoS-art).
When considering the weight of influence in Humanities vs. Social Sciences in WoS-art, the percentages of documents are statistically significantly higher in Social Sciences than in Humanities, in the cases of three influence dimensions: social, informative and political.

3.3. Relationships among Variables

To deepen the analysis of the social, media, informative and political influence of documents from the CSIC’s HSS area, several logistic regression models were built to detect which variables best explain it. As shown in Figure 5, the probabilities (odds ratios) of having influence, considered as the presence/absence of mentions in the considered sources, increase if a paper has certain characteristics: published in English, collected by WoS, signed by a Social Sciences centre, ranked in the high citations quartile in Google Scholar, published in open access, with international collaboration and recently published. These variables allow us to correctly classify, on average, 73% of the cases analysed; that is, the ability to predict the presence/absence of influence is quite strong (76% and 71%, respectively). This percentage means the average effect of all the variables combined in the prediction of the influence.
Furthermore, the probability of finding a publication with any of the four types of influence analysed increases with the presence of some interactions. For instance, Figure 6 shows how the predicted probabilities increase with the synergy between the publication of an article by a Social Sciences centre and in conCIENCIA (Social Area and C-art). Additionally, this article published in conCIENCIA will have more influence when it is highly cited in Google Scholar, as it was expected (GS Quartile and C-art).
The detailed results are shown in Appendix C.

4. Discussion and Conclusions

One of the objectives of scientific activity is to ensure that results contribute to the advancement of knowledge, and that their impact and influence transcend the academic field and reach different sectors of society. In order to address this issue, the study of the activity of the CSIC’s HSS area was performed, which is similar to that of other European research institutions [54], enabling the exploration of new sources and obtaining a series of results that are discussed below.
The bibliometric study of documents with DOIs confirmed that the greatest international orientation exists in the WoS database (publication in English and with high international collaboration). For this reason, it is important to use additional data sources, such as institutional ones, to gain a broader vision that goes beyond the results included in international databases (see, for example, Sivertsen) [55]. Nevertheless, considering that we needed the DOIs as a necessary parameter for our analyses, most of the books and book chapters we collected were in English.
Surprisingly, the highest accessibility is found in C-art. Indeed, its OA percentages are much higher than those found in other studies using national sources to determine publication availability or the percentages found in the WoS database (e.g., Pölönen et al. [56]. However, the OA WoS-art we analysed accounted for 64% of the CSIC’s HSS documents, slightly higher than the total figure for Spain, which was 62% for 2017–2021 [57]. Regardless of how widespread OA practices are at the CSIC, these high figures are also explained by our selection of documents with DOIs. Production with foreign institutions (international collaboration) is twice as high in WoS as in conCIENCIA, which is in line with the broader context of CSIC and Spain in the same period, while the low figures for C-art and C-book were as expected. Nevertheless, regarding the impact, not only WoS-art show high figures, but also C-book (73–78% of high-impact documents in conCIENCIA), and the same happens with respect to citation in Google Scholar (high percentages of cited documents in both cases).
There are statistically significant differences between Humanities and Social Sciences in terms of coverage in different databases. Coverage is more balanced in WoS and less balanced in conCIENCIA (where the Humanities have more weight). This can be explained by the trend towards publication in national journals, which is greater in the Humanities, as has also been found in previous studies [58,59]. In addition, there are also differences regarding publication in English (the largest presence of Spanish in C-art is in Humanities), publication in OA (the highest comes from Social Sciences) and impact measured through conCIENCIA (the greatest number of documents with a high impact is found in Humanities). However, the percentage of documents cited in Google Scholar and the average number of citations are higher in Social Sciences than in Humanities (with statistically significant differences in both cases). Nonetheless, as these are non-normalised impact indicators, differences must be considered as merely descriptive, given that the citation habits of both of them are very different.
Since traditional bibliometric indicators only show the academic impact, it is important to consider other possible influences, especially in the social sphere. Costas et al. (2015) indicate that the impact of HSS through Altmetric.com is greater than that of natural sciences, and that in this sense altmetric impact can be an appropriate complement to citation analysis. Altmetric indicators have thus been explored as a means of social, media, informative and political influence of research, and both their advantages and their disadvantages have been highlighted.
The analysis of alternative sources has revealed that social influence is the most prominent, because documents with a Twitter mention range from 59% for WoS-art to 23% for C-art. WoS-art results are consistent with the overall CSIC’s output, which reaches 69% in all areas. This prevalence of Twitter as the most used altmetric source seems to be a constant in studies that use the Altmetric.com source [23,26]. Some authors have said that Twitter is of particular interest because it is widely used outside the academy [60]. Therefore, it seems to be a promising source of information on public interest in science. Likewise, it is also used by academics, so tweet counts can also reflect academic impact [61]. Previous studies of mentions on Twitter have shown that, on average, 35% of Social Science articles are usually tweeted about one or more times, while Humanities articles are tweeted about only half as much [62]. Although in our case we analyse HSS centres and not subject areas, the superiority of the figures shows the important social influence that research in the CSIC’s HSS area has.
Media influence indicates that, almost 8% of WoS-art have been mentioned in digital press. Figures reach 4% for C-book and 2% for C-art. Informative influence of C-book, although scarce, are proportionally important (8%). This agrees with the findings of the work by Taylor [63], which analyses the influence of books and book chapters on social media and their relationship with OA (OA increases the number of mentions), a trend that is confirmed in our study through logistic regression analyses.
Within the study of influence, it is especially interesting to analyse the political one, such as mentions collected by the Overton database, since they offer the potential to broaden the spectrum of social results capable of being subject to a solid quantitative evaluation. Using Overton also improves the accuracy and the generally underestimated volume of citations of policy documents collected by Altmetric.com. Therefore, this new source is a promising tool for the study of research impact on policy documents, in line with Pinheiro et al. [64]. Likewise, as other recent studies show, the use of this source is especially relevant in Humanities and Social Sciences. According to Fang et al. [65] HSS publications appear to have the highest probability of being cited by the Overton-indexed policy documents. These findings reflect a tendency towards research outputs in the fields of Humanities and Social Sciences to attract a greater deal of attention from the policymakers.
The results of our study show that the influence of the CSIC’s HSS production in policy documents is not very high in absolute values, except in the case of WoS-art, since 14% of these publications have received mentions in policy documents. These values are in line with those found in other studies [65] which show that 12% of the Humanities and Social Science publications have received at least one mention in policy documents.
The analysis of the presence/absence of influence shows statistically significant differences between Humanities and Social Sciences in the social, informative and political dimensions, with higher proportions in Social Sciences publications. Thus, the first hypothesis was confirmed through this comparison. However, recent studies have detected that the profiles of influence in some disciplines can be very different. In this sense, authors show that the percentages of impact on social media for WoS articles are 32% in economics, 49% in sociology and 55% in communications [23]. As stated by these authors, the high proportion of documents with mentions seems to be associated with the objects of study, since the fields connected to scientific communication and open science seem to have an important societal impact.
For the political influence case, the absolute values are higher in Social Sciences, although no statistically significant differences have been found (perhaps due to the low number of publications with mentions). These results agree with studies that analyse the scope of the source used (Overton), showing that its coverage is mainly related to Social Sciences, Economics and the Environment [47,64].
To analyse the possible factors that contribute to the influence of research publications in various audiences (including the four influence dimensions), different logistic regression models are performed. In this sense, these models illustrate the importance of international visibility through either use of the English language, inclusion in WoS or collaboration with foreign institutions. Likewise, documents in OA and those with academic impact show an important societal influence. Additionally, documents from Social Sciences centres and recent publications also arouse the greatest interest on social media. Lastly, there are synergies between some variables, for example, the publication from Social Sciences centres is decisive to increase the influence of C-art, as well as the publication with high impact in Google Scholar. All these statistical tests point to certain differences between Humanities and Social Sciences in terms of the influence of their research, as indicated in the first hypothesis, and detect which variables help to increase the visibility of publications in other audiences, as stated in the second hypothesis.
In conclusion, the study carried out has shown that the use of multiple sources makes it possible to better collect local output and obtain additional data not included in international databases, such as those related to the non-academic influence of publications. This is especially important for both the Humanities and the Social Sciences, where the scientific results published in articles indexed in international databases are, in proportion, much lower than in other areas.
Although the inclusion of various sources was helpful in meeting the proposed objectives, this study is not without limitations. First, the use of an institutional database requires careful, thorough information processing, because a significant proportion of documents is included directly in the database by researchers themselves. This circumstance highlights the need to guarantee data integrity, consistency, quality and completeness in systems of this type [55]. Second, it should be taken into account that this study focused on publications with DOIs. The proportion of articles without a DOI identifier is minimal, but in the case of books and book chapters, the number of documents with DOIs is much lower, so, with the sources used, a complete picture of books and book chapters cannot be obtained.
This study is a starting point for future research to explore and assess several tools and sources of information for detecting the influence of HSS research on different audiences. As suggested by Yang et al. [4], altmetrics (even considering their drawbacks [66]) could be used as a complement to scientific impact assessment and to inform peer reviews, as it is recognised that the outcome of the research under review has attracted some attention outside the scientific community. A systematic, comprehensive analysis will also facilitate the proposal of new metrics applied to the evaluation of research in this field, highlighting the importance of the Humanities and Social Sciences for society as a whole.

Author Contributions

Conceptualization, D.D.F., F.M. and B.G.-A.; methodology, D.D.F., F.M. and B.G.-A.; software, D.D.F., F.M. and B.G.-A.; validation, D.D.F., F.M. and B.G.-A.; investigation, D.D.F., F.M. and B.G.-A.; data curation, B.G.-A.; writing—original draft preparation, D.D.F.; writing—review and editing, D.D.F., F.M. and B.G.-A.; visualization, D.D.F., F.M. and B.G.-A.; supervision, D.D.F. and F.M.; funding acquisition, D.D.F. All authors have read and agreed to the published version of the manuscript.

Funding

This work has been developed in the framework of the research project “New metrics for the evaluation of open science activities and their application to CSIC” (CSIC-202210I192). It has also received funding from the Spanish Ministry of Science and Innovation through the DOSSUET project (Diagnosis of open science in the Spanish university system and mechanisms for its transformation and improvement), grant number PID2019-104052RB-C21.

Data Availability Statement

Additional data to those presented in this study are available on request from the corresponding author. Raw data is not publicly available due to copyright restrictions.

Acknowledgments

The authors acknowledge the Metric Information Studies Laboratory (LEMI-UC3M) for the script for obtaining indicators from Altmetric.com. Thanks are also due to José Luis Ortega, a researcher at the Institute of Advanced Social Studies (IESA-CSIC), who provided the Google Scholar citations of all documents with DOIs. The authors would also like to acknowledge the support of the rest of the ACUTE team, María Bordons and Luz Moreno.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A. Institutes in the HSS Area

Area 1 of HSS consists of 16 institutes:
  • School of Arabic Studies (EEA).
  • Spanish School of History and Archaeology (EEHAR).
  • Institution Milá I Fontanals (IMF).
  • Institute for Economic Analysis (IAE).
  • Archaeology Institute of Merida (IAM).
  • Institute of Heritage Sciences (INCIPIT).
  • Institute of Economics, Geography and Demography (IEGD).
  • Padre Sarmiento Institute of Galician Studies (IEGPS).
  • Institute of Advanced Social Studies (IESA).
  • Institute of Philosophy (IFS).
  • Institute for Innovation and Knowledge Management (INGENIO).
  • Institute of History (IH).
  • Institute of Language, Literature and Anthropology (ILLA).
  • Institute of Languages and Cultures of the Mediterranean and the Near East (ILC).
  • Institute for Public Goods and Policies (IPP).
  • School of Hispano-American Studies (EEHA).
  • Centre for Human and Social Sciences (CCHS).
The production of the now extinct Institute of Islamic Studies and the Near East (IEIOP) and the López Piñero Institute for the History of Medicine and Science has also been considered in the period analysed.
In 2020, Area 1 had 316 scientists, 92 support technicians and 57 training researchers, who participated in 237 national research projects, 30 Horizon 2020 projects and 89 contracts and agreements with companies and institutions (CSIC, 2021).

Appendix B. Bibliometric Indicators

Publications 11 00031 g0a1 550
Figure A1. Comparison of impact according to conCIENCIA vs. impact according to Google Scholar.
Figure A1. Comparison of impact according to conCIENCIA vs. impact according to Google Scholar.
Publications 11 00031 g0a1

Appendix C. Relationship between Variables

Table
Table A1. Logistic regressions for the presence/absence of influence with some interactions between variables.
Table A1. Logistic regressions for the presence/absence of influence with some interactions between variables.
Model 1Model 2: Type * Social AreaModel 3: Type * GS Quartile
BS.E.Sig.ORBS.E.Sig.ORBS.E.Sig.OR
English Lang.0.9460.161<0.0012.5760.9440.163<0.0012.5700.9630.162<0.0012.620
WoS-art vs.
 C-art−0.9120.134<0.0010.402−1.2010.175<0.0010.301−1.5310.339<0.0010.216
 C-book−0.7800.207<0.0010.458−0.9260.249<0.0010.396−0.6590.495 0.518
OA0.4670.126<0.0011.5960.4790.127<0.0011.6150.4700.126<0.0011.599
GS Quartile0.4860.057<0.0011.6250.4850.057<0.0011.6250.4200.071<0.0011.521
Internat.Coll0.3600.123<0.011.4330.3430.123<0.011.4090.3610.123<0.011.435
Social Area0.7330.120<0.0012.0810.5040.149<0.0011.6550.7200.120<0.0012.055
Recent year0.1690.054<0.011.1840.1780.054<0.0011.1950.1640.054<0.011.178
Type * Social Area
 C-art by Social Area 0.7010.265<0.012.016
 C-book by Social Area 0.3760.443 1.456
Type * GS Quartile
 C-art by GS Quartile 0.2470.122<0.051.280
 C-book by GS Quartile −0.0610.180 0.941
Constant−343.422108.403<0.010.000−361.212108.961<0.0010.000−333.023108.706<0.010.000
Nagelkerke R Square0.363 0.367 0.365
Note: p-value is shown only when there is a significant difference. “*” means interaction between variables.

Notes

1
Five percent of all the documents included in this study were randomly reviewed, and mistakes were found in around 14% of the Google Scholar citations. In fact, half of the books and book chapters contained citation errors, possibly because the researchers themselves entered the data manually.
2
Ninety percent of the articles indexed in WoS were included in conCIENCIA. The remaining 10% were articles that had not been assigned to a specific CSIC institution or were pending administrative review for inclusion in the institutional database.
3
There were thirty-four documents in conCIENCIA that appeared in two languages at the same time (English and another language).
4
There were six C-art items for which access type was not stated.

References

  1. Garfield, E. Citation indexes for science. A new dimension in documentation through association of ideas. Science 1955, 122, 108–111. [Google Scholar] [CrossRef] [PubMed]
  2. Waltman, L. A review of the literature on citation impact indicators. J. Inf. 2016, 10, 365–391. [Google Scholar] [CrossRef]
  3. Priem, J. Altmetrics. In Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact; Conin, B., Sugimoto, C.R., Eds.; The MIT Press: Cambridge, MA, USA, 2014. [Google Scholar] [CrossRef]
  4. Yang, S.; Zheng, M.; Yu, Y.; Wolfram, D. Are Altmetric.com scores effective for research impact evaluation in the social sciences and humanities? J. Inf. 2021, 15, 101120. [Google Scholar] [CrossRef]
  5. Bornmann, L. Measuring the societal impact of research. EMBO Rep. 2012, 13, 673–676. [Google Scholar] [CrossRef] [PubMed]
  6. DORA. San Francisco Declaration on Research Assessment. 2012. Available online: https://sfdora.org/read/ (accessed on 6 June 2022).
  7. ENRESSH. CA15137—European Network for Research Evaluation in the Social Sciences and the Humanities (ENRESSH). 2015. Available online: https://www.cost.eu/actions/CA15137/ (accessed on 7 June 2022).
  8. Hicks, D.; Wouters, P.; Waltman, L.; de Rijcke, S.; Rafols, I. Bibliometrics: The Leiden Manifesto for research metrics. Nature 2015, 520, 429–431. [Google Scholar] [CrossRef] [PubMed]
  9. Wilsdon, J.; Allen, L.; Belfiore, E.; Campbell, P.; Curry, S.; Hill, S.; Jones, R.; Kain, R.; Kerridge, S.; Thelwall, M.; et al. The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management; SAGE Publications: Newbury Park, CA, USA, 2015. [Google Scholar] [CrossRef]
  10. Hicks, D. The four literatures of social science. In Handbook of Quantitative Science and Technology Research; Moed, H., Ed.; Kluwer Academic: Dordrecht, The Netherlands, 2004; pp. 473–496. [Google Scholar]
  11. Nederhof, A.J. Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review. Scientometrics 2006, 66, 81–100. [Google Scholar] [CrossRef]
  12. Kulczycki, E.; Engels, T.; Pölönen, J.; Bruun, K.; Duskova, M.; Guns, R.; Nowotniak, R.; Petr, M.; Sivertsen, G.; Starčič, A.; et al. Publication patterns in the social sciences and humanities: Evidence from eight European countries. Scientometrics 2018, 26, 3. [Google Scholar] [CrossRef]
  13. Jonker, H.; Vanlee, F.; Ysebaert, W. Societal impact of university research in the written press: Media attention in the context of SIUR and the open science agenda among social scientists in Flanders, Belgium. Scientometrics 2022, 127, 7289–7306. [Google Scholar] [CrossRef]
  14. Toledo, E.G. La evaluación de las Humanidades y de las Ciencias Sociales en revisión. Rev. Española Doc. Cient. 2018, 41, e208. [Google Scholar] [CrossRef]
  15. Sīle, L.; Pölönen, J.; Sivertsen, G.; Guns, R.; Engels, T.C.; Arefiev, P.; Dušková, M.; Faurbæk, L.; Holl, A.; Kulczycki, E.; et al. Comprehensiveness of national bibliographic databases for social sciences and humanities: Findings from a European survey. Res. Eval. 2018, 27, 310–322. [Google Scholar] [CrossRef]
  16. Bonaccorsi, A.; Daraio, C.; Fantoni, S.; Folli, V.; Leonetti, M.; Ruocco, G. Do social sciences and humanities behave like life and hard sciences? Scientometrics 2017, 112, 607–653. [Google Scholar] [CrossRef]
  17. Zhou, P.; Thijs, B.; Glänzel, W. Is China also becoming a giant in social sciences? Scientometrics 2008, 79, 593–621. [Google Scholar] [CrossRef]
  18. Van Noorden, R. Online collaboration: Scientists and the social network. Nat. News 2014, 512, 126–129. [Google Scholar] [CrossRef] [PubMed]
  19. Mohammadi, E.; Thelwall, M. Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows. J. Assoc. Inf. Sci. Technol. 2014, 65, 1627–1638. [Google Scholar] [CrossRef]
  20. Chen, K.-H.; Tang, M.-C.; Wang, C.-M.; Hsiang, J. Exploring alternative metrics of scholarly performance in the social sciences and humanities in Taiwan. Scientometrics 2015, 102, 97–112. [Google Scholar] [CrossRef]
  21. Zahedi, Z.; Costas, R.; Wouters, P. How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications. Scientometrics 2014, 101, 1491–1513. [Google Scholar] [CrossRef]
  22. Costas, R.; Zahedi, Z.; Wouters, P. Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective. J. Assoc. Inf. Sci. Technol. 2015, 66, 2003–2019. [Google Scholar] [CrossRef]
  23. De Filippo, D.A.; Sanz-Casado, E. Bibliometric and Altmetric Analysis of Three Social Science Disciplines. Front. Res. Metrics Anal. 2018, 3, 34. [Google Scholar] [CrossRef]
  24. Neylon, C.; Wu, S. Article-Level Metrics and the Evolution of Scientific Impact. PLoS Biol. 2009, 7, e1000242. [Google Scholar] [CrossRef]
  25. Priem, J.; Hemminger, M. Scientometrics 2.0: Toward new metrics of scholarly impact in the social web. First Monday 2010, 15. [Google Scholar] [CrossRef]
  26. Haustein, S.; Costas, R.; Larivière, V. Characterizing Social Media Metrics of Scholarly Papers: The Effect of Document Properties and Collaboration Patterns. PLoS ONE 2015, 10, e0120495. [Google Scholar] [CrossRef]
  27. Ortega, J.L. Relationship between altmetric and bibliometric indicators across academic social sites: The case of CSIC’s members. J. Inf. 2015, 9, 39–49. [Google Scholar] [CrossRef]
  28. Sugimoto, C.R.; Work, S.; Larivière, V.; Haustein, S. Scholarly use of social media and altmetrics: A review of the literature. J. Assoc. Inf. Sci. Technol. 2017, 68, 2037–2062. [Google Scholar] [CrossRef]
  29. Torres-Salinas, D.; Cabezas-Clavijo, A.; Jiménez-Contreras, E. Altmetrics: New indicators for scientific communication inWeb 2.0. Comunicar 2013, 41, 53–60. [Google Scholar] [CrossRef]
  30. Robinson-García, N.; Torres-Salinas, D.; Zahedi, Z.; Costas, R. Nuevos datos, nuevas posibilidades: Revelando el interior de Altmetric.com. Prof. Inf. 2014, 23, 4. [Google Scholar]
  31. Fang, Z.; Costas, R.; Tian, W.; Wang, X.; Wouters, P. An extensive analysis of the presence of altmetric data for Web of Science publications across subject fields and research topics. Scientometrics 2020, 124, 2519–2549. [Google Scholar] [CrossRef]
  32. Ortega, J.L. The life cycle of altmetric impact: A longitudinal study of six metrics from PlumX. J. Inf. 2018, 12, 579–589. [Google Scholar] [CrossRef]
  33. Gregorio-Chaviano, O.; Repiso, R.; Calderón-Rehecho, A.; León-Marín, J.; Jiménez-Contreras, E. Dialnet Métricas como herramienta de evaluación bibliométrica: Aportes al análisis de la actividad científica en Ciencias Sociales y Humanidades. Prof. Inf. 2021, 30. [Google Scholar] [CrossRef]
  34. Calderón-Rehecho, A. ¿Qué relevancia tiene Dialnet Métricas en las ciencias sociales y humanidades? Anu. ThinkEPI 2022, 16. [Google Scholar] [CrossRef]
  35. Robinson-Garcia, N.; van Leeuwen, T.N.; Ràfols, I. Using altmetrics for contextualised mapping of societal impact: From hits to networks. Sci. Public Policy 2018, 45, 815–826. [Google Scholar] [CrossRef]
  36. De Filippo, D.; Serrano-López, A.E. From academia to citizenry. Study of the flow of scientific information from projects to scientific journals and social media in the field of “Energy saving”. J. Clean. Prod. 2018, 199, 248–256. [Google Scholar] [CrossRef]
  37. Cabezas-Clavijo, Á.; Torres-Salinas, D. Indicadores de uso y participación en las revistas científicas 2.0: El caso de PLoS ONE. Prof. Inf. 2010, 19, 4. [Google Scholar] [CrossRef]
  38. Schloegl, C.; Gorraiz, J. Comparison of citation and usage indicators: The case of oncology journals. Scientometrics 2010, 82, 567–580. [Google Scholar] [CrossRef]
  39. Eysenbach, G. Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact. J. Med. Internet Res. 2011, 13, e123. [Google Scholar] [CrossRef] [PubMed]
  40. Serrano-López, A.E.; Ingwersen, P.; Sanz-Casado, E. Wind power research in Wikipedia: Does Wikipedia demonstrate direct influence of research publications and can it be used as adequate source in research evaluation? Scientometrics 2017, 112, 1471–1488. [Google Scholar] [CrossRef]
  41. Gumpenberger, C.; Glänzel, W.; Gorraiz, J. The ecstasy and the agony of the altmetric score. Scientometrics 2016, 108, 977–982. [Google Scholar] [CrossRef]
  42. Orduña-Malea, E.; Martín-Martín, A.; Delgado-López-Cózar, E. La bibliometría que viene: ALMetrics (Author Level Metrics) y las múltiples caras del impacto de un autor. Prof. Inf. 2016, 25, 3. [Google Scholar]
  43. Martín-Martín, A.; Orduna-Malea, E.; López-Cózar, E.D. Author-level metrics in the new academic profile platforms: The online behaviour of the Bibliometrics community. J. Inf. 2018, 12, 494–509. [Google Scholar] [CrossRef]
  44. Sugimoto, C. “Attention Is Not Impact” and Other Challenges for Altmetrics. The Wiley Network. 2015. Available online: https://www.wiley.com/en-us/network/publishing/research-publishing/promoting-your-article/attention-is-not-impact-and-other-challenges-for-altmetrics (accessed on 12 September 2022).
  45. Thelwall, M.; Nevill, T. Could scientists use Altmetric.com scores to predict longer term citation counts? J. Informetr. 2018, 12, 237–248. [Google Scholar] [CrossRef]
  46. Torres-Salinas, D.; Romero, E. InfluScience: Científicos y científicas socialmente influyentes [Project PID2019-109127RB-I00]. InfluScience. 2019. Available online: https://influscience.eu/ (accessed on 15 June 2022).
  47. Szomszor, M.; Adie, E. Overton—A bibliometric database of policy document citations. arXiv 2022, arXiv:2201.07643. [Google Scholar] [CrossRef]
  48. De Filippo, D.; Sastrón-Toledo, P. Influence of research on open science in the public policy sphere. Scientometrics 2023, 128, 1995–2017. [Google Scholar] [CrossRef]
  49. Tahamtan, I.; Bornmann, L. Altmetrics and societal impact measurements: Match or mismatch? A literature review. Prof. Inf. 2020, 29. [Google Scholar] [CrossRef]
  50. Torres-Salinas, D.; Robinson-García, N.; Arroyo-Machado, W. Coverage and distribution of altmetric mentions in Spain: A cross-country comparison in 22 research fields. Prof. Inf. 2022, 31, 2. [Google Scholar] [CrossRef]
  51. CSIC. Guía de Uso del Sistema de Información de Contribución Científica del CSIC. conCIENCIA (v.2). [Madrid]: CSIC. 2012. Available online: http://documenta.sitios.csic.es/alfresco/download/d/workspace/SpacesStore/8f47c5ea-9201-437a-bafa-f34de61669aa/Manual_Ayuda_conCIENCIA.pdf;jsessionid=0E4AB0D87A4B5CBFD0B453CDA0D671EB (accessed on 1 April 2022).
  52. Morillo, F.; Santabárbara, I.; Aparicio, J. The automatic normalisation challenge: Detailed addresses identification. Scientometrics 2013, 95, 953–966. [Google Scholar] [CrossRef]
  53. CSIC. Manual de la Productividad por Cumplimiento de Objetivos (PCO); Vicepresidencia de Investigación Científica y Técnica: Madrid, Spain, 2017. [Google Scholar]
  54. Bordons, M.; Morillo, F.; Moreno-Solano, L.; Gil Sánchez, J.; González-Albo, B. La Actividad Científica del CSIC a Través de Indicadores Bibliométricos (WoS 2016–2020); CSIC, CCHS-IFS: Madrid, Spain, 2021. [Google Scholar] [CrossRef]
  55. Sivertsen, G. Developing Current Research Information Systems (CRIS) as Data Sources for Studies of Research. In Springer Handbook of Science and Technology Indicators; Glänzel, W., Moed, H.F., Schmoch, U., Thelwall, M., Eds.; Springer: Cham, Switzerland, 2019. [Google Scholar] [CrossRef]
  56. Pölönen, J.; Laakso, M.; Guns, R.; Kulczycki, E.; Sivertsen, G. Open access at the national level: A comprehensive analysis of publications by Finnish researchers. Quant. Sci. Stud. 2020, 1, 1396–1428. [Google Scholar] [CrossRef]
  57. Clarivate Analytics (2022), Web of Science Database. Available online: https://www.webofscience.com/wos/woscc/advanced-search (accessed on 19 October 2022).
  58. Ossenblok, T.L.B.; Engels, T.C.E.; Sivertsen, G. The representation of the social sciences and humanities in the Web of Science--a comparison of publication patterns and incentive structures in Flanders and Norway (2005–9). Res. Eval. 2012, 21, 280–290. [Google Scholar] [CrossRef]
  59. De Filippo, D.; Aleixandre-Benavent, R.; Sanz-Casado, E. Toward a classification of Spanish scholarly journals in social sciences and humanities considering their impact and visibility. Scientometrics 2020, 125, 1709–1732. [Google Scholar] [CrossRef]
  60. Mohammadi, E.; Thelwall, M.; Kwasny, M.; Holmes, K.L. Academic information on Twitter: A user survey. PLoS ONE 2018, 13, e0197265. [Google Scholar] [CrossRef]
  61. Haustein, S.; Larivière, V.; Thelwall, M.; Amyot, D.; Peters, I. Tweets vs. Mendeley readers: How do these two social media metrics differ? IT Inf. Technol. 2014, 56, 207–215. [Google Scholar] [CrossRef]
  62. Thelwall, M. Altmetric Prevalence in the Social Sciences, Arts and Humanities: Where are the Online Discussions? J. Altmetrics 2018, 1, 4. [Google Scholar] [CrossRef]
  63. Taylor, M. An altmetric attention advantage for open access books in the humanities and social sciences. Scientometrics 2020, 125, 2523–2543. [Google Scholar] [CrossRef]
  64. Pinheiro, H.; Vignola-Gagné, E.; Campbell, D. A large-scale validation of the relationship between cross-disciplinary research and its uptake in policy-related documents, using the novel Overton altmetrics database. Quant. Sci. Stud. 2021, 2, 616–642. [Google Scholar] [CrossRef]
  65. Fang, Z.; Dudek, J.; Noyons, E.; Costas, R. Science Cited in Policy Documents: Evidence from the Overton Database. 2020. Available online: http://altmetrics.org/wp-content/uploads/2020/11/02_submission_Fang_Dudek_Noyons_Costas-altmetrics20.pdf (accessed on 23 October 2022).
  66. Erdt, M.; Nagarajan, A.; Sin, S.-C.J.; Theng, Y.-L. Altmetrics: An analysis of the state-of-the-art in measuring research impact on social media. Scientometrics 2016, 109, 1117–1166. [Google Scholar] [CrossRef]
Publications 11 00031 g001 550
Figure 1. Distribution of the scientific production in HSS.
Figure 1. Distribution of the scientific production in HSS.
Publications 11 00031 g001
Publications 11 00031 g002 550
Figure 2. Indicators of scientific activity, accessibility and collaboration by documental typology in Humanities vs. Social Sciences (percentages).
Figure 2. Indicators of scientific activity, accessibility and collaboration by documental typology in Humanities vs. Social Sciences (percentages).
Publications 11 00031 g002
Publications 11 00031 g003 550
Figure 3. Indicators of impact by documental typology in Humanities vs. Social Sciences.
Figure 3. Indicators of impact by documental typology in Humanities vs. Social Sciences.
Publications 11 00031 g003
Publications 11 00031 g004 550
Figure 4. Indicators of influence by documental typology in Humanities vs. Social Sciences (percentages).
Figure 4. Indicators of influence by documental typology in Humanities vs. Social Sciences (percentages).
Publications 11 00031 g004
Publications 11 00031 g005 550
Figure 5. Probability (odds ratio) for the presence/absence of influence based on document characteristics.
Figure 5. Probability (odds ratio) for the presence/absence of influence based on document characteristics.
Publications 11 00031 g005
Publications 11 00031 g006 550
Figure 6. Predicted probabilities for the presence/absence of influence based on different interactions between variables.
Figure 6. Predicted probabilities for the presence/absence of influence based on different interactions between variables.
Publications 11 00031 g006
Table
Table 1. Stages of this study, dimensions, indicators and sources used.
Table 1. Stages of this study, dimensions, indicators and sources used.
StageDimensionIndicatorsSource
Bibliometric analysisScientific activityNo. of docs. by documental typologyWeb of science/conCIENCIA
% of docs. by language
Thematic specializationNo. of docs. in HSS area
% of docs. in Humanities
% of docs. in Social Sciences
International collaboration% of docs. in international collaboration
Accessibility% of docs. in open accessUnpaywall
Impact% of docs. with citationsGoogle Scholar
% of docs. with low, medium and high impactconCIENCIA
Altmetric analysisSocial influence% of docs. with mentions on TwitterAltmetric.com
Media influence% of docs. with mentions in media
Informative influence% of docs. with mentions in Wikipedia
Political influence% of docs. with mentions in policy documentsOverton
Relationships among variablesProbability of having influenceDifferent independent variables: scientific activity, thematic specialization, international collaboration, accessibility, impact by Google Scholar and publication yearAll
Table
Table 2. Comparisons of bibliometric indicators by documental typology (percentages in columns).
Table 2. Comparisons of bibliometric indicators by documental typology (percentages in columns).
Dimension/IndicatorsWoS-ArtC-ArtC-Book
Language
English85.6352.7279.74
Spanish13.9748.3314.38
International collaboration
With collaboration51.5223.7324.84
Accessibility
With open access64.4772.4136.60
Impact according to conCIENCIA
Low3.8519.8611.76
Medium19.0331.465.88
High72.8723.9077.78
No information4.2524.784.58
Impact in Google Scholar
Cited docs.84.4152.0262.09
Total docs.988569153
Table
Table 3. Comparisons of influence indicators by documental typology (percentages in columns).
Table 3. Comparisons of influence indicators by documental typology (percentages in columns).
% of Docs. with Mentions
InfluenceSourceWoS-ArtC-ArtC-Book
SocialTwitter58.9122.6731.37
MedialMedia7.591.933.92
InformativeWikipedia3.342.287.84
PoliticalOverton13.775.803.92
Total 988569153
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

De Filippo, D.; Morillo, F.; González-Albo, B. Measuring the Impact and Influence of Scientific Activity in the Humanities and Social Sciences. Publications 2023, 11, 31. https://doi.org/10.3390/publications11020031

AMA Style

De Filippo D, Morillo F, González-Albo B. Measuring the Impact and Influence of Scientific Activity in the Humanities and Social Sciences. Publications. 2023; 11(2):31. https://doi.org/10.3390/publications11020031

Chicago/Turabian Style

De Filippo, Daniela, Fernanda Morillo, and Borja González-Albo. 2023. "Measuring the Impact and Influence of Scientific Activity in the Humanities and Social Sciences" Publications 11, no. 2: 31. https://doi.org/10.3390/publications11020031

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop