How Efficiently Does the EU Support Research and Innovation in SMEs?

Abstract

The European Regional Development Fund devoted around 66 billion Euros to the financial support of innovation and productivity in European enterprises over the 2014–2020 programming period. In this framework, we assessed the implementation of the Operational Programmes dedicated to fostering research and innovation, particularly in small and medium-sized enterprises. With this aim, we used a network slack-based data envelopment analysis model paired with cluster analysis that encompasses a multitude of performance framework indicators to assess 53 Operational Programmes from 19 countries. Our findings suggest that compared to transition and less developed regions, more developed regions present a higher room for improvement. Also, less developed regions present a better performance when they employ their funding against more developed regions, suggesting that further funding should be channelled for leveraging research and innovation in the former regions. Finally, Operational Programme managers should focus on solving the problems both inherent to the poor outcomes in terms of enhancing the number of researchers working in improved research infrastructures and promoting the technology transfer between research institutions and enterprises.

1. Introduction

There is a rising awareness of the significance that innovation can play in both productivity growth and economic wellbeing [1,2,3,4,5].

Small and Medium-sized Enterprises (SMEs) are the cornerstone of the European economy, matching 99% of all enterprises, and generate around 100 million jobs in the European Union (EU). Being accountable for more than half of the EU’s Gross Domestic Product (GDP), any upgrading in their innovation can have an important impact on the EU Research and Innovation (R&I) scenery [6].

The quest for a solution to the challenges involved in the enhancement of enterprises’ innovation capacity is not new [7]. This is particularly pertinent since it is believed that a company’s success can be determined by its capacity to innovate [8,9,10]. Decades of research exploring the elements that drive this process led to a wide range of perspectives on innovation capacity [11]. Overall, three main sets of components can particularly affect innovation capacity in enterprises [11]: human resources and internal and external dimensions.

A paradigm shift in how innovation might be viewed consists of open innovation. In effect, open innovation in SMEs has been drawing the attention of many scholars over the years [12]. Through this form of innovation, firms may receive knowledge inflow and outflow to improve their innovation capabilities [13]. A few scholars have further developed this concept, realising that data and knowledge interactions can occur in three forms [14,15,16]: (a) incoming (outside-in), associated with the collection of knowledge from the external environment to generate innovations within the organization; (b) combined, associated with co-generation; and (c) outcoming (inside-out), represented by knowledge exchange. The study of open innovation generally considers distinct rationales, including the triple helix framework of collaboration linkages between universities, industries, and governments [17,18,19], governance models [20,21], and absorptive capacity [22,23].

Thomä and Zimmermann [24] proposed a valuable framework for organising the way SMEs innovate. These authors differentiated two types of innovation. One is based on Research and Development (R&D) carried out by specific departments or individuals within a company to generate new scientific knowledge. External collaborations or scientific organizations are used to supplement it as necessary. This method, known as the ‘Science, Technology, and Innovation’ (STI) mode, needs financing to establish a connection between profitability and innovation. The second approach is focused on experience, learning-by-Doing, and Using and Interacting (DUI). Informal techniques of learning and comprehension predominate in this case. Employees that work closely together generate innovations, as does the company environment through interaction with customers, suppliers, and other stakeholders. In effect, businesses do not entirely operate in either mode but rather use a combination of the two, with the balance dictated by the sort of business and the hurdles it faces.

SMEs are confronted with several practical hurdles when it comes to innovation. Access to external capital may be difficult for SMEs, especially when risky projects are at stake [25,26,27]. The degree to which this is a barrier varies according to the age of the company, business size, research intensity, growth orientation [28], and, in many cases, geography [29]. Other barriers may involve the difficulty in employing highly skilled workers [30,31,32,33,34], leadership problems [35], reduced absorptive capacity [36] and difficulties in the capture of value [37]. Nevertheless, financial obstacles are seen as the most significant impediments to innovation [38].

SMEs seem to have a much lower propensity and capability to create internal R&D than bigger corporations. They are, in principle, significantly more reliant on public R&D support to promote their in-house R&I capacity [39]. In this context, public policy has increased both financial and non-financial assistance to SMEs in response to the hurdles they must face while attempting to innovate.

The framework and basis for EU policy on SMEs started with the European Charter for Small Enterprises proposal by the General Affairs Council in 2000 and its endorsement by the EU Council at Fiera. It was subsequently backed by a comprehensive SME policy with the implementation of the Small Business Act (SBA) in 2008 [40]. The SBA established ten principles that include initiatives to strengthen SMEs, such as enabling funding, improving access to procurement procedures, and fostering female-led start-ups. The Europe 2020 Strategy, which was published in 2010, contains seven Flagship Initiatives, one of which is the achievement of the Innovation Union. Since then (for the 2014–2020 multiannual financial framework), a broad range of SME programs have been developed, many of which are aimed at fostering innovation.

The European Regional Development Fund (ERDF) offered about 66 billion Euros of European Union (EU) financial support to promote innovation and productivity in European enterprises during the 2014–2020 programming period [41]. Specifically, SMEs were the core of the ERDF financial assistance to enterprises.

As of 2014, member states (MS) were required to assess the efficacy, efficiency, and impacts of these financial assistance programmes. Despite the existing abundance of publications for assessing them (see Section 2), there are still issues that lack scholarly attention, especially when assessments occur over the planning periods. Ortiz and Fernandez [42] highlighted that the design and implementation of R&I policies still present significant obstacles for policymakers. This is particularly true in the monitoring and evaluation stages, mainly because of the lack of relevant data, benchmarking assessment studies, and administrative skills/capabilities. Indeed, monitoring processes during the 2014–2020 programming period have placed a high focus on assessing process-oriented outcomes (called herein “procedural efficiency”), lacking data on metrics to evaluate the direct impacts of the investments supported [42].

Additionally, in the case of R&I policies, the evaluation procedure assumes a prominent role in helping countries and/or regions in the improvement of future policy instruments by recognizing the accomplishments and failures in the preceding policy stage [43].

In this framework, policymakers may wish to set the appropriate policies to make the necessary adjustments while completing this type of evaluation using the Data Envelopment Analysis (DEA) approach. In this way, it is possible to obtain insightful information, which allows for correcting possible deviations from best practices. When compared to other approaches (for example, macroeconomic evaluations, microeconomic studies that use control groups, and case study evaluations), the DEA model used herein can be especially useful for management authorities (MA). This modelling approach makes it possible to identify the benchmarks and modifications that must be made to enhance the execution of this type of funding while also grasping the disparities between the different regions’ categories.

Since the EU provides significant funding for R&I activities in European MS, and SMEs are the basis of the European economy, we have devoted our evaluation to the programmes subsidised by the ERDF, mainly dedicated to R&I in SMEs.

In this regard, our work is novel in four ways: (1) It proposes the usage of the Network Slack-Based Measure (SBM) DEA model in conjunction with cluster analysis to appraise 53 Operational Programmes (OPs) from 19 EU countries dedicated to fostering R&I in SMEs; (2) It separates efficiency analysis into two stages: the procedural efficiency, mainly concerned with OPs’ implementation, and the potential R&I capacity generation obtained with the funds spent; (3) It makes it possible to assess this sort of funding across the programming period, allowing for the implementation of policy actions to address detected inefficiencies, thus offering supplementary evidence that can be useful in the intermediate monitoring phases of the OPs. (4) It addresses the diverse regional characteristics of the OPs, according to the most recent regional development NUTS2 categorization, which was used to allocate the ERDF for the 2014–2020 programming period. Therefore, it considers less developed regions (GDP lower than 75% of EU GDP), regions of transition (GDP lower than 90% and higher than 75% of EU GDP), and more developed regions (GDP higher than 90% of EU GDP), which are categorized into three separate clusters.

All in all, the main research questions posed in this are:

RQ1. “Does a higher level of procedural efficiency necessarily lead to a higher R&I potential capacity generation?”

RQ2. “Which performance framework indicators preclude the efficient application of funds committed to boosting R&I in EU SMEs?”

RQ3. “Which performance framework indicators showed higher resilience to efficiency classification in face of their potential shifts”?

RQ4. “Which OPs were most frequently considered as a reference of best practices over the programming period under scrutiny?”

RQ5. “Which type of regions showed a higher performance in supporting R&I in SMEs?”

After the previous Introduction, this paper has the following structure. Section 2 provides a literature review on cohesion policy assessments of R&I in SMEs and other enterprises. Section 3 defines the underpinning assumptions related to the methodology applied to assess the execution and outcomes of the OPs under consideration. Section 4 covers the key considerations followed for choosing the inputs and outputs utilised in the efficiency assessment of OPs and some statistics on the data that instantiated the model. Section 5 analyses the findings and considers possible policy suggestions. Section 6 delivers the main conclusions and suggests potential recommendations based on the findings obtained.

2. Literature Review

More than 1000 evaluations have been performed by EU MS since 2015, focusing on specific funds, topics, and regions, monitoring the implementation progress, and/or evaluating the effect of initiatives, both for the 2007–2013 and 2014–2020 programming periods [40].

The frequency of assessments performed by MS varies greatly [40]. This is due to significant differences in the quantity and type of investment financing, the number of OPs in each MS, and the methodology proposed in the evaluation plans [40]. Furthermore, some countries choose to undertake many minor assessments, whilst others prefer to do aggregate assessments.

Most of these evaluations are usually centred on implementation issues and assess target accomplishment, with the primary concern being the consistency of projects and activities with programmes’ objectives, as well as the effectiveness and efficiency of their execution. They also examine whether current funds are being spent and whether the specified objectives, particularly those of the performance framework, are being met. The impact evaluations are carried out later in the programme cycle when most activities have already taken place.

Since our study will assess the efficiency of OPs implementation within the context of Thematic Objective (TO) 1, we have outlined the OP evaluations completed by EU MS from 2015 to date that explicitly mention R&I in their abstract description (see

Table A1. List of cohesion policy programme evaluations completed by the Member States mentioning R&I concerns from 2015 to date (Available online: https://cohesiondata.ec.europa.eu/2014-2020-Evaluation/Cohesion-policy-programme-evaluations-2015-to-date/iz3t-u7bv) (accessed on 12 April 2022).

Title in English	Date of Publication	Authors	Methods (a)
			DR	LR	MD/DA	I	FG/FW	S	CS	B	E	SM	O
Impact evaluation of support for RDI and businesses under the Centre OP, 2014–2020, in France FR, Code: FRE103; ID: 1822	01/03/2021	Edater, Erdyn (Available online: www.edater.fr, accessed on 12 April 2022; Available online: www.erdyn.com (accessed on 12 April 2022)	X		X	X	X	X				X
Evaluation of measures to develop RDI capacity under the Kujawsko-Pomorskie OP, 2014–2020 PL, Code: PLE400; ID1812	01/03/2021	LB&E Sp. z o.o. Available online: www.lbie.eu, (accessed on 12 April 2022)			X	X	X	X
Evaluation of the impact of support for research under Cohesion policy in 2014–2020 on competitiveness and innovation in Poland ES, Code: ESE118; ID: 1064	01/02/2021	IDEA—PAG Uniconsult—Imapp consulting (Idea, Available online: www.ideaorg.eu (accessed on 12 April 2022); Available online: www.pag-uniconsult.pl (accessed on 12 April 2022); Available online: www.imapp.consulting (accessed on 12 April 2022)
Evaluation of the impact of support for research under Cohesion policy in 2014–2020 on competitiveness and innovation in Poland PL, Code: PLE395; ID1740	01/02/2021	IDEA—PAG Uniconsult—Imapp consulting, Available online: www.ideaorg.eu (accessed on 12 April 2022); Available online: www.pag-uniconsult.pl (accessed on 12 April 2022); Available online: www.imapp.consulting (accessed on 12 April 2022)			X	X			X				X
Impact evaluation of investment in education and RDI under the Structural Funds OP, 2014–2020 in Lithuania LT, Code: LTE39; ID: 1409	01/08/2020	PPMI Group, Available online: www.ppmi.lt, (accessed on 12 April 2022), info@ppmi.lt; Available online: www.vpvi.lt/en, (accessed on 12 April 2022)	X		X	X	X	X	X
Analysis of integration of higher education, research and technological innovation policies under the Toscana Ops, 2014–2020 ES, Code: ESE118; ID: 1064	01/03/2020	Massimo Bressan; Simone Bertini; Nicola Sciclone; collaborazione di Maria Luisa Maitino and Valentina Patacchini IRPET—Available online: http://www.irpet.it/, (accessed on 12 April 2022)			X								X
Evaluation of R&D and Innovation Potential of the Slovak Republic in 2007–2013 and 2014–2020 SK, Code: SKE24; 1267	01/02/2020	Grant Thornton Advisory s.r.o. and Visions Consulting, s.r.o. Available online: www.grantthornton.sk (accessed on 12 April 2022); Available online: www.visions.cc (accessed on 12 April 2022)	X		X
Evaluation of ERDF policy areas in France in preparation for the 2021–2027 programming period FR, Code:FRE86; ID:1613	01/01/2020	Strasbourg Conseil and l’ADIT, Teritéo & Taran Consulting, Technopolis & Edater, Vizéa, Teritéo & Oréade—Brèche, Spatial Foresight France, ASTER Europe Conseil and Rouge Vif Territoires	X		X	X	X	X				X
Monitoring and analysis of TO1 Research, Development and InnovationSpain, Code: ESE118; ID: 1064	01/10/2019	Ministry of Finance Available online: www.dgfc.sepg.hacienda.gob.es (accessed on 12 April 2022)	X		X
Evaluation of measures supporting business RDI under the EU Structural Funds Investments OP, 2014–2020 LT, Code: LTE38; ID: 1235	01/08/2019	UAB Visionary Analytics Available online: www.visionary.lt (accessed on 12 April 2022)			X	X	X	X	X			X	X
Evaluation of support for smart specialisation in RDI in Podlaskie under the regional OP, 2014–2020 PL, Code: PLE251; ID905	01/07/2019	Artur Kotliński, Maciej Maj, Kamil Pluta, Emilia Pokropińska, Jarosław Sawicki (biuro@instytut-ipc.pl; office@openfield.pl)	X		X	X	X	X	X			X
Interim evaluation of the Research and Innovation OP 2014–2020 SK, Code: SKE23; ID: 1266	01/03/2019	Grant Thornton Advisory s.r.o. and Visions Consulting, s.r.o.Available online: www.grantthornton.sk, (accessed on 12 April 2022); Available online: www.visions.cc (accessed on 12 April 2022)	X		X	X		X
Research and Innovation Strategy for Smart Specialisation in Tuscany IT, Code: ITE46; ID660	01/12/2017	SIRIS Academic info@sirisacademic.com	X		X
On-going evaluation of the implementation of action 1.2 on RDI under the regional OP Śląskie Voivodeship 2014-2020	01/03/2017	ECORYS Polska Sp. z o.o.	X			X	X	X				X	X
Progress evaluation and sectoral analysis by Thematic Objective RDI ES, Code: ESE25; 276	01/12/2016	Dirección General de Fondos Comunitarios (fondoscomunitarios@sepg.hacienda.gob.es)										X
Experience analysis and identification of good practices in the area of digital skills development support in the process of detailed implementation rules for OP Digital Poland 2014–2020 preparation as well as for coordination of thematic objective 2 PL, Code: PLE53; ID: 212	01/06/2016	Sonia Buchholtz, dr Anna Buchner, dr hab. prof. SWPS Mirosław Filiciak, dr Justyna Jasiewicz, Paweł Kabicz, dr Anna Mierzecka, Joanna Pospieszynska—Burzyńska, Piotr Szczerba, dr Alek Tarkowski, Rafał Trzeciakowski	X	X		X	X		X	X		X

^(a) Note: DR—desk research; LR—literature review; MD/DA—monitoring data/data analysis; I—interviews; FG/FW—focus groups/facilitated workshops; S—surveys; CS—case studies; B—benchmarking; E—expert consultation; SM—statistical methods; O—other.

included in Appendix A).

Even though the examination of Table A1 indicates that MS use systematic methods in their evaluations of TO 1, just a few (primarily dedicated to impact assessments) consider more robust methods (such as statistical techniques or other). Therefore, despite MS’ efforts to improve cohesion policy evaluation, this suggests there is still room for further improvement.

A search was conducted utilizing the Google Scholar database from 2017 to 2022 to obtain a clear idea of the several approaches usually used in assessing Cohesion Policy programmes, specifically dedicated to innovation in SMEs. The keywords “EU innovation SMEs *” and “EU fund innovation assessment *” were used to locate relevant papers. In the end, only renowned journals, highly cited books, and Joint Research Centre’s publications on the subject were retained.

From the literature review conducted, it was ascertained that there are several approaches to assessing the cohesion policy, each with its advantages and disadvantages [44]. The two most utilised methodologies to evaluate the impact of cohesion policy are macroeconomic and econometric models. In the vein of R&I macroeconomic impact assessment, Computable General Equilibrium Models combined with input-output analysis and econometric approaches are usually used (e.g., [45,46,47]). Despite enabling the study and analysis of the main effects of EU funding on economic development, these models cannot convey specific information on management performance [48]. Furthermore, they disregard the distribution of EU funds among distinct thematic objectives and activity sectors in each region or country. The dominant research stream is built on econometric analyses (e.g., [4,49,50,51,52,53]). However, it also yields conflicting results [54], leading various scholars to question its usage [54,55,56]. Other approaches can also be used, but with similar inherent limitations (e.g., [57,58]).

An alternative would be to explore ‘microeconomic studies’ that apply ‘control groups’ to compare the efficacy of Structural Fund (SF) beneficiaries against other homogeneous control groups (see Table A1 from the Appendix A). Finally, for the evaluation type that relies upon case study assessments, a blend of data involving surveys, OPs’ monitoring reports, and quantitative approaches are employed to study the effects of SFs at distinct levels [59].

In general, the assessment methods adopted do not enable contrasting any unit under assessment (either a country, region, or OP) with its counterparts. They do not allow identifying the changes that must take place to make an inefficient country, region, or programme efficient [60]. Furthermore, these approaches usually need the fulfilment of statistical assumptions (e.g., normality, lack of multicollinearity, and homoscedasticity). As a result, the use of nonparametric approaches can be especially helpful and suitable, specifically as performance framework indicators may be employed with nonparametric techniques. These convey a group of indicators in each OP according to which the European Commission, in collaboration with the MS, assesses the performance of the OPs in each MS.

In general, stochastic approaches are utilised to obtain the frontier production function [61]. However, the vast majority can only handle a single output [60]. Nonparametric approaches such as DEA can readily tackle several inputs and outputs and are also used to obtain the efficiency frontier. Besides, unlike stochastic approaches, DEA does not depart from any functional form of the production function or the estimation of the error term. DEA assumes that the higher the deviation from the efficient frontier, the higher the inefficiency of the unit under evaluation.

Furthermore, DEA can aid in the identification of the main reasons that preclude efficiency, offering decision-makers relevant insights on how to address them. In this context, [62] utilised DEA to examine the relative spatial disadvantages of the EU’s Level II areas. Gómez-García et al. [61] used employment and productivity rates as outputs, as well as the Stochastic Frontier and DEA approach to assess the pure technical efficiency and global technical efficiency of Objective 1 in the application of SFs in EU regions from 2000 to 2006. Anderson and Stejskal [63] used DEA to appraise the efficiency of diffusion of innovation of EU MS regarding their European Innovation Scoreboard rankings. Also, [60] used the Value-Based DEA methodology that couples DEA with Multiple Criteria Decision Aiding and considers the main factors that may impact the implementation efficiency of SFs in distinct EU regions and countries. More recently, [64] employed the SBM model coupled with cluster analysis for evaluating 102 OPs from 22 EU countries dedicated to the promotion of a Low-carbon economy in SMEs.

Despite its usefulness, the DEA method is underexplored in the assessment of the European cohesion policy devoted to R&I—see

Table 1. Studies on the assessment of R&I.

Authors	Main Purpose	Methodologies	Variables
Radicic and Pugh [49]	Assess the input and output additionality of national and EU R&D programmes both individually and in conjunction considering SMEs from 28 EU MS	Inverse probability weighted regression adjustment estimator	R&D personnel, R&D expenditure, Patents, Innovative sales, Export, Turnover, Age, Competition, Innovation capacity, Resources, R&D department, Micro firms, Small firms, Medium firms
Di Comite et al. [45]	Evaluate the system-wide economic impact of the strategic financial tools used by the EU (the SFs and the cohesion funds) for the accomplishment of the regional policy	RHOMOLO—Spatial computable general equilibrium model	Inter-regional transportation costs; Government investment; Government consumption; Entry costs to intermediates; Fixed costs final demand firms; Worker training; Shock to tangible capital cost; Shock to intangible capital cost.
Anderson and Stejskal [63]	Appraise how efficiently EU MS diffuse innovation regarding their European Innovation Scoreboard rankings	Charnes-Cooper-Rhodes DEA model	Inputs—New doctorate graduates, Lifelong learning, Opportunity-driven entrepreneurship, Public sector R&D expenditure, Venture capital investments, Private sector R&D expenditure, Non-R&D Innovation expenditure; Outputs—Employment in knowledge-intensive
			activities, Sales of new-to-the market and new-to-the-firm innovations
Santos et al. [50]	Identifying disparities in investment project features between enterprises with funded and non-funded applications, as well as understanding which types of projects are chosen for a subsidy	Counterfactual analysis and Propensity Score Matching estimators	Firm size; Micro or small firm; firm age; the number of partners; Foreign capital; Experience in the Portuguese innovation incentive system; Export intensity; Share of skilled jobs; R&D; Patented; Productivity; Regional export intensity; Time fixed effect; Sector fixed effect; Region fixed effect
Thum-Thysen et al. [51]	Explore whether the determinants of investment in intangible and tangible assets in Europe are different	Regression analysis with accelerator term	Data for investment in intangible and tangible assets, Capital stocks and proxies for the potential drivers and barriers (regulatory framework; availability of human capital; other forms of public intervention; financial conditions)
Bedu and Vanderstocken, [57]	Review the R&D assistance system established in Aquitaine, France’s leading region in the amount of total expenditure devoted to innovation	Coarsened exact matching model	Non-R&D personnel, Private R&D spending, R&D personnel, Technological intensity, Total Assets, Total Personnel, Turnover, Wages, Pre Technological intensity and the Pre Total personnel, Date of funding, High and Med/High Technology, Low and Med/Low Technology, Knowledge Intensive Services, Less Knowledge Intensive Services, Regional subsidy, Total subsidies, Non-regional subsidies, Pre R&D personnel, Pre Private R&D spending, Pre Technological intensity, Pre Total personnel, Pre Non-R&D personnel, Pre Turnover, Pre Total Assets, Pre wages
Fattorini et al. [52]	Analyse the effect of EU cohesion policy on European manufacturing companies, considering the heterogeneous distribution of company inefficiencies within a region and combining regional policies	Panel data methodology	Firm-level financial data for the period 2007–2015 from the ORBIS database and Total Factor Productivity growth as a dependent variable
Gustafsson et al. [58]	Study the short- and long-run impacts of public support in the shape of grants on venture expansion and consequent investment financing	Multi-level discontinuous growth model	Grants, Investment, Revenue, Venture size
Parrilli et al. [65]	Examine whether the application of innovation modes depends on regional assets and capabilities	Matching approach with multiple treatments	Leader and strong innovators, moderate innovators and modest innovators.
Stojčić et al. [4]	Examine the connection between public funding and public procurement for innovation and firm-level innovation output and outcome additionality in 8 Central and Eastern EU countries	One-to-one nearest neighbour matching and	Portion of sales coming from new-to-the-market and new-to-the-firm products; Portion of sales coming from new-to-the-firm or new-to-the-market products; Control variables: Medium firm, Large Firm, Patent application, Organizational Innovation, Marketing Innovation, Enterprise group, EU market, Other markets, Human capital
		the inverse probability weighted regression adjustment estimator
Sein and Prokop [53]	Assess if government financing and collaboration based on the triple-helix and quadruple-helix concepts directly or indirectly stimulate company product and process innovation, considering the purpose of firms’ R&D.	Partial least squares structural equation modelling	R&D, Triple-Helix/Quadruple-Helix cooperation, Product/Process Innovation, Government funding
Hérvas-Oliver et al. [66]	Investigate the particularities of SME innovation mechanisms in catching-up Southern, Central and Eastern European countries and contrast them with selected among Europe’s most developed countries to highlight major disparities	ANOVA tests and logistic regressions	Dependent (product innovation, radical product and process innovation), Independent STI- and DUI-type variables (R&D internal—engagement in intramural R&D; Buy Equipment—engagement in machinery acquisition; DUI collaborations, STI collaborations and DUI&STI collaborations) and control variables (University degree, International markets, Size and Industry—categorical).
Hérvas-Oliver et al. [67]	Evaluate if the policy target of 3% of GDP devoted to the promotion of R&I matches the practice of SMEs’ R&I in EU’s diverse regions	Logistic regression	Dependent (Regional SME Innovation), Independent (Public R&D funding, Private R&D spending, Non-R&D expenditures, SME collaboration, publications), Control—country dummies
Barbero, et al. [47]	Create a general equilibrium model to assess the possible macroeconomic impacts of meeting the TO1 R&D personnel targets outlined in the ERDF OPs for a set of NUTS-2 regions in Greece, Italy, Portugal, and Spain.	Panel data stochastic frontier approach with output-oriented technical inefficiency and RHOMOLO	GDP, Employment, capital stock, region-specific fixed effects, R&D personnel in the regions; human capital
Diukanova et al. [46]	Ex-ante analysis of the macroeconomic impacts of cohesion policy funding applied to support R&I and to assist low-carbon initiatives in Apulia, southern Italy	Input-output analysis and RHOMOLO	Main socio-economic indicators in Apulia: GDP per capita, GDP (and shares of Italian and EU28 GDP); Economically Active Population rate, Unemployment rate, Employment rate, Long-term unemployment, Youth unemployment, Primary, Secondary and Tertiary education attainment

.

Moreover, neither of the reports examined in Table A1 (Appendix A) employ the DEA approach in their evaluations. When conducting the efficiency assessment using the DEA method, MA can identify the OPs’ benchmarks in accordance with the desired practices. This enables the identification of the required changes that must be performed on the set of indicators of the performance framework that will allow inefficient OPs to become efficient across the programming period.

Also, as far as we are aware, the Network SBM methodology has not hitherto been employed in the assessment of the execution of OPs under the R&I theme (or in the assessment of any other TO). As a result, we will especially address the efficiency evaluation of R&I OPs using the Network SBM approach in conjunction with cluster analysis, as well as undertake a sensitivity analysis of the results produced.

3. Methodology

In DEA, there are two sorts of models: radial and non-radial. The Charnes–Cooper–Rhodes—CCR [68] and Banker–Charnes–Cooper—BCC [69] models represent radial models. These models cope with proportionate variations in inputs or outputs. As a result, the CCR and BCC efficiency scores thus obtained show the highest proportionate input (output) decrease (increase) rate possible that is equal for all inputs (outputs). Nevertheless, in real-world enterprises, this type of assumption is not realistic. For instance, if we make use of labour, raw materials, and capital as inputs, these might have a substitutional effect among them and do not necessarily change proportionately. Besides, these models only allow considering input- or output-oriented versions. Therefore, if the model is input-oriented, it is (unrealistically) assumed that the inefficiency linked to the usage of a particular input (resource) is necessarily linked to the inefficiency related to the use of another input by the same Decision-Making-Unit (DMU) under evaluation. Alternatively, if the model is output-oriented, it is presumed that a DMU can generate different outputs (outcomes) simultaneously with the same generation capability, thus ignoring that the generation efficiency for distinctive outputs can be diverse. As a result, in this paper, we employ the Network DEA model based on SBM model [70]. As opposed to the CCR and BCC models, the SBM model offers a more comprehensive examination of efficiency because it is non-radial (presuming that inputs and outputs can change non-proportionally), and it can be input-, output, and non-oriented. Contrary to radial models, which ignore slacks, the SBM model offers information on the modifications needed for each inefficient DMU’s input and output values to become efficient. Moreover, the SBM model possesses certain desirable characteristics, in particular, monotonicity (the efficiency score is monotone, decreasing in each slack both in inputs and outputs) and unit invariance (the efficiency score holds irrespective of the units of data) [70].

In addition, unlike the additive model [71], which is also non-radial, the SBM model enables calculating an efficiency score based on the slacks computed. Furthermore, the SBM model may be used with clustering analysis by considering groups of DMUs that fit specific criteria to appraise the DMU’s efficiency according to the cluster-frontier, therefore mitigating the influence of DMU’s heterogeneity on efficiency [72,73,74].

3.1. The SBM Model

Let the set of n DMUs be given as$\left(DM{U}_1,DM{U}_2,\dots , DM{U}_n\right)$. We define the (m × n) matrix of inputs (resources) and the (s × n) vector of outputs (outcomes), respectively, as X = [x_ij, i = 1, 2, …, m, j= 1, 2, …, n] and Y = [y_rj, r = 1, 2, …, s, j = 1, 2, …, n]. The rows of these for each DMU_k are ${\mathit{x}}_k^T$ and ${\mathit{y}}_k^T$, for its inputs and outputs, respectively, with ^T signalling the transposition of a vector.

The production possibility set for n DMUs can be given as:

$$\begin{gathered} P=\{\left({\mathit{x}}_k,{\mathit{y}}_k\right): {\sum }_{j=1}^n{\lambda }_j{y}_{rj}\ge y_{rk}, \mathrm{r}=1, \dots , \mathrm{s}, {\sum }_{j=1}^n{\lambda }_j{x}_{ij}\le x_{ik}, \mathrm{i}=1, \dots , \mathrm{m}, \\ {\lambda }_j\ge 0, j=1, \dots ,n\} \end{gathered}$$

where λ = ${\left({\lambda }_1,\dots ,{\lambda }_n\right)}^T$ is an intensity vector.

The inequalities of this production possibility set can be written as equalities by introducing the slack variables as follows:

$$\begin{array}{l}{x}_{ik}={\sum }_{j=1}^n{x}_{ij}{\mathsf{\lambda }}_j+s_i^{-}, \mathrm{i}=1, \dots , \mathrm{m}\\ y_{rk}={\sum }_{j=1}^n{y}_{rj}{\mathsf{\lambda }}_j-s_i^{+}, \mathrm{r}=1, \dots , \mathrm{s}\\ {\lambda }_j\ge 0, j=1, \dots , n,\\ s_i^{-}\ge 0, i=1, ..., m,\\ s_i^{+}\ge 0, r=1, ..., s,\end{array}$$

where $s_i^{-}$ and $s_i^{+}$ correspond to the input and output slacks, respectively.

Therefore, the non-oriented version of the SBM model can be written as [70]:

$$\begin{array}{c}\underset{\lambda ,{\mathit{s}}^{-}, {\mathit{s}}^{+}}{\min }\rho =\frac{1-\frac{1}{m}{\sum }_{i=1}^m{s}_i^{-}/x_{ik}}{1+\frac{1}{s}{\sum }_{r=1}^s{s}_r^{+}/y_{rk}}\\ \mathrm{s}.\mathrm{t}.\\ x_{ik}={\sum }_{j=1}^n{x}_{ij}{\mathsf{\lambda }}_j+s_i^{-}, \mathrm{i}=1, ..., \mathrm{m}\\ y_{rk}={\sum }_{j=1}^n{y}_{rj}{\mathsf{\lambda }}_j-s_i^{+}, \mathrm{r}=1, ..., \mathrm{s}\\ {\lambda }_j\ge 0, j=1, \dots , n,\\ s_i^{-}\ge 0, i=1, ..., m,\\ s_i^{+}\ge 0, \mathrm{r}=1, ..., \mathrm{s}\end{array}$$

(1)

An increase in $s_i^{-}$ or $s_i^{+}$, ceteris paribus, will reduce the value of ρ, with 0 < ρ < 1. The value of ρ in (1) can additionally be written as:

$$\rho =(\frac{1}{m}{\sum }_{i=1}^m\frac{x_{ik}-s_i^{-}}{x_{ik}}){(\frac{1}{s}{\sum }_{r=1}^s\frac{y_{rk}+s_r^{+}}{y_{rk}})}^{-1}$$

(2)

The value given by $\frac{x_{ik}-s_i^{-}}{x_{ik}}$ provides the percentage decrease of input i while $\frac{1}{m}{\sum }_{i=1}^m\frac{x_{ik}-s_i^{-}}{x_{ik}}$ corresponds to the average percentage decrease of inputs. Analogously, $\frac{y_{rk}+s_r^{+}}{y_{rk}}$evaluates the percentage increase of output r while $\frac{1}{\mathrm{s}}{\sum }_{r=1}^s\frac{y_{rk}+s_r^{+}}{y_{rk}}$ gives the average percentage increase in outputs. Hence, ρ corresponds to the ratio of the average inefficiencies of inputs and outputs.

Model (1) can be transformed into model (3) by employing the Charnes–Cooper transformation through a positive scalar variable t:

$$\begin{array}{l}\underset{t, \lambda ,{\mathit{s}}^{-}, {\mathit{s}}^{+}}{\min }\mathsf{\tau } =t-\frac{1}{m}{\displaystyle \sum _{i=1}^m}t{s}_i^{-}/x_{ik}\\ \mathrm{s}.\mathrm{t}. t+\frac{1}{s}{\sum }_{r=1}^{s}t{s}_r^{+}/y_{rk}=1,\\ {\mathit{x}}_k=X\lambda +{\mathit{s}}_{}^{-},\\ {\mathit{y}}_k=Y\mathsf{\lambda } -{\mathit{s}}_{}^{+}, \\ {\mathit{\lambda }}_{}\ge \mathit{0}, {\mathit{s}}_{}^{-}\ge \mathit{0}, {\mathit{s}}_{}^{+}\ge \mathit{0}, t>0.\end{array}$$

(3)

Let ${\mathit{S}}_{}^{-}$= $t{\mathit{s}}_{}^{-}$, ${\mathit{S}}_{}^{+}$= $t{\mathit{s}}_{}^{+}$ and Λ = $t{\mathit{\lambda }}_{}$. So, problem (3) turns out to be:

$$\begin{array}{l}\underset{t,\lambda ,{\mathit{s}}^{-}, {\mathit{s}}^{+}}{\min }\tau =t-\frac{1}{m}{\displaystyle \sum _{i=1}^m}{S}_i^{-}/x_{ik}\\ \mathrm{s}.\mathrm{t}. t+\frac{1}{s}{\sum }_{r=1}^s{S}_r^{+}/y_{rk}=1,\\ t{\mathit{x}}_k=X\mathit{\Lambda }+{\mathit{S}}_{}^{-},\\ t{\mathit{y}}_k=Y\mathit{\Lambda }-{\mathit{S}}_{}^{+},\\ {\mathit{\Lambda }}_{}\ge \mathit{0}, {\mathit{S}}_{}^{-}\ge \mathit{0}, {\mathit{S}}_{}^{+}\ge \mathit{0}, t >0\end{array}$$

(4)

The optimal solution to problem (4) is given by:

$${\rho }^{*}={\tau }^{*}, {\lambda }^{*}={\mathbf{\Lambda }}^{*}/t^{*}, {\mathit{s}}_{}^{-*}={\mathit{S}}_{}^{-}/ t^{*}, {\mathit{s}}_{}^{+*}={\mathit{S}}_{}^{+}/ t^{*}.$$

Definition 1.

A DMU_k is called SBM-Efficient if ${\rho }^{*}=1$, i.e., if ${\mathit{s}}_{}^{-*}$ = 0 and ${\mathit{s}}_{}^{+*}$ = 0.

Definition 2.

An SBM-inefficient DMU_k has a set of efficient reference DMUs, which is obtained by choosing the indices of the DMUs with ${\lambda }_j^{*}>0$.

Consider the SBM-inefficient DMU_k as follows: E_k ={j: ${\lambda }_j^{*}>0$, j = 1, …, n}. The DMU that can be seen as a reference of best practices, which is in a point of the efficiency frontier, for the SBM-inefficient DMU_k is given as:

$$({\widehat{\mathit{x}}}_k, {\widehat{\mathit{y}}}_k)=({\mathit{x}}_k-{\mathit{s}}_{}^{-*}, {\mathit{y}}_k+{\mathit{s}}_{}^{+*})=({\sum }_{jϵE_k}^{}{\lambda }_j^{*}{\mathit{x}}_j,{\sum }_{jϵE_k}^{}{\lambda }_j^{*}{\mathit{y}}_j)$$

(5)

3.1.1. The Network SBM Model

Traditional DEA techniques see DMUs as a “black box” that converts inputs into outputs. Nevertheless, there are also DEA techniques, named Network DEA, that view the entire system as being made of distinct stages (i.e., subDMUs) with linkages across them that relate to intermediate outputs created, which are then consumed inside the system. These Network DEA techniques provide finer-grained evaluation and increase the discriminating capability of the assessments performed [75]. Besides, through this approach, it is possible to assess the impact of each stage-specific inefficiencies on the global efficiency of the DMU.

Tone and Tsutsui [75] suggested the network slacks-based measure (NSBM), which relies on SBM [70] and weighted SBM [76]. This method evaluates the whole and stage efficiencies of the DMUs under scrutiny. The approach considers a global production possibility set that specifies the links between the multiple stage processes. Specifically, it is expected that a DMU is made up of p sub-processes (p = 1, …, P), with each sub-process consuming external inputs to create certain outputs. Besides external inputs and outputs, there are also intermediate factors linking processes. The generalized non-oriented NSBM model of [75], considering Variable Returns to Scale (VRS), can be given (by considering m_p external inputs and s_p external outputs) as:

$$\begin{array}{c}\underset{{\lambda }^{\mathrm{p}},s^{\mathrm{p}-}, s^{\mathrm{p}+}}{\min }{\mathsf{\rho }}_k=\frac{{\sum }_{p=1}^P{w}^p\left(1-\frac{1}{m_p}{\sum }_{i=1}^{m_p}\frac{s_i^{\mathrm{p}-}}{x_{ik}^p}\right)}{{\sum }_{p=1}^P{w}^p\left(1+\frac{1}{s_p}{\sum }_{r=1}^{s_p}\frac{s_r^{\mathrm{p}+}}{y_{rk}^p}\right)}\\ \mathrm{s}.\mathrm{t}.\\ x_{ik}^p={\sum }_{j=1}^n{x}_{ij}^p{\mathsf{\lambda }}_j^p+s_i^{\mathrm{p}-}, \mathrm{i}=1, ..., m_p, \mathrm{p}=1, \dots , \mathrm{P},\\ y_{rk}^p={\sum }_{j=1}^n{y}_{rj}^p{\mathsf{\lambda }}_j^p-s_i^{\mathrm{p}+}, \mathrm{r}=1, ..., s_p, \mathrm{p}=1, \dots , \mathrm{P},\end{array}$$

Intermediate linking constraints (7) or (8),

$$\begin{array}{c}{\sum }_{j=1}^n{\mathsf{\lambda }}_j^p=1, {\mathsf{\lambda }}_j^p\ge 0, j=1, \dots , n, \mathrm{p}=1, \dots , \mathrm{P},\\ {\sum }_{p=1}^P{w}^p=1, w^p\ge 0, \mathrm{p}=1, \dots , \mathrm{P},\\ s_i^{\mathrm{p}-}\ge 0, \mathrm{i}=1, ..., m_p, \mathrm{p}=1, \dots , \mathrm{P},\\ s_i^{\mathrm{p}+}\ge 0, r=1, ..., s_p, p=1, \dots , P,\end{array}$$

(6)

where ρ_k is the overall efficiency score of DMU_k, ${\lambda }_j^p$ is the intensity factor related to process p, and $x_{ij}^p$ is the value of external input i used by the process p of DMU_j such that $x_{ij}^{}$ = ${\sum }_{p\in P_I\left(i\right)}^{}{x}_{ij}^p$, where P_I(i) is the set of processes that use external input I and $y_{jr}^p$ is the value of external output r generated by process p of DMU_j, such that $y_{jr}^{}$ = ${\sum }_{p\in P_O\left(r\right)}^{}{y}_{jr}^p$, where P_O(r) is the set of processes that produce external output r, and $w^p$ is the relative weight of process p, which is obtained according to its significance.

The intermediate linking constraints can be represented according to the “free” link or to the “fixed” value cases [75].

In the first case, the linking factors are obtained freely (i.e., are viewed as discretionary factors) whilst maintaining continuity between inputs and outputs, i.e.,:

$${\sum }_{j=1}^n{z}_{qj}^{p,h}{\mathsf{\lambda }}_j^p={\sum }_{j=1}^n{z}_{qj}^{p,h}{\mathsf{\lambda }}_j^h, \mathrm{q}=1, ..., \mathrm{Q}$$

(7)

where $z_{qj}^{p,h}, q=1, \dots , Q$, is the value of intermediate factors linking process p to process h.

Therefore, the linking flow may expand or contract in the optimal solution to (6).

In the second case, the linking activities remain with the same values (i.e., are viewed as non-discretionary factors):

$$\begin{array}{c}{z}_{qk}^{p,h}={\sum }_{j=1}^n{z}_{qj}^{p,h}{\mathsf{\lambda }}_j^p, \mathrm{q}=1, ..., \mathrm{Q}, \forall \mathrm{p},\mathrm{h}, \mathrm{as} \mathrm{outputs} \mathrm{from} \mathrm{p},\\ z_{qk}^{p,h}={\sum }_{j=1}^n{z}_{qj}^{p,h}{\mathsf{\lambda }}_j^h , \mathrm{q}=1, ..., \mathrm{Q}, \forall \mathrm{p},\mathrm{h}, \mathrm{as} \mathrm{inputs} \mathrm{to} \mathrm{h}.\end{array}$$

(8)

Definition 3.

A DMU_k is overall SBM efficient if ${\rho \prime }_k^{*}=1$, where ${\rho \prime }_k^{*}$ is the optimal value to problem (6).

Definition 4.

The non-oriented stage efficiency score given within [0, 1] is as follows:

$${\mathsf{\rho }}_p^{\prime }=\frac{1-\frac{1}{m_p}{\sum }_{i=1}^{m_p}\frac{s_i^{\mathrm{p}-*}}{x_{ik}^p}}{1+\frac{1}{s_p}{\sum }_{r=1}^{s_p}\frac{s_r^{\mathrm{p}+*}}{y_{rk}^p}}, \mathrm{p}=1, \dots , \mathrm{P},$$

(9)

where $s_i^{\mathrm{p}-*}$ and $s_r^{\mathrm{p}+*}$ are the optimal output slacks to problem (6).

Theorem 1.

A DMU is globally efficient if and only if it is efficient for all stages [75].

Model (6) can be linearized by employing a similar method applied to convert problem (1) into a linear problem.

3.1.2. The Network SBM Model with Cluster Analysis

Conventional DEA models presume that all DMUs are homogeneous. As a result, all DMUs are thought to offer the reference set for building meta-frontiers. In reality, DMUs are not always homogenous, thus impacting the correctness of the DEA results obtained [77]. Therefore, clustering benchmarking might be particularly suitable when dealing with DMUs with diverse characteristics [73,74]. Cluster benchmarking is a methodology for categorizing a group of DMUs into clusters based on shared features. The clusters can be formed applying a clustering approach (in statistics) that is suitable to the problem under consideration, given exogenously, utilizing expert information, or based on the level of scale efficiency [75]. DMUs belonging to the same cluster are more similar than those belonging to other clusters [74].

The major objective is to exploit the similarity of DMUs in the same cluster as well as the discrepancy of DMUs in distinct clusters. The production frontiers must be established independently to complete the efficiency evaluation considering the best practices according to the appropriate clusters. The technology gap ratio (TGR) may be calculated by contrasting the results of the DEA model that considers all DMUs in the same group with the results obtained after clustering. The non-oriented version of the SBM model is used to calculate the meta-frontier and cluster-frontiers (see Problem (6)). Generically, the overall TGR_k of DMU_k is then calculated as follows [78]:

$$TG{R}_k=\frac{{\rho \prime }_k{}_{}^{meta \left(VRS\right)*}}{{\rho \prime }_k{}_{}^{cluster \left(VRS\right)*}}, \mathrm{p}=1, \dots , \mathrm{P},$$

(10)

where ${\rho \prime }_k{}_{}^{meta \left(VRS\right)*}$ is the efficiency value of DMU_k according to the SBM non-oriented model based on the meta-frontier considering VRS, and ${\rho \prime }_k{}_{}^{cluster \left(VRS\right)*}$ is the efficiency value of DMU_k computed with the SBM non-oriented model using the cluster-frontier and considering VRS. The TGR for each process can be obtained analogously.

The TGR value indicates the gap between the cluster- and meta-frontier. It is applied to assess the technological efficiency gap of the same DMU based on distinct frontiers. Furthermore, TGR might signal the necessity to separate various groups [78]. The lower the TGR value, the higher the clustering needed, and vice-versa.

The value of TGR_k should range between 0 and 1 [73,74]. Nevertheless, the TGRs of some DMUs in different stages (i.e., divisions) may be bigger than one when using the concave frontier because it might include unfeasible input-output arrangements, consequently leading to erroneous and unreasonable assessments. Therefore, [79] suggested a non-concave meta-frontier that envelopes those input-output conjunctions that belong to the technology set of at least one of the technologies, removing the area with unfeasible input-output conjugations. They proposed a two-step methodology to obtain a non-concave meta-frontier, thus making it possible to attain a TGR in distinct processes of the network structure within [0, 1]. First, the technical efficiency of a DMU is measured against its own cluster technology. Then, the technical efficiency of a DMU is measured against the other cluster technologies. The technical efficiency score thus obtained is lower if the other cluster technology enables the DMU to generate a higher output level. In that situation, the other cluster technology characterizes the meta frontier for this DMU.

A TGR_k closer to one indicates a tiny gap between the meta- and the cluster-frontier. The meta-frontier indicates the underlying efficiency level of the entire assessed group of DMUs, whereas the cluster-frontier depicts the true efficiency level of each cluster; the greater the value of TGR_k, the narrower the gap between the meta- and the cluster-frontier.

4. Data and Assumptions

TO 1 covers innovation in global terms. The entire budget devoted to TO 1 is roughly 66 billion euros, with ERDF (national and EU contributions) covering 94% [41]. TO 1 receives 22% of the entire (national and EU) ERDF funding, corresponding to 62.2 billion euros supported by the ERDF, of which 20.6 billion are national, and 41.6 billion are EU funds. TO 1 is divided into two main investment priorities (IP) [41]: IP (1a) strengthening R&I infrastructure and capability to foster R&I excellency, as well as deploying centres of competence, predominantly those of European prominence; (1b) encouraging corporate investment in R&I, constructing connections and co-benefits between enterprises, research centres, and the university education sector, particularly stimulating incentives to invest in new products and services, transfer of technology, social innovation, eco-innovation, public service tenders, demand stimulus, networking, clusters, and open innovation via smart specialisation, and sponsoring technological and applied research, pilot lines, initial product validation, high tech manufacturing capacities, and initial production, with a focus on essential technological solutions and all-purpose technology dissemination. Since our study is focused on enterprises, in particular SMEs (most firms supported by ERDF), we will address priority (1b) in our assessment, and we will consider the corresponding dimensions of intervention given in

Table 2. Dimensions of intervention in TO 1.

Code	Dimension
1	Generic productive investment in SMEs
2	Research and innovation processes in large enterprises
3	Productive investment in large enterprises linked to the low-carbon economy
4	Productive investment linked to the cooperation between large enterprises and SMEs for developing information and communication technology(ICT) products and services, e-commerce and enhancing demand for ICT
57	Investment in infrastructure, capacities and equipment in large companies directly linked to research and innovation activities
62	Technology transfer and university-enterprise cooperation primarily benefiting SMEs
63	Cluster support and business networks primarily benefiting SMEs
64	Research and innovation processes in SMEs (including voucher schemes, process, design, service and social innovation)
65	Research and innovation processes, technology transfer and cooperation in enterprises focusing on the low carbon economy and to resilience to climate change
66	Advanced support services for SMEs and groups of SMEs (including management, marketing and design services)
67	SME business development, support to entrepreneurship and incubation (including support to spin offs and spin outs)
69	Support to environmentally-friendly production processes and resource efficiency in SMEs
76	Development and promotion of cultural and creative assets in SMEs
77	Development and promotion of cultural and creative services in or for SMEs
82	ICT Services and applications for SMEs (including e-Commerce, e-Business and networked business processes), living labs, web entrepreneurs and ICT start-ups)

Available online: https://cohesiondata.ec.europa.eu/2014-2020-Categorisation/ESIF-2014-2020-categorisation-ERDF-ESF-CF-planned-/3kkx-ekfq (accessed on 1 January 2022).

.

We engaged several stakeholders in the selection of the factors used in the assessment by hosting a guided workshop with various policymakers and MA on the theme “Evaluating Co-financed Intervention Policies in Enterprises.” Furthermore, we conducted a separate examination of regional and national programs, classified into clusters based on the categories of the regions under consideration (i.e., less developed regions, regions of transition, and more developed regions). In this case, whenever the regional categories assigned to the OPs simultaneously belonged to distinct types of regions, these have been classified as regions of transition. Because these are the most up-to-date statistics for the accomplishment indicators, the numbers evaluated are cumulative values from multiple years released on 19 November 2021. The OPs with no missing data were analysed (i.e., the OPs with missing data were not contemplated) in our evaluation, which resulted in the study of 19 countries and 53 programs.

The input and output factors considered for performing the efficiency assessment of implementing the European Structural Investment (ESI) funds devoted to R&I interventions in SMEs were chosen from a list of common indicators legally required by the EU [80]. These were more often available and are described below (see also

Table 3. Inputs and Outputs selected.

	Researchers Working in Improved Infrastructures	Enterprises Supported	Enterprises Working with Research Institutions	Enterprises Supported for New Market to Products	Eligible Cost Decided	Total Eligible Spending
Description	Number of researchers working in improved research infrastructures	Number of enterprises supported	Number of enterprises cooperating with research institution	Number of enterprises supported to introduce new-to-the-market products	Financial resources assigned	Eligible costs validated
Type of factor	Output P2	Output P1	Output P2	Output P1	Input P1	IntermediateOutput P1/Input P2
Unit	Number of researchers Full Time Equivalent	Number of enterprises	Number of enterprises	Number of enterprises	Euro	Euro
Source	(b)	(b)	(b)	(b)	(a)	(a)
Explanation	Increase the success of research institutions in attracting competitive and private research funding	Increase the translation of R&I processes into new and improved processes and services	Increase the translation of R&I processes through technology transfer from Higher Education Institutions	Increase the translation of R&I processes into new and improved commercial products	Reflects concerns about the pace of programmes’ implementation	Reflects concerns about the pace of programmes’ implementation
Classification	Output Indicator	Process Indicator	Output Indicator	Process Indicator	Financial Indicator	Financial Indicator

(a) Data from the ERDF/ESF/cohesion fund programmes. Available online: https://cohesiondata.ec.europa.eu/2014-2020-Categorisation/ESIF-2014-2020-categorisation-ERDF-ESF-CF-planned-/3kkx-ekfq (accessed on 11 April 2022); (b) List of SFs achievement data. Available online: https://cohesiondata.ec.europa.eu/2014-2020/ESIF-2014-2020-Achievement-Details/aesb-873i (accessed on 11 April 2022).

).

In addition, according to [41], at the TO level, 2014–2020, common R&I indicators concentrate on financial aspects (e.g., financial resources assigned), which are always present at any stage, and on procedural aspects (e.g., ‘the number of enterprises receiving support’). Nevertheless, there are other indicators, referred to as real outputs, such as the ‘number of researchers working in improved research infrastructure facilities’ and the ‘networks created between enterprises and research institutions facilitating technological and knowledge transfer’ [41]. Therefore, we have considered an NSBM model with the structure given in Figure 1.

4.1. The Pace of Programmes’ Implementation

The EU has been pursuing an R&I-led strategy relying on a paradigm in which R&D investment is a key source of higher economic achievements [67]. Under this strategy, the EU has attempted to increase R&D expenditure, with a target of 3% of its GDP. Hence, the financial implementation of SFs, which is an essential prerequisite for efficient policy execution, is included in this study, with special attention paid to the rate of programme implementation [60,81]. Costs that do not meet the appropriate qualifying conditions will not be reimbursed under this framework. These must be confirmed by a certified Controller, who is the entity or responsible person for verifying at the national scale that the co-financed products and services were purchased, that the related spending has been paid, and that the required EU Programme and national regulations were followed.

On the one hand, the “total eligible spending” relates to the eligible expenses submitted by the supported projects that this Controller has certified. As a result, the larger its value, the higher each project’s financial execution. Consequently, it is employed as an output at the first stage of the analysis, which seeks to assess procedural aspects only. Then, it is viewed as an input at the second stage of the assessment, which seeks to evaluate the actual R&I capacity generated with the funding used. On the other hand, the “eligible costs decided” are those that have financial resources allotted to the projects chosen for financing (project pipeline), being thus considered an external input.

4.2. Enterprises Supported

This indicator refers to the number of firms that have received ERDF assistance (whether or not the assistance is considered state aid). According to [41], this factor of evaluation should be used with indicators referring to the number of enterprises supported by introducing new-to-the-firms and new-to-the-market products for innovation in enterprises. The more firms are supported, the higher the possibility of enhancing their R&I capacities and their adoption of innovative technologies, thus being regarded as a procedural external output.

4.3. Enterprises Working with Research Institutions

Previous publications emphasise the necessity of efficient cooperation across organizations to build trust-based channels, which help promote knowledge transfer [82]. Indeed, areas with well-coordinated institutional frameworks relying on the triple helix but also the creation of novel cooperation mechanisms [83] are inextricably related to strong local innovation processes [18]. As a result, we employ indicator CO26, ‘Number of firms interacting with research institutions,’ which evaluates network engagement and is regarded as a proxy for potential technology and knowledge transfer. This indicator was obtained from the set of common output indicators on ‘Productive Investment’ available from [41] and corresponds to the number of enterprises that collaborate with research institutions on R&D initiatives. The initiatives involve at least one enterprise and one research institution. Support may be provided to one or more of the collaborating parties (research institution or industry), but it must be conditioned on cooperation. The collaboration might be fresh or old. The collaboration should endure at least until the end of the project. This indicator is viewed as a real external output at the second stage of the analysis.

4.4. Number of Enterprises Supported to Introduce New-to-the-Market Products

The launch of a new-to-the-market product is acknowledged as a sign of innovation. Besides measuring the outcomes in terms of innovation, it also shows its successful market introduction. There are several concepts and metrics that can be employed for assessing innovation [84]. Out of these, a few consider expenditure as inputs, patents as outputs, and launching new-to-the-market products/services as a way of measuring market acceptance (also as an output). All measures have their own limitations. It is not guaranteed that investing in R&D will end up reaching innovation. Besides, patents do not necessarily generate added value to the organization and do not give any account for the degree of market acceptance. Moreover, as mentioned earlier, one of the limitations of the performance framework indicators used during the 2014–2020 programming period is the lack of data regarding the direct impact of the investments supported. Hence, we will use a proxy of the potential created for introducing a new-to-the-market product the ‘number of enterprises supported to introduce new-to-the-market products,’ which measures the assistance provided to businesses in developing a ‘new-to-the-market’ product in all its markets. This is the CO29 indicator from the list of common indicators obtainable from [41].

In this indicator, process innovation might also be included if the process concurs with the creation of the ‘new-to-the-market’ product. Applications that do not seek to generate a new-to-the-market product are not eligible.

A product is new-to-the-market if it is not offered on a market (i.e., it does not have to be new to all markets) that delivers the same functions, or the technology used by the new product is radically different from the technology used by already existing products [41]. Products may be both tangible and immaterial (including services and processes). To avoid double-counting, we have not used the indicator measuring the number of new-to-the-firm products. This indicator is regarded as an external procedural output during the first stage of evaluation.

4.5. Number of Researchers Working in Improved Research Infrastructure Facilities

Previous studies have found that firms with increased accessibility to more advanced technology and highly skilled workers outperform firms devoid of these (see, e.g., [84,85]). Therefore, we have considered indicator CO25 from the list of indicators presented in [41], ‘the number of researchers working in improved research infrastructure facilities,’ as a proxy that allows measuring both highly skilled workers and the access of SMEs to more advanced technology.

This measure of research activity is contemplated in the framework of research infrastructure facilities (either physical or non-physical) improved due to ERDF funding. Existing job openings in research infrastructure facilities that (1) directly carry out R&I activities and (2) are directly impacted by the assistance provided by the OP. The positions must be filled (empty positions are not considered) and are in Full-Time Equivalent. The R&D project has to enhance the facilities or the quality of equipment, i.e., maintenance or substitution without quality improvement is not accounted for. This indicator is employed as a real external output at the second stage of the analysis.

Table A2. Data.

Programme	Improved resinfra	Enterprises Supported	Enterprises Working RI	Enterprises New Market Products	Eligible Cost Decided	Total Elegible Spending	Region Category
Alentejo—ERDF/ESF	10.00	83.00	14.00	2.00	58,217,962.00	14,239,865.00	Less developed
Alsace—ERDF	189.83	1909.00	46.00	24.00	27,153,237.00	19,104,006.00	More developed
Andalucía—ERDF	232.00	0.00	92.00	0.00	9,634,024.00	1,424,378.00	Transition
Aragón—ERDF	340.00	696.00	267.00	0.00	2,345,000.00	1,399,896.00	More developed
Baden-Württemberg—ERDF	76.50	120.00	443.00	113.00	135,801,370.00	79,412,867.00	More developed
Bayern—ERDF	113.00	0.00	13.00	0.00	117,832,446.00	43,420,521.00	More developed
Berlin—ERDF	837.59	658.00	66.00	649.00	422,930,140.00	327,681,024.00	More developed
Brandenburg—ERDF	1143.00	75.00	3.00	0.00	271,983,485.00	120,411,834.00	Transition
Bremen—ERDF	32.00	121.00	0.00	35.00	70,055,519.00	24,997,078.00	More developed
Brussels Capital Region—ERDF	18,538.90	145.00	83.00	65.00	25,935,325.00	6,337,597.00	More developed
Calabria—ERDF/ESF	72.00	764.00	493.00	171.00	94,015,191.00	58,952,072.00	Less developed
Campania—ERDF	72.00	1065.00	158.00	640.00	478,090,524.00	146,549,902.00	Less developed
Canarias—ERDF	40.00	138.00	0.00	0.00	53,714,494.00	8,898,407.00	Transition
Castilla y León—ERDF	333.00	0.00	3076.00	0.00	158,014,788.00	69,516,817.00	More developed
Cataluña—ERDF	2379.00	75.00	0.00	0.00	331,751,374.00	27,603,147.00	More developed
Cohesion Policy Funding—EE—ERDF/ESF/CF	1324.41	9677.00	1838.00	310.00	490,919,405.00	264,062,219.00	Less developed
Competitive Central-Hungary—ERDF/ESF	338.00	278.00	0.00	17.00	166,743,374.00	56,338,663.00	More developed
Competitiveness and Cohesion—HR—ERDF/CF	10.00	331.00	142.00	192.00	266,718,752.00	110,046,783.00	Less developed
Competitiveness Entrepreneurship and Innovation—GR—ERDF/ESF	847.00	340.00	0.00	0.00	8,175,750.00	817,920.00	Transition
Competitiveness Programme—RO—ERDF	136.06	229.00	0.00	101.00	190,831,345.00	117,272,917.00	Transition
Corse—ERDF/ESF	16.30	7.00	54.00	0.00	13,196,290.00	4,897,175.00	Transition
East Wales—ERDF	195.00	417.00	60.00	2.00	99,686,500.00	23,052,166.00	More developed
England—ERDF	77.68	19,146.00	4169.00	2475.00	953,058,658.00	547,408,267.00	Transition
EU Cohesion Policy—SI—ERDF/ESF/CF/YEI	119.83	932.00	631.00	660.00	847,532,572.00	406,679,368.00	Transition
EU Structural Funds Investments—LT—ERDF/ESF/CF/YEI	161.28	2458.00	771.00	469.00	677,736,126.00	244,424,833.00	Less developed
Extremadura—ERDF	3662.00	1181.00	1027.00	0.00	39,801,270.00	13,896,966.00	Less developed
Growth and Employment—LV—ERDF/ESF/CF/YEI	1352.50	2442.00	115.00	566.00	199,074,160.00	77,431,566.00	Less developed
Integrated Infrastructure—SK—ERDF/CF	59.30	378.00	0.00	70.00	947,644,425.00	411,180,483.00	Transition
La Rioja—ERDF	116.08	168.00	0.00	0.00	12,685,184.00	12,014,052.00	More developed
Lazio—ERDF	255.00	562.00	159.00	0.00	109,131,144.00	42,966,355.00	More developed
Lorraine et Vosges—ERDF/ESF/YEI	1673.27	56.00	219.00	48.00	88,132,818.00	25,084,802.00	Transition
Lubelskie Voivodeship—ERDF/ESF	15.00	282.00	138.00	0.00	69,778,912.00	38,587,792.00	Less developed
Luxembourg—ERDF	75.00	0.00	116.00	40.00	11,319,879.00	7,899,337.00	More developed
Madeira—ERDF/ESF	330.00	10.00	3.00	10.00	27,625,031.00	11,617,474.00	More developed
Mecklenburg-Vorpommern—ERDF	803.55	484.00	91.00	394.00	308,202,199.00	172,946,706.00	Transition
Multi-regional Spain—ERDF	14,838.00	19,250.00	0.00	0.00	2,675,711,496.00	1,278,171,878.00	Transition
National fund for investments in growth and jobs—ERDF	100.00	151.00	124.00	0.00	7,920,685.00	1,880,444.00	More developed
Niedersachsen—ERDF/ESF	108.96	330.00	0.00	175.00	326,011,096.00	199,221,241.00	More developed
País Vasco—ERDF	21.00	1498.00	157.00	0.00	60,617,604.00	43,386,190.00	More developed
Pays de la Loire—ERDF/ESF	2840.20	1217.00	1707.00	0.00	84,317,622.00	48,898,141.00	More developed
Poitou-Charentes—ERDF/ESF	215.78	469.00	30.00	15.00	80,943,391.00	11,433,896.00	Transition
Réunion—ERDF	107.30	359.00	220.00	211.00	44,644,207.00	29,796,583.00	Less developed
Saarland—ERDF	563.00	98.00	13.00	0.00	26,565,980.00	12,378,578.00	More developed
Sachsen—ERDF	1751.31	1457.00	169.00	0.00	655,719,135.00	358,044,509.00	Transition
Sachsen-Anhalt—ERDF	1006.43	0.00	27.00	0.00	218,397,233.00	125,743,286.00	Transition
Schleswig-Holstein—ERDF	303.40	214.00	273.00	176.00	93,357,335.00	50,000,231.00	More developed
Sicilia—ERDF	54.00	189.00	25.00	0.00	19,355,072.00	8,514,315.00	Less developed
Smart growth—PL—ERDF	811.50	6633.18	3590.18	0.00	7,525,498,071.00	2,654,718,365.00	Less developed
Thüringen—ERDF	842.50	354.00	40.00	62.00	246,794,570.00	110,393,584.00	Transition
Toscana—ERDF	238.00	4470.00	1216.00	2289.00	281,981,499.00	189,360,205.00	More developed
Valenciana—ERDF	247.74	1575.00	0.00	0.00	93,316,144.00	71,327,346.00	More developed
Wallonia—ERDF	109.50	7232.35	1211.35	131.00	310,322,894.00	147,512,404.00	Transition
West Wales and The Valleys—ERDF	50.00	981.00	849.00	68.00	365,867,527.00	174,174,256.00	Less developed

in Appendix A has data on these indicators.

From the analysis of

Table 4. Descriptive Statistics per region category.

Region Category	Statistics	Researchers Working in Improved Infrastructures	Enterprises Supported	Enterprises Working with Research Institutions	Enterprises Supported for New Market to Products	Eligible Cost Decided	Total Eligible Spending
Overall Mean	Mean	1136	1730	452	192	394,392,646	170,821,334
	S.D.	3220	3972	897	474	1,080,808,911	402,568,084
	Minimum	10	0	0	0	2,345,000	817,920
	Maximum	18,539	19,250	4169	2475	7,525,498,071	2,654,718,365
	Count	53	53	53	53	53	53
More Developed	Mean	1242	640	340	156	117,091,229	59,557,103
	S.D.	3836	1004	727	485	117,296,122	78,064,181
	Minimum	21	0	0	0	2,345,000	1,399,896
	Maximum	18,539	4470	3076	2289	422,930,140	327,681,024
	Count	23	23	23	23	23	23
Transition	Mean	1407	2973	396	233	463,529,105	226,371,871
	S.D.	3508	6340	1021	604	656,185,882	319,537,468
	Minimum	16	0	0	0	8,175,750	817,920
	Maximum	14,838	19,250	4169	2475	2,675,711,496	1,278,171,878
	Count	17	17	17	17	17	17
Less Developed	Mean	592	2,034	722	202	794,593,629	295,030,424
	S.D.	1047	2891	1010	229	2,033,465,229	714,241,070
	Minimum	10	83	14	0	19,355,072	8,514,315
	Maximum	3662	9677	3590	640	7,525,498,071	2,654,718,365
	Count	13	13	13	13	13	13

Note: S.D.—standard deviation.

, the average overall financial execution rate (i.e., the ratio between the total eligible spending and the total eligible cost) is low (43.33%). It is even lower for the less developed regions (37.13%), presenting higher values for the regions of transition (48.84%) and more developed regions (50.86%). Additionally, the highest average R&I generation potential might be found in the regions of transition (with the highest average number of researchers working in improved infrastructures of 1407). In contrast, the average number of enterprises supported for working with research institutions is higher in the less developed regions (722).

A DEA requirement refers to inputs and outputs that must have an isotonic relation that can be corroborated using correlation analysis [86]. This means that the link between inputs and outputs should be consistent, i.e., increasing the value of any input ceteris paribus should not decrease any output but should instead lead to an increase in the value of at least one output. The factors support an isotonic relation if the correlation between inputs and outputs is positive (and significant). Because the normality assumption for the application of the tests for the significance of Pearson’s correlation was not validated in this situation, we chose to obtain the Spearman correlation coefficients and the associated significance tests for both P1 and P2 processes—see

Table 5. Correlation matrix.

Process	Variable	Researchers Working in Improved Infrastructures	Enterprises Supported	Enterprises Working with Research Institutions	Enterprises Supported for New-to-the-Market Products	Eligible Cost Decided	Total Eligible Spending
P1	Enterprises Supported	-	1.00	-	0.35 ** (0.0099)	0.49 ** (0.0002)	0.55 ** (0.0000)
	Enterprises new market products	-	0.35 **(0.0099)	-	1.00	0.47 ** (0.0004)	0.49 ** (0.0002)
	Eligible cost decided	-	0.49 ** (0.0002)	-	0.47 ** (0.0004)	1.00	0.95 ** (0.0000)
	Total eligible spending	-	0.55 ** (0.0000)	-	0.49 ** (0.0002)	0.95 ** (0.0000)	1.00
P2	Researchers working in improved infrastructures	1.00	-	0.04 (0.7889)	-	-	0.11 (0.4218)
	Enterprises working with research institutions	0.04 (0.7889)	-	1.00	-	-	0.29 * (0.0380)
	Total eligible spending	0.11 (0.4218)	-	0.29 * (0.0380)	-	-	1.00

Note: Spearman correlation coefficients are presented in this table; in parentheses are the associated p-values of bi-directional tests. * Significant correlation at the 0.05 level. ** Significant correlation at the 0.01 level.

.

5. Analysis and Discussion of Results

Results were computed with MaxDEA 8 Ultra software, assuming that the linking factor considered in the analysis can be obtained freely. The values of the TGR for the distinct region categories and the corresponding meta- and cluster-frontiers were computed for all R&I OPs under scrutiny. We applied the Kruskal–Wallis test to validate the existence of significant differences in the efficiency scores of the three clusters for both processes.

Table 6. Results of Kruskal–Wallis test.

Processes	Region (3 Clusters)	Count	Mean Rank	p-Value
	More developed	23	23.61
P1	Transition	17	24.18	0.031
	Less Developed	13	36.69
	More developed	23	20.70
P2	Transition	17	36.47	0.005
	Less Developed	13	25.77

reports the results of the Kruskal–Wallis test, that are significant at a 5% level, for procedural efficiency and R&I efficiency. From this test, we can conclude that the used approach is adequate.

Basic descriptive statistics for these measures are shown in

Table 7. Descriptive Statistics of the results obtained for the OPs.

Region Category	Procedure	Statistics	Efficiency (Meta)	Efficiency (Cluster)	TGR (Nodes)	Efficiency (Network Meta)	Efficiency (Network Cluster)	TGR (Network)
More Developed		Mean	0.33	0.33	1.00	0.15	0.15	1.00
		Median	0.10	0.10	1.00	0.01	0.01	1.00
		Standard Deviation	0.38	0.38	0.00	0.30	0.30	0.00
		Minimum	0.00	0.00	1.00	0.00	0.00	1.00
		Maximum	1.00	1.00	1.00	1.00	1.00	1.00
		Count	23	23	23	23	23	23
Transition		Mean	0.29	0.34	0.57	0.19	0.33	0.24
		Median	0.06	0.12	0.69	0.01	0.09	0.07
	P1	Standard Deviation	0.41	0.43	0.30	0.39	0.44	0.36
		Minimum	0.00	0.02	0.09	0.00	0.01	0.03
		Maximum	1.00	1.00	1.00	1	1	1
		Count	17	17	17	17	17	17
Less Developed		Mean	0.22	0.66	0.40	0.10	0.35	0.18
		Median	0.13	1.00	0.38	0.00	0.07	0.13
		Standard Deviation	0.20	0.41	0.26	0.20	0.46	0.17
		Minimum	0.03	0.03	0.07	0.00	0.00	0.00
		Maximum	0.71	1.00	1.00	0.56	1.00	0.56
		Count	13	13	13	13	13	13
More Developed		Mean	0.22	0.22	1.00	0.15	0.15	1.00
		Median	0.03	0.03	1.00	0.01	0.01	1.00
		Standard Deviation	0.38	0.38	0.00	0.30	0.30	0.00
		Minimum	0.00	0.00	1.00	0.00	0.00	1.00
		Maximum	1.00	1.00	1.00	1.00	1.00	1.00
		Count	23	23	23	23	23	23
Transition		Mean	0.24	0.54	0.33	0.19	0.33	0.24
		Median	0.07	0.38	0.19	0.01	0.09	0.07
	P2	Standard Deviation	0.37	0.42	0.36	0.39	0.44	0.36
		Minimum	0.00	0.03	0.03	0.00	0.01	0.03
		Maximum	1.00	1.00	1.00	1	1	1
		Count	17	17	17	17	17	17
Less Developed		Mean	0.19	0.34	0.37	0.10	0.35	0.18
		Median	0.01	0.05	0.27	0.00	0.07	0.13
		Standard Deviation	0.34	0.46	0.27	0.20	0.46	0.17
		Minimum	0.00	0.00	0.01	0.00	0.00	0.00
		Maximum	0.96	1.00	0.96	0.56	1.00	0.56
		Count	13	13	13	13	13	13

for both processes under evaluation. Figure 2 illustrates the efficiency scores based on the meta- and cluster-frontiers for procedural efficiency, R&I efficiency, and the overall network efficiency, respectively.

It should be noted that more developed regions had their cluster frontiers tangent to their meta-frontiers, meaning that the TGR for these regions is one and that these regions are already producing at the same level of efficiency of the meta-frontier regardless of the process under evaluation (Figure 2a–c). The other mean TGR values were 0.4 and 0.57 in terms of Procedural efficiency, with the situation getting worse for R&I efficiency. For example, the TGR values obtained for this latter process are 0.37 and 0.33 for less developed and transition regions, respectively (the median TGR values for both processes and less developed and transition regions corroborate these conclusions). These outcomes appear to confirm the ‘European paradox’ since the OPs reveal higher levels of TGR for procedural efficiency than for R&I potential capacity development, specifically in less developed regions and regions of transition. This paradox is related to the incapacity of transforming the results of technological research and skills into actual innovations and competitive advantage [49]. In a similar vein, [87] and [49] found that the European Innovation System is more successful at doing pure research, which is primarily motivated by public R&D financing, than at producing innovation outputs. In summary, there is an ‘innovation gap’ in that encouraging innovation inputs through public support does not always result in innovation outputs.

The overall network TGR is even lower for these regions, with values ranging between 0.18 and 0.24. These findings indicate the existence of a big gap between the two frontiers, particularly for the less developed regions (when compared to the meta-frontier, the cluster-frontier had the number of efficient OPs—score equal to 1—increased from 0 to 7 and 0 to 4 for Procedural and R&I efficiencies, respectively—see Figure 2a,b). In the regions of transition, the cluster-frontier had the number of efficient DMUs showing only one additional efficient OP according to procedural efficiency, thus attaining four efficient OPs, whereas according to R&I efficiency, three OPs became additionally efficient, ending up with six efficient OPs (Figure 2a,b).

Figure 3 depicts the number of OPs at various efficiency score subintervals based on the cluster- and meta-frontiers, respectively. The number of OPs classified as efficient regarding procedural efficiency expanded from 7 to 15 (Figure 3a,b), whereas in the case of R&I efficiency, it only increased from 7 to 14 efficient OPs (Figure 3c,d). The overall network efficiency shows an increase of 5 to 10 efficient OPs in the cluster-frontier (Figure 3e,f).

These findings are heavily driven by the TGR achieved (both for Procedural efficiency and R&I efficiency) by R&I OPs in less developed regions. These generate only about 40% of the possible efficiency given the technology available for this type of OP, according to the average European R&I OPs considered (19 countries are represented), as shown in Table 7. Nonetheless, the regions of transition obtain 57% of the potential efficiency on average in terms of procedural efficiency (with a median of 69%, meaning that 50% of OPs have a potential efficiency smaller than or equal to 69%). However, only 33% of the potential efficiency (with the median equaling 19%, 50% of the OPs have potential efficiency smaller than or equal to 19%) in terms of R&I efficiency (Table 7).

According to the cluster frontiers, 15 and 14 out of 53 OPs were relatively efficient for Procedural and R&I efficiencies. Nevertheless, the overall network efficiency only showed 10 efficient OPs, implying that only about 19% of the OPs considered were efficient. This indicates that the majority of OPs assessed had simultaneous inefficient procedural and R&I capacities.

The four OPs more frequently viewed as benchmarks are “Aragón” (22), “Brussels Capital Region” (19), “Cohesion Policy Funding—EE” (19), and “Competitiveness Entrepreneurship and Innovation—GR” (18)—see

Table 8. Main characteristics of network efficient OPs.

DMU	Cluster	Number of Times as Benchmark	Researchers Working in Improved Infrastructures	Enterprises Supported	Enterprises Working with Research Institutions	Enterprises Supported for New-to-the-Market Products	Eligible Cost Decided	Total Eligible Spending	Innovation * Performance *
Aragón—ERDF	More developed	22	340	696	267	0	2,345,000	1,399,896	Moderate Innovator
Brussels Capital Region—ERDF	More developed	19	18,539	145	83	65	25,935,325	6,337,597	Innovation leader
Cohesion Policy Funding—EE—ERDF/ESF/CF	Less Developed	19	1324	9677	1838	310	490,919,405	264,062,219	Strong Innovator
Competitiveness Entrepreneurship and Innovation—GR—ERDF/ESF	Transition	18	847	340	0	0	8,175,750	817,920	Moderate Innovator
England—ERDF	Transition	12	78	19,146	4169	2475	953,058,658	547,408,267	Innovation Leader
Extremadura—ERDF	Less Developed	13	3662	1181	1027	0	39,801,270	13,896,966	Emerging innovator
Multi-regional Spain—ERDF	Transition	2	255	562	159	0	109,131,144	42,966,355	Moderate innovator
Sicilia—ERDF	Less Developed	3	54	189	25	0	19,355,072	8,514,315	Moderate innovator
Smart growth—PL—ERDF	Less Developed	6	812	6633	3590	0	7,525,498,071	2,654,718,365	Emerging innovator
Wallonia—ERDF	Transition	1	110	7232	1211	131	310,322,894	147,512,404	Strong Innovator

* According to the Regional Innovation Scoreboard in 2021 (Hollanders, 2021).

. Out of these, two belong to more developed regions, one belongs to a less developed region, and another one belongs to a region of transition. By cross-checking these findings with the Regional Innovation Scoreboard in 2021 [88], “Brussels Capital Region” is ranked 14th in the Top list of regional “innovation leaders,” whereas “Aragón” is viewed as a “moderate innovator.” The OP “Cohesion Policy Funding—EE,” a national programme, belongs to a country classified as a “Strong Innovator”. Finally, the OP “Competitiveness Entrepreneurship and Innovation” belongs to Greece, which is considered a “moderate innovator”.

Curiously, the less developed regions, which are both efficient and “emerging innovators,” manage to attain high values both for the number of researchers working in improved infrastructures and the number of enterprises working with research institutions.

Finally, only the efficient OPs coming from regions either classified as “Innovation leaders” or “Strong Innovators” scored in all the indicators considered in the analysis—see Table 8. These findings might be driven by the fact that these regions are better equipped in terms of innovative institutions, as well as advanced education and knowledge transfer, also suggesting that these ended up profiting the most from the EU funds awarded [19].

Globally, more developed regions show a higher potential for improvement of 85%, whereas transition and less developed regions had a smaller potential for improvement of 67% and 65%, respectively—see Table 7. These findings are in line with [63]. They concluded that the most innovative EU MS had much lower efficiency of diffusion of innovation scores when compared to some apparently weak innovating MS.

Figure 4 shows that the number of enterprises supported for developing new-to-the-market products in both efficient and inefficient OPs is not as significant as the other factors that help explain the overall network efficiency (Figure 4c). The mean funds dedicated to the eligible cost of efficient OPs (1,205,112,294 €) were significantly higher than that of inefficient ones (205,853,193 €). The same can also be concluded regarding the mean eligible spending of efficient OPs (492,283,983 €) and that of inefficient ones (96,062,570 €)—see Figure 4c). Moreover, the mean values of the number of researchers working in improved R&I infrastructures and the number of enterprises working with R&I institutions of efficient OPs are particularly high for those OPs that can generate R&I capacity efficiently. This is evident compared to the values attained for efficient OPs showing higher procedural efficiency (Figure 4a,b). In addition, the number of enterprises supported is slightly higher for efficient Procedural OPs than for OPs that manage to reach higher R&I capacity efficiently (Figure 4a,b). This suggests that supporting a higher number of enterprises does not necessarily lead to the generation of technology transfer between R&I institutions and enterprises or to the improvement of R&I conditions. It seems that the continuous focus on public R&D spending to rampage R&I capacity in SMEs may well hinder the adoption of new means of stimulating R&I, many of which are better capable of helping EU SMEs [67]. SMEs’ innovation capacity is frequently influenced by immaterial variables, such as the strength of the local and regional R&I mechanisms [65]. Cooperation and networking, either at the firm or organisational levels, are critical for the development and knowledge transfer, which is at the heart of SMEs’ R&I [67]. R&I often occurs in SMEs as a result of a variety of partnerships, including with scientific R&D institutions, along with those based on DUI [24]. As a result, [89] highlight the need to create improved interactions between regional innovation and development policies and funding programmes at both strategic and implementation levels.

5.1. Potential Improvements

The SBM model identifies the required changes that inputs and outputs should undertake for inefficient OPs to become efficient (

Table 9. Improvement potential for the OPs—Overall network cluster efficiency.

Factor	Satatistics	Original	Projection	Variation
Researchers working in improved infrastructures	Mean	1136	6603	481%
	Mean (MD)	1242	12,188	881%
	Mean (T)	1407	1863	32%
	Mean (LD)	592	2920	393%
Enterprises Supported	Mean	1730.00	2116.95	22%
	Mean (MD)	640.00	1007.68	57%
	Mean (T)	2973.00	3671.46	23%
	Mean (LD)	2034.00	2046.68	1%
Enterprises working with research institutions	Mean	452	673	49%
	Mean (MD)	340	499	47%
	Mean (T)	396	473	19%
	Mean (LD)	722	1243	72%
Enterprises supported for new market to products	Mean	192	203	6%
	Mean (MD)	156	166	6%
	Mean (T)	233	249	7%
	Mean (LD)	202	210	4%
Eligible cost decided	Mean	394,392,646	275,299,518	−30%
	Mean (MD)	117,091,229	31,951,742	−73%
	Mean (T)	463,529,105	275,473,478	−41%
	Mean (LD)	794,593,629.00	705,610,405.90	−11%
Total eligible spending	Mean	170,821,334	117,524,353	−31%
	Mean (MD)	59,557,103	22,078,911	−63%
	Mean (T)	226,371,871	138,216,958	−39%
	Mean (LD)	295,030,424	259,329,806	−12%

Note: Negative values refer to improvements in input overuse and positive values refer to output shortages.

). These results are depicted in Figure 5 and Figure 6, both by OP and by region type, where OPs are represented from left to right in decreasing order of efficiency.

The number of researchers working in improved infrastructures shows the largest improvement potential of about 481% (i.e., it should increase on average from 1136 to 6603 researchers)—see Table 9 and Figure 5. In any case, the more developed regions and the less developed regions show the highest room for improvement potentials of 881% and 393%, respectively. In contrast, the lowest improvement potential belongs to the regions of transition (32%)—see Table 9. These results are consistent with those of [29], who concluded that this sort of barrier had the largest restraining effect in more developed regions. Additionally, other studies also highlight the importance of the lack of skills as a hurdle to innovation (e.g., [32,33,34]).

The number of enterprises working with research institutions also presents an overall high potential for improvement (49%), particularly for the less developed regions (72%), followed by more developed regions (47%)—see Table 9 and Figure 5. These outcomes highlight the need to pursue more effective and efficient cooperation with research institutions (see [90]). Under the current output levels, the more developed regions show lower efficiency in terms of the use of funding as they require a reduction of 73% and 63% of eligible costs decided and total eligible spending, respectively, to become efficient—see Table 9 and Figure 6. The number of enterprises supported requires further improvements, particularly for more developed regions (57%) and regions of transition (23%)—see Table 9 and Figure 5. The number of enterprises supported for new-to-the-market products is the factor requiring the lowest adjustments possible (ranging from 4 to 7%)—see Table 9 and Figure 5. Hence, to foster R&I in enterprises, MA should focus on further increasing the number of researchers working in improved infrastructures and the number of enterprises working with research institutions. This is particularly important in less developed and more developed regions. Besides, MA should channel their funding towards less developed regions since these present a better performance when they employ their funding both in eligible costs decided (−11%) and eligible spending (−12%) against more developed regions (−63%)—see Table 9 and Figure 6. These findings are consistent with the European Commission’s Smart Specialisation Strategies for 2021–2027, which identify the major obstacles and next measures required to promote innovation-led growth, particularly in less developed and industrial transition regions [43].

Overall, further support should be given to SMEs in less developed regions regarding the information provided, namely concerning existing funding options. Actions should be promoted to raise awareness of how the market, both at national and international levels, new technologies, and new regulations operate [43,91,92]. Additional efforts should also involve the expansion of the target group enterprises by attracting companies that are not currently innovating but are innovation-oriented. Finally, MA should strengthen the support of partnerships between SMEs and R&I institutions, particularly in less developed regions.

5.2. Sensitivity Analysis

Because the SBM-DEA technique is nonparametric, an alternative way usually employed for performing sensitivity analysis involves removing one aspect of evaluation (input or output) at a time and evaluating the shifts in efficiency obtained [93].

We performed four regression models, with the dependent variables being the scores obtained by removing each evaluation factor at a time and the independent variable being the original score. The slope and the related coefficient of determination (or R-square) are presented for each model. The sensitivity of efficiency to changes in the factors considered may thus be evaluated by the gap between the value one and the slope of the regression function, implying that the sensitivity of efficiency to adjustments in the factors is directly associated with this gap [73,94].

The sensitivity analysis results are depicted in Figure 7 and

Table 10. Sensitivity analysis results.

Variables	Slope	|1-Slope|	R2	Classification
Enterprises Supported	0.8825	0.1175	0.8883	Output P1
Enterprises supported for new-to-the-market products	0.9232	0.0768	0.8989	Output P1
Researchers working in improved infrastructures	0.8561	0.1439	0.8141	Output P2
Enterprises working with research institutions	0.6253	0.3747	0.5826	Output P2

. The factor which shows higher impacts on efficiency is the “number of enterprises working with research institutions” since the omission of this variable leads to the highest value of |1-slope|. The “number of enterprises supported” and the number of “researchers working in improved infrastructures” have an analogous effect on efficiency. The “number of enterprises supported for new-to-the-market products” has the smallest impact on efficiency (Figure 7b). Then again, results suggest that further measures should be adopted to promote cooperation between enterprises and research institutions.

6. Conclusions and Policy Implications

The main goal of this paper was to assess the efficiency of SFs’ implementation of 53 OPs from 19 EU countries in the framework of TO 1, considering procedural and R&I potential capacity generation. The data employed is the most up-to-date and refers to the programming period between 2014–2020. The chosen data is highly suitable to instantiate DEA nonparametric models. The key advantage of utilising this sort of assessment methodology lies in the fine-grain information that they may provide to MA on the inefficiency of the OPs when compared to their benchmarks. In this sense, by identifying the benchmarks of inefficient OPs, the DEA methodology offers particularly relevant information on the best practices that these OPs should undertake to become efficient.

Even though DEA has undeniable convenience over other conventional methodologies (e.g., microeconomic studies with control groups and case study evaluation), there is still a dearth of academic interest in its usage in the context of SFs’ efficiency evaluation. As a result, one of the innovative aspects of our work involves the use of the Network SBM-DEA model in conjunction with cluster analysis to evaluate the efficiency of the implementation of SFs allocated to foster R&I in enterprises, mainly SMEs. The Network SBM-DEA model makes it possible to identify the benchmarks (in terms of the best practices) and the necessary changes that need to be implemented for the factors considered in the evaluation to enhance the OPs implementation. Additionally, it encompasses a two-stage analysis: a procedural efficiency analysis and an evaluation of the potential R&I capacity generation of the programmes under scrutiny. This sort of evaluation is especially important if the programmes are in progress during the monitoring stages since it allows MA to predict the influence that prospective changes in indicators may have on the efficiency scores of OPs. Furthermore, this technique entails the use of sensitivity analysis of the results computed, enabling an understanding of which factors have the greatest influence on efficiency changes.

Unlike other approaches and methodologies employed during the ex-post or ex-ante assessments of cohesion policies, the DEA methodology enables efficiency appraisal of OPs’ implementation throughout the programming period. This makes it possible to adopt the necessary policies to reach efficiency during the period underway. Additionally, DEA can provide actual support in the design of future cohesion policy instruments because it allows an understanding of the main accomplishments and failures of the preceding implementation programming phases. Additionally, it delivers valuable information on what can be done to overcome the identified failures.

In summary, the main conclusions with respect to our research questions are provided.

RQ1. “Does a higher level of procedural efficiency necessarily lead to a higher R&I potential capacity generation?”

 (i)

OPs show higher levels of TGR for procedural efficiency than R&I potential capacity generation, particularly for less developed regions and regions of transition.

(ii)

Our findings seem to corroborate the ‘European paradox,’ since there is an ‘innovation gap’ to the extent that fostering innovation inputs through public funding does not necessarily lead to innovation outputs.

RQ2. “Which performance framework indicators preclude the efficient application of funds committed to boosting R&I in EU SMEs?”

 (i)

Overall, the two main factors that preclude R&I OPs from becoming efficient are the number of researchers working in improved R&I infrastructures and the number of enterprises working with R&I institutions.

(ii)

These results highlight the need to tackle the problem of a lack of skills, a factor that has been often identified as one of the biggest hurdles to innovation. MA should also strengthen the support of partnerships between SMEs and R&I institutions, particularly in less developed regions.

RQ3. “Which performance framework indicators showed higher resilience to efficiency classification in face of their potential shifts”?

From the results of the sensitivity analysis, it can be established that:

 (i)

The “number of firms that are supported for new-to-the-market products” has the least influence on efficiency.

(ii)

More steps should be taken to enhance collaboration and networking between firms and research organizations, because this indicator has the highest influence on efficiency.

RQ4. “Which OPs were most frequently considered as a reference of best practices over the programming period under scrutiny?”

“Aragón” (22), “Brussels Capital Region” (19), “Cohesion Policy Funding—EE” (19), and “Competitiveness Entrepreneurship and Innovation—GR” (18) were the four OPs most frequently considered as a reference for best practices. Two of them belong to more developed regions, one to a less developed region, and one to a region of transition.

RQ5. “Which type of regions showed a higher performance in supporting R&I in SMEs?”

(i)

In general, more developed regions exhibit a higher room for progress (85%), while transition and less developed regions had a smaller capacity for progress of 67% and 65%, respectively. Therefore, our findings show that successful policies should reflect not only the SMEs R&I actions towards their innovation capacity enhancement, but also the region where they are located.

7. Limitations and Future Research Directions

While this study provided novel insights into the success of the efficiency in the implementation of R&I cohesion policy, mainly in European SMEs, it also had its caveats. Despite the performance framework providing a set of implementation indicators, it is not possible to reach a complete match between the data obtained for the OPs’ achievements and their financial implementation. Furthermore, the data provided is frequently incomplete, and as a result, our assessment ended up considering a limited number of indicators and OPs.

Although our study addressed an evaluation approach to use during the monitoring stages of the programming period, further research should contemplate the evaluation in the ex-post phase, particularly focusing on the economic impacts of the OPs in TO 1. This is particularly relevant because input, intermediate, and output impacts are likely to occur across distinct periods. Varga et al. [95] concluded that the evaluation of the economic impacts in a smart specialisation policy context is critical, but it is still a challenging endeavour. In this sense, economic impact assessment seeks to estimate the potential effects of policy interventions on economic indicators, such as GDP or employment, as well as the influence of policies on territorial disparities. As a result, the goal of economic impact assessment differs significantly from the evaluation level of implementation of the OPs in that it includes both direct and indirect aggregate effects of the projects (contemplating Keynesian demand multiplier effects, supply effects generated by inter-industry linkages, or knowledge spill-overs).