Two-Stage Data Envelopment Analysis Models with Negative System Outputs for the Efficiency Evaluation of Government Financial Policies

Abstract

The main purpose of this study is to provide a comparative analysis of several possible approaches to applying data envelopment analysis (DEA) in the case where some decision making units (DMUs) in the original sample have negative system outputs. In comparison to the traditional model of Charnes, Cooper, and Rhodes (CCR) and the CCR model with a scale shift to measure second-stage outputs, the range directional measure (RDM) model produces the most appropriate results. In this paper, an approach is proposed for estimating returns to scale. The study applies a two-stage DEA model with negative second-stage outputs to assess the public support for research, development, and demonstration projects in the energy sector in 23 countries over the period from 2010 to 2018. The assessment of government performance depends on its contribution to the growth of energy efficiency in the national economy and the reduction of its carbon intensity. Intermediate outputs (patents in the energy sector) are included in the analysis as both outputs of the first stage and inputs of the second stage. Taking the similarity between the calculations obtained without stage separation and the system efficiency calculations from the two-stage model as a measure of model adequacy, the RDM model shows the highest similarity scores.

1. Introduction

In recent years, there has been growing interest in network and multi-stage Data Envelopment Analysis (DEA) models in the scientific literature [1,2,3]. Unlike conventional models that deal with Decision Making Units (DMUs) as systems of the “black box” type, in which only inputs and outputs are known, multi-stage and network models make it possible to use a priori knowledge about the structure of the economic agent under study. This allows one to consider the economic agent (DMU) as a system with several subsystems connected to each other by various functional links. In the simplest case, a production facility can be considered as a system or process consisting of two subsystems/sub-processes interconnected by so-called intermediate outputs: the outputs of the first stage, which are also the inputs of the second stage. This approach is currently quite popular and is practiced for modeling the efficiency of banks [4,5,6,7], insurance companies [8,9,10], innovative and high-tech enterprises [11,12,13], production and logistics chains [14,15,16], educational and medical institutions [17,18,19,20,21], and many other types of economic units.

A growing body of literature has been devoted to the modeling, using multi-stage DEA models, the performance of various institutional systems on a macroeconomic scale, such as national/regional innovation systems, or national/regional environmental management systems [22,23,24,25,26,27,28]. Macroeconomic-scale systems, as a rule, have a complex structure, for better understanding of which it is reasonable to decompose them into smaller subsystems. A common feature of such models is the fact that the inputs of the first stage are considered to be government spending for achieving certain political goals, including financial spending in the form of various subsidies, grants, or direct government funding. Stated policy objectives are systemic outputs, while intermediate outputs may be some measure of public spending performance, such as the number of innovation/environmental projects completed, the number of intellectual property created, the number of activities carried out, etc. Examples of political goals could be sustainable development, environmental improvement, energy efficiency, income inequality reduction, etc. Most often, they are measured in terms of the growth or decline of an indicator relative to some base period.

In real practical cases, the stated political goals may not be achieved, at least in the time under study [29,30,31]. This is due not necessarily to the inefficiency of public spending, but also to the presence of multidirectional processes in the macroeconomic system. For example, the efforts of the state aimed at increasing the energy efficiency of the economy can be nullified by a general decline in business activity and a fall in gross domestic product (GDP). Funds aimed at improving the state of the environment may not bring positive changes if the capacities of hazardous industries increase. From a mathematical perspective, this means that DMU system outputs can take negative values.

Approaches to solving the problem of negative data in the DEA have been known for a long time. For example, the paper of Portela, Thanassoulis, and Simpson is highly cited in DEA literature as one of the first papers on the topic of dealing with negative data in DEA [32]. Further, several approaches were proposed based on modified slack-based measures [33,34,35,36,37], semi-oriented radial measures [38,39,40], and others [41,42].

However, the features of two-stage DEA models with systemic negative outputs (i.e., negative outputs only at the second stage) have not yet been sufficiently studied in the literature. In particular, there has been little discussion on the question of what approaches to use and how to calculate the return on scale, coefficients of technical efficiency, scale efficiency for the first and second stages, and the entire DMU.

The aim of this paper is to compare several approaches to solving two-stage DEA models with negative system outputs. We consider only additive efficiency aggregation in two-stage models, since multiplicative types do not allow us to determine targets.

Constructions of economic functions in the paper (production functions, isoquants, and so on) were made on the basis of methods proposed in the paper [43] using the software FrontierVision, version 2.1 [44].

The remainder of this paper is structured as follows: Section 2 gives a brief literature review on the topic of negative data in network DEA models. Section 3 provides a description of different approaches and presents our approach, followed by the case study in Section 4. Section 5 concludes and proposes the direction for future research.

2. Literature Review

Among the first DEA models dealing with negative data, we should mention the study by Scheel [45], where he suggests treating absolute values of negative outputs as inputs and absolute values of negative inputs as outputs, and then the study of Portela et al. [32], which proposes a range directional measure (RDM) approach. A few years later, a modified slack-based measure was introduced by Sharp et al. [33] and a semi-oriented radial measure (SORM) by Emrouznejad et al. [38]. Cheng et al. [41] proposed a modified traditional radial DEA model where the original values of inputs and outputs were replaced with absolute values as a basis for quantifying the share of improvements needed to reach the frontier. Tohidi and Khodadadi [46] extended the RDM model for the evaluation of cost and allocative efficiencies in the presence of negative data. Later in the literature, these models were extended and developed for the case of multi-stage and network DEA models.

For example, Lu et al. [47] introduce a two-stage dynamic DEA model for evaluating the operating efficiency and profitability of credit unions associated with farmer’s associations in Taiwan. Capital expenses ($X_1$), labor expenses ($X_2$), and common elements ($X_3$) are taken as inputs at the first stage. They generate common loans ($Z_1$), policy loans ($Z_2$), and non-interest income ($Z_3$) to be used as intermediate outputs along with non-performing loans ($Z_4$) representing undesirable output. At the second stage, authors take the intermediate outputs to produce the final income (Y), which obviously can be negative. The authors compare numerical results obtained with RDM, proportional distance function measure (PDFM), and SORM models and come to the conclusion that SORM is the best for estimating the operating performance of credit unions.

Izadikhah and Saen [48] suggest a new DEA non-radial model for finding the efficiency measure in the presence of negative data. In the proposed model, all inputs, intermediate measures, and outputs are assumed to be negative or positive. The authors consider a numerical example for the supply chain, which produces equipment for expendable medical devices. In this example, one of the intermediate products (rate of increasing partnership cost in green production plans) and two system outputs (rate of increasing number of green products, rate of increasing revenue) can have negative values.

Kong et al. [49] modified the RDM two-stage DEA model to measure the efficiency of DMUs in the presence of both negative data and undesirable outputs. They applied the proposed model to the problem of evaluating Taiwanese bank efficiency at both the operational and profitability stages. As inputs in the first stage (operational), they consider operational expenses, loanable funds, and capital stock. The operational stage has one final output (service fee) and three intermediate outputs (performing loans, investments, and nonperforming loans). One intermediate output (nonperforming loans) is considered undesirable. As system outputs, the model uses interest income and investment revenue, where at least the second output can have negative values. Comparing the results of the numerical example performed by two different models, the study obtained that the operational efficiency scores calculated by two independent RDM models was higher than that measured with the help of the proposed model, while the profitability efficiency scores obtained by the RDM-type model were higher in comparison with two independent RDM models.

The study by Tavana et al. [50] builds a new two-stage DEA model that can handle negative input, intermediate, and output variables. The authors have developed an extended RDM model [32] to a two-stage RDM model and showed that it has some useful properties. They also provided a simple example in order to demonstrate the ability of the proposed model to evaluate the efficiency of two-stage processes in the presence of negative data. In the next stage of research, they build a novel dynamic two-stage DEA model that allows for negative input–intermediate–output data, as well as for both desirable and undesirable carryovers (links between time moments). They obtained new dynamic model that has been used for further extending the proposed two-stage RDM model. Also, it was shown that proposed model can be applied for several management areas, for example, banking.

Kianfar et al. [51] propose multiobjective programming for two- and three-stage DEA problems with negative data. They introduce two separate objective functions for semi-negative and semi-positive inputs and outputs. The authors argue that, in cases where DMUs have more than two stages, solving a multiobjective programming model is necessary to calculate the overall efficiency. The strength of the proposed method is the possibility to extend it for a case of a k-stage network in the presence of negative data.

Kao [52] proposes a generalized radial model that can be used in the presence of negative data by applying a more general production possibility set that only requires the aggregate input and aggregate output to be positive. These constraints helps to deal with the unreasonable situation when inputs are consumed in the production process while the outputs are being accumulated, or another unreasonable situation where the products are produced in the production process while the resources are being generated. The model is valid for the assumptions of both constant and variable returns to scale, and can be extended to network systems. This paper introduces the simplest extension of the two-stage system. For the two-stage system under constant returns to scale, the system efficiency obtained from the proposed model can also be decomposed into the product of the two sub-system efficiencies.

Li et al. [53] divided the operational process of Internet banks into the value operation stage (stage 1) and the value creation stage (stage 2). Next, they applied the two-stage network DEA model, where net assets, R&D investment, and the number of employees are inputs. Total deposits (including interbank deposits) are an intermediate output. Non-interest income, net interest income, and non-performing loan ratio are system outputs. In this model, non-interest income can have negative values. The authors apply the efficiency coefficient method (or the extremum method) to positively standardize the 2018–2019 data with the indicator data.

The paper of Babaie Asil et al. [54] introduces a two-stage bounded additive model to evaluate the efficiency of airlines and uses fuel cost, maintenance expenses, labor expenses, and fleet size as input values; available ton miles and available seat miles as intermediate values; and revenue passenger mile, revenue ton mile and net income as output values. In the real-life experiment, many companies have negative net income, which makes this problem a two-stage DEA model with negative outputs. The authors propose a two-stage model with constant returns to scale (CRS), and show that it deals with negative outputs quite effectively.

Azadi et al. [55] developed a new network RDM model to measure the sustainability and resilience of healthcare supply chains, which consist of suppliers of medical goods and equipment and hospitals. The model can address different types of data, including negative ones.

3. Materials and Methods

3.1. DEA Background

Consider a set of n observed DMUs $(X_j,Y_j)$, $j=1,\dots ,n$, where output vector $Y_j={(y_{1j},\dots ,y_{rj})}^{\mathsf{T}}\in {\mathbb{E}}^r$ is produced from input vector $X_j={(x_{1j},\dots ,x_{mj})}^{\mathsf{T}}\in {\mathbb{E}}^m$. A production possibility set (PPS) is determined on a set of axioms using the observed set of production units. For the constant returns-to-scale model, PPS is written in the following form:

$$T=\left\{(X,Y)|X\ge {\displaystyle \sum _{j=1}^n}{\lambda }_j{X}_j,Y\le {\displaystyle \sum _{j=1}^n}{\lambda }_j{Y}_j,{\lambda }_j\ge 0,j=1,\dots ,n\right\}$$

(1)

Using the constraints of set (1), the output-oriented CRS model [56] for DMU${}_o$ is written as follows:

$$\begin{array}{cc}\hfill \max & \eta +\epsilon \left({\displaystyle \sum _{k=1}^m}{s}_k^{-}+{\displaystyle \sum _{i=1}^r}{s}_i^{+}\right)\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{X}_j{\lambda }_j+S^{-}=X_o\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{Y}_j{\lambda }_j-S^{+}=\eta Y_o\hfill \\ \hfill & S^{-}\ge 0,S^{+}\ge 0,\hfill \\ \hfill & {\lambda }_j\ge 0,j=1,\dots ,n\hfill \end{array}$$

(2)

where ${\lambda }_j$, $j=1,\dots ,n$ are intensity variables; $S^{-}={(s_1^{-},\dots ,s_m^{-})}^{\mathsf{T}}$ and $S^{+}={(s_1^{+},\dots ,s_r^{+})}^{\mathsf{T}}$ are slack variables; the unrestricted variable $\eta$ determines the radial expansion of the output vector $Y_o$; $\epsilon$ is a non-Archimedean infinitesimal constant, which prevents weights from being zero. Without using $\epsilon$, problem (2) has to be solved in two stages [57]. The output efficiency score for DMU${}_o$ is obtained as $1/{\eta }^{*}$, where ${\eta }^{*}$ is optimal for (2).

The variable returns-to-scale (VRS) model [58] can be obtained from model (2) by restricting the sum of $\lambda$-variables equal to one.

3.2. Two-Stage Models with Negative Outputs

In the two-stage DEA model with a basic structure, it is assumed that each DMU${}_j$ in the first stage uses input $X_j$ to produce intermediate product $Z_j={(z_{1j},\dots ,z_{dj})}^{\mathsf{T}}\in {\mathbb{E}}^d$. Then, $Z_j$ is considered input in the second stage to produce output $Y_j$.

In the two-stage CRS model, the production possibility set T can be written as follows:

$$\begin{array}{c}{T}_{\mathrm{CRS}}=\left\{(X,Y,Z)\right|{\displaystyle \sum _{j=1}^n}{X}_j{\lambda }_j\le X,{\displaystyle \sum _{j=1}^n}{Z}_j{\lambda }_j\ge Z,{\displaystyle \sum _{j=1}^n}{Z}_j{\mu }_j\le Z,\hfill \\ {\displaystyle \sum _{j=1}^n}Y{\mu }_j\ge Y,{\lambda }_j\ge 0,{\mu }_j\ge 0,j=1,\dots ,n\}\hfill \end{array}$$

(3)

According to the structure of set (3), we can write the following aggregated two-stage output-oriented CRS model to measure the efficiency of DMU${}_o$:

$$\begin{array}{cc}\hfill \max & w_1{\eta }_1+w_2{\eta }_2\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{X}_j{\lambda }_j\le X_o\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{Z}_j{\lambda }_j\ge {\eta }_1{Z}_o\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{Z}_j{\mu }_j\le Z_o\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}Y{\mu }_j\ge {\eta }_2{Y}_o\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{Z}_j{\lambda }_j\ge {\displaystyle \sum _{j=1}^n}{Z}_j{\mu }_j\hfill \\ \hfill & {\lambda }_j\ge 0,{\mu }_j\ge 0,j=1,\dots ,n\hfill \end{array}$$

(4)

where weights $w_1$ and $w_2$ represent the preferences over the two stages and satisfy $w_1+w_2=1$; the last constraint represents the linkage between stages. Note that the infinitesimal constant $\epsilon$ has been ignored in this model to make the expression simpler.

In model (4), the individual efficiencies of the first and second stages are determined as $1/{\eta }_1^{*}$ and $1/{\eta }_2^{*}$, respectively. The overall system efficiency can be calculated as $1/(w_1{\eta }_1^{*}+w_2{\eta }_2^{*})$.

For the VRS case, the production possibility set T of the two-stage model has the following form:

$$\begin{array}{c}{T}_{\mathrm{VRS}}=\left\{(X,Y,Z)\right|{\displaystyle \sum _{j=1}^n}{X}_j{\lambda }_j\le X,{\displaystyle \sum _{j=1}^n}{Z}_j{\lambda }_j\ge Z,{\displaystyle \sum _{j=1}^n}{Z}_j{\mu }_j\le Z,{\displaystyle \sum _{j=1}^n}Y{\mu }_j\ge Y,\hfill \\ \hfill {\displaystyle \sum _{j=1}^n}{\lambda }_j=1,{\displaystyle \sum _{j=1}^n}{\mu }_j=1,{\lambda }_j\ge 0,{\mu }_j\ge 0,j=1,\dots ,n\}.\end{array}$$

(5)

Based on the corresponding PPS (5) output-oriented VRS model is written as:

$$\begin{array}{cc}\hfill \max & w_1{\eta }_1+w_2{\eta }_2\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{X}_j{\lambda }_j\le X_o\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{Z}_j{\lambda }_j\ge {\eta }_1{Z}_o\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{Z}_j{\mu }_j\le Z_o\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}Y{\mu }_j\ge {\eta }_2{Y}_o\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{Z}_j{\lambda }_j\ge {\displaystyle \sum _{j=1}^n}{Z}_j{\mu }_j\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{\lambda }_j=1,{\displaystyle \sum _{j=1}^n}{\mu }_j=1,\hfill \\ \hfill & {\lambda }_j\ge 0,{\mu }_j\ge 0,j=1,\dots ,n\hfill \end{array}$$

(6)

The advantage of model (6) over other network DEA formulations is that returns to scale and frontier projection can be obtained with the help of conventional DEA methods.

It is known that the CRS model cannot handle data with negative inputs and outputs. However, it can be applied if only negative outputs are present in the data and all inputs are nonnegative. This can be illustrated in Figure 1. This figure represents a two-dimensional cut of the frontier (output isoquant) for a three-dimensional one-input/two-output CRS model that is constructed using only DMUs with positive outputs. Points A, B, and C are efficient in this model. Unit D has one positive and one negative output. The efficiency scores of this unit can be measured relative to their radial projection $D_1$ onto the frontier as $O{D}_1/OD$. However, if all outputs of a DMU are negative, as for unit E, then the standard CRS model becomes unbounded. For this reason, all coordinates of unit E should be multiplied by $-1$ to obtain unit $E_2$. After that, the efficiency score of unit E can be measured relative to the radial projection of unit $E_2$ as $-O{E}_2/O{E}_1$. This means that, for units with all negative outputs, the efficiency score is negative.

In the presence of negative outputs in the VRS model, Kao [52] proposed restricting total virtual input ${\displaystyle {\displaystyle \sum _{k=1}^m}}{v}_k{X}_{kj}$ and total virtual output ${\displaystyle {\displaystyle \sum _{i=1}^r}}{u}_i{Y}_{ij}$ to being nonnegative. This leads to the following model:

$$\begin{array}{cc}\hfill \min & {\displaystyle \sum _{k=1}^m}{v}_k{X}_{ko}+u_0\hfill \\ \hfill & {\displaystyle \sum _{i=1}^r}{u}_i{Y}_{io}=1,\hfill \\ \hfill & {\displaystyle \sum _{k=1}^m}{v}_k{X}_{kj}+{\displaystyle \sum _{i=1}^r}{u}_i{Y}_{ij}+u_o\le 0,j=1,\dots ,n,\hfill \\ \hfill & {\displaystyle \sum _{k=1}^m}{v}_k{X}_{kj}\ge 0,j=1,\dots ,n,\hfill \\ \hfill & {\displaystyle \sum _{i=1}^r}{u}_i{Y}_{ij}\ge 0,j=1,\dots ,n,\hfill \\ \hfill & v_k\ge \epsilon ,k=1,\dots ,m,\hfill \\ \hfill & u_i\ge \epsilon ,i=1,\dots ,r.\hfill \end{array}$$

(7)

This model is, in fact, the model with restricted multipliers. Figure 2 explains how efficiency scores are measured in model (7). The frontier of the VRS model is depicted with dashed lines. In model (7), the production possibility set expands in such a way that ray $C{D}_1$ is parallel to segment $OE$ and ray $A{E}_1$ is parallel to line $OD$.

Hence, efficient units in the VRS model may become inefficient in model (7). Furthermore, model (7) does not guarantee the efficiency estimation for all DMUs because, for some units with negative outputs, it may become infeasible. For example, the darkened area in Figure 2 indicates the region where the projection onto the frontier does not exist; therefore, the model (7) for unit F becomes infeasible.

The traditional approach to dealing with negative data in VRS technology is to use the translation invariance property of DEA models. Translation invariance means that the shift of inputs and outputs by a scalar does not affect the efficiency scores. For example, the input-oriented VRS model is translation invariant in the outputs. After transformation

$${\tilde{y}}_{ij}=y_{ij}-{\displaystyle \underset{k}{min}}{y}_{ik},j=1,\dots ,n,$$

for all outputs i that contain negative values, all DMUs are located in a positive orthant and the traditional VRS model can be applied.

However, using this approach, units with negative outputs may be assessed as efficient; see Figure 3. In this figure, unit A has a negative output. Then, after the shift of output y, the origin moves to point $O_1$ and unit A is assessed as efficient. Such a situation is unacceptable from a managerial point of view, since unit A outperforms unit E with positive outputs.

Some non-radial DEA models (RAM, SBM, etc.) are translation invariant; hence, they can handle negative values in data [35,36,37]. However, they suffer from the same weaknesses as illustrated in Figure 3.

There exist DEA models with VRS technology that do not require a scale shift. The most popular is the RDM model, which is successfully used for multi-stage DEA models and, at the same time, is able to work with negative indicators.

Next, we describe how the RDM model operates under a two-stage DEA [50]. Introduce two ideal points $I_1$ and $I_2$ for the first and second stages as follows:

$$\begin{array}{cc}\hfill I_1& =\left\{{\overline{x}}_1,\dots ,{\overline{x}}_m,{\overline{z}}_1^1,\dots ,{\overline{z}}_d^1\right\},\hfill \\ \hfill I_2& =\left\{{\overline{z}}_1^2,\dots ,{\overline{z}}_d^2,{\overline{y}}_1,\dots ,{\overline{y}}_r\right\},\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill {\overline{x}}_i& =x_{ij},i=1,\dots ,m.\hfill \\ \hfill {\overline{z}}_q^1& =z_{qj},q=1,\dots ,d,\hfill \\ \hfill {\overline{z}}_q^2& =z_{qj},q=1,\dots ,d,\hfill \\ \hfill {\overline{y}}_k& =y_{kj},k=1,\dots ,r.\hfill \end{array}$$

Let

$$\begin{array}{cc}\hfill \left(R_p^X,R_p^{Z1}\right)& =\left(R_{1p}^X,\dots ,R_{mp}^X,R_{1p}^{Z1},\dots ,R_{dp}^{Z1}\right),\hfill \\ \hfill \left(R_p^{Z2},R_p^Y\right)& =\left(R_{1p}^{Z2},\dots ,R_{dp}^{Z2}{R}_{1p}^Y,\dots ,R_{rp}^Y\right),\hfill \end{array}$$

be the directional vectors from DMU${}_p$ to ideal points $I_1$ and $I_2$ in the first and second stages, respectively. Components of these vectors are written as follows:

$$\begin{array}{cc}\hfill R_{ip}^X& =x_{ip}-{\overline{x}}_i,i=1,\dots ,m,\hfill \\ \hfill R_{qp}^{Z1}& ={\overline{z}}_q^1-z_{qp},q=1,\dots ,d,\hfill \\ \hfill R_{qp}^{Z2}& =z_{qp}^2-{\overline{z}}_q^2,q=1,\dots ,d,\hfill \\ \hfill R_{kp}^Y& ={\overline{y}}_k-y_{kp},k=1,\dots ,r.\hfill \end{array}$$

The RDM two-stage DEA model is written in the following form:

$$\begin{array}{cc}\hfill \max & w_1{\theta }_1+w_2{\theta }_2\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{x}_{ij}{\lambda }_j\le x_{ip}-{\theta }_1{R}_{ip}^X,i=1,\dots ,m,\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{z}_{qj}{\lambda }_j\ge z_{qp}+{\theta }_1{R}_{qp}^{Z1},q=1,\dots ,d,\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{\lambda }_j=1,{\lambda }_j\ge 0,j=1,\dots ,n,\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{z}_{qj}{\mu }_j\le z_{qp}-{\theta }_2{R}_{qp}^{Z2},q=1,\dots ,d,\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{y}_{kj}{\mu }_j\ge y_{kp}+{\theta }_2{R}_{kp}^Y,k=1,\dots ,r,\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{\mu }_j=1,{\mu }_j\ge 0,j=1,\dots ,n,\hfill \\ \hfill & {\displaystyle \sum _{j=1}^n}{z}_{qj}{\lambda }_j\ge {\displaystyle \sum _{j=1}^n}{z}_{qj}{\mu }_j,q=1,\dots ,d,\hfill \end{array}$$

(8)

where importance weights $w_1$ and $w_2$ show the preferences in the two stages and satisfy $w_1+w_2=1$. The first three constraints correspond to the first stage of technology; the following set of constraints determine the second stage; while the last constraint links the two stages.

Model (8) is invariant with respect to translations and units of measurement. The efficiency scores in the first and second stages are acquired as ${\rho }_1=1-{\theta }_1^{*}$ and ${\rho }_2=1-{\theta }_2^{*}$, where ${\theta }_1^{*}$ and ${\theta }_1^{*}$ are optimal variables in model (8). The total efficiency is calculated as the weighted sum of the efficiency measures for each stage, and can be determined as $\rho =w_1{\rho }_1+w_2{\rho }_2=1-(w_1{\theta }_1^{*}+w_2{\theta }_2^{*})$.

In DEA applications, calculating returns to scale (RTS) is highly important because it helps DMUs measure the relationship between their inputs and outputs to analyze how their investments are generating proper returns or to adjust production. Therefore, VRS technology is generally more preferable than CRS.

3.3. Estimation of RTS in a Two-Stage Model with Negative Outputs

Although the RDM model is based on VRS technology, it cannot be used to measure the returns to scale, since the projection of the DMU on the efficient frontier may have negative values. Therefore, we propose the following approach to measure the returns to scale in the presence of negative outputs.

Let $J^{+}$ be a subset of production units with nonnegative inputs and outputs:

$$J^{+}=\left\{j|X_j\ge 0,Y_j\ge 0,j=1,\dots ,n\right\}.$$

(9)

Define the production possibility set $T^{+}$ on the basis of set $J^{+}$ as follows:

$$T^{+}=\left\{(X,Y)|X\ge {\displaystyle \sum _{j\in J^{+}}}{X}_j{\lambda }_j,Y\le {\displaystyle \sum _{j\in J^{+}}}{Y}_j{\lambda }_j,{\displaystyle \sum _{j\in J^{+}}}{\lambda }_j=1,{\lambda }_j\ge 0,j\in J^{+}\right\}.$$

(10)

For units from $J^{+}$, returns to scale can be measured in technology $T^{+}$ using conventional methods [59,60].

Define the set of units with negative outputs as $J^{-}=J\setminus J^{+}$. Next, for each DMU${}_p$, $p\in J^{-}$, we try to find the projection onto the efficient frontier by solving the following output-oriented model:

$$\begin{array}{cc}\hfill \max & \eta +\epsilon \left({\displaystyle \sum _{k=1}^m}{s}_k^{-}+{\displaystyle \sum _{i=1}^r}{s}_i^{+}\right)\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{X}_j{\lambda }_j+S^{-}=X_p,\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{Y}_j{\lambda }_j-S^{+}=\eta Y_p,\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{\lambda }_j=1,\hfill \\ \hfill & S^{-}\ge 0,S^{+}\ge 0,\hfill \\ \hfill & {\lambda }_j\ge 0,j=1,\dots ,n.\hfill \end{array}$$

(11)

If an optimal solution is found in (11), then returns to scale of DMU${}_p$ is measured at the projection point $\left(X_p-S^{-*},{\eta }^{*}{Y}_p+S^{+*}\right)$ of set $T^{+}$. If model (11) is infeasible for DMU${}_p$, then we find the closest point ${\tilde{Z}}_p=({\tilde{X}}_p,{\tilde{Y}}_p)$ of the set $T^{+}$ by solving the following optimization problem:

$$\begin{array}{cc}\hfill & \min \rho \hfill \\ \hfill & {∥w∥}_p\le \rho ,\hfill \\ \hfill & \left(Z_p+w\right)\in T^{+},\hfill \end{array}$$

(12)

where $Z_p=(X_p,Y_p)$. We consider two cases of the p-norm, where $p=1$ and $p=\infty$. For the first case, ${∥w∥}_1={\sum }_i\left|w_i\right|$, consider the following LP problem:

$$\begin{array}{cc}\hfill \min & {\displaystyle \sum _{k=1}^{m+r}}{w}_k\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{X}_j{\lambda }_j\le X_p+w^x,\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{Y}_j{\lambda }_j\ge Y_p-w^y,\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{\lambda }_j=1,\hfill \\ \hfill & {\lambda }_j\ge 0,j=1,\dots ,n,\hfill \\ \hfill & w^x\ge 0,w^y\ge 0,\hfill \end{array}$$

(13)

where $w^x={(w_1,\dots ,w_m)}^{\mathsf{T}}$ and $w^y={(w_{m+1},\dots ,w_{m+r})}^{\mathsf{T}}$.

Proposition 1.

Model (13) minimizes the distance with respect to the 1-norm between unit $(X_p,Y_p)\notin T^{+}$ and set $T^{+}$.

Proof. 

To prove the equivalence of (12) and (13), we represent space ${\mathbb{E}}^{m+r}$ as a sum of orthants

$${\mathbb{E}}^{m+r}={\mathbb{E}}_1^{m+r}\cup \dots \cup {\mathbb{E}}_l^{m+r}\cup \dots \cup {\mathbb{E}}_N^{m+r},$$

(14)

where

$${\mathbb{E}}_l^{m+r}=\left\{{\displaystyle \sum _{i\in L_l^{+}}}{\alpha }_i{e}_i-{\displaystyle \sum _{k\in L_l^{-}}}{\beta }_k{e}_k|{\alpha }_i\ge 0,{\beta }_k\ge 0\right\}.$$

Here, $e_i$ and $e_k$ are $(m+r)$-identity vectors with one in position i and k, respectively, and $j=1,\dots ,N$. The index sets $L_l^{+}$ and $L_l^{-}$ satisfy the conditions $L_l^{+}\cap L_l^{-}=\varnothing$, and $L_l^{+}\cup L_l^{-}=L=\{1,\dots ,m+r\}$. Let the first orthant ${\mathbb{E}}_1^{m+r}$ be ${\mathbb{E}}_{+}^{m+r}$; the numbering of the rest is arbitrary.

Allowing for (14), we can reduce problem (12) to the solution of a family of the following problems $P_l$, $j=1,\dots ,N$:

$$\begin{array}{cc}\hfill \min f_l=& {\displaystyle \sum _{k=1}^{m+r}}{w}_k\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{x}_{ij}{\lambda }_j\le x_{ip}+w_i,i\in L_l^{+}\cap I^x,\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{x}_{ij}{\lambda }_j\le x_{ip}-w_i,i\in L_l^{-}\cap I^x,\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{y}_{ij}{\lambda }_j\ge y_{ip}+w_i,i\in L_l^{+}\cap I^y,\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{y}_{ij}{\lambda }_j\ge y_{ip}-w_i,i\in L_l^{-}\cap I^y,\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{\lambda }_j=1,{\lambda }_j\ge 0,j=1,\dots ,n,\hfill \\ \hfill & w_i\ge 0,i=1,\dots ,m+r,\hfill \end{array}$$

(15)

where $I^x=\{1,\dots ,m\}$ and $I^y=\{m+1,\dots ,m+r\}$. Note that some of the problems $P_l$ may be unbounded.

Let $f_l^{*}$ be the optimal objective value of the problem $P_l$. Then, the minimal value $f^{*}={\min }_{l\in L}{f}_l^{*}$ yields the optimal value of the problem (12). Next, we will show that it is sufficient to solve only the problem $P_1$ to find $f^{*}$.

The problem $P_1$ from the family (15) is written as (13). Consider any other problem $P_l$ from the family (15) with a finite optimal objective value. The dual problem $D{P}_l$ can be written as follows:

$$\begin{array}{cc}\hfill \max g_l=& {\displaystyle \sum _{i=1}^r}{u}_i{y}_{ip}-{\displaystyle \sum _{i=1}^m}{v}_i{x}_{ip}+u_0\hfill \\ \hfill & {\displaystyle \sum _{i=1}^r}{u}_i{y}_{ij}-{\displaystyle \sum _{i=1}^m}{v}_i{x}_{ij}+u_0\le 0,j\in J^{+},\hfill \\ \hfill & 0\le v_i\le 1,\hspace{1em}\hspace{1em}i\in L_l^{+}\cap I_x,\hfill \\ \hfill & 0\le v_i,\hspace{1em}\hspace{1em}\hspace{1em} i\in L_l^{-}\cap I_x,\hfill \\ \hfill & 0\le u_i\le 1,\hspace{1em}\hspace{0.17em}\hspace{0.17em}i\in L_l^{-}\cap I_y,\hfill \\ \hfill & 0\le u_i,\hspace{1em}\hspace{1em}\hspace{1em} i\in L_l^{+}\cap I_y.\hfill \end{array}$$

(16)

For the problem $D{P}_1$, the constraints on variables $u_i$ and $v_i$ are written as:

$$\begin{array}{cccc}& 0\le v_i\le 1,\hfill & \hfill & i=1,\dots ,m,\hfill \\ & 0\le u_i\le 1,\hfill & & i=1,\dots ,r.\hfill \end{array}$$

From the structure of (16), it follows that the feasible set of $D{P}_l$ includes the feasible set of $D{P}_1$. Therefore, the optimal objective values satisfy $g_1^{*}\le g_l^{*}$. This implies that optimal objective values of primal problems $P_l$ obey $f_1^{*}\le f_l^{*}$.

Since we took $P_l$ arbitrary, $f_1^{*}={\min }_{l\in L}{f}_l^{*}$. This completes the proof. □

The coordinates of the closest point according to the 1-norm are determined as ${\tilde{Z}}_p=(X_p+w^{x*},Y_p-w^{y*})$, where $w^{x*}$ and $w^{y*}$ are optimal in (13).

For the infinity norm, ${∥w∥}_{\infty }={\max }_i\left|w_i\right|$, problem (12) can be written in the following equivalent LP form:

$$\begin{array}{cc}\hfill & \min \rho \hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{X}_j{\lambda }_j\le X_p+\rho d_1^{x\phantom{(}},\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{Y}_j{\lambda }_j\ge Y_p+\rho d_1^y,\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{\lambda }_j=1,\hfill \\ \hfill & {\lambda }_j\ge 0,j=1,\dots ,n,\hfill \end{array}$$

(17)

where $d_1^{x\phantom{(}}=(1,\dots ,1)\in {\mathbb{E}}^m$, $d_1^y=(-1,\dots ,-1)\in {\mathbb{E}}^r$.

Proposition 2.

Model (17) minimizes the distance with respect to the infinity norm between unit $(X_p,Y_p)\notin T^{+}$ and set $T^{+}$.

Proof. 

Choose an arbitrary direction $d_2=(d_2^{x\phantom{(}},d_2^y)\in {\mathbb{E}}^{m+r}$ such that ${∥d_2∥}_{\infty }=1$ and $d_2\ne d_1$. Consider the following linear program:

$$\begin{array}{cc}\hfill & \min \rho \hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{X}_j{\lambda }_j\le X_p+\rho d_2^{x\phantom{(}},\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{Y}_j{\lambda }_j\ge Y_p+\rho d_2^y,\hfill \\ \hfill & {\displaystyle \sum _{j\in J^{+}}}{\lambda }_j=1,\hfill \\ \hfill & {\lambda }_j\ge 0,j=1,\dots ,n.\hfill \end{array}$$

(18)

Problem (18) determines the minimal distance between point $(X_p,Y_p)$ and set $T^{+}$ according to the infinity norm along the direction $d_2$. Observe that the finite solution of problem (18) exists for some direction $d_2$, but not for any direction.

Take the direction $d_2$ for which the finite solution of (18) exists. By construction, we have ${∥d_1∥}_{\infty }={∥d_2∥}_{\infty }=1$.

Let ${\rho }_1^{*}$ and ${\rho }_2^{*}$ denote the objective values of Problems (17) and (18), respectively. Due to the optimality of ${\rho }_1^{*}$ and ${\rho }_2^{*}$, we obtain

$$\begin{array}{cc}& Z_1=Z_p+{\rho }_1^{*}{d}_1\in T^{+},\hfill \\ & Z_2=Z_p+{\rho }_2^{*}{d}_2\in T^{+}.\hfill \end{array}$$

Assume that ${\rho }_2^{*}<{\rho }_1^{*}$. Consider the vector $d3=d1-d2$; according to the construction of $d_1$ and $d_2$, vector $d_3\ge 0$, and at least one component of $d_3$ is strictly positive.

Taking into account the monotonicity of set $T^{+}$, we obtain

$$Z_3=Z_2+{\rho }_2^{*}(d_1-d_2)=Z_p+{\rho }_2^{*}{d}_1\in T^{+}.$$

Thus, the unit $Z_3$ satisfies the constraints of Problem (18) and ${\rho }_2^{*}<{\rho }_1^{*}$, violating the minimality of ${\rho }_1^{*}$ in (17). From this, it follows that ${\rho }_1^{*}\le {\rho }_2^{*}$. This completes the proof. □

The closest point ${\tilde{Z}}_p\in T^{+}$ is obtained as ${\tilde{Z}}_p=(X_p+{\rho }^{*}{d}_1^{x\phantom{(}},Y_p-{\rho }^{*}{d}_1^y)$, where ${\rho }^{*}$ is the optimal objective value of (17).

Since point ${\tilde{Z}}_p\in T^{+}$, the returns to scale can be evaluated at the projection point $\left({\tilde{X}}_p-S^{-*},{\eta }^{*}{\tilde{Y}}_p+S^{+*}\right)$ obtained by solving (11).

This approach is illustrated in Figure 4.

Units B, C, and D are from set $J^{+}$, and returns to scale for them are measured using well-known DEA methods. Next, for DMUs with negative outputs, we try to find an output projection onto the efficient frontier of the set $T^{+}$. For unit F from set $J^{-}$, such a projection exists; hence, the returns to scale are measured for F at the projection point $F_1$. For unit A from set $J^{-}$, the output model (11) is infeasible, and then the nearest point $A_1$ from set $T^{+}$ is found with the help of (17). And after that, for point $A_1$, the returns to scale can be estimated at projection point B of the efficient frontier. At the efficient point of the frontier, the returns to scale can be evaluated using conventional methods [59,60].

4. Results

4.1. Data and Variables

As a numerical example, we consider the problem of evaluating the efficiency of government spending on decarbonization goals supporting research, development, and demonstration (RD&D) projects in different fields of energy efficiency. This problem was initially considered in the paper [61], where it was solved by the simplest method.

Statistical data show that public support for new energy technologies can reach a significant share of GDP in many countries. For example, according to the International Energy Agency (IEA) [62], government spending on energy efficiency and renewable energy technologies in Spain is around 0.5% of GDP, while in Finland it is almost 0.13% of GDP. However, it is not always the case that public spending achieves the stated objectives and actually leads to increased energy efficiency and reduced emissions. There is contradictory evidence in the scientific literature. Some studies provide results that prove that public spending on energy research, development and demonstration has played a significant role in reducing environmental pollution and GHG emissions [31,63], while others claim that changes in environmental efficiency due to public spending have been unsatisfactory in most countries [29,64,65].

Our sample includes 23 countries—members of IEA, namely Australia, Austria, Belgium, Brazil, Canada, Czech Republic, Denmark, Finland, France, Germany, Hungary, Ireland, Italy, Japan, South Korea, Netherlands, Norway, Poland, Spain, Sweden, Switzerland, the UK, and the USA. This study considers the volumes of government spending on RD&D as inputs. Patents are usually the main results of scientific research of an applied nature, therefore the number of patents in the field of “clean” energy and the number patents in the field of hydrocarbon energy received in each country are considered as intermediate outputs. The shifts in the energy and carbon intensity of the national economy over time are considered system outputs.

The data for government spending and patents was taken from IEA database [62]. Energy intensity and carbon intensity were obtained from the World Bank Databank [66,67].

Obviously, there is a time lag between investment in R&D and the production of scientific results. Similarly, there is a time lag between obtaining a scientific result and its implementation into practice. Therefore, in our study, we use the means for RD&D budgets (as share of GDP) for the period 2010–2012, the number of patents received in 2013–2015, and changes in carbon intensity and energy efficiency as the difference between the means in 2010–2012 and the means in 2016–2018. According World Bank methodology, energy efficiency is measured as GDP per unit of energy use (constant 2017 PPP $ per kg of oil equivalent), and carbon intensity is measured as CO${}_2$ metric tons for unit of GDP (calculated based on [67]). The number of patents is measured per 10,000 residents to take into account differences in the size of the economies and populations of the countries included in the sample.

Table 1. Descriptive statistics for inputs [68].

Statistics	Type of Technology
	Fossil Fuel	Renewable Energy and Energy Efficiency	Others
Mean	0.007	0.041	0.022
Std Dev	0.013	0.074	0.029
Variance	0.001	0.005	0.001
Min	0.000	0.008	0.000
Max	0.060	0.357	0.144
Kurtosis	11.675	17.499	15.52
Asymmetry	3.256	4.058	3.706

Note: The data represents public spending on energy-related RD&D projects, % GDP, mean for 2010–2012.

and

Table 2. Descriptive statistics for intermediate and system outputs [68].

Statistics	Patents on Fossil Fuel	Patents on Renewable Energy and Energy Efficiency	Change in Energy Efficiency	Change in Carbon Intensity
Mean	0.896	4.967	0.252	0.041
Std Dev	1.155	4.764	1.120	0.053
Variance	4.871	15.720	5.445	0.207
Min	0.048	0.069	−0.057	−0.072
Max	4.919	15.789	5.388	0.135
Kurtosis	6.366	0.000	22.949	−0.383
Asymmetry	2.425	1.059	4.788	−0.187

Note: The number of energy-related patents is measured per 10,000 residents for 2013–2016; the change in carbon intensity and energy intensity are measured as the differences between mean for 2010–2012 and mean for 2016–2018.

present descriptive statistics for inputs, intermediate outputs, and system outputs of the model.

Table 2’s data shows that outputs for several countries in the data sample are negative. This means that not all countries achieved the declared goals of increasing energy efficiency and reducing emissions in their economies during 2010–2018. From a mathematical point of view, working with negative outputs requires the use of special methods, which will be discussed below.

4.2. Results and Discussion

The efficiency scores of the first and second stages, together with the overall efficiency scores calculated by the CCR model, CCR with a shift in the scale of measurement of negative system outputs (CCR+), and the RDM model, which are all presented in

Table 3. Efficiency scores in CCR model, CCR model with a shift in outputs (CCR+), and RDM model.

Country	CCR Model			CCR+ Model			RDM Model
	Stage 1	Stage 2	Overall	Stage 1	Stage 2	Overall	Stage 1	Stage 2	Overall
Australia	21.7%	85.3%	53.5%	21.7%	48.8%	35.3%	100.0%	97.6%	98.8%
Austria	77.9%	7.3%	42.6%	77.9%	8.4%	43.2%	97.2%	50.7%	74.0%
Belgium	61.2%	20.3%	40.8%	61.2%	22.4%	41.8%	99.3%	62.5%	80.9%
Brazil	32.8%	100.0%	66.4%	32.8%	100.0%	66.4%	100.0%	100.0%	100.0%
Canada	41.4%	9.2%	25.3%	41.4%	5.1%	23.3%	79.9%	66.3%	73.1%
Czech Republic	16.0%	100.0%	58.0%	16.0%	61.0%	38.5%	100.0%	100.0%	100.0%
Denmark	69.8%	13.9%	41.9%	69.8%	7.4%	38.6%	94.1%	77.2%	85.6%
Finland	86.1%	8.5%	47.3%	86.1%	3.1%	44.6%	100.0%	65.7%	82.8%
France	42.1%	17.4%	29.8%	42.1%	10.5%	26.3%	75.1%	66.6%	70.8%
Germany	100.0%	0.8%	50.4%	100.0%	6.0%	53.0%	100.0%	43.0%	71.5%
Hungary	100.0%	1.4%	50.7%	100.0%	26.3%	63.2%	100.0%	79.0%	89.5%
Ireland	44.3%	0.7%	22.5%	44.3%	19.1%	31.7%	100.0%	55.0%	77.5%
Italy	17.7%	100.0%	58.8%	17.7%	69.5%	43.6%	91.9%	100.0%	96.0%
Japan	100.0%	19.5%	59.8%	100.0%	15.9%	58.0%	100.0%	59.6%	79.8%
Netherlands	75.7%	48.1%	61.9%	75.7%	17.8%	46.8%	97.4%	100.0%	98.7%
Norway	100.0%	3.9%	51.9%	100.0%	3.3%	51.7%	100.0%	56.1%	78.0%
Poland	7.9%	2.2%	5.1%	7.9%	21.3%	14.6%	87.9%	81.2%	84.5%
South Korea	100.0%	0.3%	50.1%	100.0%	0.5%	50.2%	100.0%	32.5%	66.3%
Spain	1.0%	100.0%	50.5%	1.0%	100.0%	50.5%	50.9%	100.0%	75.5%
Sweden	100.0%	9.6%	54.8%	100.0%	9.4%	54.7%	100.0%	53.4%	76.7%
Switzerland	49.7%	42.1%	45.9%	49.7%	25.4%	37.6%	86.2%	100.0%	93.1%
UK	76.5%	33.6%	55.0%	76.5%	15.5%	46.0%	100.0%	90.0%	95.0%
USA	100.0%	6.4%	53.2%	100.0%	3.3%	51.6%	100.0%	53.2%	76.6%

.

Returns to scale (decreasing, constant, or increasing) estimates in the first and second stages are presented in

Table 4. Estimated RTS in stages 1 and 2.

Country	Returns to Scale
	Stage 1	Stage 2
Australia	IRS	CRS **
Austria	DRS	DRS
Belgium	IRS	DRS
Brazil	IRS	CRS **
Canada	IRS	DRS *
Czech Republic	IRS	CRS
Denmark	DRS	DRS
Finland	DRS	DRS
France	DRS	DRS
Germany	CRS	DRS
Hungary	CRS	CRS **
Ireland	IRS	CRS *
Italy	IRS	CRS
Japan	CRS	DRS *
Netherlands	IRS	DRS
Norway	CRS	DRS *
Poland	IRS	CRS **
South Korea	CRS	CRS *
Spain	DRS	CRS
Sweden	CRS	DRS
Switzerland	DRS	DRS
UK	IRS	DRS
USA	CRS	DRS

*—RTS is calculated using Model (11), **—RTS is calculated using Model (17).

. In the first stage, all inputs and outputs for all DMUs are positive. Hence, the RTS is calculated at this stage with the help of traditional methods. For the second stage, we used the approach proposed in Section 3.3. First, the set $T^{+}$ is constructed for DMUs with positive inputs and outputs, and RTS is estimated for all units from this set. In our dataset, nine DMUs (Australia, Brazil, Canada, Hungary, Ireland, Japan, Norway, Poland, and South Korea) have negative outputs. Hence, for these units, Model (11) is solved in order to obtain an efficient projection point where RTS can be measured. For Canada, Ireland, Japan, Norway, and South Korea, such projections exist, and the estimated RTS is marked in Table 4 using the symbol ‘*’. For Australia, Brazil, Hungary, and Poland, Model (11) is infeasible. Therefore, for such units, it is necessary to determine the closest point from the set $T^{+}$. Next, the RTS is determined for this point; such DMUs are indicated in Table 4 by the symbol ‘**’.

As can be seen, the differences in the results between the models are quite significant. For obvious reasons, only the first stage of the CCR model and the first stage of the CCR+ model show complete agreement between the results of the calculations; the differences between these models only appear in the methods of calculating the efficiency score in the second stage.

Let us consider how the efficiency scores change at the second stage for DMUs negative outputs. From Table 3, one can see that, as a result of shifting the scale for output indicators, the efficiency scores of some DMUs increase unnecessarily. For example, the efficiency score of Hungary increased 18 times (from 1.4% to 26.3%). A similar situation is observed for two other DMUs: for Ireland, the efficiency score increased from 0.7% to 19.1%, and for Poland, from 2.2% to 21.3%. This is due to the fact that after the scale shift, the efficient frontier changes significantly and, as a result, the DMUs are estimated relative to other reference units.

To figure out the differences between the results of the models, let us look at two sets of parameters: (a) the sum of squares of deviations of efficiency scores on each stage ($S{S}_1$ and $S{S}_2$, respectively), as well as the system efficiency scores ($S{S}_{\mathrm{total}})$; and (b) Pearson’s linear correlation coefficients between the first and second stage efficiency scores ($r_1$ and $r_2$, respectively), as well as between the system efficiency scores ($r_{\mathrm{total}}$).

Let us use two sets of parameters to estimate the differences between the results of the models: (a) the sum of squares of deviations of efficiency scores on each stage ($S{S}_1$ and $S{S}_2$, respectively), as well as the system efficiency scores ($S{S}_{\mathrm{total}})$; and (b) Pearson’s linear correlation coefficients between the first and second stage efficiency scores ($r_1$ and $r_2$, respectively), as well as between the system efficiency scores ($r_{\mathrm{total}}$). The first set of estimates will allow us to measure the magnitude of the overall difference between the models, while the second set of estimates will allow us to measure the differences between the efficiency scores for each of the DMUs (countries). Calculated estimates are presented in

Table 5. Sums of squares of deviations and correlation of DMU efficiency scores results obtained by comparing different models.

Models	${\mathit{SS}}_1$	${\mathit{SS}}_2$	${\mathit{SS}}_{\mathbf{total}}$	${\mathit{r}}_1$	${\mathit{r}}_2$	${\mathit{r}}_{\mathbf{total}}$
CCR & CCR+	0	68.23%	17.00%	1	0.9039	0.8163
CCR & RDM	427.35%	524.73%	355.17%	0.5575	0.8064	0.4143
CCR+ & RDM	427.35%	609.56%	416.48%	0.5570	0.7162	0.0896

.

As can be seen, the CCR+ and RDM models differ most in terms of the efficiency scores obtained. The question of model selection is therefore of particular importance in order to ensure that the results of the calculations make sense in practice, correspond to the real picture, and adequately reflect the state of the national innovation systems in the field of energy-efficient technology development.

In our view, the measure of the discrepancy between the calculations obtained without subdividing into stages (in the situation where the DMU is modeled as a “black box” system) and the calculations of the system efficiency score according to the two-stage model can serve as a criterion for the adequacy of the model. Since the main point of considering the two-stage model is to find sources of inefficiency, the resulting efficiency scores themselves should be as close to each other as possible. We use this reasoning to figure out the efficiency scores for the CCR, CCR+, and RDM models without taking into account the intermediate outputs (number of patents), and compare them with the efficiency scores presented in Table 3. As in the previous case, the sums of squared deviations and correlation coefficients between the results obtained by models of the same type are used for comparison. The resulting estimates are shown in

Table 6. Sums of squares of deviations and correlations of DMU efficiency scores obtained by comparing black box and two-stage models.

Models	${\mathit{SS}}_{\mathbf{total}}$	${\mathit{r}}_{\mathbf{total}}$
CCR	281.86%	0.4946
CCR+	394.65%	0.2618
RDM	70.13%	0.2829

.

As we can see, according to the set of estimates of the difference between the results of the models, the RDM model is the most preferable. First, it gives minimal differences in the results of the efficiency assessment of the whole sample of DMUs. The RDM model is in second place in terms of the difference between the efficiency estimates for each country. In addition, only the RDM model allows for the determination of returns to scale for all DMUs in the first and second stages (Table 4), which is also important for making correct decisions regarding the improvement of the efficiency of national innovation systems in the development and deployment of energy-efficient technologies.

Interpreting the results of all three models from a practical point of view, we can also say that the efficiency scores calculated according to the RDM model are more meaningful and better reflect the processes in the innovation systems of the countries in the sample. The higher efficiency achieved in the first stage indicates the more productivity for corresponding countries in generating energy innovations and registering patents related to renewable energy technologies. The nations with the best efficiency scores at the second stage demonstrate an excellent utilization of energy technology patents in practical applications, hence facilitating the rapid implementation of energy innovations aimed at reducing the energy and carbon intensity of their economies. The low level of efficiency at the second stage of the innovation process can be attributed to the ongoing predominance of hydrocarbon energy within the economies of these nations. This is the case in Norway, the only country where the number of fossil fuel patents exceeds the number of patents for clean energy. Despite the considerable attention paid to clean energy innovation, these technologies are still not widespread in the economic system, which is not yet allowing the goals of decarbonizing and improving the energy efficiency of the economies of that nations to be achieved.

On the other hand, the weak effect of a high number of patents on energy intensity and carbon reduction in some countries can be linked to the accumulation of information in emerging domains of technical progress that have yet to attain the level of industrial implementation. According to the literature, this is the case in Japan and South Korea [69,70].

At the same time, for countries with low levels of patenting activity, the valuable results can be achieved by introduction of organizational innovations in environmental management (management strategies, environmental management systems, standards, etc.), changes in consumer behavior, and the development of less energy-intensive sub-sectors of the economy. In the context of Brazil, notable achievements include the effective establishment of a national framework of energy efficiency requirements specifically residential buildings [71], along with advancements in load energy management and the wide acceptance of these initiatives within society [72].

Regarding returns on scale, all DMUs show constant (39.1%) or decreasing returns to scale (60.9%) in the second stage. Decreasing returns to scale may indicate excessive patenting of energy technologies, i.e., patenting not for the purpose of direct technology deployment, but for other purposes (e.g., using blocking patenting to circumvent competitors in the market or reputational patenting to build a positive image). The situation is more favorable in the first stage, where many DMUs have increasing (43.5%) or constant (30%) returns to scale. Taken together, these results suggest that the main source of inefficiency in the government support system for innovative energy-efficient technologies lies in the practical implementation of new developments.

5. Conclusions

The main purpose of this study is to provide a comparative analysis of several possible approaches to applying the DEA methodology in the case where some DMUs in the original sample have negative second-stage outputs. In comparison with the traditional CCR model and the CCR model with a scale shift to measure second-stage outputs, the RDM model produces the most logical results. If we take as a measure of model adequacy the similarity between the calculations obtained without stage separation (i.e., in a situation where the DMU is modeled as a “black box” system) and the system efficiency calculations from the two-stage model, the RDM model shows the highest similarity scores. In addition, RDM is the only possible model that allows the calculation of economies of scale at each stage of a DMU with a complex structure.

From a theoretical point of view, these results help to reveal the advantages and disadvantages of different DEA models in the specific case of the presence of negative outputs in the last stage of the DMU with a complex structure. From a practical point of view, the results of this study can help policy-makers in the design and implementation of national energy efficiency and decarbonization strategies of national economies.

Nevertheless, it is worth noting that the models developed based on existing data are still simplified, and can be improved in several directions. It should be noted that some of demonstration projects can be placed in the implementation phase, i.e., it is more reasonable to consider their budgets as supplementary inputs at the second phase. Although, the share of budgets spent on demonstration projects is not available in the existed dataset, it can be taken into account by introducing coefficient $\alpha \in [0,1]$, which represents the share of budgets spent in total spending. Coefficient $\alpha$ can be specified by an expert or can be determined by solving an additional optimization problem. This case leads us to the problem of shared inputs. The elimination of this main limitation of the model is the subject of further research by the authors. Another interesting direction of the further research is the extension of existed approaches dealing with negative data to other efficiency measures [73,74].