Network DEA and Its Applications (2017–2022): A Systematic Literature Review

Abstract

Data Envelopment Analysis (DEA) is one of the fastest growing approaches to solving management problems for the multi-criteria evaluation of the efficiency of homogeneous production systems. The general trend in recent years has been the development of network DEA (NDEA) models, which can consider the complicated structure of Decision Making Units (DMUs) and, therefore, can be more informative from the point of view of management science than traditional DEA models. The aim of this study is the systematization and clarification of general trends in the development of NDEA applications over the past 6 years (2017–2022). This study uses the methodology of a systematic literature review, which includes the analysis of the dynamics of the development of the topic, the selection of the main clusters of publications according to formal (citation, branches of knowledge, individual researchers) and informal (topics) criteria, and the analysis of their content. This review reveals that, most frequently, network structures are used for bank models, supply chain models, models of eco-efficiency of complex production systems, models of innovation processes, and models of universities or their departments and healthcare systems. Two-stage models, where the outputs of the first stage are the inputs of the second (intermediate outputs), are the most commonly used. However, in recent years, there has been a noticeable tendency to complicate DEA models and introduce hierarchical structures into them.

1. Introduction

Nowadays, Data Envelopment Analysis (DEA) is one of the fastest growing approaches to solving management problems for the multi-criteria evaluation of the efficiency of homogeneous economic agents. Since the first studies, dating back to 1978 [1], the methodology has undergone significant improvements, expanding the range of its practical applications [2].

DEA is a nonparametric method that is used to measure the relative efficiency of a homogeneous set of Decision Making Units (DMUs). Let $X_j={(x_{1j},\dots ,x_{mj})}^{\mathsf{T}}\ge 0$ be the input vector and $Y_j={(y_{1j},\dots ,y_{rj})}^{\mathsf{T}}\ge 0$ be the output vector of j-th DMU ($j=1,\dots ,n$). At least one input and one output are assumed to be non-zero. Based on a set of axioms and using the observed set of DMUs, a production possibility set (PPS) can be constructed. For DEA models with variable return to scale (VRS), the production possibility set is written in the following form:

$$T_{\mathrm{VRS}}=\left\{(X,Y)|\sum _{j=1}^n{X}_j{\lambda }_j\le X,\sum _{j=1}^n{Y}_j{\lambda }_j\ge Y,\sum _{j=1}^n{\lambda }_j=1,{\lambda }_j\ge 0,j=1,\dots ,n\right\}.$$

(1)

PPSs displaying the constant, non-increasing, and non-decreasing returns to scale (CRS, NIRS, and NDRS) are generated by the $T_{\mathrm{VRS}}$. They are defined as follows:

$$\begin{array}{cc}\hfill T_{\mathrm{CRS}}& =\left\{(X,Y)|(X,Y)=\delta (X^{\prime },Y^{\prime }),(X^{\prime },Y^{\prime })\in T_{\mathrm{VRS}},\delta \ge 0\right\},\hfill \\ \hfill T_{\mathrm{NIRS}}& =\left\{(X,Y)|(X,Y)=\delta (X^{\prime },Y^{\prime }),(X^{\prime },Y^{\prime })\in T_{\mathrm{VRS}},0\le \delta \le 1\right\},\hfill \\ \hfill T_{\mathrm{NDRS}}& =\left\{(X,Y)|(X,Y)=\delta (X^{\prime },Y^{\prime }),(X^{\prime },Y^{\prime })\in T_{\mathrm{VRS}},\delta \ge 1\right\}.\hfill \end{array}$$

(2)

For measuring the relative efficiency of the selected DMUs, there are several approaches. The most known are the radial measures. For example, the traditional input measure of efficiency for VRS technology yields the following model written in the envelopment form:

$$\begin{array}{cc}\hfill min& \theta -\epsilon \left(\sum _{i=1}^m{s}_i^{-}+\sum _{k=1}^r{s}_k^{+}\right)\hfill \\ \hfill \mathrm{subject} \mathrm{to}& \sum _{j=1}^n{X}_j{\lambda }_j+S^{-}=\theta X_0,\hfill \\ \hfill & \sum _{j=1}^n{Y}_j{\lambda }_j-S^{+}=Y_0,\hfill \\ \hfill & \sum _{j=1}^n{\lambda }_j=1,\hfill \\ \hfill & S^{-}={(s_1^{-},\dots ,s_m^{-})}^{\mathsf{T}}\ge 0,\hfill \\ \hfill & S^{+}={(s_1^{+},\dots ,s_r^{+})}^{\mathsf{T}}\ge 0,\hfill \\ \hfill & {\lambda }_j\ge 0,j=1,\dots ,n,\hfill \end{array}$$

(3)

where $S^{-}$ and $S^{+}$ are the input and output slack vectors, respectively; the $\epsilon$ is a non-Archimedean infinitesimal that is used to constrain input and output weights from being zero. The optimal objective value of Model (3) is the efficiency score for unit $(X_0,Y_0)$ in input-oriented VRS model. From Model (3), it follows that ${\theta }^{*}\le 1$ and, consequently, the efficiency score of a DMU${}_0$ is from 0 to 1.

The non-radial SBM models [3] eliminate the assumption of proportionate changes in inputs and outputs and deal with slacks directly. The following model evaluates SBM score ${\rho }^{*}$ under the VRS assumption:

$$\begin{array}{cc}\hfill min\rho =& \frac{1-(1/m){\sum }_{i=1}^m{s}_i^{-}/x_{i0}}{1+(1/r){\sum }_{k=1}^r{s}_k^{+}/y_{k0}}\hfill \\ \hfill \mathrm{subject} \mathrm{to}& \sum _{j=1}^n{X}_j{\lambda }_j+S^{-}=\theta X_0,\hfill \\ \hfill & \sum _{j=1}^n{Y}_j{\lambda }_j-S^{+}=Y_0,\hfill \\ \hfill & \sum _{j=1}^n{\lambda }_j=1,\hfill \\ \hfill & S^{-}={(s_1^{-},\dots ,s_m^{-})}^{\mathsf{T}}\ge 0,\hfill \\ \hfill & S^{+}={(s_1^{+},\dots ,s_r^{+})}^{\mathsf{T}}\ge 0,\hfill \\ \hfill & {\lambda }_j\ge 0,j=1,\dots ,n,\hfill \end{array}$$

(4)

From Model (4), it holds that $0\le {\rho }^{*}\le 1$.

The general trend in recent years has been the development of network DEA models, which, from the point of view of management science, are more informative than traditional DEA models. Traditional DEA models consider DMUs as “black-box” systems, where only inputs and outputs are known. The transformation of inputs into outputs is unknown, and for this reason, intermediate indicators are lost. It leads to the impossibility of distinguishing and identifying which part of the DMU is responsible for its overall inefficiency. If the researcher a priori knows any additional information about the structure of DMU or the process of its operation, then the natural way of improving the traditional model will be to divide the DMU into two or more sub-processes that are interconnected by the so-called intermediate products $Z_p$. Intermediate products are the outputs of one sub-process and, simultaneously, are inputs to another sub-process. With the introduction of such a complicated structure of DMU into the model, the results become more informative and open new possibilities for better strategy planning and management of the modeled economic agents.

The first studies in this area appeared as early as 1994 and 1997 [4,5], but they used the terminology “modified” models. The terms “multi-stage” and “network” models began to be traced in the metadata of articles since 1998, and the growth in interest in these types of models began in 2009 and became especially noticeable starting in 2016 (see Figure 1). This trend demonstrates that network DEA (NDEA) has become a prominent research area of interest with a wide spectrum of practical applications.

In the network DEA models, the sub-processes of DMU can be arranged in series, parallel, or mixed structures. In the series network model, each sub-process produces some intermediate outputs, which are then consumed by the consecutive sub-process. The simplest series structure is the two-stage one. Consider the case where all inputs $X_j$ are consumed by the first sub-process to produce intermediate products $Z_j={(z_{1j},\dots ,z_{pj})}^{\mathsf{T}}\ge 0$. All intermediate products are supplied to the second sub-process to produce the final outputs $Y_j$. This means that no intermediate products leave the system. The technology of each sub-process is assumed to be selected from (1) and (2). Then, the corresponding PPS for the VRS case is written in the form:

$$\begin{array}{c}{T}_{\mathrm{VRS}}^S=\left\{(X,Z,Y)\right|\sum _{j=1}^n{X}_j{\lambda }_j\le X,\sum _{j=1}^n{Z}_j{\lambda }_j\ge Z,\sum _{j=1}^n{\lambda }_j=1,{\lambda }_j\ge 0,j=1,\dots ,n\hfill \\ \hfill \sum _{j=1}^n{Z}_j{\mu }_j\le Z,\sum _{j=1}^n{Y}_j{\mu }_j\ge Y,\sum _{j=1}^n{\mu }_j=1,{\mu }_j\ge 0,j=1,\dots ,n\}.\end{array}$$

(5)

This concept is easily generalized to multi-stage processes.

In the parallel network structure, sub-processes are not interconnected through the intermediate outputs. Each sub-process consumes some portion of the inputs and produces a certain amount of output. In other words, inputs are shared between sub-processes to produce the final outputs of the DMU.

The hierarchical organization of sub-processes at different levels is proposed in [6]. It was shown that the overall structure can be transformed into a parallel structure composed of those units at the bottom of the original hierarchical structure. The efficiency score of the DMUs with hierarchical structures can be measured as a weighted average of the sub-units at the bottom level.

Dynamic network DEA models can assess the efficiency score using “carry-over” variables [7]. This model evaluates not only the efficiency of the network system over the one selected period but also dynamic changes in efficiency.

Several attempts have been made to systematize and categorize the growing number of publications in the field of NDEA in recent years. For example, categorization of the papers on the topic of network DEA, based on the network structure, was introduced by Kao [8]. Reviews of network DEA models were carried out by Chen et al. [9] and Koronakos [10].

Nevertheless, a significant recent development of the methodology and its practical applications has not been adequately covered in previous reviews. Therefore, the aim of this study is the systematization and clarification of general trends in the development of NDEA applications over the past 6 years (2017–2022). This study applies a systematic literature review (SLR) methodology, which includes the selection of papers by following strict criteria and bibliometric and content analysis.

The rest of this paper is organized as follows: Section 2 presents the method and sources of the systematic literature review. Section 3 presents the results and discussions. The final section, Section 4, concludes the paper with a summary of the key findings, study limitations, and directions for future work.

2. Materials and Methods

Traditionally, a systematic literature review consists of three main stages: (1) search and screening of literary sources; (2) descriptive analysis of a sample of publications; (3) qualitative content analysis of selected literary sources [11,12,13,14,15,16]. To ensure a representative sample of publications and guarantee that it includes the maximum number of papers on the topic of interest, usually several databases of literature sources (Scopus, Web of Science, Science Direct, etc.) are used, which are searched for several variable sets of keywords. Next, a screening of the selected literary sources is carried out to check the compliance of publications with the specified time period and subject area. The duplicate records, which often occur due to differences in the spelling of bibliographic references, are eliminated.

A descriptive analysis of a sample of publications involves studying the dynamics of the number of publications, the distribution of publications by fields of knowledge, the authors, the geography of authors, citations, etc. [17,18]. Descriptive analysis aims to point out the main trends in the development of the chosen scientific direction, to determine the papers and scientists who have had the greatest influence on this development, and to identify the most vital problems. This stage narrows the set of papers for subsequent qualitative content analysis.

In our study, the search for scientific documents for descriptive analysis was carried out using the Google Scholar database as a repository with the largest coverage of bibliographic data. Taking into account the fact that DEA network models can also be called multi-stage in the literature, and the abbreviation DEA itself is often used instead of the full name of the methodology, and search queries were formulated as follows: “network Data Envelopment Analysis” OR “network DEA” OR “two-stage DEA” OR “two-stage Data Envelopment Analysis” OR “three-stage Data Envelopment Analysis” OR “three-stage DEA” OR “multi-stage DEA” OR “multi-stage Data Envelopment Analysis”. This search was made in the titles of the documents.

From the bibliography obtained with the help of Google Scholar, we selected the literature based on the following principles:

• We selected only articles published in peer-reviewed journals.
• Articles were written in English.
• We did not take into account references that could not be found or accessed (broken links or journal’s website no longer being online).
• We did not consider retracted articles and article updates: correction, erratum, etc.
• We did not select articles that are currently in press.
• We chose only articles devoted to multi-stage and network DEA models.

It is necessary to note the importance of the last point because the “two-stage” (or “three-stage”) keyword in the query results in articles that may not be devoted to NDEA but represent a two-stage analysis where regression methods are applied at the second stage.

The workflow of this systematic literature review is presented in Figure 2.

3. Results

3.1. Statistics on NDEA Publications

In total, there were 598 journal articles selected for the period of 2017–2022. The distribution of the selected articles by year is represented in Figure 3. This figure illustrates the continuous and rapid growth in the field of network DEA models in recent years. Over the period from 2017 to 2022, the number of articles increased more than sixfold. The average annual growth is 47%.

Figure 4 represents the share of the selected articles for literature review in the total number of publications in Google Scholar output. Since we did not include, in our study, two-stage or three-stage analysis where regression models were used, this graph shows that the proportion of articles in the sample that used network DEA terminology in the correct context is increasing. This indicates the maturity of terminology and this scientific direction as a whole.

From the metadata obtained via Crossref API, we found that about 30% of papers received financial support for their research. This value is quite large and shows that network DEA models and the development of this area, in general, are generating interest not only from researchers but also from funding organizations.

Table 1. Journals that have published the greatest number of NDEA articles (2017–2022).

No.	Journal	Articles	Percent
1	Sustainability	29	4.85%
2	Annals of Operations Research	19	3.18%
3	Journal of Cleaner Production	19	3.18%
4	Omega	18	3.01%
5	Socio-Economic Planning Sciences	18	3.01%
6	Computers and Industrial Engineering	17	2.84%
7	European Journal of Operational Research	17	2.84%
8	Expert Systems with Applications	16	2.68%
9	Journal of the Operational Research Society	13	2.17%
10	International Journal of Environmental Research and Public Health	11	1.84%
11	RAIRO—Operations Research	10	1.67%
12	Energies	9	1.51%
13	International Journal of Data Envelopment Analysis	9	1.51%
14	Energy	8	1.34%
15	International Journal of Industrial Mathematics	7	1.17%
16	Operational Research	6	1.00%
17	International Transactions in Operational Research	6	1.00%
18	Mathematical Problems in Engineering	6	1.00%
19	Central European Journal of Operations Research	5	0.84%
20	Environmental Science and Pollution Research	5	0.84%
21	PLoS ONE	5	0.84%
22	Scientia Iranica	5	0.84%
23	Journal of Intelligent and Fuzzy Systems	5	0.84%
24	Advances in Mathematical Finance and Applications	5	0.84%
25	Journal of mathematical extension	5	0.84%
Total		273	45.65%

shows the top 25 journals that published the greatest number of articles on NDEA in the period from 2017 to 2022. It is reasonable that in the top positions are the leading journals in the field of management science and operations research, such as Annals of Operations Research, Omega, Socio-Economic Planning Sciences, European Journal of Operational Research, etc. However, it is rather surprising that the Sustainability and Journal of Cleaner Production was also featured at the top of the table because they are primarily aimed at environmental and sustainability assessment. The obtained result confirms that practical applications of network DEA models contribute significantly to this area. Not without reason, eco-efficiency is one of the most popular application fields of NDEA, see discussion below.

Table 2 shows which publishers issued journals with the most articles on the subject. The most utilized publisher is Elsevier BV with 176 articles, while the top three publishers (Elsevier BV, Springer Science and Business Media LLC, and MDPI AG) cover more than half of the total number of NDEA papers published from 2017 to 2022.

Figure 5 reveals the descriptive statistics involving the number of authors in the selected papers. According to our dataset, only 10% of all selected articles were written by a single author, while about 20% were published by two authors. The vast majority of articles (about 95%) have five or fewer authors. Figure 5 shows the distribution of the selected articles by the number of authors.

The next step of our review was co-keywords analysis. The technique is not, in fact, scientometric, but is widely used as a tool for rapid visualization and mapping of the subject area. The keywords/terms for analysis were selected based on a final sample of 598 journal articles as identified by the authors themselves and extracted from titles and abstracts using text mining algorithms. Terms extracted from article titles and abstracts are ranked by how often they occur together. The size of the bubbles reflects the number of publications. By default, the thematic clusters defined automatically are given in color.

Table 2. Publishers that have published the greatest number of NDEA articles (2017–2022).

No.	Publisher	Articles	Percent
1	Elsevier BV	176	29.43%
2	Springer Science and Business Media LLC	81	13.55%
3	MDPI AG	72	12.04%
4	Informa UK Limited	45	7.53%
5	AZADTABRIZ—Islamic Azad University	24	4.01%
6	Emerald	22	3.68%
7	Hindawi Limited	19	3.18%
8	Wiley	17	2.84%
9	Inderscience Publishers	12	2.01%
10	EDP Sciences	10	1.67%
11	IOS Press	6	1.00%
12	Sinaweb as Publisher	6	1.00%
13	Public Library of Science (PLoS)	5	0.84%
14	American Institute of Mathematical Sciences (AIMS)	5	0.84%
Total		500	83.61%

Table 2. Publishers that have published the greatest number of NDEA articles (2017–2022).

No.	Publisher	Articles	Percent
1	Elsevier BV	176	29.43%
2	Springer Science and Business Media LLC	81	13.55%
3	MDPI AG	72	12.04%
4	Informa UK Limited	45	7.53%
5	AZADTABRIZ—Islamic Azad University	24	4.01%
6	Emerald	22	3.68%
7	Hindawi Limited	19	3.18%
8	Wiley	17	2.84%
9	Inderscience Publishers	12	2.01%
10	EDP Sciences	10	1.67%
11	IOS Press	6	1.00%
12	Sinaweb as Publisher	6	1.00%
13	Public Library of Science (PLoS)	5	0.84%
14	American Institute of Mathematical Sciences (AIMS)	5	0.84%
Total		500	83.61%

As a result of metadata processing using the VOSviewer software product, we obtained the following distribution of topics and keywords in 10 clusters (https://app.vosviewer.com/?json=https://drive.google.com/uc?id=1ehC6PyclAuKQ-cZ2NV2YudM0vlAKzEFU) (accessed on 8 April 2023), with a total of 1141 items and 45,180 links (see Figure 6):

• NDEA methodology (red), 265 publications;
• Eco-efficiency (green), 215 publications;
• Reviews (blue), 183 publications;
• Selection criteria (yellow), 171 publications;
• Banking industry (purple), 83 publications;
• Science (turquoise), 81 publications;
• Life cycle assessment (orange), 60 publications;
• R&D (beige), 41 publications;
• Technological production (lilac), 25 publications;
• Ecology and nature (gray), 17 publications;

Figure 6. Analysis of terms/topics/keywords by co-occurrence (2017–2022). Source: author’s own elaboration.

Figure 6. Analysis of terms/topics/keywords by co-occurrence (2017–2022). Source: author’s own elaboration.

Figure 6 presents the graph of co-occurrence relationships among the 1141 keywords; the keywords in the different clusters are displayed in different colors. If keywords are grouped into the same cluster, they are more likely to reflect identical topics. Each cluster has a different number of subject keywords. The thickness of line is proportional to the closeness of connections between two keywords; the thicker the line between two words is, the closer the relationship is.

Next, we analyzed using the co-citation method, and the results are shown in Figure 7 (https://app.vosviewer.com/?json=https://drive.google.com/uc?id=1D5e5rArsqFSxczhZOwFP7B3Up49yfG7P) (accessed on 8 April 2023). This technique makes it possible to assess the semantic proximity of two publications (authors, countries, journals, etc.). Co-citations define it as the frequency with which two entities are cited together by other works. The more often, the more likely the articles are to be closer in meaning.

When forming a cluster with the limitation of only title field binary accounting, we narrowed the field for building a cluster, deliberately excluding from the selection keywords that create noise associated with NDEA itself. We built the second group of clusters based on a sample of authors using the following method. For each of the 37 authors, the total strength of co-authorship links with other authors will be calculated. The authors with the highest total link strength will be selected.

The graph shows 8 clusters of 30 items with a total of 65 links and 49 links. The citation core is Emrouznejad, Ali with a total of six articles [19,20,21,22,23,24]: two in 2017 and four in 2021. It can also be seen that the links are united into one cluster with seven items, two clusters of five items each, one cluster that unites four items, and three clusters, each including three links between three authors. Furthermore, the author who has connections with four other clusters is Zhu, Joe [25,26,27,28,29], an author of seven articles: one in 2017, two in 2018, and four in 2021.

3.2. Practical Applications of NDEA

By analyzing the context of publications, firstly, we can single out a separate and rather large category of articles that are devoted to the theoretical problems of the NDEA methodology and do not address any specific practical case. The share of such articles in the total number of publications ranges from 21% in 2018 to 32% in 2019 (see Figure 8).

Note that the results presented in Figure 8 coincide with the results of the graph of co-occurrence relationships (Figure 6) and mark theoretical papers as the most representative group of papers in the sample.

The rest of the studies develop the NDEA methodology for specific practical applications, the spectrum of which is extremely wide. The largest number of publications is devoted to the functioning of banks, energy systems, supply chains, transport and logistics systems, educational systems, healthcare systems, hospitals, airports, and airlines, as well as the problems of eco-efficiency of various economic agents and the modeling of innovative processes (see Figure 9). This also correlates well with the results of co-occurrence analysis (Figure 6).

Most papers developing NDEA models for applications in banksrepresent the bank as a two-stage DMU, with the stage (or subsystem) of attracting capital and the stage of investing capital. As an intermediate output of the first stage, the studies in [30,31,32,33,34,35,36,37] consider customer deposits. The paper in [38] proposes a profit-oriented approach and divides overall bank efficiency into operational efficiency and profitability efficiency. Operational expenses, loanable funds, and capital stock are inputs of the first stage. Performing loans, investments, nonperforming loans, and service revenues are outputs of the first stage. Intermediate outputs here are only performing loans and investments because nonperforming loans and service revenues, measured by total revenues from net commissions and service provisions, cannot generate more profit. A similar but simplified approach is used by Anand and Kumar [39], taking as inputs of the first stage—fixed assets ($x_1$), number of staff ($x_2$), and borrowed funds ($x_3$) as a sum of borrowings and deposits; as intermediate outputs— investments ($z_1$) and advances ($z_2$); as outputs of the second stage—income by interest ($y_1$) and income by non-interest ($y_2$).

The paper in [40] introduces a more complicated structure of the bank with a series of sequential processes. Division 1 uses fixed assets ($X_1$) and employees ($X_2$) to generate deposits ($Z_1$), loans ($Z_2$), obligations ($Z_3$), and operating costs ($Z_4$). Division 2 then converts $Z_1$ into non-operating costs ($Z_5$), division 3 converts $Z_2$ into income ($Z_6$), and division 4 converts $Z_3$ into fees ($Z_7$). Finally, division 5 uses $Z_4$, $Z_5$, $Z_6$, and $Z_7$ to generate profit (Y).

Several studies in the field of NDEA for bank applications introduce dynamic NDEA models. For example, the authors of [41] model the structure of a bank for two consecutive periods. The first stage presents deposit collection with two inputs: the number of human forces and fixed assets. The second stage presents loaning. The inputs of this stage (intermediate outputs) are deposits and delayed claims. The amount of the loan was defined as the output of the second stage. A similar structure of the bank model can be found in [42].

The study in [35] additionally highlights the stage of making a profit (three-stage model), and takes into account the fact that unused assets can be transferred to the next periods of the bank’s operation (see Figure 10). Therefore, a dynamic model is also considered.

As the inputs of the first stage, the fixed assets, $X^{\mathrm{Fix}}$, and the salaries of employees, $X^{\mathrm{Exp}}$, are considered. The outputs of the first stage and, accordingly, the inputs of the second stage are deposits $Z^{\mathrm{Dep}}$ and capital $Z^{\mathrm{Due}}$, borrowed from other banks and financial institutions. An additional input of the second stage is the interest paid on deposits, $X^{\mathrm{Add}}$. The output of the second stage and the input of the third stage are loans provided to individuals and legal entities, $Z^{\mathrm{Tot}}$. A desirable output of the third stage is the interest received from loans, $Y^{\mathrm{Good}}$. Non-performing loans, $Y^{\mathrm{Bad}}$, are considered undesirable outputs reflecting credit risks. Unused assets are indicated on the chart as $C_j$, where j is an index of DMU under investigation. A similar model can be found in [43].

The paper in [44] proposes an approach that includes social dimension in the NDEA model of the bank by adding a third stage. In this three-stage model, the inputs of the initial sub-process are employees ($X_1$), fixed assets ($X_2$), and non-operating costs ($X_3$), which include all bank costs other than deposit interest expense. Bank deposits ($Z_1$) are considered as the output of the first sub-process and input of the second sub-process. The deposit interest expenses ($X_4$) are also considered as the input of the second sub-process. The number of transactions ($Z_2$) and the number of accounts ($Z_3$) are considered as other outputs of the first stage (free outputs). The outputs of the second sub-process are as follows: bank facilities ($W_1$), interest income ($W_2$), and non-interest income ($W_3$). Bank facilities ($W_1$) are also considered as intermediate output of the second stage. Finally, the employment rate ($Y_1$) is identified as the output of the third sub-process (social welfare).

One of the most complicated structures of the bank model can be found in [45]. It combines a multi-stage structure with a parallel (serial) structure (see Figure 11). The main idea of the proposed structure is dividing a network into some sub-networks and classifying inputs into several categories (such as financial inputs and environmental inputs).

Another significant group of papers models the operation of supply chains [46,47,48,49,50,51,52,53,54,55]. For example, Chodakowska and Nazarko [47] study the problem of optimizing the operation of the chain “online store—delivery”. The inputs at both stages (delivery company and sales company) are the wages and welfare costs of employees, as well as the costs of maintaining the operation of the enterprise. The outputs of each stage are the profits of the respective company. The intermediate output is the total operating income from delivery.

A more complicated structure of supply chain is introduced in [56]. Besides three serially connected sub-processes including production, transmission, and distribution of natural gas stages, eight parallel components in the production stage as refinery companies are also considered. Complicated network interconnections among refineries and transmission zones are considered to model the real case of the natural gas supply chain network.

Some papers, such as [49,52,54], also consider undesirable outputs in the supply chain, which are emissions from one of the stages (most frequently in manufacturing). In this case, the authors change the focus of the study from the issue of economic efficiency to the issue of sustainability.

It should be noted that, in recent years, increasing attention has been paid to the problems of eco-efficiency of supply chains in the literature. Therefore, it is rather difficult to clearly separate these two large groups of studies (focusing on supply chain eco-efficiency issues or focusing on eco-efficiency of other multi-stage processes) using the network DEA for modeling. For example, the paper in [57], by its title, stands for a problem of measuring supply chain efficiency, but in reality, it models the energy supply of cities in mid-eastern Chinese provinces from the point of view of carbon trading system efficiency.

The articles in [58,59,60,61,62,63,64,65,66,67,68] are devoted to modeling production processes from the standpoint of eco-efficiency. They divide all the production activities of economic agents into a production stage and a stage responsible for environmental impacts (cleaning emissions, creating eco-friendly innovations, etc.). For instance, the authors of [58] model the activity of an industrial enterprise as a two-stage process: the production and treatment of wastewater and gases emitted into the atmosphere (see Figure 12).

The inputs of the first stage are labor, capital, and energy, and the outputs are industrial added value, CO₂ emissions, solid waste, polluted wastewater, and emissions in the air. The volumes of pollutants in wastewater and atmospheric gases are the inputs of the second stage. Additional inputs of the second stage are equipment for wastewater and gas treatment and wastewater and gas treatment costs.

A similar approach is used in [66] for modeling provincial eco-efficiency in China. The inputs of the first stage (production process) include labor, capital stock, energy, land, and water. The free output is GDP, while the intermediate outputs are waste gas, wastewater, and SO₂. The second stage is a treatment process, which has government funding for environmental protection as a free input (in addition to intermediate outputs). The outputs of the second stage include solid waste utilization, wastewater treatment, and greening rate, where greening rate refers to the ratio of the greenbelt area within the scope of construction land to the construction land area.

Ren et al. extend the approach suggested above and include social dimension in the model of regional eco-efficiency [67]. They argue that nature provides the environmental capacity for human development and, therefore, introduce a third stage, corresponding to the social subsystem. The inputs at the first stage (economic system) include natural, human, and social capital. The outputs of the economic system are goods and services, measured by GDP. The undesired intermediate outputs of this stage are wastewater, waste, and solid waste. The second stage is the environmental treatment stage, where investment in environmental pollution treatment is an input and urban sewage treatment rate, air quality in major cities, and utilization rate of solid waste are outputs. For the social stage, inputs are the proportions of R&D input and social public expenditure. The Human Development Index is used as the system output indicator.

The same idea of three-dimensional eco-efficiency is used in [68]. However, the authors represent the economy–society–environment circular system using a matrix-type NDEA with three linked subsystems. Each subsystem has an external input and external output and links to another subsystem with internal input and output. The peculiarity of this model is that intermediate outputs do not really exist because every activity should belong to either input or output, not both. For the environmental subsystem, the authors take environmental protection investments as external input. Excellent and good rates of atmospheric quality, industrial wastewater treatment, average PM10 concentrations, and the utilization amount of industrial solid waste are taken as external outputs. As internal inputs, they use industrial/residential wastewater discharged, industrial/residential sulfur dioxide emissions, industrial/residential soot and dust emissions, and carbon dioxide emissions. The internal outputs are energy consumption, forest coverage rate, and cultivated area. For the economic subsystem, the external inputs are total fixed asset investment and technology investment; the external output is GDP. The internal inputs are the number of employees at the end of the year and energy consumption; the internal outputs are per capita disposable income, industrial wastewater discharged, industrial sulfur dioxide emissions, industrial soot and dust emissions, and carbon dioxide emissions. The social subsystem takes social security expenditure as an external input and produces per capita educational year and proportion of health technical personnel as external outputs. As internal inputs, it takes per capita disposable income, cultivated area, and forest coverage rate. The internal outputs of social subsystem are the number of employees at the end of the year, residential wastewater discharged, residential sulfur dioxide emissions, and residential soot and dust emissions.

The study in [62] evaluates the efficiency of power generation facilities, where the first stage takes into consideration the financial mission of the plants, and the second stage considers their sustainable mission. The inputs of the first stage are annual fuel consumption, book value of plant and land, plant capacity, and number of employers. The intermediate outputs are annual electricity net generation and availability factor. The system outputs that represent social dimension are annual incidence of deaths, annual incidence of heart attacks, annual incidence of asthma attacks, annual incidence of hospital admissions, annual incidence of chronic bronchitis, annual incidence of asthma ER visits. The environmental dimension is represented by such system outputs as carbon dioxide (CO${}_2$) emission, nitrous oxide (N${}_2$O) emission, methane (CH${}_4$) emission, sulfur dioxide (SO${}_2$) emission, and nitrogen oxides (NO${}_x$) emission. The economic dimension is represented as annual revenue.

The paper in [60] considers countries in terms of generation and implementation of eco-innovations. A two-stage DEA model is used at the first stage, then the eco-efficiency of the economy is assessed. At the second stage, the efficiency of the implementation of eco-innovations is measured. The inputs of the first stage are labor, land, and energy. The intermediate outputs are GDP and greenhouse gas emissions. Note that one intermediate output is desirable, and the other is undesirable. The outputs of the second stage are the number of people employed in R&D, the export of high-tech products, the number of ISO 14001 certificates for environmental management systems, and the amount of electricity produced from renewable sources.

The study in [63] models the formation and treatment of air pollution in the city and takes SO${}_2$ spatial transmission as the intermediate output or input indices between decision-making units.

The papers in [69,70,71,72,73,74,75,76,77,78] introduce multi-stage models for the innovation process, which is generally divided into the research and development (R&D) stage and the implementation stage. For example, in study [72], the innovation process at high-tech enterprises is modeled using a two-stage model. The inputs of the first stage (R&D) are the costs of research equipment and the number of researchers. The second input is divided between the first and second stages in a certain endogenously determined proportion. The outputs of the first stage, and at the same time, the inputs of the second stage are patent applications and patents in force. Additional inputs of the second stage (commercialization) are the costs of technical modernization, the costs of acquiring national and foreign technologies, the costs of adapting technologies, and the costs of developing new products. The outputs of the second stage are the revenue from the sales of new products and the number of technical contracts concluded in the domestic market. Wei et al. [75] and Zhang and Cui [78] use a similar network model with shared inputs for innovation-oriented and resource-based cities.

A more recent study [79] of innovations in Chinese companies combines multiple network processes across periods and proposes a dynamic network DEA model (see Figure 13).

R&D expense (RE) and R&D labor (RL) here represent innovative inputs. New patent applications (NPs), government grants (GGs), and commercialization expenses (CEs) are inputs of the commercialization stage. The total revenue gained from the commercialization process (RV) and new product revenue (NR) are system outputs. The feature of the model is the introduction of two carry-over variables between dynamic processes. They are patent stock (PS) and capital stock (KS).

The papers in [73,76,77,80] consider regions as DMUs and model their innovation activity. The paper in [81] models innovation activities in BRICS countries. In [73], the inputs of the R&D stage are the number of employers involved in R&D and the internal costs of R&D. The intermediate outputs of the first stage (and inputs of the second one) are patent applications and new developed products. The common inputs that are shared between the first and second stages are the common base of patents and fixed assets. Additional inputs of the second stage are the costs of developing new products, the costs of modernization and the number of employees. The output of the second stage is the total revenue from the sale of new products. In [77], the authors transform the internal expenditure of R&D expense into internal expenditure stock and introduce the idea of the deprecation in R&D capital; therefore, they use a dynamic network DEA model.

The study in [74] considers the high-technology industry in China as having a three-level hierarchical structure. Five major branches—the manufacture of medicines, the manufacture of aircraft and spacecraft, the manufacture of electronic equipment and communication equipment, the manufacture of computers and office equipment, and the manufacture of medical equipment and measuring instruments—are arranged as the first level and can be further divided into different sub-industries in the second level where two sub-industries are divided into three subordinate industries. Each sub-industry has two operational processes: technology development and economic application, which can be described by the two-stage DEA model. Thus, the main difference in this study will be the decomposition of efficiency measures at each level of hierarchy. A similar task of evaluating the efficiency of the high-technology industry in China is solved in [82] using the same approach of representing different high-tech sectors as two-stage DMUs. The study in [83] proposes an improved NDEA model of innovative Chinese companies, where the commercialization stage consists of two parallel sub-stages (technology commercialization stage and product commercialization stage). The paper in [84] focuses specifically on the innovation efficiency of the pharmaceutical industry in China. The paper in [85] analyzes the efficiency of innovations in the tobacco industry. The study in [86] considers the efficiency of innovations in energy companies.

The papers in [87,88,89] consider organizations of the higher education sector as DMU and divide their activity into two stages: education and scientific research. In [89], educational activities are divided into two sub-processes: the process of the undergraduate level and the graduate level. The graduate level itself is represented by two blocks, the first of which is responsible for education, and the second for scientific research (see Figure 14).

The only input is funding. The intermediate outputs are the number of undergraduate and graduate students, $z_1$, the number of professors, $z_2$, the number of technical and administrative staff, $z_3$, and the area of university buildings used for classes and research, $z_4$. All these outputs are shared between undergraduate and graduate blocks. The only intermediate output that is not shared between these blocks is the number of students at a given level, $z_5$. There are also intermediate outputs between the second and third stages: the number of defended master’s and doctoral dissertations. The output of the first sub-process is the output index, a new variable introduced in this study that is the product of the number of students and the quality of education. The output of the second sub-process is the number of patents and the number of publications in Scopus. The coefficient, $\alpha$, which reflects the share of input received by the first sub-process of the second stage, is restricted to between 60% and 90%.

Kremantzis et al. [88] model a university department (business school) as a multi-layer hierarchical structure. The internal composition of the department consists of three main functions: teaching, research, and enterprise. Teaching is further divided into undergraduate and postgraduate teaching activities. The paper in [90] considers only the research process in universities and divides it into the faculty research process (first stage) and the student research process (second stage). The paper in [91] divides a university into financial division and academic division. For the financial division, it takes non-academic staff, takings of students’ tuition, and public funding as inputs. As the intermediate output, it takes the operation costs. For the academic division, it takes non-academic and academic staff as inputs and the number of undergraduates enrolled, the number of postgraduates enrolled, and the number of graduates (completions) as system outputs. Two variables are taken as carry-over: earnings from research activities (for the financial division) and the floor area for academic spaces (for the academic division). The paper in [92] models the education system as sequences of three sub-processes: primary, secondary, and tertiary education.

The papers in [93,94,95,96,97,98] model the efficiency of hospitals. The study in [94] divides hospital service into a medical–surgical care sub-unit and a quality sub-unit. The paper in [96] considers receiving the patient, checking their condition, and making decisions on how to continue the treatment as the first stage and an operation in the hospital as the second stage. The study in [97] divides it into the hospital stage and pharmacy stage, with both connected in series. The paper in [98] analyzes three types of production–service supply chains (medical, pharmaceutical, and rehabilitation) as an interconnected network structure. Pereira et al. [95] consider the general structure of the hospital with five main services: the inpatient service, the operating room service, the outpatient surgery service, the medical appointment service, and the emergency service. The corresponding inputs are adjusted labor costs, operating costs, clinical material costs, and outsourcing costs. The system outputs are the number of patients leaving the inpatient service, the number of programmed surgeries in the operating room service, the number of critical surgeries in the operating room service, the number of surgeries that do not involve anesthesia or respiratory assistance in the outpatient surgery service, the number of consultations conducted by a physician in the medical appointment service, and the amount of urgent medical assistance provided by the emergency service. The intermediate products represent the connections between divisions and correspond to patient flows between distinct services in terms of their respective indicators. An interesting feature of this model is that the values of the aforementioned intermediate products are not available. Authors use a bootstrapping method for generating the data.

In addition to the most representative areas of practical applications of the network DEA discussed above, the literature also presents a large number of papers on modeling the activities of enterprises and organizations in various sectors of the economy. Thus, the papers in [99,100,101,102,103] model energy systems. Zhang et al. [99] consider the case of electricity generation (first stage) and electricity transmission (second stage) in China. The inputs of the first stage are (1) installed capacity, (2) raw coal, and (3) generation investment, while the desirable output refers to the electricity generated. The inputs of the second stage are the electricity generated (intermediate output) and grid investment (additional exogenous input), the desirable output is defined as electricity sold, and the undesirable output includes electricity loss. A similar approach is used in [100,101,102] for modeling an electrical energy system in Iran as a three-stage DMU with generation, transmission, and distribution stages. The same idea is used in [103] for modeling the energy system of Mexico.

The papers in [104,105,106,107] model the efficiency of airlines and airports; the studies in [108,109] are devoted to the efficiency of different manufacturing companies. The papers in [110,111,112] analyze logistic companies and systems. The papers in [113,114,115] investigate the efficiency of different health care systems in struggling with the COVID pandemic. The study in [116] focuses on the efficiency of the national transportation industry. The paper in [117] models the fashion retail chain. In our opinion, the last paper deserves a more detailed description due to the originality of the research approach. The authors build a three-stage DEA model to evaluate the effectiveness of sales of products with a short life cycle (for example, fashionable clothing stores). The first stage describes the primary placement of the collection, the second stage is its first replenishment (usually the second week of sales), and the third stage is all subsequent replenishment of the collection until the end of the season. All stages share one entrance (the amount of clothing from the new collection), their outputs (revenue from sales), and one sinter way out (stock of unsold clothing), which at the third stage becomes an undesirable output.

4. Conclusions

This study sheds light on the NDEA literature and highlights the most developed areas of practical applications of network models. Most frequently, network models are used to describe the internal structure and processes in banks, supply chains, production systems, innovation systems, universities or their departments, and healthcare systems. Two-stage models, where the outputs of the first stage are the inputs of the second (intermediate outputs), are the most commonly used. However, in recent years, there has been a noticeable tendency to complicate models and introduce hierarchical structures into them.

The biggest challenge in modeling complex systems and the source of the main differences in the models and results is the choice of indicators, which represent inputs, intermediates, and outputs of the Decision Making Units. The lack of generally accepted indicators that could describe the resources consumed by a complex production system and its useful outputs is the main reason that some practical areas have not yet been covered by modeling.

Our review reveals a large share of the papers that use “multi-stage” terminology in order to reflect several stages of the study, where DEA is used at a certain stage, while other stages employ different methods of economic and mathematical modeling, such as regression, factor analysis, bootstrapping, etc. Another confusing fact in the terminology is the mix-up between parallel, hierarchical, and matrix models.

Some limitations of our research should be pointed out. Firstly, our systematic literature review is only based on the Google Scholar collection. Although it has the most representative collection, we should not ignore the fact that some papers on the topic could have been missed. Secondly, our review was focused on articles in the English language, overlooking potential contributions from authors writing in other languages (Persian, Chinese, Indonesian, and others). Finally, since the reviewed literature in some cases addressed two or more practical applications (for example, the eco-efficiency of the supply chain), some of the classifications provided in our research are based on the criteria of the authors.

In addition, we can note that numerous questions remain unanswered, of which the first is the mathematical approaches to the calculation of partial and system efficiency of DMUs. Due to the limited size of the paper, we leave it as a research agenda for future studies.