Next Article in Journal
Corporate Social Responsibility (CSR) in the Travel Supply Chain: A Literature Review
Next Article in Special Issue
Analysis of Earthquake Forecasting in India Using Supervised Machine Learning Classifiers
Previous Article in Journal
Cross-Sector Partnerships for Innovation and Growth: Can Creative Industries Support Traditional Sector Innovations?
Previous Article in Special Issue
Machine Learning for Conservation Planning in a Changing Climate
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Using a DEA–AutoML Approach to Track SDG Achievements

School of Economics and Management, University of Chinese Academy of Sciences, Beijing 100049, China
College of Belt and Road, University of Chinese Academy of Sciences, Beijing 100049, China
Author to whom correspondence should be addressed.
Sustainability 2020, 12(23), 10124;
Submission received: 22 October 2020 / Revised: 28 November 2020 / Accepted: 30 November 2020 / Published: 4 December 2020


Each country needs to monitor progress on their Sustainable Development Goals (SDGs) to develop strategies that meet the expectations of the United Nations. Data envelope analysis (DEA) can help identify best practices for SDGs by setting goals to compete against. Automated machine learning (AutoML) simplifies machine learning for researchers who need less time and manpower to predict future situations. This work introduces an integrative method that integrates DEA and AutoML to assess and predict performance in SDGs. There are two experiments with different data properties in their interval and correlation to demonstrate the approach. Three prediction targets are set to measure performance in the regression, classification, and multi-target regression algorithms. The back-propagation neural network (BPNN) is used to validate the outputs of the AutoML. As a result, AutoML can outperform BPNN for regression and classification prediction problems. Low standard deviation (SD) data result in poor prediction performance for the BPNN, but does not have a significant impact on AutoML. Highly correlated data result in a higher accuracy, but does not significantly affect the R-squared values between the actual and predicted values. This integrative approach can accurately predict the projected outputs, which can be used as national goals to transform an inefficient country into an efficient country.

1. Introduction

Sustainable Development Goals (SDGs) are a global plan for developing sustainability in order to eradicate poverty while preserving the environment and quality of life of all living things around the world without leaving anyone behind. According to the 2019 Sustainable Development Report, none of the countries is currently pursuing a path to achieve the 17 goals [1]. Therefore, each country needs to monitor its progress in order to develop and promote strategies that meet the expectations of the SDGs [2]. Benchmarking, by setting goals to compete with, can help identify best practices and then apply them reliably to improve a country’s performance towards reaching their SDGs. Data envelope analysis (DEA) is a benchmarking technique in which the optimal combinations of effort (input) and performance (output) result from a linear optimization problem. DEA uses the data from each decision-making unit (DMU) involved in the efficiency assessment to establish an efficient frontier and then determines the positions of each DMU with respect to the frontier. Every country must set future goals in order to achieve their SDGs; however, DEA lacks predictive capability [3,4,5,6]. Some studies have tried to integrate DEA with other methods, such as machine learning, to predict future situations.
Machine learning is a field of artificial intelligence that applies statistical techniques to give computers the ability to learn from data without being explicitly programmed. It is easy to use and understand compared to statistical methods; however, a data scientist needs to understand the perspective of a problem, address the data situation, preprocess the data accordingly, and arrive at a model. The prediction capability limitation of DEA can be fulfilled by the machine learning capability [7,8]; however, designing a good machine learning framework is a crucial step and requires an expert [9,10,11,12]. Automated machine learning (AutoML), one of the most helpful automation techniques, simplifies machine learning by automating various machine learning processes. The aim is to provide machine learning problem solving opportunities for users who are looking for a way to reduce their time and manpower requirements. It helps data scientists make better use of machine learning in their work [13]. AutoML is an emerging research field; however, more studies need to be done to ensure the more widespread application of this technique, which is possible with the increasing availability of open data [14]. We therefore urgently need to use this tool in order to gain insights from this rapidly growing data base.
No individual technique can solve all problems. DEA is used in many studies as a method of evaluating performance, but it is unable to forecast the future efficiency frontier. AutoML is a powerful forecasting tool and can be helpful in choosing the right algorithm; however, it lacks the ability to optimize. Many studies have combined DEA with machine learning to improve the performance of both these decision-support tools. Although this integrative approach can evade the limitations of its individual methods and provide more detailed performance, there are still some gaps to expand the approach. None of the previous studies combined DEA and AutoML approaches to measure efficiency in the SDGs area. As a final refinement in 2020, at the 51st session of the United Nations Statistical Commission, there are now 169 goals and 231 unique indicators for the 17 SDGs, which are expected to be achieved by 2030 [15]. Although concrete goals have not yet been formulated or decided, the principles, on the basis of which the thematic priorities of the sustainable goals have been determined, have already been agreed to [16].
This work shows a method for assessing and predicting the performance of SDGs using an integrative method between DEA and AutoML. A stratification DEA approach was applied in this study to benchmark and evaluate the performance of each country. Then, an AutoML method was processed to predict its efficiency score (ES), efficiency tier (ET), and projected output (PO) in the future based on its historical performance. The experiment regarding the ES is to test the machine learning regression algorithm; for the ET, to test the classification algorithm; and for the PO, to test the regression problem with multiple outputs. There are two empirical studies to demonstrate the integration between DEA and AutoML, with various data properties. The first study consists of long-term and highly correlated data used to measure the annual goals for the sustainable prosperity of a global development strategy called the Belt and Road Initiative (BRI) project. The second study includes short-term and less correlated data, measuring the weekly performance in fighting the COVID-19 pandemic around the world. This integrative approach can efficiently perform both analysis and prediction in these studies, with a faster build and more flexible customization, as well as being user-friendly to everyone.

2. Materials and Methods

In this study, an integrative approach is presented in which DEA and AutoML are integrated to predict ES, ET, and PO, and to support the achievement of the SDGs. The integrative approach consists of four main steps, as shown in Figure 1:
  • implementing a stratification DEA to evaluate the ES and ET, and to examine the PO that will turn an inefficient DMU into an efficient DMU;
  • applying AutoML to predict the three outcomes of the DEA using classification algorithms to determine the ET and regression algorithms to predict the ES and PO;
  • using a back-propagation neural network (BPNN) to produce a series of the same results to validate the AutoML outputs by comparing the rates of precision and accuracy along with the number of DMUs within an acceptable percentage error (PRED);
  • employing DEA by using the predicted PO from the AutoML and BPNN to evaluate the ES and compare them to validate the integrative approach.

2.1. Stratification DEA

DEA is an efficiency analysis technique that is widely used in industry to compare the efficiency of organizational units. Charnes et al. [17] developed a basic DEA model in 1978, called the CCR model after the initials of its developers, which accepts the assumption of constant returns to scale (CRS). Banker et al. [18] introduced the BCC model in 1984, which accepts the assumption of variable returns to scale (VRS). Although DEA users can consider multiple inputs and outputs, only similar DMUs are considered in a benchmark, and all DMUs must have the same inputs and outputs to ensure that applying the DEA produces a significant result. These factors are often not comparable; therefore, the inputs and outputs are multiplied by meaningful weights. A feature of the DEA, compared to other efficiency analysis techniques, is that the weights of the inputs and outputs are determined in the model without user intervention. This work used the BCC output-oriented model to measure efficiency because the DMUs of different sizes can be processed.
Barr et al. [19] proposed the idea of peeling an onion to layer and classify DMUs using a stratification DEA. Stratification is an arrangement in superimposed layers and classes. It is a process of dividing the general population into homogeneous subgroups prior to operation, and separating data from different sources to analyze patterns. Stratification can bring advantages in statistics if the population can be divided into meaningful groups called strata [20,21]. Meaningful means that the layer is relatively homogeneous with regard to one or more properties, which also influences the expression of the property that is ultimately interesting and that differs from one another as clearly as possible. When the subpopulation in a data analysis varies in a general population, it may be advantageous to select data within each subpopulation or stratum. The stratum must be mutually exclusive, i.e., each element of the population is assigned to a unique layer; furthermore, none of the general population elements can be left out. The strata of the stratified dataset are expediently selected such that they are essentially homogeneous in terms of the characteristics relevant for the selection of the strata of the population and differ from one another in terms of the expected values in order to differentiate these characteristics as much as possible. This stratification technique was integrated into DEA as a classification tool in many works [22,23,24,25,26,27]. This work therefore used this technique in combination with DEA to improve the analysis of the results.

2.2. Automated Machine Learning

Machine learning requires a programmed mechanism full of algorithms, each of which can be adjusted more precisely with so-called hyperparameters. An important task of a data scientist is to find the right algorithm for the respective problem and to set it correctly. In fact, the area of responsibility is considerably larger. In a typical machine learning application, users must use the appropriate methods for data preprocessing, feature extraction, and feature selection in order to make datasets usable for machine learning. Following these preprocessing steps, users then need to perform algorithm selection and hyperparameter optimization in order to maximize the predictive performance of the machine learning model. Since many of these steps require time-consuming and human skills on the part of the user, AutoML was designed as an artificial intelligence-based solution to meet this challenge. AutoML simplifies and accelerates the machine learning workflow and enables data scientists with limited specific programming skills to create machine learning systems. As a result, it is no longer just the expertise of data scientists and programmers to create their own machine learning systems. Even with little expertise, a high-quality machine learning model can be trained and deployed using powerful AutoML tools. Not only will the field of activity of data scientists no longer exist, but their focus will shift to more specialized or sophisticated analysis techniques. In a complex machine learning process with difficult problem-solving tasks; however, not all steps can be automated [12,28,29]. Tasks like a basic understanding of the problem or complex data engineering tasks often still need to be performed by people.
As with traditional manual machine learning, AutoML can be used in many areas to create, train, optimize, and deploy machine learning models to solve problems. Typical tasks that are performed with the models implemented with AutoML are classification tasks. For example, it was used in the health sector [28,30,31], the corporate sector [32], the environmental sector [33,34], the energy sector [35,36], and others [37,38,39]. There are many AutoML tools and solutions available today to help data scientists. While some of them are intended for local use, others allow use over cloud-based platforms. Commercial software solutions such as Google Cloud AutoML and Azure Automated ML, which focus on business applications, enable the use of various AutoML services through a cloud-based platform.
There are also open source tools that use standard machine learning algorithms, such as Auto-WEKA [40], auto-sklearn [41], and TPOT [42], or tools that apply deep learning algorithms like Mcfly [43], which focuses on multivariate times series classification, or auto-Keras [44], which focuses on image classification. Olson et al. [45] introduced a tree-based pipeline optimization tool called Tree-based Pipeline Optimization Tool (TPOT) to automate pipeline design. Optimization models from TPOT use genetic programming, an optimization technique inspired by the mechanism of biological evolution in a population that works by selecting the individuals best suited to a constraint, by crossing over and adding random variations. The individuals of the population in this case are models defined by trees that contain only the most efficient pipelines in a development. During development, the population of trees is evolved by changing existing nodes, adding new nodes, or deleting them. This tool is one of the most popular open source AutoML systems because it can automatically build machine learning pipelines with satisfactory accuracy [13]. Balaji and Allen [46] compared three open source AutoML solutions and found that auto-sklearn is best for classification but worst for regression, while TPOT is best for regression and the second for classification problems. In contrast to auto-sklearn, TPOT offers a white box operation because the Python code of the best model is generated in the form of a scikit-learn pipeline. TPOT was chosen as the AutoML solution in this work because it can handle both regression and classification problems well and the structure of the pipeline generated in TPOT is described by a general tree-based pipeline.

2.3. Integration Between DEA and Machine Learning

Zhou et al. [47] showed that the trend towards combining DEA with other analytical methods, such as a neural network (NN), is increasing significantly and that future research should focus more on such combinations in order to improve the accuracy of the analyses and the explanation of the results. The integration between DEA and NN was proposed in a bank branch performance evaluation study to overcome the drawbacks and promote the benefits of both methods [48]. The approach was used not only for performance evaluation and prediction, but also for optimization purposes. As a frontier method, DEA also calculates the PO, which turns an inefficient DMU into an efficient DMU. An NN predicts efficient frontiers instead of an ES, and then the PO is predicted. The efficient frontier indicates the number of efficient DMUs, which accordingly have an ES of 1 or 100%. We adapted this concept by integrating DEA and AutoML instead of NN to perform these tasks and fill the previously mentioned gap. The previous integrative frameworks can be summarized in six categories:
  • first evaluate the ES with DEA and then predict the ES with an NN [49,50,51];
  • first forecast the inputs and outputs with the NN and then evaluate the future ES with DEA [4,6];
  • first evaluate the ES using DEA, then manually group the DMUs based on the ES ranges, and finally predict the groups using an NN [52];
  • first evaluate the ES and PO with DEA and then predict the ES and PO with an NN [3,53,54,55];
  • first evaluate the ET and PO using a stratification DEA and then predict the ET and PO with an NN [5];
  • first evaluate the ET using a stratification DEA to preprocess the NN learning datasets and then predict groups using each stratified learning datasets with an NN [56].
This work combined the fourth and fifth framework to assess and predict the ES, ET and PO. DEA was used as a first step in evaluating the ES to understand the current state of the DMUs and identifying the PO to establish the goal of improving the performance of an inefficient DMU. Original inputs and outputs were used in this step. The second step was to follow the concept proposed by Barr et al. [19] and the stratification technique proposed by Seiford and Zhu [20] to determine the ET of each country based on their performance. ET1 was accredited when the assessed countries achieved a 100% efficiency in the first round. These countries were then removed from the analysis and the DEA model re-executed. This process was repeated and stopped when the DEA’s rule of thumb was met. The rule of thumb was established to ensure that the DEA model can produce a result with a good discrimination between efficient and inefficient DMUs. The number of DMUs should be more than twice the total number of input and output variables to improve the discriminatory power [57,58]. For example, if the total number of input and output variables is 20, the number of DMUs in the final round evaluation should be more than 40. The remaining inefficient DMUs are automatically assigned to tier n + 1 if n is the last lap number. We then employed BPNN and AutoML to predict three targets, including the ES, ET and PO, which were the main results of the DEA experiment. Original inputs and outputs were used, along with the ES and ET resulting from DEA, to predict the ES and ET. To predict the PO, which is the efficient frontier projection, only the original inputs and PO of the DEA experiment were used as the learning dataset. We used the root mean square error (RMSE), PRED, and the coefficient of determination (R-squared) to assess the performance of the regression algorithm, and the precision and accuracy rates to evaluate the performance of the classification algorithm. Finally, DEA was employed to assess the ES using the original inputs and PO predicted from the BPNN and AutoML. Normally, the ES calculated from the projection frontier should be 100%, since the projection frontier is the optimal value for inputs and outputs. In this step, the multi-target regression algorithm was validated. We also applied the distance formula using Pythagorean theorem to find the distance between the efficiency points of a DMU and the maximum efficiency, which is 100% in each SDG. This allowed us to compare the performance of all DMUs in terms of the overall perspective.

2.4. Data Description of the Sustainability Study in the BRI Region

The Belt and Road Initiative (BRI), originally called One Belt One Road, was presented by the Chinese government in fall 2013 [59]. The entire project now affects more than 60% of the world’s population, around 35% of the global economy, and deals with around half of the global energy consumption and carbon emissions [60]. It is therefore important to focus on the progress of sustainability development in this region. An empirical study of this work aims at the area of prosperity, which consists of SDGs 7–11 [61,62]; however, only SDG7, SDG8, and SDG9 focus on the area of prosperity and relate to the energy, economic, and infrastructure developments [63] that correlate with the strategy of the BRI [64]. Lim et al. [65] also stated that SDG7, SDG8, and SDG9 are important to support overall growth, especially in economic terms.
The promotion of sustainable growth in the green economy and the creation of decent jobs while respecting human rights and global borders are important for all developing, emerging, and industrialized countries. SDG7 is a guarantee for affordable, reliable, sustainable, and modern access to energy. SDG8 includes sub-goals for economic growth, productivity growth, and decent job creation. It therefore calls for a global improvement in resource efficiency in consumption and production and strives to decouple economic growth from environmental destruction. Investing in scientific and technological research on sustainable infrastructure can stimulate economic growth, create jobs, and promote prosperity. SDG9 is about building resilient infrastructure and promoting sustainable industrialization and innovation. Research on energy efficiency and economic growth, as well as investments in energy infrastructure and clean energy technologies, should be encouraged.
This empirical study includes a performance assessment of the BRI countries in SDG7, SDG8, and SDG9 to find the best practices for improving performance development in the fields of energy, economics, and infrastructure. Based on the data available from 2011 to 2017, 80 countries in the BRI region were included in this sustainability study. The inputs and outputs of each SDG were extracted from the denominators and numerators of the indicators. Lamy and Millet [66] expressed energy efficiency as ratios depending on the measured aspect, and the authors also extracted the inputs and outputs from the denominators and numerators of the ratios. All indicators are defined by the United Nations and Table 1 shows the description and variables extracted from the indicators in SDG7, SDG8, and SDG9. Based on the available data, we were able to measure some indicators in Targets 7.1–7.3, 8.1–8.2, 8.4–8.5, 8.10, 9.1–9.2, 9.4, and 9.c using the variables shown in Table 2. Energy efficiency is one of the main targets of SDG7, which is to double the global improvement in energy efficiency, ensure access to energy and affordability, and increase the share of renewable energies. The energy intensity of Indicator 7.3.1 is used to measure energy inefficiency; however, the aim of Target 7.3 is to improve energy efficiency. Energy intensity is defined as the energy supplied to the economy per unit of value of economic output. A lower ratio indicates that less energy is used to create an output unit. We therefore reversed it to calculate it as GDP per energy consumption for the evaluation of energy efficiency. The unemployment rate of Indicator 8.5.2 and the carbon emissions of Indicator 9.4.1 are undesirable outputs, and there are four methods to deal with undesirable outputs in DEA [67]. We used the multiplicative inverse transform to process these. In summary, there were two inputs and four outputs for SDG7, four inputs and outputs for SDG8, and three inputs and seven outputs for SDG9. The correlation matrix in Table 3, which measured the R-squared, Spearman’s rank correlation coefficient (ρ), and statistical significance of all variables, shows that all variables were significantly correlated. In addition, access to electricity, the adult population, and manufacturing employment had the highest mean R-squared, and air freight, renewable energy consumption, and commercial bank branches had the lowest mean R-squared. Access to electricity, domestic material consumption, and manufacturing employment had the highest mean ρ, and the consumption of renewable energy, air freight, and passengers had the lowest mean ρ.

2.5. Data Description of the COVID-19 Pandemic Study

A UN report highlighted the need to learn from the current global health crisis called COVID-19 and to implement the SDGs more consistently and faster than before [68]. COVID-19 is currently the subject of intense research; however, there is limited literature using DEA in studies. Shirouyehzad et al. [69] evaluated the effectiveness of COVID-19 management of the outbreak response using DEA. The authors used population and International Health Regulations Core Capacity Scores as inputs and confirmed cases as undesirable outputs to measure the effectiveness of contagion control, and used confirmed cases, recovered cases, and deaths as the inputs, outputs, and undesirable outputs, respectively, to evaluate the effectiveness of the medical treatment. Ghasemi et al. [70] applied DEA to measure government performance during the COVID-19 outbreak. In the first step, the authors rated the effectiveness of preventing the spread of the coronavirus using population and population density as inputs and confirmed cases as the undesirable output. In this study, an index called the stringency index was used to analyze the prevention of spread. The authors rated the effectiveness of preventing coronavirus deaths in the second step, using the first step score, population, and population aged 65 and over as the inputs, and deaths as the undesirable output. The stringency index is one of the policy indices in the Oxford COVID-19 Government Response Tracker (OxCGRT) analysis, which shows how the infection situation is developing in different countries in relation to mitigation and containment measures [71]. In total, the researchers examined 17 indicators and aggregated the indicators under four parameters, including the overall government response index, the stringency index, the containment and health index, and the economic support index.
Good health is the goal and result of SDG3, which comprises a total of 28 indicators. In addition, nine indicators measure mortality rates and five indicators from Target 3.3 measure new and active cases caused by diseases. Half of the world’s population does not yet have access to basic health services [1]; therefore, government support in the health sector is included in SDG3. Since this empirical study is an assessment of management performance during the COVID-19 pandemic, we have adjusted all variables to focus only on the COVID-19 pandemic. In summary, there were three inputs including active cases, which were the weekly base load work, the containment and health index, which represented the base severity of the lockdown restrictions and closings in each country, and the newly confirmed cases, with one desirable output being active cases in the next three weeks, and two undesirable outputs being the newly recovered cases and new deaths in the next three weeks. The results of this part illustrate the effectiveness of the medical treatment for COVID-19.
SDG11 also targets security and sustainable communities. Target 11.5 points to the number of people affected by disasters; additionally, Target 11.b also focuses on policies and plans to manage and reduce disaster risk. These targets highlight issues related to public and potential disasters. The COVID-19 pandemic is a new type of infectious disease hitting the 21st century, affecting a wide range of people. To control the spread of COVID-19, many governments are applying social distancing and local lockdown restrictions. These restrictions affect the behavior of people in activities and travel [72,73]. Hadjidemetriou et al. [74] stressed that the reduction in human mobility had a significant impact on the confirmed COVID-19 deaths. Badr et al. [75] also found that the decline in mobility patterns correlated strongly with the slowdown in the growth rates of COVID-19 cases. Zhou et al. [76] mentioned that the extent of the impairment of mobility could help delay the peak number of transmission cases. Community mobility is therefore a key strategy for governments to deal with COVID-19. The Google COVID-19 Community Mobility Report shows a change in the mobility of people to categorized locations compared to base days. In this study, three places were categorized, namely, workplaces, residential places, and public places. The daily visits to public places were combined by people visiting four categorized locations, namely, retail and recreation, grocery and pharmacy, transit stations, and parks. Not only is the Google Mobility Report useful, the Google trend also shows during COVID-19. Pier et al. [77] evaluated Google searches for otolaryngology-related terms in the United States and found a significant change during COVID-19. Kurian et al. [78] also measured a correlation between COVID-19 cases and Google Trends in the United States and concluded that they were highly correlated. In addition, Google Trends could be used to monitor a new outbreak area. Husnayain et al. [79] analyzed search terms related to COVID-19, hand-washing, and face masks in Taiwan on Google and indicated that Google Trends could be used to monitor public nervousness about COVID-19. Effenberger et al. [80] reported that the public interest represented by Google Trends correlated with the number of new COVID-19 cases. In summary, there were four inputs, namely, days from the first confirmed case, new tests, trending inquiries about “covid” reflecting the awareness of people in each country, and the OxCGRT’s overall government response index, which served as the basis for the severity of government policies in each country, with three desirable outputs reflecting the results of the restrictions—daily visits to workplaces, residential places, and public places—and one undesirable output, which was newly confirmed cases representing the unwanted outcome of all responses from governments. This part shows a performance evaluation of the preparation for coping with the current infectious disease COVID-19.
The incubation period of COVID-19, which is the time between infection and the appearance of the first symptoms, averages five to six days; however, it can take up to two weeks for a symptom to appear [81]. The newly confirmed case dataset was therefore created with a delay of one week after the week evaluated. After an incubation period, fever, muscle pain, and a dry cough may occur. The disease often manifests itself in a general, severe feeling of illness and back pain [82]. The WHO–China Joint Mission on COVID-19 reported that the median time from first symptoms to recovery from mild cases is approximately two weeks [83]. This information was used to create a lag time for the newly recovered case dataset; therefore, the newly recovered case dataset was two weeks after the newly confirmed case dataset. Wang et al. [84] developed an algorithm to predict the days from hospitalization to death, and the result was 13 days. This result was used to create a delay time for the new death dataset; therefore, the new death dataset was also two weeks after the newly confirmed case dataset. In summary, the newly confirmed case dataset was collected from week t + 1, and the newly recovered case and newly death datasets were pooled from week t + 3 if the week evaluated was week t.
Based on data availability, 61 countries were assessed from 12 April to 23 May or from Weeks 16 to 21, 2020, using all of the variables listed in Table 4. Only Uganda was death-free during the study period. Although Taiwan was the fourth county to experience COVID-19 in this study, they had the lowest active cases at Week 24. Taiwan also had the lowest cumulative number of recovered cases, newly confirmed cases, and deaths during the investigation period when the country without deaths was excluded. Belarus had the lowest scores on the overall government response index and the containment and health index, followed by Taiwan. The United States had the highest active cases at Week 24. They also had the highest cumulative number of recovered cases, newly confirmed cases, and deaths during the period studied. In contrast, Costa Rica had the lowest cumulative COVID-19 tests in this study. Although El Salvador had the highest score on the overall government response index, they had the lowest trending inquiries about “covid”, which was a factor that reflected people’s interest. Restricted mobility to public places changed the most in Bolivia, while Panama was the country where mobility to workplaces changed the most, and second place after Singapore in mobility to residential places for all weeks. The correlation matrix in Table 5 shows that the variables in the COVID-19 pandemic study were less correlated than those in the BRI sustainability study. Only the newly confirmed case variable correlated significantly in rank with all variables. In addition, the newly confirmed cases, as well as active cases in week t + 3 and week t had the highest mean in both R-squared and ρ. Trend inquiries about “covid”, days from the first confirmed case, and daily visits to workplaces had the lowest mean value in both R-squared and ρ.

3. The BRI Sustainability Study

This section presents a study that compares and assesses sustainability progress in relation to SDG7, SDG8, and SDG9 in order to support policymakers in developing strategies to achieve the targets. The SDG7 performance assessment considered two inputs, namely energy consumption and the national population, and four outputs, namely access to clean fuels, access to electricity, renewable energy consumption, and GDP. For SDG8, we defined total workforce, adult population, and total employment as the inputs, GDP and domestic material consumption as the desirable outputs, and the number of unemployed as the undesirable output. National population, total employment, and gross value added were inputs; the number of air freight and passengers, employment, value added in manufacturing, internet users, and mobile phone subscriptions were the desirable outputs; and carbon emissions were an undesirable output in the performance assessment of SDG9. Annual national population was the variable that served as input for all SDGs, as people are the key factor for sustainability. Employment also played an important role in SDG8 and SDG9, showing that employment is essential for economic development. SDG7 and SDG8 also shared an important output, GDP, as these SDGs mainly focus on the area of prosperity.

3.1. Result of the DEA Experiment

We measured the effectiveness of SDG7, SDG8, and SDG9 from 2011 to 2017 in 80 BRI countries, and the results are shown in Table A1 in Appendix A. There were seven countries that performed efficiently in all periods studied, including Austria, China, Italy, Luxembourg, Malta, Russia, and Saudi Arabia. These countries were in the 30 countries with the highest GDP per capita in 2017. Six of these countries were also in 30 countries with the highest value added per employment in manufacturing and the highest energy consumption per capita. These results are consistent with the selection of inputs and outputs as the national population, employment, and GDP were the main drivers for SDG7, SDG8, and SDG9.
In the first step of the DEA experiment, we measured the ES of the first evaluation round. When compared over the years, the mean ES was relatively stable across all the SDGs, as shown in Table 6a. Only SDG8 had the lowest mean and lowest minimum, but always the highest standard deviation (SD) in ES. The mean ES was around 55% and the minimum ES less than 15% each year. SDG8 mainly focuses on economic growth and labor productivity to alleviate poverty; therefore, the progress in economic development of each country in the BRI region seems to be quite small and very different. In addition, Luxembourg was one of the three largest GDP per capita countries in the world, while Mozambique was among the five lowest GDP per capita countries in the world, according to the United Nations in 2017. SDG7 had the highest mean and the highest minimum, but the lowest SD in ES. The mean ES was 97% and the minimum ES was around 50%. This means that most of the countries performed pretty well at providing people with clean and affordable energy and reducing their energy intensity. Although the mean ES in SDG9 was lower than in SDG7, it was still relatively high and satisfactory. The mean ES was 94% and the minimum ES was 36%. This shows that the BRI region considered sustainable industrialization, innovation, and infrastructure equally. In summary, the trends in the mean ES and the minimum ES in SDG7 and SDG8 showed increasing trends, while SD showed a decreasing trend, meaning that countries in the BRI improved performance to get closer. Although SDG9 showed an uptrend from the mean, the minimum showed a downtrend and SD showed a negative uptrend. These showed that the inequality in SDG9 was increasing.
We analyzed the sustainability performance of each country and, in a second step, grouped them based on their performance. In SDG7, SDG8, and SDG9, the total number of input and output variables were six, eight, and ten, so the number of DMUs to analyze in the final round should be greater than 12, 16, and 20, respectively, according to the DEA’s rule of thumb. As shown in Table 6b, all periods studied produced the same number of ET. The results of this ET assessment were comparable to those of the ES assessment because they shared the same characteristics. The number of DMUs in each ET was similar from year to year. SDG8 had the largest number of ET corresponding to six tiers; furthermore, the number of DMUs in each ET from ET 1–5 was quite comparable. Only ET5 contained half of the units compared to the other tiers. The trends in ET1, ET3, and ET5 increased, but ET4 and ET6 had downtrends while ET2 was stable. These showed the same characteristics in relation to ES, leading to the conclusion that SDG8 had the highest variance in ES and the trend of ES increased. SDG7 and SDG9 had the same number of ET, namely, three tiers. ET1 in SDG7 made up almost 70% of the total DMUs, ET2 had about a quarter, while ET3 covered less than 10%. Although the trends in ET1 and ET3 increased, ET2 showed a downward trend. The large number of DMUs in ET1 made the mean ES high, and the narrow distribution in ET2 and ET3 resulted in a low SD in SDG7. Around 60% of the DMUs in SDG9 were in ET1, less than 40% in ET2, and almost 5% in ET3. The proportion of DMUs in ET1 and ET2 was certainly not as different as in SDG7, and the proportion in ET3 was very low. This clearly showed that there were two classified groups in SDG9. While ET1 was in the uptrend, ET2 and ET3 built downtrends. The number of DMUs in ET1 of SDG9 was lower than in SDG7, so the mean ES of SDG9 was also lower than that of SDG7. The higher number of DMUs in ET2 and ET3 of SDG9 resulted in a higher SD than that of SDG7.

3.2. Result of the AutoML Experiment

This section shows an experiment predicting the ES, ET, and PO in 2017. We used the collected data and the results of the DEA from 2011 to 2016 to create the learning dataset for training and testing in the AutoML approach. We also employed BPNN to predict those targets using the same learning dataset and compare its performance against AutoML. Clementine software was used to perform traditional BPNN because it is widely recognized and used in many academic studies [85,86,87,88,89]. The ES and ET in 2017 were not that different from previous years, so the DEA’s results could be a capable learning dataset for BPNN and AutoML to get accurate results. Table 7a shows the comparison of the regression evaluation in the ES prediction. The mean ES predicted by BPNN and AutoML were quite similar; however, the SD in the AutoML results were higher than in the BPNN. The maximum ES is normally 100%, but the maximum ES of the BPNN in SDG7 and SDG9 were lower. These two SDGs characteristically had a low SD in ES, so this could affect the algorithms. The minimum ES of BPNN in SDG7 and SDG9 obviously differed from the actual values, but both BPNN and AutoML were able to predict close to the actual minimum ES in SDG8. The RMSE was measured to compare the accuracy of the BPNN and AutoML; moreover, AutoML outperformed BPNN in all predictions. The RMSE of BPNN was the lowest in SDG7 and lower in SDG9 than in SDG8. Although the AutoML’s RMSE was also lowest in SDG7, it was higher in SDG9 than in SDG8. Not only RMSE, but also the number of DMUs with a percentage error less than 20% (PRED(0.20)), 10% (PRED(0.10)), and 5% (PRED(0.05)) from AutoML were higher than BPNN in all SDGs. PRED in SDG7 and SDG9 of both methods were quite similar with the exception of PRED (0.05) in SDG9. PRED and R-squared of BPNN in SDG8, which had the highest SD in ES, were much lower than those of AutoML. AutoML’s predicted ES correlated better with the actual score than BPNN’s because all of the AutoML’s R-squared values were much higher than the BPNN’s in all SDGs. We can say that AutoML predicted the ES better than the popular BPNN technique in this BRI sustainability study.
We then predicted ET using the inputs, outputs, and ET of the DEA experiment as the learning dataset. The precision and accuracy of the outputs predicted from the classification AutoML algorithms were also better than the outputs from the BPNN, as shown in Table 7b. BPNN classified all DMUs in SDG7 with a very low SD as ET1, but AutoML was able to identify all ETs and classified the DMUs based on their performance. Although the BPNN could predict greater than 50% accuracy in SDG7, AutoML could predict greater than 90% accuracy. SDG8 contained six tiers and AutoML again recognized them all and achieved an accuracy of over 85%. It was unlikely that BPNN could only set four tiers, resulting in an accuracy of around 30%. BPNN classified more than half of the DMUs as ET3, but only 13 DMUs that matched. BPNN could detect only ET1 and ET2 in SDG9, but only 59% of the DMUs were correct. Although SDG9 had a higher SD than SDG7 in ES, AutoML still performed well and resulted in an accuracy of 85%. These show that AutoML could also beat BPNN in this classification problem.
We predicted the PO using BPNN and AutoML to study the efficient frontier projection that turned an inefficient DMU into an efficient DMU. Only the original inputs and PO of the DEA experiment were used as the learning dataset. Table 7c shows the evaluation metrics of both methods for comparing the prediction performance. All of AutoML’s RMSE and PRED values were better than the BPNN’s ones, with the exception of the manufacturing value added in SDG9. These results were consistent with the previous finding that AutoML’s regression algorithm could overwhelm the BPNN. Next, we used these predicted POs from the BPNN and AutoML along with the original inputs to measure performance in 2017 using the DEA application. This measurement would compare the performance in the multivariable prediction algorithm, since a machine learning model is basically good at supporting a single-target prediction and ignoring the multiple relationships between target variables. This study aims to optimize all targets together and not just a single target, as all variables are related to each other. Nevertheless, the multiple regression feature was missing from the selected TPOT library, so AutoML predicted the PO as a single target and ignored the relationship of the other outputs when fitting the predictive model. Figure 2 shows the comparison of the ES and the distance to the efficient frontier of all DMUs between BPNN and AutoML. For this experiment, the BPNN’s ES in diamond shape was mostly closer to 100% than that of AutoML in the line shape. The mean ES of all SDGs in BPNN was 0.998, while in AutoML it was 0.988. In addition, the mean minimum ES of BPNN was 0.991 and for AutoML it was 0.872. Although the BPNN could generate more efficient DMUs in each SDG, AutoML could generate more efficient DMUs in all SDGs. While all DMUs of SDG8 in BPNN achieved an ES of 100%, only 25 DMUs in all SDGs could perfectly reach the efficient frontier with zero distance. There were 32 DMUs from AutoML that could perfectly distance zero from the efficient frontier; however, the mean distance in AutoML was still higher than that in the BPNN. Since the original TPOT library was unable to predict regressions with multiple outputs, this may result in a more accurate prediction, but poorer mean ES and distance.

4. The COVID-19 Pandemic Study

This section provides a study on the integrative approach to addressing the COVID-19 pandemic in the context of the SDGs to understand the situation and develop guidelines for management and risk mitigation. To assess the effectiveness of medical treatment in SDG3, three inputs, one desirable output and two undesirable outputs, were applied to the DEA. Performance in preparing to deal with COVID-19, which was defined as a segment in SDG11, was assessed based on four inputs, three desirable outputs, and one undesirable output. The newly confirmed case variable was used in both SDGs, and only this variable correlated significantly in rank with all variables. Some variables were measured at weeks t + 1 and t + 3, when the week evaluated was week t, due to incubation, convalescence, and hospitalization to death.

4.1. Result of the DEA Experiment

Table A2 in Appendix A shows the results of the DEA in this study. The results are the ES in medical treatment associated with SDG3 in Target 3.3 and in preparedness for COVID-19 related to SDG11 in Targets 11.5 and 11.b, as well as the distance between the assessed DMU and the DMU with the best performance. Based on the data available, three countries including Belarus, Taiwan, and Uganda performed best overall in the fight against COVID-19 over all periods studied. Belarus, Taiwan, Uganda, and the United States showed the best effectiveness of medical treatments, while Bolivia, Belarus, Costa Rica, Latvia, Morocco, Panama, El Salvador, Taiwan, Uganda, and Uruguay were best prepared for COVID-19. When looking at the ratio of newly recovered cases to newly confirmed cases during the period studied, Taiwan had the highest number. It also had the lowest ratio of newly confirmed cases to new COVID-19 tests, which was a result in the area of prevention, and it also reduced the number of active cases well. Uganda, the only country with no fatalities, also performed well in terms of the ratio of newly confirmed cases to new COVID-19 tests. Belarus did well for the ratios of deaths to active cases and deaths to newly confirmed cases. Morocco and Uruguay had done well in reducing the number of active cases due to their good prevention practices.
Table 8a shows that the mean ES in both SDG3 and SDG11 were unstable from year to year. SDG3 produced a smaller mean and a smaller minimum, but a higher SD in ES. SDG3 in this study focuses only on the medical management of COVID-19 to measure recovery, deaths, and remaining active cases. It ranged from the country with the highest confirmed and recovered cases and deaths in the world to countries with no deaths; therefore, the medical treatment situation was different in each country. The results in SDG3 show that many DMUs could handle this situation well, but more DMUs deteriorated at Week 20 due to the lowest mean ES over that period. The maximum of the minimum ES in SDG3 was below 15%, which was quite low compared to SDG11, at almost 40%. SDG11 aims to assess the effectiveness of the preparations that most countries had made regarding social distancing and local lockdown restrictions; therefore, the SD was lower than that of SDG3. The mean ES for all periods were nearly 80%, except for Week 20, which was less than 75%. This means that most countries were pretty well prepared for COVID-19. In summary, the mean ES and SD of SDG3 were fairly stable over the periods studied, while the minimum ES showed a small downward trend. This means that medical treatment performance did not change significantly over the period studied and the worst countries did not improve. The mean ES of SDG11 showed a slight downward trend, the minimum ES showed an upward trend, but the SD showed a downward trend. These describe that there were more countries that were poorly prepared for COVID-19, but the worst countries tried to improve and narrow the gap with more efficient countries.
Based on the DEA’s rule of thumb, plus six and eight of the total input and output variables in SDG3 and SDG11, the final round of DEA should have more than 12 and 16 DMUs for analysis, respectively. We used the stratification DEA to classify the ETs and the results varied over time, as shown in Table 8b. In addition, they were relatively identical to the results of the ES assessments. SDG3 had more ETs than SDG11 because the variance in SDG3 was higher than in SDG11. ET1 had the largest number of DMUs in Weeks 18, 19, and 21, while ET2 had the largest number in Weeks 16 and 17. Week 20 was completely different from the others as all ETs had similar numbers of DMUs; moreover, this week had the lowest mean ES. This lowest ES could come from the largest number of DMUs in ET5. Week 21 was also special as ET1 was more than twice the size of the other ET, resulting in the highest mean ES. In addition, ET1, ET4, and ET5 generated upward trends, while ET2 and ET3 generated downward trends. These express that most countries performed well in terms of medical treatment, but the gap between countries with high and low efficiency had widened. There were four ETs in SDG11; additionally, the number of DMUs in ET4 was lowest each week. Results varied slightly from Weeks 16 to 20, with ET2 being greatest and ET3 greater than ET1. In contrast, the number of DMUs in ET1, ET2, and ET3 at Week 21 were comparable. The trends in ET1 and ET3 were positive, while ET2 and ET4 were declining. These show that most countries performed well in preparing for COVID-19 and that performance improved.

4.2. Result of the AutoML Experiment

This part shows an experiment to predict the ES, ET, and PO at Week 21 using the collected data and DEA results at Weeks 16 to 20 as the learning dataset. When comparing the results of the regression score in ES prediction between the BPNN and AutoML, the final conclusion was the same as in the BRI sustainability study, as shown in Table 9a. The mean ES predicted by the BPNN was very different from the AutoML in SDG3, but was quite similar in SDG11. In contrast, the predicted ES of both methods in the BRI study was quite similar. Only the maximum ES predicted by the BPNN in SDG3 was 100%, but AutoML in the BRI study was able to achieve 100% in all SDGs. The minimum ES predicted by BPNN in SDG3 was completely different from the actual values, but the minimum ES predicted by BPNN and AutoML in SDG11 was close to the actual values. Not only were the RMSE of AutoML in SDG3 and SDG11 lower than that of BPNN, but all of AutoML’s PREDs were also better than those of the BPNN, which were the same as in the BRI study. Nonetheless, the AutoML’s PREDs were lower in this study than in the BRI study. PREDs and R-squared in SDG11 were better than in SDG3, which could be due to the lower SD. The ES predicted by AutoML correlated better with the actual score than that predicted by the BPNN, and the R-squared of the AutoML in this study was similar to the BRI study. We can summarize that AutoML performed better than BPNN in this COVID-19 prediction experiment. The ET of each SDG was predicted by the BPNN and AutoML using the inputs, outputs, and ET of the DEA experiment as a learning dataset to compare the performance in the classification algorithm, as shown in Table 9b. Both the BPNN and AutoML approaches were able to capture all ETs equally; however, AutoML was more accurate than BPNN. Therefore, AutoML could also perform better than BPNN on this COVID-19 classification problem. We then predicted the PO of SDG3 and SDG11 using only the original inputs and PO from the DEA experiment as the learning dataset. The evaluation metrics in Table 9c show the comparison of the BPNN and AutoML approaches. As with the results in the BRI study, all of the AutoML’s RMSE, PRED, and R-squared scores were better than the BPNN’s ones, with the exception of PRED (0.05) on daily visits to workplaces and R-squared on daily visits to public places in SDG11. Finally, we used this predicted PO along with the original inputs to measure Week 21′s performance using the DEA application and to compare performance on regression with multiple outputs. Figure 3 illustrates the comparison of the ESs and the distance to the efficient frontier of all DMUs between BPNN and AutoML. The ESs assessed by the BPNN were closer to 100% than those of AutoML in all SDGs. The mean ES of the BPNN was 0.973 in SDG3 and 0.983 in SDG11, while the mean ES of AutoML was 0.816 in SDG3 and 0.980 in SDG11. The minimum ES of BPNN was 0.899 in SDG3 and 0.912 in SDG11, while the minimum ES of AutoML was 0.272 in SDG3 and 0.778 in SDG11. These values were lower than the results in the BRI study. There were 31 efficient DMUs in SDG3 and 22 efficient DMUs in SDG11 from BPNN. In contrast, AutoML’s efficient DMUs were 19 DMUs in SDG3 and 32 DMUs in SDG11. There were 12 DMUs from BPNN and 11 DMUs from AutoML that could perfectly distance zero from the efficient frontier; however, the mean distance in BPNN was quite higher than that in AutoML. As previously described, the data in the BRI study were better correlated than the data in this COVID-19 study; therefore, the results of the BRI study showed a higher ES and a higher ratio of efficient DMUs to total DMUs. Due to the limitation of TPOT, BPNN performed slightly better than AutoML in this regression problem with multiple outputs. Moreover, the integrative approach can be used to measure current performance in preparing for and dealing with COVID-19. Then each country understands its own capabilities and can choose to follow best practices to fight the COVID-19 pandemic. The projected results can be used as a target in action.

5. Conclusions

Since the DEA does not have predictive capabilities, it is integrated with an AutoML application to predict the future performance and projected outputs, thus developing such a predictive ability. This article introduces the integrative method to assess and predict the performance of a country’s SDGs. There are two empirical studies with different data properties to demonstrate the integrative approach. Three prediction targets were used to measure the performance of AutoML in the regression, classification, and multi-target regression algorithms. A BPNN was used to validate the AutoML outputs. There were three main outcomes, namely, a performance measurement analysis, performance evaluation of the predictive methods, and a goal analysis.
Most countries in the BRI region performed reasonably well on SDG7, but progress on SDG8 was quite limited. Countries in the BRI region were considered equally in SDG9, but inequality in SDG9 increased as the countries were clearly divided into two groups. Most countries performed well in treating COVID-19 and preparing for COVID-19; additionally, performance did not change significantly over the period studied.
In particular, policymakers should improve access to clean fuels and electricity to improve performance in SDG7. Domestic material consumption and employment in manufacturing are the key outputs leading to high performance in SDG8 and SDG9. To improve the performance of medical treatment for COVID-19 in SDG3, recovered cases and deaths are the primary goals that everyone should focus on. Daily visits to workplaces and residential areas are two main criteria by which the performance of preparing to deal with COVID-19 in SDG11 should be measured. These factors should be used as national goals in support of the achievement of the SDGs.
A low SD results in poor regression and classification prediction performance for the BPNN, but does not significantly affect AutoML. AutoML can outperform a BPNN on regression and classification prediction problems; however, a BPNN can handle the problem of multi-target optimization better than AutoML due to the limitation of TPOT in the multi-output regression. Long-term data with a high correlation can provide a more accurate and precise prediction result than short-term data with a lower correlation; however, the R-squared values of both data properties were quite similar.
The integrative approach can accurately predict the projected outputs that can be used as national goals to transform an inefficient country into an efficient country to meet the expectations in the SDGs. The use of this method to reduce the population size and variance in the population analysis can be expanded in future work.

Author Contributions

Conceptualization, B.S. and D.W.; methodology, B.S. and D.W.; software, B.S.; validation, B.S.; formal analysis, B.S.; investigation, B.S.; data curation, B.S.; writing—original draft preparation, B.S.; writing—review and editing, B.S.; supervision, D.W. All authors have read and agreed to the published version of the manuscript.


This work was supported in part by the Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant 2020AAA0108400, in part by the National Natural Science Foundation of China under Grant 71825007, in part by the Chinese Academy of Sciences Frontier Scientific Research Key Project under Grant QYZDB-SSW-SYS021, and supported by the International Partnership Program of Chinese Academy of Sciences, Grant No. 211211KYSB20180042 and the Strategic Priority Research Program of CAS under Grant XDA2302020.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table A1. DEA experiment results in the sustainability study of the BRI.
Table A1. DEA experiment results in the sustainability study of the BRI.
2011 2012 2013 2014 2015 2016 2017
AT *1.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.000
CN *1.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.000
IT *1.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.000
LU *1.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.000
MT *1.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.000
RU *1.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.000
SA *1.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.0001.0001.0001.0000.000
* Efficient DMUs in all studied periods.
Table A2. DEA experiment results in the COVID-19 pandemic study.
Table A2. DEA experiment results in the COVID-19 pandemic study.
16th Week17th Week18th Week19th Week20th Week21st Week
BY *1.0001.0000.0001.0001.0000.0001.0001.0000.0001.0001.0000.0001.0001.0000.0001.0001.0000.000
TW *1.0001.0000.0001.0001.0000.0001.0001.0000.0001.0001.0000.0001.0001.0000.0001.0001.0000.000
UG *1.0001.0000.0001.0001.0000.0001.0001.0000.0001.0001.0000.0001.0001.0000.0001.0001.0000.000
* Efficient DMUs in all studied periods.


  1. United Nations. The Sustainable Development Goals Report 2019; The Sustainable Development Goals Report; United Nations: San Francisco, CA, USA, 2019. [Google Scholar]
  2. van Zanten, J.A.; van Tulder, R. Beyond COVID-19: Applying “SDG logics” for resilient transformations. J. Int. Bus. Policy 2020. [Google Scholar] [CrossRef]
  3. Kwon, H.B.; Lee, J. Two-stage production modeling of large U.S. banks: A DEA-neural network approach. Expert Syst. Appl. 2015, 42, 6758–6766. [Google Scholar] [CrossRef]
  4. Shokrollahpour, E.; Lotfi, F.H.; Zandieh, M. An integrated data envelopment analysis–artificial neural network approach for benchmarking of bank branches. J. Ind. Eng. Int. 2016, 12, 137–143. [Google Scholar] [CrossRef] [Green Version]
  5. Kwon, H.B.; Marvel, J.H.; Roh, J.J. Three-stage performance modeling using DEA–BPNN for better practice benchmarking. Expert Syst. Appl. 2017, 71, 429–441. [Google Scholar] [CrossRef]
  6. Shabanpour, H.; Yousefi, S.; Saen, R.F. Forecasting efficiency of green suppliers by dynamic data envelopment analysis and artificial neural networks. J. Clean. Prod. 2017, 142, 1098–1107. [Google Scholar] [CrossRef]
  7. Hong, H.K.; Ha, S.H.; Shin, C.K.; Park, S.C.; Kim, S.H. Evaluating the efficiency of system integration projects using data envelopment analysis (DEA) and machine learning. Expert Syst. Appl. 1999, 16, 283–296. [Google Scholar] [CrossRef]
  8. Nandy, A.; Singh, P.K. Farm efficiency estimation using a hybrid approach of machine-learning and data envelopment analysis: Evidence from rural eastern India. J. Clean. Prod. 2020, 267, 122106. [Google Scholar] [CrossRef]
  9. Villanueva Zacarias, A.G.; Reimann, P.; Mitschang, B. A framework to guide the selection and configuration of machine-learning-based data analytics solutions in manufacturing. Procedia CIRP 2018, 72, 153–158. [Google Scholar] [CrossRef]
  10. Lee, K.M.; Yoo, J.; Kim, S.W.; Lee, J.H.; Hong, J. Autonomic machine learning platform. Int. J. Inf. Manag. 2019, 49, 491–501. [Google Scholar] [CrossRef]
  11. Quemy, A. Two-stage optimization for machine learning workflow. Inf. Syst. 2020, 92, 101483. [Google Scholar] [CrossRef] [Green Version]
  12. Zhang, Q.; Hu, W.; Liu, Z.; Tan, J. TBM performance prediction with Bayesian optimization and automated machine learning. Tunn. Undergr. Sp. Technol. 2020, 103, 103493. [Google Scholar] [CrossRef]
  13. Waring, J.; Lindvall, C.; Umeton, R. Automated machine learning: Review of the state-of-the-art and opportunities for healthcare. Artif. Intell. Med. 2020, 104, 101822. [Google Scholar] [CrossRef] [PubMed]
  14. Goldstein, E.B.; Coco, G.; Plant, N.G. A review of machine learning applications to coastal sediment transport and morphodynamics. Earth-Sci. Rev. 2019, 194, 97–108. [Google Scholar] [CrossRef] [Green Version]
  15. UN General Assembly. Global Indicator Framework for the Sustainable Development Goals and Targets of the 2030 Agenda for Sustainable Development (A/RES/71/313, E/CN.3/2018/2, E/CN.3/2019/2 and E/CN.3/2020/2); UN General Assembly: New York, NY, USA, 2020. [Google Scholar]
  16. Janoušková, S.; Hák, T.; Moldan, B. Global SDGs assessments: Helping or confusing indicators? Sustainability 2018, 10, 1540. [Google Scholar] [CrossRef] [Green Version]
  17. Charnes, A.; Cooper, W.W.; Rhodes, E. Measuring the efficiency of decision making units. Eur. J. Oper. Res. 1978, 2, 429–444. [Google Scholar] [CrossRef]
  18. Banker, R.D.; Charnes, A.; Cooper, W.W. Some models for estimating technical and scale inefficiencies in data envelopment analysis. Manage. Sci. 1984, 30, 1078–1092. [Google Scholar] [CrossRef] [Green Version]
  19. Barr, R.; Durchholz, M.; Seiford, L. Peeling the DEA Onion: Layering and Rank-Ordering DMUs Using Tiered DEA; Southern Methodist University Technical Report: Dallas, TX, USA, 2000. [Google Scholar]
  20. Seiford, L.M.; Zhu, J. Context-dependent data envelopment analysis—Measuring attractiveness and progress. Omega 2003, 31, 397–408. [Google Scholar] [CrossRef]
  21. Karamoozian, A.; Wu, D.; Chen, C.L.P.; Luo, C. An Approach for Risk Prioritization in Construction Projects Using Analytic Network Process and Decision Making Trial and Evaluation Laboratory. IEEE Access 2019, 7, 159842–159854. [Google Scholar] [CrossRef]
  22. Bougnol, M.L.; Dulá, J.H. Validating DEA as a ranking tool: An application of DEA to assess performance in higher education. Ann. Oper. Res. 2006, 145, 339–365. [Google Scholar] [CrossRef]
  23. Sharma, M.J.; Yu, S.J. Performance based stratification and clustering for benchmarking of container terminals. Expert Syst. Appl. 2009, 36, 5016–5022. [Google Scholar] [CrossRef]
  24. Lim, S.; Bae, H.; Lee, L.H. A study on the selection of benchmarking paths in DEA. Expert Syst. Appl. 2011, 38, 7665–7673. [Google Scholar] [CrossRef]
  25. Lu, W.M.; Lo, S.F. Constructing stratifications for regions in China with sustainable development concerns. Qual. Quant. 2012, 46, 1807–1823. [Google Scholar] [CrossRef]
  26. Hong, J.D.; Jeong, K.Y. Cross efficiency based heuristics to rank decision making units in data envelopment analysis. Comput. Ind. Eng. 2017, 111, 320–330. [Google Scholar] [CrossRef]
  27. Lim, S. Context-dependent data envelopment analysis with cross-efficiency evaluation. J. Oper. Res. Soc. 2012, 63, 38–46. [Google Scholar] [CrossRef]
  28. Orlenko, A.; Moore, J.H.; Orzechowski, P.; Olson, R.S.; Cairns, J.; Caraballo, P.J.; Weinshilboum, R.M.; Wang, L.; Breitenstein, M.K. Considerations for automated machine learning in clinical metabolic profiling: Altered homocysteine plasma concentration associated with metformin exposure. In Proceedings of the Pacific Symposium on Biocomputing, Big Island, HI, USA, 3–7 January 2018; pp. 460–471. [Google Scholar]
  29. Toh, T.S.; Dondelinger, F.; Wang, D. Looking beyond the hype: Applied AI and machine learning in translational medicine. EBioMedicine 2019, 47, 607–615. [Google Scholar] [CrossRef] [Green Version]
  30. Villela, M.A.; Bravo, C.A.; Shah, M.; Patel, S.; Jorde, U.P.; Stehlik, J.; Castellanos, A. Prediction of Outcomes after Heart Transplantation Using Machine Learning Techniques. J. Heart Lung Transplant. 2020, 39, S295–S296. [Google Scholar] [CrossRef]
  31. Faes, L.; Wagner, S.K.; Fu, D.J.; Liu, X.; Korot, E.; Ledsam, J.R.; Back, T.; Chopra, R.; Pontikos, N.; Kern, C.; et al. Automated deep learning design for medical image classification by health-care professionals with no coding experience: A feasibility study. Lancet Digit. Heal. 2019, 1, e232–e242. [Google Scholar] [CrossRef] [Green Version]
  32. Jang, H. A decision support framework for robust R&D budget allocation using machine learning and optimization. Decis. Support Syst. 2019, 121, 1–12. [Google Scholar] [CrossRef]
  33. Lautenschlager, F.; Becker, M.; Kobs, K.; Steininger, M.; Davidson, P.; Krause, A.; Hotho, A. OpenLUR: Off-the-shelf air pollution modeling with open features and machine learning. Atmos. Environ. 2020, 233, 117535. [Google Scholar] [CrossRef]
  34. Adams, M.D.; Massey, F.; Chastko, K.; Cupini, C. Spatial modelling of particulate matter air pollution sensor measurements collected by community scientists while cycling, land use regression with spatial cross-validation, and applications of machine learning for data correction. Atmos. Environ. 2020, 230, 117479. [Google Scholar] [CrossRef]
  35. Liu, H.; Chen, J.; Hissel, D.; Su, H. Remaining useful life estimation for proton exchange membrane fuel cells using a hybrid method. Appl. Energy 2019, 237, 910–919. [Google Scholar] [CrossRef]
  36. Kosir, S.; Heyne, J.; Graham, J. A machine learning framework for drop-in volume swell characteristics of sustainable aviation fuel. Fuel 2020, 274, 117832. [Google Scholar] [CrossRef]
  37. Soares, E.F.; Campos, C.A.; de Lucena, S.C. Online travel mode detection method using automated machine learning and feature engineering. Futur. Gener. Comput. Syst. 2019, 101, 1201–1212. [Google Scholar] [CrossRef]
  38. Zeng, Y.; Zhang, J. A machine learning model for detecting invasive ductal carcinoma with Google Cloud AutoML Vision. Comput. Biol. Med. 2020, 122, 103861. [Google Scholar] [CrossRef]
  39. Tsamardinos, I.; Fanourgakis, G.S.; Greasidou, E.; Klontzas, E.; Gkagkas, K.; Froudakis, G.E. An Automated Machine Learning architecture for the accelerated prediction of Metal-Organic Frameworks performance in energy and environmental applications. Microporous Mesoporous Mater. 2020, 300, 110160. [Google Scholar] [CrossRef]
  40. Kotthoff, L.; Thornton, C.; Hoos, H.H.; Hutter, F.; Leyton-Brown, K. Auto-WEKA: Automatic Model Selection and Hyperparameter Optimization in WEKA. In Automated Machine Learning: Methods, Systems, Challenges; Hutter, F., Kotthoff, L., Vanschoren, J., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 81–95. ISBN 978-3-030-05318-5. [Google Scholar]
  41. Feurer, M.; Klein, A.; Eggensperger, K.; Springenberg, J.T.; Blum, M.; Hutter, F. Auto-sklearn: Efficient and Robust Automated Machine Learning. In Automated Machine Learning: Methods, Systems, Challenges; Hutter, F., Kotthoff, L., Vanschoren, J., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 113–134. ISBN 978-3-030-05318-5. [Google Scholar]
  42. Olson, R.S.; Moore, J.H. TPOT: A Tree-Based Pipeline Optimization Tool for Automating Machine Learning. In Automated Machine Learning: Methods, Systems, Challenges; Hutter, F., Kotthoff, L., Vanschoren, J., Eds.; Springer International Publishing: Cham, Switzerland, 2019; pp. 151–160. ISBN 978-3-030-05318-5. [Google Scholar]
  43. van Kuppevelt, D.; Meijer, C.; Huber, F.; van der Ploeg, A.; Georgievska, S.; van Hees, V.T. Mcfly: Automated deep learning on time series. SoftwareX 2020, 12, 100548. [Google Scholar] [CrossRef]
  44. Jin, H.; Song, Q.; Hu, X. Auto-Keras: An Efficient Neural Architecture Search System. In Proceedings of the Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA, 4–8 August 2019; Association for Computing Machinery: New York, NY, USA, 2019; pp. 1946–1956. [Google Scholar]
  45. Olson, R.S.; Urbanowicz, R.J.; Andrews, P.C.; Lavender, N.A.; Kidd, L.C.; Moore, J.H. Automating biomedical data science through tree-based pipeline optimization. In Proceedings of the Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Springer: Berlin/Heidelberg, Germany, 2016; Volume 9597, pp. 123–137. [Google Scholar]
  46. Balaji, A.; Allen, A. Benchmarking Automatic Machine Learning Frameworks. arXiv 2018, arXiv:1808.06492. [Google Scholar]
  47. Zhou, H.; Yang, Y.; Chen, Y.; Zhu, J. Data envelopment analysis application in sustainability: The origins, development and future directions. Eur. J. Oper. Res. 2018, 264, 1–16. [Google Scholar] [CrossRef]
  48. Athanassopoulos, A.D.; Curram, S.P. A comparison of data envelopment analysis and artificial neural networks as tools for assessing the efficiency of decision making units. J. Oper. Res. Soc. 1996, 47, 1000–1016. [Google Scholar] [CrossRef]
  49. Wu, D.; Yang, Z.; Liang, L. Using DEA-neural network approach to evaluate branch efficiency of a large Canadian bank. Expert Syst. Appl. 2006, 31, 108–115. [Google Scholar] [CrossRef]
  50. Agarwal, R.; Green, R.; Brown, P.J.; Tan, H.; Randhawa, K. Determinants of quality management practices: An empirical study of New Zealand manufacturing firms. Int. J. Prod. Econ. 2013, 142, 130–145. [Google Scholar] [CrossRef]
  51. Hanafizadeh, P.; Khedmatgozar, H.R.; Emrouznejad, A.; Derakhshan, M. Neural network DEA for measuring the efficiency of mutual funds. Int. J. Appl. Decis. Sci. 2014, 7, 255–269. [Google Scholar] [CrossRef]
  52. Mostafa, M.M. Modeling the efficiency of top Arab banks: A DEA-neural network approach. Expert Syst. Appl. 2009, 36, 309–320. [Google Scholar] [CrossRef]
  53. Kwon, H.B.; Lee, J.; Roh, J.J. Best performance modeling using complementary DEA-ANN approach: Application to Japanese electronics manufacturing firms. Benchmarking 2016, 23, 704–721. [Google Scholar] [CrossRef]
  54. Kwon, H.B. Exploring the predictive potential of artificial neural networks in conjunction with DEA in railroad performance modeling. Int. J. Prod. Econ. 2017, 183, 159–170. [Google Scholar] [CrossRef]
  55. Kwon, H.B. Performance modeling of mobile phone providers: A dea-ann combined approach. Benchmarking 2014, 21, 1120–1144. [Google Scholar] [CrossRef]
  56. Misiunas, N.; Oztekin, A.; Chen, Y.; Chandra, K. DEANN: A healthcare analytic methodology of data envelopment analysis and artificial neural networks for the prediction of organ recipient functional status. Omega 2016, 58, 46–54. [Google Scholar] [CrossRef]
  57. Golany, B.; Roll, Y. An application procedure for DEA. Omega 1989, 17, 237–250. [Google Scholar] [CrossRef]
  58. Amirhossein, K.; Desheng, W. A hybrid risk prioritization approach in construction projects using failure mode and effective analysis. Eng. Constr. Archit. Manag. 2020, 27, 2661–2686. [Google Scholar] [CrossRef]
  59. Jin, X.; Li, D.D.; Wu, S. How will China shape the world economy? China Econ. Rev. 2016, 40, 272–280. [Google Scholar] [CrossRef]
  60. Singpai, B.; Wu, D.D. An integrative approach for evaluating the environmental economic efficiency. Energy 2020, 118940. [Google Scholar] [CrossRef]
  61. Asian Development Bank. Key Indicators for Asia and the Pacific 2017; Asian Development Bank: Manila, PA, USA, 2017. [Google Scholar]
  62. OECD Measuring Distance to the SDG Targets 2019; OECD Publishing: Paris, France, 2019; ISBN 9789264480940.
  63. Morton, S.; Pencheon, D.; Squires, N. Sustainable Development Goals (SDGs), and their implementation. Br. Med. Bull. 2017, 124, 81–90. [Google Scholar] [CrossRef] [PubMed]
  64. Li, K.-W. China’s Economic Reform Path. Redefining Capital. Glob. Econ. Dev. 2017, 331–358. [Google Scholar] [CrossRef]
  65. Lim, M.M.L.; Søgaard Jørgensen, P.; Wyborn, C.A. Reframing the sustainable development goals to achieve sustainable development in the anthropocene—A systems approach. Ecol. Soc. 2018, 23. [Google Scholar] [CrossRef] [Green Version]
  66. Lamy, C.; Millet, P. A critical review on the definitions used to calculate the energy efficiency coefficients of water electrolysis cells working under near ambient temperature conditions. J. Power Sources 2020, 447, 227350. [Google Scholar] [CrossRef]
  67. Liu, W.; Zhou, Z.; Ma, C.; Liu, D.; Shen, W. Two-stage DEA models with undesirable input-intermediate-outputs. Omega 2015, 56, 74–87. [Google Scholar] [CrossRef]
  68. United Nations. Shared Responsibility, Global Solidarity: Responding to the Socio-Economic Impacts of COVID-19; United Nations: San Francisco, CA, USA, 2020. [Google Scholar]
  69. Shirouyehzad, H.; Jouzdani, J.; Khodadadi Karimvand, M. Fight Against COVID-19: A Global Efficiency Evaluation based on Contagion Control and Medical Treatment. J. Appl. Res. Ind. Eng. 2020, 7, 109–120. [Google Scholar] [CrossRef]
  70. Ghasemi, A.; Boroumand, Y.; Researcher, I.; Shirazi, M. How Do Governments Perform in Facing COVID-19? MPRA: Munich, Germany, 2020. [Google Scholar]
  71. Petherick, A.; Hale, T.; Phillips, T.; Webster, S. Variation in Government Responses to COVID-19|Blavatnik School of Government; Blavatnik School of Government Working Paper; Blavatnik School of Government: Oxford, UK, 27 May 2020. [Google Scholar]
  72. Parady, G.; Taniguchi, A.; Takami, K. Travel behavior changes during the COVID-19 pandemic in Japan: Analyzing the effects of risk perception and social influence on going-out self-restriction. Transp. Res. Interdiscip. Perspect. 2020, 7, 100181. [Google Scholar] [CrossRef]
  73. de Haas, M.; Faber, R.; Hamersma, M. How COVID-19 and the Dutch ‘intelligent lockdown’ change activities, work and travel behaviour: Evidence from longitudinal data in the Netherlands. Transp. Res. Interdiscip. Perspect. 2020, 6, 100150. [Google Scholar] [CrossRef]
  74. Hadjidemetriou, G.M.; Sasidharan, M.; Kouyialis, G.; Parlikad, A.K. The impact of government measures and human mobility trend on COVID-19 related deaths in the UK. Transp. Res. Interdiscip. Perspect. 2020, 6, 100167. [Google Scholar] [CrossRef]
  75. Badr, H.S.; Du, H.; Marshall, M.; Dong, E.; Squire, M.M.; Gardner, L.M. Association between mobility patterns and COVID-19 transmission in the USA: A mathematical modelling study. Lancet Infect. Dis. 2020, 3099, 1–8. [Google Scholar] [CrossRef]
  76. Zhou, Y.; Xu, R.; Hu, D.; Yue, Y.; Li, Q.; Xia, J. Effects of human mobility restrictions on the spread of COVID-19 in Shenzhen, China: A modelling study using mobile phone data. Lancet Digit. Health 2020, 2, e417–e424. [Google Scholar] [CrossRef]
  77. Pier, M.M.; Pasick, L.J.; Benito, D.A.; Alnouri, G.; Sataloff, R.T. Otolaryngology-related Google Search trends during the COVID-19 pandemic. Am. J. Otolaryngol. 2020, 41, 102615. [Google Scholar] [CrossRef] [PubMed]
  78. Kurian, S.J.; ur Rehman Bhatti, A.; Alvi, M.A.; Ting, H.H.; Storlie, C.; Wilson, P.M.; Shah, N.D.; Liu, H.; Bydon, M. Correlations Between COVID-19 Cases and Google Trends Data in the United States: A State by State Analysis. Mayo Clin. Proc. 2020. [Google Scholar] [CrossRef] [PubMed]
  79. Husnayain, A.; Fuad, A.; Su, E.C.-Y. Applications of Google Search Trends for risk communication in infectious disease management: A case study of the COVID-19 outbreak in Taiwan. Int. J. Infect. Dis. 2020, 95, 221–223. [Google Scholar] [CrossRef]
  80. Effenberger, M.; Kronbichler, A.; Shin, J.I.; Mayer, G.; Tilg, H.; Perco, P. Association of the COVID-19 pandemic with Internet Search Volumes: A Google TrendsTM Analysis. Int. J. Infect. Dis. 2020, 95, 192–197. [Google Scholar] [CrossRef]
  81. Lauer, S.A.; Grantz, K.H.; Bi, Q.; Jones, F.K.; Zheng, Q.; Meredith, H.R.; Azman, A.S.; Reich, N.G.; Lessler, J. The incubation period of coronavirus disease 2019 (COVID-19) from publicly reported confirmed cases: Estimation and application. Ann. Intern. Med. 2020, 172, 577–582. [Google Scholar] [CrossRef] [Green Version]
  82. Chan, J.F.W.; Yuan, S.; Kok, K.H.; To, K.K.W.; Chu, H.; Yang, J.; Xing, F.; Liu, J.; Yip, C.C.Y.; Poon, R.W.S.; et al. A familial cluster of pneumonia associated with the 2019 novel coronavirus indicating person-to-person transmission: A study of a family cluster. Lancet 2020, 395, 514–523. [Google Scholar] [CrossRef] [Green Version]
  83. World Health Organisation. Report of the WHO-China Joint Mission on Coronavirus Disease 2019 (COVID-19); World Health Organisation: Geneva, Switzerland, 2020; Volume 2019. [Google Scholar]
  84. Wang, L.; Li, J.; Guo, S.; Xie, N.; Yao, L.; Cao, Y.; Day, S.W.; Howard, S.C.; Graff, J.C.; Gu, T.; et al. Real-time estimation and prediction of mortality caused by COVID-19 with patient information based algorithm. Sci. Total Environ. 2020, 727, 138394. [Google Scholar] [CrossRef]
  85. Emrouznejad, A.; Shale, E. A combined neural network and DEA for measuring efficiency of large scale datasets. Comput. Ind. Eng. 2009, 56, 249–254. [Google Scholar] [CrossRef]
  86. Shih, D.H.; Yen, D.C.; Lin, H.C.; Shih, M.H. An implementation and evaluation of recommender systems for traveling abroad. Expert Syst. Appl. 2011, 38, 15344–15355. [Google Scholar] [CrossRef]
  87. Ghiassi, M.; Burnley, C. Measuring effectiveness of a dynamic artificial neural network algorithm for classification problems. Expert Syst. Appl. 2010, 37, 3118–3128. [Google Scholar] [CrossRef]
  88. Chang, C.L.; Chen, C.H. Applying decision tree and neural network to increase quality of dermatologic diagnosis. Expert Syst. Appl. 2009, 36, 4035–4041. [Google Scholar] [CrossRef]
  89. Chen, H.Y.; Chuang, C.H.; Yang, Y.J.; Wu, T.P. Exploring the risk factors of preterm birth using data mining. Expert Syst. Appl. 2011, 38, 5384–5387. [Google Scholar] [CrossRef]
Figure 1. Methodology framework.
Figure 1. Methodology framework.
Sustainability 12 10124 g001
Figure 2. DEA experiment result calculated from the predicted projection in the sustainability study of the BRI, with the blue diamonds representing the back-propagation neural network (BPNN) and the red lines representing automated machine learning (AutoML): (a) efficiency scores of SDG7; (b) efficiency scores of SDG8; (c) efficiency scores of SDG9; (d) distance to the efficient frontiers of SDG7, SDG8, and SDG9.
Figure 2. DEA experiment result calculated from the predicted projection in the sustainability study of the BRI, with the blue diamonds representing the back-propagation neural network (BPNN) and the red lines representing automated machine learning (AutoML): (a) efficiency scores of SDG7; (b) efficiency scores of SDG8; (c) efficiency scores of SDG9; (d) distance to the efficient frontiers of SDG7, SDG8, and SDG9.
Sustainability 12 10124 g002
Figure 3. DEA experiment results calculated from the predicted projection in the COVID-19 pandemic study, with the blue diamonds representing the BPNN and the red lines representing AutoML: (a) efficiency scores of SDG3; (b) efficiency scores of SDG11; (c) distance to efficient frontiers of SDG3 and SDG11.
Figure 3. DEA experiment results calculated from the predicted projection in the COVID-19 pandemic study, with the blue diamonds representing the BPNN and the red lines representing AutoML: (a) efficiency scores of SDG3; (b) efficiency scores of SDG11; (c) distance to efficient frontiers of SDG3 and SDG11.
Sustainability 12 10124 g003
Table 1. Description of the indicators in SDG7, SDG8, and SDG9.
Table 1. Description of the indicators in SDG7, SDG8, and SDG9.
IndicatorExtracted DenominatorExtracted Numerator
(7.1.1 *) Proportion of population with access to electricityPopulationElectricity access
(7.1.2 *) Proportion of population with primary reliance on clean fuels and technologyPopulationClean fuel access
(7.2.1 *) Renewable energy share in the total final energy consumptionTotal energy consumptionRenewable energy
(7.3.1 *) Energy intensity measured in terms of primary energy and GDPGDPTotal energy consumption
(7.a.1) International financial flows to developing countries in support of clean energy research and development and renewable energy production, including in hybrid systemsN/AClean energy finance flows
(7.b.1) Installed renewable energy-generating capacity in developing countries (in watts per capita)PopulationElectricity generated by renewable resources
(8.1.1 *) Annual growth rate of real GDP per capitaPopulationGDP
(8.2.1 *) Annual growth rate of real GDP per employed personEmploymentGDP
(8.3.1) Proportion of informal employment in total employment, by sector and sexEmploymentInformal employment
(8.4.1) Material footprint, material footprint per capita, and material footprint per GDPGDPMaterial footprint
(8.4.2 *) Domestic material consumption, domestic material consumption per capita, and domestic material consumption per GDPPopulationDomestic material consumption
(8.5.1) Average hourly earnings of female and male employees, by occupation, age and persons with disabilitiesTotal working hoursTotal income
(8.5.2 *) Unemployment rate, by sex, age and persons with disabilitiesLabor forceUnemployment
(8.6.1) Proportion of youth (aged 15–24 years) not in education, employment or trainingYouth populationUneducated or unemployed youth
(8.7.1) Proportion and number of children aged 5–17 years engaged in child labor, by sex and ageChild populationChild employment
(8.8.1) Frequency rates of fatal and non-fatal occupational injuries, by sex and migrant statusEmploymentNon-fatal occupational injuries and illnesses
(8.8.2) Level of national compliance with labor rights (freedom of association and collective bargaining) based on International Labour Organization (ILO) textual sources and national legislation, by sex and migrant statusN/ANational compliance with labor rights
(8.9.1) Tourism direct GDP as a proportion of total GDP and in growth rateGDPTourism value added
(8.10.1 *) (a) Number of commercial bank branches per 100,000 adults and (b) number of automated teller machines (ATMs) per 100,000 adultsAdult populationNumber of commercial bank branches
(8.10.2) Proportion of adults (15 years and older) with an account at a bank or other financial institution or with a mobile-money-service providerAdult populationAdults with an account at a financial institution
(8.a.1) Aid for Trade commitments and disbursementsN/ACommitments for aid for trade
(8.b.1) Existence of a developed and operationalized national strategy for youth employment, as a distinct strategy or as part of a national employment strategyN/ANumber of national strategies for youth employment
(9.1.1) Proportion of the rural population who live within 2 km of an all-season roadTotal rural populationRural population who live within 2 km of an all-season road
(9.1.2 *) Passenger and freight volumes, by mode of transportPopulationAir, road or railways passengers or goods transferred
(9.2.1 *) Manufacturing value added as a proportion of GDP and per capitaPopulationManufacturing value added
(9.2.2 *) Manufacturing employment as a proportion of total employmentEmploymentManufacturing employment
(9.3.1) Proportion of small-scale industries in total industry value addedIndustry value addedSmall-scale industries value added
(9.3.2) Proportion of small-scale industries with a loan or line of creditNumber of small-scale industriesSmall-scale industries with loan or line of credit
(9.4.1 *) CO2 emission per unit of value addedGross value addedCO2 emissions
(9.5.1) Research and development expenditure as a proportion of GDPGDPResearch and development expenditure
(9.5.2) Researchers (in full-time equivalent) per million inhabitantsPopulationResearchers
(9.a.1) Total official international support (official development assistance plus other official flows) to infrastructureN/ATotal official flows for infrastructure
(9.b.1) Proportion of medium and high-tech industry value added in total value addedGross value addedMedium and high-tech industry value added
(9.c.1 *) Proportion of population covered by a mobile network, by technologyPopulationMobile phone subscriptions or Internet users
* Measured indicators based on the data available.
Table 2. Descriptive statistics of the variables in the sustainability study of the Belt and Road Initiative (BRI).
Table 2. Descriptive statistics of the variables in the sustainability study of the Belt and Road Initiative (BRI).
1Energy consumptionU.S. EIAktoe7Input56,703217,2881,995,057399
2PopulationWorld Bankpeople7, 8, 9Input48,895,906155,696,3801,386,395,000416,268
3Labor forceWorld Bankpeople8Input23,336,08388,036,169787,399,317180,756
4Adult populationWorld Bankpeople8Input36,366,959126,249,2671,137,712,323354,703
5Total employmentWorld Bankpeople8, 9Input22,120,92184,072,312752,753,746169,227
6Gross value addedUNconstant 2015 US$9Input300,768,954,7261,180,630,638,27812,600,000,000,0002,910,456,947
7Access to clean fuelsWorld Bankpeople7Output27,966,63290,083,247816,996,879278,423
8Access to electricityWorld Bankpeople7Output43,190,262154,469,2651,386,395,000416,268
9Renewable energy ConsumptionU.S. EIAktoe7Output646918,439127,5441
10GDPUNconstant 2015 US$7, 8Output313,461,796,8661,187,748,354,77712,600,000,000,0003,299,234,529
11Domestic material consumptionWorld BankKG8Output659,176,464,0993,493,224,391,62234,800,000,000,0006,756,860,686
12UnemploymentWorld Bankpeople8Output (U)1,215,0274,032,20436,101,08310,040
13Commercial bank branchesWorld Bankbranches8Output442611,90299,93720
14Air freightWorld Bankmillion ton-km9Output904300023,3240.0001
15Air passengersWorld Bankpeople9Output14,766,27748,542,713551,234,509899
16Manufacturing employmentWorld Bankpeople9Output5,350,64724,121,994219,519,43928,306
17Manufacturing value addedUNconstant 2015 US$9Output67,113,438,034345,141,666,4223,660,000,000,000189,954,006
18CO2 emissionsU.S. EIAktCO29Output (U)215,5101,015,7219,259,2591,354
19Internet usersWorld Bankpeople9Output19,199,11073,370,618752,812,485113,488
20Mobile phone subscriptionsWorld Bankpeople9Output49,323,520146,141,8511,469,882,500478,860
Table 3. Correlation matrix of the variables in the sustainability study of the BRI.
Table 3. Correlation matrix of the variables in the sustainability study of the BRI.
Sustainability 12 10124 i001
⁺⁺ Correlation is significant at the 0.01 level (2-tailed).
Table 4. Descriptive statistics of the variables in the COVID-19 pandemic study.
Table 4. Descriptive statistics of the variables in the COVID-19 pandemic study.
1Active casesJohns Hopkins Universitycases3Input26,611122,3611,171,80523
2Containment and health indexUniversity of Oxfordscore3Input751310025
3Days from the first confirmed caseJohns Hopkins Universitydays11Input732213128
4New COVID-19 testsUniversity of Oxfordcases11Input103,594284,7202,651,203839
5Trending inquiries about “covid”Googlescore11Input5218947
6Government response indexUniversity of Oxfordscore11Input73129621
7Newly recovered cases at week t + 3Johns Hopkins Universitycases3Output490711,70692,8632
8New deaths at week t + 3Johns Hopkins Universitypeople3Output (U)311114312,4520
9Active cases at week t + 3Johns Hopkins Universitycases3Output (U)34,848156,5401,402,5255
10Daily visits to workplacesGooglescore11Output4116801
11Daily visits to residentialGooglescore11Output188412
12Daily visits to public placesGooglescore11Output53221001
13Newly confirmed COVID-19 cases at week t + 1Johns Hopkins Universitycases3Input714023,867210,5120
11Output (U)
Table 5. Correlation matrix of the variables in the COVID-19 pandemic study.
Table 5. Correlation matrix of the variables in the COVID-19 pandemic study.
Sustainability 12 10124 i002
⁺⁺ Correlation is significant at the 0.01 level (2-tailed), ⁺ Correlation is significant at the 0.05 level (2-tailed), ⁻ Correlation is not significant.
Table 6. DEA experiment summary in the sustainability study of the BRI: (a) yearly efficiency score; (b) number of decision-making units (DMUs) in each efficient tier.
Table 6. DEA experiment summary in the sustainability study of the BRI: (a) yearly efficiency score; (b) number of decision-making units (DMUs) in each efficient tier.
2011201220132014201520162017 2011201220132014201520162017
SD0.0880.0820.0800.0800.0800.0850.086 ET224222318181613
Max1111111 ET34555665
SDG8Mean0.5550.5550.5590.5620.5560.5600.570 ET215121616161315
SD0.3000.2950.2950.2950.2890.2890.288 ET314181318162117
Max1111111 ET419171814131213
Min0.1280.1230.1240.1300.1310.1310.132 ET510131314141413
SDG9Mean0.9360.9330.9390.9360.9460.9430.941 ET69775667
Max1111111 ET233312832292829
Min0.3860.3810.3640.3750.3360.3320.330 ET34332224
Table 7. Performance comparison of prediction methods in the sustainability study of the BRI: (a) regression evaluation metrics in the efficiency score prediction; (b) precision and accuracy in the efficient tier classification; (c) regression evaluation metrics in the frontier projection prediction.
Table 7. Performance comparison of prediction methods in the sustainability study of the BRI: (a) regression evaluation metrics in the efficiency score prediction; (b) precision and accuracy in the efficient tier classification; (c) regression evaluation metrics in the frontier projection prediction.
SD0.0000.0810.2460.2730.0160.099ET2 11/165/911/139/1925/33
Max0.9781.0001.0001.0000.9981.000ET3 4/513/4915/21 1/1
Min0.9740.5140.1300.1420.9260.381ET4 1/1011/12
RMSE0.0860.0260.1630.0470.1160.052ET5 13/14
PRED(0.20)0.9500.9880.5000.9750.9250.975ET6 6/6
Projected OutputsRMSEPRED
SDG7Clean fuels29,508,6710.0880.0250.0000.9034,571,9570.9000.8500.6131.000
Unemployment8.96 × 10−60.3000.1500.0750.8665.50 × 10−60.7630.5750.4130.948
Bank branches55160.3000.1880.1380.85412180.8500.6500.2750.995
SDG9Air freight16090.0630.0250.0130.8789250.3630.1380.0500.979
Air passengers23,297,8740.0880.0250.0250.96717,770,1310.3880.1500.0130.994
Manufacturing employment2,731,6180.2500.1380.0380.991379,4690.8000.5880.4131.000
Manufacturing value added94,784,937,9490.1500.0630.0250.985345,878,375,4490.0250.0000.0000.997
CO2 emissions0.00010.3250.1500.0880.8514.26 × 10−50.5880.3750.3000.922
Internet users19,810,6100.1500.0880.0380.97617,691,6650.5630.1750.0880.999
Mobile phone36,174,1090.1750.0750.0500.98832,220,5960.8750.5750.3880.996
Table 8. DEA experiment summary in the COVID-19 pandemic study: (a) weekly efficiency score; (b) number of DMUs in each efficient tier.
Table 8. DEA experiment summary in the COVID-19 pandemic study: (a) weekly efficiency score; (b) number of DMUs in each efficient tier.
wk 16wk 17wk 18wk 19wk 20wk 21 wk 16wk 17wk 18wk 19wk 20wk 21
SD0.3010.2620.2980.3190.2960.288 ET2181713141312
Max1.0001.0001.0001.0001.0001.000 ET3141415121310
Min0.1050.1370.0880.0520.0630.088 ET4911712128
SD0.1650.1690.1680.1760.2130.209 ET2272625242519
Max1.0001.0001.0001.0001.0001.000 ET3151617181619
Min0.2900.3680.3950.3850.1710.258 ET4765455
Table 9. Performance comparison of prediction methods in the COVID-19 pandemic study: (a) regression evaluation metrics in the efficiency score prediction; (b) precision and accuracy in the efficient tier classification; (c) regression evaluation metrics in the frontier projection prediction.
Table 9. Performance comparison of prediction methods in the COVID-19 pandemic study: (a) regression evaluation metrics in the efficiency score prediction; (b) precision and accuracy in the efficient tier classification; (c) regression evaluation metrics in the frontier projection prediction.
Projected OutputsRMSEPRED
SDG3Recovered at week t + 368010.1150.0490.0330.91943030.2300.1310.0330.932
Deaths at week t + 30.2540.2950.1640.0160.6170.1280.4100.3610.2300.917
Active at week t + 30.0720.0160.0160.0160.0710.0480.0660.0490.0160.879
Confirmed at week t + 10.1360.0160.0160.0000.3920.1120.0330.0160.0160.551
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Singpai, B.; Wu, D. Using a DEA–AutoML Approach to Track SDG Achievements. Sustainability 2020, 12, 10124.

AMA Style

Singpai B, Wu D. Using a DEA–AutoML Approach to Track SDG Achievements. Sustainability. 2020; 12(23):10124.

Chicago/Turabian Style

Singpai, Bodin, and Desheng Wu. 2020. "Using a DEA–AutoML Approach to Track SDG Achievements" Sustainability 12, no. 23: 10124.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop