Quantifying Electricity Supply Resilience of Countries with Robust Efficiency Analysis

Abstract

The interest in studying energy systems’ resilience is increasing due to a rising awareness of the importance of having a secure energy supply. This growing trend is a result of a series of recent disruptions, among others also affecting electricity systems. Therefore, it is of crucial importance for policymakers to determine whether their country has a resilient electricity supply. Starting from a set of 12 indicators, this paper uses data envelopment analysis (DEA) to comprehensively evaluate the electricity supply resilience of 140 countries worldwide. Two DEA models are applied: (1) the original ratio-based Charnes, Cooper, and Rhodes (CCR) model and (2) a novel hybrid framework for robust efficiency analysis incorporating linear programming and Monte Carlo simulations. Results show that the CCR model deems 31 countries as efficient and hence lacks the capability to differentiate them. Furthermore, the CCR model considers only the best weight vectors for each country, which are not necessarily representative of the overall performance of the countries. The robustness analysis explores these limitations and identifies South Korea, Singapore and Canada as the most resilient countries. Finally, country analyses are conducted, where Singapore’s and Japan’s performances and improvement potentials are discussed.

1. Introduction

Electricity is a crucial commodity to foster the economic development and well-being of a country [1]. Governments are increasingly aware of the need to improve the energy efficiency of electricity production (i.e., decrease the total amount of energy required to produce the desired quantity of electricity), as it leads to better supply security and reduced greenhouse gas emissions. Even though recently the global average efficiency slowly improved [1], major electricity supply disruptions still happen (e.g., the 2012 India blackout [2] or the 2015 Turkey blackout [3]). The financial, social and environmental consequences of such disruptions can cause great damage to the economy of a country, its government and citizens [4,5]. Resilience aims at minimizing the impact of these adverse consequences by defining pre- and post-event strategies, making outages less likely or smaller in extent [6,7].

A resilient electricity supply is fundamental to guarantee a well-functioning modern society [8]. In this regard, one of the key interests of policymakers is to assess how their country performs compared to others. This kind of international comparison supports them in identifying improvement potentials and quantifying achievements or progress towards predefined objectives and targets [9]. Many international country performance assessments are indicator-based [10], because indicators are suitable to model multi-dimensional problems [11]. Countries are commonly ranked based on the indicators only or on aggregated measures, sometimes also called indices or composite indicators, that combine the individual indicators [12]. Within the energy sector, a wide range of operational research methods are used to build such indices [13]. Examples include the analytical hierarchy process (AHP) [14,15], technique for order preference by similarity to ideal solution (TOPSIS) [16], outranking methods such as the preference ranking organization method for enrichment evaluation (PROMETHEE) [17] or elimination et choix traduisant la réalité (ELECTRE) [18], weighted averages [19], multi-attribute value/utility theory (MAVT/MAUT) [20] and data envelopment analysis (DEA) [21].

In the context of electricity supply resilience, comparative country evaluations and rankings are missing [22]. Hence, building upon the framework proposed by Gasser et al. (2017) [23] and Gasser et al. (2020) [24] to define a set of 12 indicators that cover resilience holistically, this paper assesses the electricity supply resilience of 140 countries following a security of supply perspective. Due to the fact that the indicators have positive and negative preference orders, and the requirement to not involve preferences from decision-makers, DEA, a family of non-parametric methods to derive efficiencies of decision-making units (DMUs) [25], is particularly suitable to rank the countries. In fact, for DEA, the indicator weights are endogenously determined (directly calculated from the performance matrix itself) [26]. The final scores of the DMUs are commonly called efficiencies, as they represent the ratio of the weighted output indicators’ performances to the weighted input indicators performances. Hence, a DMU that is deemed efficient is also resilient, as the indicator set represents electricity supply resilience. DEA efficiency and resilience have thus equal meanings in this context and the terms can be used interchangeably.

The DEA methodology hereby developed allows the following important questions to be answered, which are relevant to (inter-) governmental agencies as well as research institutions:

• What are the best performing (i.e. most resilient) countries and what are the reasons for this achievement (see Section 4.1)?
• Why are some countries inefficient, how can they improve their scores and which are their benchmarks (see Section 4.2)?
• How robust is the performance of the countries (see Section 4.3)?
• What is the univocal ranking of the countries (see Section 4.4)?
• How well does a country perform in comparison to another one (see Section 4.5)?
• How does the performance of countries vary according to changes in selected indicators (see Section 4.6)?

The current paper is organized as follows. In Section 2, a detailed literature review about energy-related country comparisons using DEA is provided, which leads to the formulation of the research gaps. Section 3 describes the case study and the methodology. The latter includes the original ratio-based Charnes, Cooper, and Rhodes (CCR) DEA model [27] and a hybrid framework for robust efficiency analysis incorporating linear programming and Monte Carlo simulations [28]. The CCR model was considered because it is the most commonly used DEA model. The hybrid framework explores the limitations of the CCR model by performing a robustness assessment through selecting random weight vectors obtained via a Hit-And-Run algorithm. To the authors’ best knowledge, the present study represents the first application of such an analysis to a country ranking. In Section 4, comparative results answering the research questions are presented and discussed. Furthermore, improvement potentials for Singapore and Japan are analyzed. Section 5 provides the main conclusions of the study.

2. Literature Review—Energy-Related Country Comparisons with Data Envelopment Analysis

The first notions of DEA can be traced back to early publications by Farrell (1957) [29] and Brockhoff (1970) [30], but the seminal paper by Charnes et al. (1978) provided the first application of linear programming to estimate an efficiency frontier [27]. A comprehensive overview of DEA in past decades is given in a review by Liu et al. (2013) [31]. The popularity of DEA is also reflected by its numerous applications in different fields such as the evaluation of socio-economic, environmental and productivity performance [31]. There are also diverse applications in the energy sector [32,33,34,35,36,37], as shown in

Table 1. Literature review of publications using data envelopment analysis (DEA) to analyze energy-related topics.

Source	Scope	Geographical Coverage	Number of DMUs	Inputs	Outputs
Apergis et al. (2015) [34]	Energy efficiency	Organisation for Economic Co-operation and Development (OECD)	20	Labor; energy consumption; capital stock	GDP; CO2 emissions
Bampatsou et al. (2013) [41]	Capacity of an economy to produce a higher GDP given fixed energy inputs	European Union	15	Fossil and non-fossil fuel energy consumption	GDP
Cai et al. (2019) [42]	Carbon emissions efficiency in Chinese cities	China	280	Labor; capital; energy and water consumption	GDP; CO2 emissions
Camarero et al. (2013) [43]	Impact of CO2, SO2 and NOx air-pollutants on the environment	OECD	22	CO2, SO2 and NOx emissions	GDP
Chang (2014) [44]	Energy intensity	European Union	27	Capital stock; labor force; energy consumption	GDP
Cui et al. (2014) [45]	Energy efficiency	Global	9	Employees; energy consumption; energy services	CO2 emissions; industrial profit
Gómez-Calvet et al. (2016) [46]	Abatement opportunities of CO2, SO2 and NOx air-pollutants	European Union	27	CO2, SO2 and NOx emissions	GDP
Halkos and Petrou (2019) [47]	Energy recovery from waste	European Union	28	Energy consumption; labor; capital; population density	GDP; Greenhouse Gases (GHG), NOx and SOx emissions
Hsieh et al. (2019) [48]	Environmental assessment	European Union	28	Labor; capital; energy consumption	GHG and SOx emissions; GDP
Hu and Kao (2007) [49]	Energy-saving target ratio	Asia-Pacific	17	Energy; labor; capital	GDP
Li and Wang (2014) [40]	Environmental efficiency	Global	95	Capital stock; labor force; energy consumption	GDP; CO2 emissions
Liou and Wu (2011) [50]	Effect of economic development on energy use efficiency and CO2 emissions	Global	57	Labor; capital; energy consumption	GDP; CO2 emissions
Pang et al. (2015) [39]	Clean energy use and total-factor efficiencies	Global	87	Capital stock; labor force; energy consumption	GDP; CO2 emissions
Ramanathan (2005) [51]	Energy consumption and carbon dioxide emissions	Middle East and North Africa	17	Fossil fuel energy comsumption; carbon emissions	Non-fossil fuel energy consumption; GDP
Robaina-Alves et al. (2015) [52]	Resource and environment efficiency	Europe	26	Capital stock; labor force; energy consumption	GDP; GHG emissions
Song et al. (2013) [53]	Energy efficiency	Brazil, Russia, India, China and South Africa (BRICS)	5	Energy consumption; economically active population; capital	GDP
Wang et al. (2019) [54]	Relation between CO2 emissions and GDP	Global	25	Gross capital formation; labor force; energy consumption	GDP; CO2 emissions
Wang (2015) [38]	Energy systems’ sustainability	Global	109	CO2 emissions; energy intensity	Share of renewables
Wegener and Amin (2019) [37]	Greenhouse gas emissions minimization	Canada and USA	23	Wells; employees; capital expenditures; total assets	GHG emissions; production
Zeng et al. (2017) [55]	Economic; energy supply; environmental	Baltic States	3	Energy intensity; energy weight in HICP; electricity prices; import dependency; diversification of import sources; diversification of energy mix	Energy balance of trade; share of renewables; carbon intensity
Zhang et al. (2011) [56]	Total-factor energy efficiency	Developing countries	23	Labor force; energy consumption; capital stock	GDP
Zhou and Ang (2008) [57]	Energy efficiency performance	OECD	21	Capital stock; labor force; consumption of coal, oil, gas and other	GDP; CO2 emissions
Zhou et al. (2014) [58]	Energy efficiency of transport sector	China	30	Labor; consumption of coal, gasoline, kerosene, diesel oil, electricity and other	Passenger kilometers; tonne-kilometers; CO2 emissions
Zhou et al. (2016) [59]	Energy efficiency	Global	32	Capital stock; labor force; fossil and non-fossil energy consumption	GDP; CO2 emissions
This study	Electricity supply resilience	Global	140	System Average Interruption Duration Index (SAIDI); accident risks; import dependence; average outage time	Control of corruption; political stability and absence of violence/terrorism; mix diversity; equivalent availability factor; GDP per capita; insurance penetration; government effectiveness; ease of doing business

.

A detailed, global study by Wang (2015) compared the sustainability of the energy systems in 109 countries [38]. However, this study was based on only three, rather generic, indicators from the World Bank: (1) the CO₂ emissions intensity in kg per 2005 USD of Gross Domestic Product (GDP), (2) the energy intensity in kg of oil equivalent per GDP (constant 2005 Purchasing Power Parity (PPP)), and (3) the share of electricity produced from renewables in %. Overall, results demonstrated that the energy systems in high-income countries have a better sustainability performance.

Another worldwide study analyzes emission reductions, energy conservation and economic output of 87 countries based on the GDP, the capital stock, the labor force, the energy consumption and the overall CO₂ emissions [39]. European countries were found to perform better than non-European ones. Li and Wang (2014) used the same indicators and applied them to 95 countries [40]. They identified tremendous gaps between countries according to income groups, and similarly high-income countries were ranked top.

Gómez-Calvet et al. (2016) studied the opportunities for abatement of CO₂, SO₂ and NO_x air-pollutants by looking at the evolution of environmental performance over time [46]. They found that the environmental efficiency of European countries had improved over the period 1993–2010. The same three air-pollutants were also analyzed by Camarero et al. (2013) and they reached similar conclusions, with the exception that the eco-efficiency of NO_x emissions did not improve [43]. Regarding energy efficiency improvements, Wang et al. (2019) compared the CO₂ emissions in relation to GDP growth from 25 countries and found that India and China are the two worst countries in terms of energy efficiency [54]. A similar study focusing on energy recovery from waste for European Union (EU) member states was produced by Halkos and Petrou (2019) [47]. Furthermore, Robaina-Alves et al. (2015) derived the efficiencies of European countries based on the maximization of the ratio between the GDP (desired output) and GHG emissions (undesired output) [52]. Their key finding was that since the ratification of the Kyoto Protocol, countries have taken steps to reduce emissions and this is reflected in the evolution of the eco-efficiency level of some countries.

Several other publications address energy-efficiency issues in general. Cui et al. (2014) found that energy efficiency is mostly driven by investments into energy technologies research and tax exemptions for technology companies [45]. Based on the analysis of 28 European countries, Hsieh et al. (2019) recommend that “the EU’s strategy for environmental energy improvement should be to pay attention to the benefits of renewable energy utilization, reducing GHG emissions, and enhancing the development of renewable energy utilization to help achieve the goal of lower GHG emissions” [48]. Apergis et al. (2015) show that capital-intensive countries are more energy efficient than labor-intensive ones [34]. Zhang et al. (2011) compare 23 developing countries according to their total-factor energy efficiency, which is defined as the ratio between the targeted energy input and the actual energy input [56]. Similarly, Chang (2014) studies the difference between the targeted and actual energy intensities of 27 EU member countries in order to make conclusions about the potentials for improvement [44]. With a data set of 57 countries, Liou and Wu (2011) found that economic development is interrelated with energy use efficiency and CO₂ emission control [50]. Furthermore, Zhou et al. (2014) used the DEA to rank 30 administrative regions of China according to the energy efficiency of their transport sector [58]. Results show that the Eastern area generally performs better than the Central and Western areas. Song et al. (2013) found that the economies of Brazil, Russia, India, China and South Africa (BRICS) have low energy efficiencies but the trend is increasing quickly [53]. Cai et al. (2019) quantify the carbon emissions of 280 Chinese cities to find that only nine of them are efficient [42]. Results show that coastal regions are performing better than central and western regions. Finally, Zhou et al. (2016) develop novel energy-efficiency measures that seem to handle undesirable outputs better and are more effective at identifying inefficient production behaviour [60]. Their data set consists of 32 countries.

Economic efficiency has also been analyzed as the capacity of an economy to produce higher GDP for a given total energy input. In particular, Bampatsou et al. (2013) study the effect of different energy mixes and find that adding nuclear energy into a country’s energy mix affects negatively its economic efficiency, due to fewer efforts invested in energy saving and conservation [41].

In summary, as seen in Table 1, most of these studies deal with energy or eco-efficiency. Out of the 25 studies analyzed (including the present one), 15 of them (60%) use the same set of inputs: labor force, capital stock and energy consumption [34,39,40,42,44,47,48,49,50,52,53,54,56,57,59]. Furthermore, out of these 15 studies, four consider only the GDP as an output [44,49,53,56], while the rest consider the GDP as a desirable output and GHG emissions as an undesirable output [34,39,40,42,47,48,50,52,54,57,59]. Within DEA country comparisons, efficiency is, therefore, usually measured as a minimization of the labor force, capital stock, and energy consumption (inputs) in order to maximize the GDP (desirable output) and minimize the GHG emissions (undesirable output). Further studies use only a subset of these indicators, such as only the energy consumption as an input and the GDP as an output [41], or only the GHG emissions as an input and the GDP as an output [43,46]. Overall, only five studies do not directly use the GDP as an output [37,38,45,55,58]. Three studies differentiate between fossil fuel and non-fossil fuel energy consumption [38,51,55]. The fossil fuel energy consumption would be classified as an input (hence to be minimized), while the non-fossil fuel or renewable sources are classified as outputs (hence to be maximized).

Finally, to the authors’ best knowledge, no study using DEA to comprehensively rank countries with regard to their electricity supply resilience performance has been published. This implies that a broader set of indicators needs to be considered, including grid reliability, accident risks, diversity of generation and availability of production technologies. Furthermore, the present study uses DEA models to assess ranking robustness, and to explore scenario-based improvement strategies. Therefore, the research gaps filled by the present study are:

• The use of DEA models to develop rankings that represent the electricity supply resilience of countries.
• The development of novel DEA algorithms to better understand why some countries are efficient and others are not. These new models are applied for the first time in a real-life case study.
• The examination of ranking stability by means of robustness analysis.
• The study of country-specific improvement strategies from an optimization point of view.

3. Case Study Description and Methodology

In this section, first the general scope of the case study and the indicator set selection process are described. Second, the quantification of the data and preparation for DEA are presented. Finally, the DEA concept, its notation and formulas are explained.

3.1. Indicator Set Selection, Quantification and Data Set Preparation

The first step was to conduct a literature review in order to identify relevant indicators. As shown by Gasser et al. (2017) [23], most of the indicators used in the security of electricity supply studies are related to resilience too. Hence, the abundance of security of supply studies is a promising starting point (e.g., [10,61,62,63,64,65]). On top of these, further ones related to resilience were considered as well (e.g., [8,66]). This resulted in an extensive list of resilience-related indicators. Subsequently, all of these were assessed according to four quality criteria [67]: (1) their relevance to resilience, (2) the credibility of the data source, (3) the availability of the data and (4) the comparability of the data between countries. The more countries in the list, the more likely data becomes unavailable or incomparable. Thus, as the present case study is about the electricity sector, the limiting data was the electricity production by fuel type. The most comprehensive data for this comes from the International Energy Agency (IEA) and is available for 140 countries worldwide [68]. Therefore, the final set of 140 countries consists of 12 indicators fulfilling the four assessment criteria (see

Table 2. Performance matrix with selected countries and 12 indicators. The table including all 140 countries is available in Table S1 in the electronic supplementary information (ESI). For the preference order of the values, an upward pointing arrow indicates better performance for higher values (positive preference order), whereas a downward pointing arrow indicates better performance for lower values (negative preference order).

0-Country	1-SAIDI	2-Accident Risks	3-Control of Corruption	4-Political Stability and Absence of Violence/Terrorism	5-Electricity Mix Diversity	6-Electricity Import Dependence	7-Equivalent Availability Factor	8-GDP per Capita	9-Insurance Penetration	10-Government Effectiveness	11-Average Outage Time	12-Ease of Doing Business
Measurement Unit	Hours per Customer per Year	Fatalities/GWeyr	Percentile Rank	Percentile Rank	Normalized Shannon Index	Ratio Consumption/Production	%	2010 USD per Capita	% of GDP	Percentile Rank	Hours	Distance to Frontier (100 = Best, 0 = Worst)
Preference Order of the Values	↓	↓	↑	↑	↑	↓	↑	↑	↑	↑	↓	↑
Albania	111.8	7.03	38.46	58.10	0.00	1.02	39%	4543	0.90%	53.85	2.15	66.06
Cambodia	34.2	3.53	12.02	49.52	0.39	1.16	64%	1021	0.30%	25.00	1.40	52.34
Canada	1.0	0.02	95.19	95.24	0.60	0.81	56%	50,108	4.23%	94.71	0.71	80.34
China	1.3	9.65	48.56	26.19	0.42	0.95	74%	6498	1.63%	68.27	3.25	63.43
Congo, Dem. Rep.	92.6	7.01	8.17	4.76	0.01	0.81	39%	384	0.30%	2.88	2.03	34.54
Denmark	0.4	0.04	98.08	76.67	0.57	1.14	54%	60,037	2.69%	97.60	0.80	83.91
Eritrea	92.6	0.95	7.69	19.05	0.01	0.86	85%	528	0.40%	4.81	2.03	26.16
Finland	0.2	0.03	99.52	87.14	0.71	1.20	73%	45,208	2.18%	96.15	0.50	80.34
France	0.1	0.01	88.94	51.43	0.39	0.82	80%	41,768	3.09%	88.46	1.00	75.19
Germany	0.3	0.07	93.27	68.57	0.74	0.89	71%	45,252	3.36%	93.75	1.50	78.52
Haiti	92.6	1.44	10.10	22.38	0.12	0.41	81%	728	1.00%	0.96	2.03	38.63
Iceland	0.5	0.00	95.67	95.71	0.25	0.97	53%	45,939	2.20%	90.87	1.00	78.33
Iraq	2352.0	0.97	4.81	2.86	0.31	0.64	83%	5120	1.63%	9.62	2.33	44.56
Italy	0.7	0.05	57.69	58.57	0.76	1.09	68%	33,912	2.06%	69.23	0.28	71.16
Japan	0.4	0.08	91.35	89.05	0.65	0.96	78%	47,142	2.55%	95.19	4.00	75.36
Kenya	188.5	2.88	13.94	9.52	0.46	0.81	69%	1134	1.88%	43.27	11.42	54.19
Libya	1883.4	0.50	0.96	3.33	0.30	0.28	85%	5447	0.40%	1.92	3.11	32.84
Luxembourg	0.2	0.02	97.12	98.10	0.49	2.97	54%	108,965	1.79%	93.27	1.00	68.77
Myanmar	92.6	4.20	20.67	10.48	0.33	0.84	58%	1643	0.10%	10.10	2.03	38.68
Nepal	92.6	7.02	32.21	14.29	0.01	1.12	39%	690	1.63%	12.98	2.03	59.99
Netherlands	0.3	0.08	94.71	80.48	0.57	1.03	80%	51,285	8.35%	97.12	1.00	75.21
New Zealand	2.4	0.02	100.00	99.05	0.56	0.94	56%	36,236	4.64%	98.56	1.71	86.42
Niger	290.0	0.79	30.77	13.33	0.31	2.07	84%	384	0.70%	31.25	1.50	45.39
Nigeria	2900.5	1.37	12.50	6.19	0.21	0.83	77%	2535	0.20%	16.35	6.38	46.40
North Korea	92.6	5.30	9.13	10.95	0.32	0.84	52%	1068	1.63%	2.40	2.03	62.67
Norway	1.8	0.00	99.04	91.43	0.10	0.84	40%	89,595	2.21%	98.08	0.82	82.49
Paraguay	15.9	7.03	15.87	48.57	0.00	0.20	39%	3822	1.20%	17.31	2.03	59.82
Qatar	0.7	0.12	78.37	84.29	0.00	0.94	85%	74,531	1.50%	77.40	1.75	65.32
Singapore	0.0	0.12	96.63	96.19	0.11	0.98	85%	51,809	1.69%	100.00	0.00	84.60
South Korea	0.0	0.07	66.83	53.81	0.56	0.97	85%	25,021	4.12%	79.81	0.00	83.52
South Sudan	92.6	0.95	0.48	2.38	0.02	0.94	85%	332	1.63%	0.48	2.03	35.70
Spain	0.3	0.05	69.71	55.71	0.84	0.91	66%	30,486	2.75%	85.10	0.50	73.87
Sweden	1.9	0.01	98.56	80.95	0.53	0.82	58%	55,159	1.88%	96.63	1.46	80.23
Switzerland	0.1	0.01	97.60	96.67	0.41	0.92	58%	75,594	4.12%	99.52	1.00	75.80
Syria	92.6	0.52	1.92	0.00	0.31	0.84	84%	919	0.30%	5.29	2.03	41.53
Togo	92.6	5.10	25.48	38.10	0.34	15.06	53%	554	1.10%	11.06	2.03	46.30
Turkmenistan	92.6	0.12	5.77	42.86	0.00	0.73	85%	6937	1.63%	19.23	2.03	62.67
UK	0.4	0.05	93.75	61.43	0.75	0.98	76%	41,196	2.44%	94.23	2.00	82.57
Uruguay	5.6	4.34	89.42	85.24	0.49	0.80	46%	13,950	1.55%	72.60	1.75	61.69
USA	0.6	0.07	89.90	67.14	0.67	0.96	79%	51,593	4.22%	89.90	2.00	82.03
Venezuela	92.6	4.66	4.33	15.71	0.39	0.65	56%	12,793	3.89%	10.58	2.03	35.30
Vietnam	21.4	2.79	41.83	50.00	0.49	0.92	68%	1685	0.74%	55.29	1.98	59.04
Yemen	92.6	0.62	3.37	0.48	0.29	0.74	85%	775	0.20%	3.37	2.03	44.58

) [24,69]. The indicators cover resilience holistically, i.e., both the pre- and post-event phases, and represent, among others, the quality, reliability and interconnectivity of the electricity system, the generation diversity, the fuel supply security and self-sufficiency, the available financial resources, the equivalent availability factor (EAF), the average outage times and geopolitical factors such as corruption, government effectiveness and political stability. Overall, this study covers more than 96% of the world’s population and 99.6% of the world’s electricity consumption.

The data come from a variety of reliable, credible and widely recognized sources, ranging from governmental agencies to international organizations and private companies [68,70,71,72,73,74,75]. Furthermore, the values of each indicator originate from a unique source, making it homogeneous and comparable. However, there are some missing values, which were inserted as the mean of the other values in order not to distort the data [76]. Inserting the mean of the other values is one of the three most standard techniques employed for dealing with data incompleteness in different scientific disciplines [77]. The other two procedures consist in (1) excluding the incomplete cases and (2) replacing the unknown value with the entire range of all possible values on a given indicator (input or output). These two other procedures were neglected for the following reasons. First, the countries with missing performances were not excluded from the analysis as the data was not available only for a limited subset of inputs or outputs, whereas for the vast majority of countries it has been reliably collected. In the context of DEA, the practical usefulness and credibility of the results increase with the greater number of DMUs. Considering that we have 140 countries for only 12 indicators, the ratio country to indicators is very high [32,78]. This increases the discriminatory power of DEA, ultimately resulting in less bias. In addition, by using the novel hybrid framework for robust efficiency analysis (i.e., a Hit-And-Run Monte Carlo simulation), subjectivity is again minimized. Second, analyzing the entire range of admissible values, it would have markedly deteriorated the robustness of the results. For example, such a hypothetical country would be allowed to attain the best and the worst performances on a given indicator. Due to the application of a ratio-based efficiency model, such a unit would attain the extreme comprehensive performances (e.g., the first and the last ranks depending on the precise performance adopted in a given scenario) only because of treating the missing values in this particular way.

The next step in the preparation of the data for further analysis is the normalization. In fact, the algorithm used in this research is a weighted sum over another weighted sum (called the ratio-based efficiency measure, see formula 1), which makes normalization a necessary step to render the different measurement units comparable [79]. The normalization method adopted in the present case study is the target one, which consists in dividing all the indicators by their maximum value, because it conserves the ratios. Hence, each indicator has a maximum value of 1 and a minimum value in the range [0, 1]. Additionally, the ratios are conserved (see Table S2 in the electronic supplementary information (ESI)).

Finally, DEA requires the indicators to be classified into either being inputs or outputs (see formula 1). As the present case study represents a general benchmarking problem, where DEA is employed for decision-making, the inputs are the indicators with a negative preference order (i.e., to be minimized) and the outputs are the ones with a positive preference order (i.e., to be maximized) [78]. Hence, the classification of the indicators is straightforward and univocal. This results in a total of 4 inputs (i₁, i₂, i₆ and i₁₁) and 8 outputs (i₃, i₄, i₅, i₇, i₈, i₉, i₁₀ and i₁₂) (see Table 2).

3.2. Ratio-Based Efficiency Analysis with the Charnes, Cooper, and Rhodes (CCR) Model

DEA is a method for assessing efficiencies of DMUs, which in the present case study are countries. Considering a set of DMUs ($D=\left\{DM{U}_1,\dots ,DM{U}_K\right\}$, where $K$ is the number of DMUs), the efficiency of $DM{U}_{\mathrm{o}}\in D$ is calculated as the ratio between virtual output and virtual input, which are quantified as weighted sums of individual outputs and inputs, respectively [27]. The equation is:

$$E_o\left(v,u\right)=\frac{{\sum }_{n\in OUT}{u}_n{y}_{no}}{{\sum }_{m\in IN}{v}_m{x}_{mo}}$$

(1)

where:

• $E_o\left(v,u\right)$ is the efficiency of $DM{U}_o\in D$;
• $x_{mo}$ is the amount of $m$-th input consumed by $DM{U}_o\in D$, $m\in IN$ (by default a set of inputs is defined as $IN=\left\{1,\dots ,M\right\}$);
• $y_{no}$ is the amount of $n$-th output produced by $DM{U}_o\in D$, $n\in \mathrm{OUT}$ (by default a set of inputs is defined as $OUT=\left\{1,\dots ,N\right\}$);
• $v_{IN}=\left\{v_{\mathrm{j}}:j\in IN\right\}$: a vector of input weights (by default $v_{IN}=\left\{v_1,\dots ,v_M\right\}$);
• $u_{OUT}=\left\{u_{\mathrm{j}}:j\in OUT\right\}$: a vector of output weights (by default $u_{OUT}=\left\{u_1,\dots ,u_N\right\}$).

In a standard DEA setting, the aim is to divide the DMUs into efficient and inefficient ones. For this purpose, one has to find for each $DM{U}_{\mathrm{o}}\in D$ a weight vector that maximizes its efficiency score. Hence, the ratio-based efficiency analysis with the CCR model consists in solving the following primal optimization problem [27]:

$$\begin{gathered} \max E_{\mathrm{o}}^{*}={{\displaystyle \sum }}_{n \in OUT}{u}_n{y}_{no} \\ \mathrm{subject}\mathrm{to}:{{\displaystyle \sum }}_{m \in IN}{v}_m{x}_{mo}=1; \\ {\sum }_{n \in OUT}{u}_n{y}_{nk}\le {\sum }_{m \in IN}{v}_m{x}_{mk};k=1,\dots ,K; \\ {v}_m,u_n\ge 0;m\in IN,n\in OUT. \end{gathered}$$

(2)

By definition, the DMUs with efficiency score $E_{\mathrm{o}}^{*}$ equal to 1 are considered as efficient. The rest of the DMUs are inefficient (efficiency scores between 0 and 1 exclusive), because other DMUs or their conical combination achieve higher scores under the same conditions. Note that $E_{\mathrm{o}}^{*}$ indicates a multiplier that should be applied to all inputs $x_{mo}$, $m\in IN$ so that $DM{U}_{\mathrm{o}}\in D$ becomes efficient (e.g., in case $E_{\mathrm{o}}^{*}=0.8$, $DM{U}_{\mathrm{o}}$ would become efficient by decreasing its inputs by 20%).

Moreover, DEA allows to identify benchmarks to be followed and improvement strategies for the inefficient DMUs. These can be determined by solving the following dual optimization problem:

$$\begin{gathered} \min {\mathsf{\theta }}_{\mathrm{o}} \\ \mathrm{subject}\mathrm{to}:{{\displaystyle \sum }}_{k=1,\dots ,\mathrm{K}}{\mathsf{\lambda }}_{\mathrm{k}}{x}_{mk}\le {\mathsf{\theta }}_{\mathrm{o}}{x}_{m0};m\in IN; \\ {\sum }_{k=1,\dots ,\mathrm{K}}{\mathsf{\lambda }}_{\mathrm{k}}{y}_{nk}\ge y_{n0};n\in OUT; \\ {\mathsf{\lambda }}_{\mathrm{k}}\ge 0,\mathrm{k}=1,\dots ,\mathrm{K}. \end{gathered}$$

(3)

On the one hand, for an efficient $DM{U}_{\mathrm{o}}\in D$, ${\mathsf{\theta }}_{\mathrm{o}}=1$ and ${\mathsf{\lambda }}_{\mathrm{k}=\mathrm{o}}=1$. On the other hand, for an inefficient $DM{U}_{\mathrm{o}}\in D$, all DMUs with ${\mathsf{\lambda }}_{\mathrm{k}}>0$ are contained in the reference set of $DM{U}_{\mathrm{o}}$ and can be used for constructing a hypothetical reference unit with greater or equal outputs and lower inputs than $DM{U}_{\mathrm{o}}$. The differences between the inputs of such a reference unit and $DM{U}_{\mathrm{o}}$ indicate the improvements of inputs that are expected from $DM{U}_{\mathrm{o}}$ for attaining the efficiency. Overall, the CCR model allows tackling research questions 1 and 2.

The aforementioned analysis represents an input-oriented perspective. It derives the required reduction of inputs, if any, that would ensure efficiency (i.e., the best ratio between the virtual outputs and inputs for at least one feasible vector of weights associated with these factors), assuming that the outputs of a given DMU remain unchanged. Note that in DEA, it is also possible to conduct an output-oriented analysis, hence finding the improvements of outputs needed for reaching the efficiency while holding the current amount of inputs (for details, see [27]).

3.3. In-Depth Analysis of Status of Efficiency

Explanation of the efficiency status requires construction of arguments, which can be used to justify its validity and logic. In this section, the task of generating explanations of the outcomes of DEA in view of the following procedures is considered:

• in case $DM{U}_{\mathrm{o}}\in D$ is efficient, identification of the minimal subsets of indicators that make it efficient (such minimal subsets of inputs and outputs are called efficiency reducts);
• in case $DM{U}_{\mathrm{o}}\in D$ is inefficient, identification of the smallest subsets of other DMUs that underlie its inefficiency (such minimal subsets of DMUs are denoted as efficiency constructs).

It is to be noted that the efficiency reducts and constructs are new methodological developments, as explained in the following sentences. On the one hand, to determine all efficiency reducts for some efficient $DM{U}_{\mathrm{o}}\in D$, an additive method is implemented (see Algorithm 1). It consists of a progressive verification if $DM{U}_{\mathrm{o}}$ is efficient when using different subsets of inputs $IN$ and outputs $OUT$, while starting with the smallest ones, and eliminating from further consideration the proper supersets (a superset is a set that includes another set. For example, i₁, i₂ and i₃ is a superset of i₁ and i₃, and i₂ and i₃.) of these subsets of indicators that already guaranteed the efficiency [80]. For each efficient $DM{U}_{\mathrm{o}}$ there exists at least one efficiency reduct (in the worst-case scenario, it contains all inputs and outputs).

Algorithm 1. Additive method for identifying all efficiency reducts.

Require: sets of inputs $IN$ and outputs $OUT$

Ensure: $ERs$, all efficiency reducts for $DM{U}_{\mathrm{o}}\in D$

1:  $IO$ = all subsets containing at least one input from $IN$ and at least one output from $OUT$ ordered with respective to the increasing cardinality

2:  for each $I{O}_k\in IO$ do

3:  Solve equation (2) for $DM{U}_{\mathrm{o}}\in D$ with inputs and outputs reduced to $I{O}_k$ to derive an optimal solution $E_{\mathrm{o}}^{*}(I{O}_k$)

4:  if $E_{\mathrm{o}}^{*}(I{O}_k$) = 1 then

5:  $ERs$ = $ERs\cup I{O}_k$

6:  Remove all supersets of $I{O}_k$ from $IO$

7:  end if

8:  end for

On the other hand, to identify an efficiency construct for some inefficient $DM{U}_{\mathrm{o}}\in D$, the aim is to find a subset of other DMUs that once removed from the analysis would make $DM{U}_{\mathrm{o}}$ efficient. This can be attained by solving the following mixed-integer linear programming (MILP) problem:

$$\begin{gathered} \min f_w={\displaystyle {\displaystyle \sum }_{k=1,k\ne 0}^K}{b}_k \\ \mathrm{subject}\mathrm{to}:{\sum }_{n\in OUT}{u}_n{y}_{no}={\sum }_{m\in IN}{v}_m{x}_{mo}=1; \\ {\sum }_{n\in OUT}{u}_n{y}_{nk}\le {\sum }_{m\in IN}{v}_m{x}_{mk}+C{b}_k\left(k=1,\dots ,K,k\ne 0\right); \\ {b}_k\in \left\{0,1\right\}\left(k=1,\dots ,K,k\ne 0\right); \\ {v}_m,u_n\ge 0;m\in IN,n\in OUT; \end{gathered}$$

(4)

where $C$ is a large positive constant. If $b_k=1$, $DM{U}_{\mathrm{k}}$ needs to be eliminated to make $DM{U}_{\mathrm{o}}$ efficient. Hence, the optimal solution of the above MILP (denoted with *; e.g., $f_w^{*}$) indicates one of the efficiency constructs $I{C}_w=\left\{DM{U}_{\mathrm{k}}\in D:b_k^{*}=1\right\}.$ It is possible to identify other constructs by adding the constraints that forbid finding again the solutions found in the previous iterations ($w$, $w-1$,…, $1$): ${\sum }_{DM{U}_{\mathrm{k}}\in I{C}_w}{b}_k\le f_w^{*}-1$ [81].

3.4. Robust Efficiency Analysis

The standard ratio-based efficiency analysis derives the efficiency scores from the best-case scenario for each DMU. As a result, such scores may not be representative, because they may only be achieved for a very limited number of weight vector combinations, which—in addition—are different for each DMU. Therefore, some DMUs may be considered as efficient even though they do not perform particularly well in general. Moreover, the subsets of efficient and inefficient DMUs can be large, and standard DEA methods offer poor arguments to discriminate within these subsets [78]. For this purpose, a variety of measures that reflect how DMUs perform across all feasible vectors of input/output weights are accounted for [28]. These results refer to three different perspectives: cardinal ratings (efficiency scores; answers research question 3), pairwise one-on-one comparisons (preference relations; answers research question 4), and ordinal comparisons of all DMUs (efficiency ranks; answers research question 5). Specifically, linear programming (LP) to derive the following exact outcomes is used:

• maximal $E_{\mathrm{o}}^{*}$ and minimal $E_{o,*}$ efficiency scores for $DM{U}_{\mathrm{o}}\in D$ attained in the set of all feasible input/output weights (note that $E_{\mathrm{o}}^{*}$ corresponds to the score derived from the standard analysis);
• a necessary efficiency preference relation ${\underset{\_}{\succ }}_E^N$, which holds for a pair $(DM{U}_{\mathrm{o}},DM{U}_{\mathrm{k}})\in D\times D$ in case $DM{U}_{\mathrm{o}}$ attains efficiency at least as good as $DM{U}_{\mathrm{k}}$ for all feasible input/output weights;
• the best $R_{\mathrm{o}}^{*}$ and the worst $R_{o,*}$ efficiency ranks for $DM{U}_{\mathrm{o}}\in D$, which are derived from the analysis of, respectively, minimal and maximal subsets of DMUs that attain better efficiency than $DM{U}_{\mathrm{o}}$ for some feasible input/output weights.

The measures convey useful knowledge on the performances of DMUs in the most and least advantageous scenarios as well as for all feasible weight vectors combinations. Nonetheless, the difference between extreme outcomes can, in general, be quite large, whereas the necessary relation can leave many DMUs incomparable. For this reason, a stochastic efficiency analysis based on the Monte Carlo (MC) simulation to estimate the probability of different outcomes is applied [82]. Hence, a large representative set of feasible weight vectors ${\left(v,u\right)}^S$ (with $W$ being the number of samples) is derived, using a dedicated algorithm such as Hit-And-Run [83]. Hit-And-Run samplers have been proven to perform well for problems of larger sizes [83]. Note that each vector from the feasible weight space is assigned equal chances to be hit (uniform distribution). For each $\left(v,u\right)\in {\left(v,u\right)}^S$, the efficiency score $E_o\left(v,u\right)$ for each $DM{U}_{\mathrm{o}}\in D$ is computed, which allows us to approximate the following stochastic acceptability indexes:

• an efficiency acceptability interval index $EAII\left(DM{U}_{\mathrm{o}},b_i\right)$, which is the share of feasible weight vectors for which $DM{U}_{\mathrm{o}}\in D$ attains an efficiency score in the interval $b_i\subset \left[0,1\right]\left(\mathrm{i}=1,\dots ,\mathrm{B}\right)$, where $\mathrm{B}$ is the number of subintervals (${\bigcup }_{\mathrm{i}=1}^{\mathrm{B}}{b}_i=\left[0,1\right];b_i\cap b_j=\varnothing ,i\ne j$). This represents the distribution of scores, providing the performance robustness assessment that answers research question 3;
• an expected (average) efficiency $E{E}_{\mathrm{o}}={\sum }_{\left(v,u\right)\in {\left(v,u\right)}^S}{E}_o\left(v,u\right)/W$ for $DM{U}_{\mathrm{o}}\in D$;
• a pairwise efficiency outranking index $PEOI\left(DM{U}_{\mathrm{o}},DM{U}_{\mathrm{k}}\right)$ for $\left(DM{U}_{\mathrm{o}},DM{U}_{\mathrm{k}}\right)\in D\times D$, which is the share of feasible weight vectors for which $DM{U}_{\mathrm{o}}$ is not worse than $DM{U}_{\mathrm{k}}$ in terms of the efficiency score, i.e., $E_o\left(v,u\right)\ge E_k\left(v,u\right)$. This answers research question 4 as it indicates how well countries perform in comparison with each other;
• an efficiency rank acceptability index $ERAI\left(DM{U}_{\mathrm{o}},r\right)$, which is the share of feasible weight vectors for which $DM{U}_{\mathrm{o}}\in D$ attains $r$-th rank. This answers research question 5 as it allows to rank the countries;
• an expected (average) rank $E{R}_o={\sum }_{r=1}^{K}r·ERAI\left(DM{U}_o,r\right)$ for $DM{U}_{\mathrm{o}}\in D$.

Most importantly, all of the above results indicate how stable the scores, rankings, and relations observed for different DMUs are across all feasible weight vectors including those that are disadvantageous for each DMU. Hence, it is complementary to the CCR model as it provides a more likely and plausible representation compared to the standard efficiency analysis. Moreover, these outcomes offer arguments (e.g., average efficiencies or ranks), which enable univocal rankings. For the computational details, discussion on the properties and detailed interrelations between the outcomes computed with LP and MC simulation, see [28]. Overall, the robust efficiency analysis allows us to explore research questions 3, 4 and 5.

4. Results and Discussion

In this section, the results are discussed according to the research questions formulated in Section 1.

4.1. What Are the Best Performing (i.e., Most Resilient) Countries and What Are the Reasons for This Achievement?

The main interest of decision-makers might be knowing which countries are the best. Therefore, the results of the CCR model are given in

Table 3. Charnes, Cooper, and Rhodes (CCR) maximum efficiencies of the 140 countries.

Country	CCR	Country	CCR	Country	CCR	Country	CCR
Algeria	1.000	Armenia	0.987	Mauritius	0.858	Brazil	0.700
Australia	1.000	Taiwan	0.983	Cyprus	0.846	Hungary	0.700
Bulgaria	1.000	Israel	0.979	Guatemala	0.843	Pakistan	0.698
Canada	1.000	UK	0.970	Uzbekistan	0.838	Mongolia	0.692
Costa Rica	1.000	Tunisia	0.967	Serbia	0.831	Vietnam	0.683
Czech Republic	1.000	Brunei Darussalam	0.953	Nicaragua	0.825	Peru	0.678
Estonia	1.000	Uruguay	0.953	Turkey	0.821	Latvia	0.671
Finland	1.000	Portugal	0.946	Hong Kong	0.815	El Salvador	0.667
France	1.000	India	0.943	Syria	0.811	Honduras	0.665
Germany	1.000	Venezuela	0.942	Bangladesh	0.807	Botswana	0.654
Haiti	1.000	United Arab Emirates	0.941	Bosnia and Herzegovina	0.806	Montenegro	0.648
Iceland	1.000	Azerbaijan	0.936	Belgium	0.806	Angola	0.625
Italy	1.000	Iran	0.936	Congo, Rep.	0.805	Gabon	0.624
Jamaica	1.000	Oman	0.914	Tanzania	0.803	Suriname	0.599
Kuwait	1.000	Malaysia	0.912	South Africa	0.797	North Korea	0.584
Libya	1.000	Argentina	0.906	Iraq	0.796	Kyrgyzstan	0.576
Luxembourg	1.000	Slovenia	0.905	Dominican Republic	0.791	Malta	0.567
Netherlands	1.000	Slovakia	0.895	Austria	0.791	Zimbabwe	0.564
New Zealand	1.000	Saudi Arabia	0.895	Senegal	0.785	Cambodia	0.552
Norway	1.000	Poland	0.890	Panama	0.779	Myanmar	0.551
Paraguay	1.000	Trinidad and Tobago	0.888	Georgia	0.776	Sudan	0.542
Qatar	1.000	Ukraine	0.886	Bolivia	0.770	Mozambique	0.531
Romania	1.000	Yemen	0.883	Colombia	0.763	Croatia	0.514
Russia	1.000	Denmark	0.880	Kosovo	0.761	Albania	0.502
Singapore	1.000	Morocco	0.877	Ghana	0.756	Zambia	0.488
South Korea	1.000	Indonesia	0.876	Egypt	0.747	Nigeria	0.486
Spain	1.000	Chile	0.874	Eritrea	0.745	Cameroon	0.483
Sweden	1.000	Bahrain	0.867	Greece	0.743	Tajikistan	0.478
Switzerland	1.000	Jordan	0.865	China	0.742	Ethiopia	0.461
Turkmenistan	1.000	Cuba	0.865	Kenya	0.739	Nepal	0.429
USA	1.000	Philippines	0.864	Sri Lanka	0.729	Niger	0.396
Ireland	0.993	Cote d’Ivoire	0.864	Ecuador	0.720	Congo, Dem. Rep.	0.373
Mexico	0.992	Thailand	0.860	Lebanon	0.716	Namibia	0.268
Moldova	0.992	Kazakhstan	0.859	Lithuania	0.707	Benin	0.245
Japan	0.991	Belarus	0.858	South Sudan	0.706	Togo	0.040

. According to this model, 31 out of the 140 countries are deemed as efficient (having a value of 1). All of these countries have at least some inputs and outputs performing well enough so that with specific weight vectors no other countries do better.

Among the inefficient countries, Togo, Benin, Namibia and the Democratic Republic of Congo (DRC) score the lowest, with maximal CCR efficiencies of 0.040, 0.245, 0.268 and 0.373, respectively. This means that even in the best case, i.e., with the most advantageous weight vector combination, other countries perform significantly better. While the CCR model provides a clear differentiation of scores for inefficient countries, it does not allow us to differentiate the efficient ones, with 31 receiving an efficiency of 1. Therefore, building a univocal ranking is impossible with the CCR model. The differentiation between efficient countries comes in the context of ranking robustness assessment presented in Section 4.3 and Section 4.5.

After the efficient countries were identified, in the next step it was analyzed why they are efficient. This can be done by identifying the individual indicators that make the corresponding country efficient, i.e., efficiency reducts, as shown in

Table 4. Minimal subsets of indicators (efficiency reducts) that make the corresponding country efficient. The numbers refer to the indicator number listed in Table 2. The full table for all efficient countries is available in the electronic supplementary information (ESI), Table S3.

Country	Inputs	Outputs	Country	Inputs	Outputs	Country	Inputs	Outputs
Algeria	1;2;6;11	7	Norway (continued)	2;6	3	South Korea (continued)	6;11	9
Australia	1;6	5;7;8		2;6	10		6;11	12
Canada	6	3;4		2;6	12		6;11	3;5
	6	4;5		2;11	4		6;11	4;5
	6	5;8		6;11	3		6;11	5;8
	6	9;10		6;11	10		6;11	7;8
	1;6	10	Qatar	1;6	7;8	Spain	6	3;5
	1;6	12		2;6	7;8		6	5;10
	2;6	10;12		6;11	7;8	Sweden	6	3
	2;11	9	Singapore	1;2	4		6	10
	6;11	10		1;2	8		1;6	5;7;8
	2;6;11	12		1;6	3		2;6	5
Estonia	1;2;6	3;7;12		1;6	10		2;6	4;7
Germany	6	3;5		2;11	4		2;6	7;12
	6	5;8		2;11	8	Switzerland	6	5;8;9
Haiti	1;6	5;7		6;11	3		1;2	3
	6;11	7		6;11	4		1;2	4
Italy	2;11	5		6;11	8		1;2	5
Jamaica	1;6	7		6;11	10		1;2	7
Kuwait	1;6	7	South Korea	1;2	3		1;2	8
	6;11	7		1;2	5		1;2	9
Libya	6	5		1;2	7		1;2	10
	6	7		1;2	9		1;2	12
	2;6	12		1;2	10		2;6	9
Luxembourg	2;11	5;8		1;2	12		2;6	4;7
	1;2;11	8		2;11	3		2;6	7;8
Netherlands	6	9		2;11	5		2;11	5
New Zealand	2;6	9;12		2;11	7		6;11	4;7;8;9
Norway	2	8		2;11	9		6;11	7;8;9;10
	6	8		2;11	10	Turkmenistan	2;6	7
	6	4;10		2;11	12	USA	1;6	5;7;8;9
	1;6	10;12

. For example, it is possible to find priorities (weight vectors) that make the United States of America (USA) efficient when considering inputs 1 and 6 and outputs 5, 7, 8 and 9 only (the numbers correspond to the indicator numbers). In fact, on the inputs, the USA’s SAIDI (i₁) of 0.6 hours per customer per year and its electricity import dependence (i₆) are well-performing. On the outputs, the USA’s electricity production mix (i₅) is diverse, its EAF (i₇) is high and both its GDP per capita (i₈) and insurance penetration (i₉) are comparatively high on an international scale. In other words, these indicators represent the strengths of the USA and no other country performs better under such priorities. However, this efficiency might not be obvious to reach as the USA requires, under specific weight vectors, the combination of at least two inputs and at least four outputs.

Considering another example, Singapore can become efficient under certain weight vectors if inputs 1 and 2 and output 4 are considered. This efficiency reduct corresponds to the SAIDI, where Singapore is the best country in the world as it only experiences less than a minute of electricity supply interruption per customer per year [84], the low severe accidents risks, indicating that Singapore’s electricity production mix is safe from the point of view of human fatalities, and its outstanding political stability and absence of violence/terrorism. Hence, Singapore requires only two inputs and one output to reach efficiency, indicating that these are stronger compared to those from the USA.

Furthermore, a rather surprising result is given by Libya. Considering its low indicator performances (e.g., SAIDI of 1883.4 hours per customer per year, high corruption and low political stability, GDP per capita, insurance penetration, government effectiveness and ease of doing business), it is unexpected that Libya still is deemed efficient. However, as it has the best ratios of i₅/i₆ or i₇/i₆, these subsets of indicators still make it efficient for specific weighting vectors.

Knowing which countries are the best might be the first requirement of a decision-maker, but not all countries perform at the top. Therefore, it is necessary to also look at the inefficient countries, including the reasons why they are inefficient and how they can improve themselves, which leads to Section 4.2 below.

4.2. Why Are Some Countries Inefficient, How Can They Improve Their Scores and What Are Their Benchmarks?

Table 5. Projection of inefficient countries onto the efficiency frontier. Only the shares of Denmark, Japan and Uruguay are hereby displayed. For these three countries, the shares from other countries are null. The full table for all countries is available in the ESI, Table S4.

Country	Canada	Czech Republic	Germany	Norway	Singapore	South Korea	Spain	Sweden	Switzerland
Denmark	0.251	0.000	0.000	0.008	0.078	0.183	0.145	0.000	0.446
Japan	0.039	0.416	0.427	0.000	0.000	0.000	0.000	0.000	0.225
Uruguay	0.691	0.000	0.000	0.000	0.000	0.000	0.000	0.240	0.000

shows the projections of inefficient countries onto the efficiency frontier. For example, the projection for Uruguay represents 0.691 of Canada and 0.240 of Sweden (the sum of the shares does not necessarily have to be equal to 1, because conical combinations of existing units are tolerated). This means that the closest virtual country to Uruguay, that is situated on the efficiency frontier, is composed of 0.691 times the indicator values of Canada plus 0.240 times the indicator values of Sweden. The distance to this virtual country represents the closest path for Uruguay to become efficient. In other words, Canada and Sweden are its benchmarks. The higher the share, the closer the original country already is to its benchmark. In this example, Uruguay is already closer to Canada compared to Sweden. The same analysis can be made with other countries. For example, Denmark can become efficient with contributions from six countries and Japan from four.

Furthermore, the necessary improvements for all inputs or outputs that need to be applied to make a certain country efficient are given in

Table 6. Necessary improvements to make a country efficient. These improvements need to be achieved on all inputs cumulatively or all outputs cumulatively. The full table for all countries is available in the ESI, Table S5.

Country	i1	i2	i3	i4	i5	i6	i7	i8	i9	i10	i11	i12
Denmark	0.000	−0.001	0.133	0.295	0.091	−0.009	0.327	0.075	0.249	0.202	−0.007	0.175
Japan	0.000	−0.002	0.030	0.008	0.007	−0.001	0.029	0.004	0.098	0.055	−0.182	0.122
Uruguay	−0.002	−0.448	0.044	0.042	0.095	−0.002	0.103	0.333	0.239	0.204	−0.068	0.194

. The values are negative for the inputs, as they need to be decreased, and positive for the outputs because they have to be increased. These necessary improvements have to be applied to all the inputs together or all the outputs together.

Finally, as DEA is a measure of efficiency in relation to other DMUs, it is also possible to become efficient when the country list is changed.

Table 7. Minimal subset of countries (efficiency constructs) to be removed in order to make the respective country efficient. For Denmark, Japan and Uruguay, the countries not listed all contain zeros. The full table for all countries is available in the ESI, Table S6.

Countries	Canada	Czech Republic	Germany	Norway	Singapore	South Korea	Spain	Sweden	Switzerland
Denmark	0	0	0	0	0	0	0	0	1
Japan	0	0	1	0	0	0	0	0	0
Uruguay	1	0	0	0	0	0	0	0	0

represents the efficiency construct to be removed in order to make a given country efficient. In the case of Uruguay, it is inefficient because Canada is included in the analysis. In other words, even under the most favorable conditions for Uruguay, Canada will perform better. Another example is Denmark, which has a high level of electricity supply resilience at an absolute level, but it can do even better as under similar conditions, Switzerland scores higher. Furthermore, for Denmark, Japan and Uruguay, only one country has to be removed to make them efficient. This indicates that these three countries are close to being efficient. On the contrary, for example, the Democratic Republic of Congo (DRC) requires the removal of 119 countries to become efficient (see Table S6). In other words, the DRC is far away from the efficiency frontier. It is important to note that, for policymaking, the efficiency constructs should not be misused. In fact, it would not make sense to adapt the list of countries in order to artificially increase the score of a country of interest. The efficiency constructs should rather be seen as a tool to identify benchmarks towards which inefficient countries should aim for.

The results presented in Section 4.1 and Section 4.2 demonstrated that the CCR model successfully identified (1) the benchmarks, (2) the leading indicators for each country from an efficiency perspective, (3) the distance from the efficiency frontier, (4) the required indicator improvements to make inefficient countries efficient and (5) the list of countries making others inefficient. However, one has to bear in mind that these findings are based on the most advantageous weight vectors for each country, clearly representing best-case scenarios.

Therefore, the analysis is extended using robust efficiency analysis models to address the following questions:

• What is the average or most likely performance of a country? What is the expected distribution of its performance (Section 4.3)? This answers research question 3.
• Is there a univocal ranking of countries (Section 4.4)? This answers research question 4.
• How does each country perform against all others (pairwise comparisons) (Section 4.5)? This answers research question 5.

Identifying benchmarks allows us to analyze weaknesses and develop successful policies. However, the results of Section 4.1 and Section 4.2 are based on the CCR model, i.e., a best-case scenario, as explained in Section 3.4. Decision-makers might be interested in a more unbiased view of the results in which ranking robustness is assessed. This leads to Section 4.3 below.

4.3. How Robust Is the Performance of the Countries?

Instead of analyzing only the best-case scenario, the Hit-And-Run Monte Carlo-based robust efficiency model accounts for the wide variability of preference models and it calculates a distribution of performance scores of the countries, thus providing a measure of robustness [83,85].

Table 8. Maximum and average efficiency scores for the robust efficiency analysis model, as well as the efficiency acceptability interval indices. The number after the countries’ names corresponds to their ranks computed based on the average efficiency. The full table for all countries is available in the ESI, Table S7.

Country	Simulation-Based Monte Carlo		Simulation-Based Monte Carlo Efficiency Acceptability Interval Indices (in %)
	Maximum	Average	[0.0–0.1]	[0.1–0.2]	[0.2–0.3]	[0.3–0.4]	[0.4–0.5]	[0.5–0.6]	[0.6–0.7]	[0.7–0.8]	[0.8–0.9]	[0.9–1]
South Korea (1)	1.000	0.911	0.0	0.0	0.1	0.2	0.5	1.1	4.3	10.9	18.8	64.1
Singapore (2)	1.000	0.852	0.0	0.1	0.3	0.6	1.6	4.6	9.4	15.6	19.9	47.8
Canada (3)	1.000	0.668	3.7	6.1	7.1	6.9	8.0	7.5	8.6	7.8	9.6	34.7
Spain (4)	1.000	0.594	2.8	5.2	7.0	7.9	9.1	10.7	15.5	21.0	16.2	4.6
Finland (5)	1.000	0.583	2.2	4.6	6.3	7.5	9.4	12.3	20.2	28.8	7.0	1.7
Norway (6)	1.000	0.576	5.2	8.2	8.9	8.6	9.4	9.3	9.9	11.6	13.4	15.4
Switzerland (7)	1.000	0.571	5.6	8.5	9.0	9.3	8.5	9.2	8.1	9.5	20.3	12.0
Italy (8)	0.919	0.554	1.2	3.5	5.1	7.8	11.3	23.7	31.0	14.1	2.3	0.1
Netherlands (9)	1.000	0.523	5.2	8.9	9.5	9.9	10.5	11.0	13.1	17.9	10.9	3.0
France (10)	1.000	0.516	6.8	10.3	9.9	9.5	10.1	9.1	10.9	15.9	12.9	4.6
Denmark (11)	0.786	0.489	4.8	8.2	9.3	10.6	11.4	13.7	27.8	14.2	0.0	0.0
Iceland (12)	0.984	0.477	6.9	10.5	10.4	10.9	11.1	10.8	15.3	20.1	3.9	0.2
Sweden (14)	1.000	0.471	10.1	12.5	11.4	10.6	9.0	8.8	9.4	13.2	10.3	4.7
Germany (16)	0.975	0.438	10.1	13.1	11.9	12.0	10.2	10.8	12.0	11.9	7.1	0.8
New Zealand (17)	0.915	0.435	11.1	13.1	12.5	11.4	9.3	9.4	11.5	13.7	7.7	0.2
USA (24)	0.949	0.381	13.7	15.1	13.9	11.9	11.1	12.0	11.5	8.1	2.7	0.1
UK (26)	0.916	0.366	14.4	15.4	14.0	12.5	11.4	12.9	11.0	6.9	1.4	0.0
Qatar (35)	0.914	0.318	16.0	17.5	16.7	15.1	13.3	12.2	6.8	1.9	0.3	0.0
Japan (42)	0.886	0.263	26.6	21.6	14.5	12.3	10.0	6.4	5.0	2.8	0.8	0.0
Luxembourg (43)	0.690	0.260	7.2	17.3	40.4	30.0	4.1	0.7	0.2	0.0	0.0	0.0
Algeria (54)	0.769	0.225	16.7	29.3	28.2	16.5	6.8	1.8	0.5	0.1	0.0	0.0
Turkmenistan (75)	0.775	0.143	44.3	30.0	16.2	6.5	1.9	0.8	0.2	0.1	0.0	0.0
United Arab Emirates (77)	0.823	0.141	52.4	23.2	11.8	5.4	3.7	2.3	0.9	0.2	0.0	0.0
Costa Rica (78)	0.863	0.138	57.8	20.4	8.7	5.4	3.0	2.5	1.4	0.6	0.2	0.0
Uruguay (83)	0.749	0.115	60.6	23.0	8.4	4.3	2.0	1.1	0.5	0.1	0.0	0.0
Vietnam (91)	0.548	0.100	63.1	24.3	8.4	3.1	1.0	0.1	0.0	0.0	0.0	0.0
Yemen (98)	0.650	0.083	69.5	22.9	5.7	1.3	0.4	0.1	0.0	0.0	0.0	0.0
Syria (99)	0.600	0.082	69.5	23.5	5.4	1.1	0.4	0.1	0.0	0.0	0.0	0.0
Haiti (102)	0.736	0.077	74.8	18.4	4.4	1.4	0.5	0.3	0.1	0.0	0.0	0.0
Niger (107)	0.299	0.070	77.4	21.7	0.9	0.0	0.0	0.0	0.0	0.0	0.0	0.0
Cambodia (110)	0.379	0.068	77.7	17.6	4.2	0.5	0.0	0.0	0.0	0.0	0.0	0.0
Eritrea (111)	0.485	0.061	81.9	15.2	2.3	0.5	0.2	0.0	0.0	0.0	0.0	0.0
South Sudan (112)	0.475	0.059	82.6	14.8	2.0	0.5	0.1	0.0	0.0	0.0	0.0	0.0
Venezuela (115)	0.511	0.054	85.3	11.2	2.8	0.6	0.1	0.0	0.0	0.0	0.0	0.0
China (116)	0.471	0.051	86.7	9.5	2.7	0.9	0.2	0.0	0.0	0.0	0.0	0.0
Paraguay (119)	1.000	0.048	88.2	8.0	2.1	0.9	0.4	0.2	0.1	0.0	0.0	0.0
Myanmar (123)	0.362	0.043	90.5	8.2	1.3	0.1	0.0	0.0	0.0	0.0	0.0	0.0
North Korea (124)	0.344	0.042	90.1	8.5	1.2	0.2	0.0	0.0	0.0	0.0	0.0	0.0
Kenya (128)	0.407	0.036	93.4	5.5	1.0	0.2	0.0	0.0	0.0	0.0	0.0	0.0
Libya (130)	0.979	0.035	93.1	5.2	0.9	0.3	0.2	0.0	0.0	0.1	0.0	0.0
Iraq (132)	0.568	0.034	93.7	5.1	0.8	0.2	0.1	0.0	0.0	0.0	0.0	0.0
Nepal (126)	0.258	0.031	94.8	5.0	0.2	0.0	0.0	0.0	0.0	0.0	0.0	0.0
Nigeria (138)	0.355	0.020	98.3	1.4	0.2	0.0	0.0	0.0	0.0	0.0	0.0	0.0
Congo, Dem. Rep. (139)	0.189	0.018	98.8	1.2	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
Togo (140)	0.033	0.015	100.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0

shows the results for the maximum and average performance of the scores obtained with 10,000 model runs. Furthermore, the efficiency acceptability interval indices (10 bins of equal size) are displayed. The sum of the efficiency acceptability interval indices per country is equal to 100%. The table shows only a selection of 45 from the 140 countries, ordered by decreasing values of average efficiency.

According to this model, South Korea, Singapore and Canada are the most efficient countries as in 64.1%, 47.8% and 34.7% of the simulations, respectively, their efficiency is situated in [0.9, 1]. As these countries perform well on multiple inputs and multiple outputs, their efficiency score is high for many weight vectors. On the other end of the spectrum, Togo, the DRC and Nigeria have, for more than 98% of the Monte Carlo simulations performed, efficiencies situated in [0, 0.1]. In Togo’s or the DRC’s case, neither of their indicators is well-performing. Regarding Nigeria, only its EAF (i₇) performs at an average level, which in comparison with other countries, still makes it one of the worst-performing ones.

When it comes to the distribution of performance, South Korea, Singapore and Canada consistently rank at the top, as they attain efficiencies higher than 0.7 for most of the simulations. This indicates that these three countries clearly outperform the others leading to lower efficiencies for all other countries. In the middle, there are countries that show more balanced distributions of scores. For example, Germany, Sweden, New Zealand and France attained efficiencies in all intervals and the sum of their three highest interval values represent between 37% and 40%. Hence, the efficiency of these countries is highly dependent on the weight vector considered. If more weight is placed on their well-performing indicators, they will score higher, and reciprocally. This aspect is not revealed by the CCR model, as it only retains the single most advantageous weight vector. When analyzing France in more detail, it can be seen that all its indicators are performing well except its political stability and absence of violence/terrorism (i₄), where it ranks 60th, and its relatively low electricity mix diversity (i₅, 77% of its electricity production comes from nuclear energy). Hence, France mostly performs well, but there exist some disadvantageous weight vectors, resulting in more balanced distribution of scores. Finally, about half of the countries are clustered at the bottom. In fact, for most weight vectors, these countries attain efficiencies situated in [0, 0.1] and only rarely exceed 0.5.

The maximum efficiency column in Table 8 corresponds to the extreme value of the 10,000 Monte Carlo simulations (the convergence of the results was verified by increasing the number of simulations stepwise and the confidence interval of 95% for Stochastic Multicriteria Acceptability Analysis (SMAA) analyses is not exceeded [85]). While the CCR model deemed 31 countries as efficient, the stochastic analysis only identified 13 as efficient, indicating that the Monte Carlo simulation could not find the best scenario for each country based on the chosen number of runs. Both scores would of course be equal, if an infinite amount of simulations would have been performed [28]. Also, as the CCR model is a linear optimization problem, the simulation-based maximum efficiencies will always be lower or equal to the CCR efficiencies. The difference of both scores, along with the efficiency acceptability interval indices, provides an indication of the likelihood of finding weight vectors that result in a high country score. The largest differences between the CCR (best-case) and the simulation-based maximum efficiency were found for Venezuela, Kenya and Luxembourg with differences of 0.430, 0.332 and 0.310, respectively. Among the countries considered efficient by the CCR, the largest differences were found for Luxembourg, Haiti and Algeria (0.310, 0.264, 0.231). This means that, even though these countries can be efficient based on the CCR model, the corresponding weight vectors are limited. This indicates that relying on the CCR model only might be misleading, as some countries are considered efficient even though efficiency is reached for very few and unlikely weight vectors.

Finally, Table 8 also shows the average efficiency, taken as the arithmetic average of all the simulations. Once again, South Korea, Singapore and Canada are the most efficient countries with average efficiency scores of 0.911, 0.852 and 0.668, respectively. On the lower end is again Togo, the DRC and Nigeria. It is important to note that this average efficiency can be used to rank the countries as it is highly unlikely that two or more countries have equal values. Furthermore, a ranking based on the average makes more sense because it is not just driven by the best-case scenario of the CCR model.

Interestingly, even though Libya, among others, reaches maximal efficiency in the CCR model, its average efficiency is extremely low (0.035). In fact, it is lower than numerous countries that do not reach efficiency. Therefore, Libya reaches unitary efficiency only for a very small number of randomly selected weight vectors and hence should not be seen as a country with an overall high electricity supply resilience. As shown in Table 4, only three efficiency reducts make it efficient. These are indicators {5, 6}, {6, 7} and {2, 6, 12}. In particular, its electricity import dependence is excellent as it produces much more electricity than it consumes and, therefore, can, in the case of shortage, easily cover its own demand. But this particular situation does not mean that Libya’s electricity supply is resilient holistically, as 8 out of its 12 indicators perform poorly. On the other end, Denmark is the inefficient country with the highest average efficiency (0.489). In fact, Denmark actually scores high on all of its indicators, except for its EAF (i₇, 49% of Denmark’s electricity production comes from wind energy which has a low EAF) and its insurance penetration (i₉). Nevertheless, for each weight vector, there are still other countries that perform better. However, based on the average efficiency, one can safely state that Denmark’s electricity supply resilience is higher than Libya’s. Hence, compared to the CCR model, the stochastic model allows us to distinguish which countries are more robustly deemed efficient than others, even though they all may be considered as efficient in the CCR model. Furthermore, the simulation-based analysis ranks many inefficient countries higher than several efficient ones in the CCR model. Therefore, for policymaking, it is crucial to consider the CCR model in combination with the robust efficiency model, as it gives a more realistic and broader interpretation.

To confirm the robustness of these results, a sensitivity analysis was performed which investigated how the outputs of a model are affected by varying its inputs and which inputs have the highest effect on the results. If the variations in the inputs result in small variations of the outputs, then the model is considered robust. In the present study, the sensitivity analysis was applied on the Monte Carlo simulation by removing each indicator one by one and keeping the others. The Spearman’s rank correlation coefficients (rho) was calculated between the average country efficiencies for the complete indicator set and one indicator removed at a time. It can be seen from

Table 9. Spearman’s rank correlation coefficients (rho) between vectors of average efficiencies when removing the 12 indicators one by one. All correlation coefficients are significant at the 0.01 level.

Indicator Removed	Correlation
1 (input)	0.998
2 (input)	0.872
3 (output)	0.999
4 (output)	0.998
5 (output)	0.995
6 (input)	0.871
7 (output)	0.996
8 (output)	0.998
9 (output)	0.999
10 (output)	1.000
11 (input)	0.874
12 (output)	0.999

that all the coefficients are very high and significant at the 0.01 level. The lowest value is for i₂, where the correlation factor between the vectors of average efficiencies is ca. 0.897. This still represents a very high correlation and the general trend in the ranking is preserved. Thus, it can be concluded that the results are robust.

The distribution of performance of the countries presented in this section shows how robust the results are, which is a key interest of decision-makers. Once the robustness is analyzed, it is important for decision-makers to verify the rank of the countries. In fact, a certain score could lead to a high, average or low rank, depending on the performance scores of the other countries. In Section 4.4, it will become clear that there is no one-to-one relationship between scores and ranks.

4.4. What Is the Univocal Ranking of the Countries?

The previous three sections discussed country scores. However, a certain rank can be achieved by different scores and a certain score can result in different ranks. In order to analyze the rank distribution, this section shows the country ranks computed with the simulation-based Monte Carlo analysis (see

Table 10. Best and expected ranks for selected countries obtained by the simulation-based Monte Carlo analysis. The three highest efficiency rank acceptability indices (ERAI) are shown with their corresponding ranks. The ERAIs are expressed in %. The number in parenthesis after a country’s name is its overall rank according to the expected rank ER0. The full table for all countries is available in the ESI, Table S9.

Country	Best (Ro*)	Expected (ERo)	Rank	ERAI	Rank	ERAI	Rank	ERAI
South Korea (1)	1	3.6	1	46.8	2	25.5	3	5.3
Canada (2)	1	4.1	3	21.5	1	20.9	6	18.1
Singapore (3)	1	4.6	2	42.0	1	25.2	3	4.9
Switzerland (6)	1	8.0	8	14.2	9	12.5	5	8.3
Denmark (11)	2	12.4	10	14.5	11	14.3	9	13.9
USA (28)	1	30.1	30	4.6	29	4.4	26	4.2
UK (32)	2	32.8	35	4.3	30	4.2	30	4.2
Japan (55)	2	53.8	50	2.9	50	2.9	50	2.9
Turkmenistan (77)	1	76.8	66	2.7	77	2.6	70	2.5
Costa Rica (79)	1	79.1	102	2.7	103	2.6	99	2.4
Uruguay (83)	2	83.9	89	3.6	87	3.2	95	3.0
United Arab Emirates (88)	3	87.3	75	2.9	94	2.0	111	1.9
Syria (98)	10	98.9	105	3.5	86	2.8	78	2.7
Niger (99)	44	99.1	89	3.6	87	3.5	87	3.5
Libya (129)	1	126.8	138	12.0	139	11.4	137	10.7
Nigeria (139)	32	135.4	140	31.6	139	19.2	138	8.6
Congo, Dem. Rep. (140)	120	137.9	140	42.1	139	16.5	137	9.0

). For the optimal weight vectors, the 31 efficient countries obviously have rank 1 as their best. All the inefficient countries rank at best second. Additionally, the expected rank ($E{R}_o$) allows us to differentiate all countries, including the efficient ones. As with the average efficiency, the $E{R}_o$ can be used to build a univocal country ranking. Therefore, the $E{R}_o$ once again confirms that the top place is taken by South Korea ($E{R}_o$ of 3.6). As shown in Section 4.3, the close followers are Canada and Singapore ($E{R}_o$ of 4.1 and 4.6, respectively). In conformity with previous results, the DRC is the country with the lowest $E{R}_o$ (137.9).

Furthermore, it is interesting to analyze the distribution of the ranks. The countries with the most concentrated ranks are the top- and bottom-performing ones (e.g., South Korea, Singapore, the DRC, Canada and Nigeria, among others). In fact, these are clustered at the top or at the bottom, hence it is more likely for them to have large ERAIs. On the contrary, the United Arab Emirates (UAE), Costa Rica and Luxembourg show only small ERAIs. In fact, the sum of their three largest ERAIs is only 6.8%, 7.7% and 7.8%, respectively. An extreme case is Libya, which according to the weight vectors considered, can attain every possible rank. However, Libya’s $E{R}_o$ is 126.8, indicating that it most likely does not rank high. It can be seen that Denmark performs much better than Libya as its $E{R}_o$ is 12.4. Switzerland, the country that makes Denmark inefficient, ranks slightly better at 8.0. Finally, the tie between Niger and Syria is also reflected through their almost identical $E{R}_o$ (99.1 and 98.9, respectively).

After the analysis of scores and ranks of the countries, it would be valuable for decision-makers to identify close competitors. These would help to identify strengths and weaknesses, therefore supporting the development of realistic targets and appropriate policies. This leads to Section 4.5 below.

4.5. How Well Does One Country Perform in Comparison to Another?

Pairwise efficiency outranking indices (PEOI) are used to compare the performance of countries between each other. It represents the share of simulations where a country performs at least as good or better than another (see

Table 11. Pairwise efficiency outranking indices (PEOI) for selected pairs of countries. The values indicate the shares (in percentage) of weight vector samples for which a country has an efficiency score not worse than another (i.e., at least as good). Due to the large number of countries in the data set, the complete table is available in the ESI, Table S8. Countries hereby shown include the best- and worst-performing ones, ties and some of the ones discussed in this paper.

Country	Canada	Congo, Dem. Rep.	Denmark	Libya	Niger	Nigeria	Singapore	South Korea	Switzerland	Syria	Togo
Canada	100.0	100.0	100.0	100.0	100.0	100.0	34.1	33.9	97.1	100.0	100.0
Congo, Dem. Rep.	0.0	100.0	0.0	28.0	0.5	46.1	0.0	0.0	0.0	0.1	32.3
Denmark	0.0	100.0	100.0	99.9	100.0	100.0	9.0	5.1	27.9	100.0	100.0
Japan	0.0	100.0	5.6	100.0	97.6	100.0	3.9	3.8	0.3	99.2	100.0
Libya	0.0	72.0	0.1	100.0	4.4	98.2	0.0	0.0	0.0	1.9	62.3
Niger	0.0	99.5	0.0	95.6	100.0	99.6	0.0	0.0	0.0	49.9	99.5
Nigeria	0.0	53.9	0.0	1.8	0.4	100.0	0.0	0.0	0.0	0.0	42.3
Singapore	65.9	100.0	91.0	100.0	100.0	100.0	100.0	41.7	76.1	100.0	100.0
South Korea	66.1	100.0	94.9	100.0	100.0	100.0	58.3	100.0	77.8	100.0	100.0
Switzerland	2.9	100.0	72.1	100.0	100.0	100.0	23.9	22.2	100.0	100.0	100.0
Syria	0.0	99.9	0.0	98.1	50.1	100.0	0.0	0.0	0.0	100.0	97.7
Togo	0.0	67.7	0.0	37.7	0.5	57.7	0.0	0.0	0.0	2.3	100.0
UK	0.0	100.0	11.3	100.0	100.0	100.0	7.9	6.0	0.8	100.0	100.0
Uruguay	0.0	100.0	0.3	92.0	66.8	99.1	0.1	0.3	0.0	66.0	100.0
USA	0.0	100.0	13.5	100.0	100.0	100.0	9.0	7.8	1.2	100.0	100.0

). PEOIs become extremely useful for identifying close competitors. In country rankings, relevant comparisons can be made in smaller peer groups, i.e., with similarly performing or geographically neighboring countries. For example, Syria and Niger are in a tie (50.1%). Hence, these two countries can benchmark their performance and closely monitor each other over time. If a country turns out to outperform its peer, then the policies of the two countries can be analyzed and successful strategies identified.

Furthermore, as expected, the top-performing countries (i.e., South Korea, Singapore and Canada) only rarely score lower than other countries. Similarly, on the other end of the spectrum, Togo, the DRC and Nigeria rarely achieve higher efficiencies than the other countries. Furthermore, even though Libya is deemed efficient, it performs better than Denmark in ca. 1‰ of the simulations only (the CCR model deemed Denmark as inefficient). Once again, this confirms that Denmark has a better electricity supply resilience than Libya.

Another interesting case is how Switzerland performs compared to the top three countries. Switzerland wins against Singapore and South Korea in ca. 22%–24% of the simulations, but wins against Canada for only 2.9% of the cases, even though the average efficiency of Canada is lower than that from South Korea and Singapore. This is due to the fact that the Euclidian distance between Switzerland’s and Canada’s indicator performances is smaller, meaning that Switzerland and Canada have more similar indicator values compared to Singapore and South Korea. Hence, for the weight vectors that are advantageous for Switzerland, Canada almost always performs slightly better, whereas South Korea and Singapore do not.

4.6. How Does the Performance of Countries Vary According to Changes in Selected Indicators?

The previous sections presented results on the performance of countries, their potentials for improvement and their position in a univocal ranking. In this section, DEA is used as a means to make country-specific improvement potential evaluations. In this way, policymakers can detect early warning signals and explore different future pathways, leading to more effective decisions and subsequent implementation of strategies to reach the targets. In the present study, two types of country analyses were applied:

• Obtain a new country ranking, based on updated indicator values according to specific scenarios (Singapore, Section 4.6.1).
• Determine the minimal required improvements on the indicators in order to become an efficient country (Japan, Section 4.6.2)

4.6.1. Country Analysis: Singapore’s Electricity Supply Resilience

Located in Southeast Asia, the small sovereign city-state and island country of Singapore is often referred to as the Switzerland of Asia (e.g., [86]). It portrays a high standard of living [87], a strong economy [88] and political stability [89]. Furthermore, Singapore is one of the largest and most competitive financial centers in the world [90]. Additionally, it is one of the world’s top five oil trading and refining hubs [91] and is home to the world’s second busiest container port [92]. Regarding the energy sector, Singapore imports mainly petroleum products, crude oil and natural gas. Furthermore, natural gas is the source for 95.2% of the electricity produced [93]. Currently, the Singaporean government is trying to diversify its energy supply, in order to be able to better cope with supply disruptions and price increases [94]. Its Economic Strategies Committee (ESC) published a report aiming at ensuring energy resilience and sustainable growth [95]. It contains five key strategies: (1) diversifying the energy sources, (2) enhancing infrastructure and systems, (3) increasing energy efficiency, (4) strengthening the green economy and (5) pricing energy right.

Overall Singapore, deemed as efficient with the CCR model (see Table 3), has the second highest average efficiency (see Table 8) and the third best expected rank (see Table 10). Its outstanding infrastructure is ranked the second best in the world [96], which is reflected by its low SAIDI (i₁) of less than a minute per customer per year, and by the fact that there are almost no fatalities related to electricity production (i₂). Furthermore, being among the top performers in controlling the levels of corruption (i₃) and political stability (i₄), Singapore has a stable environment that enables clear policymaking and transparent directives. However, its electricity generation mix diversity (i₅), consisting of 95.2% of natural gas, makes it particularly vulnerable to potential disruptions. Additionally, on its electricity import dependence (i₆), Singapore’s electricity grid is currently not strongly connected in the region, even though this might change in the near future as there are growing efforts to establish electricity interconnections in Southeast Asia, in particular between Singapore, Malaysia and Indonesia [97]. Nowadays, Singapore basically produces what it consumes. If there were a shortage for any reason, only limited amounts of electricity could be imported from its neighbors. Furthermore, its EAF (i₇) is high, which is attributed to the fact that the electricity is mainly produced by natural gas (natural gas has an EAF of 0.85 [74]). Also, Singapore excels with a high GDP per capita (i₈) and an outstanding government effectiveness (i₁₀). This is the result of a series of successful developments undertaken in the past decades [98]. However, its insurance penetration as a percentage of GDP (i₉), a central part for accelerating recovery processes [99], is not expected to increase in the future [75]. Regarding the average outage time (i₁₁), Singapore is the top-performing country having almost no interruptions at all and even if there happens to be one, it usually is so short that it is hardly noticeable by the population. Lastly, Singapore’s ease of doing business (i₁₂) is already the second highest in the world [100].

Based on these premises, two scenario analyses were developed and discussed in the following two sections (Section 4.6.2 and Section 4.6.3).

4.6.2. Scenario 1: 8% Solar Photovoltaic Electricity Production

One of the weaknesses of Singapore’s electricity supply is its generation mix diversity (i₅) consisting of 95.2% of natural gas, which makes it particularly vulnerable to potential disruptions. To address this issue, one of the government’s current strategies is to diversify its sources [101], by for example increasing the share of renewables [102]. In particular, by 2030, Singapore has a potential of producing 8% of its electricity by solar photovoltaics (PV). This change affects the following indicators:

• i₂; improvement from 0.124 to 0.115 fatalities/GWeyr: solar PV has lower fatality rates than natural gas [71].
• i₅; improvement from 0.11 to 0.22: replacing natural gas generation by solar PV improves the mix diversity.
• i₇; deterioration from 0.85 to 0.79: solar PV has a lower EAF than natural gas [74].

Therefore, as the performance of i₂ and i₅ increases, but that from i₇ decreases, it is not yet clear if this will have a positive effect on Singapore’s electricity supply resilience. However, results show that, in this particular case, it is advisable to pursue this strategy as Singapore’s score effectively improves (see

Table 12. Results for scenario 1. The efficiency acceptability interval indices (EAIIs) and the ERAIs are given in %. Even though Singapore does not get the first position, having 8% of solar photovoltaic (PV) generation improves its electricity supply resilience.

Efficiency Interval	Average Efficiency	[0.0–0.1]	[0.1–0.2]	[0.2–0.3]	[0.3–0.4]	[0.4–0.5]	[0.5–0.6]	[0.6–0.7]	[0.7–0.8]	[0.8–0.9]	[0.9–1]
South Korea	0.905	0.0	0.3	0.0	0.2	0.3	1.1	4.3	11.9	19.7	62.2
Singapore new	0.872	0.1	0.2	0.2	0.3	0.9	2.7	6.2	15.6	22.8	51.0
Canada	0.680	2.6	6.5	6.8	6.5	7.5	8.3	8.1	8.4	8.6	36.7
Singapore original	0.852	0.0	0.1	0.3	0.6	1.6	4.6	9.4	15.6	19.9	47.8
ERAI	ERo	Rank 1	Rank 2	Rank 3	Rank 4	Rank 5	Rank 6	Rank 7	Rank 8	Rank 9	Rank 10
South Korea	3.7	43.2	26.8	6.2	3.1	4.1	2.3	0.9	1.3	0.9	0.7
Canada	4.0	20.4	11.3	22.2	7.3	8.2	17.9	3.6	1.5	4.2	1.3
Singapore new	4.3	27.7	40.6	5.1	4.5	2.7	2.6	0.9	1.4	1.6	1.9
Singapore original	4.6	25.2	42.0	4.9	3.2	2.6	2.5	2.3	1.4	1.6	1.0

). In fact, even though Singapore keeps its second position for the average efficiency and the third rank according to the ERAIs, its performance improved, hence reducing the gap to South Korea and Canada.

Overall, this shows that it is not possible to predict a priori if an increase of the share of renewables is good or bad for resilience. It has to be studied on a case-by-case basis. In the present example, it turned out that increasing the share of solar PV generated electricity to 8% is positive for Singapore’s electricity supply resilience.

4.6.3. Scenario 2: Singapore in 2030

The second scenario for Singapore assumes that in the year 2030 it not only reaches an 8% share of solar PV, but that the following two indicators change as well:

• i₆; improvement from 0.98 to 0.92: Singapore’s electricity grid is currently not strongly connected in the region, but its import dependence is expected to decrease as a result of planned interconnections with Malaysia and Indonesia [97] and according to the projected production and consumption in 2030 [103].
• i₈; improvement from 51,809 to 67,360 USD/capita: according to predictions, Singapore’s GDP will increase to 67,360 USD/capita in 2030 [104].

With the new values for i₂, i₅, i₆, i₇ and i₈, Singapore’s expected performance in 2030 will improve even more (see

Table 13. Results for scenario 2. The EAIIs and the ERAIs are given in %. According to this scenario, Singapore has the best electricity supply resilience in the world.

Efficiency Interval	Average Efficiency	[0.0–0.1]	[0.1–0.2]	[0.2–0.3]	[0.3–0.4]	[0.4–0.5]	[0.5–0.6]	[0.6–0.7]	[0.7–0.8]	[0.8–0.9]	[0.9–1]
Singapore new	0.912	0.0	0.1	0.1	0.2	0.5	1.5	5.5	8.7	17.2	66.2
South Korea	0.878	0.0	0.1	0.1	0.0	0.4	1.7	5.5	17.8	26.3	48.1
Canada	0.662	3.1	7.2	7.2	7.2	7.0	7.6	7.2	10.3	11.3	31.9
Singapore original	0.852	0.0	0.1	0.3	0.6	1.6	4.6	9.4	15.6	19.9	47.8
ERAI	ERo	Rank 1	Rank 2	Rank 3	Rank 4	Rank 5	Rank 6	Rank 7	Rank 8	Rank 9	Rank 10
Singapore new	3.2	47.2	29.7	3.8	3.3	3.1	1.7	1.1	1.5	1.0	1.2
Canada	3.9	18.8	12.2	24.2	9.1	7.0	17.6	3.5	0.9	3.7	1.2
South Korea	4.1	27.5	39.0	7.1	4.5	3.4	1.7	1.9	1.4	1.4	2.0
Singapore original	4.6	25.2	42.0	4.9	3.2	2.6	2.5	2.3	1.4	1.6	1.0

), and it will overtake South Korea and Canada, i.e., reaching the first position as the most resilient country in the world regarding electricity supply.

4.6.4. Country Analysis: Japan’s Electricity Supply Resilience

In the three decades following 1960, Japan’s economy has boomed with average GDP growth rates of up to 12.9% [105] and is currently the third-largest in the world [106]. This lead it to become the fifth best country worldwide, according to a global index from the U.S. News and World Report [107]. Its population enjoys the highest life expectancy in the world [108] proving that the quality of life is high [87]. Japan also ranks particularly high for entrepreneurship (2nd [107]) which portrays its numerous innovations and comparatively high number of patent applications [109]. Regarding the energy sector, Japan has gone through tremendous changes since the 2011 Tōhoku earthquake and tsunami [110]. In fact, due to the Fukushima Daiichi nuclear disaster, the nuclear produced electricity was replaced almost instantaneously by mostly imported oil, gas and coal [68], making Japan more vulnerable to supply disruptions. As this option comes with risks and drawbacks (e.g., import dependence, environmental concerns [111], financial burden [112]), the Japanese government published a revised version of its Strategic Energy Plan (SEP) in 2014 [113]. Its goals are to ensure a stable supply, enhance economic efficiency on the premise of safety and pursue environmental suitability.

Based on these premises, Japan’s low SAIDI (i₁) and low fatality rates related to electricity production (i₂) reflect the overall high quality of its infrastructure [96]. Furthermore, having low levels of corruption (i₃) [114] and high political stability (i₄), Japan has a stable environment that enables clear policymaking and transparent directives. Even though Japan’s performance on the electricity mix diversity (i₅) is far from alarming, it shows slightly lower scores compared to some years ago, as its mix diversity has since the Fukushima Daiichi nuclear disaster increasingly been dependent on imported fossil fuels. Regarding the electricity import dependence (i₆), due to its geographical location, Japan’s electricity grid is currently isolated [115], which means that the production simply follows the consumption pattern. Recently, an interconnected Northeast Asia (NEA) grid has received increasing attention. However, modest economic benefits are a major problem for its implementation [115,116]. Additionally, if this plan were to be realized, it is likely that Japan would overall import electricity which would further increase its dependence on its neighbors. Furthermore, its EAF (i₇) is high, which is attributed to the fact that the electricity is mainly produced by fossil fuels (Fossil fuels have EAFs of 0.85 [74]). Also, Japan shows high GDP per capita (i₈) and government effectiveness (i₁₀). Its insurance coverage (i₉) is expected to grow [75], as a result of increased awareness of potential losses due to frequent recent natural catastrophes [117]. In fact, Japan is the fourth most exposed country in the world [118]. This is probably also the reason why its average outage time (i₁₁) is long (4 hours), even though there is on average only one interruption per citizen every tenth year [100]. Lastly, Japan currently ranks 34th in the ease of doing business ranking [100].

Overall, Japan performs well on most of its indicators. Its average efficiency is 0.263 (42nd, see Table S7) and it ranks 55th according to its expected rank. However, it still does not reach efficiency (CCR score of 0.991, see Table 3). Based on these premises, Section 4.6.5. describes Japan’s scenario analysis and its results.

4.6.5. Scenario 3: Required Electricity Generation Portfolio to Make Japan Efficient

Unlike the two scenarios for Singapore, where the original calculations were made with an updated data set, the scenario for Japan is an optimization scenario, where the aim was to find the minimal improvement that makes Japan efficient. The first step was to determine which indicators should be varied. The second step consisted in calculating the minimum required performance changes for these indicators in order to make Japan efficient. The chosen scenario investigated if by varying only the electricity generation portfolio Japan can become efficient. As a consequence, the minimum improvements for indicators i₂, i₅ and i₇ were calculated to make Japan efficient (see

Table 14. Required minimum performance on i2, i5 and i7 in order to make Japan efficient.

Indicator	i2: Severe Accident Risk	i5: Electricity Mix Diversity	i7: Equivalent Availability Factor
Unit	Fatalities/GWeyr	Normalized Shannon Index	%
Original performance	0.0782	0.6526	78.14
Required performance	0.0765	0.6664	79.80

). These improvements were obtained by running the CCR model and considering a constant, proportional improvement over the three indicators.

The third step was to calculate to what electricity generation portfolio these new values correspond. Considering the 10 fuel sources that are currently producing electricity in Japan (see

Table 15. 10 closest electricity generation portfolios to Japan’s current one (listed in increasing order of Euclidian distance).

Technology Share	Coal	Oil	Natural Gas	Biofuels	Waste	Nuclear	Hydropower	Geothermal	Solar PV	Wind
Original	32.96%	9.85%	39.36%	3.32%	0.66%	0.91%	8.76%	0.25%	3.44%	0.50%
Portfolio 1	32.19%	10.05%	38.84%	4.67%	0.52%	1.38%	8.70%	1.90%	1.75%	0.02%
Portfolio 2	30.71%	9.65%	40.27%	4.15%	1.89%	0.10%	8.16%	2.98%	1.64%	0.46%
Portfolio 3	32.34%	9.02%	37.30%	4.99%	2.75%	1.09%	10.12%	1.45%	0.86%	0.07%
Portfolio 4	32.20%	8.50%	41.23%	3.98%	1.02%	0.76%	6.24%	2.62%	1.78%	1.67%
Portfolio 5	31.14%	9.51%	41.61%	4.69%	0.52%	0.58%	6.25%	2.02%	1.76%	1.91%
Portfolio 6	33.73%	8.43%	38.17%	2.10%	2.13%	3.92%	9.18%	0.73%	0.64%	0.96%
Portfolio 7	32.04%	11.35%	36.59%	4.01%	2.38%	2.33%	10.30%	0.19%	0.29%	0.52%
Portfolio 8	31.72%	7.28%	40.74%	5.02%	2.88%	0.98%	5.88%	1.83%	3.53%	0.13%
Portfolio 9	34.52%	7.01%	38.91%	1.96%	3.31%	1.51%	8.13%	2.34%	1.28%	1.02%
Portfolio 10	31.58%	9.76%	37.66%	1.66%	3.60%	3.38%	11.27%	0.66%	0.31%	0.11%

), there are numerous portfolios that result in the required performance on indicators i₂, i₅, i₇. From an optimization point of view, this is equivalent to an underdetermined system (3 indicators for 10 technologies; fewer equations than unknowns). Hence, for each of the portfolios fulfilling the constraints on the three indicators of Table 14, its Euclidian distance was calculated. Performing the calculations for 10 million randomly selected portfolios, Table 15 shows the 10 closest portfolios to Japan’s current one (smallest Euclidian distances). In fact, these represent the ones that require the least amount of change in order to make Japan efficient.

Although the 10 portfolios do not show large differences, they can provide different policy perspectives. For example, if the target is to reduce the amount of fossil fuels (coal, oil and natural gas), then portfolios 3 and 10 are most suitable. These two portfolios come with an increase of biomass (biofuels and waste combined), nuclear, hydropower and geothermal electricity, whereas solar PV and wind decrease. Portfolio 8 is the only one that increases the share of solar PV, which already today is on a sharp rise [68]. However, no portfolio jointly increases the shares of solar PV and wind. Overall, portfolio 8 is closest to the Japanese government’s goals [113,119,120], as (1) it decreases the amount of coal and oil, (2) only slightly increases the share of natural gas (currently the cleanest fossil fuel, especially if used in combination with carbon capture and storage [121]), (3) increases the shares of biomass, solar PV, geothermal and nuclear electricity, but (4) decreases the share of hydropower and wind electricity.

5. Conclusions and Policy Implications

Starting from a set of 12 indicators, this study uses two DEA models to assess the electricity supply resilience of 140 countries. First, the classical CCR model deemed 31 countries as efficient (score of 1), and hence resilient. For these countries, it is possible to find at least one weight vector under which no other country performs better. To gain insights into these efficient countries, a novel algorithm that allows us to calculate their efficiency reducts was developed. This demonstrated which minimal combinations of indicators can make a country efficient. Furthermore, another novel algorithm was developed to identify the efficiency constructs of each inefficient country. In other words, the minimal subsets of countries that make it inefficient was computed.

Second, a robust efficiency analysis was applied. To the authors’ best knowledge, the present study represents the first application of such an analysis to a country ranking. A distribution of efficiency scores for each country is calculated, which provides information about ranking stability as it depicts the likelihood of a country scoring in a certain performance bin. Additionally, it allows calculating both the average efficiency and the expected rank of a country that can be used to establish a univocal country ranking. The robustness analysis also allows computing the pairwise efficiency outranking indices.

Finally, scenario analyses for Singapore and Japan were carried out. For Singapore, the analysis consisted in verifying if its current energy policies lead to an even higher resilience, even though Singapore is already efficient according to the CCR model. Results showed that increasing electricity production from solar PV is beneficial for Singapore’s electricity supply resilience. In contrast, as Japan is an inefficient country, an optimization problem was solved to determine the minimal required improvement on selected indicators in order to make it efficient. From a policymaking perspective, this is equivalent to finding the optimal way to allocate resources in order to increase its rank. By strictly considering technologies that are already producing electricity, results showed that it is possible to reach efficiency by only slightly changing the production shares.

Overall, this study showed that combining the CCR model, including its efficiency reducts and constructs, with the robust efficiency analysis provides a holistic assessment methodology that can be applied to the present electricity supply resilience assessment of 140 countries, but also similar problems in other domains to support robust decision-making by stakeholders. In fact, even though the CCR model is the most widely used, its results are limited and can be misleading. While the CCR model provides a clear differentiation of scores for inefficient countries, it does not differentiate between the efficient ones. Therefore, building a univocal ranking is impossible. Furthermore, the CCR model provides a best-case scenario, as it computes the most advantageous weight vector for each country separately. As a result, such scores may not be representative, because they might only be achieved for a very limited number of weight vector combinations. Therefore, the authors believe that by using the hereby developed methodology, policymakers would have a broader view of how the alternatives under study perform. Many policies are indeed based on the results of indices obtained by aggregating average values without considering uncertainty or robustness of the results. This might lead to ill-informed decisions. Accounting for uncertainty in input data and problem structure brings a dynamic component to the usual indices that are static.

By considering the CCR and robust efficiency analysis simultaneously, decision-makers can identify close competitors. This provides important learning lessons from comparable countries (so-called benchmarks). Furthermore, this methodology stimulates a multi-disciplinary approach when considering improving the overall performance of a country. In fact, as the indicators are interrelated, multiple specialists should share knowledge in order to tackle the complexity of today’s world. Through collaboration between multiple parties, including research institutions, industry and governmental agencies, it would be possible to develop improvement plans and policies to reach predefined targets. The methodology proposed in this paper could provide an interactive discussion platform to lead the decision-making process.