Energy Policy Concerns, Objectives and Indicators: A Review towards a Framework for Effectiveness Assessment

Abstract

This work presents a review that aims to characterize the policy evaluation practices regarding the public policies on energy, with a focus on the metrics: concerns, objectives, and indicators. As key novelty, emphasis was put into finding attributes and metrics that can be used to assess effectiveness, not only efficacy or efficiency. The concerns and objectives were organized into four categories: Institutional, Environmental, Economic, and Social. For every category, detailed and condensed concerns were identified. It was attempted to find indicators for every condensed concern, which resulted in 15 core indicators.

1. Introduction

Differences among the policy objectives and the policy results are often found. This situation can be intensified if the energy policies (EP) are formulated or implemented, without considering the particular characteristics and capabilities of different countries or regions. EP evaluation has previously been proposed as a tool not only to compare the policy achievements with their objectives but also to assess the “goodness” of policy programs. This is becoming increasingly important as countries, regions, and even municipalities are pressed to develop carbon neutrality plans. Regardless of the jurisdiction, authorities are organizing to face the challenge of climate change mitigation. The efforts can be seen from top-level international agreements, as the Paris Agreement [1], and the Sendai Framework [2], to city-level climate-neutral plans (e.g., the plans developed in Berlin, Germany [3], Ann Arbor, USA [4], Bayside, Australia [5] or Freiburg, Germany [6]). Additionally, the United Nations 2030 Agenda [7] has established the Sustainable Development Goals (SDGs) as crosscutting international alignment principles. They represent a set of universal goals that meet the urgent environmental, political, and economic challenges facing the world. To meet these goals, Governments will need better planning tools and to monitor progress towards the targets; more specifically, tools that enable evaluating policies in multidisciplinary ways, allowing decision makers to analyze the policies in an integral way.

As an intended contribution to the planning and management of climate action plans, this work addresses the problem of EP evaluation, with a focus on effectiveness [8]. It starts with the conceptual differences between effectiveness, efficiency, and efficacy, and then moves on to find specific indicators for a comprehensive assessment. Such a set would be useful for ex-post evaluations, but in some cases, it could also be useful for ex-ante evaluation and thus assist the design of better policy programs. Therefore the main motivations of this work have been to create a support document that enables a more structured and comprehensive planning process (and that does not leave important aspects forgotten); planning programs that have a higher chance of succeeding, in practice; and supporting the good use of resources (public and/or private).

With the aim of evaluating EP and provide information that facilitates their adjustment to distinctive characteristics, a set of concerns was intended to be identified through this review. Concerns that were organized under the different proposed categories and subcategories. These concerns were organized under condensed concerns by identifying their final objective.

An indicator identification process was followed. Indicators were selected as the assessment metrics that can provide a comparison between the policies defined objectives and their achievement, therefore selected as an effectiveness measurement tool. The identification of concerns ends objectives has also been done through this report, serving as cornerstones for the identification of indicators.

The objective of this work was to review and systematize with a scientific-based methodology, the available information regarding EP effectiveness and its assessment indicators. The review led to the identification of trends in the subject. The main contribution of the document is the identification of the multidisciplinary attributes to assess energy policy effectiveness. No other review has been found available in the literature in which EP effectiveness is characterized holistically.

The paper is structured into 6 sections. Section 2 addresses the concepts of policy success: efficacy, efficiency, and effectiveness. The different EP effectiveness categories reported through the selected documents are presented in Section 3. Section 4 focuses on the systematization of energy policy concerns and objectives. Section 5 presents energy policy indicators, where the concerns of Section 4 have been matched with corresponding indicators. Section 6 includes a Discussion, and finally, the conclusions are presented in Section 7.

2. Policy Success: Efficacy, Efficiency and Effectiveness

A sensible goal of policy evaluation is to assess whether or not they are “successful”. “Success” is defined as “the accomplishment of an aim or purpose” [9]. This is very much in line with the definition of “efficacy”, which is defined as “the ability, especially of a medicine or a method of achieving something, to produce the intended result” [10].

A strict or “hard” approach to the evaluation of “success” or “efficacy” could therefore rely on the comparison between what is achieved and what was the target or goal; e.g., a program that has the target of promoting the installation of 80,000 solar collectors would be “successful” or “efficacious” if it achieves 80,000 or more installations, while it would be “unsuccessful” if it achieves equal or less than 79,999 installations.

The example above illustrates as well that such a “hard” approach to success evaluation is probably too strict. Under this approach, failing by a large amount or by a small amount is put under the same label. It could also serve as a stimulus to establish artificial and sometimes counter-productive low targets, to increase the probability of achieving the results.

Therefore, more flexible and comprehensive metrics are needed to assess policy success. This is where the concept of “effectiveness” may come into play. Although, sometimes seen as equivalent to “success”, it is formally defined as “the degree to which something is successful in producing a desired result” [11]. The word “degree” opens room for a more gradual evaluation, clearly allowing for differentiation between failing by a large amount vs. failing by a small amount, as well as for succeeding by a small amount vs succeeding by a large amount.

Furthermore, “effective” has already been used, e.g., in the field of medicine as different from “effective” in the sense that it is something that is evaluated from the real-world use or implementation, including informal feedback from doctors and patients, rather than from controlled experiments alone [12].

It may be of interest to bring into consideration also the concept of “efficiency”. This is clearly distinct from both efficacy as from effectiveness, assessing what was achieved concerning the means that were put into the program. While of course being efficacious is more important than being efficient, being efficient is also important. It implies that resources (e.g., funding to invest), which typically are limited, are being well used. By being efficient, more effect can be achieved from a given amount of resources.

As a more nuanced concept than efficacy, effectiveness thus has the potential to encompass the components of taking into consideration contextual variables and of assessing “success” in a quite modulated manner. It can even be advocated that, while privileging the evaluation of the achievement of results, a truly effective program should assess those results concerning the contextual variables (geographical, economic, and cultural) rather than by just comparing it to the initial goals of the program. This is so because these often are not declared and, even when they are, their choice is inherently somewhat arbitrary. Effectiveness can thus incorporate some dimension of efficiency into the assessment.

3. Materials and Methods

The academic databases Sciencedirect, Google Scholar, and JSTOR, together with the web browser Google were used for gathering documents. All the sources were useful to identify documents that characterize energy policy effectiveness. The expressions under which the search was done were “Energy policy evaluation” (academic and open-source); “Energy policy effectiveness” (academic source); “Energy policy assessment” (academic source); “Energy policy efficiency” (academic source). The selection of the documents to be reviewed was done by their title, abstract content, and keywords. After doing so, the criteria for the document selection for the analysis was having at least one concern that assesses energy policies. A summary of this process is presented in

Table 1. Search terms by source and documents reviewed and adopted by each.

Search Term	Source	Documents Reviewed	Documents Adopted
Energy Policy Evaluation	Sciencedirect	18	9
Energy Policy Evaluation	Google	17	10
Energy Policy Assessment	Sciencedirect	16	10
Energy Policy Effectiveness	Sciencedirect	9	5
Energy Policy Effectiveness	Google Scholar	20 (3 repeated)	15
Energy Policy Effectiveness	JSTOR	5	3
Energy Policy Evaluation	Google Scholar	19 (3 repeated)	13
Energy Policy Evaluation	JSTOR	5 (1 repeated)	1

.

The detail of each search source is presented in Appendix A, for which reference and order numbers have been provided and that have been used for the organization and analysis of the documents presented in the document.

Analysis: Geographical Scope

When analyzing the scope or level of application of the energy policies evaluations (local, regional, national, international, or a combination among them), it is noted in

Table 2. Energy policy evaluation documents found per geographical level of analysis.

Level	Quantity	Document Numbers (from Tables in Appendix A)
National	40	1–4; 6–11; 13; 16–18; 20; 21; 24–26; 28; 30–32; 34; 35; 45; 46; 50; 52; 53–55; 58–64; 66.
Regional (State)	9	14; 15; 19; 27; 36; 38; 39; 47; 56.
International	11	29; 33; 37; 40; 41; 43; 44; 49; 51; 57; 65.
Combined	3	12; 23; 22.
Local	3	5; 42; 48.

that the highest number of the reviewed documents regards the national level (with 40 documents), with a considerable difference above the other alternatives. The national scope is followed by the international scope (with eleven documents). Some documents regard all the levels of application or a combination of more than one level. Another observation is that only three locally focused documents were found.

Concerning the time frame shown in

Table 3. Energy policy evaluation documents found per time-frame.

Time Framing of the Process	Quantity	Documents Numbers (from Tables in Appendix A)
Ex ante	10	1; 5; 15; 22; 23; 29; 31; 40; 47; 51.
Ex post	34	4; 8–10; 18; 21; 27; 32; 33; 35; 36; 38; 39; 41–63; 65; 66.
In progress	7	19; 28; 30; 34; 46; 48; 64.
Combined	2	14; 17.
Not inferred	13	2; 3; 6; 7; 11–13; 16; 20; 24–26; 37.

, almost all of the retrieved documents state or enabled us to infer the period of the evaluation (summing 53 documents), against 13 documents where that was not possible. The time framing of the process, according to the reviewed documents, can be ex ante, in progress, or ex post. The ex post energy policy evaluation approach was the most often found (34 documents), followed by the ex ante approach (10 documents). It can be noticed that only two of the reviewed documents (“Moving Toward Energy Efficiency: A Results-Driven Analysis of Utility-Based Energy Efficiency Policies”, Theel and Westgaard, 2017 [13] and “Evaluating Consistency in Environmental Policy Mixes through Policy, Stakeholder, and Contextual Interactions Lieu et al., 2018 [14] declared a combined approach (with more than one possible timing period).

Table 4. Energy policy evaluation documents found per stakeholder participation.

Stakeholder Participation	Quantity	Documents Numbers (from Tables in Appendix A)
Yes	19	3–6; 13; 14; 17; 20; 23; 29; 30; 37; 40; 53; 54; 58; 59, 64.
No	47	1; 2; 5-12; 16; 18; 19; 21; 22; 24–28; 31–36; 38; 39; 41–52; 55–57; 60–63; 65; 66.

shows the participation of stakeholders in the EP evaluation. Stakeholder participation is related to the involvement of actors along the evaluation process. Less than one-third of the documents retrieved report some form of participation from stakeholders (19 out of 66 documents).

The methodological approaches for evaluation were also reviewed and are presented in

Table 5. Energy policy evaluation documents found per methodological approach.

Methodology	Quantity	Documents Numbers (from Tables in Appendix A)
Statistical	17	10; 14; 20; 27; 32; 33; 36; 38; 39; 42–44; 48; 52; 56; 61; 63.
Scenario	12	1; 5; 9; 16; 18; 21; 22; 34; 53; 60; 65; 66.
Qualitative-based approach	19	2–4; 11; 13; 17; 23–25; 28; 31; 37; 40; 41; 45; 46; 57; 58; 64.
Indicators	4	8; 12; 35; 59.
Others	14	6; 7; 15; 19; 26; 29; 30; 47; 49-51; 54; 56; 63.

. As expected, very diverse approaches were found from document to document. The qualitative-based methodology was found as the most popular approach (19 documents). Descriptive and qualitative analysis methods have been considered as qualitative-based. The second more frequent approach is statistical (17 documents). Scenario and modeling techniques have also been identified (12 documents). The category others include the approaches that were found only once e.g., cost–benefit analysis (CBA), return on assets (ROA), etc.

In

Table 6. Energy policy evaluation documents found per expressed objectives.

Motivation	Quantity	Documents Numbers (from Tables in Appendix A)
Policy evaluation	34	6–8; 13–16; 18–22; 27; 30; 32–34; 37; 39; 40; 45; 46; 49; 53–55; 57–62; 64-66.
Policy review	7	23; 24; 26; 29; 35; 36; 51.
Support policy makers	4	3; 5; 17; 20.
Evaluate policy success	16	1; 9–12; 38; 41–44; 47; 48; 50; 52; 56; 63.
Creation of a methodology/framework	5	2; 4; 25; 28; 31.

, it can be seen that when analyzing the motivation stated in the reviewed documents, the most common one is to conduct a policy evaluation itself, observed in 34 documents. Four documents clearly defined their interest in supporting policymakers, while 16 documents target the evaluation of policy success. It can be noted the interrelation between the identified motivations.

4. Results

4.1. Energy Policy Concerns and Objectives

The identification of the concerns most frequently included in the literature and how they are defined led to forming a list of more than 60 concerns. They have been classified into four broader categories, and some subcategories have been formed.

4.1.1. Environmental Concerns

As shown in

Table 7. Energy Policy Environmental concerns found.

Document	GHG	Other Impacts
Lieu et al., 2018 [14].	-	Environmental stakeholders
Abotah, 2015 [15].	Low carbon economy	Environmental protection
Chapman et al., 2016a [16].	GHG emissions	RE Deployment, RE Tech Efficiency
Chapman et al, 2016b [17].	GHG emissions	-
Chen, 2011 [18].	-	Eco-label system
Alyamani et al., 2016 [19].	GHG emissions	-
Lee & Shih, 2010 [20].	-	Environmental protection
Nikolaev & Konidari, 2017 [21].	GHG emissions	-
Hafeznia, Aslani, Anwar, & Yousefjamali, 2017 [22].	-	Climate information
S. J. Li et al., 2017 [23].	-	RE energy production
Yu et al., 2015 [24].	CO2 emissions	Energy savings
Aized et al., 2018 [25].	GHG emissions	-
Arababadi et al., 2017 [26].	GHG emissions	-
Burke & Stephens, 2017 [27].	-	RE minimum requirements
Chang & Fang, 2017 [28].	-	Sustainability in energy supply
García-Álvarez et al., 2018 [29].	-	General environmental awareness
Mirjat et al., 2017 [30].	-	General environmental awareness
Atalla et al., 2018 [31].	Climate change mitigation	-
Delmas & Montes-sancho, 2011 [32].	GHG emissions	Renewable resources
Menz & Vachon, 2006 [33].	-	Resources capacity and quality
Arbolino et al., 2019 [34].	GHG emissions	Renewable share
Thornley & Cooper, 2008 [35].	-	Resource capacity
Jenner et al. 2013 [36]	-	Resources share
Dong, 2012 [37].	GHG emissions	-
Sancho, 2010 [38].	GHG emissions	-
Bersalli, et al., 2020 [39].	GHG emissions	-
Stagl, 2006 [40].	Global warming and climate change	Renewable energy use; Less-pollutant transportation
Hoffman et al., 1977 [41].	GHG emissions	Environmental impact, energy resources
Carley, 2009 [42].	-	Natural resource endowments, RE share
Kilinc-Ata, 2016 [43].	GHG emissions	-

, two subcategories were defined: greenhouse gas (GHG) and other environmental impacts. The burden among the categories could be very thin, as some concerns are related to both, as renewable energy (RE) sources related concerns (e.g., RE standards, RE Deployment and RE Tech Efficiency).

Greenhouse gas (GHG) emissions reduction was found as the predominant concern for evaluating energy policies under an environmental perspective and gathered under the subcategory Environmental-GHG. The GHG emissions concern consideration has been incentivized by the continuous and reinforced influence produced by the definition of international agendas, national targets, agreements, and other international efforts to reduce GHG emissions and minimize climate change.

Other impacts include the concerns that assess the conservation and improvement of the environment, excepting GHG. RE related concerns (e.g., RE standards, RE Deployment and RE Tech Efficiency) have been included in this subcategory.

4.1.2. Economic Concerns

A large set of concerns was found under the economic category, as shown in

Table 8. Energy Policy Economic concerns found.

Document	Cost	Competitiveness
Chapman et al., 2016a [16].	Energy price performance	Market growth and maturity
Theel & Westgaard, 2017 [13].	Energy price and demand	GDP
Abotah, 2015 [15].	Economic feasibility	-
Qiang et al., 2016 [44].	Cost-Effectiveness	-
Lieu et al., 2018 [14].	Supply chain Stakeholders and Service providers	-
Chapman et al., 2016b [17].	Generation Cost, Electricity Price Impact, Levelized cost of electricity (LCOE)	GDP Impact and Market Impact
Energy and Environmental Economics, 2014 [45].	Net Cost	-
Chen, 2011 [18].	Energy demand, energy price	-
Bukarica & Tomšić, 2017 [46].	-	Market barriers
Alyamani et al., 2016 [19].	Energy demand, cost, and price	Utility profits and revenue
Lee & Shih, 2010 [20].	-	Macroeconomic parameters
Collado & Díaz, 2017 [47].	Energy savings	-
Nikolaev & Konidari, 2017 [21].	Internalized total cost	Attractiveness in RES investment
A.J. Chapman [48].	Generation Cost, Electricity Price	GDP Impact and Market Impact
X. Zhao et al., 2016 [49].	Power demand, energy substitution, technological change	-
S. J. Li et al., 2017 [23].	Economic data	-
S. Ma et al., 2020 [50].	Carrier price, demand	Patents
Shrimali et al., 2017 [51].	Cost-effectiveness potential	Investment incentives
Aized et al., 2018 [25].	Costs, supply, and demand	-
Burke & Stephens, 2017 [27].	Financial measures	-
Chang & Fang, 2017 [28].	Economic context	-
Liu et al., 2018 [52].	Costs and profits	-
García-Álvarez et al., 2018 [29].	Electricity price trends, economic situation, security of supply	-
Mirjat et al., 2017 [30].	Economic efficiency	-
Atalla et al., 2018 [31].	Welfare and Economic impact	-
Lund, 2007 [53].	Cost-effectiveness	Market State
Dong, 2012 [37].	GDP per capita, Oil imports, electricity consumption	-
Agnolucci, 2007 [54].	Financial support	-
Mitchell et al., 2006 [55].	Risk reduction	-
Sancho, 2010 [38].	GDP	-
Qiu et al., 2019 [56].	City Infrastructure	-
Bersalli et al., 2020 [39].	Energy Production, Energy dependence rate	Access to domestic credit, Income per capita
Economics, 2011 [45].	Cost-effectiveness	-
Yuan et al., 2011 [57].	Cost-effectiveness
Rosenow 2016 [58].	Energy Savings	-
Nicholson-crotty & Carley, 2016 [59].	Energy generation	-
Stagl, 2006 [40].	Energy efficiency; energy costs	-
Zhang et al., 2014 [60].	Energy costs, renewable energy costs (and subsidies), Carbon prices	-
Hoffman et al., 1977 [41].	Cost, generation, dependence on imports	-
Carley, 2009 [42].	Gross state product per capita, electricity use, electricity price	-
Neij & Kerstin, 2006 [61].	Technology characteristics, electricity costs	-
J. Zhao et al., 2011, [62].	Technology payback period	-
Kilinc-Ata, 2016 [43].	Carrier prices, imports, GDP growth, electricity consumption	-
Verbruggen, 2009 [63].	Cost-effectiveness	-
Banerjee & Solomon, 2003 [64].	-	Consumer response and manufacturer/marketer response
Kim & Lee, 2012 [65].	-	Consumer choice, technology, and location characteristics

. It should be noticed that almost all the reviewed documents included at least one aspect under this category. The economic dimension regards all the concerns that assess policies in monetary terms, from which two subcategories were formed.

The Economic-cost subcategory includes the concerns related to direct monetary influence, particularly related to costs and expenses. Both levels, macroeconomic and microeconomic, have been assessed; e.g., the energy price is included, referring to the economic aspects in a micro economical view. The gross domestic product (GDP) impacts of energy decisions refer to the macro level of an economy.

Regarding Economic-competitiveness, this subcategory brings together market orientation concerns (e.g., attractiveness of investment, market growth and market maturity aspects). The market characteristics provide information that allows the comparison among different markets, informing about their national and international competitiveness.

4.1.3. Social Concerns

Regarding the Social category, all concerns with a social orientation have been gathered in

Table 9. Energy Policy Social concerns found.

Document	Equity	Employment
Andrew J. Chapman et al., 2016a [16].	Health, subsidy allocation, costumer’s participation	Employment
Rogge et al., 2017 [66].	Distribution of energy costs (equity)	-
IRENA et al., 2014 [67].	Equity (Consumer´s impact)	-
Abotah, 2015 [15].	Customer interaction and satisfaction	-
Andre John Chapman, 2016b [17].	Impact of subsidies in energy prices, distribution of costs and benefits	Employment
(W. Li et al., 2016) [68].	Perceptions of importance and satisfaction	-
Nikolaev & Konidari, 2017 [21].	Energy equity	Employment
Wang et al., 2017 [69].	Consumers’ preferences, vehicle acceptance, and WTP	-
Hafeznia et al. [22].	Social acceptance of RE technologies	Knowledge and skill of workforce (expertise)
Enzensberger [70].	Stakeholder´s consideration	-
Edwards, 2018 [71].	Social acceptance	-
Burke & Stephens, 2017 [27].	Energy democracy	-
Chang & Fang, 2017 [28].	Equity among households: Energy elasticity	-
Liu et al., 2018 [52].	Consumer surplus	-
Mirjat et al., 2017 [30].	Social context	-
Atalla et al., 2018 [31].	Electricity price elasticity (Household behavior)	-
Delmas & Montes-sancho, 2011 [32].	Customers, Renewable preferences	Unemployment
Arbolino et al., 2019 [34].	Health	-
Harmelink et al., 2006 [72].	Acceptance by the actors	-
Yao et al., 2014 [73].	Demographic Information	-
Sancho, 2010 [38].	-	Unemployment
Qiu et al., 2019 [56].	Demographic information	-
Ling-yun & Jia-jia, 2016 [74].	Public Health and people´s welfare	-
Delmas & Montes-sancho, 2011 [32].	Customers’ preferences, Investors	-
Bürer & Rolf, 2009 [75].	Investors	-
J. Zhao et al., 2011 [62].	Household income; neighborhood	-
Stagl, 2006 [40].	Warm spaces	-
Neij & Kerstin, 2006 [61].	Participation and Commitment of actors	-

, including the concerns in which equity is recognized. Two sub-categories were defined. Firstly, Social-equity include concerns related to social sustainability factors, as the acceptance of certain energy technologies and the impacts of subsidies on energy bills. It can be noticed how equity has been considered in several documents, being a key component of distributive energy justice, and that should contribute to new policy initiatives. Some other concerns include energy democracy, energy equity and consumer surplus. This last is an important reference to measure the welfare of consumers [29].

The second subcategory, Social-employment, as can be inferred from its name, is the subcategory to account for the employment rates affected by the energy policies. The knowledge and skills (expertise) of the workforce have also been included in this subcategory.

4.1.4. Institutional Concerns

Finally, under the institutional concerns assessment (

Table 10. Energy Policy Institutional concerns found.

Document	Concern
Rogge et al., 2017 [66].	Actor´s and institution´s participation
IRENA et al., 2014 [67].	Political accountability, source of finance and regulatory simplicity
Andrew J. Chapman et al., 2016c [76].	Governance, policy processes and priorities
Theel & Westgaard, 2017 [13].	Policy Environment
Abotah, 2015 [15].	Institutional and Government support
Bukarica & Tomšić, 2017 [46].	Political will, public participation, public dialogue, regulatory framework
Yoon & Sim, 2015 [77].	Policy framework (policy environment and policy process)
Nikolaev & Konidari, 2017 [21].	Administrative and financial capacity, Feasibility of implementation
Arababadi et al., 2017 [26].	Policy budget
Burke & Stephens, 2017 [27].	Policy implementation mechanisms
Liu et al., 2018 [52].	Government fiscal revenue
García-Álvarez et al., 2018 [29].	Policy Environment
Mirjat et al., 2017 [30].	Policy Environment
Delmas & Montes-sancho, 2011 [32].	Type of Governor
Menz & Vachon, 2006 [33].	Policy regime
Dong, 2012 [37].	Other policies (number)
Agnolucci, 2007 [54].	Policy certainty and planning constraints
Qiu et al., 2019 [56].	Policy options
Rosenow et al., 2016 [58].	Policy characteristics
Carley, 2009 [42].	Political institutions, other policies (presence)
Harmelink et al., 2008 [78].	Policy process
J. Zhao et al., 2011 [62].	Advertisement
S. Ma et al., 2020 [50].	Subsidies
Verbruggen, 2009 [63].	Robustness, efficiency

), the variety of concerns found through the reviewed papers was larger than in the previous categories; however, no subcategories were created, due to a lack of a clear structure. The identified concerns describe the policy institutional framework, including concerns as flexibility with policy instruments, actors, and institutional environment, among others.

There is a recent recognition of the importance of the institutional context. Concerns like political commitment, policy choice, and citizen engagement characterize policies from an institutional perspective.

Even if the institutional performance does not measure policy success itself, it contributes to explain the policy environment. Institutional context tends to be assessed qualitatively rather than quantitatively. The lack of quantitative procedures to evaluate some dimensions of the institutional context provides the opportunity to develop scientific-based methodology proposals that enable to evaluate the concerns found under this category.

Table 11. Number of appearances per concern by subcategory.

Category	Quantity of Documents	Documents Numbers (from Tables in Appendix A)
Environmental-others	14	2, 7, 15, 17, 18, 21, 24, 25–29, 31, 33
Environmental GHG	14	1, 5, 9, 15, 18, 21–23, 36, 38, 39, 44, 47, 49
Economic-cost	36	1, 2, 5, 8–10, 14–19, 22, 24-29, 31, 32–36, 39, 40, 42–49, 60
Economic-competitiveness	15	1, 3, 5, 7, 9, 10, 14, 18, 31, 34–37, 49, 62
Social Equity	25	1, 6, 9, 11, 12, 15, 18, 20, 24–26, 28, 29, 31, 36, 37, 39, 40, 42, 48, 52, 53, 58, 60, 66
Social-Employment	7	1, 9, 10, 18, 31, 36, 47
Governance	26	3, 4, 9–15, 23, 24, 26–28, 36, 38, 40, 44, 45, 48, 51, 56, 57, 60, 62, 63, 65

shows the number of appearances per concern by subcategory. It is noted that Economic-cost is the subcategory that was found more often in the reviewed documents, reflecting the importance given to it by the consulted sources. The second most popular subcategories are Governance and Social equity, which regards the energy impacts on society and energy distributional impacts.

Table 12. Single vs. multi-category orientation.

Category	Quantity of Documents	Documents numbers (from Tables in Appendix A)
Single	24	4, 6–8, 13, 16, 19–21, 30, 32, 34, 37, 41, 42, 46, 50, 52, 54, 57, 58, 61, 64, 66
Multi-category	42	1–3, 5, 9, 10–12, 14, 15, 17, 18, 22, 23, 24–29, 31, 33, 35, 36, 38–40, 43–45, 47–49, 51–53, 55, 56, 59, 60, 62, 63, 65
Environmental	2	21, 41
Economic	12	7, 8, 16, 19, 32, 34, 46, 50, 52, 54, 61, 64
Social	7	6, 20, 30, 37, 42, 58, 66
Institutional	3	4, 13, 57

shows an analysis of the categories. It was noted that most documents mention more than one category (42 documents). From the remaining 24 documents (single category approach), 12 documents report an Economic orientation, being the largest one.

4.2. Identification of Gaps

After the review of the concerns that have been used to characterize the effectiveness of the energy policies, some areas have been identified as not fully covered regarding an integral review of EP effectiveness characterization.

A list of missing concerns has been elaborated (

Table 13. Additional concerns by category.

Environmental Concerns	Economic Concern	Institutional Concern
Water resources	Externalities pricing	Amount and quality of Staff
Land resources	Grid state	-
Waste Management	-	-

) in which additional concerns have been included. The criterion followed to add concerns was if the outcomes of the concern could be modified by a policy and their relevance for the ep effectiveness characterization.

Not all the subcategories needed extra concerns, e.g., the Social subcategories (Social-Equity and Social-Employment) have been considered as complete, as the concerns identified through the review do provide a wide perspective of the social approach.

Under the environmental category, the concern of environmental protection has been often found through the revised documents. However, it has been considered relevant to detail it through more specific concerns (as water and land resources) to be able to better measure them. They are relevant concerns for the environmental evaluation of policy and closely related to the impacts of energy on the environment.

The externalities pricing is a concern that can be reported under the environmental or economic category, due to their interrelationship in the concern itself. For the monetarization process that externalities involve, it has been decided to keep this concern under the economic category.

It was found a lack of consideration of some aspects that may be relevant for the policy appraisal, as available funds (for technology, innovation, and infrastructure investment), the economic situation of the private sector, and the economic situation by regions, economic and fiscal parameters (like inflation, escalation, exchange rates, and taxes). All these concerns can be gathered as economic background. Additionally, the grid state is a relevant concern for its relation to the energy service provision.

Finally, the concern under the institutional category (Amount and quality of Staff) intends to involve a relevant aspect of the institutional framework: the governing human resources. They differ from Knowledge and skill of the workforce (reported in the Social concerns), as this last regards the technical workforce involved in the policy implementation, while the second one, regards the governance-related workforce.

4.3. Inferred Energy Policy Objectives and Condensed Concerns

With the intention of better organizing the concerns and facilitating the identification of their evaluation metrics, a condensation process was taken to eliminate redundancies, conglomerating them into condensed concerns. The condensed concerns have been identified after questioning where there could be an impact if the extracted concern does not comply. In addition, for every condensed concern, its end objective has been identified (as shown in Figure 1). The identification of the ends objective aims to put the review more in line with the typical multi-criteria decision making (MCDM) language and to ease the further identification of indicators.

The economic extracted concerns are the largest in number when compared with other categories. It was possible to narrow down the initial list to three condensed concerns: affordability, accessibility, and economic competitiveness.

Affordability includes all the concerns that may affect the state of being cheap enough for people to be able to pay for electricity or other forms of energy [79]. Accessibility refers to the fact of energy or electricity being able to be reached or obtained easily [80].

Competitiveness brings together the concerns that relate to being able to compete successfully with other companies, countries, organizations, etc. [81].

The concerns under the environmental spectrum have been condensed through two concerns. The first one reflects the policy’s impact on the environment. Its ends objective relates to environmental sustainability, defined as the quality of causing little or no damage to the environment and therefore able to continue for a long time [82]. The second condensed concern denotes the impacts on climate change.

In the social category, Equity, as the situation in which everyone is treated fairly and equally [83], gathers the majority of the social concerns. Health is however kept as a separated condensed concern.

Governance effectiveness and efficacy as a single condensed concern included all the concerns under the institutional perspective. Governance is defined as how an organization is managed at the highest level and the systems for doing so [84] and assessed through its effectiveness and efficacy.

The organization of the extracted and condensed concerns is documented in Figure 1 The economic, environmental, and social categories include more than one condensed concern.

5. Indicators for Energy Policy Effectiveness

5.1. Indicators’ Relevance and Types

The relevance of indicators has been recognized in the energy field [85,86,87,88,89]. Agents have created ways to assess different dimensions of the energy industry through indicators, giving them several functions and purposes [90,91,92,93]. They can be used as assessment metrics, providing a comparison between the defined objectives and their achievement, therefore assessing the effectiveness of the state of systems, which can be applied to assess the success of policies.

Indicators can be differentiated by function, serving as monitoring, guideline, or communication instrument. With a monitoring purpose, indicators evaluate the policy progress towards their objective’s achievement. When indicators are used to monitor the status of an energy policy, they can be helpful for the improvement of the policy. An in progress evaluation enables the identification of the need for modifying the track to achieve the settled objectives in case the progress is not fully satisfactory.

At the beginning of the policy cycle, indicators can provide a deeper orientation of where to add more resources or where to focus while designing a policy. They may allow a deeper understanding of some elements and even help to identify relations that may not be shown through basic statistics or data. Indicators, as communication instruments, can be useful tools for policymakers to express elaborated ideas or concepts to other relevant stakeholders of the sector or out of it; e.g., the indicators issued by national governments to inform regional or local authorities.

Indicators can also be differentiated by whether they are individual or composite (also known as aggregated). The first ones are commonly used to show the presence or state of a situation or condition individually for a topic [94]. Despite the usefulness of individual indicators, sometimes the information provided may not be enough for the purposes of the evaluation or an integrative assessment of more than one element. Hence, the condition of the state of a topic may require the evaluation of more than one criterion, requiring the use of composite indicators. For some interrelated information, composite indicators are a feasible alternative. They are the result of individual indicators being compiled into a single index to summarize complex and multi-dimensional situations [95]. Some well-known examples of composite indicators in the energy field are the energy poverty and energy access indicators. For energy deployment decisions, policymakers require to know the state of the energy sources, which involves different concerns. That may require an aggregated indicator. For this example, aggregated and individual indicators may be required to contribute to providing information about the energy sources, including why and how they were successful, the social perceptions around them, together with the economic details and other relevant concerns [24].

5.2. Indicators Review Methodology

The indicators’ documents search was initially done in the open browser Google. The first 50 results of the search under energy policy indicators were reviewed in detail. Many of them were about energy or indicators, however, they either analyze or propose indicators related to buildings, utilities or other very specific topics of the energy sector or are not related to energy policies. Therefore, only the documents with general energy policy goals were considered, resulting in 23 documents presented in Appendix B.

Subsequently, a complementary search was done on the academic database Sciencedirect. From the first 50 documents retrieved under Energy Policy Indicators, most of the documents retrieved, do not propose new sets of indicators (from the ones previously identified). The majority of the related documents focus on analyzing the indicators proposals done by international organizations or governments (identified through the open sources search) or documents that focus on the analysis of the implementation of the indicators in a particular country or region.

When searching the corresponding indicators for every condensed concern through the retrieved documents, some concerns could not be matched through the recovered documents. A detailed search was done to find at least one indicator for the concerns. This led to finding additional documents listed in Appendix B. For this search, the concerns themselves were used as keywords, together with the term “indicator”, “assessment” and “evaluation”.

5.3. Indicator Identification

In Section 4 all the extracted concerns were organized into condensed concerns, for which the indicator identification process was followed, identifying, at least one indicator for every condensed concern. The end objective of each concern has been enlisted, and at least one indicator as an assessment metric of each objective has been identified.

Table 14. Identified indicators for the condensed concerns—Economic.

Concern	End Objective	Indicator	Units	Source	Definition
Affordability	Max. The capability of energy payment	Willingness to Pay	Numerical value	Measuring Willingness to Pay for Electricity [96]	Indicator of how much personal satisfaction or well-being individuals derive from different outcomes
		Spending on energy as a proportion of the household income	Percentage	Assessing the performance of renewable energy support policies with quantitative indicators [97].	Percentage of income destined to pay electric energy
Accessibility	Max. The capability of energy provision	Energy access	Aggregated value	Energy Access Score [98].	Aggregated indicator of eight subindicators
		Energy security	Vary by subindicator	Energy Trilemma Index [99].	Indicator part of the Energy Trilemma Index, assessment of Security of Supply and Energy Delivery
Economic competitiveness	Max. The economic competitiveness	Harmonized Competitiveness	Numerical value	Harmonised competitiveness indicators [100].	Indicators that include price and cost competitiveness
		Competitiveness Index	Index	The Global Competitiveness Report [101].	Index that analyzes competitiveness along 12 pillars by country
		Country Context	Vary by subindicator	Energy Trilemma Index [99].	Part of the Energy Trilemma Index. Includes Investability, Stable Regulatory environment, Air, land and water impact, RD&D and innovation, Coherent and predictable framework.

shows the indicators for the condensed concerns with an economic orientation. For all the concerns, more than one indicator has been identified.

Table 15. Condensed Environmental Energy Policy Effectiveness concerns and identified indicators.

Concern	End Objective	Indicator	Units	Source	Definition
Environmental sustainability	Min. The environmental impacts	Environmental Sustainability Indicator	Vary by indicator	Environmental Sustainability Indicator [102].	Indicators on air, climate, water, nature, and human influence.
		Environmental Performance Index	Index	Environmental Performance Index [103].	Ranks 180 countries on 24 performance indicators across ten issue categories covering environmental health and ecosystem vitality
		Environmental Sustainability	Vary by subindicator	Energy Trilemma Index [99].	Part of the Energy Trilemma Index. Includes the assessment of Energy resource productivity, GHG Emission and C02 emissions
Impact on climate change	Minimize the impact on climate change	GHG emissions from energy production per unit of GDP	GHG/GDP (C02/economic unit)	Indicators for Energy Sustainable Development [89].	Composed by GHG emissions from energy production over GDP

shows the proposed indicators for the condensed concern with environmental orientation, environmental sustainability, for which three alternatives were identified; the proposed indicators for the two condensed concerns with social orientation are shown in

Table 16. Condensed Social Energy Policy Effectiveness concerns and identified indicators.

Concern	End Objective	Indicator	Units	Source	Definition
Equity	Max. the consumer´s welfare	Energy Equity Indicators	Vary by subindicator	Energy Equity Indicators [104].	Set of indicators that evaluates Access, investment and Resilience
		Energy Equity	Vary by subindicator	Energy Trilemma Index [99].	Includes the assessment of Access, Quality of Supply and Affordability and competitiveness
Health’s impacts	Min. The energy impact on health	Accident fatalities per energy produced by fuel chain	Number of fatalities by fuel chains per energy or electricity produced annually	Energy indicators for sustainable development [87].	Annual fatalities by fuel chain

; the indicators for the Governance effectiveness and efficacy condensed concern are shown in

Table 17. Condensed Governance Energy Policy Effectiveness concerns and identified indicators.

Concern	End Objective	Indicator	Units	Source	Definition
Governance effectiveness and efficacy	Max. the policy effectiveness	Institutional Feasibility indicators	Vary by subindicator	Institutional Feasibility indicators [24].	Aggregated and individual indicators for six dimensions of governance
		Worldwide Governance Indicators	Percentile Rank (0 to 100)	Worldwide Governance Indicators [105].	Indicators that Summarize dataset of the quality of governance

.

6. Discussion

6.1. Contributions of the Review

Wide variety of documents (66 documents) that meet the eligibility criteria have been analyzed. Each of them presents an alternative to characterize energy policies. The data gathered derives from multiple sources that attempt to give a solution to the same question: how to evaluate energy policy effectiveness?

The results generated a wider perspective on the different concerns that assess energy policy effectiveness. Four core categories (Economic, Environmental, Social, and Institutional) were organized and synthesized through more specific categories (eight condensed concerns organized by their ends objective), presented in Figure 2.

It was found that five documents of the review [21,27,30,32,52] consider at least one extracted concern under each of the four core categories, being the most complete documents presented in this review. Nevertheless, no single document covered the seven condensed concerns that have been identified. Therefore, our results are an expansion of those approaches, giving the possibility to apply a more integral evaluation.

The identification of the core metrics of the concerns was also done, leading to the exploration of the energy policy indicators spectrum. The main contribution of this review is the organization of attributes focused on assessing energy policy effectiveness in a more detailed way. The Energy Trilemma [99] metrics have been very useful to identify options to assess some of the identified condensed concerns; however, the Trilemma lacks the consideration of one of the core categories resulting from this review, the Institutional one.

The potential of the data from this review gives the possibility to construct more complete and integral assessment procedures and to reframe already existing energy policy evaluation methodologies with the addition of new attributes (and its metrics).

This review can be directly useful for the national and international current governance situation, in which different levels of authority are implementing approaches for climate change mitigation, whereas EP evaluation is required, e.g., for the SDGs achievement assessment, a set of indicators currently exists [106]. However, a policy effectiveness evaluation tool or methodology (in coherence with the SDGs) may contribute to assess in more detail every policy. Therefore, this report contributes to establishing a base for the future development of a policy effectiveness evaluation methodology that supports the alignment of policies to achieve the SDGs or any other international, national, or local frameworks.

6.2. Limitations of the Review

The search considered the key search engines Elservier, Google, Google scholar, and JSTOR, so in principle, all major publications obeying the criteria were covered. However, theoretically, it could happen that the consideration of additional databases (e.g.,CORE, Bielefeld Academic Search Engine (BASE), Directory of Open Access Journals (DOAJ), Social Science Research Network (SSRN), World Wide Science, Semantic Scholar, etc.) could provide some additional results. In addition, the non-published documents (studies non-accepted for publication) have not been reviewed; however, this situation does not pose a serious threat to the validity of the review [107]. The search protocol applied has a reasonable degree of easiness and reachability, resulting in a sound number of documents that could be gathered.

The data generated is suitable to be analyzed with statistical methods or other quantitative techniques as HOMA [108] or MASEM [109]. Meta-analysis techniques have not been applied in this study, which has a qualitative nature. However, useful information may be retrieved in the future by such kind of analysis.

7. Conclusions

This review aimed to characterize the policy evaluation practices in the Energy field, with a focus on the metrics: concerns, objectives, and indicators. In addition to efficacy and efficiency, an emphasis was put into trying to find indicators that can assess effectiveness. Effectiveness enables a more holistic, field-tested, and modulated assessment of the achievements.

It was found that from the different categories (Environmental, Economic, Social, and Institutional), the Economic category is the one most often included, being present in nearly all documents (36 out of 66). Ex post is the most frequent evaluation type in terms of the moment (34 out of 66). Regarding the level of disaggregation reported for EP evaluations, local studies were found in three documents, while the national ones had the lead (40 out of 66). Considerations like the stakeholder’s participation, evaluation motivation, and methodological approach varied with less focus across the gathered documents.

The extracted concerns were condensed in a more compact list that eliminated redundancies, organizing them by their end objective. This resulted in seven condensed concerns: accessibility; affordability; competitiveness; impact on climate change; other impacts on the environment; equity; and health.

A total of 15 indicators were identified as possible alternatives to fully characterize the concerns: 2 for affordability, 2 for accessibility, 3 for economic competitiveness, 3 for environmental impacts, 2 for equity, 1 for health, and 2 for governance effectiveness and efficacy. This list is a considerable expansion from what could be found in any single previous document. It is proposed as a holistic assessment of the effectiveness of energy policies. The information organized through this review could be the object of further future works, e.g., on the computation of every single indicator—either ex ante or ex post, but with the goal of enabling ex ante estimation as much as possible. This would further support the elaboration of solid policies, adapted to the local institutional and geographical contexts.