A Methodology for Determination and Definition of Key Performance Indicators for Smart Grids Development in Island Energy Systems

Abstract

There is a growing interest over the last decades in the field of autonomous island grids that is driven mainly by climate reasons. The common objective among the members of the European Union (EU) is the increase of Renewable Energy Sources (RES) penetration in the energy mixture, as well as turning the grid into a smart grid. Consequently, more and more state-of-the-art solutions are being proposed for the electricity generation and the optimization of the energy system management, taking advantage of innovations in all energy related sectors. The evaluation of all available solutions requires quantitative assessment, through the adoption of representative Key Performance Indicators (KPIs) for the projects that are related to smart grid development in isolated energy systems, providing the relevant stakeholders with a useful comparison among the proposed solutions. The evaluation approach that is described in this paper emphasizes the role of the various stakeholder groups who face the proposed solutions by different points of view. Apart from the domains of interest that are also observed in previous approaches, the proposed list also contains a set of legal KPIs, since the regulatory framework can either represent a serious barrier or grant a strong incentive for the implementation of state-of-the-art energy technology and grid management solutions in different countries.

1. Introduction

1.1. Isolated Energy Systems Overview and Current Trends

The electrification and energy provision in islands is a quite complex issue, since the need for energy sustainability poses serious environmental and technical feasibility considerations that always need to be taken into account. Thus, several research projects and studies [1,2,3,4,5] attempt to provide a range of solutions trying to meet cases of high diversification in terms of energy requirements and state-of-the-art technologies utilized. One of the major issues faced is the current cost of island electrification, especially of small and non-interconnected islands, which is very high when compared to that of mainland grids. Secondly, the current energy (mainly electricity) production is very polluting due to the use of heavy fuel oil-based generators in most of the cases. Moreover, most parts of the island grid infrastructure are characterized by small individual grids where new technology solutions need to be evaluated before levelling up in interconnected grids. Relevant studies have revealed that island grids, as a kind of isolated communities, confront quite common problems that require similar strategies, even if they are too far away from one another or characterized by different climate conditions, irrespective of whether they are interconnected or not. For example, a study by Cross et al. [6] has shown that distant islands, like the Isle of Man, Malta, Jersey, and Terceira are facing common challenges concerning the social and economic sustainability as well as the CO₂ emissions, and they share similar thoughts for solutions.

The new trend is turning the small grids or even the end-parts of interconnected systems into smart, in order to support the increase of Renewable Energy Sources (RES) penetration in the energy mix. This trend is applied in various ways and in multi-scale applications, from the level of a building (net-zero or even energy positive buildings) to that of a whole district or a region (smart micro-grids). A grid of an island can act as a useful representative use case for smart micro-grids, since it shares common characteristics and needs of any micro-grid. The constant difference between offer and demand, as well as the need for sustainability, independency, and energy cost reduction, are always the objectives of such type of grids.

Many studies have been recently conducted throughout the world concerning the transformation of island grids into smart ones. Some analyze the aspects that influence the selection of the optimal governance policy or energy strategy [7], while others look closer to the optimized integration of a (state-of-the-art) technology in a grid [8,9,10,11,12,13,14]. The smartening of a grid, in most of the cases, demands the utilization of multi-parameter decision-making software with high accuracy [15,16] to take into account all different parameters that play a role in a decision [17]. Many others focus on specific case studies of island grids around the world. Except for the Faroe Islands archipelago, which was the first one to demonstrate a smart grid with large-scale utilization of wind power [18], many other studies have been conducted for the optimization of smart island grids [6,11,19,20,21,22,23].

The common sharing guideline among the European Union (EU) members is to increase the RES penetration in the energy mix, as well as to reduce the energy consumption so as to diminish the carbon footprint of each end-user, from the scale of a single device up to a whole interconnected grid. Smartening the grid with the use of smart meters, predictive algorithms, and Demand Side Management (DSM) algorithms provide automation, reduce peaks, and distribute more cognitively the energy generation from various types of power plants (fossil fuel or RES based). These guidelines have been defined by the European Commission (EC) (2020 Energy Strategy) [24,25] and have been individualized for the case of island grids with the set-up of dedicated EU structures, like the Smart Islands Initiative [26].

1.2. KPI’s Role and Methodological Background

The holistic evaluation of any newly proposed technological solution is a very important step during the procedure of its development and optimization and before its real-life application in a large scale. The use of indicators is valuable not only to describe accurately a specific characteristic of a technological intervention, but also to evaluate this in a simple way on a fair basis, facilitating its comparison (in many aspects, as it will be evident later in the article) to other ones designed to meet the same scopes. KPIs are indexes that measure the effectiveness of a project or a venture and/or its proposed solutions towards the achievement of the pre-defined specific key objectives [27]. The process of selecting KPIs also assists the clarification of project objective’ degree of success. These indicators should be [28]:

• meaningful: a KPI relates with one or several expected innovation impacts, and therefore makes sense, since it can contribute to reach the program overarching goals;
• understandable: the KPI definition is clearly related to the expected impacts of the studied innovation; and,
• quantifiable: experimental values coming from field testing at an appropriate scale are used to develop ad-hoc simulation tools that are able to estimate the expected innovation impacts.

The KPIs are performance indicators that can assess (a) characteristics of a technology solution; (b) the impact of a technology on its environmental surrounding; (c) its economic feasibility; (d) its social approval either by the policy-making bodies or by the local society; and, (e) the advances and/or the relevant legal framework requirements that need to be met, before being implemented in a large scale. Each value of the selected KPIs in Research and Development (R&D) is very important as they can form the basis for an analytic evaluation of a technology solution by being in position to valorize the various proposed solutions according to their performance and the specific needs of each case that they were designed to serve. As an example, for the case of Photovoltaics (PV) investments in several grid tied and off-grid use case scenarios,

Table 1. The Payback Period of five Photovoltaics (PV) investments [28].

Economic Parameters	10 kW Grid Tied System, Florida	5 kW Grid Tied System in Kaui, Hawaii	5 kW Grid Tied System, Arizona	8 kW off Grid System, Jamaica	8 kW off Grid System, Roatan Honduras
Cost of Electricity ($/kWh)	0.117	0.55	0.111	0.39	0.52
System Size (Watts)	10,000	5000	5000	8000	8000
Peak Sun Hours (hours)	5	5	6.5	5	5
System Efficiency	83%	83%	83%	75%	75%
Installed Cost Per Watt ($/Watt)	1.375	2.75	2.75	4	4
Rebate and Incentive	30%	30%	30%	0%	0%
Initial Investment ($)	13,750	13,750	13,750	20,000	20,000
Rebates and Incentives ($)	−4125	−4125	−4125	0	0
Actual System Cost ($)	9625	9625	9625	20,000	20,000
Estimated Annual Electricity Savings ($)	1772	4166	1093	4271	5694
Payback Period (years)	5.43	2.31	8.81	4.68	3.51

examines the impact of the various background conditions on the value of one of the most significant KPIs that is taken into account by all investors/users, i.e., the “Payback Period”, which indicates the time period that is needed for the cumulative gains of an investment to equal the cumulative costs:

The scope of the determined KPIs, specifically for the case of smart grids, is not only to present the performance of various technological solutions, but also to identify the margins of further development that is available, towards optimizing the smart and efficient operation of a grid in an efficient, cost-effective, user-friendly, and environmentally friendly way, respecting as much as possible the social needs of the local communities where each system is expected to be applied. Thus, various aspects need to be taken into consideration when a technology solution is assessed, as those of (a) each stakeholders’ opinion given their different perspectives; (b) the technical performance of each solution; (c) its contribution to system security and sustainability; (d) the economic feasibility of the investment; (e) the environmental impact as compared to similar technologies; (f) the legislative burdens for the application of the proposed technologies; and, (g) the effects on the local residents quality of life and their opinion, since citizen’ engagement on the examination and the adoption of a solution is a prerequisite for the solution’s further development and application on a larger scale (i.e., that of a community).

1.3. Smart Grid Evaluation Frameworks

The topic of the present study is oriented towards the evaluation of the integrated state-of-the-art technologies in smart grids that, as stated above, are becoming a priority for the EC in order to decrease the carbon footprint as well as the overall cost of energy. Thus, the present literature survey is mainly based on the evaluation methodologies developed by many large projects that are funded by the EC. The Expert Group of the EC Task Force for Smart Grids circulated a report in 2010 [29], aiming at defining an assessment framework for the evaluation of Smart Grid projects according to a set of criteria, in line with the requirements that were put forward by the EC in the Proposal for a regulation on guidelines for Trans-European energy infrastructure [25]. The report suggests the use of a list of KPIs using six different criteria (level of sustainability, capacity of transmission and distribution grids, network connectivity and access, security and quality of supply, efficiency and service quality in electricity supply and grid operation, contribution to cross-border electricity markets, and increase in interconnection capacities). These criteria can reflect the contribution of each project against six functions/services (enabling the network to integrate users with new requirements, improving market functioning, enhancing efficiency in day-to-day grid operation, ensuring network security, system control and quality of supply, better planning of future network investment, enabling and encouraging stronger and more direct involvement of consumers in their energy usage, and improving customer service) of the “ideal” Smart Grid.

Table 2. Technology assessment methodologies by European Commission (EC)-funded projects.

Project Acronym/Name	Project Description	Methodology of Evaluation	Ref
DREAM: Distributed Renewable resources exploitation in electric grids through advanced hierarchical management	Demonstrates an industry—quality reference solution for Distributed Energy Resources (DER) aggregation—level control and coordination, based on commonly available Information and Communication Technology (ICT) components, standards, and platforms for all actors of the Smart Grids.	Two possible approaches for the KPI development: (1) a procedure for the KPI selection and definition starting from the use case goals and then moving to the trial sites/demo site goals or (2) starting from the pilot goals, developing KPIs and use case solutions accordingly. A combination of the two approaches was finally adopted.	[27]
DISCERN: Distributed intelligence for cost-effective and reliable solutions	Examined cost-effective network solutions for future network development. The starting point for DISCERN [29,30,31] was the EEGI framework which was adopted for practical purposes and operational use by the Distribution System Operators (DSOs).	Organized structured and detailed workshops purposed to define the list of KPIs from EEGI, developing the KPI framework and their detailed definitions. The participation of the maximum possible number of DSOs in these workshops aimed at the presentation of a consolidated partners’ point of view within the respective countries and regulatory frameworks represented in the project.	[29,31]
INERTIA: Integrating active, flexible and responsive tertiary prosumers into a smart distribution grid	Provided an overlay network for coordination and active grid control, running on top of the existing grid and consisting of distributed and autonomous intelligent Commercial Prosumer Hubs.	The project established an Integrated Energy Performance Model that extended the existing ones, by incorporating and integrating multiple dimensions, i.e., the physical sub-system, the human sub-system, the Enterprise sub-system and the general surrounding environment. Through direct incorporation of the Enterprise as a specific actor, this performance model was better adjusted to specific business domains and provided the basis for the optimal balance between DSM, Energy Performance and Enterprise Performance.	[32]
EPIC-HUB: Energy positive neighborhoods infrastructure middleware based on energy—hub concept	Focused on efficient Management, Control and Decision-Support Energy Policies at neighborhood-level, defining an interoperable Middleware solution and a structured vision for the communities to use and share renewable energy sources, energy storage, and micro-generation, in order to consistently realize energy savings, reduce CO2 emissions and optimize energy usage.	The performance measures are distinguished: Key Results Indicators (KRI), Performance Indicators (PI) and Key Performance Indicators (KPI) and the concomitant use of them according to different aspects and dimensions of the project, such as time. The specific distinction provides a useful and functional taxonomy that can be used in the establishment of a complete and multi-dimensional performance framework.	[33]
inteGRIDy: Integrated smart Grid cross—functional solutions for optimized synergetic energy distribution, utilization storage technologies	Aims at integrating cutting-edge technologies, solutions and mechanisms in a scalable Cross-Functional Platform (CFP) of replicable solutions, towards connecting existing energy networks with a diverse group of stakeholders consisting of both generation and consumption profiles.	The KPI determination in inteGRIDy was based in a 4-axis principle: the different focus between global and local level, concerning whether the evaluation of technologies is made on a single pilot or not; the stakeholder’s point of view; the thematic pillars which represent the science and technology areas where the main innovative activities are tested; the domain that an indicator tries to address.	[34]

indicates the basic ideas that are developed by EC-funded projects concerning smart grids. Each project consortium developed a methodology based on the same main structure, in order to reach compact conclusions, but also applied advanced assessment specifications according to the special needs of each project.

Apart from the relevant projects presented in Table 2, there are various worth mentioning scientific studies that are dedicated to smart grids evaluation. Among them is the study of Mia Ala-Juusela et al. [35]. In this study, the concept of an energy positive neighborhood and the metrics and tools to measure the energy positivity level of an area is presented for the first time. Thanos et al. [36] defined a number of different performance metrics that could be used to evaluate Demand Response (DR) programs that are based on peak reduction, demand variation, and reshaping, as well as economic benefits. The study of Door Hans van Nes [37] introduced the idea of Key Exception Indicators to adjust KPIs, which acquire data automatically, some of which include bugs or failures. Finally, many studies have concluded in lists of KPIs to define specific standards for electricity quality in specific countries or in the European Union in general [38,39,40,41].

The EEGI is one of the European Industrial Initiatives under the Strategic Energy Technology Plan. The EEGI’s mission is to establish an adequate European grid (both transmission and distribution systems), in order to achieve the European energy policy strategic objectives [42], thus the KPIs that are defined by EEGI mainly concern the assessment of grid-based technologies.

Other initiatives, like CITYKeys [43] and SCIS [44], are also relevant to smart grids evaluation and they can be fitted to characterize autonomous grids, even though they are mainly city level referred. Many of the KPIs that are defined by these initiatives can also prove to be appropriate for the assessment of smart island energy systems.

2. Methodology

The scope of this study is to present a new methodology for the identification of KPIs, aiming at the evaluation of new technologies that are related to smart grid applications in isolated systems. The two characteristics that make it unique as an approach lie on the facts that (a) the stakeholders are placed in a prominent position when it comes to the determination of the criteria of the evaluation and (b) the inclusion of the legal aspect as one of the KPI domains, since, to the best of our knowledge, no legal KPIs can be identified in the existing literature.

The proposed methodology is adopted in the framework of the EU funded research project SMILE [45]. The project demonstrates nine different smart grid technologies on three different islands (Orkney from Scotland, Samsø from Denmark and Madeira from Portugal). The end goal of the project is to foster the market introduction of these nine technologies. The procedure followed for the definition of the assessment methodology involved the following categorization steps: (1) division of the technology solutions into groups called thematic pillars; (2) definition of the main groups of stakeholders; and, (3) listing of the KPIs into separate domain lists, to make clear their link to the aspect they are expected to evaluate.

The specific methodology is generic, because it is structured in order to satisfy all possible assessment requirements of a proposed technology solution. Thus, it could be preferred as well in a wide variety of technology projects other than those concerning the smart isolated grids, especially if the proposed technology solutions are in a mid-high Technology Readiness Level (TRL).

2.1. Thematic Pillars

The thematic pillar categorization is not rigid, as it may vary according to the topic and the proposed solutions. The proposed categorization consists of five (5) pillars that are supposed to contain the majority of the state-of-the-art solutions when referring to smart island grids.

The innovative energy solutions proposed for an autonomous grid can be categorized into five (5) main thematic pillars. These are the following:

• DR services with the use of predictive algorithms;
• smartening the distribution grid through advanced monitoring and predictive models;
• energy storage provided with the use of Battery Energy Storage Systems (BESS) or heating storage, as well as storage management through models and algorithms;
• smart integration of grid users through the transportation sector, using the flexible capacity of electric vehicles and boats; and,
• domestic heating/cooling systems, using renewable technologies coupled with energy/heat storage options.

If the project does not only concern an isolated grid but also focuses on methodologies to decrease load and increase RES penetration, then other thematic pillars can be proposed to fit other priorities, such as desalination, industrial production, water management, etc.

This high-level segmentation sets the first methodological layer for the proposed KPI analysis. The thematic pillars should be taken into account when evaluating any new innovative concept. These pillars represent the main categorization of the solutions tested, so that the evaluation of a pilot/demonstrator can be held according to them.

It should be underlined that, not all studies refer to all of the technology pillars mentioned. For example, a study could focus on many solutions of even one of the enlisted technology pillars, which would make it compulsory to change the list of pillars into a more focused one. For instance, a project concerning only storage could define other type of pillars, well fitted to the requirements and orientation of each project (e.g., hydro, Hydrogen, second life batteries etc.). All in all, the selected pillars should provide a cognitive structure for the holistic project evaluation, though the current proposed one is assumed to be pretty generic.

2.2. Stakeholder’s Perspective

As mentioned, the assessment of a technology solution becomes more useful and meaningful on the condition of the examination of the various perspectives of the relevant key stakeholders. Their goals and requirements are needed for the definition of the main strategy to be followed for the overall evaluation in terms of advancements and new expertise gained. The four categories of stakeholders that are referred below can represent all the stakeholders’ points of view, concerning the development of smart grids.

2.2.1. Transmission/Distribution System Operator (TSO/DSO)

TSOs and DSOs are responsible for the management and operation of the transmission and distribution network of electricity, respectively. The operators are responsible for control rooms and various ICT systems for power transmission/distribution management and automation in the High Voltage/Medium Voltage/Low Voltage (HV/MV/LV) grid electricity network. In addition, depending on the legislation of each country, a DSO might be responsible for the reduction of energy consumption requests; in the competitive electricity market, the distribution of electricity is sometimes a monopoly that is controlled by the regulating authorities.

2.2.2. Consumers (End Users)

The role of the customer in the energy system can change from a passive user, i.e., simply using energy from the energy grid, to an active participant in the energy system, i.e., reacting to signals in the market and delivering energy services to the grid and market participants. Actually, one of the main objectives of Smart-Grid related projects is to ensure and promote the active participation of end users in market and grid operations; thus, special focus should be paid to the evaluation of End Users engagement within the context of the project. The consumers can be sorted as residential, non-residential, and industrial, in order to examine the end-users’ role in the grid level in more detail:

• Residential consumers: Their main interest is the low price, with a probable environmental care about the electricity mixture. Questionnaires can be used in order to deal with the acquisition of local residents’ point of view.
• Non-residential consumers: Their main interests are grid security and sustainability, as well as the provision of energy (electric, thermal) for a low price. This category includes facilities, offices, urban lighting, and generally non-residential buildings.
• Industrial consumers: They have multiple roles. On the one hand, they are large-scale end-users who are often connected to the HV grid, so they demand high quality electricity power. Since they play a vital role in the economy of a country, they can put pressure on the decision makers for the provision of low-cost electricity power, with a view to getting an advantage in the transnational competition. On the other hand, they can provide the operators with ancillary services because of their capability to considerably increase/reduce the load.

2.2.3. Market Operator (MO)

This category includes electricity producers, aggregators, and suppliers. More specifically, the traditional utility operators and their expected new business roles are considered. Energy Service Companies (ESCOs) and DR Aggregators are the responsible parties to manage the technology utilized for DR and negotiate on behalf of their customers with the operator for the provided services.

Energy Service Companies (ESCOs), Aggregators and retailers are interested (a) in monitoring and analysing the behaviour of the end-users; (b) in validating the operational credibility of the technological installations supporting alternative DR schemes, in order to identify potential profile deviations; and, c) in evaluating the impact of the benefits generated by the applied policies. Towards this direction, it is essential for the study to evaluate the impact of the different strategies (DR, Storage and Electric Vehicle management) to the different market stakeholders.

Furthermore, the term ‘prosumers’ refers to agents that both consume and produce energy at local level. The growth of small and medium-sized agents using solar photovoltaic panels, smart meters, vehicle-to-grid electric vehicles, home batteries, and other ‘smart’ devices, induces the increase in flexibility in the electricity networks. As the number of prosumers increases, the electricity sector is likely to undergo significant changes over the coming years, offering possibilities for greening of the system. However, demand reduction implications on the grid have not been implemented yet; managing a grid incurs mainly a fixed cost and the more the use of the grid reduces, the more the percentage cost of the grid maintenance increases and is undertaken by the remaining users of the grid. The main interests of a Market Operator are the profits in an energy venture, a short payback period of the initial capital cost, and a large investment lifetime.

2.2.4. Institutions/Policy-, Law-, and Regulation-Making Bodies

The institutions represent an important stakeholder group to consider. They gather policy-, law-, and regulation-making bodies at the EU, as well as at the national or local level. They define the rules under which the energy market is working and they roll-out its gradual privatization. They are responsible for the transposition of the EU regulations to a national level. So far, a clear and consistent strategy for smart grids has not been yet adopted either in a national level or in EU level. Despite the rapid improvement of individual technologies, such as renewable energy generators or about specific energy issues (e.g., environmental impact), little progress on the overall vision for a modernized smart grid is detected regarding energy management. The envisioned strategy for smart grids will integrate the appropriate technologies, solve the grid related issues, and provide the desired benefits to stakeholders and society [46]. Consequently, all the responsible institutions need to be asked to provide their vision and their opinion. Figure 1 provides an indicative explanation of each stakeholder’s point of view.

2.3. KPI Domains

The other basic axis of the proposed assessment framework lies on the definition of the domains, namely technical, economic, environmental, social, and legal ones. These domains (or dimensions) are complementing each other to set up the holistic performance framework. The KPI domains are defined as:

• KPIs measuring Technical Performance, such as the energy consumption, the RES generation ratio, the peak load reduction, etc.
• KPIs measuring Economic Performance, such as the average cost of energy consumption, the average estimation of cost savings, etc.
• KPIs of Environmental impact, such as CO₂ emissions reduction
• KPIs of Social impact such as the degree of users’ satisfaction from DR services.
• KPIs of Legal infrastructure, such as the level of support for electricity/heat integration in the legal framework in the case that there are specific provisions.

The specific domain categorization is not the only one that can be determined. There are other domain frameworks too, either close to the one presented (e.g., SCIS [44]), or quite different (e.g., CITYKeys [43]). The present study proposes the following one, as a more holistic in studies of medium to high TRL.

The Legal domain is a new aspect that is presented in this study for the first time and many stakeholders demand it nowadays.

2.3.1. Technical Domain

KPIs in the Technical Domain measure the effectiveness of a given use case with respect to the operating parameters and technical constraints acting on the HV/MV/LV grids and active/passive users. They identify and quantify the benefits that a technology solution offers to existing assets and on the quality of service provided to customers.

Technical KPIs are derived by gathering the electrical metrics on the network (e.g., voltages/currents collected along feeders and active/reactive power measured at the interface with the transmission system) and on customers and producers (e.g., active/reactive energy/power exchanged with the network). In some cases, the KPIs need to be supported by numerical simulations on the basis of a grid model and the actual measurements that are collected on the grid (KPIs aiming at evaluating the technical performance of a particular asset e.g., batteries or the model based evaluation of DER capacity in a local network).

The interest in these KPIs varies, depending on the perspective of the various stakeholders. For instance, system operators (TSOs and DSOs) are mainly concerned about KPIs that are related to the HV/MV/LV network operation, while customers are focused on KPIs assessing the performance of a new approach/strategy at their premises. However, other factors exist that could affect the relevance of the KPIs considered in the different situations, as, for example, the regulatory framework in force that can promote an improvement of the quality of service with reference to specific technical indexes, such as System Average Interruption Duration/Frequency Index (SAIDI/SAIFI), or business cases applying in each particular scenario, also in relationship with the target performances that are defined in the economic domain.

2.3.2. Economic Domain

The economic performance evaluation takes into account the business efficiency of each application and usage scenario from the market stakeholder perspective. Among the objectives of a study is to provide market viable solutions, defining business oriented KPIs to evaluate the day-to-day performance of the tools and applications under examination. For example, the residents of apartments would like to have a view of the economic benefit from their flexible consumption behavior. They may be willing to sacrifice part of their comfort to achieve lower energy bills and they would like to know what the cost/benefit ratio is. Likewise, the business stakeholder (DR Aggregator) would like to know the actual benefit from the implementation of DR strategies in a portfolio of customers.

Once again, the overall business and economic analysis is closely related to the definition of business stakeholders in the project, along with the selection of business models and it is associated scenarios to be examined at the demonstration sites of the project.

2.3.3. Environmental Domain

KPIs in the Environmental Domain are essential for understanding and evaluating the environmental impact of energy/storage and smart grid distribution related solutions they and are important for a smart system planning and operation. The environmental KPIs can be used to evaluate the efficiency of the energy systems demonstrated in environmental terms, according to the phase when the measurement is taken. For example, there are KPIs that are calculated during the operational phase (e.g., Noise Pollution Exposure), as well as those that are measured in the end-of-life phase (e.g., Energy Return on Investment). The main focus is on operational phase evaluation through the definition of KPIs that set the framework for day to day evaluation, while the Life Cycle Analysis (LCA) methodology will be applied for the determination of environmental aspects and potential impacts of a product or system from raw material extraction through production, use, and disposal, also evaluating possible recycling routes following a Cradle-to-Cradle approach.

2.3.4. Social Domain

The social aspects of energy projects were found to be the less popular among the employed KPIs in previous similar studies, although some studies and platforms are mainly devoted to them [47]. The selected indicators reveal that attitudes towards energy are interrelated with demand-response mechanisms [48], and such KPIs can be used to evaluate the extent up to which the end-users (citizens in most cases) are willing to participate and be self-motivated for further demonstration and application of the demonstrated solutions. The potential of end customers to actively participate in DR schemes is often a core aspect. In general, the social domain visualizes the impact of a technology, scheme or policy to social factors like local wealth, unemployment, satisfaction, or to even more specific factors, like the effect on the use of public transport, the health care system, etc. A popular approach that is used in literature for expressing the social KPIs is the Likert scale [49], since it is a sensible way to quantify a qualitative value.

2.3.5. Legal Domain

KPIs in the Legal Domain mainly monitor the legislative framework concerning the application and evolution of the proposed technological solutions. This specific domain is not commonly used, but it is of great importance for the R&I, as it allows for assessing the existing legal and regulatory framework and identifying the modifications that are needed for the deployment of the technology. The legal framework can also have a strong impact on the feasibility of a technology. Indeed, an early legislative support of a new technology can give a serious asset to its developer and its user on the market. Generally, market actors need a steady legislation to take the decision to invest capital. Therefore, legal certainty and clarity are of paramount importance. The Legal KPIs mainly evaluate the adaptability and adoptability of the legal and regulatory framework. This technology-legal framework alignment is difficult to be objectively quantified, so the subjective point of view of several stakeholders is requested as an input, usually in the form of a percentage scale or Likert scale.

2.4. The Interest of Each Stakeholder in the Assessment of Each Domain

The domain categorization does not have a direct connection to each stakeholder. This means that each stakeholder needs KPIs by all domains in order to overall assess the proposed solution according to his/her point of view.

Table 3. The interest of each stakeholder to the assessment of each domain.

Domains of KPIs	TSOs/DSOs	Market Operators	Consumers	Institutions
Technical domain	TSOs and DSOs are mostly interested in ensuring an adequate level of quality of supply to the grid-connected customers, taking into consideration each of the specific grid characteristics. Critical peaks of demand should be avoided, constantly monitoring users’ consumption to avoid grid breakdowns and efficiently addressing fraud challenges. In other words, Operators are interested in the operational impact of any scenario to the grid conditions.	With reference to the technical domain, Market Operators (MOs) are interested in the various technologies available for power generation and storage, as well as to the proposed DR strategies. Technology performance is crucial for any investment decision. Moreover, a better exploitation of assets devoted to improving the regulating capabilities of Virtual Power Plant (e.g., energy storage systems) would reduce the required investment costs and increase the incomes.	The quality of the power delivered is a matter of interest mainly to non-residential consumers. Especially factories and large workplaces can withstand neither power interruptions, nor large voltage variations or harmonics. Residential consumers are not as dependent to quality of service as the non-residential ones, but certainly demand it.	Policy Bodies are interested in monitoring the contribution of the projects (pilots) to the smart grid functions, which are directly related to Smart Grid policy objectives. Among others, these include the Security and quality of supply, the connectivity and access to all categories of network users, the capacity of transmission and distribution grids to connect and transfer electricity from and to users.
Economic domain	The aforementioned concerns of the TSOs and DSOs in the technical domain are also having an economic aspect, as any potential inefficiencies in the quality of supply to the grid customers, may cause significant charges from the side of the regulation authorities. Moreover, DSOs are responsible for proposing an energy strategy, giving directions about the future of the energy mixture, bearing in mind the overall cost.	Main goal of the Market operators is to maximize the profit from their investment. This means that they care for all the economic aspects of any possible technology in which they could invest. They compete to sell DR services to the utility operator and provide compensation to consumers, in order to modify their preferable consumption pattern. In this respect, they will make use of economic indicators to identify operational needs, market opportunities or critical situations and deploy appropriate DSM strategies. Any available RES promotion paying policies (feed-in tariff, etc.) are under close observation as they play a decisive role in the overall feasibility of an investment. Real-time views for revenue protection, unexpected EV and solar loads identification are some of the metrics that would make sense for utilities in such case.	The main expectation of the residential consumers is a direct economic benefit either in the form of cost reduction or in terms of at hand compensation, depending on the DR schema category they participate. Non-residential consumers also demand the lowest possible final cost, as the energy cost is one of the main factors that are included in the final cost of any kind of business, and thus is very important to the international competition.	From the perspective of policy makers, economic domain indicators should reflect the efficiency and quality of service achieved in electricity supply and grid operation. Measures of interest indicatively include: Demand side participation in electricity markets and in energy efficiency measures, societal CBA, which go beyond the costs and the benefits incurred by the project promoter, as well as the monetary value of reduced CO2 emissions based on the amount of CO2 reduction and the current CO2 allowances price.
Environmental domain	TSOs and DSOs are highly interested the effect of the new smart technologies on the environment, either when applied or when they replace conventional systems, since their electric grid, under supervision, influences significantly the cities and citizens’ quality of life. Moreover, they need to confront with the current EU legislation policies promoting the low CO2 technologies.	Market Operators are expected to apply schemes contributing in making grid distribution smarter and more efficient (e.g., DR programs by Large Scale Enterprises (LSEs) or third-party energy aggregators). Environmental KPIs related to demand determine the quality of response from the customers. Moreover, the environmental indicators are necessary for the Market Operators in order to provide the environmental profile asked by both government and end-users (market).	Both residential and commercial end-users are highly interested in knowing more about the environmental impact of any technology solution proposed. Environmental parameters are linked to and to a certain extent reflect the, demographical, physical and contextual characteristics such as types of premises and profile of users, weather conditions, national/local characteristics, idiosyncrasies and legislation etc.	Governing Bodies are interested in the levels of sustainability and would like to monitor it in a quantified manner (including the reduction of greenhouse emissions and the environmental impact of electricity grid infrastructure). International agreements are directing the local energy policies which include the increase in RES penetration and the reduction of the CO2 emissions.
Social domain	The social approach is necessary for the definition of the quality standards of the delivered services, as comfort and satisfaction are seriously taken into consideration.	Even more than the social approach of the DSOs, Market Operators (especially the utility-scale) depend on the social comfort and satisfaction by the delivered services, as it plays a crucial role in the determination of the marketing strategy to prevail over the competition.	All kinds of consumers can be motivated to change their energy behaviour through different social approach techniques, especially if there is direct monetary benefit. It further allows them to understand and feel comfortable with the energy infrastructures at home (RES, batteries, smart-meters, etc.) and improve their energy attitude.	Governing Bodies are interested in the social approach via the filter of the general evaluation of their general policy that has to be acceptable to the highest possible population percentage.
Legal domain	Being responsible for the operation of the grid, TSOs and DSOs are tempted to apply the most suitable mix of technologies according to their needs. The legislative framework sets the drivers and barriers to network operators’ freedom to optimize the grid operation, as long as they do not engage in the energy market.	Market operators are affected by the legal framework. They purchase technologies that have to cope with specific official standards. Yet, the permission to use a technology and the rules under which the market operates are set by the legal and regulatory framework. A very serious point here is the profitability of an investment. Generally, in a global market, the sooner an innovative technology is applied, the bigger market share it will acquire. This is why legislators or the executive are often pressed by companies to adjust the legal or regulatory framework according to the technology progress as soon as possible, if needed.	Consumers historically are the least involved in the legal domain. They seldom have to alter their position according to the changes in the legal framework, at least not as much as other actors. However, the recent drivers towards more active consumers (sometimes labelled prosumers) might entail an increasing interest from these actors in the changes brought to the legal and regulatory framework. The winter package intensifies this process [50].	Institutions are at the source of the legal and regulatory framework. They are responsible for the development of a legal infrastructure that takes into consideration all the allying/opposing interests in order to define the barriers among which these interests can be expressed. It could be said that the legal KPIs evaluate the efficiency of the institutions, and specifically their ability to set up adapted market rules for the integration of new technologies to the energy market.

shows the connection of each stakeholder and each domain.

2.5. The Approach of KPI Determination

The goal of the proposed approach, is to assist each stakeholder to raise the most interesting questions from his/her point of view, as well as the most intelligible presentation of the answer. Therefore, the determination of the KPI list should be a result of the questions that are made by each stakeholder in order to evaluate each technology solution according to all the possible domains of interest. This is schematically depicted in Figure 2.

The methodology described can prove to be the most reasonable way to include every viewpoint of evaluation of any possible reader (who will represent in reality a different stakeholder) of the study.

In addition, the presentation of the KPIs in this way is more likely to be understood through a domain categorization. A separate list for each stakeholder would not be helpful, since most of the KPIs are of interest for more than one stakeholder. Domain categorization is the most usual practice, even though the list of domains is not always the same.

Thus, after the provision of the various stakeholders’ point of view about the proposed solutions through the five (5) above described domains, a final list of KPIs can be made after the interaction among the demonstrators. Figure 3 schematically depicts the overall assessment methodology proposed. The procedure of KPI identification and assessment is accomplished in three consecutive phases. In Phase 1, the various stakeholders of each demonstrator propose the KPIs that interest them, for the evaluation of the various technology pillars. In the meantime, the demonstrators are in close collaboration, giving feedback to each other and interacting in order to make an optimized integration of the KPIs proposed. In Phase 2, the KPIs are grouped in a final list, which is presented divided into the five (5) afore-explained domains. Finally, the consolidated list of KPIs is returned to the demonstrators for calculation. Figure 3 depicts the methodology for the case of a project with three (3) island demonstrators as in “SMILE” project.

3. Results—KPI Identification

A typical result of the proposed methodology i.e., a list of KPIs categorized per domain with the corresponding stakeholders group of interest are presented in this Section. It should be underlined that the specific list is an indicative paradigm that is based on the evaluation that has been conducted in the framework of the SMILE project. Accordingly, the list may be modified according to the specific topic of each project that is implemented. Nonetheless, most of the KPIs proposed can certainly prove to be useful for the majority of such projects. The technical, environmental, economic, social, and legal KPIs are listed in the

Table 4. List of Technical KPIs.

Name of KPI	Description	Unit	Stakeholders in Charge
Share of RES: (a) electricity, (b) heating/cooling and domestic hot water (DHW)	RES penetration for covering a) electrical and b) thermal needs	%
Share of DER (decentralized/distributed energy resources)	Share of DER in the energy mix	%
Peak shaving from the side of consumption	Reduction of the power peaks	% of peak power reduction
Generation Forecasting Accuracy	Confidence or fuzziness (risk) in RES generation forecasting	RMSE (root mean square error)
Energy Losses	Yearly amount of energy lost on grid’s conductors, transformers, etc.	kWh/year
Voltage variations	Difference between the actual voltage supplied to MV/LV users and the nominal value (indicatively would better be between −5% and 5% [51])	%
On-site Energy Ratio	Relation between the annual energy supply from local renewable sources and the annual energy demand	%
Maximun Hourly Surplus-Deficit (MHS-Dx)	The maximum value of how much bigger the hourly local renewable supply is than the demand during that hour (per year)	KWh
Reduced Energy Curtailment of RES/DES	The difference between the energy curtailments before and after the integration of a/all the proposed solutions.	%
Grid Congestion	Grid sustainability to peaks	%
Battery degradation rate	The rate at which the battery performance is deteriorating over a year/cycle	%
System Average Interruption Frequency Index (SAIFI).	Measures the average frequency of power-supply interruptions in the system (indicatively would better be <1.5 interruptions per customers and year [52])	$\frac{interruptions}{customer·year}$
System Average Interruption Duration Index (SAIDI).	Measures the average cumulative duration of power-supply interruptions in the system (indicatively would better be <150 min per customer and year [52])	$\frac{minutes}{customer·year}$
Unbalance of the three-phase voltage system	Difference in the voltage of the three phases	%
Harmonic distortion	The Total Harmonic Distortion unit (THDu) indicates the distortion of the voltage wave. There are other THD factors that give relative information about the power, the current etc. (indicatively, would better be ≤5%)	%
Storage Energy Losses	Losses because of energy storage solutions	%
Degree of self-supply	Measures the percentage of PV generation which is used for self-supply, and not injected to the grid.	%
Frequency Control	Calculates the percentage of times that the average value of the fundamental frequency measured over periods of 10 s goes out of the stated ranges.	%

,

Table 5. List of Environmental KPIs.

Name of KPI	Description	Unit	Stakeholders in Charge
EROI	Energy Return on (Energy) Investment taking into consideration the component’s whole life time (indicatively for PVs it is usually >6 [53])	MWh (usable energy)/MWh (energy used to obtain that energy resource)
CO2 tonnes saved	Tonnes saved per annum as compared with gas and grid electricity	tonnes CO2
Noise Pollution Exposure	Noise pollution in residential areas, compared to previous condition.	%
Reduced Fossil Fuel Consumption	Reduction in the fossil fuels consumption for heating, transportation and power generation	TOE/year
Carbon Footprint of Heating House	Examines the carbon footprint for heating a house with(out) the project’s proposed solutions	Kg CO2/year

,

Table 6. List of Economic KPIs.

Name of KPI	Description	Unit	Stakeholders in Charge
Life-cycle cost of energy generation (€/ΜWhel or €/ΜWhth)	The sum of all the costs throughout the lifetime of the energy investment, normalized to the energy generated.	(€/ΜWhel or €/ΜWhth)
Internal Rate of Return (IRR)	Profitability of an investment (indicatively a wide value range lies between 5 and 30% for PV investments [54])	%
ROI	Return on investment (indicatively can rise above 20% for PVs [55])	%
Payback Period	The period of time needed for the cumulative gains from an investment to equal the cumulative cost (indicatively could vary between 5 and 20 years for PVs [28]).	Years
Annuity Gain	Measures the annual profits of an investment throughout its lifetime.	€/y
Total capital cost per kW installed	Examines the initial cost of an investment depending on the size of the capacity being installed	€/kW
Feed in Tariff	Energy policy which provides guaranteed price to RES energy investors	€
Heating Prices	The current price heating energy.	€/kWh
Load purchasing from mainland	The amount of money for the power that has to be purchased from the mainland	€
Fossil Fuel purchasing from mainland	The amount of money for the fossil fuels that have to be purchased from the mainland for heating, transportation and power generation	€
Transportation Cost	Calculation of the fuel cost for electric transportation (indicatively ranges around 12 €/100 km for a typical family car depending on the cost of gas and whether it is in urban or rural environment [56])	€/100 km

,

Table 7. List of Social KPIs.

Name of KPI	Description	Unit	Stakeholders in Charge
Improved access to online services	The extent to which access to online services was improved	Likert scale
Increased environmental/sustainability education	The extent to which the project has used opportunities for increasing environmental awareness and educating about sustainability and the environment	Likert scale
City’s unemployment rate	Residents unemployed as a share of all economically active residents	%
DR scheme sensibility	Are consumers satisfied with the DR policy?	Likert scale
EV scheme sensibility	Are consumers going to be using EVs within the next 15 years	Likert scale
Thermal Comfort	Evaluation of the performance of the heating solutions proposed	Likert scale
Degree of Landscape Impact	Refers to the possible opposition from citizens. A wind turbine or battery may look ugly or obstruct the view to the horizon. An aesthetical measure.	Likert scale

and

Table 8. List of Legal KPIs.

Name of KPI	Description	Unit	Stakeholders in Charge
Local grid balancing legal framework development	The extent to which local grid balancing technologies’ regulation is suitable at EU level and at the partners’ islands level	%
Micro-grids legal framework	The extent to which micro-grids regulation is suitable at EU level and at the partners’ islands level	%
Suitable Energy Storage Regulation	The extent to which energy storage regulation is suitable at EU level and at the partners’ islands level	%
Monitoring and Evaluation	The extent to which the progress of policies/strategies/projects is evaluated and is adapted according to the findings	Likert scale

, respectively. What is more, the description of some characteristic KPIs out of the extended KPIs list in these tables is accompanied by indicative values as examples. The symbols of the stakeholders in the last column of each table have been explained above in Figure 1.

4. Conclusions and Further Considerations

This study presents an evaluation framework for the development of smart grids in island energy systems. In particular, for the definition of the necessary KPIs, a three-axis framework is proposed that includes: (a) the technology pillars; (b) the stakeholders; and, (c) the domains of interest.

Any project performing a study and comparing different situations (the old technical equipment against the state-of-the-art) needs quantified results presented in the most proper way in order to render the comparison comprehensible regarding certain points that require credible evaluation. The determination of the respective list of KPIs for the holistic evaluation of any type of technical interventions, towards the above-mentioned directions is not generic for all projects; though the current proposed list can be regarded as flexible enough to fit to, and be followed in any similar type of project. On the other hand, the methodology for the KPI determination should be holistic posing the right questions:

• What technology solutions are tested?
• What aspects of these solutions should beg under concern and evaluation?
• Who is interested? Who sets the evaluation criteria?

The answers to these questions lead to the most well-fitted and correct categorizations. Thus, the evaluation approach should in a way always integrate the thematic pillars, the domains, and the stakeholders. The third one is the key aspect, which is, most of the time neglected, but can enhance the formulation of a holistic approach being in position to address an evaluation platform from every possible viewpoint of interest. The specific study proposes a typical and decent categorization, to achieve that. With the introduction of this proposed methodology, the stakeholders are responsible to define criteria of assessment for all domains of each thematic pillar, according to their interest; thus providing a holistic list of KPIs that suitable for the specific conditions and needs of a project, which can be replicated in any similar project, by the introduction and/or modification of the defined list of domains. In addition, the Legal domain, which is a novelty of the study as lacking in the existing literature, adds essential information concerning the adjustability of the existing legal infrastructure to the needs of the tested technology solution.

The presented final list of 45 KPIs could be used as a typical sample for project evaluation concerning autonomous grids, which can however be used to additional types of similar smart-grid projects. The results of such an assessment can be consolidated to deliver higher-level results, which can assess the overall performance of the solution, even its general impact on a society. This would need a generalization from the specific demo-site level, up to the whole island level, or even reaching the level of EU. The optimization and sensibility of such a generalization can be the object of a further near-future study.

The evaluation process through the use of KPIs is of great importance, as it indicates the success degree of the research and the potential development. All of the interested stakeholders can inspect the KPIs values and get a clear picture of the progress that is made. In that respect and in order to improve and strengthen the impact of solutions demonstrated, the evaluation has to be carried out inductively, i.e., from part-level to whole-level approach). Such a route approach can also achieve the successful passage from the specific case studies to a more generalized scheme. That is the reason why the evaluations of each case study need to be generalized, taking benefit from the smaller-scale experience that is gained by similar case studies towards a greater scale (i.e., from pilot grid level up to whole island level, see Figure 4).