An Indicator-Based Methodological Framework for Assessing an eMaaS Scheme

Abstract

Mobility as a Service (MaaS) and, more recently, electric Mobility as a Service (eMaaS) have increasingly been put forward to meet the economic, social, and environmental challenges linked to mobility. First, however, monitoring and evaluating such a scheme’s performance is crucial, mainly through the definition of appropriate indicators. In this study, a standardised methodological approach is presented for the assessment of an eMaaS scheme. In addition, this methodological approach contains a range of innovative Key Performance Indicators (KPIs). The proposed KPIs refer to the evaluation of the scheme based on four pillars: (a) society, (b) users, (c) operators, and (d) internal operation. The methodology for evaluating the proposed KPIs includes identifying the available sources for data collection. For example, data can be collected through questionnaire surveys, focus group discussions, and the system’s central dashboard. An appropriate set of indicators to evaluate a system from various perspectives is necessary to assess an eMaaS scheme in real-life conditions. Furthermore, the evaluation of the overall operation of the scheme will contribute to drawing valid conclusions (e.g., user acceptance, economic viability) for the implementation of eMaaS in urban areas.

1. Introduction

Individual motorised transport represents a significant share of trips and passenger kilometres in most cities. Worldwide, 1.3 billion vehicles are now in use, and many of these are privately owned [1]. To this end, cities worldwide are implementing policies to limit private vehicle use and decrease car ownership to achieve a mode shift away from private vehicles. Such policies include, amongst others, the promotion of shared mobility and the increased uptake of transport-sharing schemes such as Mobility as a Service (MaaS) [2,3].

MaaS promises a more efficient and user-centred mobility paradigm that replaces privately owned transport and optimises the use of mobility resources. MaaS is a model for supplying a wide range of passenger transport services (such as car-sharing, car rental, underground, rail, bus, bike-sharing, and taxis) through a single platform. MaaS platforms typically provide an intermodal journey planner, a booking system, easy payment, and real-time information [4]. Overall, MaaS is an emerging, newly born transport initiative with limited implementation so far, as none of the implemented MaaS systems worldwide has yet demonstrated long-term commercial viability [5,6].

Based on [7], road transport accounts for more than 20% of the total greenhouse gas (GHG) emissions, making electrification of transportation a cornerstone for achieving climate and energy targets. Furthermore, electrification is an alternative energy source for decarbonising transportation systems and supply chains, particularly in last-mile operations. The transportation industry is working to lower carbon emissions by shifting towards electric vehicles (EVs), causing a rapid and significant transformation in the e-mobility landscape. Recent works and advancements in the electrokinetic field and the increasing popularity of EVs are driving this change [8,9,10]. Therefore, vehicle electrification is one of the most promising solutions to reduce CO₂ emissions from road transport effectively [11].

Achieving sustainability is a top priority for both people and the environment. Therefore, finding ways to decrease traffic congestion and lower net GHG emissions is crucial. Two crucial components of transportation systems that can play a key role in achieving the above objectives are shared mobility and electrification, as they have the potential to enhance numerous aspects of transportation. eMaaS, or electric mobility as a Service, is an extension of MaaS that goes beyond the basic concept by providing an “eco-friendly” and accessible route. The main focus of eMaaS is electric vehicle-sharing services, which offer the most potential in the mobility services market. This system could transform transportation in the future by providing a personalised on-demand mobility model while producing no emissions [12].

eMaaS, like many other domains, is data-rich and data-driven. Data from various sources, such as vehicles, chargers, operational systems (like car-sharing platforms), and users who have opted in, can be used to enhance and improve the services provided to users. To effectively analyse the collected data and draw conclusions about the performance of an eMaaS system, it is necessary to establish an impact assessment framework. This framework will aim to identify the institutional factors, both key enablers and potential barriers, that play a significant role in the successful development and implementation of eMaaS. Creating personalised solutions and recommendations that align with the objective is crucial for accepting and adopting innovative systems. In addition, consistent monitoring and evaluation of the system’s performance through accurate and reliable indicators, is also crucial.

Based on the literature, the evaluation of a MaaS system is based on four primary levels: (a) society, (b) users, (c) operators such as transport operators and providers, and (d) internal operation.

When assessing a system’s potential success, it’s essential to consider how accepted it is by society. This involves examining citizens’ perceptions and intentions towards the system, especially those who might use it. Implementing a full-scale MaaS system requires considering various factors, including user characteristics such as gender, age, and income, as well as their access to public transportation services and knowledge of MaaS [13].

To gather information on the potential users’ attitudes towards the MaaS system, stated and revealed preference surveys are conducted, as there is currently no data available on the market. Participants are asked to express their willingness to subscribe to hypothetical customised MaaS scenarios [14]. Numerous questionnaires, interviews, and studies on stated preferences have been conducted in different countries to determine the traits and characteristics of potential users (or adopters) [15,16].

Various studies have examined the possibility of people using and paying for MaaS systems. These studies have examined the level of interest and willingness to utilise MaaS services. The uncertainties of becoming a MaaS customer may impact how individuals respond to subscriptions, bundling, and willingness to pay [17,18]. Other studies have tried to identify the characteristics of mode usage linked to these systems’ probable acceptance [6,19,20]. On the other hand, some researchers looked into how MaaS could change the travel habits of people who may use it. The main question they were trying to answer was whether MaaS could encourage more sustainable travel behaviour [18,21].

In some stated preference studies, thematic indicators or factors were identified as related to the service’s needs, requirements, or added values rather than individual or mode use characteristics [22,23]. Specific studies focusing on preferences have explored the expectations of MaaS users, including the desired characteristics of an ideal service [21]. However, participants may find it challenging to anticipate a novel service or method of organising their transportation needs [24].

The second aspect of the evaluation involves examining how satisfied current users are with the system. Please note that this service is currently in its initial stage and is only available as a limited service for pilot projects in certain regions rather than as a universal service. Most research on MaaS has concentrated on examining pre-conditions, needs, requirements, and the initial awareness stage in the innovation-decision process, as there have been few opportunities for empirical study thus far. So far, there is limited information about the demographic profile of MaaS users from pilot programmes and services [22,25,26,27].

A commonly used method in research is to conduct pilot projects and gather data from them. Numerous studies have been conducted to examine the attractiveness of the MaaS market in various contexts. These studies involved selecting specific participants and MaaS packages [28,29]. Analysing the results of pilot projects and conducting ex-post empirical analysis can offer valuable insights from specific experiences and case studies [30,31,32]. The information gathered may include details about the trip patterns of MaaS users and the travel habits of respondents both before and after the pilot was introduced [33].

A few empirical studies on actual MaaS users have also explored users’ motives (or drivers). Motives can also be described as the perceived benefit or added value from a business point of view. Studies have aimed to determine the service features contributing to creating value and user satisfaction. These include available modes, pricing, ease of use, customer experience, customisation, flexibility, convenience, improved safety and reliability, innovation, and environmental responsibility [25,31,34]. Some studies have looked at how the values and attributes mentioned above can hinder adoption if they do not provide a perceived improvement over the current solution [16,35].

Another critical aspect regarding the impact assessment of the MaaS system pertains to the evaluation of the system by the different stakeholders participating in the scheme or potential participants, such as transport providers, operators, and also public authorities, which are considered policymakers in the broader area in which the system operates, i.e., regions, municipalities, and public transport authorities.

Qualitative and quantitative research methods have been utilised to better understand the preferences and attitudes of public and private operators in the transportation industry towards MaaS. These methods include focus groups, questionnaire surveys, personal interviews, and round table discussions [31].

Questionnaire surveys covered various topics related to MaaS, such as understanding and characteristics of the concept, important modes, service features, geographical areas, and stakeholders. In addition, deployment issues, including enablers and obstacles, and potential positive and negative impacts were also included [16,36,37].

Some studies conducted personal interviews with actors who are currently part of a MaaS system or are interested in joining one in the future. The purpose of these studies was to analyse a range of factors that can be grouped into two main categories: (i) perceived uncertainties about roles, mandates, business models, and proof of concept, and (ii) issues related to cooperation and collaboration, such as questions about responsibilities, a lack of clear leadership, and fears of being dominated by other actors and losing control over development [31,38,39,40]. In addition, operators/providers need to consider the viability of MaaS, such as whether it presents a business opportunity and if there are reasonable profitability margins. This is vital information to have when assessing a MaaS system [5,26,31].

The fourth and last aspect of the assessment involves evaluating the system’s internal operations. While there has been much research on user preferences and attitudes towards MaaS, including focus groups, surveys, and round table discussions with stakeholders, more research must be done on the system’s internal operations.

Studies have been conducted to determine the potential adoption of various MaaS systems by analysing users’ travel patterns, trip characteristics, and habits [22,23,41]. Other studies have examined how the travel behaviour of an average commuter differs from that of a MaaS user [21,34,37,41]. Demographic characteristics, such as gender, age, education level, etc., and user profiles, such as the number of trips made, modes used, and subscription programme, also played an essential role in the assessment of the internal operation of the system [5,28,42].

Since MaaS is a relatively new concept, there is a shortage of readily accessible information regarding its impact and MaaS-related services. For example, there is a lack of research investigating the reasons behind the observed behaviour patterns (usually intentional) related to MaaS. In addition, evaluations have primarily focused on how users’ behaviour changes, such as their willingness to switch transportation modes or pay for transportation-related expenses. However, there has been a limited assessment of the impact on businesses, and this information is not widely accessible. Furthermore, there needs to be more information available for eMaaS systems, which are still in the early stages of implementation.

This study addresses the research gap mentioned above by introducing a new methodological assessment framework to assess and evaluate an eMaaS system. The methodological framework will concentrate on different factors and viewpoints essential for assessing an eMaaS programme in actual situations. Overall, this study aims to address the following research questions:

• What are the most important aspects and factors that should be used to evaluate an eMaaS system?
• How can the above aspects and factors be measured and assessed?
• How can the proposed methodological framework identify institutional factors (enablers and barriers) affecting the development and implementation of an eMaaS system in Greece?

This paper presents and analyses key methodological steps implemented in an eMaaS project to meet the above research objectives. The project’s primary goal is to implement a new, innovative electric sharing system for Greece by integrating electric bike-sharing, electric scooter-sharing, electric car-sharing, and park-and-ride services. The eMaaS project will implement a pilot study in Thessaloniki, Greece. After the end of the pilot operation of the project, the overall evaluation of the system’s operation will be assessed by using the proposed assessment framework presented in this study.

The remainder of this paper is as follows. First, in Section 2, the methodological approach of the present study is explained. Then, the proposed indicator system is presented in detail in Section 3. Finally, this research’s main outputs and conclusions are summarised in Section 4.

2. Materials and Methods

This study aims to develop a standardised methodological approach for assessing an eMaaS scheme. Based on the literature review, the most important aspects and factors commonly used to evaluate a MaaS system were identified. In addition to the parameters already used for assessing the system based on the literature, additional parameters related to the evaluation of electromobility, micromobility, and sustainable urban mobility, in general, were added.

To properly evaluate the system’s performance, we must not only identify which parameters to assess but also determine the method of assessment. MaaS has increasingly been put forward to meet the economic, social, and environmental challenges linked to mobility. However, as with any project, monitoring the system’s performance is essential, particularly by defining indicators. Indicators and, more specifically, Key Performance Indicators (KPIs), due to their specific properties and characteristics, are a tool with broad acceptance and valuable methodological tools for measuring and evaluating the performance of transport systems. Transport systems are assessed against policy goals using KPIs. These measurable indicators track performance over time for a specific objective. KPIs provide clear targets, progress milestones, and valuable insights to decision-makers, helping them evaluate system performance, identify areas for improvement, and make informed decisions [41,43,44,45,46,47,48].

Based on the above, the methodological approach is based on various KPIs commonly mentioned about evaluations of transport-related interventions in general and those argued regarding MaaS and MaaS-related services. The proposed KPIs refer to the assessment of the scheme based on four pillars, as identified by the literature review: (a) users of the eMaaS scheme, (b) society-city (potential users), (c) eMaaS system (internal operation), and (d) operators.

The proposed assessment framework, as presented in Figure 1, in addition to introducing a series of innovative KPIs, also describes the process of calculating these indicators alongside identifying the available sources for data collection. The proposed sources and data collection methods for evaluating the relevant KPIs are presented below for each pillar.

2.1. Users of the eMaaS System

This topic concerns collecting data and information on investigating the degree of satisfaction of existing users of the eMaaS system. In general, user satisfaction is the degree to which a user’s needs, expectations, and preferences are met by a product or service, in this case, the eMaaS system. In this category, data are collected on topics such as

• Demographic profile of existing users (age, gender, income, etc.);
• The overall level of user satisfaction;
• Level of user satisfaction concerning the individual elements of the system;
• Identification of the system’s weaknesses;
• Suggestions for system improvement.

All relevant data can be collected through questionnaire surveys to existing users of the eMaaS system.

2.2. Society—City

The second topic concerns evaluating the system from the perspective of society, i.e., potential users. Essentially is related to the investigation of the degree of acceptance, perceptions, and intentions of citizens for the eMaaS system and includes the collection of data on issues related to the following:

• Demographic profile of citizens (age, gender, income, most frequent transport mode used, etc.);
• Attitudes and perceptions towards electric vehicles, shared vehicles, and micro-mobility;
• Intention to use the eMaaS system (demographic profile and trip characteristics of potential users);
• Identification of potential benefits and barriers concerning using the eMaaS system
• Degree of acceptance of the eMaaS system.

Data and relevant information can be collected through questionnaires to the general population (potential users) where the eMaaS system is applied.

2.3. eMaaS System

This pillar concerns the evaluation of the internal operation of the eMaaS system and the collection of data on operational elements, such as

• Usage data;
• Trip characteristics and trip patterns;
• Users profiles;
• Multimodality, i.e., different modes used.

Data for the above can be easily collected through the central dashboard of the system, where data related to the internal operation of the eMaaS system (such as rentals, trip characteristics, origin and destination data, trajectories, etc.) are collated and stored.

2.4. Operators

The last pillar concerns the collection of data and information about the evaluation of the system by transport operators/providers and public authorities currently involved, as well as potential participants in the scheme. Data usually refer to the following:

• Operational planning, exploitation, and economic viability of the system—based on cost/revenue data;
• Satisfaction of operators participating in the scheme based on collaboration characteristics, knowledge, data, and revenue exchange flow;
• Intention to participate (enabling factors and barriers regarding potential participation);
• Intention to promote and financially support the system by public authorities at the municipal and regional levels.

Data for the above can be collected from the system’s central dashboard (cost/revenue data) and through focus groups and personal interviews with potential participants in the scheme, as well as transport operators and providers already involved.

3. The Proposed Indicator System

This paper presents a new indicator system to help stakeholders and authorities evaluate a transport initiative like eMaaS. It should be noted that despite being developed in the context of a pilot project being implemented in Thessaloniki, Greece, this indicator system could be easily transferred and applied to other projects implemented worldwide, mainly due to the selection of indicators not relying on sophisticated or case-specific input data.

The proposed indicator system includes a targeted selection of 67 indicators, deriving from the pillars of users, society, system, and transport operators. It should be highlighted here that several indicators in the proposed system stem from the pool of indicators generally used to assess MaaS systems. At the same time, the rest are new indicators that have not been considered in evaluating MaaS systems, particularly in the assessment of eMaaS systems.

The proposed indicator system is organised into six (6) levels, i.e., pillars, objectives, indicators, the unit of measurement, data source, and anticipated goal, and is illustrated in

Table A1. The proposed system of indicators.

Pillar	Objective	Indicator	Unit	Data Source	Goal
Users	Examination of trip characteristics of system users	iU1: Number of rentals per user	No.	eMaaS Platform	↑
		iU2: Total usage cost per user	EUR	eMaaS Platform	↓
		iU3: Average travel time per user	min.	eMaaS Platform	↓
		iU4: Average parking time duration per user	min.	eMaaS Platform	↓
		iU5: Average walking distance per user	m	eMaaS Platform	↑
		iU6: User type (occasional, subscription-based)	No.	eMaaS Platform	↑
	Investigation of the degree of user satisfaction	iU7: Share of respondents with a positive attitude towards the system	%	Questionnaire Survey (system’s users)	↑
		iU8: Share of respondents with a positive attitude towards the system’s app	%	Questionnaire Survey (system’s users)	↑
		iU9: Share of respondents with a positive attitude towards available subscriptions	%	Questionnaire Survey (system’s users)	↑
		iU10: Share of respondents with a positive attitude towards the overall cost of using the system	%	Questionnaire Survey (system’s users)	↑
		iU11: Share of respondents with a positive attitude towards the number of available shared vehicles	%	Questionnaire Survey (system’s users)	↑
		iU12: Share of respondents with a positive attitude towards the spatial distribution of available shared vehicles	%	Questionnaire Survey (system’s users)	↑
Society (Potential Users)	Investigation of respondents’ familiarity with system components	iS1: Share of respondents who are aware of electromobility	%	Questionnaire Survey (general population)	↑
		iS2: Share of respondents who are aware of micromobility	%	Questionnaire Survey (general population)	↑
		iS3: Share of respondents who are aware of shared mobility	%	Questionnaire Survey (general population)	↑
	Investigation of respondents’ attitude towards system components	iS4: Share of respondents with a positive attitude towards electromobility	%	Questionnaire Survey (general population)	↑
		iS5: Share of respondents with a positive attitude towards micromobility	%	Questionnaire Survey (general population)	↑
		iS6: Share of respondents with a positive attitude towards shared mobility	%	Questionnaire Survey (general population)	↑
	Examination of respondents’ intention to use the system and identification of main reasons for potential use/no-use	iS7: Share of respondents who are willing to use the system	%	Questionnaire Survey (general population)	↑
		iS8: Share of respondents who are willing to use the system for environmental reasons	%	Questionnaire Survey (general population)	↑
		iS9: Share of respondents who are willing to use the system for economic reasons (e.g., discharge of ownership costs)	%	Questionnaire Survey (general population)	↑
		iS10: Share of respondents who are willing to use the system for flexibility reasons (e.g., decrease of time spent for parking search)	%	Questionnaire Survey (general population)	↑
		iS11: Share of respondents who are not willing to use the system due to its pricing policy	%	Questionnaire Survey (general population)	↓
		iS12: Share of respondents who are not willing to use the system due to its limited services (e.g., limited area coverage, low availability of transport modes, etc.)	%	Questionnaire Survey (general population)	↓
		iS13: Share of respondents who are not willing to use the system due to the lack of transport infrastructure (e.g., lack of an integrated and extended cycle network)	%	Questionnaire Survey (general population)	↓
		iS14: Share of respondents who are not willing to use the system due to the system’s mobile app unattractiveness	%	Questionnaire Survey (general population)	↓
		iS15: Share of respondents who are not willing to use the system due to the absence of integration of public transport	%	Questionnaire Survey (general population)	↓
System	Measurement of system usage	iSy1: Number of system’s users—registered users	No.	eMaaS Platform	↑
		iSy2: Number of rentals per day	No.	eMaaS Platform	↑
		iSy3: Number of rentals per month	No.	eMaaS Platform	↑
	Examination of the characteristics of the trips made using the system	iSy4: Average trip length (average distance travelled) of trips made by any means of transport	m	eMaaS Platform	↑
		iSy5: Average trip length (average distance travelled) of trips made by electric car	m	eMaaS Platform	↑
		iSy6: Average trip length (average distance travelled) of trips made by electric bike	m	eMaaS Platform	↑
		iSy7: Average trip length (average distance travelled) of trips made by electric scooter	m	eMaaS Platform	↑
		iSy8: Average trip duration of trips made by any means of transport	min.	eMaaS Platform	↑
		iSy9: Average trip duration of trips made by electric car	min.	eMaaS Platform	↑
		iSy10: Average trip duration of trips made by electric bike	min.	eMaaS Platform	↑
		iSy11: Average trip duration of trips made by electric scooter	min.	eMaaS Platform	↑
		iSy12: Average speed of trips made by any means of transport	km/h	eMaaS Platform	↑
		iSy13: Average speed of trips made by electric car	km/h	eMaaS Platform	↑
		iSy14: Average speed of trips made by electric bike	km/h	eMaaS Platform	↑
		iSy15: Average speed of trips made by electric scooter	km/h	eMaaS Platform	↑
		iSy16: Total daily vehicle kilometers travelled (vkm) by any means of transport	vkm	eMaaS Platform	↑
		iSy17: Daily vehicle kilometers travelled (car)	vkm	eMaaS Platform	↑
		iSy18: Daily bike kilometers travelled	vkm	eMaaS Platform	↑
		iSy19: Daily scooter kilometers travelled	vkm	eMaaS Platform	↑
	Identification of trip patterns (Origins-Destinations—ODs) and traffic flows	iSy20: Most frequent Origins and Destinations of trips made by electric car	X, Y (coordinates)	eMaaS Platform	-
		iSy21: Most frequent Origins and Destinations of trips made by electric bike	X, Y (coordinates)	eMaaS Platform	-
		iSy22: Most frequent Origins and Destinations of trips made by electric scooter	X, Y (coordinates)	eMaaS Platform	-
		iSy23: Electric car traffic flows (traffic volumes) per network road segment, during different time periods of the day	Traffic volume	eMaaS Platform	↑
		iSy24: Electric bike traffic flows (traffic volumes) per network road segment, during different time periods of the day	Traffic volume	eMaaS Platform	↑
		iSy25: Electric scooter traffic flows (traffic volumes) per network road segment, for different time periods of the day	Traffic volume	eMaaS Platform	↑
	Examination of multi-modal usage of the system	iSy26: Multimodality rate (number of trips made by only one means of transport divided by number of trips made by two or more means of transport)	Νο. (ratio)	eMaaS Platform	↓
Transport Operators	Examination of system’s operational characteristics	iSt1: Number of mobility services available	No.	eMaaS Platform	↑
		iSt2: Number of beneficiary operators	No.	eMaaS Platform	↑
		iSt3: Total system revenues	EUR	eMaaS Platform	↑
		iSt4: Revenues per operator	EUR	eMaaS Platform	↑
		iSt5: Revenues per transport service	EUR	eMaaS Platform	↑
		iSt6: Share or rentals where more than one transport service has been used (multimodality	%	eMaaS Platform	↑
	Investigation of the degree of satisfaction of companies participating in the system	iSt7: Share of operators with a positive experience on the overall performance of the system	%	Personal interviews	↑
		iSt8: Share of operators that are likely to continue to participate in the scheme after the end of the project	%	Personal interviews	↑
		iSt9: Share of operators with a positive experience regarding the allocation of the revenues	%	Personal interviews	↑
		iSt10: Share of operators with a positive experience on cooperating with the other participating operators	%	Personal interviews	↑
	Examination of operators’ intention to participate in the system and identification of main reasons for potential participation or not	iSt11: Number of companies that could in the future participate in the scheme	No.	Focus group	↑
		iSt12: Likelihood of participation in the scheme	%	Focus group	↑
		iSt13: Reasons for not participating in the scheme	-	Focus group	-
		iSt14: Incentives to participate in the scheme	-	Focus group	-

Where ↑ indicates that in the context of the operation of the eMaaS system the value of indicator is expected to increase compared to the current situation while ↓ indicates that the value of the corresponding indicator is expected to decrease.

in the Appendix A.

The first pillar concerns the evaluation of the eMaas system from the user perspective. It aims to examine users’ trip characteristics, such as the number of rentals, travel time, and distance, and the degree of user satisfaction regarding specific features of the eMaaS system, such as the availability and spatial distribution of vehicles and the rental cost of vehicles. The pillar consists of 12 KPIs, the calculation of which is based on data primarily originating from two sources: the eMaaS platform, where all the data concerning the operation of the system is stored and the questionnaire survey, which aims to collect users’ views on the operation of the system and is addressed to all the users of the eMaaS system.

Apart from the system’s actual users, the indicator system’s second pillar concerns the eMaaS system’s evaluation from the perspective of society, i.e., potential users. The objectives of this pillar are the investigation of respondents’ familiarity and attitude towards system components such as micromobility and electromobility and the identification of respondents’ intention to use the system in the future by capturing the potential reasons behind this decision. The pillar consists of 15 KPIs whose calculation is mainly based on the questionnaire survey addressed to the general population where the pilot project occurs, i.e., the city of Thessaloniki, Greece.

The third pillar concerns the evaluation of the internal operation of the eMaaS system. The 25 KPIs belonging to this pillar aim to measure the actual usage of the system (number of rentals and users), examine trip characteristics (speed, trip length and duration, vehicle kilometres travelled), identify trip patterns and traffic flows (most frequent origins and destinations), and examine the multi-modal usage of the system, i.e., the number of trips made with more than one means of transport. Data required for the calculation of the relevant indicators are also, in this case, derived from the operational data of the system through the eMaaS platform.

The fourth and last pillar of the proposed indicator system concerns the evaluation of the system by both transport operators/providers and public authorities currently involved in the system as well as potential participants in the scheme. Regarding the evaluation of the system by the first group of operators, this is undertaken through the examination of the system’s operational characteristics (data from the eMaaS platform) as well as the investigation of the degree of satisfaction of companies participating in the system through personal interviews. For the latter group of operators, their intention to participate in the system is examined by implementing focus groups for selected operators in the area covered by the scheme. The last pillar includes 14 ΚPIs.

To understand the potential impact of an eMaaS system, specific goals and trends were assigned to each of the 67 indicators. This is because indicators alone cannot identify problems, draw conclusions, or highlight solutions without being compared to a reference or goal. The goals were determined based on relevant findings from the literature and judgements made during the research study. These goals indicate a decrease or increase in the corresponding indicator, which can help determine whether the eMaaS system is functioning properly and being adopted by society. This information is critical for planners and decision-makers to make informed decisions.

4. Discussion and Conclusions

The way people get around is evolving as new business models are being created and put into practice. This leads to the emergence of innovative products and services in the mobility industry. Lately, MaaS has become more well-known in the world of passenger transport. MaaS is a creative way of getting around that combines various modes of transportation. It offers consumers a flexible, personalised, on-demand, and seamless way to travel from one place to another. This is all done through a single interface.

As electric mobility and MaaS continue to gain popularity, eMaaS has the chance to become a leading solution for current mobility issues. The concept of eMaaS goes beyond MaaS by providing users with seamless and multimodal transportation options and striving to be more eco-friendly than simply reducing car ownership, which is the primary goal of MaaS. The concept of eMaaS is a new and promising form of mobility that can help create a more sustainable transportation system. However, it is important to understand the motivations, expectations, perceptions, and concerns of the key players involved in achieving success.

Transportation planners, transport operators, policymakers, and citizens are curious about the future developments of the eMaaS market. Research has shown that MaaS has the potential to decrease our reliance on personal vehicles, lower transportation-related emissions, and improve the reliability of our transportation system. However, possible societal, operational, financial, and regulatory barriers could slow its success. For MaaS to be genuinely beneficial, all parties’ needs must be met. This includes the MaaS actors and the MaaS customers (i.e., travellers) [2,21,36,42].

Additionally, any barriers to implementing MaaS must be identified and addressed. It is hard to gather detailed information about the practical use of MaaS since it is a new concept. Qualitative research is one way to better understand the MaaS ecosystem and its expectations, perceptions, concerns, and attitudes.

In this paper, we examine the main factors that impact the effectiveness of a MaaS system. We conducted a literature review focusing on the monitoring and performance of MaaS systems, as there was a lack of information available for eMaaS systems. Our literature review analysis revealed that the evaluation of MaaS systems is mainly based on factors and characteristics organised around four pillars: (a) society, (b) users, (c) transport operators and providers, and (d) internal operation. For each of the above pillars, different types of qualitative and quantitative research results were presented for the current state of the art regarding MaaS systems that are currently being implemented or have been implemented in the past.

The analysis showed that the majority of the surveys were conducted to study people’s attitudes towards the implementation of a MaaS system. Specifically, the surveys focused on their willingness to use the system in the future and the conditions, especially pricing policies, under which they would be willing to use it.

Regarding the views and level of satisfaction of actual users of the system, most studies, considerably fewer in number due to limited pilot applications, have focused on collecting data on user satisfaction about specific system features such as availability, pricing, ease of use, and customer experience, as well as the potential shift of users to more sustainable modes of transport. A critical aspect of evaluating the system is the stakeholders’ opinion, either as participants in the scheme or as public authorities overseeing such a scheme. Data collected from questionnaire surveys, focus groups, and interviews were used to better analyse the preferences and attitudes of public and private stakeholders towards MaaS and assess the system’s operation from a business perspective. However, most studies on this topic did not thoroughly address the viability of a MaaS system and the business opportunity it may represent. Significant shortcomings were also identified concerning the evaluation of the system from the point of view of its purely operational performance. Most studies analysed aspects of trip characteristics and patterns while failing to address issues related to multimodality and the complementarity of available transport modes.

Then, based on the results of the literature analysis and, in particular, the shortcomings identified, an integrated impact assessment framework was formulated. This assessment framework is based on collecting data and information for factors and characteristics organised around the four pillars, which have been identified based on the literature review. However, the novelty of this framework is twofold. Initially, in addition to identifying the characteristics to be assessed, a range of innovative KPIs are proposed. The process of calculating these indicators, along with the identification of the available sources for data collection, is also defined by the framework. However, the most important innovation of the proposed methodology is that it has been developed to evaluate an eMaaS system. The aforementioned indicator-based assessment framework was formulated in the context of the eMaaS project currently running in the city of Thessaloniki, Greece, whose pilot application will be assessed using the proposed framework.

The overall objective of the proposed indicator-based assessment framework is to evaluate in-depth the eMaaS system under consideration after the end of the pilot operation; based on the evaluation, to assess the consequences of introducing eMaaS concepts on a broader scale from an individual (user), organisational, and societal perspective, respectively; to assess the potential socio-economic and other impacts of eMaaS; and to provide a basis and support for stakeholders’ decision making.

Consequently, there remain important challenges in applying the indicator-based assessment framework in practice. Most challenges are related to the volume of data generated from the system’s pilot operation. The primary sources for data collection, as identified by the methodology, are the data concerning the actual operation of the system and questionnaire surveys addressed to the existing and potential users of the system. Questionnaire surveys aim to gather extensive data from a large portion of the population. However, the amount of data collected depends on the number of system users. Calculating relevant indicators and drawing safe conclusions about the system’s operation may prove challenging if it fails to attract enough users. Nevertheless, the significance of the proposed methodology remains, which is based on extensive research and a literature review on the evaluation and operation of MaaS systems.

As the evaluation of the eMaaS systems is still at an early stage, the proposed methodology is expected to significantly contribute to this direction and foster the development and expansion of these systems by identifying enablers and barriers hindering the evolution of eMaaS. Overall, the particular assessment framework of indicators can be the starting point for a more robust and comprehensive assessment of the characteristics and functional elements of an eMaaS system.