Assessing Knowledge Performance for the Fast-Track Delivery of Sustainable Mobility Solutions

Abstract

European cities are motivated to act towards the achievement of climate-neutral mobility solutions. Often, though, they are facing many challenges when bringing (innovative) sustainable mobility solutions forward. Capacity building that fills the skills gaps and/or enables the acquisition of new ones related to the planning and implementation of such solutions can empower local/regional authorities to identify them, adopt them and eventually deliver them properly. The aim of this paper is to present the Key Performance Indicator (KPI) framework that has been used for the assessment of the effectiveness of the Learning and Exchange Programme applied in an EU-funded project. It presents the methodological steps for the adoption of the KPIs, as well as the tools used for the selection of the KPI data and the KPI monitoring at the project level. It also presents the results from the application of the framework for assessing the knowledge performance towards the deployment of sustainable mobility solutions. It finally reflects on recommendations for applying the KPI framework to other cases and thematic contents.

1. Introduction

European cities are motivated to act towards the achievement of climate-neutral mobility solutions [1]. A policy that could achieve transformative changes requires interdisciplinary and transdisciplinary approaches [2], but cities are, often, facing many challenges when it comes to bringing sustainable mobility ideas forward and “fast tracking” these ideas towards deployment [3]. Gaps can be identified in governance schemes, lack of funding, limited (human) resources, but also in limited knowledge/experience for data-driven approaches that can support all stages of mobility solutions implementation [4].

Capacity building that fills the gaps and enables the acquisition of skills related to the implementation of innovative mobility solutions can empower local/regional authorities to identify, adopt and eventually deliver mobility solutions that are considered innovative for the local/regional context and need [5].

The definition of capacity building is broad, referring not only to the process of developing and strengthening abilities and skills but also to the process of building relationships and values [6]. The Organization for Economic Co-operation and Development (OECD) places great emphasis on the “quality of organizations” and the “enabling environment” that allows the term “capacity development” to go beyond the increase in knowledge and skill of the individuals, to incentive-building and governance transformations of the organizations [7]. Under the same strategic vein, the United Nations (UN) has identified the importance of capacity building and has appointed the United Nations Environmental Programme (UNEP) to act as a missionary for capacity building as a means of achieving sustainable development [6].

Looking closely at the transport field, Glaser, Brommelstroet, & Bertolini [2] identify learning for transportation policy as an integral part of larger processes, such as innovation. Martins, Kalakou & Pimenta [3] see the capacity building of local authorities as a “hearts and minds approach” for individual, organizational and institutional capacity that will address principal barriers to achieving more sustainable transport strategies (p. 11) and they defined capacity building as “the strengthening of the dynamics in an organization that drive its effectiveness in implementing plans” (p. 13). But how is capacity building delivered within an analytical framework?

Thapa, Matin and Bajracharya [8] have defined a sequential capacity-building flow for the development of the necessary skills and abilities around the use of Earth Observation and geospatial information technologies, which consists of four elements: needs assessment, design, implementation, and monitoring (Figure 1). Starting from capacity assessment that allows for clear gap definition and learning objective formulation, the capacity building curriculum is designed to address the learning objectives. Following the execution of the capacity-building activities, monitoring and evaluation take place to assess the performance of the capacity-building activities against criteria of quality and relevance and foreseeing mitigation actions in case the criteria are not met.

Similarly, the United Nations Development Programme (UNDP) [9] proposes five steps for the capacity development cycle adding, at the beginning, actions that will bring the commitment of people to the process [9].

Anderston, et al. [10] propose a capacity-building approach beginning from a bottom level of knowledge gain (in the form of external knowledge that fills the gaps) to organizational transformation, where organizations are empowered to deliver/adopt changes. In their underlying framework of learning, the learning taxonomy should not stop on classifying the learning objectives, rather than going to an upper level of evaluation capacity, namely “the ability to conduct an effective evaluation” [11] (p. 1).

Evaluation of capacity building has been highlighted as a common denominator and Martins, Kalakou and Pimenta agree that evaluating capacity building is crucial in order to map the program’s progress against its goals, grasp the reactions of the learner and obtain their feedback, identify gaps and weaknesses in the learning programme itself and propose actions to address these weaknesses [3]. The European Commission (EC) also clearly states: “Assessing institutions and capacity is thus a central element of preparing and implementing any kind of support” [12] (p. 7). Horton answers the question of “Why should capacity building be evaluated?” by also discussing the purpose of “accountability” (the obligation to produce evaluation reports), but eventually the importance of the “improvement” of ongoing or future capacity-building activities is stressed [13] (p. 9).

An intense research dialogue around capacity building assessment initiated since 2000 and an evaluator can refer to a commonplace of sources that cover a variety of fields/sectors [12,14,15]. An underlying philosophy for these frameworks has to do with the consideration of the organizations as systems that are embedded in a context (environment), receive resources and use the capacity of these resources to deliver products/services. Therefore, any changes related to any of these items (content, resources, capacities) should be considered.

Complementary to the above, a number of guides are also available for setting up and applying monitoring and evaluation frameworks for capacity building [16,17,18,19,20], with substantial work carried out for the health sector and the measurement of “community capacity” in the public health arena (for an extended review over the latter, the reader can refer to Liberato et al.’s work [21]).

Within the pool of capacity-building evaluation frameworks/guides, though, not a substantial amount of work is found for the transport sector. As the most outstanding cases, the SUITS project (Sustainable Urban Integrated Transport Systems: Transferable tools for S-M local authorities) capacity building evaluation framework [3] and Transport Innovation Living Lab approach for capacity needs assessment are mentioned [5]. SUITS evaluation framework [3] assessed the perceived drivers and barriers of capacity building development that aims to transform the transport organizations of the cities engaged in the project. The project identifies four capacity elements: inputs (referring to the expenditure of people, materials and funds), processes (activities transforming the inputs to outputs), outputs (results produced by the processes) and outcomes (knowledge, skills and behaviour). The evaluation framework defined the tools/methods for data collection and the categories and sub-categories of capacity to be assessed (categories: organizational, political, legal and societal; sub-categories: communicational, financial, managerial and technical), each one linked with a list of Key Performance Indicators (KPIs). The framework foresaw a rating conversion methodology, in order to estimate the performance level of each city, based on estimated KPI values.

Assessment of the capacities of cities to plan and implement mobility innovations is performed by Teko and Lah within nine Living Labs (LLs). Key characteristics and elements of the LLs are used for the definition of the assessment framework and further developed into indicators: “extent of real-life contextualization” (as LLs “depict a real-life scenario” capacity needs are defined in a real-life context), “level of participation” (as LLs allow “active user participation”, also in the assessment of capacity needs), “diversity of stakeholders involved” (as LLs “involve multiple stakeholders”, capacity needs assessment is conducted from a variety of relevant stakeholders) and “time span of engagement” (short, medium or long term “length of time involved in defining capacity needs”). As a result, capacity needs can be refined reflecting collective decisions, the interest of stakeholders in the assessment process is sustained and capacity-building interventions are well tailored to the identified needs [5] (Table 1 on p. 4).

Gordon and Chadwick [15] and Simister and Smith [16] contribute to our discussion on how to identify a direct link between capacity built and community benefits achieved due to increased capacities. Literature suggests that, when trying to evaluate any learning process, we should move along a pathway, beginning from “inputs” (what are the efforts placed?), going to “outputs” (what are the results of these efforts?), then “outcomes” (how effective are the inputs in delivering benefits?) and, finally, “impacts” (changes answering to higher-level objectives) [3,9,12,15,16,18].

The current study follows the performance of a Learning and Exchange Programme, that was designed and implemented for European cities as part of an EU-funded project, to help them accelerate the deployment of innovative sustainable mobility solutions. The monitoring and evaluation of the knowledge exchange and transfer was carried out through a Key Performance Indicator (KPI) based assessment framework, applying a loop of “input-to-impact” pathway for the KPI feed, analysis and evaluation throughout the 2-year duration of the Learning and Exchange Programme. The KPI assessment allowed for the strengths and weaknesses of the Programme to be identified and relevant mitigation measures to be applied when necessary.

2. Materials and Methods

The paper focuses on a capacity-building monitoring and assessment process, undertaken by the authors as part of an EU-funded project, which aimed to support local authorities in Europe to quickly implement innovative—for them—mobility solutions. The project followed the capacity-building flow of Thapa, Matin and Bajracharya presented above (Figure 1) [8], adapted as per the iterations needed to accommodate changes in the design of the capacity-building activities:

Step 1: A carefully performed needs assessment is of utmost importance for the priority definition and activity design of a capacity-building programme that reflects the specific individual/organizational learning conditions and priorities [22]. Within this concept, the capacity building programme examined in this paper first initiated with a “diagnostic” phase to address the actual challenges faced by cities. The participating cities have defined their understanding of “smart” and “clean” innovations and have identified the barriers they need to overcome for rapid implementation. “Innovation profiles” were finally extracted in a “fingerprint” visualization of the cities’ performance across eight success factors, the so-called “8Ms”: mood, motivation, mass, momentum, mechanisms, measures, methods and money [23].

Step 2: The project designed and applied a tailored Learning and Exchange Programme (from now on called the “Programme”), which aimed at fulfilling the learning needs identified and helping cities overcome the barriers obstructing the deployment of innovative mobility solutions. The Programme included audiences and connections throughout Europe, resources such as databases of solutions and best practices and peer-learning activities that revealed new opportunities for sustainable mobility innovation [24]. The learning exchange was established through four clusters:

• Cluster 1—Sustainable and Clean Urban Logistics;
• Cluster 2—Cycling in the Urban and Functional Urban Area;
• Cluster 3—Integrated Multi-modal Mobility Solutions;
• Cluster 4—Traffic and Demand Management.

At the same time, four cross-cutting themes have been identified for dealing with key learning components running across all clusters:

• Behaviour Change;
• Digitalisation and Data Management;
• Funding, Financing and Business Models;
• Governance, Participation, Cooperation and Co-creation.

Step 3: The Programme was wrapped around five Learning Sequences (LSs), each with a defined object to achieve: from cities setting their goals within the project to cities concretizing their actions towards the deployment of innovative mobility solutions and eventually accelerating their innovation. Five intensive learning weeks (the Capacity Building Weeks—CBWs) marked the end of each LS, and interactions between these weeks were facilitated through, i.e., webinars, online workshops and access to asynchronous learning material.

Step 4: A robust monitoring and evaluation framework was developed, to capture the innovation performance of the project, in a two-fold manner [25]:

• Branching exploration, aiming to understand the innovation profiles of the cities;
• Incremental iterative refinement, aiming to monitor and refine the mobility solutions. This was carried out through the development and monitoring, by the authors, of a KPI framework that monitors and evaluates the progress, efficiency and benefits of the Programme. The current paper focuses on this part of the project’s evaluation framework.

Based on the “capacity building-to-impact pathway” presented by Hailey, James & Wrigley [20] and the Ripple Model presented by James [19], the authors present in Figure 2 and adapted a generic impact pathway for the Programme [26]. In this, “capacity building interventions ripples” flow outwards, like a drop of rain, from capacity building efforts to behavioural change of the learners, and, in turn, to higher-level changes.

A predictive approach is used with Key Performance Indicators (KPIs) set for every level (input, output, outcome, impact) to monitor the progress towards pre-defined objectives of the project, always within the spectrum of delivering sustainable mobility innovations [26]:

• At the “input” level, KPIs measure the efforts placed in terms of delivery of peer-learning activities; engaging learners (representatives of European local authorities) and peers (experts, private commercial bodies, project partners) to the Programme and identifying case-specific learning expectations and obstacles/barriers that hinder the delivery of sustainable mobility innovations in each city context.
• At the “output” level, relevant KPIs monitor the direct results that the peer-learning activities have in the individuals’ knowledge, skills and competencies (the level to which identified challenges were solved and learning needs were addressed) and in the type and magnitude of new synergies established between the cities and EU networks. Proxy indicators (such as satisfaction of the learner) are also used at this level.
• At the “outcome” level, the defined KPs follow the achievements and changes at the organizational level that come as a result of the enhanced individual competencies and the new opportunities for synergies. This includes, for example, the level to which organization efficiency in governance and financing of sustainable mobility solutions is enhanced and the number of Deployment Plans developed for orchestrating the implementation of those mobility solutions.
• Finally, “impact” indicators consider high-level changes in the organization structure, along with community-level benefits attributed to the actual implementation of the sustainable mobility solutions (i.e., behavioural changes towards more sustainable modes of transport and reduction of greenhouse emissions).

The KPI framework used in the project defined a list of a total of 51 KPIs. A loop of data collection for the KPI calculation was established, which allowed for regular monitoring and reporting of the Programme’s performance. The latter allowed, in turn, the activation of responsive and/or formative mechanisms, whenever an inadequate performance of the Programme in addressing the learning needs was identified. The KPI monitoring process systematically considered the feedback from twenty-three European cities engaged in the project. Target values were also attributed before the launch of the Programme to 17 indicators and the Programme’s progress towards these values has been referenced. These target values created quantifiable traces along all levels of the Programme’s capacity-to-impact pathway: from the extent of engagement—input (in terms of the number of people and private/commercial bodies actively participating in the project)—to the magnitude of discussions around innovative mobility solutions and data—output (in terms of number of mobility solutions and new data sources exchanged within the project)—which, in turn, would facilitate the delivery of a number of Deployment Plans in different spatial contents—outcomes (i.e., 24 Deployment Plans delivered in the project, of which at least 50% addressing peri-urban or rural areas). Eventually, at an impact level, the improvement of the cities’ competence in delivering sustainable mobility solutions is quantified as the number of cities moving positively over a “global EU spectrum”, described in detail in [25,26]. As the proposed KPI framework was applied in an EU-funded project, the selection of target values was carried out at the project’s proposal phase and bound to the contractual agreement between the consortium and the European Commission. Baseline values were not set before the launch of the Programme, as this required city-specific data, which was not possible to obtain (as the cities were eventually engaged in the Programme after the launch of the project). This meant that target values had to be delivered in a case-agnostic way, considering no previous achievements (17 KPI values set to zero at the launch of the project). For KPIs for which no target value was defined but a baseline value was necessary, relevant estimations were conducted at the outset of the project.

Data mapping and storing processes were defined at an early stage of the project. Data collection methods were decided and the relevant tools for gathering data were created. Data collection was performed in a consistent format, either through individual data points (online forms/questionnaires) or directly within logbooks (spreadsheets or Word documents) created for data collection and storage. Each KPI was matched to the data streams and mechanisms were set for organization (organization in folders and classification). Validity of the logbook data and compatibility of information among cities was ensured by the design of the logbook itself (i.e., directly connecting the logbook fields with the entries in the online survey response sheets) and through regular logical tests performed by the logbook managers.

Interviews, questionnaire surveys and structured observations were used as methods for data collection. The main target group consisted of representatives of the cities engaged in the project. These were mostly recruited from the cities’ mobility and logistics departments (78% of the cities’ representatives), but other domains were also brought in (i.e., public space, energy, environment, urbanism and construction, management and administration, IT). The KPI framework requested for each survey a unique record per city, in many cases binding collectively the feedback received from different city representatives. Therefore, having the same people participating in all the survey/interview rounds and having a person assigned as “the city’s respondent” (collecting information from and consulting its colleagues) was important. Such city–internal coordination was not always possible, though, and relevant subjectivity issues are reported in the discussion and conclusions of the current paper.

The following tools were used for the monitoring of the KPIs:

• Needs Assessment Survey (NAS): As already mentioned in Step 1 of the project’s capacity-building flow, this was conducted at the beginning of the project to collect information on the cities’ innovation profile. Data collected regarding cities’ needs, obstacles and opportunities regarding innovative sustainable solutions were used for calculating the baseline values for some indicators.
• Event Forms (EF): The Event Form was introduced as an online questionnaire targeting the event organizers (partners) (see Supplementary material, Table S1). Both quantitative (i.e., number of participants) and qualitative data (i.e., level of participation, as perceived by the organizer) were collected through the forms. The event organizers filled in the forms usually within a period of 2 weeks after the event implementation.
• Innovation Diaries (ID): The ID was introduced as an online questionnaire targeting the cities that were engaged in the project (see Supplementary material, Table S2). The ID initially collected information related to challenge definition (barriers that hinder the rapid deployment of innovative mobility solutions), idea formation (getting inspired from city peers) and learning action framing (what exactly cities need to overcome the identified challenges). As the Programme moved forward from problem definition to planning formulation (Deployment Plans), the ID content was adjusted accordingly. Nonetheless, questions related to the city’s progress/satisfaction with the Programme’s activities remained as a key content in all IDs. The cities’ representatives filled in the Innovation Diary usually within a month after the end of each CBW.
• Registration and participation forms, collecting data on participants and their working profiles.
• Complementary to the above, the KPI analysis has been facilitated by the content analysis of the following project documents and outputs:
• Dissemination Tracker, introduced by the project’s communication and dissemination activities, aims at monitoring the project’s dissemination efforts, including attendance of the partners and the cities’ representatives external to the project events.
• Deployment Plans, developed and delivered by the cities as a final product of the project, for accelerating the deployment of selected innovative mobility solutions.
• Transferability assessment templates, introduced as a short questionnaire during the project’s study visits, for capturing the transferability potential of each study visit case to the cities/regions engaged in the project.
• Other project activities, such as the Exploitation Strategy implementation.

Longitudinal data were collected with the “panel survey” method following the same sample of individuals during the Programme’s lifetime. Four (4) rounds of data collection were carried out, following the first four Learning Sequences of the Programme. Given the evaluation environment (project), convenience sampling was applied: partners/event organizers replying to the Event Form questionnaires and representatives from the cities engaged in the project replying to the Innovation Diaries (ID). The sampling size for the Innovation Diaries was fixed and matched the number of the cities engaged in the project (23 in total), although not all cities provided an ID record in each round (the average reply rate was around 70–75%).

For the communication of the KPI monitoring to the project consortium and the activation of the iteration process of the evaluation framework, four internal periodical Activity Reports were produced, one after the conclusion of each of the Learning Sequences 1, 2, 3 and 4. The Activity Reports monitored those KPIs that were relevant for the learning period under analysis, as not all KPIs were monitored each time. A review of the performance of the KPIs was conducted through infographics and a link to the project’s quantitative impact targets was carried out when relevant. At the same time, the insight from the Event Organizers was provided to allow for a better understanding of the event process and formats, as well as the “inclusiveness” (engagement of external) and interactiveness of the events delivered. At the end of each Activity Report, the main findings from the KPI analysis were provided on the basis of promising results and points that need further attention. This allowed the whole partnership to keep track of the progress of the Programme towards the predefined targets and even plan/proceed to changes in the content/format of the engagement activities when the KPI results indicated such a need.

3. Results

3.1. Data Collected and Relevant Challenges Identified

Table 1. Data collected for KPI monitoring.

Data Type	Number of Collected Data	Learning Sequences
Needs Assessment Survey questionnaires	29	1
Event Forms (questionnaires)	94	1, 2, 3, 4
Innovation Diaries (questionnaires)	70	1, 2, 3, 4
Dissemination Tracker records	64	1, 2, 3, 4, 5
Deployment Plans (reports)	23	4, 5

summarizes the data that were collected for the KPI monitoring of the Programme. These are counted as collected forms/questionnaires, or records or documents/reports. The correlation with the Learning Sequence in which data were collected is also made.

Issues and points of attention regarding data collection are summarized below. These are mostly related to the Needs Assessment Survey (NAS) and the Innovation Diaries (IDs), as their completion required input from the cities’ representatives. It should be highlighted here that the majority of the cities (19 out of the 23) were engaged in the project on a voluntary basis. Four (4) cities acted as “Ambassador cities” in the project and were involved as partners.

• The NAS was launched at the beginning of the project with an initial composition of cities. At the end of the first year of the project, though, some cities could not remain engaged due to changes in their policy priorities. Other cities were recruited, for which the NAS had to be initiated again.
• The Innovation Diaries (IDs) requested special guidance for their completion. A dedicated workshop took place before the completion of the ID1 (during the first Capacity Building Week—CBW#1) to present the concept of the Innovation Diaries and explain the type of information requested by the cities. During the workshop, cities’ representatives were advised to complete one (1) ID for their city each time. In cases where several city representatives were engaged in the project, this required the consolidation of the replies into one document. This was, however, not always coordinated internally (perhaps because this requirement was not communicated between the different city representatives who filled in the ID), and some cities provided two IDs at the same time. In such cases, consolidation of the data was carried out during data analysis.
• Four (4) rounds of the Innovation Diary survey have taken place, with input requested by the cities engaged in the project. Special attention was paid that we did not create “survey fatigue”. For this reason, ID3 was provided in a simpler and shorter format than IDs 1, 2 and 4, despite not completely serving the project’s monitoring purposes. Another reason for this was the need to have the last ID (ID4) in its most extended version so that the cities could reflect upon all aspects of their learning.
• A typical difficulty that was also encountered had to do with the engagement of the Local Affiliates on ID data collection, as (a) not all LAs replied to the Innovation Diaries and (b) it was not always the same cities that replied to each ID.
• One minor issue encountered regarding the monitoring of participation in online events had to do with the inability to identify registered participants joining the event when these used only their first name or the name of their organization. A cross-check with the event organizer/coordinator was then necessary.

To ensure data validity, the evaluation process and findings were regularly checked by and discussed with colleagues from the project’s consortium. Peer debriefing was also applied to enhance the reliability and validity of data collection and analysis.

3.2. KPI Analysis and Key Findings

In this section, a discussion of the key results of groups of KPIs is performed, showcasing the “input-to-impact” pathway of the proposed assessment framework. Indicatively, some KPI (info)graphics are also displayed, although the presentation of all 29 (info)graphics that were produced for the KPI framework application falls outside the scope of this paper.

3.2.1. Delivery of the Learning and Exchange Programme

The delivery of the Programme progressed as initially planned, with a total of 106 capacity-building sessions implemented during five Learning Sequences (Figure 3). The Programme differentiated between core activities (plenaries, study visits, horizontal topic sessions for all and cluster-based peer-learning sessions and hands-on workshops) and intermediate activities, organized as remote learning sessions around core interests identified by each cluster, horizontal learning needs, coordination activities and specific requests by the cities.

Included in the core activities, five matching and exchange events between cities and mobility innovation suppliers took place, as well as one learning event that coached cities in participatory forms of planning and implementation. The matching and exchange events had a great added value to relationship building between cities and providers, which eventually led to 15 concrete contracts that supported the cities towards the development and implementation of their Deployment Plans.

The Programme was flexibly built on a combination of methodologies (webinars, study visits, co-learning workshops, co-creation workshops, peer-review workshops and speed networking) that allowed for each LS objective to be reached. Specific care was given for events/sessions that were delivered online (i.e., CBWs #1 and #2 were held online due to the ongoing pandemic) in order to properly adapt to the online format (i.e., shorten the duration, foresee enough breaks, use of online co-learning/design tools such as online whiteboards, etc.).

The learning and exchange events enabled an intense discussion around the learning needs of the cities, especially at the beginning of the Programme. In total, 200 learning needs were either expressed by the cities or recorded by the event organizers. Indicatively, during LS1, where 51 learning needs have been discussed, 30% of them have been covered through the Programme.

City representatives have also expressed their satisfaction in a more qualitative term, as the majority of them (63%) indicated that the Programme covered to a high or very high level their needs/questions around the deployment of their innovative mobility solution. The rest (37%) positioned themselves with an average score regarding this statement.

3.2.2. Participation and Active Engagement

Twenty-three (23) cities were eventually engaged in the Learning and Exchange Programme and allocated to the cluster of their interest. Special care was placed on achieving a rather balanced composition of cities within the clusters, meaning having both cities with economies that advance sustainable mobility solutions and cities lagging behind on that matter. A re-shuffling and re-definition of the clusters was necessary to be performed due to the entrance of new cities in the project until the end of the first year, but the Programme allowed for flexibility and, eventually, the balanced composition of the clusters was not compromised.

Regarding the participation of cities in the learning and exchange events, the online events during CBWs #1 and #2 allowed for more people to connect, but interactions between the cities were, by default, more limited and the full benefits of the face-to-face exchange were not reached.

In total, more than 600 people were somehow involved in the Programme, either as city representatives or invited speakers, mobility solution suppliers and local actors (Figure 4). Cities representatives, were, of course, present in each Learning Sequence, as the main beneficiaries of the Programme. The involvement of the rest groups of engaged people followed the objectives of each Learning Sequence: suppliers were mostly represented in LS1, where relevant matchmaking events were organized in the form of “speed dating”. During LS4, the cities prepared their Deployment Plans, and, therefore, members of the cities’ stakeholders’ groups that provided input to the plans were involved in this process. Noteworthy, people within the cities’ administrations who were reached by their colleagues and received knowledge from the Programme were counted once towards the end of the Programme (through ID4), but they were engaged throughout the other Learning Sequences as well.

The Programme strongly encouraged an integrated approach to solving challenges and addressing needs around the planning and implementation of innovative mobility solutions. A transdisciplinary and interdisciplinary approach, involving actors from sectors other than the mobility and logistics sector, was possible, as experts from various fields of expertise were invited to the project and cities received feedback from 42 local actors outside the mobility sector for their Deployment Plans (DPs). Overall, there were 162 people outside the mobility sector who either received knowledge from the Programme or provided their feedback to the DPs.

The Programme provided a stage for people representing various local contexts and working backgrounds, to come together and exchange their knowledge and expertise over common challenges and needs. This was highly appreciated by all participating cities.

3.2.3. Synergies and Networking

The Programme was further complemented by the “Activity Fund”, which offered cities support for preliminary studies, the organization of further in-depth exchange activities, and access to tailored expert advice from a Pool of Suppliers. The latter was set up to draw external expertise to the Programme’s community. In total, 48 private/commercial bodies were connected to the project through the pool of suppliers and 15 contracts were signed between 9 cities and these suppliers, for direct support to their Deployment Plans.

One of the key characteristics of the Programme was its extroversion: gaining knowledge and experience from other EU projects/initiatives and networks and sharing its insights with a wider EU community. In total, the Programme established links with 37 EU projects and networks and has established 4 interactions with Smart Cities Marketplace (SCM), i.e., as partners participating in SCM events/sessions or invitation of speakers representing SCM initiatives/projects to the CBWs.

Furthermore, the Programme also aimed at inspiring the LAs to act as ambassadors of their innovation to their wider (local) network and already six (6) cities have undertaken this role.

As a result of the added value of connecting with EU networks/initiatives that was communicated through the Programme’s activities, one city has connected with SMC, one city has become a CIVITAS member and two cities have connected with EIT (European Institute of Innovation and Technology) Urban Mobility.

Going one step further than the above, one city also brought its mobility solution explored within the project as a pilot case in EIT calls, while 15 new proposals/projects were brought forward from 9 cities for receiving funding at the national or EU level. Seven (7) of these new proposals have already received funding.

3.2.4. Progress towards the Acceleration of Mobility Solutions Deployment Plans

An important discussion for innovative solutions has been initiated through the Programme, bringing forward more than 122 mobility solutions as an inspiration. This led to an increase in knowledge of network members on innovative mobility solutions and increased capacities in the selection of mobility solutions that address the city’s needs, which has been recorded through the final Innovation Diary on a 1–5 Likert scale (Figure 5).

Eventually, 23 solutions were identified from the cities/regions as those explored within the project and 23 Deployment Plans were registered and approved.

The Programme offered a great opportunity for identifying and addressing specific challenges related to the implementation of sustainable and innovative mobility solutions. Several challenges were discussed and addressed during the course of the Programme and, as the latter progressed towards the development of the Deployment Plans, eventually, 58 barriers were linked to the mobility solutions explored within the project. Most of these barriers (48%) had a local identity, followed by 17% that had both local and national identity and 15% that were linked to all levels (local, national and European). Despite the diversity of the cities’ characteristics and the mobility innovations chosen, common challenges were identified in various domains, such as [27] the following: data management (reluctance for data sharing, challenges in data integration and standardization, as well as in data privacy and security); user’s acceptance and changing of their attitudes; stakeholders engagement; skills (lack of specific skills and/or specialized personnel); decision-making process (bureaucracy); legal framework (GDPR and cybersecurity, frequent changes in legislation and technologies advancing more than the legislation); business models (coordination between public authority right and private business); technology (how to integrated technological advancements); policy (lack of political commitment); funding (how to secure adequate funding); and cooperation (lack of inter-departmental cooperation).

The Programme enabled 74% of the barriers that were eventually included in the Deployment Plans, to be solved or partially solved. This was carried out in a direct manner (i.e., directly increasing the skills of the city's personnel or providing “soft” funding for maturing the selected innovative mobility solution towards its funding from other sources), but mostly in an indirect way. The latter was carried out by offering inspiration and guidance for the cities to launch the proper processes and undertake the necessary actions/initiatives on their own resources through a new way of thinking and an enhanced way of inter-departmental cooperation and liaison with external innovative mobility solutions providers.

Eventually, a significant increase in network members’ capacity for overcoming identified barriers and implementing innovative mobility solutions is recorded (Figure 5), as 10 cities have indicated through the last Innovation Diary a higher capacity for implementing innovative mobility solutions than the one reported at the beginning of the project (moving from a “before” average score from all cities of 2.6 to an “after” score of 3.5). Ten (10) cities have also indicated a very high or high capacity for finalizing the implementation of their innovative mobility solution after the end of the project.

During the course of the project, two cities had already launched the implementation of their mobility solutions and another five were under preparation for implementation (i.e., preparing procurement documents).

The process of Deployment Plan (DP) development was highly appreciated by the cities, as it allowed for a structured definition of the challenges related to and actions that need to be undertaken for the implementation of innovative mobility solutions. Accompanied by dedicated workshops on acceleration factors, the DPs also allowed the cities to reflect upon the condition under which an innovation can be considered as “shovel ready” for implementation.

3.2.5. Cross-Cutting Skills Supporting the Deployment Plans

The Skill Streams events of the Programme allowed for common data, governance, funding and engagement issues to be discussed and related good practices to be shared.

The discussion over data seemed of high importance for many cities and LS2 had a specific focus on mobility data integration and management. In total, 24 new data sources were discussed during LS1 and LS2. Four (4) cities have already shared open data with their fellow cities during the project and 8 cities were willing to do so after the project’s closure. As a result, 11 network members indicated an increase in knowledge of data gathering, management and analysis (Figure 5) (moving from a “before” average score of all cities of 2.4 to an “after” score of 3.5). Eventually, 9 new data sources and/or new methodologies for data integration were included in the Deployment Plans and 10 cities have already launched their data collection.

Funding (lack of funding) was a key common challenge for the implementation of innovative mobility solutions for many cities. LS3 allowed for a targeted learning approach in relation to funding mechanisms and business models, enabling a significant increase in the knowledge of network members on developing investment and/or business/operating plans (Figure 5) (moving from a “before” average score of all cities of 2.8 to an “after” score of 3.6).

Regarding governance on territorial planning, 7 project network members reported an increase in their capacity (Figure 5) (moving from a “before” average score of all cities of 3.0 to an “after” score of 3.6), while having already observed changes in their city governance model.

Social innovation and the ecosystemic approach to engage all actors was placed at the focus of the Programme during CBW4. Although many cities have indicated that they had already high or very high capacities in citizens’ and stakeholders’ engagement (Figure 5) (“before” average score of all cities = 3.5), an improvement is recorded for 6 cities (“after” average score of all cities = 4.0). As a result of the Programme’s support to better understand who the stakeholders are, how to engage them and what their influence is in the planned mobility solutions, 10 cities are already trying to improve their local engagement activities.

3.2.6. Achievements of Higher-Level Objectives

Higher-level achievements are connected with changes in the organization structure and community benefits offered through the deployment of the project’s innovative mobility solutions per se.

Cities were asked to position themselves in a “global” spectrum of ‘starters’ (cities facing a rapid transition curve and ready to interact and learn from other cities), ‘sharers’ (“capacity conscious” cities, who can share knowledge but also have learning needs) or ‘leaders’ (a relative leader in a specific topic, but still with room to benefit from further advise). A positive movement of in total 9 cities is observed in the spectrum: 8 cities moving from ‘starter’ to ‘sharer’ status and 1 city moving from ‘sharer’ to ‘leader’ status.

Cities were also asked to provide quantitative data for the estimation of KPIs related to modal shifts and reduction of greenhouse gas emissions, attributed to the implementation of the mobility solutions per se. Unfortunately, the cities could not provide such data rather than indicate in a more generic approach the direct connection of their solutions to the objectives and target values of their Sustainable Urban Mobility Plans. It is worth mentioning, though, that all cities had a greenhouse gas emission reduction goal, while the majority of the cities had the goal of a modal shift toward more energy-efficient modes (i.e., electric vehicles, bicycles, walking, public transport when the shift is carried out from private cars).

4. Discussion

The current paper presents a KPI Framework for assessing the performance of a learning and exchange programme dedicated to increasing local authorities’ capacities around the planning and implementation of innovative mobility solutions. Key findings from the application of the Framework in the case of a Learning and Exchange Programme developed as part of an EU-funded project are also provided. This analysis allowed for the Programme to be properly monitored and assessed within the requirements of the project. Considering, on one hand, the interconnection of capacity-building efforts and sustainable mobility (for, among other things, developing skills for sustainable mobility, integrating sustainable practices, engaging the community, achieving multi-stakeholder cooperation, bringing along technological innovation, etc.) and, on the other hand, the connection between evaluation processes and the delivery of responsible and responsive capacity building efforts (as already discussed in the Introduction), this analysis walks along with initiatives that aim at fostering more sustainable mobility systems.

Nonetheless, the results from the Framework’s application should be cautiously generalized due to the content-specific character of the project; therefore, the discussion that follows hereafter focuses primarily on the Framework itself and the data requirements supporting it.

The KPI framework was based on a “capacity building-to-impact” pathway, clustering 51 KPIs as input, output, outcome or impact indicators. A loop for the KPI monitoring was applied and iterations of data collection enabled a regular understanding of the performance of the Programme, but, also, allowed for responsive and/or formative mechanisms to take place for tackling raising issues.

The KPI framework monitored learning gaps in various skills (technical, administrative, financial, social) and the way these were addressed by the learning and exchange provisions of the Programme. Thus, it heavily depended on the contribution of the cities that were engaged in the Programme, although room for structured observations was also allowed. This dependency further favoured the engagement, but it also brought forward challenges in the data collection. These were primarily related to the challenges of keeping the interest of all the cities to provide data vivid throughout a rather extended data collection period, as, in total, five rounds of data collection were requested, from mid-2021 to mid-2023.

The above is also related to a discussion over the “objectivity” of observations related to the increase in organization capacities. While individual capacity can be easily evaluated, organizational capacity monitoring implies a twofold procedure: the first being the actual transfer and adoption of knowledge within the organizational structures and the second being the evaluation in terms of “collective” progress. Actual observations of the knowledge transfer and the organizational transformation would provide an objective perspective but, other than requiring significant resources and a more extended list of KPIs that better map the organizational functions [28], they are difficult to implement as part of (rather short) projects (since it takes time for capacity utilization to become apparent). In such cases, data rely on individuals’ responses to knowledge transfer mechanisms (Has the individual exchanged knowledge with their fellow workers in a consistent way?) and perceptions over the progress of organization transformation (Do the individual’s replies in the evaluation survey consider changes at an individual or a collective level?), which always contain a level of subjectivity.

Another weakness of the KPI framework that should be discussed deals with the gap between the expectation to have adequate metrics for the impact indicators versus the maturity of the cities to provide relevant data for their calculation. Reporting on impacts such as carbon emission reductions or modal shifts to more sustainable or energy-efficient modes of transport is rather challenging even when the mobility solutions are already in place, let alone when their implementation needs time to mature (which was the case in our KPI framework application). Complex (modelling) processes that are usually required either for an ex-ante or an ex-post estimation of emissions reduction, modal shifts, etc., contradict engagement on a voluntary basis that respects the limited resources brought forward by the “volunteer”. Time limitations set in projects of rather short duration (i.e., 2 years or even less) also add to this issue. Simister and Smith seem to have a similar opinion: “The duration between capacity building interventions and desired end results can be very long. For example, one Southern capacity building provider interviewed as part of the research are only now seeing the fruits of work carried out fifteen years ago” [16] (p. 7). They might have had an argument, though, on the basis of “maybe you have decided to measure too far”, as the extent to which the monitoring and evaluation of capacity building should go is a critical decision assigned to the evaluator right from the very beginning. To this end, the authors seem to mostly support “illustrations” of changes, rather than measurements of it [16] (pp. 8–9).

James also comments on a dimension that is seen by the authors as an additional challenge for the assessment of our impacts, stating that, “as a project moves from input to effects, to impact, the influence of non-project factors becomes increasingly felt thus making it more difficult for the indicators to measure change brought about by the project” [19] (p. 10), and thus stressing that capacity building expected impacts are heavily influenced by changes in the external environment as well.

5. Conclusions

The authors opine that the proposed KPI framework can provide a useful tool for the evaluation of similarly structured learning programs, given, of course, content-related adjustments and considerations of the challenges expressed above. This would mean that research should be advanced with the investigation of KPIs and data collection methods for evaluating broader and structural organizational changes in the management of sustainable and innovative mobility solutions. KPIs that enable an objective analysis or long-term impacts on organization sustainability and resilience (and even on career trajectories of participants), with the use of time series data, would be considered of particular added value in such cases.

Tracking the impacts of a structured learning process at the community level—in our case, this relates to sustainability goals reached as a result of the implementation of mobility solutions (i.e., modal shift to more sustainable modes of transport, resulting, in turn, in reduction of noise and emission)—was not made possible through our proposed evaluation framework, and thus still remains a major challenge for future research. Evaluation frameworks require a detachment of the achievements attributed to the object under evaluation (in our case the Learning Programme) from achievements attributed to other, external initiatives or activities (similar or not). But then, in an ecosystemic approach, such as the one for sustainable mobility, where different initiatives interact with and are combined with each other to achieve the highest possible benefit, this requirement might be considered highly controversial.