Evaluating Social Impact of Smart City Technologies and Services: Methods, Challenges, Future Directions

Abstract

This study examines motivations, definitions, methods and challenges of evaluating the social impacts of smart city technologies and services. It outlines concepts of social impact assessment and discusses how social impact has been included in smart city evaluation frameworks. Thematic analysis is used to investigate how social impact is addressed in eight smart city projects that prioritise human-centred design across a variety of contexts and development phases, from design research and prototyping to completed and speculative projects. These projects are notable for their emphasis on human, organisational and natural stakeholders; inclusion, participation and empowerment; new methods of citizen engagement; and relationships between sustainability and social impact. At the same time, there are gaps in the evaluation of social impact in both the smart city indexes and the eight projects. Based on our analysis, we contend that more coherent, consistent and analytical approaches are needed to build narratives of change and to comprehend impacts before, during and after smart city projects. We propose criteria for social impact evaluation in smart cities and identify new directions for research. This is of interest for smart city developers, researchers, funders and policymakers establishing protocols and frameworks for evaluation, particularly as smart city concepts and complex technologies evolve in the context of equitable and sustainable development.

1. Introduction

Smart city systems collect an abundance of data about urban life, monitoring everything from energy use and environmental conditions to personal mobility and consumption habits. This data can be used to build a picture of how a city operates, to improve performance and to inform policy, but less is understood about the short- and long-term social effects of smart cities, the extent to which smart city initiatives are accepted by citizens, or how they shape the urban experience and influence the lives of residents. For example, how do residents of urban areas experience new technologies and services like autonomous shuttles or delivery drones, or participate in virtual decision making about their neighbourhood? Do smart city processes exclude certain groups of people who believe that their technological skills are inadequate for participation and might autonomous city infrastructures exclude people altogether? Do citizens feel their needs are prioritised over technological development or the other way around? In the emerging “autonomous city”, in which Cugurullo observes “artificially intelligent urban technologies are taking the management of urban services out of the hands of humans” [1] (p. 2), it is all the more pressing to understand the social impacts of smart cities and how they can be studied. In a similar vein, calls to reframe smart cities towards a people-centric approach that utilises smart technologies as tools to tackle citizen needs and social problems (the so-called smart city 2.0 or 3.0), requires the evaluation of impacts from the perspective of diverse stakeholders [2,3].

To date, social impacts in smart city discourse can be characterised in two ways: they tend to be portrayed as positive and they lack specificity. Across smart city projects, the “social” is implicit in goals like improving quality of life (QOL) and well-being but the term “social” is often undefined in the given context. According to Beretta, technology is perceived as leading automatically to a common good and an improvement in the city [4]. However, there is a “deficit of proof”, as described by Paskaleva et al., when it comes to the impacts of smart city initiatives [3]. The evaluation of social impacts in particular has received little attention in comparison to areas such as technology, mobility and energy [5]. While social indicators are frequently included in global, quantitative macro-level indexes, like the Lisbon Ranking and Digital Economy & Society Index (DESI), they do not offer the same information as social impact assessment, which examines how things are felt or experienced at different levels, from individuals to households or communities [6].

Defining social impact can be challenging in smart city projects, but it is vital for future cities and the increasing automation of cities. Social impact can be highly subjective and context-dependent, but understanding this subject can offer important insights into how diverse stakeholders experience and value change [7]. We draw from concepts of social impact assessment (e.g., [6,8,9]) and others to argue that this perspective is necessary if smart city design is to be socially inclusive, equitable, sustainable and responsive to citizen’s needs. Indeed, this is a requirement outlined in many smart city policies and standards, such as ISO 37100 (Sustainable cities and communities) and ISO 37120 (Sustainable development in communities—Indicators for city services and quality of life). However, there are currently no clear standards to evaluate social impact in smart cities. To address this, we examine eight smart city projects that have social impact embedded in their purpose. This allows us to gain new perspectives on what is missing from quantitative measures of smart city performance.

Our research questions are:

• What is considered “social impact” in smart cities?
• How are social impacts currently reflected in smart city frameworks? What are the strengths and weaknesses of the evaluation methods and indicators being used?
• What new perspectives are emerging on social impact in research-led, smart city projects and how could that impact be evaluated?

To answer these questions, we first report how social impact has been defined and assessed, and then investigate how this topic is reflected in smart city evaluation frameworks. Second, we review eight projects from around the world to compare their approaches to social impact and propose how that impact could be evaluated. These projects were originally selected from an open call for a NordiCHI 2020 conference workshop exploring this topic. They represent innovative research led by university and public sector partners in Europe, North America and Australia. The various types of projects and their global distribution allow us to compare a broad range of smart city research, contexts and stages of development, including community engagement, prototyping, completed and speculative projects. Data consists of project descriptions (

Table 1. Project summaries and approaches to social impact.

Project Name, Location, Status	Project Leaders, Stakeholders and Intended Audiences	Project Description			Project Aims	How Smart City Is Defined in This Project
3.1-Living Labs
A-Solid Waste Management, Finland Ongoing Project	LUT University CroBoDDIT project University, municipality, residents and businesses	A living lab approach was piloted to improve recycling practices on a university campus. The goals were to understand how to improve recycling by staff and students, and how to achieve living lab principles of openness, empowerment of users and realism. Methods included surveys, interviews, and a co-design workshop.			■ To develop smart solutions for solid waste management ■ To pilot a living lab approach	“Smartness” may lie in the capacity of empowered citizens to understand and frame the solutions that have the most impact for them, whether or not they are technology-driven.
B-Sustainability Means Inclusivity (SuMIn), Sweden 2020–2022	Linköping University Researchers: ethnographers, engineers, and designers MSc Design students Feedback: municipal representatives Intended user group: diverse inhabitants who engage with Kungsgatan on a daily basis (e.g., residents, shop owners, municipal actors)	The project aimed at engaging citizens, technical developers and other stakeholders in a series of Living Labs in Norrköping, Sweden. The research team used cultural probes to create dialogue with citizens and involve them in the living lab where possible futures for a smart and sustainable Kungsgatan were explored together with researchers and municipal actors. The resulting future visions were used in a backcasting process to prepare briefs for prototyping data-driven public services based on the current testbed technology.			■ To raise awareness of specific needs of marginalised or overlooked groups in urban environments ■ To provide municipality and technology providers with data on how the test site is used by a diverse range of local citizens that will facilitate long-term planning ■ To highlight the diversity of needs in the test site, and develop a model for engaging with citizens to ensure greater inclusivity throughout the city ■ To foreground assumptions underlying development of smart city solutions in order to refine future development of such solutions	A space where inclusivity and citizen engagement determine whether a smart city solution is adopted in the long term.
C-SimpliCITY, Austria and Sweden 2020–2021	Salzburg Research Project partners: 1 company, 2 city administrations, 1 university, 2 research organisations Audiences: citizens of Salzburg and Uppsala Assessment: company developing the platform and research partners	Co-creation activities in Salzburg and Uppsala led to two urban platforms that aggregate local sustainability services, engage citizens and empower changes related to mobility and consumption.			■ To foster collaboration and multi-stakeholder engagement in urban sustainability ■ To identify physical and digital methods for playful citizen participation and behaviour change in the field of sustainability ■ To increase visibility of local sustainability services and motivate citizens to use them ■ To empower changes related to mobility and consumption	A balanced concept, comprising sustainability, technological innovation, a human-centred approach and a consideration for economic value.
3.2-City-Public Interfaces
D-Smart Kalasatama and Oodi Library, Finland Ongoing Project	Aalto University Stakeholders: Forum Virium Helsinki facilitators, Kalasatama residents, companies experimenting with smart solutions in Kalasatama, Oodi Library staff and visitors	Aalto University researchers studied how city employees act as intermediaries, encourage citizen participation and facilitate the development of a smart city in two initiatives: Smart Kalasatama, a smart city project and testbed neighbourhood for companies that want to experiment with smart solutions for sustainable urban living, and Oodi, the Helsinki Central Library.			■ To analyze the City of Helsinki’s efforts to develop a functional city and thereby to become smart by increasing both livability and active citizenship ■ To assess the role of city employees as facilitators of smart city development and of social inclusion and participation of diverse stakeholders	For Smart Kalasatama, development efforts focus on three areas of smartness - energy, mobility and living - with an overall aim of promoting smart economy. At Oodi Library, smart city development efforts primarily focus on the advancement of social and human capital with the aim of promoting citizen participation in public life.
E-The Building City Dashboards (BCD) Project, Ireland Completed Project	Maynooth University Project partners: Dublin City Council, Fingal County Council, Cork City Council and Cork County Council, as well as Dublinked, Ordnance Survey Ireland and the Central Statistics Office	Two cities were used as living labs to create two public-facing, functioning, open-data-driven smart city dashboards that aid cities to (a) become better informed and more effective in decision-making and planning through visual analytics, data analytics and new multimedia tools; and (b) improve data management, data practices and data literacy within stakeholder organisations.			■ To tackle problems related to data, visualisation, interaction, analytics and modelling ■ To enhance citizens’ quality of life, local conditions for business, and contribute to creating scientifically informed citizens who can meaningfully engage with data and technology	Data-driven urban management and governance.
F-Citizen Voices in Cities (CiViC) Dashboard, Australia Prototype	University of Sydney Project partners: two local authorities in the greater Sydney area and Brisbane	The project generated a proof-of-concept web platform that mines social media posts and offers search features and aggregated visualisations. The dashboard is intended to support city authorities to understand current issues in local communities and assess the long-term impact of development projects.			■ To capture the voices of people from difficult-to-reach demographic groups ■ To feed debates happening on social media into city planning	This project focused on “smart governance” of smart citiy initiatives, which relates to “using CS/IT to improve democratic processes and public services (e-government) and to support and facilitate better planning and decision making” (Camero & Alba, 2019, p. 86).
3.3-Forecasting
G-Roosevelt Island’s Digital Twin for Social Impact, USA Proof of Concept Prototype	Cornell Tech Audiences: planners, developers, municipalities, public	The digital twin combines a 3D model with socio-economic data and demographic information. The data-driven microsimulation is based on historic urban research with data sources including: the American Community Survey, the neighborhood master plan, privatization agreements and resident interviews.			■ To communicate through a digital twin the less visible, social aspects of urban life, such as inequality and community resiliency	Digital simulation of a city based on multiple datasets and urban sources to understand and predict social impacts of development.
H-Greensight: Scale-Up Perspectives in Urban Green for Human Centered and Livable Urban Cores, Finland Conceptual Project 2017–2019	Finland Futures Research Centre, University of Turku Project partner: City of Turku	This interdisciplinary project used futures and foresight methods to find alternatives to two of Turku’s main urban challenges: spatial fragmentation and low human centeredness. The concept of Greensight explores natural enabling systems within the construction of alternative futures and explores the role that planning and technology play in those futures.			■ To explore the "green" layer in the smart city-Turku as an anticipatory exercise focusing on the social implications impacting wellbeing and the future of life in cities	The smart city as a future imaginary where the reconciliation of pressing social issues with urban technology development must be re-thought.
Project Name, Location	Project’s Human-Centred Approach	Connection to Environmental Sustainability	Project Evalua-tion	Methods of Project Assessment	Proposed Social Impact Indicators	Possible Social Impacts
3.1-Living Labs (continued)
A-Solid Waste Manage-ment, Finland	This project used living lab principles to plan a smart city project, thereby emphasizing diversity, inclusivity and choice in a design process that leads to real outcomes that were envisioned by participants and will benefit wider society.	■ Changing behaviours regarding solid waste management	Formative evaluation of project	Framework mapping living lab principles to co-design stages	Achievement of living lab principles: ■ Openness (inclusion, diversity of participants and ideas) ■ Empowerment (users are given choice in developing solutions and see their ideas become concrete outcomes) ■ Realism (solutions to be used and benefit society)	■ Improved social cohesion among stakeholders ■ Increased alignment with recycling goals ■ Increased sense of inclusion in city service design ■ Increased sense of empowerment in making a change towards more sustainable practices
B-Sustainability Means Inclusivity (SuMIn), Sweden	This project engages citizens as active participants in the early-stage design and development of smart city technology. It creates inclusive and democratic shared social space, emphasizing different generations, from young adults to elderly people.	■ Increasing green space, walkability and cycleability ■ Reducing traffic	Formative evaluation of project	■ Living Lab methodology including evaluation of prototypes (that were inspired by future visions) through co-creation activities	■ Engagement of marginalised and overlooked groups in co-creation and the municipality and technology providers in informed discussions on the diversity of needs in the test site, its use by a diverse range of citizens ■ Drop in levels of noise, pollution, and traffic ■ Use patterns in the test site that emerge from citizen-determined data	A more inclusive and sustainable Kungsgatan based on: ■ New data-driven public services ■ Less noise and traffic ■ Increase in walkable, bikeable and car-free green areas ■ Better intergenerational and intercultural communication ■ Creation of open and democratic community spaces ■ Improved connections with nature and biodiversity
C-SimpliCITY, Austria and Sweden	While sustainability has been a key focus of smart city strategies in both Salzburg and Uppsala, the human-centred aspect was not substantially integrated in previous initiatives. The platforms adressed the optimisation of digital tools for city administration; increased visibility of services and easier access to local offers for citizens. It emphasizes a range of stakeholders through codesign, and incorporates learnings about how stakeholders value what they learned through the project (e.g. building trust, increased collaboration, better understanding of smart city goals).	■ Changing behaviours in support of urban sustainability ■ Increasing adoption of sustianability services especially related to mobility	Formative and summative evaluation of the project	■ Iterative co-design process ■ Quantitative pre-post analysis and descriptive statistics ■ Qualitative interviews, urban probes	■ Indicators related to satisfaction with usability, user experience, platform features, usage intensity ■ Changes in behaviours (mobility, consumption, uptake of local sustainability services), in organisational priorities (whether city administrators and service providers changed approaches to strategy or services) ■ Perceived community attachment because of localised campaigns or city tours and perceived trust and collaboration	■ Added value for all stakeholders involved ■ Increased sustainable behaviour ■ More human-centered approaches and sustainability initiatives ■ Increased trust and collaboration ■ For people: changes in behavioural attitudes, physical and emotional expressions ■ For communities: revealed co-constructed norms and values of the "fluid" community
3.2-City-Public Interfaces (continued)
D-Smart Kalasatama and Oodi Library, Finland	The research examines how city employees strive to improve citizen engagement and inclusion in smart city initiatives by providing opportunities for participation, bringing different actors together, negotiatinig between them and adapting technologies and services for different needs.	Helsinki city strategy and the studied initiatives include sustainability goals but they have not been analysed in this project	Summative evaluation of project’s social impact	■ Interviews (city employees and other stakeholders) ■ Field notes from meetings, workshops and public talks ■ Documents, studies and media reports about the initiatives	■ Stakeholder engagement (diversity and representativeness of participants) ■ Empowerment of residents and users (inclusion of diverse groups, active participation)	■ A more inclusive city ■ Improved participation of citizens ■ Improved facilitation skills of city employees ■ Opening up of city venues to the public
E-The Building City Dashboards (BCD) Project, Ireland	The project adopted a human-centred approach to smart city systems to balance the needs off citizens and city administrators. It focused on the user’s overall holistic experience with dashboard systems, acknowledging users’ diversity and using qualitative research to develop design ideas that addressed different user requirements and cognitive styles.		Formative evaluation of project	■ Iteration ■ Personas ■ Open evaluation by public and industry-focused users ■ User testing and feedback throughout the project allowed the impact assessment of features and changes	■ Human-computer interaction indicators for functionality, usability, and user experience	■ Influence of insights, techniques and solutions for urban dashboard developers, city managers, urban data and smart city development projects, and the open data movement
F-Citizen Voices in Cities (CiViC) Dashboard, Australia	The project sought to augment existing community engagement activities, which form an essential component of all new city development in Australia, not just smart city development. The project proposed smarter ways of using existing data to capture citizens’ voices, rather than driving top-down, technological smart city solutions.		Formative evaluation of the project	■ Iterative co-design and interviews with government authorities ■ Proof-of-concept walkthrough: focus group with government representatives	■ Access to data from local communities ■ Outreach and inclusion of non-mainstream groups in city planning	■ Better understanding and consideration of local concerns, requirements and community needs before and during the implementation of large urban developments
3.3-Forecasting (Continued)
G-Roosevelt Island’s Digital Twin for Social Impact, USA	This digital twin shifts attention from the built environment to the invisible social impacts of gentrification and privatisation. It shows changes in demographics and affordable housing over the past 45 years and projects 50 years into the future.				■ Communication of demographic change, e.g., affluence and age of households ■ Communication of social effects of privatization processes	■ Better informed public about potential impacts of privatization processes ■ Increased community resiliency and participation
H-Greensight: Scale-Up Perspectives in Urban Green for Human Centered and Livable Urban Cores, Finland	Greensight applies a multi perspective approach. It seeks to examine the many layers of life, the human and non-human ones, to explore impacts, deficiencies and their critical role in understanding social change. As technology and digitalization continue to expand, nature relations need further consideration as an important counterbalancing factor for achieving human centricity.	■ Framing cities as part of the environment, not separate ■ Using nature areas as intervention points when integrating the human experience of life in cities with physical and natural environments	Proposed framework for envision-ing alternative futures	■ Foresight methods: weak signals, trend analysis, data & statistical analysis, surveys, workshops, observations, site photographs and videos	■ Urbanization, population density, spatial distribution of and access to green space ■ Satisfaction of the basic human need to experience life outdoors	■ Understanding urban phenomena and change ■ Re-evaluation of values and priorities ■ The human factor placed at the centre ■ The dimensions of future societies of intangible needs

and Appendix A) supplied by ten of the project researchers and authors of this paper. Third, through an analysis of the projects, we identify emerging perspectives on social impact and corresponding gaps in smart city frameworks. Fourth, we outline future directions and areas of opportunity for the evaluation of social impact in smart cities. We contend that more coherent, consistent and analytical approaches are needed to build narratives of change and to comprehend impacts before, during and after smart city projects.

Our findings provide analytical approaches to the evaluation of social impacts in smart city initiatives. We also propose criteria for new evaluation practices that allow for greater understanding of social impacts at the same time that they support technological development. This is of interest for smart city developers, researchers, funders and policymakers establishing protocols and frameworks for evaluation of complex urban technologies. By addressing these current priorities and challenges, we can foster and support social impact in smart cities. This is particularly important for public stakeholders who are responsible for ensuring equality and justice in urban development in general and smart and sustainable development in particular.

2. Social Impact and the Evaluation of Smart Cities

2.1. Social Impact

The term social is widely used in smart city discourse, often uncritically and without specificity. As the concept broadly refers to society and to groups of people and their well-being, it is easily applied to almost all aspects of smart cities. Linking the concept of happiness and well-being (as important indicators of social progress) with access to a wide range of public and private digital services, not only creates a system of complex interdependencies but also makes it difficult to evaluate correlations. For example, it is challenging to establish how much the social layer of the city is improved through digitalisation and the implementation of autonomous systems, or to what extent the way people live and work is transformed by digitalising and automating the city and not the other way around. Smart city projects are often assumed to have a positive social effect. This is underlined by the belief that “all that is technological is ‘good’ and might lead ‘automatically’ to the transformation and improvement of the city” [4] (p. 116). Thus, projects motivated by efficiency (e.g., reducing traffic congestion or increasing connectivity) or economic growth (e.g., creating new jobs and supporting entrepreneurship) are described as having social benefits but without explaining how they contribute to citizens’ QOL or well-being in measurable ways (cf. [10]). The lack of precision with which the term social is used reflects a lack of research about who benefits from smart cities and whose responsibility it is to advance a social agenda [11]. Note that while smart cities are intended to “work for [their] inhabitants”, little is actually known about what citizens want and need [5] (p. 667). Trencher points out that further research is needed to define and demonstrate the characteristics of citizen-centric smart cities, while acknowledging that this will vary across cultures and locations [2].

2.1.1. Social Impact Assessment

Social impact assessment (SIA) and environmental impact assessment (EIA) are well-established practices that have become standard in the planning of major development and infrastructure projects [9]. However, they do not appear to be widely adopted in the context of smart city projects [12]. Vanclay offers the following definition:

“Social impact assessment is the process of analysing (predicting, evaluating and reflecting) and managing the intended and unintended consequences on the human environment of planned interventions (policies, programs, plans, projects) and any social change processes invoked by those interventions so as to bring about a more sustainable and equitable biophysical and human environment”.

[6] (p. 190)

Slootweg et al. differentiate between “change processes (being tangible, objectively verifiable and measurable processes) and impacts (as subjective, context-dependent final variables of impact studies)” [8] (p. 27) (emphasis added in italics). Similarly, Vanclay argues that “social impacts must be experienced or felt” and this may be “corporeal”, “perceptual” or “emotional” [6] (p. 201). It is not enough to measure, for example, increased cultural diversity or adoption numbers for autonomous vehicles. Rather, social impact is how those factors are experienced by the people studied. This could be, for example, through a greater sense of belonging to a city, or a greater sense of trust in service providers and technologies. Vanclay also outlines different levels of impact, from the individual to the household, organisation, specific community or society more broadly [6]. As illustrated in Section 2.2 below, smart city indexes tend to provide quantitative data about measurable, society-level change. However, they do not provide any indication of who has been studied or the social impacts they experience.

In the context of design and urban development projects, change can be evaluated by comparing a defined scenario before, during and after an intervention. Summative evaluation assesses outcomes and impacts, while formative evaluation is integrated in the planning process and conducted throughout the project [13]. According to the OECD, outcomes are short-term changes and impacts are “positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended” [13] (p. 24). The Design Council (UK) describes social impact as the “degree of social change” that people experience as a result of design [14] (p. 5). That social change is made evident in direct impacts, such as how design improves health or increases inequalities, as well as in the “often invisible ‘ripple effects’, like creating new relationships, sparking new ways of framing challenges, and building confidence in participants” [14] (p. 4). This illustrates that social impacts can occur outside of the immediate scope of a project, and that they can be unintended, intangible and difficult to quantify. Defining social impact also depends largely on who is asked—indicators determined by a few stakeholders cannot capture the full social impact of a project [7]. We observe that overall, there is a lack of evaluation of both social “outcomes” and “impacts” and argue that all of these approaches—subjective and objective, qualitative and quantitative, studying what is directly and indirectly linked to projects and including a variety of stakeholder perspectives—are needed to understand the effects of smart city projects on the people involved.

2.1.2. Smart City Context

Smart cities have particular qualities that influence how social impact should be conceived and assessed. Key distinctions of smart cities include: the latest urban technologies, such as sewage systems, water supply networks and mass-transit systems; ICT combined with infrastructures, architecture and everyday objects or our bodies [15,16]; economy and governance driven by innovation, creativity and entrepreneurship, enacted by smart people [17]; and a focus on urban development that enhances lives of citizens [18]. Cugurullo observes that smart cities are difficult to define in part because smart technologies change so rapidly [19]. This is, as seen in the recent shift from “automation to autonomy” with the introduction of artificial intelligence, manifested in autonomous vehicles, robots and urban management platforms [19]. Camero and Alba emphasise that the focus should not be technology itself but rather how technology supports the development of smart cities [20]. Similarly, Bakici et al. argue that the smart city “connects people, information and city elements using new technologies in order to create a sustainable, greener city, competitive and innovative commerce, and an increased life quality” [21] (p. 139). Sharifi concludes that, “while there is still no universally accepted definition, a wide consensus exists that a smart city is one that utilises ICT-enabled solutions in various socio-economic, institutional, and environmental domains to increase […] (QOL), sustainability, and resilience and to maintain the competitive capacity of cities in an increasingly interconnected network of cities” [22] (p. 1). Other concepts describe generations of smart city development, from version 1.0, driven by technology, business and economy, to 2.0 and 3.0 driven by governments and citizens addressing social challenges, increasingly through co-creation methods [2,3]. It is challenging to delineate and assess social impact given how multi-faceted these definitions of smart cities and their stakeholders are.

2.2. Social Impact in Smart City Frameworks

Many frameworks, indexes and matrices have been developed to evaluate and compare smart cities. Sharifi creates a typology based on 34 smart city assessment tools, noting that the most common areas evaluated are: “economy, people, governance, environment, mobility, living, and data” [22] (p. 1). The shared objective of these assessment schemes “is to evaluate the level of success in increasing QOL and sustainability of cities and to examine their success in maintaining competitive capacity” [22] (p. 2). Inconsistent terminology and indicators can make these frameworks difficult to compare. QOL and well-being relate closely to social impact and are used often and with overlapping meanings. For example, QOL is one of three pillars used by the OECD in the measurement of well-being, along with material living conditions and sustainability [23]. In everyday language, QOL is defined as “the degree to which a person or group is healthy, comfortable, and able to enjoy the activities of daily living”, which is similar to well-being as “the state of being happy, healthy, or prosperous” [24]. The use of these terms, while inconsistent, illustrates that these smart city evaluation tools do consider social impact in different forms and to varying degrees. Below, we look at how some of the most prominent frameworks take social impact into account, as well as how several address social impact in a more focused way.

2.2.1. Current Frameworks Including Social Factors

At a global level, several quantitative approaches have been developed that include the evaluation of social aspects in relation to technology: Lisbon Ranking and Digital Economy & Society Index (DESI). Lisbon Ranking is based on indicators jointly proposed by the United Nations Economic Commission for Europe (UNECE) and the International Telecommunications Union (ITU) to investigate the smartness and sustainability of cities. This includes social factors, such as education, health, safety, housing, culture and inclusion. The Lisbon Ranking uses statistical analysis like hierarchical clustering and principal component analysis, with Eurostat as a main data source [25]. Examples of data relevant to social impact include variables of gender pay gap, voter turnout in national and EU parliamentary elections, persons employed between the ages of 20 and 64 and Gini coefficient of disposable income. DESI is a composite index that summarises relevant indicators of Europe’s digital performance and tracks the evolution of states (as opposed to cities), across five dimensions: connectivity, human capital, use of Internet, integration of digital technology and digital public services. Variables relevant to social impact are, for example, the level of internet user skills, digital public services for citizens, user centricity score of eGovernment services and the number of science and technology graduates. The results are mainly intended for national policymakers [26]. In addition, the European Commission has published the EU Regional Social Progress Index (EU-SPI), which covers categories of basic human needs, foundations of well-being and opportunities [27]. While not directly addressing smart cities, the EU-SPI’s 50 indicators are examples of how the “social” can be measured through statistical analysis of phenomena such as personal security, access to information, communication and freedom of choice. In addition, the ISO 37122:2019 standard covers indicators for smart cities and includes, for example, cultural issues such as number of book and e-book titles per 100,000 people, percentage of public recreation services that can be booked online and safety (percentage of city area covered by surveillance cameras).

Social factors are also visible to some extent in general assessment frameworks, such as Kourtit et al.’s six angles for approaching impacts in smart cities (social and human capital, competitiveness, transport and ICT, civic participation, QOL and natural resources) [28]. This is also true in Giffinger et al.’s focus areas of a smart city: smart economy (competitiveness); smart people (human and social capital); smart governance (participation); smart mobility (transport and information and communications technology (ICT)); smart environment (natural resources); and smart living (QOL) [29]. One of the most well-known tools is the Smart Cities Index, which includes characteristics (living, economy, people, governance, mobility and environment), factors and indicators (see [5]) and is used to rank cities according to level of services provided.

All of these quantitative indexes provide important information about urban development. However, while they represent social change, they do not capture social impact [6,8]. To do so, they would need to include measures for assessing how interventions are felt or experienced and by whom.

2.2.2. Social Sustainability

Measuring social impacts can be connected to the wider aim of sustainable development in smart cities. Our research links to previous findings such as the principles of social sustainability—equity, security, inclusion and adaptability—outlined in the city of Vancouver’s Social Development Plan (2005) and the framework for social sustainability assessment (see [30] (p. 10)). Another example is the Social Sustainability Assessment Framework (SSAF), originally developed for urban generation by Colantonio and Dixon [31], which could be applied to smart city contexts as well [11]. In this framework, social sustainability “concerns how individuals, communities and societies live with each other and set out to achieve the objectives of development models, which they have chosen for themselves, taking also into account the physical boundaries of their places and planet earth as a whole” [31] (p. 18). The SSAF includes ten dimensions and policy areas, including demographic change, social mixing, inclusion and cohesion [31]. More recently, the City Model Canvas (CMC) assesses value created in an economically, environmentally and socially sustainable way through smart services based on fourteen elements, such as value propositions and social benefits [32].

Even though the concept of social impact is not mentioned as such in most of these frameworks, the idea that smart cities are intended to improve QOL and well-being is clear. This is seen, for example, in Giffinger et. al.’s indicators, such as “social and ethnic plurality” as an indicator of “smart people”, and “social cohesion” as an indicator of “smart living” [29] (p. 12). For Bremer et al., when thinking of cities as a service, smart cities aim to create positive impacts using available forms of urban capital (industrial, human, social and ecological) [33]. They maintain that people should not only be considered as capital but also as the centre of smart cities (“People first”). Similarly, Trivellato argues that the focus of smart cities should be on social sustainability, where basic needs (e.g., food, housing and health) and basic values of equity and democracy (education, social inclusion and cohesion, integration) are prioritised [11].

The connection between sustainability and social issues has been recognized recently in the smart city research domain. For example, Bouzguenda et al. review the role of digital citizen participation (DCP) in advancing social sustainability and argue that DCP could significantly contribute to social sustainability [34]. Related to impact assessment, Orejon-Sanchez et al. compare technical and social indicators in smart city development and conclude that Spanish smart cities have synergies with tourism and quality of life [35]. This is evident in particular when it comes to social cohesion, urban planning, international outreach and technology implementation in urban development [35]. Research about the introduction of autonomous vehicles highlights the social implications of their adoption. It suggests that social impacts might be identifiable, for example, in shifts in attitudes on safety and shared ownership, new patterns of mobility and even land use as the need for parking lots diminishes and more space becomes available for housing and parks [36,37]. There are several further social challenges linked to the implementation of autonomous services, such as transport pods, as they would traverse different kinds of social spaces like train stations, public parks and educational institutions [38]. Evaluating both positive and negative impacts of autonomous technologies before their large-scale implementation will be crucial to the achievement of social benefits that smart technologies promise for future cities [39,40,41].

2.2.3. Two Challenges in Evaluating Social Impact

Smart city indexes present two main challenges when evaluating social impact. The first is that they risk oversimplifying complex phenomena. Relying on quantifiable indicators [5] or focusing on individual rather than multiple smart city projects, programmes or strategies [11] make it difficult to see the situated, contextual and qualitative nature of how people experience social impact over time. The numbers may seem clear but the associated social phenomena are not. The second risk is that when determining what is measured, competing priorities, such as “enthusiasm for the opportunities generated by smart technologies” [11] (p. 338) or corporate goals [32], will overshadow social goals and the perspectives of certain stakeholders, in particular those who might be most affected by smart city projects. This has repercussions for city authorities and decision makers who are the primary audience for most smart city assessment tools [22]. It also affects the residents and users of smart cities who should be included in decision making processes about the future of their city [42,43].

Although we can conclude that “smartness” in cities is intended to create a positive social impact, there is a gap when it comes to evaluating this impact. Social impact assessment accounts for both positive and negative experiences of change. From the frameworks above, the lack of a unified methodology and consistent terminology for assessing social impact is apparent. Clearly defining social impact in the context of smart cities and recognising its complexity are needed in order to design impact evaluation strategies. The eight projects discussed below provide examples of how researchers and stakeholders address social impact, allowing for a more nuanced discussion of evaluation in Section 5 and Section 6.

3. Methodology

To decode social impact evaluation in a smart city context, our research design takes a global approach. We selected eight projects from six countries, conducted by different types of stakeholders using a variety of methods. This ensures a greater diversity of perspectives and definitions of social goals than in the indexes reviewed above. To ensure the validity of our results and conclusions, we assessed the projects systematically using thematic analysis over the five phases described below (summarised in Figure 1). All phases were conducted by the first three authors of this paper (we refer to ourselves below as “primary researchers”) with input throughout from the other authors.

3.1. Data Identification

The eight projects studied here were originally submitted for discussion in a five-hour conference workshop, Decoding The Smart City, hosted by the primary researchers at NordiCHI 2020 [44]. This method was chosen because of the primary researchers’ backgrounds in design research. The workshop was designed to produce ideas for context-sensitive social impact evaluation strategies. It was aimed at practitioners and researchers working on smart city projects and interested in developing their capacities in social impact assessment. Participants were invited to submit position papers about their smart city projects in order to be accepted in the workshop. A total of 10 projects were submitted. The workshop consisted of three activities: (1) co-creating a definition of social impact, by mapping priorities and affinities using visual collaborative methods; (2) discussing evaluation strategies for social impact using participants’ projects; and (3) ideating scenarios with speculative future casting methods. Participants designed social impact evaluation strategies for the speculative scenarios, mapping advantages and limitations of each strategy. The scenarios were created based on three real-life case studies of smart city projects presented by guest speakers.

After the workshop, the organisers invited all participants to contribute to the collaborative study presented in this paper. This process resulted in eight projects, which provided the data for analysing how social impact is addressed in research-led smart city projects. Participants submitted an image and a 500-word description of their projects and their approach to “social” (see Appendix A). Participants were also invited to compile information in a table, based on the workshop structure and discussions. This included: project stakeholders, aims, definition of smart city, human-centred approach, connections to environmental sustainability and, where possible, methods of assessment, potential social impacts and social impact indicators (see Table 1).

The fact that the projects are based on an open call for contributions to a NordiCHI 2020 workshop likely influenced the types of projects that we were able to use as examples. However, as the workshop was held fully online, there were fewer obstacles for participation compared to face-to-face workshops held in association with a conference. As a consequence, we were able to choose projects spanning six countries and a wide range of smart city initiatives at different implementation stages and representing various forms of collaboration between research institutions and organisations.

3.2. Data Analysis

The primary researchers conducted the analysis in five phases (see Figure 1). During the first phase (familiarise), the three of us independently reviewed the project descriptions and the table to gain an overview of the approaches to human-centred smart cities and social impact.

In the second phase (code), the primary researchers met virtually and used the online platform Miro to conduct a collaborative, thematic analysis [45]. This deductive process involved coding each project based on the categories of: project aims, research questions, approaches (priorities, values, ideologies), methods, stakeholders, outcomes, evaluation strategies and indicators. The three of us used affinity diagrams to cluster the notes, compare the categories across the projects and identify recurring themes. Throughout, we worked independently and together to write up findings based on the thematic analysis. We met frequently and discussed topics related to the possible social changes, impacts and indicators of each project.

In the third phase (categorise), the primary researchers identified three different categories of projects based on common methods, priorities and uses of technology: living labs, city–public interfaces and forecasting (see Table 1). The latter category focuses on prediction of social impacts, whereas the first two include projects that are in progress or have been completed. These categories are used to organise the projects in Table 1.

In the fourth phase (compare), new questions emerged from the discussion of the affinity diagrams and a comparison of the projects. The primary researchers grouped these questions under the three overarching topics structuring the analysis (Section 4 below): approaches to smart city research; how “the social” is addressed; and how social impact is assessed (approaches to evaluation). The three of us noted that because many of the projects were ongoing, they involved formative methods of assessment (evaluating and refining work in progress). Furthermore, many had no formal plans to assess social impact, meaning that some projects had no information in the table under “social impact indicators” or “potential social impacts”. To address this, the primary authors proposed “potential” social impacts and indicators where none were identified (see Table 1). They are speculative and intended for discussion only—we recognize that social impacts and indicators are ideally established in consultation with stakeholders [46]. However, by imagining possible social impacts and how they could be identified, all of the authors were able to point to gaps in the evaluation of the eight projects and in existing evaluation frameworks (Section 2 above).

In phase five (review), the primary authors analysed the themes and gaps that emerged in phases two and three. This enabled the three of us to identify a range of approaches to social impact and to develop recommendations and criteria for future social impact assessment (see Section 5 and Section 6 Discussion and Conclusions). Throughout the five phases, all authors provided feedback on the analysis through shared drafts.

3.3. Data Summary

Table 1 summarises the eight projects analysed in terms of how they approach smart city research, how they address “the social”, and how they assess social impact. Projects are labelled A–H and their full descriptions and images are provided in Appendix A.

4. Findings

4.1. Approaches to Smart City Research

All of the projects are linked to research institutions, meaning that they are largely done in support of understanding and contributing to smart cities from an academic perspective. At the same time, the majority were conducted in partnership with municipalities, and involved citizens and local businesses in the development of new products, services and systems (see stakeholders in Table 1). While these projects may differ from those initiated solely by government or private sector organisations, they play an important role in smart city innovation. They approach social concerns in ways that have the potential to influence other aspects of smart city development. They highlight gaps in the frameworks discussed above (Section 2) and lead us to propose criteria for social impact evaluation in smart cities.

The projects represent a diverse selection of current projects from around the world (Australia, Austria, Finland, Ireland, Sweden, the USA), at different phases of development (conceptual, completed or ongoing), with various aims tied to smart technology. Through the analysis, three broad categories of projects emerged based on common priorities and social goals, project structures, methods and use of technology: living labs, city–public interfaces and forecasting.

4.1.1. Living Labs (Projects A–C)

Living labs bring together diverse stakeholders, including citizens, private and public sector organisations, to further research and innovation in real-life settings [47]. These three projects emphasise inclusive, human-centred design, collaborative processes and prototyping in the development of smart city systems concerned with sustainability. The third project is not identified as a living lab, but shares similarities in its iterative, co-design approach with multiple stakeholders.

4.1.2. City–Public Interfaces (Projects D–F)

The projects in this group examine how citizen engagement with smart city services and data can be increased through different interfaces with the public. The first looks at the role of front-line city employees in facilitating citizen participation in smart city initiatives. The second two focus on the role of digital tools (dashboards) in making city data more accessible or more representative of diverse citizen perspectives. These projects emphasise openness, access, inclusion and relevance to users.

4.1.3. Forecasting (Projects G–H)

The projects in this group use speculation, simulation and foresight methods to anticipate future social impact in smart cities. The two projects stress human-centred decision making in urban planning and address complex relationships between well-being, the built environment and natural ecosystems.

4.2. How “Social” Is Addressed—Perspectives from Smart City Projects

Within the eight projects, we identified four common perspectives that offer insights into how social impact is construed in current smart city initiatives.

4.2.1. Human- and Eco-Centred

The projects emphasise social impact by placing human, organisational and natural stakeholders at the centre. Their foci can be categorised in two ways. Some are human-centred: they engage minorities or groups that are difficult to reach with traditional methods (C), raise awareness of the needs of marginalised or overlooked groups (D, F, G) or employ human-centred design (E) and community engagement during planning (A, B). Some demonstrate eco-centric approaches: they foster collaboration in urban sustainability development (C), tackle waste management (A) and propose ways of thinking about sustainable smart cities in the future (H). Their relationships to technology vary: some describe current societal changes using smart technology (G), some make an intervention with technology to have an impact on society (B, C), some analyse the impact when smart technology is implemented (E, F), while others frame new relationships with technology (H). Unlike many of the frameworks outlined in Section 2.2, the projects do not assume that social impact is an implicit benefit of smart cities, but rather something that needs to be co-determined with a variety of actors. Thus, the focus of the projects is on citizens’ and other stakeholders’ needs rather than technological solutions, and on user engagement during research and project development, rather than on demonstrating that final products have spin-off social impacts. In these ways, they present alternatives to a technology-before-people approach in smart city projects [11].

4.2.2. Inclusion, Participation and Empowerment

Projects recognise a multiplicity of actors and complexity of relationships, emphasising inclusion, participation and empowerment. Different scales and hierarchies of stakeholders, from individuals and households to communities, companies, academic and government organisations, reveal various understandings of smart city project objectives (C, D) and potentially of social impact. The projects provide specific, concrete examples of citizen needs and opinions that may run counter to policy and planning objectives, and which are not accounted for in assessment schemes that rely on broad categories of indicators. This points to the limitations of smart city indexes and quantitative assessments that do not represent a diversity of perspectives, or ‘levels’ as outlined by Vanclay [6].

4.2.3. New Roles for Citizens and Their Data

New voices and new information about citizen values and behaviours are integrated into the projects through innovative methods and spaces of engagement (e.g., coffee probes and living labs in A, B) and representation (e.g., social media scanning in F). Smart city technologies are being used to amplify citizen issues. In three cases, this is accomplished through platforms, specifically dashboards, that add public debates to planning processes or allow citizens to engage with smart city services and data in new ways (C, E, F). In one case (G) the digitisation and modelling of historical and ethnographic data make visible the future challenges of residents in affordable housing. To address power imbalances within smart city planning, the projects explore new stakeholder roles. In one case, the role of citizens shifts from passive to active (B). Two cases look at giving voice to citizen concerns, rather than setting direction from the top down (F, G), while another examines the role of city employees as facilitators between citizens and institutions (D). Multiple projects propose collaborative methods and other channels to bring together citizens with decision makers and planners (e.g., A, B, D, E). Goals include increasing openness, trust and adding value for all stakeholders (A, C). The more active presence of citizens suggests that new methods to evaluate social impact are needed to reflect citizen priorities, as well as the social impact of participatory design and inclusion in planning processes. Evaluation with the goal of being responsive and accountable to citizens could involve methods, indicators and modes of communication that differ from existing evaluations for government reporting. As suggested by Paskaleva et al., impact evaluation frameworks should be co-produced with relevant stakeholders [3].

4.2.4. Connections between Impact, Social and Environmental Sustainability

The projects propose new relationships between sustainability and social impact. The project researchers argue that the role of people must be emphasised in order to meet sustainability goals (B, C). One frames sustainability as inseparable from inclusivity (B) while another focuses on long-term social sustainability through community resiliency (G). A human-centred design approach aims to increase participation in planning and using sustainable services to shift stakeholders towards more sustainable behaviours (C). Future forecasting and backcasting emerge as two approaches to imagining sustainable smart cities (B, H). The SuMin project uses envisionment to inform current technology development (B), while the Greensight project expands the social realm to consider the role of non-humans in urban life (H). The latter positions cities as integrated within natural systems and looks to a future where green bio-economy resources are as foundational to the development of cities as new technologies are today.

These connections between impact and environmental and social sustainability are in line with the findings of Timeus et al. [32] and Beretta [4], suggesting the need for assessment methods that make this dynamic visible and account for human and non-human actors, as well as local contexts. A better understanding of social impacts would help to identify connections between social and sustainable development. This will become increasingly important for cities aiming at carbon-neutrality and supports the Do No Significant Harm principle (DNSH), part of the EU-led Recovery and Resilience Facility Regulation that defines assessment of environmental impact [48]. At the same time, evaluating environmental and social impacts requires a wide range of expertise, and combining them is challenging [14]. Furthermore, the relationship between environmental and social impacts highlights the larger influences on smart city projects. They cannot be considered in isolation. Evaluation should therefore be undertaken in relation to city strategies and other smart city initiatives [11], and broader policy and environmental contexts.

4.3. How “Social” Is Assessed—Approaches to Evaluation in Smart City Projects

The projects present a rich and diverse set of approaches to the “social” in smart cities. Analysis of Table 1 reveals the complexity of assessing the different forms of social impact they represent. Even though there appears to be some variance in how the projects interpret aims, indicators and methods of assessment, the overlap between the projects suggests that social impact is understood in a similar way. In the projects, social impact can be described as the effect that each project has on specific stakeholders, particularly regarding community and often in relation to the natural environment. Nevertheless, what is being evaluated varies, whether it is the social impact of the process and/or outcomes, or the wider impact of a project or strategy.

4.3.1. Evaluating Design and Development Processes

The projects do not involve formal evaluation, that is, the comparison of value created over time by an intervention in a defined context. However, several projects (C, E, F) feature development processes as methods of evaluation in themselves (e.g., iteration, co-design, prototyping and usability testing). In other words, formative evaluation took place as they assessed activities and validated prototypes against project goals and participants’ priorities to improve and sometimes reorient projects. To formalise this, indicators may be needed to evaluate project development itself. For example, two projects (A, B) use the living lab principles of openness, empowerment of users and realism to assess their process. This is interesting in the context of Gibson’s critique that in the built environment, the tendency has been to evaluate end-performance rather than processes [49], or Letaifa’s observation that the main focus of smart city evaluations has been on technological development and not on the improved service [50]. The future-oriented projects point to the need for indicators that assess policy and priority setting in scenarios that are still unknown, based on a balance between ecological and social sustainability (H) or between the current and future needs of tenants and property developers (G). Most of the projects are ongoing or recently completed, and there is no discussion of summative evaluation or the documentation of long-term impacts. This indicates that formal impact evaluation may not be common practice.

4.3.2. Qualitative Indicators and the Tensions between Strategic Goals, Project Outcomes and Impacts

The choice of indicators in the projects raises questions. Several kinds of qualitative indicators are used: behavioural change and emotional attachment (C), for example, relate to social and cultural change. Others, like functionality and liveability (D) relate to more practical aspects of everyday life. The projects illustrate the challenges of translating high-level, strategic goals into measurable, project-level indicators. Strategic goals tend to use language that is relevant to projects (e.g., “trust and collaboration” (C) and “healthy and sustainable urban development” (H)), but which is too abstract and unsuitable to the practice of evaluation. Goals (sometimes understood as indicators) of this type set vague and unrealistic expectations, especially for small-scale projects without the resources to assess them. Indicators need to be adapted to project scope with appropriate scales (e.g., “usability” (E) can be assessed with concrete, measurable indicators while “democracy and freedom of expression” (D) are more difficult to measure, or require a larger set of indicators). The confusion between indicators for measuring strategic goals and those for assessing project outcomes and impacts, together with the “misalignment of objectives from a range of stakeholders” (C), increases the difficulty for practitioners to conduct evaluations. Rather than trying to evaluate projects using predetermined, strategy-level indicators, researchers could establish whether those indicators are relevant at the beginning of a project. They could then develop tailored indicators that reflect stakeholder priorities and are realistic given the context and resources available for evaluation.

4.3.3. Quantitative Indicators and Inadequate Representation of Social Impacts

The projects show the difficulties caused when indicators are created by interpreting strategic goals and other indicators. In some of the projects, strategic goals are broken down into quantitative indicators but the rationale is not always clear. For example, noise, pollution and traffic reduction are used to represent improved QOL (B); number of media posts are used to represent augmented community engagement (F); and amount of time saved is used to represent functionality (D). Another approach is to transform one qualitative indicator into another, easier-to-grasp, but reduced qualitative indicator. Openness (A) and empowerment (A, C), for example, are reframed as diversity of participants and possibility to choose. A consequence of this interpretation process is that social impacts are not adequately represented through the quantitative indicators or they are oversimplified.

5. Discussion

Smart cities are characterised by high levels of complexity, social diversity and contextual uncertainty. As others have shown, it is difficult to capture this complexity in evaluations [4,5,11,51]. The existing smart city frameworks that we reviewed rely on quantitative indexes to represent social change, but do not capture social impact. The eight projects analysed reveal common perspectives and novel approaches to understanding social impact, but they also illustrate their own challenges with evaluation.

5.1. A Definition of Social Impact in Smart Cities

While social impact is generally understood as important, its evaluation lacks consistent and coherent definitions, measurements and methods. Drawing from other fields [6,13,14], we offer the following definition: “Social impact in smart cities refers to experiences of social change as a direct or indirect result of smart city projects”.

5.2. Opportunities for Evaluating Social Impact in Smart Cities

Based on our analysis, we outline the following areas of opportunity for evaluating social impact. Our projects show that a mix of qualitative and quantitative methods is necessary in human- and eco-centred smart city design. The projects present innovative, often open-ended and co-creative research methods that reach new audiences and lead to more inclusive planning processes. These methods generate rich, context-specific data that are difficult to quantify or evaluate using indicators derived from higher-level strategies. More qualitative methods are therefore needed to help build narratives of change and to understand impacts before, during and after smart city projects. This corroborates previous findings by Gupta et al. highlighting the risk of oversimplifying complex phenomena when using quantifiable indicators [5], and by Slootweg et al. and Vanclay who point out that social impact must be experienced and felt [6,8].

Appropriate methods and indicators can help to discern when impacts are attributable to individual projects, broader city strategies or other factors altogether. Leaving responsibility for acknowledging social impact up to each project can be an advantage, but it also decreases reliability of evaluation and highlights the importance of acknowledging that “stakeholders may superimpose their own meanings on top of knowledge for their own agenda and benefit” [52] (p. 876). A lack of rigorous assessment methods or consensus on what constitutes social impact in smart city projects leaves strategic goals open to interpretation by practitioners who, together with pressure “to transform tacit, working knowledge into explicit, measurable (downloadable) factors of organisational performance and then further reduce it to KPIs” [52] (p. 881), allows for biases and performativity in organisations.

Although it may seem obvious, social impact evaluation must prioritise human over technological and corporate goals [11,32]. By doing this, it can help to make visible the variety of stakeholders and experiences of smart city projects. Moreover, by assessing evaluation results in the wider context of city strategies and related smart city projects, further relationships can be revealed, for example, between sustainability and social impact [4,11]. In turn, this will challenge assumptions that smart city technologies automatically improve QOL. As noted earlier, social impact in smart cities may be negative or positive. Recent research has highlighted the tensions between fear and adoption of AI technology and autonomous vehicles [39], and factors contributing to AI anxiety [40]. These kinds of considerations must be taken into account in future development, implementation and impact assessment of citizen-centred, smart city projects.

5.3. Criteria for Social Impact Evaluation in Smart Cities

In summary, we propose the following criteria for social impact evaluation in the context of smart cities. There is a need for social impact evaluation that:

• Recognises the diversity of stakeholders (human and non-human) and power dynamics involved, paying special attention to those mediated through technology and those represented by government and corporate interests.
• Accounts for different understandings and experiences of social impact beyond existing smart city indicators and strategic priorities.
• Is explicit about how “social” is defined in smart city initiatives and how its impacts can be observed.
• Makes apparent the links between social and environmental sustainability.
• Assesses project development processes and their values (e.g., community engagement, inclusion, living lab principles).
• Accounts for complexity, including the relationships between local and wider project contexts (e.g., city strategies, regional ecosystems).
• Is focused, measurable and appropriate to the scale of the project and to the resources and capacity of the evaluators.
• Uses qualitative and quantitative measures when relevant, rather than trying to adapt strategic goals, reporting requirements and broad smart city frameworks to individual projects.
• Reports on impact in ways that are meaningful to stakeholders beyond governments, funders and other decision makers.

5.4. Approaches to Social Impact in Smart City Projects

While this article is concerned with the evaluation of social impact in smart cities, we also observed that social impact is not always the primary aim of smart city projects. Bringing together the eight projects and the literature reviewed, we observe the following range of approaches in which social impact is:

• Absent. Social impact is not considered in the project or evaluation.
• Part of the aims but not the evaluation. Social impact is part of general project aims, but there are no social impact indicators or evaluation.
• Part of the aims and part of the evaluation. Social impact is part of general project aims, and social impact indicators and evaluation methods are used for some aspects.
• A natural part of the evaluation. Social impact is evaluated for all aspects no matter what the project aims are.
• The main aim of the project or process and the evaluation. Social impact is the main goal of the entire project (i.e., over technological, economic, environmental aims).

By categorising projects in this way, researchers and practitioners may find it easier to identify similar projects, and to consider if social impact could play a larger role in their work. As suggested, a more holistic evaluation that moves beyond the outcomes of an individual project or process evaluation could be beneficial, whether through a comparison with other smart city projects or strategies, or even between the different levels of evaluation outlined above.

6. Conclusions

Drawing on the previous literature, this article highlights the importance of evaluating social impacts of smart city technologies and services. New initiatives like the implementation of autonomous mobility systems have social implications that may influence their long-term success and adoption. Based on a review of existing smart city evaluation frameworks and an analysis of eight research-led, smart city projects, we argue that in order to evaluate social impact, the concept needs to be more clearly understood in the smart city context. Our discussion of evaluation frameworks (Section 2) draws primarily on indexes designed for the high-level (global) comparison of smart cities. We found little in-depth analysis of social impact in specific smart city projects.

However, the eight projects we analysed provide concrete examples of socially-driven smart city research and initiatives across six countries. We identified four common approaches with projects that: focus on human- and/or eco-centred work; prioritise inclusion, participation and empowerment; identify new roles for citizens and their data; and make connections between impact, social and environmental sustainability. Furthermore, we observed three themes related to assessing the social: the importance of formative evaluation and using design and development processes to evaluate smart city projects as they unfold; the challenges of creating appropriate qualitative indicators in the face of higher-level strategic goals and confusion between outcomes and impacts; and the inadequacy of quantitative indicators in the representation of social impacts. Based on our analysis of the indexes and the projects, we offer a definition of social impact in smart cities and outline five ways that social impact is approached in smart city projects. We also propose nine criteria for the evaluation of social impact that reflect the diversity of contexts, stakeholders, and qualitative and quantitative methods used in smart city projects.

We recognise several limitations and areas for further research based on this article. The first regards the selection of projects. The eight projects share an emphasis on social impact, but they are also quite varied. A more productive comparison might be achieved with projects that fall within one of the five approaches outlined above (Section 5.4) or which focus on a shared goal (e.g., affordable housing). Furthermore, we have concentrated on projects led by research institutions with support from the public sector. Most projects were undertaken in collaboration with municipalities, businesses and citizens and many go beyond prototypes to completed products, services and systems. They may represent more experimental, theoretical and research-driven approaches than projects led by the private or public sector. While the selected projects illustrate specific kinds of smart city projects, their innovative approaches to the “social” point to gaps in mainstream smart city indexes and they could have wider influence for smart city development. Future research could compare examples from the private sector to examine how indicators, evaluation methods and views of social impact differ in market-led smart city initiatives and thereby assess the generalisability of our results. Another way to advance this research would be learning from social impact assessment in other domains, such as industrial and environmental contexts, social innovation, business and development studies. Policymakers, public agencies and funders could also support this research by including the evaluation of social impact in their selection criteria and reporting requirements.

To conclude, human- and eco-centric smart city projects can engage diverse stakeholders and produce social impacts that are not accounted for in current smart city evaluation frameworks. More holistic approaches are required to assess social impact in smart city contexts. Coherent and consistent methods that balance open-ended assessment of individual projects with the need to situate those projects in relation to city strategies, funding priorities and smart city indices, are key challenges for evaluating social impact.