Next Article in Journal
Small Municipalities Attracting Rural Newcomers and Fostering Local Cohesion: Innovative Approaches for Rural Regeneration in Italy
Previous Article in Journal
Enhancing Drought Tolerance in Wheat Cultivars through Nano-ZnO Priming by Improving Leaf Pigments and Antioxidant Activity
 
 
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Multi-Level Governance and Emergency Management: Building a Priority Assessment Matrix in the Pandemic Outbreak

1
Department of Knowledge Engineering (dEGC), UFSC-Federal University of Santa Catarina, Florianopolis 88040-900, Brazil
2
Department of Engineering for Innovation, University of Salento, 73100 Lecce, Italy
*
Author to whom correspondence should be addressed.
Sustainability 2023, 15(7), 5836; https://doi.org/10.3390/su15075836
Received: 22 February 2023 / Revised: 24 March 2023 / Accepted: 25 March 2023 / Published: 28 March 2023
(This article belongs to the Section Sustainable Engineering and Science)

Abstract

:
The execution of effective responses during a large-scale emergency requires gathering appropriate information and implementing priority assessment processes. In particular, the design of comprehensive and multi-dimensional evaluation methods is crucial, especially in countries characterized by the presence of multiple levels of authority and regional governance. In this article, we illustrate the development of a structured method that was used to support emergency decisions during the COVID-19 emergency in Brazil. We applied a design science roadmap in order to produce an artifact development process, based on extensive expert involvement and multiple sources, to identify the core requirements and information needs at healthcare management and governance levels. This study provides an assessment matrix and measurement scale based on twenty indicators and five key criteria (i.e., gravity, urgency, trend, amplitude, and impact). The study provides an evidence-based method for assessing the emergency status of, and defining recommendations for, operators and policymakers. This method can be adopted for other emergency management scenarios and contexts in order to support the analysis and implementation of responses; this has a specific relevance for multi-level governance contexts.

1. Introduction and Problem Statement

Currently, a number of natural and human-related factors can generate large scale emergencies, which could potentially affect a great number of people; this could have a ruinous impact on local and global communities. Emergency management is concerned with anticipating or reducing the probability of accidents, and dealing with the repercussions of an event in order to reduce negative impacts. The interest in emergency management has significantly increased in the last years; this is a consequence of the COVID-19 outbreak, which has triggered a global movement among academics and practitioners alike in order to define more effective approaches and methods that can handle emergency scenarios.
In fact, the pandemic required countries from around the world to take effective actions that were aimed at contact tracking, social distancing, and interrupting non-essential activities (“lockdown”) to contain the infection while properly accounting for the epidemiological, health-based, economic, and social aspects of the emergency. Such actions were implemented to varying degrees, depending on the government; they either adopted softer or more draconian measures [1,2,3], and they adopted different approaches that were either “techno-driven” or “human-driven”. These measures and approaches were implemented to control the transmission of the infection [4].
The comprehensive response to the pandemic was necessary; since the first phase of the outbreak, an array of harmoniously integrated actions occurred. These actions involved coordinating responders, providing consistent information, communicating risks, and undertaking health interventions [5]. Although the pandemic has stimulated a large stream of research on the economic and social impacts of COVID-19 [6,7,8], the emergency also represented an opportunity to investigate the varied responses to the pandemic [9], and the relevance of purposeful systems to process and retrieve real-time information [10].
The complexity of an emergency increases when the management of the emergency has to cope with articulated governance models, and multiple levels of authority which require an additional coordination effort; moreover, it also requires the development of evidence-based tools that are able to support decisions and actions taken by a multitude of policy makers and emergency actors [11,12]. Information management is central to emergency management, and this was also true of the COVID-19 disaster. In fact, information and information systems fueled and facilitated the response to the COVID-19 pandemic, which affected a wide variety of stakeholders. Data analytics underpins policy and decision making, and ongoing analysis provides governments with information concerning the efficacy and impact of these policies and decisions [13].
Successfully controlling the pandemic depended on effective and integrated leadership at different levels; the lack of studies concerning the application of multi-level governance during a pandemic indicates the need for further research [14]. In order to undertake such an endeavor, this article shares a real case of evidence-based methodology development; then, it applies the method in order to assess the pandemic emergency in Brazil. The epidemiology of COVID-19 in Brazil presented major shifts, and thus, the constant evolution and changes required continuously mapping and assessing the epidemiological features [15].
This article focuses on the development of a method for assessing emergencies and defining recommendations for operators. This method can be adopted and applied to other emergency management scenarios and contexts to support the analysis and implementation of effective responses. The remainder of the work is structured as follows. Section 2 analyzes the relevant background concerning government challenges in emergency management. Section 3 presents the research that was undertaken. Section 4 and Section 5 define the emergency assessment criteria, and these sections also note how the assessment matrix and prototypal information system were constructed. Section 6 presents a discussion of the findings, it notes the limitations of this study, and provides avenues for further research.

2. Government and Emergency Management

Governments are increasingly emphasizing emergency management as a response to public emergencies that cause extensive consequences and involve multiple agents. Particularly in large and federal states, central governments co-exist with many independent regional authorities and non-governmental actors; these bodies prioritize different interests and needs, and thus, such factors may generate a misalignment in terms of actions undertaken. This is particularly true for larger countries, wherein pressures from local authorities and multilevel governments increase the complexity of emergencies in terms of standardizing measures and coordinating response actions.
In such conditions, finding a suitable governance mechanism and leadership style pose a significant challenge for emergency management and emergency services. The success of the collaboration between authorities and the governance agenda will inevitably depend on how the collaboration process is led and managed by a multitude of decision makers and emergency responders. Emergency services have been rapidly changing in response to increasing demand, resources have been reduced, and there have been increasingly complex threats to public safety. Academics and practitioners all need to have a clear understanding of the changing role and contribution of these services, and they also need to find ways to improve their professional management methods and performance so that policy solutions to new and emerging threats may be efficiently developed and effectively implemented [16,17,18].
The principles and applications of multilevel governance have been extensively studied in the literature over the last ten years. In multilevel governance structures, control and hierarchy are replaced by asymmetric, but coordinated, interactions between actors [19]. In such structures, the actors are located at different levels, and they are invited to interact collaboratively for the common good [20]. These structures are aimed at decentralizing authority, distributing responsibilities, and enabling collaboration; thus, they can provide a basis for facing dynamic and accelerated crises [19,20,21,22,23]. The recent literature has investigated adaptive governance and decentralization [24], adaptive governance in the digital context [25], the integration of top-down and bottom-up policymaking [26], and the association between multilevel governance and problem-solving [27]. Adaptive governance is aimed at enhancing the capacity of an organization to deal with, and adapt to, changes. Strategies include utilizing internal and external capabilities, decentralizing decision-making, and seeking to inform higher-level decisions from bottom-up [28].
The concept of multilevel governance was also applied to the contexts of cities and global climate change [29]; such studies addressed adaptive governance and institutional strategies for climate-induced community relocations [30], in addition to examining how multilevel governance can support the urban politics of climate change [31]. With specific reference to the COVID-19 emergency, Ref. [1] analyzed the balance of governance and legitimacy by focusing on a case in Norway. Conversely, Ref. [28] discussed agile and adaptive governance in terms of the responses to the COVID-19 emergency. Ref. [32] discussed the use of social and behavioral science to support the COVID-19 pandemic response.
Large-scale emergencies, especially pandemics, are able to significantly challenge the local and national emergency and healthcare systems. Scholars working in the information systems field, as well as decision support, provided relevant contributions to support the modelling and prediction of infectious disease, thus enhancing the overall preparedness of national systems. For example, Ref. [33] presented a multi-agent-based, semantic-driven system for making support-based decisions in epidemic management. Conversely, Ref. [34] designed a simulation-driven hybrid tabletop exercise that allows participants to respond to a hypothetical pandemic influenza scenario and make iterative policy decisions in a group setting. More recently, in a COVID-19 context, Ref. [35] studied the importance of building contingency plans to alleviate the workload of emergency departments. Ref. [36] presented a support system for demand management in healthcare supply chains that considered epidemic outbreaks, and [37] presented a COVID-19 risk assessment DSS for practitioners.
One relevant and recent stream of the literature has also started to investigate the ethical aspects associated with technology-based assessments and the management of risks. For example, Ref. [38] studied the ethics of shared COVID-19 risks, and a framework for an ethical health technology assessment of risk in vaccine supply chain infrastructures was proposed. Ref. [38] conducted an epistemological and bibliometric analysis of ethics and shared responsibility in the context of health policy and Internet of Things applications.
The main area of investigation in our paper concerns the arising complexity of emergency-related decisions and actions when the management of the emergency has to contend with articulated governance models and multiple levels of authority. This requires an additional coordination effort and the availability of evidence-based tools that are able to support policy makers and emergency actors. This area requires further research that is aimed at successfully controlling the pandemic [14].

3. Research Process and Method

The research process was based on a design science approach, which is a pragmatic research paradigm that calls for the creation of innovative artifacts to solve real problems [39]. The approach is native to the information systems and management information systems field, and thus, it could be a relevant research strategy for the development of information-related artifacts for governance-related endeavors.
The framing of a problem, the effort required to build a concept, and the proposed solutions to a problem, are characterized by elements of innovativeness in that the process is based on the identification and integration of multiple sources of data and the expertise found in the Brazilian healthcare system. The study was coordinated by the Laboratory for Integration Engineering and Knowledge Governance (ENGIN) of the Federal University of Santa Catarina (UFSC). The Laboratory has consolidated experience in fields such as governance models and approaches, strategies, and tools for innovative knowledge management. The research project involved a large number of institutional and scientific actors and experts from the state of Santa Catarina, as well contributions from representatives of the national Brazilian government. The authors directly engaged with activities concerning the observation, expert involvement, data acquisition, and elaboration of the proposed methodology. Figure 1 shows the activities undertaken, in four main steps, which are in accordance with the adopted design science approach.
In the first step, “Problem and Goals Definition”, the context of the problem is recognized and formalized, and the main objectives, in terms of addressing the problem, are outlined. In our study, the key goal was to build an evidence-based and expert-validated assessment method to support the emergency assessment and the decisions and actions made by policy makers and health leaders. A specific aim of this study was to define a solution that would be able to address the challenge of standardizing the information gathering and measurement processes in a scenario wherein multilevel governance and multi-stakeholder coordination was required.
Next, “Artifact Development” occurred, which is the central activity of the design science process; here, the conceptual definition of the proposed solution is defined and implemented in order to address the identified problem. In our study, the development process included four sub-steps (i.e., “Health Process Analysis”, “Emergency Criteria Definition”, “GUTAI Matrix Building”, and “Integrated System Development”). Different methods were used to conduct the artifact development effort, as follows: interviews with key informants, experts, and specialists (e.g., Ministries and State Secretariats); bibliographic research (mostly governance, decision support, and emergency management literature); document analysis (e.g., WHO reports); collaboration between groups; and participant observation.
A dialogue was established between multiple public and private actors to gather and analyze relevant epidemiological data and models, and to define the key parameters for building the assessment methodology. The construction of the dataset and access to the relevant anonymized database was realized after collaborating with the integration teams at the Computer and Automation Center of the State of Santa Catarina (CIASC). Leaders of other health system data intelligence projects were also consulted. Table 1 shows the large panel of actors that were involved in building the information model and the tool building process, along with the main role that they played (i.e., whether they provided information, expertise, and/or feedback).
The researchers’ team interacted with different stakeholders that provided expertise on data analytics, epidemiology and health data, health policy, modelling and methodology design, socio-demographic dimensions, and security policies. Such interactions were mainly facilitated by sharing a power point presentation containing the main aims and objectives of the study undertaken by the researchers.
By conducting online or telephone meetings with key actors, the researchers obtained the stakeholder’s point of view on the pandemic and a number of key challenges and perspectives concerning how to deal with the outbreak. The meetings involved purposeful brainstorming sessions, which allowed for the drafting of a conceptual map of key dimensions that are concerned with the monitoring of the pandemic. Such initial requirements, concerning data collection and contextual analysis, prompted the researchers to generate a draft version of the GUTAI Matrix; this was sent via e-mail to all the stakeholders to gather further insights and suggestions for improvement. Once the suggested changes were implemented, the researchers obtained a second version of the Matrix.
The value that was added and obtained from the aforementioned interaction process was greater than that which is usually obtained from traditional design and development processes. First, the collective intelligence generated by the large number of stakeholders allowed us to gather multi-dimensional views of the health emergency. This is crucial when dealing with a complex issue such as COVID-19. Moreover, such an approach allows for increased analytic capacity, which is required for engaging in reflexive anticipatory policy design [40]. Second, in this instance, the role of facilitators and knowledge aggregators is played by the researchers rather than direct artefact developers; this allowed us to reduce the risks that are inherent to academic and self-referential approaches, although, regarding real-life societal issues, this aspect of the process may be limited.
The third step of the design process, “Artifact Demonstration and Evaluation”, aimed to show the real-life usability and potential benefits of the defined solution, and to collect feedback for improving the solution. In our study, a demonstration session was performed with the municipalities participating in the State’s Online Situation Room; notably, the largest municipality in the State of Santa Catarina, Joinville, participated in this session. This session was conducted using the available data (as of 31 May 2020); as a result, it was possible to obtain an evaluation of emergency indicators, a definition of an aggregated index, and a formulation of recommendations for city leaders.
In the design science roadmap, the main research process and the obtained findings were reported during the “Research Communication” step, wherein the solution was shared with the relevant scholarly and practitioner community. With this in mind, the present article represents the first scientific publication, at an international level, to note the activities undertaken and the results obtained. Phase 1 provides the introduction and background to this study. The communication phase (4) is formalized by this research article. The next three sections provide a detailed analysis of the artifact development and application processes (i.e., steps 2 and 3).

4. Healthcare Process Analysis and Emergency Criteria Definition

One of greatest challenges, and one of the most critical issues, facing the pandemic is the negative impact on the healthcare system, which is generally unprepared for the substantial increase in sick individuals. The first step in the artifact development process was thus to investigate how the healthcare process is managed within the national Brazilian system, with a specific focus on the pandemic scenario and the generation and gathering of critical information to be analyzed. This process includes seven core activities.
First, a pre-analysis (step 1) of the available data on contamination, as well as confirmed and suspected cases, is conducted, followed by the generation and delivery (step 2) of health system information (mostly on patients and health units) to government officials. Triaging (step 3) then aims to assess cases, and clinical and anamnesis data are generated and gathered (step 4) by health professionals. Next, exams and tests (step 5) are performed to define/validate the number of positive and negative individuals. Finally, hospitalization occurs to provide treatments to infected individuals (step 6), and the system is updated (discharge) when individuals recover from the disease or die (step 7). Figure 2 shows a visualization of this process, which was modelled by the researchers using the Bizagi Modeler®, a tool used to create BPMN standard process models.
The sequence of the identified activities generates a large set of relevant health data. Using interviews with medical experts, nurses, and hospital personnel, as well as with technology and data analysis specialists, the main information sources were identified and described. The information systems used by hospitals, health departments, and health laboratories were also studied in order to map the relevant information channels. Based on the process and flow analyses, it was possible to identify the data and critical information that was generated and used at each stage; moreover, it was also possible to assess their quality (mostly in terms of accessibility, reliability, and timeliness). Brainstorming sessions were also conducted with health and public managers with the purpose of composing a framework for major emergency-related criteria and indicators.
Method engineering and quality control disciplines propose different approaches and tools for facilitating prioritization between multiple criteria, such as the Pareto Diagram, Ishikawa Diagram, MUGDGE Matrix, SWOT Analysis, Eisenhower Matrix, and the GUT Matrix. In particular, the GUT Matrix allows for the identification of priorities in a system based on the analysis of Gravity, Urgency, and Tendency. The relevance of the GUT Matrix, in relation to the other tools applied to facilitate prioritization, is that it does not deviate from: (1) addressing the seriousness of a problem and how it affects people in the short- and long-term; (2) accounting for the severity of the problem and how long it will take for the problem to be resolved, so as to not aggravate the situation further; and (3) focusing on the potential of the problem to grow and worsen. These features of the GUT Matrix make it specifically relevant for application in a pandemic scenario, which exhibits time-sensitive, critical issues that must be effectively addressed. Hence, it is not only crucial to support to the judgment stage of the decision-making process, but also to enhance the decision itself by robustly recognizing the problem and diagnosing the situation. Moreover, reliable data which supports the elaboration and analysis of the propositions should also be used in order to generate recommendations. This would enable the identification of a potential structure that could make “unstructured decisions” [41].
In addition to the GUT proposition, and based on a previous study [42], five criteria were identified as being relevant for the assessment of the COVID-19 emergency: (1) Gravity; (2) Urgency; (3) Trend; (4) Amplitude; and (5) Impact. The reason for including two new parameters in the GUT framework was to obtain a more comprehensive decision support framework. The design science approach adopted by the researchers, including their interactions with a multitude of stakeholders, allowed us to capture multiple dimensions for evaluation in pandemic scenarios; therefore, a new parameter was discussed, validated by stakeholders, and included to account for the spatial or geographical scope of the analyzed problem (the diffusion of COVID-19 in cities and regions). Moreover, one parameter was integrated in order to address the scale and scope of the impact caused by the outbreak (which is a relevant aspect of network analysis).
The Gravity criterion refers to the saturation of beds, with particular reference to the number of ICU (Intensive Care Unit) COVID-19 beds available in the public and private health systems, and the percentage of beds that were occupied. COVID-19, in its most severe forms, requires treatment in ICUs for a period of 15 to 21 days; this causes a slow rotation of beds. It is thus crucial to monitor the occupation rate to prevent saturation and to develop more effective preventive measures while enhancing the system’s operational capacity. The Urgency criterion considers the growth of confirmed and suspected cases. This can, in turn, increase the occupation of clinical beds. This criterion is thus crucial for ascertaining a preliminary view of the situation. Knowing the occupancy rate of clinical beds allows us to monitor the progress of the disease and the stress on ICU facilities; thus, it indicates the urgency with which actions need to be taken in order to inhibit the proliferation of the disease. The Trend criterion refers to a growth rate, which assesses the total number of new confirmed cases in the last seven days, as compared with the total number of cases from the previous seven days. This criterion is important for evaluating the number of infected people, the spread of the disease, and the capacity to detect them.
The Amplitude criterion aims to analyze contamination in terms of potent diffusion in the surrounding regions. Based on such evidence, a number of actions may be taken, such as alerts and guidelines for the population, travel limitations, and the activation of specific monitoring plans. Finally, Impact refers to the relation between the number of deaths and the number of confirmed cases. Although the lethality rate is not an accurate measure of the real impact of the disease as it offers a ‘late’ view of the problem, it shows the speed of viral transmission, and it indicates possible issues in the healthcare system. In addition, the lethality parameter adds value to the decision support system in that a high number of deaths are not desirable in a healthcare system, and it can also have negative effects on the physical and mental health of healthcare professionals. Figure 3 summarizes the Gravity, Urgency, Trend, Amplitude, and Impact criteria, and the associated key indicators.
In order to quantify the four criteria, the value of the related indicators were obtained from the different information systems that have been adopted in the Brazilian healthcare management process. Figure 4 shows the architecture of the governmental information sources and systems (with related links) that were integrated into a comprehensive information platform (Boavista). The integrated platform provides data related to many different indicators that are needed to measure GUTAI criteria and to obtain five critical “indexes” as well as an aggregated GUTAI Index.
The GUTAI Index is a weighted average of the five measures associated with the identified GUTAI criteria (Gravity, Urgency, Trend, Amplitude, and Impact). It represents a synthetic value that is useful for the governance of a region or territory. The interpretation of the Index should take into account the peculiarities of the socio-demographic and epidemiological context, and it should look at the five determinants of the single synthetic value to understand the most urgent and critical aspects that require management (e.g., high lethality rate, saturation of ICU beds, etc.).

5. GUTAI Matrix Building and Integrated System Development

The health management process was studied to identify relevant information sources and systems, and to define key indicators and criteria for assessing the status of an emergency. The next step was the operationalization of those criteria; this was achieved by defining a measurement scale and a number of policy implications derived from that scale. The scale’s values were defined by the Brazilian Intensive Medicine Association [43], and they were based on international standards that were generated as a result of, and in accordance with, the impact of the pandemic in countries such as Italy and Spain. In our study, the standards suggested by BIMA (2020) were considered for use in the Brazilian context; here, we assessed the parameters defined by the state of Santa Catarina. Table 2 reports the findings of the GUTAI Matrix (i.e., the five criteria with their associated respective indicators, along with a five-level measurement scale).
In order to support and prioritize governance decisions, the five-level scale was translated into specific emergency management recommendations. In particular, the five-level scale allows us to determine numerical values for the five GUTAI criteria and to determine the aggregated GUTAI Index; therefore, the Index, which is a weighted average of the five values, can range from 1 to 5, thus implying different levels of emergency management and response strategies/priorities.
If the Index is at level 1, a Planning recommendation can be formulated, as there is still time to plan proper response strategies and actions. At level 2, Situation Control, the disease is advancing, and emergency metrics should be carefully monitored. At level 3, Attention, the emergency is worsening, and attention is required to manage the effects of the outbreak. At level 4, Quick Action, the situation is worsening, and fast decisions and effective measures are required. Finally, at level 5, Decision, the situation is serious and potentially out of control, and thus, urgent decisions need to be made. As the level increases, the actions needed to improve the operational capacity of the healthcare system require a higher priority in order to address the physical and mental health of emergency professionals; moreover, they are also required so that restrictive measures can be applied to control the spread of COVID-19. Strategies can range from different types of quarantine, social isolation, conduct guidelines, and suspensions of activities (lockdown).
Although this preliminary development effort should be the focus of future research, a prototypal web platform was drafted to support regional governance through a user-friendly graphical interface that is able to provide real time data by city, by date, and by GUTAI criteria (a submenu shows Gravity, Urgency, Trend, Amplitude, and Impact data). The prototype aims to provide different functionalities. First, the platform would offer a multi-criteria analysis of GUTAI parameters and their impact on the final GUTAI Index. The platform shows the value of the five parameters in terms of the defined unit of analysis (e.g., a city), and it illustrates the extent of the impact that each parameter has on the aggregated GUTAI value. Second, the system could provide decision makers with a recommendation tool that is able to outline suggested actions that should be prioritized, based on the obtained value of the Index. Specific types of actions are thus associated with specific ranges of the GUTAI Index. Finally, the system is intended to include a simulator function that is useful for estimating the multi-dimensional impact of given policy actions (e.g., suspension of commercial activities) on the aggregated Index and on single parameters.
Figure 5 shows the interface of the web platform. It is important to note that the focus of the research process (and of this article) should be on the activity of designing and building a method to support the decision-making process. The design and development of the prototypal platform, which can operationalize the method in this paper, is thus an additional finding of this paper; moreover, it should be the specific object of further research.
We applied the GUTAI Matrix, and the system for validation, to the Municipal Health Department of Joinville (see Figure 6), which is the largest municipality in the state of Santa Catarina. Located in the northern region of the country, Santa Catarina has a UNDP Human Development Index (HDI) of 0.808 (in 2017), the third highest in Brazil after the Federal District and São Paulo. Key information on COVID-19 was obtained from the Health Secretariat of Joinville and the Secretariat of Health of the State of Santa Catarina. Data were also directly gathered from the management information systems of hospitals and clinics, and they were processed by the Computer Center of the State of Santa Catarina using the Boavista platform. The application of the Matrix was conducted using data from 31 May 2020.
The set of recommended measures for the levels “Planning” and “Control” can be categorized as relating to “Quarantine 1” or “Quarantine 2”. “Quarantine 1” implies that, although the infection is still limited, a pandemic committee should be gathered and a prevention plan is required, especially if suspected or proven cases have been identified in the municipality or the surrounding area. Cases should be isolated during the virus incubation time (14 days for COVID-19) in their own residence, in accommodations provided by the State, or specific health units should be provided to ensure effective isolation. “Quarantine 2” implies that COVID-19 is not yet overloading the health system. In addition to the first stage, the management health system and processes should be further checked and strengthened; remote working is recommended, face-to-face classes and major events should be suspended, and public transportation should be limited to 50% of its capacity. Individuals are required to wear protection masks and to keep a distance of 2 m.
The set of recommended measures for the levels “Attention” and “Quick Actions” can be labelled as “Social Distancing” and “Temporary Social Isolation”. “Social Distancing” is justified by the fact the pandemic is in the municipality area, and there is a serious risk of accelerating contamination. The most critical urban areas should be immediately identified, and the control of social behavior should be intensified. In addition to the suggested measures in the previous stages, recommendations include the prohibition of agglomerations, meetings, and physical events/celebrations. Finally, “Temporary Social Isolation” should be required if there are serious risks of the disease exponentially spreading. In this case, for activities in which it is not possible to ensure social distancing, they should be suspended. Moreover, commercial, industrial, and service-based activities should be interrupted, except for those considered essential for the public good (e.g., food production and delivery). Institutional communication should be intensified to increase public awareness of the state of emergency, and specific attention should be dedicated to protecting the physical and mental wellbeing of health professionals.
Infection restriction measures were based on general protocols defined by most local regions and from countries across the world; however, the adoption of the Matrix provided the referents of Joinville with an evidence-based tool to better inform decisions and actions. Further research will have to provide new forms of validation methods (including experiments). The current version of the framework is based on expert group modelling and multiple sources, and thus, from this perspective, it can have a distinctive value. The distinctive findings and insights generated from this study are discussed in the following section, both in terms of theory contribution, and in terms of practitioner value.

6. Discussion and Conclusions

In this article, we have presented an articulate research process that is aimed at developing a method and tool to support emergency assessments and services by using the COVID-19 scenario in Brazil. The main assumption of this study is that the implementation of effective responses during a large-scale emergency require appropriate information gathering and priority assessment processes and tools. In particular, the design of multi-dimensional evaluation methods is crucial, especially in large countries that are characterized by the presence of multiple levels of authority and regional governance. This study provides a number of implications in terms of theory and for practitioners, which are described in the next sections.

6.1. Theory Contributions

The complexity of emergency management requires effective coordination and the development of evidence-based tools that are able to support decisions and actions taken by a multitude of policy makers and emergency actors [11,12]. Information management is crucial for emergency management, and this was also true of the COVID-19 pandemic in Brazil. Successful control over the pandemic depends on effective and integrated leadership at different levels; the lack of studies on the application of multi-level governance during a pandemic indicates the need for further research [14].
The interest in emergency management has strongly increased in recent years; this is a consequence of the COVID-19 outbreak, which has triggered a global movement among academics and practitioners to define more effective approaches and methods that can handle such emergency scenarios. This paper makes the following contribution: it develops the process of information gathering, and thus, it contributes to the development of decision support systems, which are aimed at streamlining or supporting emergency response systems [10]. The developed approach has recently been advocated in the literature as being critical for emergency and epidemic management (e.g., [33,34,37]).
Response actions and policies adopted by global governments to address the pandemic have ranged from soft to draconian [1,2,3], and they have included a range of “techno-driven” and “human-driven” approaches [4]; however, the presence of different levels of governance and multiple perspectives drive the need to define multi-dimensional criteria to assess the status of the emergency, and such criteria should be used to support decision making at the local and regional levels, which includes the potential use of adaptive governance strategies and decentralized decision-making (e.g., [28]).
Regardless of the response strategy, gathering appropriate information and emergency status assessment methods are required to provide health and emergency managers with the critical intelligence needed to orient decisions and actions. In this article, we have presented a stakeholder-driven analysis of how pressures on local authorities and multilevel governance increase the complexity of the emergency in terms of standardizing measures and coordinating responses. The extent to which it is possible to tackle and recover from the pandemic is based on the capacity of governance systems to consider the diverse actors that are involved at multiple levels; moreover, the complexity and diversity of actions required to generate effective results must also be accounted for [23]. Regarding the case of Brazil, the results provide food for thought in terms of the academic discussion concerning complex emergencies and how they can be effectively addressed with large institutional and governmental endeavors that use purposeful analytical information systems.
Another academic contribution of this paper concerns the facilitation of interactions between stakeholders which aim to make better decisions for the public good. Emergencies are able to cause extensive damage, and they might involve multiple actors. Particularly in large, federal states, central governments co-exist with many regional authorities and non-governmental actors, which have different interests and needs. Multilevel governance comprises actors located at different levels, and they are invited to interact collaboratively for the common good [20]; thus, this approach can provide a basis for addressing critical scenarios.
This study has shown the development of an evidence-based methodology, and it has application for assessing the pandemic emergency in Brazil. It focuses on finding a method for assessing emergencies and defining recommendations for operators. The background analysis and development of a methodology, which is central to the GUTAI project, has allowed us to gather new knowledge on the open approaches to decision support based on heterogeneous stakeholder participation and multi-actor involvement. Different strategies and methods have been proposed in the literature to facilitate a shared understanding of complex structures and relationships that determine the behavior of a given system. One relevant approach is represented by group model building (e.g., [44,45,46]), which is a participatory approach to understanding problems. It increases engagement in systems-based thinking, and it develops a consensus for action among a diverse group of stakeholders. In light of this, the GUTAI approach thus provides new insights into the relevance of collaborative planning and multi-stakeholder engagement; it provides more robust approaches that can be adopted by policy makers, emergency actors, and agencies at different levels of governance.

6.2. Practitioner Implications

Numerical results are not a specific outcome of this paper, but rather, a practitioner-based development in the methodology, and a preliminary application/validation of the methodology is required in selected regions/cases. The main practitioner-based aims of our article are represented by emergency managers, responders, and policy makers. The utility of the proposed framework can be envisioned in accordance with a number of applicative avenues. Moreover, it is a model that can be further elaborated upon, and it can adapt to different situations, as follows: (1) refining the decision-making model and structuring the details of the decisions and outcomes; and (2) the model can be applied to non-pandemic situations, which would verify its potential for generalization.
The developed GUTAI Matrix provides a practical tool that can support emergency governance based on a multi-criteria analysis of the emergency, using the following criteria: gravity, urgency, trend, amplitude, and impact. The continuous acquisition of rich and reliable information is essential to health surveillance; hence, the information architecture that the GUTAI Matrix is based on comprises multiple government and healthcare information systems. These systems are integrated in order to provide a holistic assessment of the health system’s ability to absorb and recover from the outbreak. Integrating resilience-based ideas into disease control and prevention can prompt governments to make more proactive and comprehensive decisions in order to protect the health of communities [47].
The approach and the tool presented in this article represent an attempt to give health managers and policy makers a tool that (if validated by the central government) may provide a way to standardize information collection and processing; moreover, it provides a set of proper response guidelines. The systems-based approach to analysis can support the development of decisions and control tools for policymakers in pandemic scenarios, and the assessment method can be adopted for use in other emergency management scenarios and contexts.
Systems analysis and systems-based strategies have great potential for addressing critical global issues, and they can also guide policy decisions by drawing on innovative methodologies, models, and tools. Although the established approaches to analysis and policy are heavily based on the separation of complex realities into specialized disciplines and views, systems-based approaches offer a more-integrated perspective and a number of proven concepts, tools, and methods that can improve our understanding of complex systemic issues [48]. The use of multi-dimensional and multi-stakeholder discussions is in line with a new open approach to developing public policies; for instance, it has been undertaken in the UK government initiative, UK Policy Lab (https://openpolicy.blog.gov.uk/what-is-open-policy-making/ (accessed on 1 February, 2023)).
The GUTAI Matrix and the research process were created to provide guidelines for practitioners; thus, they represent an interesting example of a collective intelligence exercise conducted in Brazil. This practitioner-led case is useful for the design of, and conducting, additional initiatives that can be applied to similar problematic scenarios that are based on the adoption of a systems-based approach that can identify the dimensions of evaluation and contributing stakeholders. The framework development process took a large number of dimensions and elements of investigation into account; moreover, it aimed to leverage a multi-stakeholder and collective intelligence approach. Such an approach represents an interesting and replicable example of a public-led exercise which can analyze societal problems; this approach aims to develop a more robust definition and comprehensive identification of evaluative parameters. This study also contributes to the investigation of the importance of cross-agency networking and information accessibility, which can affect the efficiency of emergency collaborations [49].

6.3. Concluding Remarks

Information management and emergency assessments were central to the COVID-19 emergency; in particular, it fueled and facilitated the response to the pandemic, which affected a wide range of stakeholders [13]. Although the GUTAI Matrix does not provide prescriptive indications, the priority assessment approach and the matrix can be useful for supporting emergency management in a multi-level governance setting. Moreover, the feedback obtained from Brazilian authorities was positive, and the tool was considered a relevant asset for analyzing data concerning the pandemic crisis. In the words of the health secretary of Joinville, Mr. Jean Rodrigues da Silva, “the GUTAI Matrix allows [us] to standardize and provide greater transparency [on the pandemic] so that everyone…know[s] where the pandemic is heading. Based on the analyzed criteria and indicators, the governance of the municipality can open or loosen social distancing and sanitary measures, [depending on how] the spread of the new coronavirus behaves”.
This study is not without limitations. First, the GUTAI Matrix is not an epidemiological model, but rather, a tool to support decision-making and emergency responses. Although the presented approach is not directly based on epidemiological models, the GUTAI Matrix can provide a complementary contribution in that it represents an integrative parameter classification and correlation framework, which can be enriched with purposeful models and health management data.
The developed tool is this not appropriate for cross-country comparative analyses, but it may be considered for interstate and inter-municipal comparisons. Second, although the five GUTAI criteria can be considered for the control of the COVID-19 disease in the analyzed territory, they rely on access to qualified data, the process of adaptation (e.g., weights for the criteria), and the integration of new indicators and metrics which become available in successive phases. The potential for the generalization of the defined approach in other geographic contexts and emergency management scenarios should be evaluated based on the identification of more general requirements (valid for every context and emergency management scenario), and specific requirements or conditions which apply to peculiar situations. The validation of this approach thus represents a core direction for future research; it also indicates that the application of this model may be extended to other experimental contexts.

Author Contributions

Conceptualization, P.d.S.F.; methodology, P.d.S.F. and F.K.-M.; investigation, P.d.S.F., F.K.-M. and A.M.; data curation, P.d.S.F., F.K.-M. and A.M.; writing—original draft preparation, P.d.S.F., F.K.-M. and A.M.; writing—review and editing, A.M. All authors have read and agreed to the published version of the manuscript.

Funding

Capes/EGC/UFSC: n° 202004266 of 11/03/2020.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Christensen, T.; Lægreid, P. Balancing governance capacity and legitimacy-how the Norwegian government handled the COVID-19 crisis as a high performer. Public Adm. Rev. 2020, 80, 774–779. [Google Scholar] [CrossRef] [PubMed]
  2. Moon, M.J. Fighting against COVID-19 with agility, transparency, and participation: Wicked policy problems and new governance challenges. Public Adm. Rev. 2020, 80, 651–656. [Google Scholar] [CrossRef] [PubMed]
  3. Zavattaro, S.M.; Hall, J.L.; Battaglio, R.P.; Hail, M.W. Introduction: COVID-19 Viewpoint Symposium, Part II. Public Adm. Rev. 2020, 80, 755. [Google Scholar] [CrossRef]
  4. Kummitha, R.K.R. Smart technologies for fighting pandemics: The techno-and human-driven approaches in controlling the virus transmission. Gov. Inf. Q. 2020, 37, 101481. [Google Scholar] [CrossRef]
  5. WHO—World Health Organization. Managing Epidemics: Key Facts about Major Deadly Diseases; World Health Organization: Geneva, Switzerland, 2018.
  6. Gharehgozli, O.; Nayebvali, P.; Gharehgozli, A.; Zamanian, Z. Impact of COVID-19 on the economic output of the US outbreak’s epicenter”. Econ. Disasters Clim. Chang. 2020, 4, 561–573. [Google Scholar] [CrossRef] [PubMed]
  7. Kim, R.Y. The Impact of COVID-19 on Consumers: Preparing for Digital Sales. IEEE Eng. Manag. Rev. 2020, 48, 212–218. [Google Scholar] [CrossRef]
  8. Martin, A.; Markhvida, M.; Hallegatte, S.; Walsh, B. Socio-economic impacts of COVID-19 on household consumption and poverty. Econ. Disasters Clim. Chang. 2020, 4, 453–479. [Google Scholar] [CrossRef]
  9. Zeemering, E.S. Functional fragmentation in city hall and Twitter communication during the COVID-19 Pandemic: Evidence from Atlanta, San Francisco, and Washington, DC. Gov. Inf. Q. 2020, 38, 101539. [Google Scholar] [CrossRef]
  10. Gasmelseid, T.M. Improving emergency response systems through the use of intelligent information systems. J. Intell. Inf. Technol. 2014, 10, 37–55. [Google Scholar] [CrossRef][Green Version]
  11. Liu, Z.; Guo, J.; Zhong, W.; Gui, T. Multi-level governance, policy coordination and subnational responses to COVID-19: Comparing China and the US. J. Comp. Policy Anal. Res. Pract. 2021, 23, 204–218. [Google Scholar] [CrossRef]
  12. Arkorful, V.E. Unpacking the blackbox of responsible pandemic governance: Of COVID-19, multilevel governance and state capacity in Ghana–A Review. Public Organ. Rev. 2022, 1–17. [Google Scholar] [CrossRef]
  13. Ågerfalk, P.J.; Conboy, K.; Myers, M.D. Information systems in the age of pandemics: COVID-19 and beyond. Eur. J. Inf. Syst. 2020, 29, 203–207. [Google Scholar] [CrossRef]
  14. Santos, N.D.A. Multi-level governance tackling the COVID-19 pandemic in China. Rev. de Adm. Pública 2021, 55, 95–110. [Google Scholar] [CrossRef]
  15. Aragão, D.P.; Junior, A.G.D.S.; Mondini, A.; Distante, C.; Gonçalves, L.M.G. COVID-19 Patterns in Araraquara, Brazil: A Multimodal Analysis. Int. J. Environ. Res. Public Health 2023, 20, 4740. [Google Scholar] [CrossRef]
  16. Wankhade, P.; Patnaik, S. Conclusion: Collaboration and Governance: It’s Very Much about ‘Process’ and ‘People’! In Collaboration and Governance in the Emergency Services; Palgrave Pivot: Cham, Switzerland, 2020; pp. 127–141. [Google Scholar]
  17. Wankhade, P.; Patnaik, S. Collaboration and Governance in the Emergency Services; Springer International Publishing: Berlin/Heidelberg, Germany, 2020. [Google Scholar]
  18. Wankhade, P.; Murphy, P. Emergency Services Management: A Research Overview; Routledge Publishing: Enaland, UK, 2023. [Google Scholar]
  19. Budd, L.; Sancino, A. A framework for city leadership in multilevel governance settings: The comparative contexts of Italy and the UK, Regional Studies. Reg. Sci. 2016, 3, 129–145. [Google Scholar] [CrossRef]
  20. Alcantara, C.; Broschek, J.; Nelles, J. Rethinking multilevel governance as an instance of multilevel politics: A conceptual strategy, Territory, Politics. Governance 2016, 4, 33–51. [Google Scholar]
  21. Henrichs, J.A.; De Meza, M.L.F.G. Multilevel governance in regional development: A case study of the Intermunicipal Consortium of Frontier. Urbe Rev. Bras. de Gestão Urbana 2017, 9, 124–138. [Google Scholar] [CrossRef][Green Version]
  22. Tortola, P.D. Clarifying multilevel governance. Eur. J. Political Res. 2017, 56, 234–250. [Google Scholar] [CrossRef]
  23. Sandström, A.; Söderberg, C.; Lundmark, C.; Nilsson, J.; Fjellborg, D. Assessing and explaining policy coherence: A comparative study of water governance and large carnivore governance in Sweden. Environ. Policy Gov. 2020, 30, 3–13. [Google Scholar] [CrossRef][Green Version]
  24. Hong, S.; Lee, S. Adaptive governance and decentralization: Evidence from regulation of the sharing economy in multi-level governance. Gov. Inf. Q. 2018, 35, 299–305. [Google Scholar] [CrossRef]
  25. Wang, C.; Medaglia, R.; Zheng, L. Towards a typology of adaptive governance in the digital government context: The role of decision-making and accountability. Gov. Inf. Q. 2018, 35, 306–322. [Google Scholar] [CrossRef]
  26. Homsy, G.C.; Liu, Z.; Warner, M.E. Multilevel governance: Framing the integration of top-down and bottom-up policymaking. Int. J. Public Adm. 2019, 42, 572–582. [Google Scholar] [CrossRef][Green Version]
  27. Thomann, E.; Trein, P.; Maggetti, M. What’s the problem? Multilevel governance and problem-solving. Eur. Policy Anal. 2019, 5, 37–57. [Google Scholar] [CrossRef][Green Version]
  28. Janssen, M.; van der Voort, H. Agile and adaptive governance in crisis response: Lessons from the COVID-19 pandemic. Int. J. Inf. Manag. 2020, 55, 102180. [Google Scholar] [CrossRef]
  29. Betsill, M.M.; Bulkeley, H. Cities and the multilevel governance of global climate change. Glob. Gov. Rev. Multilater. Int. Organ. 2006, 12, 141–160. [Google Scholar] [CrossRef][Green Version]
  30. Bronen, R.; Chapin, F.S. Adaptive governance and institutional strategies for climate-induced community relocations in Alaska. Proc. Natl. Acad. Sci. USA 2013, 110, 9320–9325. [Google Scholar] [CrossRef] [PubMed][Green Version]
  31. Bulkeley, H.; Betsill, M. Rethinking sustainable cities: Multilevel governance and the ‘urban’ politics of climate change. Environ. Politics 2005, 14, 42–63. [Google Scholar] [CrossRef]
  32. Van Bavel, J.J.; Baicker, K.; Boggio, P.S.; Capraro, V.; Cichocka, A.; Cikara, M.; Crockett, M.J.; Crum, A.J.; Douglas, K.M.; Druckman, J.N.; et al. Using social and behavioural science to support COVID-19 pandemic response. Nat. Hum. Behav. 2020, 4, 460–471. [Google Scholar] [CrossRef]
  33. Li, S.; Mackaness, W.A. A multi-agent-based, semantic driven system for decision support in epidemic management. Health Inform. J. 2015, 21, 195–208. [Google Scholar] [CrossRef]
  34. Araz, O.M.; Jehn, M. Improving public health emergency preparedness through enhanced decision-making environments: A simulation and survey based evaluation. Technol. Forecast. Soc. Chang. 2013, 80, 1775–1781. [Google Scholar] [CrossRef]
  35. Cao, Y.; Li, Q.; Chen, J.; Guo, X.; Miao, C.; Yang, H.; Chen, Z.; Li, C.; Li, L. Hospital emergency management plan during the COVID-19 epidemic. Acad. Emerg. Med. 2020, 27, 309–311. [Google Scholar] [CrossRef][Green Version]
  36. Govindan, K.; Mina, H.; Alavi, B. A decision support system for demand management in healthcare supply chains considering the epidemic outbreaks: A case study of coronavirus disease 2019 (COVID-19). Transp. Res. Part E Logist. Transp. Rev. 2020, 138, 101967. [Google Scholar] [CrossRef] [PubMed]
  37. Liu, Y.; Wang, Z.; Ren, J.; Tian, Y.; Zhou, M.; Zhou, T.; Ye, K.; Zhao, Y.; Qiu, Y.; Li, J. A COVID-19 Risk Assessment Decision Support System for General Practitioners: Design and Development Study. J. Med. Internet Res. 2020, 22, e19786. [Google Scholar] [CrossRef]
  38. Radanliev, P.; De Roure, D. Epistemological and bibliometric analysis of ethics and shared responsibility—Health policy and IoT systems. Sustainability 2021, 13, 8355. [Google Scholar] [CrossRef]
  39. Hevner, A.R.; March, S.T.; Park, J.; Ram, S. Design science in information systems research. MIS Q. 2004, 28, 75–105. [Google Scholar] [CrossRef][Green Version]
  40. Auld, G.; Bernstein, S.; Cashore, B.; Levin, K. Managing pandemics as super wicked problems: Lessons from, and for, COVID-19 and the climate crisis. Policy Sci. 2021, 54, 707–728. [Google Scholar] [CrossRef]
  41. Mintzberg, H.; Raisinghani, D.; Theoret, A. The structure of” unstructured” decision processes. Adm. Sci. Q. 1976, 21, 246–275. [Google Scholar] [CrossRef][Green Version]
  42. Freire, P.S.; Kempner-Moreira, F.; Izidório, G. Matriz GUTAI_COVID-19: Apoio à Governança Multi Level na pandemia; Instituto Stella: Florianópolis, Brazil, 2020. [Google Scholar]
  43. 2020. Available online: https://www.amib.org.br (accessed on 26 February 2021).
  44. Andersen, D.F.; Richardson, G.P.; Vennix, J.A.M. Group model building: Adding more science to the craft. Syst. Dyn. Rev. 1997, 13, 187–201. [Google Scholar] [CrossRef]
  45. Rouwette, E.A.J.A.; Korzilius, H.; Vennix, J.A.; Jacobs, E. Modeling as persuasion: The impact of group model building on attiudes and behavior. Syst. Dyn. Rev. 2011, 27, 1–21. [Google Scholar] [CrossRef]
  46. Hovmand, P.S.; Andersen, D.F.; Rouwette, E.; Richardson, G.P.; Rux, K.; Calhoun, A. Group Model-Building’ Scripts’ as a Collaborative Planning Tool. Syst. Res. Behav. Sci. 2012, 29, 179–193. [Google Scholar] [CrossRef][Green Version]
  47. DeWitte, S.N.; Kurth, M.H.; Allen, C.R.; Linkov, I. Disease epidemics: Lessons for resilience in an increasingly connected world. J. Public Health 2017, 39, 254–257. [Google Scholar] [CrossRef][Green Version]
  48. OECD—Organisation for Economic Co-operation and Development. Systemic Thinking for Policy Making: The Potential of Systems Analysis for Addressing Global Policy Challenges in the 21st Century; OECD: Paris, France, 2019. [Google Scholar]
  49. Fan, B.; Liu, R.; Huang, K.; Zhu, Y. Embeddedness in cross-agency collaboration and emergency management capability: Evidence from Shanghai’s urban contingency plans. Gov. Inf. Q. 2019, 36, 101395. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Steps of the design science research process.
Figure 1. Steps of the design science research process.
Sustainability 15 05836 g001
Figure 2. Activities involved in the macro-process of the healthcare management system.
Figure 2. Activities involved in the macro-process of the healthcare management system.
Sustainability 15 05836 g002
Figure 3. GUTAI criteria and related indicators.
Figure 3. GUTAI criteria and related indicators.
Sustainability 15 05836 g003
Figure 4. Architecture of information systems, indicators, criteria, and indexes.
Figure 4. Architecture of information systems, indicators, criteria, and indexes.
Sustainability 15 05836 g004
Figure 5. Prototypal web platform of the GUTAI COVID-19 system.
Figure 5. Prototypal web platform of the GUTAI COVID-19 system.
Sustainability 15 05836 g005
Figure 6. Application of the GUTAI Matrix to the municipality of Joinville.
Figure 6. Application of the GUTAI Matrix to the municipality of Joinville.
Sustainability 15 05836 g006
Table 1. Actors involved in building the model and their main role/contribution.
Table 1. Actors involved in building the model and their main role/contribution.
Actor (Alphabetical Order)Main Role/Contribution
Aquarela Advanced Analytics and Data Science BrigadeExpertise on data analytics
Center for Informatics and Automation of S. Catarina (CIASC)Expertise on data processing
Central Public Health Laboratory (LACEN)Pandemic/epidemiology data
Council of Secretaries Municipal Health Councils (COSEMS)Pandemic/epidemiology data
ENGIN/EGC/UFSC ResearchersStudy design/implementation
Government of Santa CatarinaOverall regulations and policy
Joinville Municipal Health SecretariatMethodology test and piloting
Mathematical Modeling TeamConsulting on parameters/modeling
Santa Catarina Court of Justice (TJSC)Security/social data and policies
Santa Catarina Medical Association (ACM)Health data and policies
Santa Catarina Public Ministry (MPSC)Security/social data and policies
Secretariat of State of Health (SES)Health data and policies
Social Good Brazil (SGB)Social-related information
State Secretariat of Administration (SEA)Demographic and market data
State Secretariat of Finance (SEF)Demographic and market data
State Secretariat of Public Security (SSP)Security data and policies
Table 2. GUTAI Matrix with criteria, indicators, and measurement scales for COVID-19.
Table 2. GUTAI Matrix with criteria, indicators, and measurement scales for COVID-19.
CriteriaSeverityUrgencyTrendAmplitudeImpact
IndicatorsICU Bed OccupationCases
COVID-19
Growth of New Confirmed CasesTerritorial Development of InfectionMortality Rate
SCALES5Very High
>80%
Very High
>80%
Very High
>2%
Very High
>80.000
Very High
>6%
4High
>50% to 80%
High
>50% to 80%
High
>1.5% to 2%
High
>60.000 to 80.000
High
>4% to 6%
3Average
>35% to 50%
Average
>35% to 50%
Average
>1% to 1.5%
Average
>40.000 to 60.000
Average
>2% to 4%
2Normal
>10% to 35%
Normal
>10% to 35%
Normal
>0.5% to 1%
Normal
>20.000 to 40.000
Normal
>0% to 2%
1Low
to 10%
Low
to 10%
Low
to 0.5%
Low
To 20.000
Low
to 0.9%
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

de Sá Freire, P.; Kempner-Moreira, F.; Margherita, A. Multi-Level Governance and Emergency Management: Building a Priority Assessment Matrix in the Pandemic Outbreak. Sustainability 2023, 15, 5836. https://doi.org/10.3390/su15075836

AMA Style

de Sá Freire P, Kempner-Moreira F, Margherita A. Multi-Level Governance and Emergency Management: Building a Priority Assessment Matrix in the Pandemic Outbreak. Sustainability. 2023; 15(7):5836. https://doi.org/10.3390/su15075836

Chicago/Turabian Style

de Sá Freire, Patricia, Fernanda Kempner-Moreira, and Alessandro Margherita. 2023. "Multi-Level Governance and Emergency Management: Building a Priority Assessment Matrix in the Pandemic Outbreak" Sustainability 15, no. 7: 5836. https://doi.org/10.3390/su15075836

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop