Improving Functional Exercises Based on Experts’ Evaluation Weights for Emergency Responses

Abstract

Social and economic changes in Southeast Asia have caused significant changes to Vietnam’s national emergency management structure, including expanding the role of firefighters from firefighting to rescue activities in various situations. It is essential to develop methods for designing exercises to identify and address the gaps in competencies related to the new roles assigned to Vietnamese firefighters. This study aimed to develop a quantitative method to effectively arrange exercise injects. It applies the analytic hierarchy process to determine the weights of response actions in functional exercises and uses comparison methods to establish the relationship between the weights of the action types, players’ capability targets, and disaster types. The results indicate that the action type weights change when the disaster type changes, whereas the capability weights within each action type remain constant. In this study, we developed a method to adjust the exercise structure based on response action weights and to optimize the number of injects that represent each action type. This innovative approach holds great significance in exercise design, increasing the likelihood of achieving the exercise capability targets, not only in Vietnam, but also in other countries.

1. Introduction

Emergency exercises play a crucial role in evaluating an organization’s readiness to achieve its set objectives. Integrating these objectives into planning and using them as criteria for assessing exercises is essential. It allows for an effective measurement of the plan’s efficiency in reaching objectives and the participants’ effectiveness in task execution. These objectives represent the predetermined performance levels set by the organization for each capability. Through post-exercise assessments and continuous improvement efforts, an entire organization can identify its shortcomings and individual incompetencies, close these gaps, and align itself with the established targets [1]. Thus, it is important to identify capacity gaps and design new exercises to address them.

Socioeconomic changes in Southeast Asia have brought about a transformation in Vietnam’s emergency management system. Specifically, the role of Vietnamese firefighters has evolved beyond firefighting to encompass a broader spectrum of rescue activities. Vietnamese firefighters must organize fire and rescue exercises to improve their capabilities. Vietnamese firefighters have been trained to have firefighting capabilities for many years and have gained several benefits. However, they started practicing rescue exercises only in 2017. The scope of their rescue operations includes a wide range of incidents, accidents, and disasters. This diversity poses challenges for practicing rescue exercises, because each type of incident, accident, and disaster has unique characteristics and requirements for rescue measures, manpower, equipment, and costs. Therefore, it is essential to design practice exercises in a systematic way that can effectively address various types of incidents, accidents, and disasters [2].

Various emergency management exercises have been developed in this domain. The emergency exercises include discussion- and operation-based exercises. The discussion-based exercises include seminars, workshops, tabletop exercises, and games, while the operation-based exercises involve drill exercises, functional exercises, and full-scale exercises [1,3]. The full-scale exercises are the most complex and offer significant benefits to the participating individuals and organizations. Vietnamese firefighters commonly perform full-scale exercises. However, these incur substantial costs, human resources, and materials, leading to cases in which they cannot be implemented [2]. Therefore, the development of effective and low-cost exercise methods is essential for augmenting full-scale exercises in Vietnam [4].

It is also crucial to develop methods for designing exercises to identify and address gaps in competencies in order to address the new roles assigned to Vietnamese firefighters. Cambodia and Laos share this situation of an expansion in the roles of firefighters, generating a pressing need for new methods for preparing, conducting, and evaluating exercises [2].

A cycle of planning, facilities, equipment, training, practice, evaluation, and improvement is necessary to ensure emergency preparedness [5,6,7,8]. Emergency exercises are considered an important component of this cycle [9,10]. A program of exercises aimed at developing coordination between relevant parties and enhancing the emergency response capacity of the participating forces requires a combination of several types of exercise. Good exercise programs should include orientation, tabletop, and functional exercises, followed by full-scale exercises [11]. Among the above exercises, functional exercises are the most suitable type of exercise for improving coordination between relevant parties, clarifying the functions and tasks of participating forces, and identifying the gaps in emergency management plans [12,13,14]. This study focused on functional exercises. A functional exercise is “designed to validate and evaluate capabilities, multiple functions and/or sub-functions, or interdependent groups of functions. Functional exercises are typically focused on exercising plans, policies, procedures, and staff members involved in management, direction, command, and control functions” [15]. However, the mobilization of actual resources is usually simulated [1]. In this context, the term “players” refers to those who perform actions in response to the exercise injects provided according to a disaster scenario. An exercise inject is a predetermined message given to the players during an exercise to prompt their reactions [16]. These injects serve as events that deliver information to the players and are specifically tailored to align with the exercise objectives. In addition to conveying details about the exercise’s progression, most injects are intended to elicit responses or actions from the players [16].

The methods for developing exercise scenarios and injects for functional exercises are still being refined. One well-known approach involves a master scenario events list (MSEL). An MSEL, either in document form or as a system, serves as a chronological timeline that outlines the anticipated actions and scripted events that the exercise designers use to introduce events into an exercise, stimulate player activities, and ensure that essential events occur to meet all of the exercise objectives. The procedural flow can also be used in more extensive and intricate exercises. Unlike the MSEL, this flow focuses solely on the expected player actions or events. The MSEL connects simulations with real-time actions, enriches the exercise experience for the participants, and mirrors incidents or activities intended to prompt player responses [1]. Using the MSEL, exercise designers can create realistic exercise scenarios and injects. They can also identify capacity gaps by comparing the actual responses of the exercise players with the expected responses specified in the MSEL.

This MSEL approach still has difficulty in selecting the types and number of injects for an exercise, such that the players can optimally enhance their desired capabilities in multiple capability domains. Usually, an inject triggers players to activate one action type, such as reporting or decision making. However, each action type is related to multiple capabilities. Thus, when an exercise designer wants to improve a specific capability, it is necessary to select the types of injects that are implicitly relevant to the targeted capability. Therefore, a quantitative method is required to facilitate this procedure.

Rescue activities have some objectives that need to be pursued simultaneously but are not easy to satisfy in total. Saving the lives of citizens involved in a natural disaster and maintaining the safety of the rescue team members are among the objectives [17]. According to the decision on the weights of these objectives, the exercise designers need to prepare an appropriate set of exercise injects in order to improve the awareness and skills of the rescue team members.

This study aims to develop a quantitative method for effectively preparing and arranging exercise injects, focusing on training Vietnamese firefighters in rescue operations. The research questions guiding this study include determining the most suitable weighting method for response actions, exploring the similarities and differences in importance weights across the different disaster types, and understanding how to arrange functional exercise injects based on the obtained weights, as follows:

• Which weighting method is the most suitable for determining the importance weights of the different response actions? The response actions consist of an action type layer and a capability layer, which will be explained later.
• What are the similarities and differences in the importance weights of the action types and capabilities across the different disaster types?
• How can we arrange the functional exercise injects based on the obtained importance weights?

The outcomes of this study can assist exercise designers in crafting effective scenarios and injects, increasing the likelihood of achieving capability targets through functional exercises. These results are not only applicable to Vietnam, but also hold relevance for other countries.

2. Methods

2.1. Strategy for Analysis

To answer Research Question 1, we reviewed the literature on weighting methods that can be used to design exercise scenarios and injects. We chose to use the analytic hierarchy process (AHP) for our design. The AHP typically considers the following three layers in its decision models: objectives, evaluation criteria, and alternatives [18,19]. A functional exercise on landslides conducted at the University of Fire Prevention and Fighting (UFPF), Vietnam, in 2022, became a reference case, and its objectives, evaluation criteria, and alternatives were specified. The evaluation criteria featured response action types, and the alternatives featured response capabilities.

To answer Research Question 2, this study calculated the importance weights across the action types and capabilities of the following three different types of natural disasters: landslides, wildfires, and flash floods. The study then identified the reasons for the weight differences across the disaster types.

To answer Research Question 3, the study analyzed functional exercises at the UFPF and demonstrated how to arrange exercise injects to improve the desired capabilities by reflecting the information obtained from the importance weights.

2.2. Methods for Estimating Importance Weights

2.2.1. Selection of Weighting Method

The selection of effective exercise injects requires an appropriate multiple-criteria decision-making (MCDM) method. MCDM has been a subject of interest to researchers for approximately 40 years. During this period, approximately 70 MCDM techniques have been investigated [20]. Each method created for MCDM has various underlying assumptions, information requirements, analysis models, and decisions [21]. This highlights the importance of choosing the most suitable method for solving a specific problem, because employing an inappropriate method can lead to misguided decisions, which can result in significant losses. Given the extensive array of MCDM techniques, selecting the correct one is a significant challenge [22].

In most MCDM models, reassessing the process of assigning weights to the criteria is crucial. Determining these weights is a significant challenge in multi-criteria decision making [23]. Numerous weighting methods have been introduced and implemented to address diverse MCDM problems. These methods can be categorized in various ways. We referred to an existing list of MCDM methods [24] and selected the relevant methods and information compiled from other studies (

Table 1. Advantages and disadvantages of weighting methods.

Weighting Method		Advantages	Disadvantages
Subjective	1. Pairwise comparison	-AHP: It can take into account the relative priorities among factors or alternatives, determining the most favorable option. Its simplicity, coupled with a wide range of applications, facilitates easy implementation to support decision makers in making precise judgments [25]. -ELETRE: It eliminates less preferred options, suitable when there are many options and few criteria. It employs straightforward logic, fully utilizes information in the decision matrix, employs a refined computational process [26], and implements an outranking approach [27].	-AHP: A solution to linear equations is not always guaranteed. As the number of levels in the hierarchy rises, the number of pairwise comparisons also increases, leading to a significantly increased time and effort required to evaluate the AHP model [25]. -ELETRE: The arbitrary nature of the threshold values impacts the final solutions [8]. It is also a time-consuming process [21] and shows uncertainty in the accuracy of rankings obtained through ELECTRE I [28]. Moreover, the method cannot effectively manage purely ordinal scales, as it necessitates a metric scale for the discordance index to assess differences [29].
	2. Point allocation	It is one of the simplest weighting methods to implement [24,30].	The obtained weights are not very precise, and the method becomes more difficult as the number of criteria increases to six or more. [24].
	3. Direct rating	It allows for the flexibility to adjust the importance of one criterion without affecting the weight of the others [24].	The weights obtained from this method are not very precise, and the method depends heavily on the decision maker first [31].
	4. Delphi method	This method leverages the expertise of individuals in the field, effectively combining the collective wisdom of expert panelists [32]. Content validity is ensured through the engagement of expert panelists and iterative rounds [33].	Much time is needed due to their iterative nature, and, over time, the expert panelists might lose interest in the research study [34]. There are no clear guidelines outlining the definitions of experts, panel size, or sampling techniques [35]. High attrition rates are expected as the number of rounds increases [36].
	5. Nominal group technique	This technique facilitates the generation of a substantial number of ideas, and those that garner the majority of votes are chosen. Participants are then tasked with selecting the five most crucial ideas from the master list and rating them on a scale of one to five based on their importance [37].	This method is not flexible, as it is designed to address only one problem at a time [37].
	6. Simple Multi-Attribute Ranking Technique (SMART), Simple Multi-Attribute Ranking Technique Exploiting Ranks (SMARTER)	SMART is simple and transparent, allowing decision makers from different backgrounds to use it [38]. The methods are designed to minimize human elicitation errors [39]. Modified versions of SMART, such as SMARTER, have also been applied to various social issues [40].	SMART does not necessarily grasp all of the details and complexity of a decision [38].
Objective	1. Entropy method	This method can measure the level of uncertainty represented by a discrete probability distribution, aiding in determining the uncertainty of each attribute or feedback [41]. The method can establish objective weights for objectives without incorporating considerations or preferences from decision makers [42].	The method exhibits high sensitivity to the entropy values of various criteria [43].
	2. Criteria Importance through Inter-criteria Correlation (CRITIC)	The CRITIC method automates the process of determining criteria weights, reducing the complexity of the evaluation [44]. This method focuses on the correlation between criteria, capturing the important relationships between them in the decision-making process [45,46].	As the matrix dimension expands, the calculation time becomes extensive [47]. This method fails to communicate the relative significance of fulfilling the decision makers’ objectives; instead, it merely reflects certain properties of the initial data [48].
	3. Mean weight	This method can be used without information from the decision maker or when there is insufficient data to make a decision [24,49].	It operates under the assumption that all criteria carry equal significance [24]. It may not accurately reflect the true priorities or significance of different criteria in a decision-making process, leading to a potential oversimplification of the complex considerations involved.
Integrated	1. Multiplication synthesis 2. Additive synthesis 3. Optimal weighting based on sum of squares 4. Optimal weighting based on relational coefficient of graduation 5. Quality function development (QFD)	The integrated method combines both subjective and objective information in the weight determination process. This helps to apply the decision maker’s knowledge and experience, along with objective information from the mathematical model. This method overcomes the disadvantages of both the subjective and objective methods [24]. Several combinations have been put forth and formulated by researchers [49,50]. QFD has been applied to work equipment safety, combined with the Delphi method and the fuzzy logic approach [51].	It can be difficult to effectively combine subjective and objective information, especially when they are heterogeneous.

).

In objective weighting methods, criteria weights are derived from the information gathered for each criterion using mathematical models. An additional example is the combined use of the T-S fuzzy fault tree and Bayesian network to predict coal mine roof accidents [52]. Objective weighting methods usually proceed without considering the responses of the decision maker [53]. The integrated weighting approach is based on a combination of subjective and objective weighting methods [24]. In these methods, the weights are calculated based on already available data.

In this study, the players’ response actions were considered as evaluation criteria, and the capability targets of the exercise were considered as alternatives. The application of the weighting method to player response actions is a new and unprecedented study on functional exercise in Vietnam; therefore, no data were available for the application of objective and integrated weighting methods. Therefore, we selected a subjective weighting method.

To determine the importance weights, we chose a weighting method based on pairwise comparison. Two methods are commonly applied to process pairwise comparison results: ELETRE and AHP. In 1968, Roy introduced ELETRE to solve the problem of ranking alternatives from the best to worst. [22]. In the 1970s, Saaty proposed the AHP, which has become one of the most widely used methods in MCDM applications, as it is easy to use [24,54].

ELETRE focuses on comparing options based on criteria, without the need to determine their priority levels. The AHP enables decision makers to assess the relative correlations between criteria using pairwise comparisons to determine their priority levels. We opted for the AHP method to determine the weights of response actions, aiming to ascertain the priority and significance of each criterion.

The AHP has many applications in group decision making [6] and is used in government, business, industry, and emergency situations [55,56,57]. The AHP is useful when people work together to solve complex problems [58] and has been applied to problems involving planning, resource allocation, priority setting [59], and solving decision problems in emergency management [60]. Safety equipment choice was studied using a framework that combined the AHP with a fuzzy model [61]. The analytic network process, which is an extension of the AHP, was used to enhance the safety performance management of construction projects [62].

2.2.2. Applying the AHP

The method for calculating the weights of the criteria using the AHP was adopted from Saaty [18,54,63]. In implementing this process, the authors followed three steps, as illustrated in Figure 1, as follows: Step 1 involved building the AHP model; Step 2 focused on establishing a pairwise comparison matrix; and Step 3 entailed estimating the importance weights.

To build an AHP model, the following three components must be defined: objectives, criteria, and alternatives. This model is illustrated in Figure 2. The elements of the model are defined as follows:

Objective: Improvement of exercises

Criteria (action types): Report, Decision Making, Suggestion, Discussion, and Maneuver.

These criteria were identified as the action types of the participants in the exercise at the UFPF in 2022.

Alternatives (capabilities): Raise awareness levels; Reduce time of participant response actions; and Increase response actions completion percentage.

One objective of emergency management exercises is to obtain participant feedback and recommendations for program improvement [64]. Therefore, the authors analyzed the experts’ comments and recommendations collected in the hot-wash discussion after the functional exercise on landslide response at the UFPF in 2022. The experts recommended the following three capability issues for improvement: raising awareness levels; reducing the time of participant response actions; and increasing the percentage of completion response actions.

2.2.3. The Survey

We created pairwise comparison questions based on the AHP model (Figure 2). Our survey questionnaire, designed following Saaty’s guidelines [54], aimed to gather expert opinions on pairwise comparisons for action types and capabilities. These pairwise comparison questions were repeated for the three types of natural disaster. The incident and natural disaster situations in the survey were not specifically described in terms of scale; number of dead, injured, and affected people; or property damage. However, the authors classified these situations as very serious on the incident and natural disaster assessment scales, ranging from less serious to serious, very serious, and especially serious. This classification allowed the experts participating in the survey to visualize the scale of the incident or natural disaster, including the number of victims, injured and missing individuals, and property damage caused by the incident or natural disaster.

Table 2. Example of pairwise comparison questions.

Question: Consider a landslide case and compare each pair of evaluation criteria. The relative importance levels were selected for each comparison and their relative importance was rated.
Report	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Decision Making
Report	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Suggestion
Report	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Discussion
Report	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Maneuver
Decision Making	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Suggestion
Decision Making	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Discussion
Decision Making	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Maneuver
Suggestion	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Discussion
Suggestion	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Maneuver
Discussion	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Maneuver

presents examples of the questions used to compare the elements of the action types. The remaining questions are listed in

Table A1. Pairwise comparison questions for capabilities in each action type.

Report	Raise awareness levels	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Reduce time of participant response actions
	Raise awareness levels	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Increase response actions completion percentage
	Reduce time of participant response actions	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Increase response actions completion percentage
Decision Making	Raise awareness levels	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Reduce time of participant response actions
	Raise awareness levels	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Increase response actions completion percentage
	Reduce time of participant response actions	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Increase response actions completion percentage
Suggestion	Raise awareness levels	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Reduce time of participant response actions
	Raise awareness levels	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Increase response actions completion percentage
	Reduce time of participant response actions	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Increase response actions completion percentage
Discussion	Raise awareness levels	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Reduce time of participant response actions
	Raise awareness levels	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Increase response actions completion percentage
	Reduce time of participant response actions	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Increase response actions completion percentage
Maneuver	Raise awareness levels	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Reduce time of participant response actions
	Raise awareness levels	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Increase response actions completion percentage
	Reduce time of participant response actions	9	8	7	6	5	4	3	2	1	2	3	4	5	6	7	8	9	Increase response actions completion percentage

in Appendix A.

We conducted an in-person survey to collect answers to the pairwise comparison questions from disaster response experts in Vietnam. The survey was conducted on 6 September 2023, with the participation of seven lecturers from the Faculty of Rescue, UFPF, who were coded as Lecturers E1 through E7.

As mentioned in the Introduction, Vietnamese firefighters regularly perform full-scale exercises. However, compared to exercise programs in other countries, such as the United States and Japan, the full-scale exercises of Vietnamese firefighters are not preceded by smaller exercises, such as tabletop exercises and functional exercises. In Vietnam, the concept of functional exercises remains relatively new. This is because the legal documents on fire prevention, firefighting, and rescue in Vietnam do not have specific concepts or regulations on functional exercises. In addition, the theoretical-based system in the fields of fire prevention, firefighting, rescue, and natural disaster response in Vietnam does not include the concept and content of training and functional exercises [2].

The survey participants were required to meet the following two criteria: having a clear understanding of functional exercises and a high level of expertise in the fields of firefighting, rescue, and natural disaster response. Owing to the above limitations, selecting experts to participate in the survey was difficult, because it was not possible to select experts from many different fields to create more objective research results. The functional exercises on landslides at the UPFP were the first functional exercises in the field of rescue and disaster response in Vietnam. The officials in charge of designing and practicing emergency exercises for the provincial fire service were assessed as not having sufficient expertise in functional exercises in rescue and natural disaster response situations to participate in the survey [2]. Therefore, we chose seven lecturers from the Faculty of Rescue at the UPFP to participate in the survey. Some information on the professional qualifications of the lecturers is as follows:

-

Academic qualifications: one doctorate and six masters in fire prevention, firefighting, and rescue.

-

Research fields: firefighting, rescue, and natural disaster response.

-

Subjects taught: basic rescue; personal techniques and rescue teams; first aid; training in using specialized rescue equipment; organizing rescue in a fire; rescue when houses and structures collapse; rescue when there is an incident at a chemical facility; organizing rescue in some special situations (earthquake, tsunami, and others); organizing rescue during motor traffic incidents; organizing underwater rescue; organizing work in professional firefighting teams; and responding to climate change.

-

Teaching experience: 7–17 years.

The survey participants had knowledge of and experience with functional exercises. Two lecturers taught functional exercises to those participating in the functional exercises on landslides at the UPFP in 2022, one lecturer participated in designing these functional exercises, and the remaining four instructors evaluated groups of players during these functional exercises.

There were experienced and less-experienced lecturers in actual disaster responses. This diversity enabled us to compare the importance weights across lecturers with different levels of disaster response experience.

After collecting the responses, we established a pairwise comparison matrix for each expert. The importance weight was calculated using this matrix. Our results indicated variations in the importance weights among the experts. To explain these differences, we conducted a follow-up interview survey with the experts in October 2023. E4, E5, and E7 were selected for the interviews because their answers were similar to the average weights across the seven experts in one disaster but different in another disaster.

2.3. Method for Calculating and Comparing Importance Weights

The importance weights were calculated using Saaty’s method [54]. A pairwise comparison matrix was used to calculate the importance weights across the action types for each disaster type and expert. The importance weights across the capabilities for completing each action type were then calculated for each disaster type by each expert. The synthesized weights of the capabilities required to respond to each type of disaster were then calculated for each expert.

All weights underwent a consistency check to ensure the accuracy of the results. This is a mandatory step in the AHP. According to Saaty (1987) [63], a consistency ratio (CR) value between 0.01 and 0.02 is deemed acceptable, while values below 0.1 indicate a high consistency of responses. If the consistency ratio exceeds 0.2, a re-survey is necessary to gather opinions from experts with that consistency ratio, as a higher consistency ratio suggests inconsistency in the importance weight, or, in other words, inaccurate results. In this study, all seven expert opinions had a consistency ratio below 0.1, demonstrating high consistency of the results.

We determined the average weights according to pairwise comparisons by the seven experts for each disaster type (landslides, wildfires, and flash floods). Next, we analyzed the differences in the importance weights across the action types. Using the same method, the authors determined the relationship between the capability weights within each action type across disaster types. Thus, we answered Research Question 2.

2.4. Improving Exercise Injects Based on Response Action Weights

Action type weights represent the degree of the impact of the action type on a player’s ability goals. The number of injects corresponding to each type of action was not considered in the previous section. Functional exercises often involve multiple injects for different types of actions. Therefore, the number of injects chosen from the different action types must be considered for a thorough understanding of their impact. We assumed that the types of actions repeated more often in an exercise would have a greater impact on capacity building. Therefore, the number of times each action is taken is an important factor that must be adjusted by exercise designers.

We used the inject structure of the landslide functional exercise at the UFPF in 2022 as a typical example to demonstrate the development processes of the injection structure based on the importance weights. The authors first calculated the weights of the players’ response actions in the actual UFPF exercise by multiplying the average weight of the actions by the number of injects that fit each action type, as shown in

Table 3. Original structure of the landslide response functional exercise.

Action Type	Report	Decision Making	Suggestion	Discussion	Maneuver
Number of injects	10	40	4	12	3

(Report, Decision Making, Suggestion, Discussion, and Maneuver).

Table 4. The sum of the weights of each action type (original).

Action Types	Average Weight (Figure 3)	Number of Injects (Table 3)	Exercise Weight for Each Action Type
Report	0.155	10	0.155 × 10 = 1.55
Decision Making	0.411	40	0.411 × 40 = 16.44
Suggestion	0.064	4	0.064 × 4 = 0.25
Discussion	0.279	12	0.279 × 12 = 3.35
Maneuver	0.091	3	0.091 × 3 = 0.27

summarizes the information needed for calculating exercise weights for different action types.

Table 5. Capability weights in each action type (average results from Table A2).

Capabilities	Report	Decision Making	Suggestion	Discussion	Maneuver
Raise awareness levels	0.084	0.339	0.132	0.719	0.170
Reduce time of participant response actions	0.499	0.476	0.361	0.090	0.514
Increase response actions completion percentage	0.417	0.186	0.507	0.191	0.316

presents a partial reproduction of

Table A2. Capability weights within each action type for landslide exercises.

Capabilities	E1					E2
	R	DM	S	D	M	R	DM	S	D	M
Raise awareness levels	0.098	0.623	0.297	0.738	0.087	0.087	0.608	0.110	0.751	0.067
Reduce time of participant response actions	0.715	0.240	0.164	0.094	0.639	0.639	0.120	0.309	0.064	0.689
Increase completion percentage of response actions	0.187	0.137	0.539	0.168	0.274	0.274	0.272	0.581	0.185	0.244
	E3					E4
	R	DM	S	D	M	R	DM	S	D	M
Raise awareness levels	0.103	0.120	0.104	0.633	0.277	0.055	0.179	0.100	0.777	0.179
Reduce time of participant response actions	0.174	0.608	0.231	0.106	0.129	0.587	0.739	0.600	0.069	0.739
Increase completion percentage of response actions	0.723	0.272	0.665	0.261	0.595	0.358	0.082	0.300	0.155	0.082
	E5					E6
	R	DM	S	D	M	R	DM	S	D	M
Raise awareness levels	0.082	0.557	0.123	0.685	0.098	0.070	0.174	0.087	0.767	0.174
Reduce time of participant response actions	0.575	0.320	0.320	0.093	0.568	0.580	0.723	0.639	0.085	0.723
Increase completion percentage of response actions	0.343	0.123	0.557	0.221	0.334	0.350	0.103	0.274	0.148	0.103
	E7					AVG
	R	DM	S	D	M	R	DM	S	D	M
Raise awareness levels	0.093	0.110	0.106	0.681	0.309	0.084	0.339	0.132	0.719	0.170
Reduce time of participant response actions	0.221	0.581	0.261	0.118	0.110	0.499	0.476	0.361	0.090	0.514
Increase completion percentage of response actions	0.685	0.309	0.633	0.201	0.581	0.417	0.186	0.507	0.191	0.316

in Appendix B and demonstrates the capability weights for each action type.

Table 6. The exercise decision weights of the capabilities (original).

Capabilities	Exercise Decision Weights
Raise awareness levels	8.19
Reduce time of participant response actions	9.13
Increase response actions completion percentage	4.55

lists the decision weights of the capabilities that incorporate the exercise inject structure. The exercise decision weight for raising awareness levels is 8.19 in Table 6. This figure was obtained as the sum of the element-by-element multiplication of the right column in Table 4 with the top row in Table 5, as follows: 1.55 × 0.084 + 16.44 × 0.339 + 0.25 × 0.132 + 3.35 × 0.719 + 0.27 × 0.17 = 8.19.

After modification by the exercise designer as shown in Table 7 for example, multiplying the vectors listed in Table 5 and Table 8 results in Table 9. The decision weights of the alternatives could be changed by varying the number of injects categorized according to the different players’ action types.

Table 7. Modified structure of the landslide response functional exercise.

Action Type	Report	Decision Making	Suggestion	Discussion	Maneuver
Number of injects	10→5	40	4	12→19	3→1

Table 7. Modified structure of the landslide response functional exercise.

Action Type	Report	Decision Making	Suggestion	Discussion	Maneuver
Number of injects	10→5	40	4	12→19	3→1

Table 8. The sum of the weights of each action type (modified).

Action Types	Average Weight (Figure 3)	Number of Injects (Table 7)	Exercise Weight for Each Action Type
Report	0.155	5	0.155 × 5 = 0.78
Decision making	0.411	40	0.411 × 40 = 16.44
Suggestion	0.064	4	0.064 × 4 = 0.25
Discussion	0.279	19	0.279 × 19 = 5.31
Maneuver	0.091	1	0.091 × 1 = 0.09

Table 8. The sum of the weights of each action type (modified).

Action Types	Average Weight (Figure 3)	Number of Injects (Table 7)	Exercise Weight for Each Action Type
Report	0.155	5	0.155 × 5 = 0.78
Decision making	0.411	40	0.411 × 40 = 16.44
Suggestion	0.064	4	0.064 × 4 = 0.25
Discussion	0.279	19	0.279 × 19 = 5.31
Maneuver	0.091	1	0.091 × 1 = 0.09

Table 9. The exercise decision weights of the capabilities (modified).

Capabilities	Exercise Decision Weights
Raise awareness levels	9.50
Reduce time of participant response actions	8.83
Increase response actions completion percentage	4.55

Table 9. The exercise decision weights of the capabilities (modified).

Capabilities	Exercise Decision Weights
Raise awareness levels	9.50
Reduce time of participant response actions	8.83
Increase response actions completion percentage	4.55

Figure 3. Action type weights. (a) Landslides. (b) Wildfires. (c) Flash floods.

Figure 3. Action type weights. (a) Landslides. (b) Wildfires. (c) Flash floods.

3. Results

3.1. Disaster Types and Importance Weights of Response Actions

3.1.1. Action Type Weights

Figure 3 shows that the weights of the action types differed across the disaster types. Panel a of Figure 3 shows that E2 and E6 considered that, for landslides, the Discussion action type had the greatest importance. Meanwhile, the remaining five experts believed that the act of Decision Making was the most important factor. Figure 3b shows that E2 and E7 believed that, for wildfires, the Decision Making action type was the most important; E4 believed that Discussion was the most important; and the remaining four experts thought that Maneuver was the most important. Figure 3c shows that E2, E5, and E7 thought that, for flash floods, Suggestion was the most important, versus Maneuver for the remaining four experts. According to the average scores of the seven experts, Decision Making and Discussion received greater priority for landslides than for the other action types, including Report and Maneuver. In contrast, Report and Maneuver were considered more important than Decision Making and Discussion for flash floods. Individual experts E1, E2, E6, and E7 shared this reversed pattern.

3.1.2. Capacity Weights within Each Action Type

Figure 4 shows the capacity weights for each action type. The weights were averaged for all seven experts. The figure shows similar patterns for the three types of disasters. Raising awareness is considered the most important factor for effective discussions, whereas reducing the response time is the most important factor for reporting, decision making, and maneuvering. Regarding Suggestions, the most important capability was increasing the completion percentage of the response action. The results from the individual experts are shown in Table A2,

Table A3. Capability weights within each action type for wildfire exercises.

Capabilities	E1					E2
	R	DM	S	D	M	R	DM	S	D	M
Raise awareness levels	0.100	0.619	0.320	0.619	0.093	0.087	0.620	0.123	0.765	0.080
Reduce time of participant response actions	0.600	0.284	0.123	0.096	0.615	0.639	0.156	0.320	0.074	0.656
Increase completion percentage of response actions	0.300	0.096	0.557	0.284	0.292	0.274	0.224	0.557	0.161	0.265
	E3					E4
	R	DM	S	D	M	R	DM	S	D	M
Raise awareness levels	0.104	0.222	0.093	0.623	0.277	0.065	0.137	0.087	0.780	0.168
Reduce time of participant response actions	0.231	0.667	0.221	0.137	0.129	0.593	0.780	0.639	0.083	0.738
Increase completion percentage of response actions	0.665	0.111	0.685	0.240	0.595	0.341	0.083	0.274	0.137	0.094
	E5					E6
	R	DM	S	D	M	R	DM	S	D	M
Raise awareness levels	0.089	0.623	0.120	0.707	0.110	0.076	0.182	0.082	0.737	0.201
Reduce time of participant response actions	0.587	0.240	0.272	0.092	0.581	0.591	0.703	0.575	0.077	0.681
Increase completion percentage of response actions	0.324	0.137	0.608	0.201	0.309	0.334	0.115	0.343	0.186	0.118
	E7					AVG
	R	DM	S	D	M	R	DM	S	D	M
Raise awareness levels	0.106	0.096	0.098	0.723	0.334	0.090	0.357	0.132	0.708	0.180
Reduce time of participant response actions	0.261	0.653	0.334	0.103	0.098	0.500	0.497	0.355	0.095	0.500
Increase completion percentage of response actions	0.633	0.251	0.568	0.174	0.568	0.410	0.145	0.513	0.198	0.320

and

Table A4. Capability weights within each action type for flash flood exercises.

Capabilities	E1					E2
	R	DM	S	D	M	R	DM	S	D	M
Raise awareness levels	0.087	0.681	0.334	0.700	0.089	0.106	0.623	0.098	0.753	0.080
Reduce time of participant response actions	0.639	0.201	0.142	0.107	0.587	0.633	0.137	0.334	0.075	0.656
Increase completion percentage of response actions	0.274	0.118	0.525	0.194	0.324	0.261	0.240	0.568	0.172	0.265
	E3					E4
	R	DM	S	D	M	R	DM	S	D	M
Raise awareness levels	0.118	0.240	0.104	0.571	0.230	0.064	0.154	0.110	0.767	0.174
Reduce time of participant response actions	0.201	0.623	0.231	0.143	0.122	0.646	0.755	0.581	0.085	0.723
Increase completion percentage of response actions	0.681	0.137	0.665	0.286	0.648	0.290	0.092	0.309	0.148	0.103
	E5					E6
	R	DM	S	D	M	R	DM	S	D	M
Raise awareness levels	0.087	0.608	0.110	0.724	0.106	0.082	0.182	0.082	0.739	0.201
Reduce time of participant response actions	0.639	0.272	0.309	0.083	0.633	0.575	0.703	0.575	0.082	0.681
Increase completion percentage of response actions	0.274	0.120	0.581	0.193	0.261	0.343	0.115	0.343	0.179	0.118
	E7					AVG
	R	DM	S	D	M	R	DM	S	D	M
Raise awareness levels	0.098	0.098	0.096	0.707	0.334	0.092	0.369	0.133	0.709	0.174
Reduce time of participant response actions	0.334	0.568	0.284	0.092	0.098	0.524	0.466	0.351	0.095	0.500
Increase completion percentage of response actions	0.568	0.334	0.619	0.201	0.568	0.384	0.165	0.516	0.196	0.327

.

3.1.3. Decision Weights of Capabilities

Figure 5 shows the decision weights of the capabilities for each disaster type. According to the scores averaged by the seven experts, reducing the response time was considered equally important regardless of the disaster type. Increasing the awareness levels was considered more important than increasing the percentage of response completion for landslides; however, this priority order was reversed for flash floods.

3.2. An Example of Improving the Exercise Inject Structure Based on the Obtained Weights

As an illustration using the weight information gathered in the preceding section, the authors have demonstrated a procedure to adjust the focus of functional exercises conducted at the UFPF in 2022. Table 3 displays the number of exercise injects assigned to the different action types in the original exercise design. Subsequently, the total weights for each action type were calculated, as listed in Table 4, culminating in the total weights for the alternatives presented in Table 6. This table reveals that the original exercise design placed the greatest emphasis on the capability of “Reduce time of participant response actions.”

Now, assume that the exercise designer wishes to prioritize “Raise awareness levels,” while maintaining the total number of injects as in the original design. This can be accomplished by adjusting the number of injects assigned to each action type criterion. Figure 4 indicates that “Discussion” is most relevant to “Raise awareness levels.” Simultaneously, it shows that “Raise awareness levels” has less relevance to the “Report” and “Maneuver” action types. Consequently, the designers should generate more injects for “Discussion” and decrease the number of injects for the “Report” and “Maneuver” action types. Table 7 exemplifies the allocation of injects according to this strategy. This modification resulted in the weights listed in Table 8 and Table 9. Table 9 verifies that the weight of “Raise awareness levels” became the highest among the capabilities, as intended.

4. Discussion

As illustrated in Figure 3, the weights of the action types varied significantly across the three disaster types, indicating distinct priorities for each type. Discrepancies in expert responses were noted, and interviews with three experts provided insights into these variations. For instance, E4 responded relatively similarly to the average weight values for flash floods and landslide disasters, particularly flash floods. However, for wildfire disasters, E4′s assessment was largely different from the average weight values. To explain this, E4 said that they did not have much experience in responding to wildfires; therefore, when participating in a wildfire response exercise, they would first prioritize the Discussion type of action, which would help them to perform other actions more accurately and efficiently. E5 and E7 did not prioritize Discussion. Through the interview process, we learned that E5 and E7 had more experience with disaster responses than E4 did. The influence of the responders’ experience, knowledge, and skills in prioritizing action types became evident, with less-experienced respondents favoring Discussion and experienced ones prioritizing actions like Maneuver, Decision Making, and Suggestions.

In the event of natural disasters, there is always a potential risk of secondary incidents. It is always possible for new and unusual problems to arise. Therefore, emergency responders’ tendency to prioritize Discussion before taking other actions may also stem partly from a cautious personality. In certain situations, it is necessary to avoid mistakes made by emergency responders who are overconfident in their judgment. Some argue that identifying the difference between experienced and inexperienced instructors is important. Knowing this difference, we can share the knowledge of the experienced trainers with the inexperienced ones to improve their capabilities. This can also be applied to evaluating the players’ experience through reaction processes during an exercise. Separate exercises, lectures, and training methods are available for each player with different levels of response experience.

The finding that the weights of the capabilities assigned to the action types do not vary across the disaster types (see Figure 4) is significant in the development of an exercise structure based on expert weighting. The decision weights for the capabilities differ across the disaster types, as shown in Figure 5. However, this difference primarily stems from variations in action type weights rather than the capability weights across the different disaster types. Thus, the capability weights can be constantly applied to different disaster exercises to calculate their weights and adjust the structure of the exercises to increase the likelihood of achieving the capability targets.

Section 3.2 suggests that exercise designers use a method to calculate the weights assigned to the different capabilities in an exercise structure. We provide an example of the development of an exercise structure based on response action weighting. This example is based on the estimated weights, thereby adjusting the number of injects to change the relative importance of the targeted capabilities. This procedure enables exercise designers to create a structure that collectively enhances either a player’s specific skills or all of the desired skills in a balanced manner by adjusting the number of injects based on the estimated weights. This study introduces a completely new approach for developing exercise structures that can be applied to the design of response exercises for many types of disasters in Vietnam and other countries.

The AHP helped to identify the differences in ideas dealing with disasters among the lecturers surveyed in this research. They worked as exercise designers for students. Visualizing these differences will help novice designers to learn from experienced designers and improve their exercise-teaching skills. Understanding these differences will also help us to rethink the desired procedure for dealing with disasters and create an effective education program for firefighting students.

The AHP is a flexible method that can elicit response action weights from the answers provided by a small number of experts. This method can be applied to improve existing exercises and to design new exercises for agile managers. The AHP model used in this study (Figure 1) can be easily modified according to the exercise context and the level of the players’ competencies. For example, the safety issue of rescue workers is included in the alternative of “raise awareness.” If the exercise designer wants to explicitly treat the safety issue, the designer can modify the alternatives to explicitly include the “safety issue.” The designer can then select and prepare exercise injects to effectively improve the rescuers’ safety.

The results of this study are applicable to designing exercises for emergency responses to incidents and natural disasters, enhancing the ability to achieve exercise objectives. The exercise results can serve as a basis for improving subsequent exercises based on the weighting of action types.

This study opens up a new direction for applying weighting methods to different types of player response actions. The weight of the players’ response actions can be considered a parameter that reflects their interest level and priority level of using certain types of actions for certain types of injects.

5. Conclusions and Limitations

This study focused on clarifying the close relationships among the goals of the exercise, injects, and players’ responses. We reviewed the weight calculation methods and selected the AHP as the best method for improving the inject structure for functional exercises. We then analyzed the patterns of weights regarding emergency action types and response capabilities. This study proposes a method for arranging exercise injects based on the weighting of action types. These research results are easily applicable to the design of functional exercises to respond to disasters in Vietnam and other countries facing similar situations.

However, certain issues must be addressed in future studies. It is difficult for exercise designers to create effective injects from scratch that stimulate the players and enhance their desired capabilities. Measuring the baseline capabilities of the players in each capability domain at the beginning of an exercise is another important issue for effectively filling the capability gaps.