Using System Dynamics Method to Measure Project Management Performance of Local Government Agencies

Abstract

Measuring project management performance is complex and requires tools to capture the dynamic nature of the processes involved. Since the conception of system dynamics in the 1950s, the method has been used to solve complex projects. Project management possesses dynamic characteristics that involve planning, human resources, implementation, and control elements; thereby, using system dynamics to measure project management performance is a realistic approach. A research study was conducted using system dynamics to develop project management performance measures to capture the complexity of the process in local government agencies. The research approach considers measuring project engineering management performance as a holistic system influenced by leadership involvement, project management processes, and project engineering manager’s ability. The Zachman architectural framework was used to develop the project-management performance system’s ontology as the system dynamics model’s foundation. A case study was conducted for three cities with local government agencies to better understand the model components and factors that influence performance. Leadership involvement, project management processes, and project manager abilities were identified as critical factors that influence the project management performance level. To validate the results of the case study, the project management performance was further studied for the City of El Paso in terms of capability, capacity, and maturity level. The research study concluded that system dynamics is a feasible method and effective tool to measure management performance for engineering projects at local government agencies.

1. Introduction

Project management has been adopted by local government agencies, with varying degrees of expertise, to deliver public works projects efficiently. To evaluate the performance of project management practices, the agencies utilize traditional budget and schedule tracking tools. Since World War II, project engineering management processes have grown increasingly sophisticated, and these traditional tools are no longer sufficient to evaluate project management performance [1,2]. Baccarini further explained project complexity elements, including organizational complexity and technological complexity, and other sub-elements such as workforce complexity. Nonetheless, traditional measurement tools are still used nowadays to measure project management performance. One possible reason is that local government agencies may not be aware of other methods to measure performance.

A comprehensive literature of existing studies showed a gap in performance measurement methods in the project engineering management area [3]. There is no consensus on how to assess project management performance in construction projects [4]. This study is conducted to measure project-management performance using system dynamics to capture the complexity of the process for engineering managers in delivering a project. The study approach focuses on how to enhance the competencies of practicing engineering managers through fostering leadership involvement, better knowledge of project management processes, and continued growth of the project manager’s ability.

This paper is organized into seven sections: introduction, background, the systems dynamics method to measure project performance, case study, results, discussion and validation, and conclusions. The introduction section describes the need for the study. The background summarizes the literature review, and the methodology explains the systems dynamics approach used to measure the project management performance. The case study section describes the application of the system dynamics approach in three local government agencies followed by a section presenting the results of the study. The discussion and validation section includes the interpretation of the results of the case study with comments for implementation complemented with the validation of the findings from the study, and the conclusion section summarizes the findings with recommendations for future research.

2. Background

Performance measurements in construction are critical because of the global economic impact of the industry; therefore, achieving a high-performance construction level requires effective project management [5,6]. Many studies with different approaches have been conducted to improve the performance of project management practices. However, most of these studies were intended for private sector usage and may not be suitable for local agencies. Considerable research studies have been done in the private sector with the primary goal of creating value for stakeholders and focusing on profitability and sustainable competitive advantage [7]. This goal is not aligned with local government entities and research studies focused on performance measures for practicing engineering managers in local agencies are limited. Furthermore, there is no consensus on the best way to assess project management performance in construction [5,8].

It is critical to select performance measurements that best fit local agencies’ management practices. Formulating a performance measure begins with the definition of its purpose. Only then, performance measures can be selected or developed with the characteristics required to achieve the agency’s objectives. With the progress of time, project management has grown in sophistication and complexity, and the project management process is complex due to the interaction among its components. Baccarini (1996) explained that project complexity is due to interrelated parts with differentiation and interdependency [1]. Moreover, complexity can change over a project life cycle, and as projects continue to reduce project timelines for execution, they become even more complex [9]. The traditional approach that relies on budget and schedule tracking tools is no longer sufficient to assess project management’s performance because of its complexity as a system [10].

As a complex system, project management possesses dynamic characteristics that consist of planning, human resources, implementation, and control elements; thereby, a system dynamics method provides a realistic approach to capture this complexity [11].

System Dynamics: In the 1950s, the concept of system dynamics was first introduced by Jay W. Forrester of the Sloan School of Management, at the Massachusetts Institute of Technology. Industrial dynamics, “described as the application of feedback concepts to social systems”, was the foundation of system dynamics. It started 50 years ago with academic initiatives that prioritized significant concerns outside academia evolving from a “theory of structure in systems as well as being an approach to corporate policy design” [12]. The Club of Rome, among other organizations, used the system dynamics model to demonstrate the connection between resource depletion and economic expansion, which generated a great deal of public interest when the club’s first report, The Limits to Growth, was released in 1972 [13]. In the 2000s, system dynamics regained its breath because this field is on an aimless plateau. Although, there was minimal evidence of a significant push into new terrain as “Lack of impact on government policies”, and it was still adhering to practices from previous decades [14].

Over the years, the concept of system dynamics has been used to simplify a complex system analysis based on cause-and-effect relationships. Moreover, in light of the human brain’s limited capability, system dynamics provides valuable assistance to develop project performance measures for managers. Human and technical system behaviors are studied using its’ techniques [15]. Project management is a complex social system because it involves individuals interacting with each other, working as a unit in a network to serve a common purpose.

People often cannot adequately understand how social systems behave due to the presence of multiple non-linear feedback mechanisms; social networks are complex and challenging to comprehend [16]. Furthermore, every person uses mental models based on assumptions and relationships as decision-making tools, and these models may be “incomplete” due to the complexity of the processes.

The system dynamics approach has been used to study various aspects of a project process including design, construction, and management. For example, system dynamics was used to study the delay and disruption of engineering projects; the research focused on the delay in approving design changes. The results showed a significant benefit of system dynamics in revealing patterns and behavior and incorporating project management decisions into solving the problems [17,18]. Moreover, System dynamics constructs a causal feedback loop to characterize the dynamic adjustment process of construction safety, which is an appropriate research approach, and anticipates the changing trend of the system and describes the dynamic development laws of the system by establishing causal feedback loops [19]. System dynamics helped to improve the understanding of the complex nature of project management performance; it identifies common problem sources and cause-effect “paths” that affect projects [20].

A system dynamic method was also used to analyze the behavior and operation of an engineering service department. In this study, a system dynamics model was developed to analyze system behavior, information feedback and formulate mathematical models of dynamic interrelationships in the engineering service department. The results indicated a need for a strategic change to establish a new culture and operation structure in the department; it provided a valuable understanding of the targeted area of improvement for managers to increase efficiency [21].

System dynamics approach has also been used to review project management’s dynamic characteristics of planning, human resources, implementation, and control elements [22,23]. Moreover, it provided a comparison between traditional approaches and system dynamics. The study noted that traditional methods are linear and assume the sum of the parts provides an estimate of the total project. The study concluded that project management performance benefits from combining traditional approaches and system dynamics methods. Furthermore, the use of system dynamics offers a complete view of the project as a whole to enhance the traditional method by incorporating more subjective factors such as the client’s behavior and the interaction on the project outcomes.

System dynamics provides a holistic approach to developing performance measures for local agencies. It addresses shortcomings of the linear approach followed by traditional project management methods; furthermore, it affords an understanding of the implemented effects of alternative responses’ actions. It is suitable for dealing with long-term and cyclical problems, for study with insufficient data, for dealing with complicated social and economic concerns yet requiring less precision than physical science, and for conducting conditional precision [24]. Therefore, it offers the most feasible venue to develop the project management performance measure.

3. Methodology

This section describes the approach and methods for developing project management performance measures using the system dynamics model. The approach adopted for this study looks at the project engineering management practices of delivering projects as a system, a holistic concept to simplify complex interactions between various project management elements. By definition, a system is a group of devices or artificial objects or an organization forming a network distributing something or serving a common purpose. Additionally, this study is focused primarily on the engineering system, which is a collection of artificial objects or parts designed to act together to perform a specific function or a set of features.

The approach considers leadership involvement, project management processes, and the competencies of the engineering managers as critical factors in the project management execution at local agencies. These critical factors influence the project management performance level, measured by the local agency’s maturity, capacity, and capability. These performance management components are used in the development of the system dynamics performance model. The Project Management Body of Knowledge (PMBOK) guidelines, combined with the quality management principle and the Project Management Maturity Model (PMMM), were utilized to develop the performance level equation, and assess project management maturity level for the system dynamics model. In developing the model, the Zachman architectural framework was used to define system components (artifacts) and boundaries.

The general hypothesis in the study is that systems dynamics can be used to identify critical performance factors that affect project management in local agencies. The system dynamics software Vensim was instrumental in developing the model. A survey was also conducted to collect first-hand information from practicing engineering managers.

3.1. Survey

Three local governments participated in this study: The City of Sunland Park, the County of El Paso, and the City of El Paso. Question-Pro was used to collect information about project management practices. The survey questionnaire consisted of nineteen questions pertaining to the agency’s organization and composition and thirty maturity level assessment questions. The maturity level assessment questionnaires covered project management processes and knowledge areas, as described in the Guide to the Project Management Body of Knowledge [25]. The Project Management Maturity Model (PMMM) was adapted to create the questionnaire to determine the maturity of the project management processes. The PMMM is an adaption of the Capability Maturity Model (CMM) for software development by the Software Engineering Institute. The PMMM provided a five-level maturity grade system parallel to those of the CMM, with level one as the lowest level and five as the highest.

3.2. Project Management Measuring Model

To develop the system dynamics model, the system ontology was first defined, and then the system dynamics performance model was developed. These steps are described as follows: Performance Measure System Ontology: The first step is to identify system components and boundaries. Enterprise architectural framework methodologies were reviewed, and one of the architectural frameworks was selected to identify the system components and boundaries. Enterprise architecture is a construction structure and a framework of a human endeavor. It is a holistic approach to the management and evolution of the enterprise. Several architectural frameworks exist today, such as the Open Group Architectural Framework (TOGAF), Model-Driven Architecture (OMG), and Department of Defense Architectural Framework (DoDAF). However, their application is limited and may not capture some types of system development. On the other hand, Zachman’s architectural framework is very flexible, thereby, it was selected for this study.

3.3. Zachman Architectural Framework

Zachman Architectural Framework (ZAF) is an enterprise framework invented by John Zachman for IBM in 1980, and it is in the public domain. The ZAF is used by Information Technology (IT) system developers to describe the IT system’s architecture. The ZAF is used to identify the needed components (artifacts) for architecture and how they relate to each other. The ZAF is an ontology, a theory of the existence of a structured set of essential elements of an object for which explicit expressions are necessary and perhaps even mandatory for creating, operating, and changing the “object.” The “object” could be an enterprise or a department, a value chain, a “sliver,” a solution, a project, a building, a product, a profession, or other subjects. According to Zachman, this ontology is derived from analogous structures found in the older disciplines of Architecture/Construction and Engineering/Manufacturing that classify and organize the design artifacts created to design and produce complex physical products (e.g., buildings or airplanes).

The ZAF uses a two-dimensional classification model based on six basic interrogatives and six distinct perspectives shown in

Table 1. Zachman Architecture Framework (ZAF) System Ontology Matrix.

	What	How	Where	Who	When	Why
Scope Business System Technology Detailed Actual	Models which fit the sub-context of the particular row and column						Planner Owner Designer Builder Contractors Workers

. The six interrogatives are what, how, where, who, when, and why. The six perspectives are planner, owner, designer, builder, implementer, and worker. These perspectives are related to stakeholder groups. The intersecting cells of the framework correspond to models that can provide a holistic view of the enterprise if documented.

To develop the system ontology, not all rows or columns need to be filled, as they are related to the system to be created. ZAF provides a view of the required essential components to construct a performance measuring system and how each component correlates to the others.

Table 2. Zachman Architecture Framework (ZAF) System Ontology Matrix for Project-Management Performance Measures [26].

	What	How	Where	Who	When	Why
Project Management Perspective	Project management performance level	Periodic performance measure of project management practice	Local government area of jurisdiction	Stakeholders: Citizen, City Council, City Manager	Annually or as needed	Improve credibility, performance reporting, transparency, accountability	Scope/ Planner
Investment	Performance level data, staff utilization, and development	Evaluation of performance, resource allocation, and investment	Department	Leadership (Department Head), Division Manager	Annually or as needed	To standardize processes and establish policy	Requirements/ Owner
Process Improvement	Maturity level, capability level, capacity level, resource management, system dynamic evaluation	Perform assessment survey, interview, observation, evaluation	Division	Division manager, project manager	Annually or as needed	Assessment of standard and policy implementations	Design/ Designer
Implementation	Performance Measure Methodology	Continuous data collection and monitoring	Department and divisions	Division manager, project manager	Annually or as needed	Process performance monitoring and improvement	Working System/Final Product
	Data	Process	Network	Organization	Timing	Motivation

shows the ontology of the project-management performance measurement system using the Zachman architectural framework. The first three rows describe the system from three perspectives: planner, owner, and designer; the last row describes the final product, the working system.

The intersecting cells of the framework correspond to the elements of the performance system. Each row in Table 2 includes information to describe a stakeholder perspective. The descriptive representation of the performance measurement system from each stakeholder perspective follows.

• First The first row is the scope of the system from the project management perspective. This row also describes the boundary of the performance measurement system:
  • What: the subject matter of the system, the project-management performance level.
  • How: the process to determine the project-management performance level. The process consists of periodic performance evaluations conducted annually or as desired by the leadership.
  • Where: the location or the network where the project management activities are conducted. In this case, it is within the local government area of jurisdiction.
  • Who: the stakeholder or the system’s owner: citizens, city council, and city’s upper management.
  • When: the performance level information is needed for strategic planning.
  • Why: the motivation behind the need to measure the performance of project management practices. The motivation is to improve credibility by improving reporting tools, transparency, and accountability of the organization.
• The second row is the perspective of the owner regarding the descriptive representation of the performance measurement system. It describes the investment requirements for the system.
  • What: input data needed for the performance system. The information is performance level data (existing or expected performance level), staff utilization, and staff development program.
  • How: the process to collect the data, in this case, through evaluating project management performance evaluation of resource distribution, and allocated investment for staff development.
  • Where: the location or network. The location is in the Public Works Department or Engineering Department.
  • Who: the stakeholder from the perspective of the owner is the user department. They are the department head and division manager.
  • When: Annual strategic planning or scheduled project-management performance evaluation.
  • Why: the motivation is to standardize the project management processes or establish a policy.
• The third row is the descriptive representation of the project management measurement system or process management from the system designer’s perspective.
  • What: the required data to perform the performance measurement. The data are the maturity level, the project manager’s capability, the project manager’s capacity, the system dynamic computer model, and resource management.
  • How: the data are collected through the assessment survey, interviews, observations, and periodic evaluation of the project management performance.
  • Where: the location is within the Public Works Department or Engineering Department
  • Who: the stakeholders are the division manager and the project manager.
  • When: Annually at the strategic planning session or as needed.
  • Why: the motivation is to assess the implementation of standards or policy.
• The fourth row is the descriptive representation of the final product or the working system implementation.
  • What: the project-management performance measure methodology for local government agencies.
  • How: the process consists of continuous data collection and performance monitoring.
  • Where: Location is at the Public Works Department or Engineering Department.
  • Who: the stakeholders involved are the division manager and the project manager.
  • When: Annually or as needed by the leadership or upper management.
  • Why: the motivation is performance improvement.

3.4. System Dynamics Performance Measurement Model

The ZAF framework provides a view of the essential components that are required to construct a performance measuring system and describes how each component correlates to the others. The ZAF framework is incorporated into the system dynamics model to map the overall scope and context of the system’s decision-making process. These two techniques when combined allow an enhanced comprehension of diagnosis processes and improvement [27,28].

System dynamics is used to develop a project-management performance measuring model and to further identify components that impact the system’s performance. The fundamental objective of using system dynamics is to gain an understanding of the system’s behavior. Each element’s behavior in the system is essential in assessing different actions at different parts of the system to accentuate or attenuate its behavioral tendency. The system dynamics model to measure project-management performance is depicted in Figure 1.

A system dynamic computer software tool, Vensim, was used to develop the project-management performance system model. The model variables were estimated based on the theoretical interpretation and commonly accepted values in the construction industry. The success of the model is based on whether or not the model can imitate the real-life event. Staff turnover is the primary contributing factor in any organization’s performance; therefore, staff retention should be the primary focus of leadership to improve performance levels. The real-life event considered in this model is the leadership (upper management) commitment to staff and project managers’ retention. The behavior of leadership commitment over time was reflected in both capability and capacity factors. The following are the variables included in the model:

3.5. Project Engineering Manager

The project engineering manager is the central focus of this performance model to measure project management practices; consequently, the longevity of the project engineering manager in the organization strongly influences performance. An organization with a high staff turnover rate struggles to successfully develop performance improvement programs because it must train new employees. Additionally, the level of intellectual property may not be maintained or improved at the organization. For each project engineering manager, the employment duration (turnover rate) is assumed to be two years; thereby, the termination rate was one project engineering manager every two years. The hiring rate is assumed as one project engineering manager annually, which is the typical hiring process rate for a local government agency. Whether an agency has one or more project engineering managers, the model looked at it as one organizational unit and used average values.

3.6. Quality Correction Factors

The concept of managing performance cannot be separated from the concept of quality management. The concept of quality management or managing for quality means to ensure product or service conformance to requirements. Managing performance parallels to managing quality since both, performance and quality, tie to the staff’s or project manager’s capacity and capability. Research indicated that 15–25% of all work performed consists of redoing prior work because products and processes were not perfect [29]. In the construction industry, commonly accepted construction change orders and time extensions could vary between 15% to 25%. These are human errors that could reduce the effectiveness and efficiency of an organization. Quality correction factors for effectiveness and efficiency are applied to the capacity and capability factors in calculating the performance level to account for the errors. The correction factors for capability are estimated at 0.85 and for capacity is 0.90.

3.7. Utilization

Utilization is obtained by dividing the ideal workload, as a number of daily projects, over the actual workload carried by project engineering managers, and it is reported in percentage. The ideal workload is determined by estimating that project activities consume approximately 1.3 h of the project engineering manager’s time daily for each project. The estimation is based on direct observations. On a regular working day, a project engineering manager ideally handles six projects. The actual workload is obtained from the project management survey indicated that a project engineering manager works on six to fifteen projects per day. Figure 2 is used as an example representation of historical workload data to develop a utilization curve as a function of the number of projects. The utilization ratio is also an indication of the commitment of top leadership to staff retention.

3.8. Knowledge Growth

Knowledge growth represents the commitment of the leadership to invest in the development of competencies for project engineering managers. Measuring the knowledge growth caused by investment in this area is difficult. One method to measure knowledge growth is through surveying at the end of a structured training program. However, an in-depth study to measure knowledge growth is not part of this study. Figure 3 is an example of representation of historical investment data to estimate the knowledge growth as a function of investment to facilitate the simulation process.

3.9. Capability Factor

The capability factor is loosely estimated by adding the initial capability, knowledge growth, and project manager, and then the result is multiplied by the quality correction capability.

Capability Factor (%) = (Initial Capability (%) + Knowledge Growth (%) + Project Manager) × Quality Correction Capability Factor

(1)

Initial Capability = Survey data of the average value of project manager knowledge of the Talent Triangle described in the PMBOK.

3.10. Capacity Factor

The capacity factor is loosely estimated by adding the value of the ratio of ideal workload over the actual workload (Figure 2) and knowledge growth (Figure 3). Then, the result is multiplied by the correction capacity factor.

Capacity Factor (%) = (Knowledge Growth (%) + Utilization (%)) × Quality Correction Capacity Factor

(2)

3.11. Project Management Maturity Process

The maturity level of the project management process was obtained from the responses collected through the survey. It is a value of the knowledge area maturity level of the project management process. The maturity level is based on the PMMM five-level maturity scale. The maturity level is then expressed as a percentage that ranges from 0 to 100.

3.12. Project Management Performance Level

Many earlier studies have shown that improved performance in an organization is achieved through their employees [30]; thereby, the organization’s project-management performance level was developed using PMBOK and quality management principles. The performance level metric was established by the following equation to capture the relationship between project management and quality management perspectives [31].

Performance Level (%) = Maturity (%) (wf1) + Capacity (%) (wf2) + Capability (%) (wf3)

(3)

Maturity = Survey data of knowledge area maturity level of project management processes.

Capacity = Number of projects that a project manager is capable of conducting over a given time

Capability = The ability of a project manager to complete the tasks.

wf = weight factor

In this simulation, the weight factors (wf) are 0.50, 0.25, and 0.25 for maturity, capacity, and capability components, consecutively. Weight factors vary for each agency or organization, and the weight factor’s determination is the subject of further research.

4. Case Study

A case study was conducted to demonstrate the applicability of the system dynamics method as a tool to measure the performance of project engineering management practices at local government agencies. The behavior of each performance system component was studied in three cities with local governments: The City of Sunland Park, the County of El Paso, and the City of El Paso. A brief description of each city with its corresponding staff profile follows.

4.1. The City of Sunland Park (NM)

As per the latest US Census Bureau in 2020, the City of Sunland Park’s population was approximately 17,121 people, with a population per square mile of roughly 1231 people. The median household income per the 2020 US Census Bureau is $33,342.00. It has an elevation of 1136 m (above sea level) with a latitude of 31.8092821 degrees and a longitude of −106.58396 degrees. Sunland Park is a city that lies in southern Dona Ana County, New Mexico, on Texas’s borders and the Mexican State of Chihuahua.

The staff profile in the City of Sunland Park is characterized as follows:

-

Project Managers: 1

-

Experience (Years): 6–10

-

License: None

-

PMI Certification: None

-

Workload/ Project Manager: 6–10 Projects

-

Typical Project Value: <$500,000

4.2. The County of El Paso (TX)

The County of El Paso encompasses eight towns/cities; Anthony town, Clint town, El Paso city, Horizon City, San Elizario city, San Elizario city, Socorro city, and Vinton village, with a population of approximately 846,477 people. The average household income as per the US 2020 Census Bureau is $48,292.00. El Paso County has a latitude of 31.8040 degrees and a longitude of −106.2051 degrees; it lies at an elevation of 1188 m above sea level, on the borders of New Mexico and the Mexican State of Chihuahua. It has a population of approximately 790 people per square mile.

The staff profile in the County of El Paso (Tx) is characterized as follows:

-

Project Managers: 5

-

Experience (Years): 6–10

-

License: 1 PE

-

PMI Certification: None

-

Workload/ Project Manager: 6–10 Projects

-

Typical Project Value: <$500,000

4.3. The City of El Paso (TX)

The City of El Paso has approximately 974,000 people, with a population per square mile of roughly 854.4 people. The latest US Census Bureau in 2020 reported that the median household income is $48,292.00. It lies at an elevation of 1188 m above sea level with a latitude of 31.8483649 degrees and a longitude of −106.43287 degrees. El Paso lies in El Paso County, Texas, on the borders of New Mexico and the Mexican State of Chihuahua. The Capital Improvement Department and the International Airport under the City of El Paso administration participated in the case study.

In the Capital Improvement Department, the staff profile is characterized as follows:

-

Project Managers: 22

-

Experience (Years): 6–10

-

License: 5 PE

-

PMI Certification: None

-

Workload/ Project Manager: 10–15 Projects

-

Typical Project Value: $5 M–$10 M

In the International Airport, the staff profile is characterized as follows:

-

Project Managers: 11

-

Experience (Years): 6–10

-

License: None

-

PMI Certification: None

-

Workload/ Project Manager: 10–15 Projects

-

Typical Project Value: $2 M–$5 M

4.4. Project Management Performance Simulations Scenario

To improve the chances of becoming a stimulus recipient, a local government agency adopts a plan to enhance its project engineering management performance. The city council asks the public works department to submit a funding request to enhance the project management performance of the department and to meet federal requirements. The director of the public works department needs to assess the current performance level of the department.

Three aspects were considered in the simulation of the project management performance: initial capability, number of project managers, and maturity process.

Table 3. Performance Model Input Variables for the Simulation.

Input Variables	City of Sunland Park	County of El Paso	City of El Paso—CID	City of El Paso—Airport
Initial Capability	67%	53%	80%	57%
Number of Project Managers	1	5	22	11
Maturity Process	20%	20%	40%	40%

summarizes the initial input variables from the simulations and survey responses of the local governments.

5. Results

Table 4. Project Management Performance Contributing Factors.

Primary Factors	Secondary Factors
Number of Project Managers	Utilization
Capability	Knowledge Growth
Capacity	Investment for Project Manager Development
Process Maturity Level	Project Manager Retention

shows the performance contributing factors, identified by the system dynamics model, that affect the project management performance level at local agencies

Eight contributing factors that affect the organization’s project performance level were identified through system dynamic analysis. These contributing factors were categorized into primary and secondary factors. The primary factors directly affecting the performance level included the number of project engineering managers, capability, capacity, and maturity level of the project management process. The secondary factors indirectly affecting the performance level were the project’s working-hours utilization, knowledge growth, investment for project manager development, and project manager retention.

In addition, Figure 4 shows the results of performance behavior simulation and its associated components. Figure 4a shows the behavior of the performance level, capability, and capacity over time for the City of Sunland Park. Figure 4b shows the results for the County of El Paso. Although the City of Sunland Park is a smaller agency, it has a higher performance level than the County of El Paso. Figure 4c shows the performance level as compared to the capability and capacity of the City of El Paso—CID. Figure 4d shows the performance level of the International Airport. The performance level of CID is higher than the Airport.

Figure 5 compares the evolution of the performance level of the City of Sunland Park, the County of El Paso, and the City of El Paso CID and Airport. The results showed that performance level differences are proportioned to the project manager’s initial capability and project-management maturity level. The project management processes for both the City of Sunland Park and the County of El Paso are at the initial level; therefore, performance levels primarily reflect the project manager’s ability.

The performance level upward tendency, from the beginning to the 4th year, is caused by the increased capability of the project engineering manager. The increased capability is caused by knowledge growth; the leadership commitment fostered the growth in knowledge of the project engineering manager. The knowledge growth decreased in the 4th year, even though leadership commitment continued. The reduction in capacity caused the downward tendency of the performance level after the 4th year. Increased workload caused a reduction in the utilization of working hours that ultimately resulted in a reduction in capacity.

In summary, the case study results showed that the City of Sunland Park and the County of El Paso performed at maturity level one, which is equivalent to 20%; the City of El Paso CID and Airport performed at maturity level two or 40%. These results are similar to the results of the study conducted by Grant and Pennypacker [32]. The maturity level is influenced by the amount of historical data to conduct the analysis. The most reliable method to assess the maturity level for a process is through a survey to validate the results.

6. Discussion and Validation

It is critical to recognize the different levels of leadership or upper management involvement in the factors identified as key performance variables in the study. In the primary factors, leadership may delegate the decision-making process to lower management levels because it does not involve a financial investment. In secondary factors, the direct involvement of top leadership is necessary to decide the amount of investment committed to utilization, knowledge growth, project management development, and retention strategies. It is required to determine the initial state of these parameters as input parameters when performing the simulation.

The change of performance factors and metrics over time can predict the future project management performance of an agency. It is also critical to recognize the influence of the top leadership commitment to these factors. In the primary factors, the top leadership commitment has indirect involvement in the project management decision-making process, although a direct involvement in the secondary factors.

The system dynamic performance model captured these three critical performance factors: the top leadership commitment, the ability of the project engineering manager, and the complexity of the project management processes. It was concluded that the increase in the number of project engineering managers has a minimum impact on the performance level. However, an increase in workload directly impacts the capacity; consequently, it also impacts the agency’s performance level. This behavior mimics real-life cases where the hiring of a project manager is based on the target workload. If the workload continues to increase, the effective utilization of working hours decreases due to the labor time available to manage the projects.

The case study demonstrated that systems dynamics could be applied in local agencies to conduct the project management performance evaluations. Leadership involvement, project management processes, and the agency’s project management ability are critical factors that influence the project management performance level. The project-management performance measurement system is based on the continuous collection of data for performance monitoring through assessment surveys, interviews, and observations. Data are required to assess the project-management process maturity level, project engineering manager capability, project engineering manager capacity, and resource management.

To validate the results of the case study and verify if the proof of concept is implementable in practice, the project management performance was further studied for the City of El Paso in the terms of capability, capacity, and maturity level. The City of El Paso was selected since the authors had direct access to project management information. In recent years, the City of El Paso was losing a project manager every two years; this condition hampered any effort to improve performance. Therefore, there was a need to assess the current project management knowledge situation and skills to identify and strengthen management areas of significant impact on the agency’s performance.

The City of El Paso was represented by a division manager from the Capital Improvement Department and a project manager from the International Airport. The Capital Improvement Department acts as the primary department in executing projects, and the International Airport is the “owner” of the projects. Project management knowledge and skills profiles were provided to determine the performance level. The profile of project management knowledge and skills was characterized by three essential talent categories:

• Technical Project Management is the technical aspect required to perform the role of a project manager (Scale from 0–100%).
• Leadership is the skills to guide, motivate, and direct a team to help an organization achieve its business goals (Scale from 0–100%).
• Strategic and Business Management is the skill required to enhance an organization’s performance and better deliver business outcomes (Scale from 0–100%).

To determine the project management performance level, there was a need to use weighting factors to establish priorities based on the goals of each agency. There are many methods available to determine the weighting factors, such as the weighted scoring method, decision matrix, and analytic hierarchy process. Based on ease of use for local governments, the prioritization matrix was selected in the study to develop weighting factors for validation. The prioritization matrix is a management and planning tool commonly used in business management to rank options.

6.1. The City of El Paso (TX): Capital Improvement Department (CID)

Project managers rely on their experience and judgment to perform the work. However, in 2018 the department introduced the project delivery manual to the project managers. The profile of project management knowledge and skills were characterized as follows:

-

Technical Project Management: 80%

-

Leadership: 80%

-

Strategic and Business Management: 80%

The performance factors calculated for the City of El Paso from thirteen projects per project manager were:

-

Maturity Level: 40%

-

Capacity: 46%

-

Capability: 80%

Table 5. Weighted Factor Matrix—City of El Paso.

Performance Factors	Improvement Cost	Complexity	Development Time	Best Management Practices	Number of Project Manager	Project Manager Experience	Training Budget	Total	Weighted Factor
Importance Score	8	8	7	6	7	8	4
Maturity Level	9	3	3	9	3	9	3	276	0.40
Capability	3	3	3	1	3	9	9	204	0.30
Capacity	3	3	3	1	3	9	9	204	0.30

shows the weighted factors obtained using the prioritization matrix.

Performance Level = Maturity Level × (0.40) + Capacity × (0.30) + Capability × (0.30) = 40% × (0.40) + 46% × (0.30) + 80% × (0.30) = 50%

(4)

6.2. The City of El Paso (TX)—International Airport

Knowledge and Skills Profile

In the International Airport, the profile of project management knowledge and skills were characterized as follows.

-

Technical Project Management: 70%

-

Leadership: 50%

-

Strategic and Business Management: 50%

Project managers rely on their experience and judgment in performing the work. This department utilized the same project delivery manual that was introduced by the Capital Improvement Department. The calculated performance factors are as follows.

-

Maturity Level: 40%

-

Capacity: 46%

-

Capability: 57%

Table 6. Weighted Factor Matrix—City of El Paso International Airport.

Performance Factors	Improvement Cost	Complexity	Development Time	Best Management Practices	Number of Project Manager	Project Manager Experience	Training Budget	Total	Weighted Factor
Importance Score	8	8	7	6	7	8	6
Maturity Level	9	3	3	9	3	9	3	282	0.38
Capability	3	3	3	1	3	9	9	222	0.31
Capacity	3	3	3	1	3	9	9	222	0.31

shows the weighted factors obtained using the prioritization matrix.

Performance Level (%) = Maturity Level × (0.38) + Capacity × (0.31) + Capability × (0.31) = 40% × (0.38) + 46% × (0.31) + 57% × (0.31) = 47%

(5)

The purpose of the validation process was to demonstrate the applicability of the methodology to determine the project management performance level of the agency.

Table 7. Summary of performance components and levels.

Agency	Number of Project Manager	Maturity Level	Maturity Weighted Factor	Capacity	Capacity Weighted Factor	Capability	Capability Weighted Factor	Performance Level
City of El Paso—CID	22	40%	0.40	46%	0.30	80%	0.30	50%
City of El Paso—Airport	11	40%	0.38	46%	0.31	57%	0.31	47%

summarizes the performance factors and performance levels of the participants.

The results indicated that local agencies perform between 47–50%. The City of El Paso CID at 50% and the City of El Paso—Airport at 47%. The performance level difference is small. The City of El Paso-Airport department’s performance level is lower because of the heavier workload assigned to the project managers and differences in the capability or skills of the project managers.

It is worth mentioning that a local agency should create specific criteria to establish weighted factors based on its own goals and priorities. The weighted factor calculation results obtained from the validation are similar to the weight factors adopted in the simulation for the simulation in the case study. In the simulation, the weight factors (wf) for maturity, capacity, and capability components were 0.50, 0.25, and 0.25, respectively.

7. Conclusions

The system dynamics approach used to develop project engineering management performance measures captured the complexity of project management in local government agencies. The model was based on multiple assumptions. Top leadership commitment was measured through investments in staff development and utilization, although further research should be conducted using historical data pertaining to the knowledge growth and investment areas.

• The system dynamic performance model identified three critical performance factors: leadership commitment, project manager ability, and project management processes. Furthermore, the performance factors’ behavior over time can be analyzed to predict an agency’s future management performance.
• Eight contributing factors that affect the organization’s project were categorized into primary and secondary factors. The primary factors were the number of project managers, capability, capacity, and maturity level of the project management process. The secondary factors were the project’s working-hours utilization, knowledge growth, investment for project manager development, and project manager retention. It is critical to recognize the influence of leadership commitment in these factors. In the primary factors, leadership has indirect involvement in the decision-making process, wherein leadership has direct involvement in the secondary factors.
• In the case study, it was found that an increase in workload directly impacts the capacity; consequently, it also affected the agency’s performance level, while an increase in the number of project engineering managers had a minimum impact on the performance level.

7.1. Contributions of the Systems Dynamic Method to Local Government Agencies

A system dynamics computer model was developed and used to conduct the performance management evaluation. For managers, the primary motivation for implementing the performance measurement system will be to improve the agency’s credibility by enhancing performance reporting tools, transparency, and accountability. The system dynamics model was developed toward this goal. The specific contributions of the methodology described in this paper are:

• Introduces local governments to a project-management performance measure approach based on a system dynamics method that serves as a framework for the standardization of the processes.
• Fosters leadership involvement to monitor performance and the development of project managers’ abilities.
• Emphasizes the importance of monitoring the project management processes, project managers’ capacity, and capability as the main performance factors that influence the agency’s project management performance.
• Assists in the strategic planning process by identifying investment priorities required to enhance project management performance.

These contributions demonstrate that there is added value for practitioners in the adoption of system dynamics for project management.

7.2. Limitations and Future Research

Establishing a relationship between the investment required for the development of core competencies in a local agency and the knowledge growth of the project manager is necessary to better estimate the increasing organization’s capability to manage engineering projects. Moreover, it is critical to analyze the relationship between the project manager’s workload and the utilization level to measure the overall capacity of the organization. Finding these relationships implies conducting more surveys for statistical analysis.

It shall not be construed that the system dynamics method described in this paper to measure project management performance is in the final form. Further development is necessary as data pertaining to knowledge growth and investment areas becomes available. Therefore, more case studies should be evaluated to allow cross-case comparisons of the results. Besides, future research should be conducted to expand the methodology to measure the management performance of government agencies that serve larger communities.