The Relationships between Organizational Factors and Systems Engineering Process Performance in Launching Space Vehicles

Abstract

The launch vehicle industry has long been considered a pioneering industry in systems engineering. Launch vehicles are large complex systems that require a methodical multi-disciplinary approach to design, build, and launch. Launch vehicles are used to deliver payloads—such as humans, robotic science missions, or national security payloads—to desired locations in space. Previous research has identified deficient or underperforming systems engineering as a leading contributor to launch vehicle failures. Launch vehicle failures can negatively affect national security, the economy, science, and society, thus highlighting the importance of understanding the factors that influence systems engineering in launch vehicle organizations in the United States. The purpose of this study was to identify and evaluate the relationships between organizational factors and systems engineering process performance. Structural equation modeling was used to develop a model of the relationships of these factors and test hypotheses. The results showed that organizational commitment, top management support, the perceived value of systems engineering, and systems engineering support significantly influence systems engineering process performance in the launch vehicle industry. Implications of this study for improving the performance of systems engineering in launch vehicle organizations are discussed.

1. Introduction

Launch vehicles are any rocket propelled systems used to deliver a payload to a desired location in space. They are used to transport spacecraft such as communications satellites, deep space robotic missions, and crewed space capsules for both private and government organizations. The National Aeronautics and Space Administration (NASA) and United States Space Force (USSF) are the major governmental users of launch vehicles in the United States (US). Many private telecommunications companies use launch vehicles to deliver satellites into orbit. Most private companies and governmental organizations do not own or operate the launch vehicles that they use. They instead outsource the development and operation of launch vehicles to private companies to deliver their spacecraft to space [1,2]. Spacecraft missions include communication, television services, science discovery, national security, and transporting humans. Satellite costs can range from several million dollars to several billion dollars [3]. Given the importance and costs of these satellites, the loss of satellites can have significant economic, scientific, societal, and national security impacts [4,5].

Spacecraft are most often lost or destroyed because of launch vehicle failures, thus indicating the need for the US Government and private satellite owners to minimize the risk of launch vehicle failure. The risk of launch failure is also a major concern for launch vehicle service providers, because a launch failure could detrimentally affect profits. Launch failures can also significantly affect the launch vehicle’s economic viability [4]. A study by J. Steven Newman, in 2001, examined 50 space flight failures and found that inadequate systems engineering contributed to all of them [6]. Newman’s study highlights the need to identify the factors affecting systems engineering in launch vehicle organizations.

Systems engineering is widely defined as a methodical interdisciplinary approach to design, build, operate, and manage complex systems [7,8,9,10,11]. In the context of launch vehicles, systems engineering is an interdisciplinary approach to design, manufacture, integrate, test, manage, and launch vehicles. Recognizing the criticality of systems engineering in the launch vehicle industry, in 2012, NASA added a systems engineering evaluation element to its launch vehicle certification policy. NASA’s launch vehicle certification policy is a plan to thoroughly evaluate the risk associated with a given commercial launch vehicle before trusting the launch vehicle to deliver a NASA spacecraft into orbit [12]. The US Space Force, another major user of commercial launch vehicles, follows a similar policy of evaluating the systems engineering of a launch vehicle provider as part of the launch vehicle certification [13].

Despite the substantial recognition that systems engineering is critical to the success of a launch vehicle, little research has examined the factors affecting systems engineering in launch vehicle organizations. Many launch vehicle failure studies have focused on the direct and proximate causes of failure, whereas little attention has been paid to systems engineering, including the underlying factors affecting systems engineering, in launch vehicle organizations. This study focused on filling that knowledge gap by identifying the underlying factors affecting systems engineering in US launch vehicle organizations. The next section of this paper presents a review of the literature on systems engineering and launch vehicle failures. The subsequent section presents a review of the methods. Analysis of the data collected from a survey is presented in the fourth section of this paper. The conclusions drawn from the data analysis are shown in the final section.

2. Literature Review

2.1. Systems Engineering and Launch Vehicle Failures

A large amount of literature has been published on launch vehicle failures; however, very little focus has been placed on how systems engineering has influenced these failures. A study by J. Steven Newman at NASA examined 50 spaceflight failures and identified systems engineering as a contributor to all of them. In Newman’s study, 41 of the 50 spaceflight failures were launch vehicle failures [6]. A second study conducted by NASA in 2005 focused on engineering lessons learned and systems engineering applications in space systems [14]. The results of the study confirmed many of Newman’s findings. Other launch vehicle failure studies by Chang [15], Harland and Lorenz [16], Isakowitz and Hopkins [17], and Leung [18] did not specifically identify systems engineering as contributing to failures; however, the identified causes of failure were associated with systems engineering. Although several studies have identified systems engineering or elements of systems engineering as contributors to launch vehicle failures, studies identifying the factors that influence systems engineering performance in launch vehicle organizations are scarce.

2.2. Value of Systems Engineering

The value of systems engineering has been a matter of much debate, because numerous complex systems were developed before the systems engineering discipline was established. Over the years, many studies have attempted to identify the value or effectiveness of systems engineering. One of the earliest studies was completed by Werner Gruhl at NASA. The study examined 32 NASA projects and aimed to identify the relationship between the investment in systems engineering and cost overruns. Projects that invested more in systems engineering were found to have smaller cost overruns, whereas projects that invested less in systems engineering had larger cost overruns [19,20]. In another study surveying many prominent engineering firms worldwide, including Boeing, Northrop Gruman, Lockheed Martin, Airbus, and Ford Motor Company, most survey respondents indicated that project costs were reduced when systems engineering was applied [21].

In the late 1990s, Boeing performed a study of three similar projects that they were conducting. The first project was to be completed without the use of systems engineering, whereas the other two used systems engineering. All three projects started simultaneously, and the two projects that used systems engineering were completed in half the time of the project that did not use systems engineering [20,22]. A similar study on automobile prototypes by Kamal Malek has shown that the development of prototypes is completed much faster when a systems engineering approach is used [23].

Eric Honour has performed extensive studies on the value of systems engineering, showing that systems engineering improves development quality, and the quality of systems engineering affects development quality [20]. Honour performed a second study on the return on investment on eight systems engineering activities: mission definition, requirements engineering, systems architecting, system implementation, technical analysis, technical management, scope management, and verification and validation. The results indicated that each activity significantly influenced project success [24].

2.3. Project Success Factors

Numerous studies have examined the critical success factors for projects. Project critical success factors are considered elements that lead to cost, schedule, quality, and customer satisfaction. The presence of these critical factors improves the likelihood of the project’s success [25,26,27,28,29,30,31]. An extensive literature review revealed that some of the most frequently found critical success factors are support from top management, a current and well-thought-out plan, good communication, adequate staff, competent leadership, sufficient and well allocated resources, effective control and monitoring, and adequate organizational structure and culture [26].

2.4. Key Systems Engineering Best Practices

Systems engineering best practices often result from lessons learned in numerous projects. The knowledge gained through experience in past projects is critical to improving systems engineering capabilities [9,10,14,32,33]. After a literature review of numerous studies on the best practices and lessons learned in systems engineering, the following themes emerged:

• Systems engineering infrastructure is critical to project success [9,10,14,32,33].
• Requirements should be unambiguous, up-to-date, and vetted with project stakeholders [9,10,14,32,33].
• An effective systems engineering plan should be implemented as early as possible [9,10,14,32,33].
• Failure to adhere to good systems engineering could lead to cost and schedule overruns [9,10,14,32,33].
• People are the primary resource for successful systems engineering [9,10,14,32,33].
• Communication is critical for successful systems engineering [9,10,14,32,33].

The following systems engineering handbooks and standards were compared: INCOSE Systems Engineering Handbook [9], NASA Systems Engineering Handbook [10], MIL-STD-499C—Systems Engineering [34], IEEE-1220—Application and Management of the Systems Engineering Process [35], EIA-632—Process for Engineering a System [36], ISO/IEC 15288—Systems and software Engineering System Life Cycle Process [37], and Capability Maturity Model Integration for Development [38]. The themes regarding best practices were found in each of these systems engineering standards and guidelines.

2.5. Systems Engineering Support (SES)

For the purposes of this study, systems engineering support is considered the framework provided by an organization for the systems engineering process. Systems engineering support includes planning, control and assessment, communication, tools and infrastructure, as well as the personnel provided by the organization to execute the systems engineering process. Systems engineering support is a major element enabling the systems engineering process in organizations [9,32,39].

2.5.1. Planning

Most of the systems engineering standards reviewed identified planning as a key element for implementing systems engineering in an organization. Leading systems engineering organizations such as INCOSE, Systems Engineering Body of Knowledge (SEBoK), NASA, and IEEE consider planning to be one of the most important aspects of implementing systems engineering. Systems engineering experts agree that planning should be implemented and well documented as early as possible. An adequate systems engineering plan should describe the systems being developed, the technical management process, the tailoring of the systems engineering life-cycle approach to be used, and integration of the technical disciplines into the systems engineering process.

2.5.2. Control and Assessment (C&A)

Systems engineering standards also identify control and assessment of the systems engineering process as another key element of implementing systems engineering. The purpose of control and assessment is to determine the effectiveness of the systems engineering process in meeting cost, schedule, and technical performance requirements for the systems under development [10,32,36,37]. Most systems engineering researchers believe that systems engineering assessment should evaluate the project’s performance on cost and schedule [19,24,40,41,42]. However, a study by Bruff, in 2009, on linking systems engineering best practices to project cost and schedule performance has shown a strong correlation between systems engineering best practices and project performance [43]. Another study by Componation in 2009 sought to relate the success of NASA projects to the systems engineering process [44]. Componation found only a correlation between cost and schedule, but not the systems engineering process.

2.5.3. Communication

Good communication has been identified as a critical factor for project success [14]. Communication has also been identified as a major systems engineering best practices. Systems engineers are expected to communicate throughout various levels of the launch vehicle organization. Communication is a key characteristic of high technology organizations [45].

2.5.4. Tools and Infrastructure (T&I)

Another systems engineering enabler identified through the literature is tools and infrastructure [32]. The instruments provided by an organization to execute the systems engineering process are considered systems engineering tools. For instance, a tool could be a process or software application. The infrastructure is described as the framework or environment in which the systems engineering tools are applied. Adequate tools and infrastructure enable good systems engineering [9,32,39].

2.5.5. Personnel

Numerous researchers have identified adequate personnel as a key systems engineering best practice and lesson learned. The personnel is the human capital provided for the systems engineering process: “the people are the primary resource for successfully developing a system” [46]. A sufficient number of competent personnel has also been identified as a critical success factor for projects [26].

2.6. Systems Engineering Culture

The culture of an organization forms the basis for the systems engineering process [43]. Culture, as described by systems engineering researchers, comprises the values, beliefs, and common practices of the organization [32,47,48,49]. Oppenheim has described systems engineering culture as “a pervasive mental state and bias for systems engineering methods applied to problem solving across the development lifecycle and all levels of enterprise processes” [39]. A healthy systems engineering culture promotes effective systems engineering and encourages system thinking. According to SEBoK, a healthy systems engineering culture is strong in the areas of leadership, trust and moral, cooperation and teamwork, empowering employees, confidence in processes, and job security.

2.6.1. Top Management Support

Research on organizational leadership and culture has shown that the leaders of an organization impart their values, assumptions, and beliefs on the organization [50,51]. The top or senior management of an organization is typically responsible for setting the organization’s goals, directions, and strategies. This aspect is also true in launch vehicle organizations. Numerous researchers have identified top management support as a critical success factor for projects [25,26,27,28,29,30,31].

2.6.2. Organizational Commitment

Another key element of organizational culture is organizational commitment: i.e., how loyal or committed employees are to the organization. Several studies in fields similar to systems engineering, such as safety culture, have shown that organizational commitment significantly affects process performance [52,53,54]. Organizational commitment is a key element of the systems engineering culture of an organization.

2.7. Literature Review Summary

The literature review identified key factors for implementing and enabling systems engineering. Factors critical to the success of systems engineering were also identified. Multiple studies have identified factors critical in implementing, enabling, and ensuring the overall success of systems engineering. However, no studies examining the relationships of these factors on the performance of the systems engineering process in launch vehicle organizations were found. This study specifically addresses this knowledge gap by focusing on answering the following questions:

• RQ1: What factors affect the performance of the systems engineering process in launch vehicle organizations?
• RQ2: What are the effects of systems engineering culture on the performance of the systems engineering process in launch vehicle organizations?
• RQ3: What are the effects of systems engineering support on the performance of the systems engineering process in launch vehicle organizations?

3. Materials and Methods

Systems engineering in launch vehicle organizations can be very complex because of the complexity of the launch vehicles themselves. Other systems engineering studies have shown that identifying the factors that influence the systems engineering process can be equally complex [20,21,22,24]. However, sufficient information from the literature review and prior experience allowed us to hypothesize a model describing the relationships of the factors affecting the performance of the systems engineering process in launch vehicle organizations (Figure 1). The constructs of this study were developed using the research conceptualization process. Through this process definitions and theories were applied to each construct of the study [55]. Establishing preliminary construct definitions were critical for providing a starting point for the inquiry of this study [56].

3.1. Research Hypotheses

Systems engineering deficiencies have been identified as a contributor to launch failures [6,14,16,17,18]. Understanding the factors that affect systems engineering in launch vehicle organizations is critical to improve the performance of the systems engineering process. By studying these factors, insight could be gained to improve the performance of the systems engineering process, thus increasing the chance of a successful space launch. To test the relationships of the constructs identified in this study, the research hypotheses in

Table 1. Research Hypotheses.

Hypothesis	Description
H1	Organizational commitment has a direct effect on systems engineering process performance.
H2	Top management support has a direct effect on systems engineering process performance.
H3	The value of systems engineering has a direct effect on systems engineering process performance.
H4	Systems engineering support has a direct effect on systems engineering process performance.
H5	Organizational commitment has a direct effect on systems engineering support.
H6	Top management support has a direct effect on systems engineering support.
H7	The value of systems engineering has a direct effect on systems engineering support.
H8	Systems engineering support mediates the relationship between organizational commitment and systems engineering process performance.
H9	Systems engineering support mediates the relationship between top management support and systems engineering process performance.
H10	Systems engineering support mediates the relationship between the value of systems engineering and systems engineering process performance.

were developed.

3.2. Research Design

Studies that investigate critical factors are often qualitative and conducted via survey instruments [26,57]. Specifically, the primary data collection tool of systems engineering studies is survey instruments. A survey instrument was developed based on the construct of this study. The survey consisted of two parts: the first part collected demographic information, and the second part collected information on the study variables. The survey statements addressing the dimensions of the study variables are shown in

Table 2. Study Dimensions and Survey Statements.

Top Management Support	1	Senior management strongly supports the systems engineering process.
	2	Senior management believes that a strong systems engineering process adds value to the organization.
	3	Senior management communicates its support for systems engineering to the organization.
	4	Senior management supports skipping a systems engineering step if it will help the organization save money.
	5	Senior management supports skipping a systems engineering step if it will help the organization meet schedule goals.
Value of Systems Engineering	1	Practicing good systems engineering reduces launch vehicle cost.
	2	Practicing good systems engineering reduces launch vehicle schedule delays.
	3	Practicing good systems engineering improves launch vehicle performance.
Organizational Commitment	1	I am willing to put in a great amount of effort beyond what is normally expected in order to help my organization be successful.
	2	I speak highly of this organization to my friends and family as a great place to work.
	3	I find that my values and my organization’s values are very similar.
	4	I am proud to tell others that I work for this organization.
	5	I really care about the fate of this organization.
	6	This is the best launch vehicle organization to work for.
Planning	1	My organization has a documented plan on how systems engineering should be implemented.
	2	My role in systems engineering is clearly identified.
	3	My organization identifies how all technical engineering disciplines are integrated.
	4	There was a systems engineering plan in place at the beginning of launch vehicle development.
Communication	1	My organization emphasizes effective communication between departments such as design, manufacturing, and operations.
	2	My organization emphasizes effective communication among the various engineering disciplines (disciplines such as avionics, structures, propulsion, environments, software, etc.).
	3	Management has an open door policy for discussing systems engineering issues.
	4	There is good communication about systems engineering items in the workplace.
	5	Documenting detailed rationale for technical decisions is highly encouraged.
Tools and Infrastructure	1	My organization follows an established systems engineering model such as Waterfall, V Model, Spiral, Agile, or Iterative.
	2	I have appropriate tools to successfully execute systems engineering in my organization.
	3	Appropriate training and guidance are provided for the systems engineering tools.
	4	The systems engineering tools provided are regularly used by my organization.
Personnel	1	My organization has employees whose sole responsibility is to facilitate the systems engineering process.
	2	My organization has the right people involved to successfully implement systems engineering.
	3	My organization has a sufficient number of people to successfully implement systems engineering.
	4	My organization understands the skills needed to successfully execute systems engineering.
	5	My organization provides access to systems engineering training.
	6	Training provided by my organization has prepared me well for my systems engineering duties.
Control and Assessment	1	There are performance measures or metrics used to evaluate the performance of systems engineering in my organization.
	2	Technical reviews are held at regular intervals to evaluate the performance of the systems engineering process, such as system requirements reviews, preliminary design reviews, critical design reviews, etc.
	3	All stakeholders are informed of the project’s progress.
	4	Resources allocated to a project are evaluated to determine whether they are adequate to achieve project success.
Systems Engineering Process Performance	1	Applying a thorough systems engineering process in my organization reduces the number of launch vehicle manufacturing problems.
	2	Applying a thorough systems engineering process in my organization reduces the severity of launch vehicle manufacturing problems.
	3	Applying a thorough systems engineering process in my organization reduces the number of launch vehicle integration and test problems.
	4	Applying a thorough systems engineering process in my organization reduces the severity of launch vehicle integration and test problems.
	5	Applying a thorough systems engineering process in my organization reduces the number of launch vehicle problems during flight.
	6	Applying a thorough systems engineering process in my organization reduces the severity of launch vehicle problems during flight.

.

Each construct had three or more corresponding survey statements. The survey statements were developed and approved as a part of doctoral research at the University of Central Florida, Orlando, FL, USA [11]. The survey was approved by the Institutional Review Board. The survey statements corresponding to the study variables were measured on a five-point Likert scale ranging from 1 = strongly disagree to 5 = strongly agree.

3.3. Data Source

This study required a population with intimate knowledge of the systems engineering processes in launch vehicle organizations. For this reason, launch vehicle organizations in the US were targeted. In addition, US launch vehicle organizations were specifically targeted to avoid any language barriers. Surveys were distributed to launch vehicle organizations throughout the US. The surveys targeted various people throughout the organizations involved in the systems engineering process. The target population included systems engineers, project managers, discipline and component level engineers, technical managers, program managers, and any other people involved in the systems engineering process. No restrictions were placed on the sizes of the organizations targeted for the survey. A minimum of 200 valid survey responses were required to complete a valid structural equation modeling (SEM) study [58,59,60].

3.4. Data Analysis Method

In this study, IBM SPSS Statistics 25 and Amos 26 were the software tools used for data analysis. IBM SPSS Statistics 25 was used to format, code, and run descriptive statistics analysis on the survey data. IBM SPSS Amos 26 was used to develop and test the measurement and research models, and identify significant factors. SEM was used to test the research hypotheses. The study followed the data analysis process below. The data analysis process is also illustrated in Figure 2.

3.4.1. Validate Survey Responses

Each survey was examined to ensure that no data were missing. Surveys with missing data would have led to different samples sizes for each variable dimension during data analysis, and would not have been suitable for regression and correlation analyses [61,62]. Invalid surveys were reviewed to ascertain whether any information could be inferred from the missing responses. Invalid surveys were retained for recordkeeping but were not included in the data analysis.

3.4.2. Descriptive Statistics Analysis

Descriptive statistics on each variable, including data such as mean, frequency, and standard deviation, was calculated. This analysis provides a basic understanding of the characteristics of the data collected. It is also useful in testing assumptions for confirmatory factor analysis (CFA) and SEM. Descriptive statistics plots were also visually examined to determine whether any information could be gained from the visual representation of the data.

3.4.3. Test Assumptions

Most statistical data analyses assume that the data are normal, linear, and homoscedastic, without multicollinearity. Verifying these assumptions is critical to account for the assumption violation in remainder of data analysis. In cases in which the data are not normal, bootstrapping could be used during SEM analysis. If assumptions testing shows evidence of multicollinearity, variables are removed from the model to satisfy that assumption.

3.4.4. Confirmatory Factor Analysis

CFA was used to evaluate the relationships between the observed dimensions and the study variables. The measurement model was a model of the relationship between the observed variable and corresponding survey statements (variable dimensions). A CFA was used because prior knowledge of the relationships existed [63]. CFA is an essential part of SEM. As part of the CFA, Cronbach’s alpha is calculated to determine the reliability of the survey statements for each corresponding survey variable during model identification. A Cronbach’s alpha value greater than 0.70 is considered adequate for a CFA [64]. To complete CFA, the following steps were performed [60]:

• Specify the model to be evaluated.
• Check whether the model is identified.
• If the model is identified, determine whether the model fit is adequate.
• If the model fit is not adequate, revise the model to improve the model fit.
• If the model fit is adequate, validate the measurement model.

The goodness of fit is a measure of how well the hypothesized model fits the data. Numerous model fit indices may be calculated to determine goodness of fit. The goodness of fit indices and criteria selected for this study are shown in

Table 3. Goodness of Fit Indices.

Model Fit Index	Criteria
$\mathrm{Chi}-\mathrm{square}({\chi }^2$)	Low
Degrees of freedom (df)	>0
Probability value (p)	>0.05
$\frac{{\chi }^2}{df}$	<5
Goodness-of-fit index (GFI)	>0.90
Tucker–Lewis index (TLI)	>0.90
Comparative fit index (CFI)	>0.90
Root mean square error approximation (RMSEA)	<0.08
90% Confidence interval (lo90—hi90)	<0.05–0.08
Probability of closeness of fit (Pclose)	>0.5

.

3.4.5. Hypothesis Testing

SEM is a robust statistical method that incorporates factor analysis, canonical correlation, and multiple regression [65,66]. SEM illustrates the relationships between observed variables and latent variables by combining various model types. This feature of SEM provided a method to quantitatively test the hypothesized structural model.

4. Results

More than 500 surveys were distributed to launch vehicle organizations throughout the US. A total of 203 valid survey responses were received, which yielded a response rate of approximately 40%. Furthermore, 27.7% of the respondents identified as systems engineers, which made up the largest percentage of systems engineering roles. The remaining 72.3% of respondents were distributed across other roles in the systems engineering process, thus addressing any sampling bias concerns. Approximately 40.9% of respondents had more than 20 years of experience. The number of projects worked on by the respondents varied, and the largest proportion of respondents (31%), by a small margin, had worked on 30 or more projects. Most of the respondents’ systems engineering experience (71.4%) was in the launch vehicle industry. Survey participants were from small, medium, and large organizations, and 60.1% came from large organizations. Participants had a range of levels of government involvement, and most (71.4%) came from government agencies. The details of the profiles of respondents are shown in

Table 4. Data Set (n = 203).

Characteristics		Frequency	%
Role	Engineering support	7	3.4
	Manager	19	9.4
	Design Engineer	5	2.5
	Test Engineer	4	2
	Manufacturing Engineer	3	1.5
	Operations Engineer	7	3.4
	Component Engineer	13	6.4
	Analyst	25	12.3
	Integration Engineer	14	6.9
	Project Manager	28	13.8
	Systems Engineer	78	38.4
Years of Experience	1–5 years	31	15.3
	5–10 years	29	14.3
	10–15 years	30	14.8
	15–20 years	30	14.8
	20 years or more	83	40.9
# of Projects	1–5 projects	47	23.2
	6–10 projects	49	24.1
	10–15 projects	26	12.8
	15–20 projects	18	8.9
	20 or more projects	63	31
LV Experience	Not all LV experience	58	28.6
	All LV experience	145	71.4
Organization Size	Small	14	6.9
	Medium	67	33
	Large	122	60.1
Government Involvement	Little	2	1
	Some	15	7.4
	A lot	41	20.2
	Government agency	145	71.4

.

Before CFA and SEM, multivariate assumptions of normality, linearity, and homoscedasticity were verified. When residuals were plotted and examined, the data were found to be in violation of the assumptions. Bootstrapping is a widely accepted technique to use for SEM when assumptions are violated [60,64]. Although research by Byrne [63] has suggested that validity can still be achieved when the data do not follow a normal distribution, we nevertheless used the bootstrapping technique to bolster analysis results.

Multicollinearity was the next assumption tested. This test identifies one or more observed variables that are strongly correlated. When two observed variables are highly correlated, they are essentially measuring the same thing and could cause problems later in the data analysis process. SEM researchers have suggested that a correlation above 0.90 between variables is considered evidence of multicollinearity [66]. No observed variables were found to have a correlation greater than 0.90; therefore, no observed variables were recommended for removal on the basis of multicollinearity.

4.1. Model Validity Test Results

Measurement models were developed and evaluated for each latent variable of this study. CFA was completed for each of the measurement models. The measurement models were revised until adequate model fit was achieved. After achieving adequate model fit of each of the measurement models, we constructed the hypothesized model (Figure 3). Final model fit results are shown in

Table 5. Model Fit of Hypothesized Model.

Model Fit Index	Criteria	Initial Model
$\frac{{\chi }^2}{df}$	<5	1.764
TLI	>0.90	0.904
CFI	>0.90	0.915
RMSEA	<0.08	0.061

.

To test the validity of the latent variables in the study, we used CFA. In SEM, validity is considered the degree to which the model results accurately measures the construct that it was designed to measure [60,64,65,67]. Two types of validity were evaluated: convergent validity and discriminant validity. Convergent validity is when evidence shows sufficient overlap of variables measuring a particular construct. The CFA results and convergent validity results are shown in

Table 6. Confirmatory Factor Analysis and Validity Test.

Construct	Indicator	Factor Loading	α	CR	AVE
Systems Engineering Process Performance	SEPP1	0.753	---	0.947	0.748
	SEPP2	0.848	---
	SEPP3	0.941	---
	SEPP4	0.896	---
	SEPP5	0.871	---
	SEPP6	0.880	---
SE Support	Communication	0.881	0.740	0.949	0.790
	Planning	0.868	0.726
	Personnel	0.977	0.794
	T&I	0.811	0.802
	C&A	0.900	0.732
Top Management Support	TMS1	0.973	---	0.933	0.874
	TMS2	0.894	---
Organizational Commitment	OC2	0.834	---	0.889	0.730
	OC4	0.938	---
	OC5	0.783	---
Value of SE	VSE1	0.840	---	0.773	0.537
	VSE2	0.749	---
	VSE3	0.586	---

NOTE: All factor loadings were statistically significant at the p < 0.001 level.

.

Three items were used to evaluate convergent validity: factor loading (item reliability), average variance extracted (AVE), and construct reliability (CR). Factor loadings above 0.5 are considered strong evidence of convergent validity [64]. All indicators in this study had factor loadings of 0.586 or greater. An AVE value greater than 0.5 shows strong evidence of convergent validity [64]. All constructs in this study had an AVE of 0.537 or greater. The final method of examining convergent validity is calculating CR for each construct. Researchers generally accept that a CR greater than 0.7 shows strong evidence of convergent validity [58]. Each construct in this study had a CR of 0.773 or greater. All three indicators, as shown in Table 6, showed strong evidence of convergent validity for the hypothesized model.

Discriminant validity is a necessary component for validating constructs. High discriminant validity is present when a construct uniquely measures a phenomenon that other constructs do not measure. A rigorous test for identifying discriminant validity is performed by comparing a construct’s square root of the AVE with its correlation with another construct. Discriminant validity is present when the correlation is less than the square root of the AVE [64]. As shown in

Table 7. Discriminant Validity.

Constructs	VSE	TMS	OC	SES	SEPP
VSE	0.733
TMS	−0.048	0.934
OC	0.282	0.262	0.857
SES	0.126	0.548	0.540	0.890
SEPP	0.401	0.345	0.507	0.577	0.868

Factor correlations. Square root of AVE on the diagonal. VSE = value of systems engineering, TMS = top management support, OC = organizational commitment, SES = systems engineering support, SEPP = systems engineering process performance.

, all correlation pairs were less than the associated square root of the AVE; therefore, the hypothesized model showed adequate discriminant validity.

4.2. Structural Equation Modeling Results

After successfully validating and achieving adequate model fit for the measurement model of the latent constructs by using CFA, we evaluated the structural model. The hypothesized structural model is illustrated in Figure 3. The structural model regression estimates are shown in

Table 8. Structural Model Regression Estimates.

	Std. Estimate (β)	S.E.	C.R.	p
OC ← Number of Projects	0.164	0.026	2.557	0.011
VSE ← Career Level	0.189	0.029	2.872	0.004
SES ← OC	0.457	0.042	9.209	***
SES ← TMS	0.466	0.030	9.374	***
SEPP ← OC	0.162	0.063	2.572	0.010
SEPP ← VSE	0.334	0.051	6.403	***
SEPP ← SES	0.459	0.071	7.682	***

*** p < 0.001.

. Goodness of fit of the model was also evaluated, and the revised structural model satisfied all model fit criteria. The model fit results are shown in

Table 9. Revised Structural Model Goodness of Fit.

Model Fit Index	Criteria	Final Revised Model
$\mathrm{Chi}-\mathrm{square}({\chi }^2$)	low	11.279
Degrees of freedom (df)	>0	11
Probability value (p)	>0.05	0.420
$\frac{{\chi }^2}{df}$	<5	1.025
Goodness-of-fit index (gfi)	>0.90	0.985
Tucker–Lewis index (TLI)	>0.90	0.999
Comparative fit index (CFI)	>0.90	0.999
Root mean square error approximation (RMSEA)	<0.08	0.011
90% Confidence interval (lo90—hi90)	<0.05–0.08	0.00–0.075
Probability of closeness of fit (Pclose)	>0.5	0.775

.

4.3. Hypothesis Testing Results

After achieving adequate goodness of fit of the revised hypothesized structural model, we completed hypothesis testing by using SEM. The hypothesis test results are shown in

Table 10. Hypothesis Testing Results.

Hypothesis	Description	β	t	Supported?
H1	Organizational commitment has a direct effect on systems engineering process performance.	0.162	2.572 *	Yes
H2	Top management support has a direct effect on systems engineering process performance.	0.067	1.583	No
H3	The value of systems engineering has a direct effect on systems engineering process performance.	0.334	6.403 **	Yes
H4	Systems engineering support has a direct effect on systems engineering process performance.	0.459	7.682 **	Yes
H5	Organizational commitment has a direct effect on systems engineering support.	0.457	9.209 **	Yes
H6	Top management support has a direct effect on systems engineering support.	0.466	9.374 **	Yes
H7	The value of systems engineering has a direct effect on systems engineering support.	0.028	0.679	No
H8	Systems engineering support mediates the relationship between organizational commitment and systems engineering process performance.	Partial Mediation
H9	Systems engineering support mediates the relationship between top management support and systems engineering process performance.	0.186	2.889 *	Yes
H10	Systems engineering support mediates the relationship between the value of systems engineering and systems engineering process performance.	No Mediation

β = standardize path coefficient, t = critical ratio, * p = < 0.001, ** p < 0.01.

. H1, H3, H4, H5, H6, H8, and H9 were all supported by the data. The model provided sufficient information to test the hypotheses.

5. Discussion

The purpose of this study was to identify and evaluate the factors affecting systems engineering process performance in launch vehicle organizations in the US. A secondary goal of this study was to develop a model explaining the relationships among systems engineering support, systems engineering culture, and systems engineering process performance. The model developed in the study identified factors that influenced systems engineering performance. Because the model showed a good fit and allowed the hypotheses to be tested, both goals were achieved.

The hypotheses that organizational commitment has a direct (H1) and an indirect (H8) effect on systems engineering process performance (H1) were supported by the data. Organizational commitment also directly affected systems engineering support (H5). Thus, the more committed employees are to the organization, the greater the systems engineering process is perceived to perform. In addition, the more committed employees are to an organization, the more support is provided to systems engineering. Several studies have shown that organizational commitment is critical to the culture of an organization, and organizational culture is the context for which the systems engineering process is executed. Therefore, it is reasonable to conclude that greater commitment among launch vehicle organization personnel is associated with greater support provided to the systems engineering process, thus resulting in better process performance.

The hypothesis testing results did not support top management support directly influencing systems engineering process performance (H2). Indicating that top management support could not be used as a predictor for systems engineering process performance. However, the data confirmed that top management support directly affects systems engineering support (H6). The hypothesis testing also indicated that top management support has an indirect effect on systems engineering process performance (H9), as mediated by systems engineering support. Numerous studies have shown that top management support is a key factor in organizational culture and a critical success factor in many projects. Therefore, we conclude that as the recognition of senior management’s support for systems engineering increases, the organization’s support for systems engineering increases, thereby increasing the performance of the systems engineering process.

The value of systems engineering was also identified as a factor affecting systems engineering. The data supported the hypothesis that the perceived value of systems engineering directly affects systems engineering process performance (H3), thus implying that as the perceived value of systems engineering increases, the performance of the systems engineering process increases. Therefore, perceiving and understanding the value of the process would be likely to enhance the performance of the process.

The results of this study suggest several implications for launch vehicle organizations and for systems engineering in general. Prior research has shown that systems engineering affects cost, schedules, technical performance, and customer satisfaction [9,10,20,32,41]. The present study identified that systems engineering process performance in launch vehicle organizations is influenced by organizational commitment, the value of systems engineering, top management support, and systems engineering support. Improving these factors could improve systems engineering process performance. For launch vehicle organizations, improving systems engineering process performance could decrease technical issues and launch failures, and ultimately result in time and cost savings.

After recognizing these factors influencing systems engineering, leaders of an organization could target these areas for improvement to enhance the performance of their organization’s systems engineering process. For example, the value of systems engineering has a significant direct effect on systems engineering process performance. The leader of an organization could choose to implement protocols to educate and ensure that the individuals in the organization recognize the value of systems engineering. This would in turn improve the performance of the organization’s systems engineering and ultimately improve the launch vehicle organization’s ability to meet cost, schedule, and technical performance goals.

6. Conclusions

Launch vehicles are very complex systems that often require equally complex systems engineering approaches to design, build, test, and launch them. Diverse systems engineering approaches can be implemented in different ways. Regardless of the systems engineering style or model, this study identified factors that affect systems engineering process performance in launch vehicle organizations. Deficient or low performing systems engineering has been identified in several studies as a leading contributor to launch vehicle failures. Given the significant influence of launch vehicle failures on human life, the economy, science, and national security, identifying factors that could enhance systems engineering process performance in launch vehicle organizations is imperative.

A systems engineering process performance model was developed to examine the relationships among perceived top management support, organizational commitment, the value of systems engineering, systems engineering support, and systems engineering process performance in US launch vehicle organizations. SEM and hypothesis testing results showed that organizational commitment, the value of systems engineering, and systems engineering support directly or indirectly influence systems engineering process performance. While top management support indirectly influence systems engineering process performance.

The data set for this study was provided voluntarily from participants in US launch vehicle organizations. Survey responses that assessed the latent variables of this study are based on participant’s perceptions. Responses may have been based on what participants believe is ideal and not what was observed or experienced in respective organizations. The survey also did not take in to account risk tolerance of the individual organizations. A difference in risk tolerance could potentially lead to different perceptions of the systems engineering process.

Despite these limitations, the results of this study identified factors that present implications for systems engineering and U.S. launch vehicle organizations. Past research has shown that systems engineering significantly impacts technical performance, cost, and schedule [7,9,10,19,20,21,22,24,41]. Prior research has also identified deficiencies in the systems engineering process as a major contributor to launch vehicle failures [6,16,17,18]. Improving the factors that affect systems engineering process performance could improve technical performance of the system, reduce cost, improve schedule, and reduce the number of launch vehicle failures.

This study’s results underscore the important roles of organizational factors in systems engineering process performance. Organizational leaders should emphasize their support for systems engineering, highlight the value of systems engineering, enhance systems engineering support, and bolster employee commitment to organizations. Improving on these factors may increase the ability of organizations to meet cost, schedule, technical performance, and customer satisfaction goals, thus ultimately improving launch vehicle success.