Project Success Criteria Evaluation for a Project-Based Organization and Its Stakeholders—A Q-Methodology Approach

Abstract

The criteria that define project success change from one project to another, also from organization to organization, making success contextual for both the project organization and its stakeholders. This paper proposes a way to bridge this gap between what project success means to an organization and to its stakeholders in the context of Research and Development (R&D) projects. To achieve this, the available literature on project success has been analyzed to convert the different aspects identified into tangible units, allowing us to define and analyze the success criteria of a project in different dimensions. Subsequently, using Q-Methodology, which allowed us to determine among subjective opinions of Project Managers (PMs) of a project-based organization and their internal stakeholders, we will determine which criteria, within the previously identified dimensions, they consider as the most important for the success of a project, aiming to identify common success criteria that can be measured and controlled in the projects. Achieving the project goal, customer satisfaction regarding the quality of the activities, and knowledge generation turned out to be the most important criteria for PMs and stakeholders.

1. Introduction

Project management has different characteristics in the public and the private sector, not just due to a clear and on occasion opposite focus, but also practices, tools, resources, stakeholders, expertise, to name a few [1]. In companies, the incentives to bring a project to fruition are based on the measurable commercial results of the businesses, while for universities, it is the maximization of measurable results derived from research in the form of articles, theses, or patents [1].

Nevertheless, when collaboration between the two sectors is pursued, in the management of common Research and Development (R&D) projects carried out in intermediate Knowledge and Technology Transfer Organizations (KTTOs) close to the Technology Readiness Level (TRL) 5 to 7, a series of endemic problems tend to appear [2]. Sometimes, those problems result in a low success rate of the projects and waste of available resources for R&D in companies and at universities. However, these intermediate KTTOs are instruments far too fundamental and essential within the innovation process to be dismissed [3]. Therefore, new ways in which to manage, measure, and analyze before, during, and after these projects, but also to know the projects’ success factors and to conduct them under these premises, are required [4].

KTTOs have proved to be a satisfactory means to enclose the public and private sector and in this way help local economies [5]. The Advanced Manufacturing Center for Aeronautics (CFAA, by its initials in Spanish) is a prominent example of the university–company model, which implements R&D projects and innovative projects to generate knowledge, new methods, and technology for their partners, mainly in aero-engine system sectors, which benefit from the priceless experience achieved throughout the development of more than 300 projects and more than 200 scientific contributions in the last 4 years [6], to increase the future opportunities of the CFAA and its partners [7]. In addition, working on TRL 5–7 [8] guarantees quick knowledge transfer, which promotes mutual benefits to companies and universities from a micro- up to a macro-economic point of view, so that universities obtain funding to conduct their research and to train their staff that enables them to respond positively to the demands of the labor market [9].

The success of the projects at the CFAA has a significant effect on future opportunities of the organization and the improvement of aeronautic technology in the region [10]. However, the lack of control, measurements, and research about evaluation of project success can be identified as an improvement opportunity in this Center.

It is worth mentioning that the success criteria by which the CFAA has been used up to now was that a project was considered successful if it could be delivered to the customer and the customer was satisfied with the outcome, mostly because the involvement of the project owner was high during the project development. This way of measuring project success can be improved, according to Munns and Bjeirmi [11], as the measurement of project success is only conducted at the end of the project life cycle, when project management outcomes are available and it is convenient to measure. It also goes hand in hand with Drucker’s definition [12] mentioned as “Effectiveness” or “doing the right things”. In the context of the definition, projects are not judged by their efficient use of resources but by the way in which the organization asks itself the question: “Does it work?”. This, however, leaves out success criteria that are aimed at ensuring the sustainability of the system, making the best use of available resources, emphasizing the most important aspects for the stakeholder and for the organization [13].

However, a safeguard can be made in that the organization that owns the project is the one who must have previously studied the different success criteria of the project they are commanding. For example, if the technical improvement requested can be viable in their productive environment, or if the productive techniques used can be sustainable and applicable in environments other than the experimental one, or if the variables measured and that guarantee the success in the management of the project (thanks to the fulfillment of the objective) are valid, applicable, and expandable in their organizations. Naturally, the organization in charge of carrying out the project can help to determine the answer to some of the questions raised, however, the holistic experience is held by the project owner and not by the organization carrying out the project.

There are numerous ways to define project success, and each of these ways differs based on the kind and scope of the project [14,15]. Typically, project success is described as the fulfillment of some externally perceived criteria [16]. Criteria, however, refer to a rule or standard by which something is assessed [17]. Project success is traditionally assessed based on the three major criteria of the so-called “iron triangle”: cost, time, and quality (or scope) [18]. Moreover, even though these ideas are distinct but related to one another, success factors and success criteria have been used synonymously in project management literature. The collection of circumstances and events that help a project to succeed are known as success factors [17], and success criteria are the successful outcomes of projects and are the parameters by which success is measured [19].

Although the desired benefits might be stated in the benefit-management plan and business case of a project, current standards do not define how success or failure criteria will be determined [20]. Without performance metrics for success, an organization cannot be effective since it is impossible to know whether the right things are being achieved. Even with initially defined success criteria, the question remains if a project can succeed in achieving goals that it was not intended to reach, since the pace of change increases and organizations encounter environments which are usually described by volatility, uncertainty, complexity, and ambiguity [21]. Even if the project goals are achieved at the end of the project, how can we measure the success of that project if we do not even agree on a definition of it in the context of project management [22,23,24,25]?

The criteria for defining project success change from one project to another, from organization to organization, and even within the same project, making success contextual to both the project organization and its stakeholders [26]. The identification and management of the project success criteria play a crucial role in achieving project success in organizations. Project managers (PMs) of the organization and internal stakeholders of the projects are active and crucial actors in identifying, evaluating, and contributing to improving project management practices in the organization.

According to research, stakeholders may have various ideas of what makes a project successful, both in terms of how important the criteria are and how the project’s achievements compare to the criteria [27,28]. According to Davis [29], the perceptions of success by stakeholders are significant, as are the perceptions of important criteria and actual performance.

The need to control the project and the outcomes that are generated during the project life cycle is of vital importance to the organizations involved in the project. An approach is based not only on the success of the project management or on the fulfillment of time or cost constraints. It is necessary to analyze from the early stages of the project the quality of the work completed, and thus be able to analyze whether these results go hand in hand with the objectives not only of the project but also of the stakeholders. A more holistic understanding of project success can be achieved by measuring success throughout the project life cycle and including stakeholders in these measures of effectiveness [11,30].

We have been able to observe the relationship between the perception of project success for stakeholders and for PMs (or their organizations), and, despite some theoretical examples [29,31,32], there is still a need to explore what are the success criteria affecting both stakeholders and the organization in the development of public–private collaborative R&D projects in KTTOs. Therefore, the present research attempts to answer the following research questions (RQ):

• RQ1: What are the most important success criteria in public–private collaborative R&D projects, according to PMs and internal stakeholders?
• RQ2: What are the different subjective perspectives according to project managers and internal stakeholders?

This paper is organized as follows: we start with a literature review necessary for the description of the concepts and dimensions of project success criteria and their impact on the evaluation of a project and how they can affect the decision-making process of PMs. This is followed by a description of the research design. The results of the study are then described, and finally, the paper ends with a summary of the conclusions that can be drawn from the study.

2. Literature Review

Projects aim to deliver benefits in various ways to an organization. These desired benefits are usually formulated in the early stages of the project in association with a benefit-management plan that documents the performance reviews and evaluation of the success of the project [33]. Once the project objectives are established, strategies are defined and tactics are implemented to execute them in order to achieve the results and enable the organization to achieve the desired benefits defined and agreed upon for the project. The relationship between project success and different aspects of project management is being addressed in a growing number of studies. According to Web of Science, more than 220 research articles, proceeding papers, reviews, and book chapters have been published during the last 10 years demonstrating that, e.g., project success remains as a vibrant school of thought, as do the earlier definitions, measurement scales and dimensions, and assessment techniques that Pinto and his colleagues developed [34]. Mir and Pinnington [23] analyzed that project management performance is correlated to project success, and that KPIs and staff are the most influential aspects of project success. Joslin and Müller [35] have also found that there is a positive relationship between the project management methodology used in a project and project success, showing that access to a comprehensive project management methodology and the ability to know which of the elements can be applied to any given project represent almost 22.3% of the variation in project success. Nixon et al. [36] also found that the performance of the project leadership has been cited as a critical success factor determining the success or failure of a project. Finally, Carvalho and Rabechini [37] verified the significant and positive impact of project sustainability management on project success dimensions, among others.

2.1. Project Success

Project success as the heart of project management, and the factors that affect it, is a commonly discussed topic in research in project management and it is therefore among the top priorities of PMs and further stakeholders. The understanding of success has changed over the years, with different focus points, as depicted in Figure 1.

Success factors are defined as elements needed to be successful. They are independent variables that can be influenced to enable success. These elements include circumstances, facts, and influences [17,34]. Success factors should be identified and analyzed, if these are in place, or if actions can be undertaken, to enable and strengthen them to reach the level required to achieve success.

Due to the complexity and uncertainties which projects naturally face by their characteristics, it is difficult to stay within all given constraints and reach the initially desired targets. However, a “project can be a success even though it takes more time or [it] is more expensive than initially expected” [38,39]. To effectively measure success, success criteria must be defined upfront and reviewed and adapted continuously, since the environment can change, or new stakeholders may emerge, or even the point of view of a stakeholder can change unforeseeably with connected benefits, or simply to ensure justification [40]. If the company’s environmental factors or organizational process assets are altered, success criteria can be revised through formal change-management processes or projects can be closed prematurely, which can also be considered a success, as no resources are wasted. Furthermore, the output of the evaluation of the successfulness of a project can differ from the point of view of the stakeholders. While a project might favor one stakeholder (internal or external), others can perceive the project to their disadvantage and are therefore dissatisfied based on different or even contrary perspectives, interests, and objectives [41]. The viewpoint, as seen, can be dependent on different backgrounds, such as cultures, industry, organization, nationality, gender, or personality.

Another element that makes the success criteria of a project particularly valuable is that it is time-dependent [26]. The results of the evaluation might be completely different from one day to another, independently if the project is still underway, or after its closure [26,42,43,44]. Priorities, needs, the point of view, or the result of an assessment can change quickly due to rapid changes in the environment [21,40]. If the project is evaluated months or years later, the assessment then should be strictly circumscribed to the project baseline and the defined assessment of its performance factors for the output (deliverables) during its lifecycle, outcome, and benefits achieved with the outputs in the defined environment.

Regarding R&D projects, measuring project success has become a fundamental concern for managers and executives in the last decades, and as a result, the issue has been extensively debated in literature [45]. However, determining whether an R&D project is successful is another subtle matter and a challenging task, as reaching top performance does not necessarily correlate with being successful. R&D projects are inherently complex, whit several dependent phases that make it even more difficult to determine their success factors.

A common approach is to decompose the success into measurable units of project success and project management success [43]. Other approaches differentiate even more in detail and use further parts as product success and business success. While project success relates to the objectives of the project plan, product success relates to meeting the product requirements and is further “reflected by use, satisfaction, and effectiveness […] in benefiting intended users” [46]. Business success is determined by the business strategy and is achieved when the funding organization has realized the expected benefits through the means of the project, as defined in the benefit-management plan.

In the end, project success cannot be expressed binarily as a “success” or a “failure”, since the output might be a success, but the desired benefit could not be reached. Furthermore, a project that is closed prematurely because it is not viable, desired, or worthwhile anymore due to changes in the environment may not be considered a failure [47]. Consequently, if areas of the evaluated level are perceived as a failure, it is important to identify the root causes, which can lay in planning failure (the gap between what was planned and what was really accomplished) and/or actual failure (the gap between what was achieved and what was accomplished) [48].

Each project is unique, and therefore specific criteria are needed to do justice to that uniqueness. However, general success and failure criteria enable the comparison of projects [46]. Nonetheless, because project success can mean different things to different stakeholders, a common definition of project success for the individual project, as well as who, when, and how to measure it, must be determined and documented [24]. The success criteria need to be specified for project management, project activities, the output, the outcome, the benefit, and the business value [49,50]. This distinction is necessary since operating project outputs, regardless of results, provide organizational benefits.

Success management can be conducted at several levels, complementing each other to the top. Furthermore, the operation has its part in the success realization of the project portfolio of the organization, and consequently, the whole picture must be addressed. Although Figure 2 aims to show relationships, the success of each level is determined by the success criteria set forth and may not be entirely complementary in practice because different puzzle parts must be balanced. As a result, satisfactorily finishing all the project’s activities does not mean that the program was a success. For this research, we focus on the area of project management. As a result, only this area and the operation area are covered, because projects typically produce deliverables that are then put into operation. In the future, more research into the other displayed areas should be undertaken.

2.2. Project Success and Stakeholders

The concept of project success and stakeholder involvement has evolved over the years. Davis provides an extensive, detailed, and exceptional analysis [29], from the 1970s, orienting the concept of success to the operational side, tools, and techniques [30] where communication with stakeholders was lacking at a general level [29] to the shift in the 1980s to examine the technical aspects of the project and how they related to the client organization [51], to conclude that success is dependent on the perception of success of the multiple stakeholders involved in the project and the time at which that success is measured [52]. This was the beginning of recognition of stakeholder involvement in project development as a Critical Success Factor (CSF) [53], taking the first steps towards understanding and recognizing the importance of project success from a stakeholder perspective in the 1990s [11], to finally recognizing that success is dependent on both internal and external stakeholders [54].

This story has been used by Turner [55] to present some necessary, but not guaranteed, conditions for project success, such as that the success criteria should be agreed with the stakeholders before the start of the project and continuously reviewed throughout the project, or that a collaborative relationship should be maintained between the stakeholders and the project manager, considering the project as a partnership, that the project manager should have the flexibility to deal with unforeseen events in the project, that the project owner should provide guidance on how the project should be carried out, and finally, that it is vitally important that the project owner takes an interest in the realization and development of the project.

Another important and related aspect is determined by correct identification and classification of stakeholders [28], and how according to this classification each stakeholder should be asked about specific issues according to their role in the project, as each one will analyze the project in particular according to their expertise.

In short, the role of stakeholders has gained importance over the years, evolving from a position of an observer to a crucial involvement in the success of the project [56], allowing them to have first-hand knowledge of the project development, to contribute their point of view in the decisions and to be an important part in determining the success of the project. When the perception of the success of the project from the point of view of the organization and the stakeholders is united, the success of the project is not automatically guaranteed, but nonetheless, the chance that the project will go ahead according to the initial objectives is much greater, making it crucial to agree from the beginning on the definition of the success criteria and the points of the project which will be evaluated as the project develops.

Furthermore, the measurement of project success is a controversial issue. According to De Wit, A. [26], it is illusory to objectively measure project success if one wants to take into account the objectives of all stakeholders throughout the project life cycle and at all levels of the management hierarchy. However, having a holistic project measurement is not the only problem, according to Millhollan and Kaarst-Brown [57]. Some of the factors that contribute to project success materialize while the project itself is being executed. Moreover, according to the same authors, some of the factors that contribute to the success of the project, both in the perception of the project manager and the stakeholders (perceived project success [58]), may not materialize until long after the project has been completed [59]. In addition, the success factors that are predefined for the project may conflict with each other during the project life cycle, as the fulfillment of some constraints may have a negative impact on the satisfaction of stakeholders’ requirements.

A way to overcome this issue is to separate the dimensions of project success into several components: the measurement of project management success in terms of the iron triangle, project success understood as the fulfillment of the overall project objectives, and project success aligned with the project outcome, creating not only a dependent relationship between project management and project objectives, but also an evaluation after the project has been completed [60,61].

2.3. Project Success Dimensions

Success criteria are accepted and dependent variables, including principles and standards by which anything can be judged and defined [17,34]. These criteria should be evaluated under six important elements that guarantee a “consistent, high-quality evaluation within a common framework”, according to the OECD. These elements are relevance, coherence, effectiveness, efficiency, impact, and sustainability [13]. Since success is dependent on the area which is inspected, these parts must be addressed individually, since although one area might be perceived as a failure, the other areas can still be perceived as successful.

Consequently, it is crucial to define the success criteria for different dimensions. Our research was limited to Project Management, Delivery Activities, Deliverables, and Operations according to Figure 2. These dimensions were chosen because of their relevance to projects and include the different stages of project development, and as mentioned before, those dimensions are where projects typically produce deliverables that are then put into operation.

2.3.1. Project Management Success Dimension

The main objective of the criteria summarized in this dimension is to allow the analysis of the project manager’s performance in implementing the project plan. These criteria were formulated in 14 statements (numbers 1–14), as presented in

Table 1. Project Management Success Criteria.

No.	Project Management Success Criteria	Reference
1	Completed within defined and agreed budget	[41,62]
2	Return on Investment of the project	[63]
3	Knowledge generation regarding project management (e.g., tools, techniques, approaches, processes)	[64]
4	Publications regarding project management (e.g., tools, techniques, approaches, processes)	[64]
5	Customer satisfaction regarding the management of the specific project	[34]
6	Project management processes were conducted within the organization’s quality standard	[65]
7	Resources for project management activities were mobilized and used as planned	[51,66]
8	Attitude towards risks (risk tolerances)	[67]
9	Risk value (impact) of suffered threads (unknown unknown)	[68,69]
10	Project goal was achieved	[51]
11	Completed within defined and agreed scope	[41,62]
12	Reputation of the organization has increased	[66]
13	Reputation of the Project Management Office (PMO) has increased	[66]
14	Completed within defined and agreed schedule	[41,62]

.

2.3.2. Delivery Activity Success Dimension

The objective of the project result delivery layer focuses on the project activities and processes necessary to create the results. These criteria were formulated in 11 statements (numbers 15–25), as presented in

Table 2. Delivery Activity Success Criteria.

No.	Delivery Activity Success Criteria	Reference
15	Resources for delivery activities were mobilized and used according to planned productivity measures	[66]
16	The activities required to produce the deliverables have a good reputation	[66]
17	Knowledge generation regarding project activities (e.g., tools, techniques, approaches, processes)	[70]
18	Publication regarding project activities	[64]
19	Client satisfaction with the quality of the activities required to produce the project deliverables	[71]
20	Delivery activities were conducted within the organization’s quality standards	[65]
21	The way the deliverables are created contributes to sustainability	[64]
22	Process improvements were identified (idea/knowledge generation)	[72]
23	Process improvements have been applied, resulting in beneficial results	[7,73]
24	Technology transfer	[63]

.

2.3.3. Deliverable Success Dimension

This dimension describes the outcome of the project, and the success criteria set out here aim to verify whether the outcome of the project is in line with the purpose agreed with the stakeholders. These criteria were formulated in seven statements (numbers 26–32), as presented in

Table 3. Deliverable Success Criteria.

No.	Deliverable Success Criteria	Reference
26	Completed within defined and agreed budget	[72]
27	Return on Investment of the project	[64]
28	Knowledge generation regarding project management (e.g., tools, techniques, approaches, processes)	[64]
29	Publications regarding project management (e.g., tools, techniques, approaches, processes)	[49]
30	Customer satisfaction regarding the management of the specific project	[64]
31	Project management processes were conducted within the organization’s quality standard	[64]
32	Resources for project management activities were mobilized and used as planned	[64]

.

2.3.4. Operations Success Dimension

Projects create unique outputs which are then transferred into operations, where they should create the defined and agreed outcomes, impacts, and the desired value. Since outcomes and impacts might not be always in the favor of all stakeholders, the impact can be either perceived as a benefit or a disbenefit. Nevertheless, its success criteria need to verify that the operation ensures that the product is operated in an appropriate way, where it is fit for use [63]. These criteria were formulated in seven statements (numbers 33–39), as presented in

Table 4. Operations Success Criteria.

No.	Operations Success Criteria	Reference
33	Operation stays within the defined budget	[41,62]
34	Operation delivers the desired outcomes (as the fundamental business objective (the fulfillment of the project objective) and that have been developed in accordance with the core competencies of the organization)	[64]
35	Operation delivers the desired impacts (expected benefits evaluated for the project portfolio as a whole)	[64]
36	Operation delivers the desired value (the performance measures in the conduct of operations)	[71]
37	Actual use by the customer	[51,74]
38	Workplace security	[69]
39	Downtime (e.g., maintenance, repair)	[69]

.

2.4. Success Measurements

To measure success, specific tolerances need to be defined. For each assessment area, success and failure criteria should be defined to gain a better understanding, avoid wrong expectations, and foster a clear and understandable communication. This can be achieved, for example, by using control charts.

To achieve this, these criteria and tolerances must be collected from all (key) stakeholders from the beginning of the project to analyze if the requirements are viable, desirable, and achievable [40]. All stated and agreed target values and limits need to be SMART (Specific, Measurable, Achievable, Realistic, and Timebound). After that, these requirements and expectations need to be balanced and managed [75,76], since different levels or areas could conflict with each other or be contrary. Then, problem solving must be applied to balance these or to identify higher priorities and to find, in the best case, acceptable solutions for all stakeholders.

Several factors can influence these processes, which are highly dependent on the situation. Furthermore, success criteria can be defined with a weighting factor or as primary or secondary [48]. The different views on the defined success criteria from (key) stakeholders should be collected and deviations should be discussed. In case parties cannot understand each other or disagree with some points which would create conflicts, a third (neutral) party can be involved, which can moderate to reach a consensus as well as conduct an assessment from an independent standpoint, or to accept knowingly an unsatisfied stakeholder with expectable consequences. All of this must be contained in the project management methodology as a formal step to control and assure that the right criteria will be considered.

Moreover, as described before, success is not a one-time definition. It evolves and develops over time. Consequently, the definition must be continually revised if needed. Along with that, it needs to be defined when and how the success can be measured. Another point which influences the assessment of success or failure is the person who conducts the assessment [24,62]. As stated before, general success criteria need to be developed for an organization to be able to compare projects. Nonetheless, distinct criteria need to be defined to do justice to the uniqueness of each project [46].

Another complementary step to assess the success of projects is to define the Key Performance Indicators (KPI) to control if the project is on track [45]. The definition and measurement of these KPIs must go hand in hand with the success criteria defined in the organization to be able to measure the performance of a project with real criteria.

2.5. Q-Methodology

Q-Methodology is one of the oldest statistical methods, originally created for psychoanalysis to understand humans’ “unconscious mental processes” [77]. Based on the history of this methodology, the goal is to find the statistical explanation of humans’ “mind” [78]. The Q-Methodology is known as a statistical tool for understanding the personal view of individuals on a specific matter, which is mostly based on the reality and pleasure–pain principles created by Sigmund Freud [77,79]. Sigmund Freud explored the pleasure–pain principle (originally in German “Lust–Unlust”), which describes the distribution of the given information based on the subconscious wishes of reality. Reality creates circumstances to the external world based on decisions made and is also closely aligned with consciousness, which is a significant point of every decision [77]. Unfortunately, reality usually differs from humans’ desire, and even an individual can disagree with the real situation in the environment. This influences subconscious processes and must be considered while analyzing results of the methodology. As stated before, Q-Methodology is not only used for psychology research but is also extended to a variety of experiments that require statistical data based on the personal point of view of the participants.

Q-Methodology is a semiquantitative technique that can help to identify stakeholders’ view on a specific topic. This methodology demands that a set of statements, a so-called Q-set (n), must be distributed within a range, i.e., ranking grid, from a given P-Sample (number of respondents) [79]. The original version of Q-Methodology consists of a set of 48 statements (Q-set (48)) which must be ranked on a grid within the range from −4 to +4, where −4 is equal to “un-pleasure” and +4 is equal to “pleasure” [78].

To be able to identify different subjective perspectives on project success from PMs and stakeholders for a public–private research organization, the study deployed Q-Methodology as the research strategy. Q-Methodology has shown its usability in the context of project management research; for example, Silvius et al. [80] use Q-Methodology to investigate the consideration of sustainability aspects in the decision-making processes of PMs, concluding that the consideration of sustainability principles is underrepresented compared to the triple-constraint criteria. According to Brown [81], Q-Methodology adds to a PM’s techniques and tools by making it possible to learn what stakeholders think about non-specified requirements, providing stakeholders and PMs more information for troubleshooting project threats. Cuppen et al. [82] used Q-Methodology to contribute to proactive risk mitigation and to reveal hidden perspectives on industrial projects, leading to better project management. Mardaras et al. [83] used it to investigate whether organizations in R&D environments have antifragile characteristics. Finally, Brown [84] provides a foundation for the systematic study of subjectivity, a person’s viewpoint, opinion, beliefs, and attitude.

3. Research Design

This paragraph presents the research strategy and research design of the study. As the literature review showed that success is a multidimensional concept and that a clear understanding of how PMs integrate the project success concept into their decision making is lacking in literature, the nature of the study is explorative.

As shown in the Conceptual Framework (Figure 3), this paper proposes a way to bridge the gap between what success means to an organization and to its stakeholders in the context of an R&D project conducted in a public/private collaboration on a KTTO. To achieve this, literature on project success was analyzed to convert the different aspects identified into tangible units, allowing us to define and organize the success criteria of a project in different dimensions. Subsequently, using Q-Methodology, the PMs of the KTTO and their main stakeholders were asked about which criteria, within the previously identified dimensions, were most important to them, which allowed us to determine the main success criteria of an R&D project for a KTTO and its stakeholders.

3.1. Q-Methodology Approach

In a Q-Methodology study, respondents are typically presented with a sample of statements about a given topic, known as the “Q-set”. The “P-set” of respondents is asked to rate the assertions from their point of view, using a quasi-normal distribution (Figure 4). The respondents disclose their subjective point of view [85] or personal perspective [86] by “Q-ranking” the assertions.

The Q-Methodology traditionally intends to give a picture of the perspectives that exist (the variety of perspectives) among the population, rather than analyzing the level of support for those perspectives among the population (the balance of perspectives). It also relies on purposive sampling and smaller sample sizes. The fact that there is a person who is assumed to have a different point of view is enough reason to include him or her in the sample. Correlation between personal perspectives then implies the existence of similar viewpoints or segments of subjectivity [87]. Q-factor analysis provides information on similarities and differences in the viewpoint on a specific subject by correlating people.

For this research, only internal stakeholder perspectives (a short characterization of the internal stakeholders can be found in

Table 5. The CFAA’s Internal Stakeholders.

Stakeholder Classification	Description	Characteristics	Quantity
Type A	Principal members	Founding Partners	2
Type B	Full members	Machines manufacturers	8
Type C	Full members	Components manufacturers	14
Type D	Limited members	Components/software/machines manufacturers	68
Collaborating partners	Limited members	Institutes, Associations, Sectorial clusters	8
University	Full members	University of the Basque Country and several research groups	1

) were collected to identify the applicable success criteria for R&D Centers, which in turn may apply for the CFAA.

To define the applicable success criteria, the Q-Methodology procedure with the subsequent steps were followed:

• Reviewing the existing success criteria of the CFAA.
• Defining new criteria, based on the literature.
• Performing a trial run of the Q-sort.
• Revising statements for the final Q-sort with the CFAA’s internal stakeholders.
• Perform Q-sorts with the selected stakeholders.
• Assessing gathered data by using the software from Aiora Zabala [88].
• Interpreting results (factors) from the software.

The first step of the defined procedure was to evaluate existing criteria of success of the project in terms of relevancy to the current research and revising unclear statements. To assess the criteria developed by the previous research at the CFAA, brainstorming was used as a method for eliminating, extending, or changing the unclear criteria. The case study was conducted by members of the CFAA’s Project Management Office.

As mentioned previously, the original version of Q-Methodology consists of 48 statements [78]. In this case, the grid was reduced to 25 cells to reflect the current number of statements. Further statements could have been identified; however, most of the participants were overwhelmed by many options, since this type of experiment had never been conducted before at the CFAA. Subsequently, 25 statements were generated and a table with the range from −4 to +4 was formed. The meaning of the ranking was slightly adjusted so that +4 represents “the most important criteria” and −4 represents “the least important criteria” in terms of their own perspective as an employee or internal stakeholder of the CFAA. As it is crucial that the distribution reflexes their own view and not the organizational viewpoint, this issue was pointed out to the participants.

To ensure that the defined Q-set is understandable, and the formulations are unambiguous, a trial version of the Q-Methodology survey was conducted. In the trial evaluation of the new criteria, six people from different positions in the CFAA participated. Before the trial run was conducted, the general principle of Q-Methodology was explained and a short introduction of the performed research was provided. The trial run was conducted in groups of two, to enable discussions, since people often struggle to express their thoughts loudly. Since the first interviews were conducted on a trial basis and the aim was to understand whether all statements are clear and easy to understand, it was not necessary to apply the procedure individually. By the end of the editing the statements, 30 final improved statements were prepared for the final Q-Methodology interviews.

3.2. Statements

A literature review of the main success criteria for each of the dimensions proposed was carried out. This understanding forms the foundation for the empirical part of our study, in which we explore how PMs consider these dimensions of success in their decision making. These criteria were found thanks to a literature review and were defined through semi-structured interviews with the organization’s project team, who assessed each of the selected criteria by the formulated questions of the OECD [13], and to ensure that the criteria to be evaluated were not decontextualized from the organization.

In the study, the respondents were asked to rank the 30 statements divided among different categories according to the PMBoK [72] on a score sheet, as in Figure 4, about how important each of the different success criteria were for them during the execution of a project (

Table 6. Final Q-set.

Item	Statements	Success Dimension	Category
1	Attitude towards risks (risk tolerances)	Project Management	Risk Management
2	Complete within defined and agreed budget	Project Management	Cost Management
3	Complete within defined and agreed scope	Project Management	Scope Management
4	Complete within defined and agreed schedule	Project Management	Time Management
5	Customer satisfaction regarding the deliverables	Deliverable	Quality Management
6	Customer satisfaction regarding the management of the specific project	Project Management	Project Management
7	Client satisfaction with the quality of the activities required to produce the project deliverables	Delivery Activities	Quality Management
8	Degree to which the deliverables meet their intended purpose	Deliverable	Quality Management
9	The activities required to produce the deliverables have a good reputation	Delivery Activities	Stakeholder Management
10	Delivery activities were conducted within organization’s quality standard	Delivery Activities	Quality Management
11	Knowledge generation regarding project activities (e.g., tools, techniques, approaches, processes)	Delivery Activities	Knowledge Management
12	Knowledge generation regarding project management (e.g., tools, techniques, approaches, processes)	Project Management	Knowledge Management
13	Project goal was achieved	Project Management	Scope Management
14	Project management processes were conducted within organization’s quality standard	Project Management	Quality Management
15	Publications regarding project activities (e.g., tools, techniques, approaches, processes)	Delivery Activities	Knowledge Management
16	Publications regarding project management (e.g., tools, techniques, approaches, processes)	Project Management	Knowledge Management
17	Reputation of the organization has increased	Project Management	Stakeholder Management
18	Reputation of the Project Management Office (PMO) has increased	Project Management	Stakeholder Management
19	Resources for delivery activities were mobilized and used according to planned productivity measures	Delivery Activities	Resource Management
20	Resources for project management activities were mobilized and used as planned	Project Management	Resource Management
21	Return on Investment of the project	Project Management	Cost Management
22	Risk value (impact) of suffered threads (unknown unknown)	Project Management	Risk Management
23	Workplace Security	Operations	Risk Management
24	The deliverables, in terms of their design and creation, are adequate in terms of direct sustainability impacts	Deliverable	Sustainability
25	The deliverables, in terms of design and creation, are adequate in terms of indirect sustainability impacts	Deliverable	Sustainability
26	The deliverables meet official standards (e.g., ISO)	Deliverable	Quality Management
27	The deliverables meet the defined quality criteria	Deliverable	Quality Management
28	The product is characterized as sustainable	Deliverable	Sustainability
29	The way the deliverables are created contribute to sustainability	Delivery Activities	Sustainability
30	Workplace Safety	Operations	Risk Management

). The statements were related to aspects to be considered as success criteria for the projects carried out in the organization. The “umbrella question” for the statements was formulated as “how important for the success of the project is to/the/that”, with each statement completing this sentence. For example: “how important for the success of the project is the attitude towards risk (risk tolerances)”.

3.3. Data Collection

Data collection was carried out through structured, individual interviews to obtain the personal perspective of each participant in the experiment and to receive their opinion on the success criteria of the project. This experiment was conducted manually by providing a Q-grid (Figure 4) with cards (statements included in Table 6) on paper.

3.4. Respondents

The respondents in the study were selected from different types of partners and industries of the CFAA and different roles inside the organization. They were all experts and experienced PMs within privately held companies, working closely with the CFAA, and developing R&D projects and having the responsibility of decision making in projects or influence on decisions. In total, 20 respondents participated in the study. The respondent classification is shown in

Table 7. Respondent classification.

Role	Group	Profile	Number of Respondents
Stakeholder	Type A	Project Manager	4
	Type B	Project Assistant	1
		Project Manager	1
	Type C	Project Manager	2
	Type D	Project Manager	1
	University	Project Sponsor	3
Organization	CFAA	PMO	3
		Project Manager	5
Total			20

.

As for the experience and expertise of the participants chosen for this exercise, this was an absolute requirement for the authors, as they needed people who were familiar with project management theory and the different elements that make it up, as well as having the necessary practical experience (minimum 5 years conducting project management work in their organizations) to be able to analyze their responses much better and have greater relevance for the conclusions of the study.

4. Results and Discussion

The individual Q-sorts of the respondents were analyzed to reveal a limited number of perspectives in which the statements were sorted by the respondents. To analyze the results, the web application [88] from Aiora Zabala was used for the evaluation of the Q-sorts. This software aims to analyze data using Q-Methodology. It also offers all the options for standard Q-analysis, such as different extraction methods, rotation, and forced and unforced distributions [89]. The data processed by the online software were analyzed according to three factors, the significance of which is indicated below.

4.1. Factor Analysis

The first step is to group the participants according to the views they have in common. For this, it is necessary to perform a factor analysis, which will show the similarities between the participants’ ranking of the statements in the Q-grid [90]. Varimax rotation was used to ensure that the factors analyzed explained the maximum amount of study variance, and Pearson has been used for the correlation coefficient. Due to the relatively small number of Q-sorts, only a factor analysis with three and four factors was conducted. Since the factors analysis with three factors are more meaningful, these built the basis for further analysis (

Table 8. Factor analysis.

Role	Participant	Factor
		f1	f2	f3
Stakeholder	UNI1	0.6904	0.2107	0.353
	UNI2	0.227	0.5606	0.071
	UNI3	0.5991	0.2246	−0.108
	TA1	0.7708	0.3909	0.157
	TA2	0.3734	0.5379	−0.496
	TA3	0.5417	0.4273	0.148
	TA4	0.6105	0.5798	−0.093
	TB1	0.7492	0.0571	0.101
	TB2	0.6559	0.0229	−0.543
	TC1	0.1002	0.842	0.082
	TC2	0.7888	0.3696	0.051
	TD1	0.1211	0.8275	0.112
Organization	PM1	0.2101	−0.0092	0.764
	PM2	0.4162	0.6386	0.118
	PM3	0.5793	0.2215	0.368
	PM4	−0.0023	0.2482	0.727
	PM5	0.3628	0.1994	0.548
	PMO1	0.4818	0.4573	0.385
	PMO2	0.7198	0.2865	0.383
	PMO3	0.2818	0.6981	0.29

). The three factors explain 62% of the total variances (

Table 9. General factor characteristics.

	Average Relation Coefficient	Number of Loads *	Eigenvalues	Explainable Variance	Reliability	Standard Error of Factor Scores
Factor 1	0.8	10	5.4	27	0.98	0.16
Factor 2	0.8	5	4.2	21	0.95	0.22
Factor 3	0.8	3	2.7	14	0.92	0.28

* Only 2 people were not grouped to a factor.

).

4.2. Factor Interpretation

Following factor interpretation, and answering RQ2 (

Table 10. Top-ranked statements for stakeholders and the organization.

Item	Statements	Success Dimension	Category
13	Project goal was achieved	Project Management	Scope Management
7	Client satisfaction with the quality of the activities required to produce the project deliverables	Delivery Activities	Quality Management
11	Knowledge generation regarding project activities (e.g., tools, techniques, approaches, processes)	Delivery Activities	Knowledge Management
30	Workplace safety	Operations	Risk Management
8	Degree to which the deliverables meet their intended purpose	Deliverable	Quality Management
21	Return on Investment of the project	Project Management	Cost Management
5	Customer satisfaction regarding the deliverables	Deliverable	Quality Management
27	The deliverables meet the defined quality criteria	Deliverable	Quality Management
17	Reputation of the organization has increased	Project Management	Stakeholder Management
9	The activities required to produce the deliverables have a good reputation	Delivery Activities	Stakeholder Management

), it was very interesting to see that achieving the project goal was the most important criteria for the success of a project. It is logical, after all. However, the fact that the focus is so strong on the achievement of the goal is curious. Finding a reason for this is rather complex as it may be due to cultural factors, personal factors, organizational culture and directives, PM background, or further influences. This strong focus might also explain why more than 80% of the projects initiated in the CFAA as of 2020 were completed and successfully delivered to the partners.

The success dimension of the project is the most important statement and is related to those grouped under project management. This may be due to the distribution of the dimensions within the Q-set, but also to the apparent importance of how the project is managed both internally and externally. Aspects related to the deliverables are also very well-represented in this sample (60%), reflecting once again that as important as it is to achieve the project objective, it is also important that the final product of the project, as well as the activities that were carried out to deliver it, are well-managed. Of these statements, 40% are considered related to quality management, which is not enough if the quality of the work carried out is not considered.

The generation of knowledge within the project is an aspect to highlight. The processes, techniques, tools, and methodologies followed to achieve the project’s objective must be documented and delivered to the project owner. This makes the subsequent implementation at the client’s premises easier.

Among the lowest-ranked statements (

Table 11. Bottom-ranked statements for stakeholders and the organization.

Item	Statements	Success Dimension	Category
16	Publications regarding project management (e.g., tools, techniques, approaches, processes)	Project Management	Knowledge Management
18	Reputation of the Project Management Office (PMO) has increased	Project Management	Stakeholder Management
29	The way the deliverables are created contribute to sustainability	Delivery Activities	Sustainability
24	The deliverables, in terms of their design and creation, are adequate in terms of direct sustainability impacts	Deliverable	Sustainability
28	The product is characterized as sustainable	Deliverable	Sustainability
14	Project management processes were conducted within organization’s quality standard	Project Management	Quality Management
25	The deliverables, in terms of design and creation, are adequate in terms of indirect sustainability impacts	Deliverable	Sustainability
26	The deliverables meet official standards (e.g., ISO)	Deliverable	Quality Management
10	Delivery activities were conducted within organization’s quality standard	Delivery Activities	Quality Management
22	Risk value (impact) of suffered threads (unknown unknown)	Project Management	Risk Management

), it is worth noting that the risk impact of the threats suffered (unknown unknown) was the lowest-ranked statement. This may be since the organization has not carried out a correct identification of risks for the projects carried out. However, it is also recognizable that risks are more common in R&D projects. Both internal and external staff should be aware of the importance of a correct risk management for the project, where these risks can be identified, catalogued, categorized, and ranked to be able to propose preventive measures.

Another aspect to highlight is the sustainability of the project. All the sustainability-related criteria defined in the Final Q-set (Table 6) are among the bottom-ranked statements for the internal stakeholders and the organization, although for the TRL in which the CFAA works, sustainability is a determining factor for the subsequent implementation of the project results at the client’s facilities. This may be due to different reasons, such as a lack of awareness of the importance of sustainability in the activities beyond the immediate achievement of project results. Different cultural aspects can be evaluated to find a reason, or even the lack of conducting risk identification exercises at the organizational level may explain why for respondents it is not a determining factor in assessing the success of a project. Further research on this aspect can be developed, not only at the level of the organizations studied, but in the general context of project management.

These results at the organizational level should lead to efforts to improve the quality of the work carried out, managing it from the initial stages of the project to achieve the ultimate objective of the project, but with a special emphasis on the quality of the activities carried out. A project management methodology that allows this control is of vital importance for the Center and its stakeholders.

It is worth recognizing that the three factor-groups included compliance with standards, which reflects the awareness of PMs and stakeholders on this issue. In general, activities that lead to understanding and conveying the importance of these standards within the work performed are of great importance for the Center and the realization of R&D projects.

Responding to RQ2, the subjective perspectives are grouped on factors, which are named and described shortly, based on the results of the Q-analysis:

4.2.1. Factor 1: High Quality-Oriented to the Output

This group is characterized by getting the job completed properly. This includes focus on workplace safety, meeting official standards for deliverables and creating customer satisfaction regarding deliverables, as well as the project activities to create the output. This group focuses on fulfilling customer expectations and increasing the reputation of the organization.

This group is mainly composed of the organization’s stakeholders, which makes it logical to appreciate that the objective of the project and the activities to be carried out should be carried out with the highest possible quality. It also focuses on the importance of creating knowledge of both project activities and project management, a further quality of R&D project management.

4.2.2. Factor 2: Traditional Project Success-Oriented

The people in Factor 2 can be described as focused on general objectives. They want to stick to plans, such as adhering to scope statements and schedules, but also official standards to make the project successful for the organization. However, external knowledge sharing is not considered as their main priority.

Customer satisfaction is an important element for this group of people. An interesting aspect for further analysis is that this group is mostly made up of stakeholders. More experience in project management, a better definition of objectives in their companies, and more demanding controls for project management may be reasons for this majority. Additionally, the people grouped under this factor have a higher correlation based on the chosen statements with the people in Factor 1 than with the people grouped under Factor 3, which further indicates a strong inclination towards Project Quality Development by this group.

4.2.3. Factor 3: External View-Oriented

This group is heavily focused on the outside, which is represented by project goal achievement and customer satisfaction, knowledge generation and publications, and increasing the reputation of the organization. Changes regarding the scope and time are accepted to ensure customer satisfaction. It is also characterized by the acknowledgment and fulfillment of the official standards of the activities developed during the project, which is a characteristic that this group choose from the statements as one of the most important to them.

In general, this group matches most of the goals of the CFAA and seeks to help on the development of applicable advanced manufacturing technologies and a quick transfer of this knowledge for both partner companies of each project and local industry.

5. Conclusions

A project can no longer be seen only as a temporary endeavor undertaken to create a unique product, service, or outcome. Although the objective is the same, the elements that constitute it increase, and must be seen as a system in which the inputs directly determine the outputs, with all the elements that compose it developed in one or several organizations with certain practices and customs, in which PMs, influenced by a series of elements, can make decisions that compromise the final success of the project according to the stakeholder’s criteria. Being able to understand this complexity and to analyze it point-by-point is what determines the success of the project.

A major concern for the organization was to identify which were the most important success criteria when evaluating or considering a project successful. The identification of these criteria in the literature, the subsequent discussion to define them, and the exercise developed in accordance with Q-Methodology provided us with the possibility to create a list of what these criteria are. With the results obtained, it is now possible for the PMs of the organization, as well as the members of the Project Management Office, to make plans, set indicators, and define methodologies and new control points in which the fulfillment of the quality criteria identified in this study can be evaluated, not only at the end of the project, but also during the whole lifecycle.

Another important aspect of the research is that it is easily applicable to other types of organizations. Thanks to the analysis of the literature, we have shown how important it is to know the stakeholders’ point of view for the success of the project, and how this is not exclusive to R&D organizations that carry out collaborative projects, but to any other organization that carries out projects.

To conclude on RQ1, the most important criteria were identified and analyzed in Section 4. About RQ2, the factor-group 1 comprised 50% of the participants. As described in chapter 4.2, this group is focused on delivering high-quality outputs. Nonetheless, as the factors show no greater majority and the participant size was limited, this can be interpreted as an indicator. However, the results can be used to improve the project approach in R&D projects, and to evaluate the success of such projects. Depending on the context of the project, these criteria might be introduced or used to re-weight existing metrics and KPIs.

6. Limitations

There are several threats to the validity of this study, particularly about subject sampling and external validity. The subjects who participated in the survey were chosen because they were the most important people in their organizations related to the CFAA and were the ones who had spent the most time working closely with them. At least one participant was chosen from each type of CFAA partner, however, having only one participant from the type D partner (Table 5) may not be appropriate, as this group of partners is the largest, with more than 50% of all CFAA partners. Furthermore, the participant group is small and might not be representative. Consequently, this research and approach should be taken further on a greater scale to validate the findings.

7. Future Research

As explained above, the criteria for defining whether a project is successful or not depend on a series of factors related to the point of view, or the moment at which the measurement is made. Future research is needed to further study the reasons behind the lack of success of the projects carried out in the organization, to determine and analyze whether they are related to the success criteria evaluated here.

Furthermore, to overcome some limitations, this research and approach should be taken further on a greater scale to validate the findings. This might strengthen the factors found or introduce new factors.

It is also important to evaluate how these success factors evolve. Changes in client requirements, organizational conditions, and the context in which the project is developed are factors that affect which criteria determine whether a project is successful or not, so it is interesting to know how and why the evolution in the success criteria determined for a project is due. A complementary part of this study could have been to compare the currently perceived success or failure of the projects of the organizations against the analyzed factors, to evaluate these as a kind of test of the factors found.

As a natural and logical next step, a project management approach in which these success criteria are considered, aimed at continuous quality assessment during the project life cycle, should be developed. Additionally, with the use of this approach and having measured the success criteria for a particular project, it would be possible to analyze whether these criteria are really adapted to the reality of the organizations or whether, on the contrary, they fail to measure some aspect not considered.