Next Article in Journal
Acknowledgment to the Reviewers of Software in 2022
Previous Article in Journal / Special Issue
Analysis of Faults in Software Systems Using Tsallis Distribution: A Unified Approach
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Are Infinite-Failure NHPP-Based Software Reliability Models Useful?

Graduate School of Advanced Science and Engineering, Hiroshima University, Hiroshima 739-8511, Japan
*
Author to whom correspondence should be addressed.
Software 2023, 2(1), 1-18; https://doi.org/10.3390/software2010001
Submission received: 18 November 2022 / Revised: 19 December 2022 / Accepted: 20 December 2022 / Published: 23 December 2022
(This article belongs to the Special Issue Dependable Computing and Security for Software Systems)

Abstract

:
In the literature, infinite-failure software reliability models (SRMs), such as Musa-Okumoto SRM (1984), have been demonstrated to be effective in quantitatively characterizing software testing processes and assessing software reliability. This paper primarily focuses on the infinite-failure (type-II) non-homogeneous Poisson process (NHPP)-based SRMs and evaluates the performances of these SRMs comprehensively by comparing with the existing finite-failure (type-I) NHPP-based SRMs. In more specific terms, to describe the software fault-detection time distribution, we postulate 11 representative probability distribution functions that can be categorized into the generalized exponential distribution family and the extreme-value distribution family. Then, we compare the goodness-of-fit and predictive performances with the associated 11 type-I and type-II NHPP-based SRMs. In numerical experiments, we analyze software fault-count data, collected from 16 actual development projects, which are commonly known in the software industry as fault-count time-domain data and fault-count time-interval data (group data). The maximum likelihood method is utilized to estimate the model parameters in both NHPP-based SRMs. In a comparison of the type-I with the type-II, it is shown that the type-II NHPP-based SRMs could exhibit better predictive performance than the existing type-I NHPP-based SRMs, especially in the early stage of software testing.

1. Introduction

In actual software development projects, clients frequently expect software products to be of high quality. As software products tend to become more complex with increasing size, software reliability is gradually gaining much attention from developers as an important attribute of software quality. Therefore, in the modern software development process, developers concentrate their human and material resources on the testing process to detect and fix inherent faults as much as possible for the purpose of improving software reliability. Due to the high cost of software testing, quantification of software reliability is also regarded as a significant concern during the verification phase.
To the best of our knowledge, the probabilistic behavior of the fault detection and correction process during the software testing phase in software reliability engineering is usually characterized by any stochastic counting process. On the other hand, after the software product is released, the probability of a product not experiencing any failure caused by software faults over a specific time interval is usually defined as quantitative software reliability. To measure the above software reliability, over the last five decades, hundreds of probabilistic models known as software reliability models (SRMs) have been implemented to quantitatively assess software reliability during the testing and operational phases. Among the existing SRMs, the non-homogeneous Poisson process (NHPP)-based SRMs are recognized as a very important class because of their mathematical tractability and high applicability. By modeling the software failure time, Kuo and Yang [1] classified NHPP-based SRMs into general order statistics SRMs and record value statistics SRMs. The same authors [1] proposed an alternative and more general classification by dividing NHPP-based SRMs into two types; finite-failure (type-I) and infinite-failure (type-II) NHPP-based SRMs with mean value functions, which are defined as the expected cumulative number of software failures. The best-known finite-failure (type-I) NHPP-based SRM was proposed by Goel and Okumoto [2], who assumed the exponential distribution as the fault-detection time distribution in software testing. The mean value function there is in proportion to the cumulative distribution function (CDF) of the exponential distribution. After that, postulating the other fault-detection time distributions, several type-I NHPP-based SRMs were proposed in the literature, such as the truncated-normal NHPP-based SRM [3], the log-normal NHPP-based SRM [3,4], the truncated-logistic NHPP-based SRM [5], the log-logistic NHPP-based SRM [6], the extreme-value NHPP-based SRMs [7,8], the gamma NHPP-based SRM [9,10], and the Pareto NHPP-based SRM [11]. At the same time, introducing a series of commonly used lifetime CDFs for modeling software failure time in reliability engineering, a few infinite-failure (type-II) NHPP-based SRMs have also been developed and widely used to quantitatively evaluate software reliability. The power-law process model [12,13,14] and the logarithmic Poisson execution time model [15,16] are classified as type-II NHPP-based SRMs. Note that the use of type-I NHPP-based SRMs does not necessarily imply that all inherent software faults are detected over an infinite time horizon. In other words, the precise number of inherent faults cannot be known, even if software testing is performed for an indefinite period. Thus, the finiteness in the type-I NHPP-based SRMs holds in the sense of the expectation of a cumulative number of detected faults. Unfortunately, a fair comparison between the type-I and type-II NHPP-based SRMs has not been made in past, because only a limited number of type-II NHPP-based SRMs have been considered in the literature.
Our research question of this paper is “Are infinite-failure NHPP-based SRMs useful?” More specifically, this paper investigates whether infinite-failure NHPP-based SRMs can guarantee better goodness-of-fit performance for the fault-count data collected in the software testing phase in comparison with finite-failure NHPP-based SRMS, and whether they can guarantee more accurate performance in predicting the number of software faults. Goodness-of-fit and predictive performance are generally recognized as the critical factors determining which SRM should be practically applied to quantitatively assess software reliability.
The original contribution of this paper is to investigate the type-II NHPP-based SRMs with the representative 11 CDFs in the literature. Three of these type-II NHPP-based SRMs are confirmed to be equivalent to the existing Cox-Lewis process, logarithmic Poisson execution time model, and power-law process, while the remaining eight type-II SRMs are novel SRMs. We confirm that the corresponding type-I and type-II NHPP-based SRMs can be obtained by importing the same software fault-detection time distribution CDFs to the finite- and infinite-failure assumptions, respectively. We make a comprehensive comparison between the existing type-I NHPP-based SRMs and their associated type-II NHPP-based SRMs. As shown in [17], it seems enough to consider 11 kinds of software fault-detection time CDFs in making goodness-of-fit and predictive performance comparisons between two different NHPP-based modeling frameworks.
The rest of this paper is organized as follows. Section 2 describes the definition of NHPP and illustrates NHPP-based software reliability modeling under the finite-failure and the infinite-failure hypotheses. We present 11 existing type-I NHPP-based SRMs based on the finite-failure hypothesis in [17] and propose 11 type-II NHPP-based SRMs with the same CDFs. The maximum likelihood approach to estimate the model parameters is summarized. We confirm that maximum likelihood estimation can be used for parameter estimation of our type-II NHPP-based SRMs for software fault-count time-domain data. We also give specific expressions of likelihood function and log-likelihood function for software fault-count time-interval data (group data), which are more common in the industry.
In the numerical examples in Section 3, we employ a total of 16 datasets collected from 16 actual development projects. In each dataset, we investigate the goodness-of-fit performance and predictive performance of type-I NHPP-based SRMs and type-II NHPP-based SRMs that have completed parameter estimation by maximum likelihood estimation. In addition, we also use these SRMs to quantitatively evaluate software reliability over a given time period, and analyze the applicability of the type-I NHPPs as well as the type-II NHPPs in predicting software reliability. In Section 4, the paper is summarized with some remarks and future directions.

2. Non-Homogeneous Poisson processes

2.1. Preliminary

As a well-known Markov process, a non-homogeneous Poisson process (NHPP) is regarded as a generalization of the classical homogeneous Poisson process (HPP). If the intensity at time point t in the definition of HPP is given by a function λ ( t ) with respect to t , then an HPP can be generalized to an NHPP. More specifically, if a stochastic counting process { N ( t ) , t 0 } is non-negative and non-decreasing, it becomes an NHPP under the following assumptions.
  • NHPP has independent increments, so the number of occurrences in a specific time interval depends on only the current time t and not on the past history of the process, which is also known as the Markov property.
  • The initial state of the process is given by N ( 0 ) = 0 .
  • The occurrence probability of one event in a given time period [ t , t + Δ t ) for an NHPP is defined by Pr { N ( t + Δ t ) N ( t ) = 1 } = o ( Δ t ) + λ ( t ) Δ t . λ ( t ) is an absolutely continuous function, and is named the intensity function of NHPP. Δ t is recognized as an infinitesimal period of time.
  • NHPP has negligible probability for two or more events occurring in [ t , t + Δ t ) , i.e., Pr { N ( t + Δ t ) N ( t ) 2 } = o ( Δ t ) , where lim Δ t 0 o ( Δ t ) Δ t = 0 and o ( Δ t ) is the higher-order term of Δ t .
  • As a typical Markov process, the Kolmogorov forward equations of NHPP can be written as
    d d t P 0 ( t ) = λ ( t ; θ ) P 0 ( t ) ,
    d d t P n ( t ) = λ ( t ; θ ) P n 1 ( t ) λ ( t ; θ ) P n ( t ) ,             n = 1 , 2 , ,
    with P 0 ( 0 ) = 1   and   P n ( 0 ) = 0 , where θ represents the free parameter vector in the transition rate function λ ( t ; θ ) . By solving the above simultaneous equations, the steady-state transition probability Pr { N ( t ) = n | N ( 0 ) = 0 } = P n ( t ) is given by
    P n ( t ) = exp ( M ( t ; θ ) ) { M ( t ; θ ) } n n ! ,     n = 0 , 1 , 2 , .
Through the Poisson property, M ( t ; θ ) is defined as the mean value function of NHPP and represents the expected cumulative number of event occurrences during the interval ( 0 , t ] .

2.2. NHPP-Based SRMs

Most textbooks [16,18,19] have pointed out that when the mean value function was used to characterize the cumulative number of software failures by time t , there were two types of NHPP-based SRMs; finite-failure NHPP-based SRMs and infinite-failure NHPP-based SRMs.

2.2.1. Finite-Failure (Type-I) NHPP-Based SRMs

In the software reliability modeling framework developed based on finite-failure (type-I) NHPP, before testing, the remaining number of software faults is assumed to obey a Poisson distribution with a positive mean μ 0 . Each software fault is assumed to be detected at independent and identically distributed (i. i. d.) random times, and is fixed immediately after it is detected. For any t ( 0 , + ) , F ( t ; α ) , a non-decreasing function, is applied to describe the time distribution of each fault detection during the software testing phase, which is also known as the cumulative distribution function (CDF). α indicates the free parameter vector in the CDF. Then, a binomial distributed random variable with probability F ( t ; α ) with a Poisson distributed population with parameter μ 0 is employed to characterize the resultant software fault-detection process. From a simple algebraic manipulation, the mean value function of NHPP can be derived as
M ( t ; θ ) = μ 0 F ( t ; α ) ,
which can also be recognized as the cumulative number of faults detected by the software testing at time point t with θ = ( μ 0 , α ) and lim t M ( t ; θ ) = μ 0   ( > 0 ) . This property is consistent with the assumption of software reliability modeling for type-I NHPP that the number of initial remaining faults expected before software testing begins is finite. In Table 1, we summarize 11 existing type-I NHPP-based SRMs with their associated CDFs and bounded mean value functions, which were employed in the software reliability assessment tool on the spreadsheet (SRATS) by Okamura and Dohi [17].
Even though the type-I NHPP-based SRMs are recognized as plausible models in term of software reliability growth phenomena, it must be acknowledged that reliability engineers sometimes feel discomfort when handling finite-failure NHPPs, since the inter-failure time distributions in the type-I NHPP-based SRMs are defective [20]. Let us suppose that the random variables T 1 ,   T 2 , , T n represent the first, second, …, n -th failure times that occur after the software testing starts at T 0 = 0 . Let the random variables X 1 , X 2 , , X n denote the inter-failure times between two consecutive failures:
T n = j = 1 n X j = T n 1 + X n ,     n = 0 , 1 , 2 , .
From Equations (3) and (5), the CDF of T n can be obtained as
G n ( t ; θ ) = P { T n t   ( the   n - th   failure   occurs   up   to   t ) } = P { N ( t ) n   ( at   least   n   failures   occur   before   time   t ) } = 0 t λ ( x ; θ ) [ M ( x ; θ ) ] n 1 ( n 1 ) ! exp ( M ( x ; θ ) ) d x = j = n [ M ( t ; θ ) ] j j ! exp ( M ( t ; θ ) ) = 1 j = 0 n 1 [ M ( t ; θ ) ] j j ! exp ( M ( t ; θ ) ) .
Then, it is straightforward to see in the type-I NHPP-based SRMs that lim t G n ( t ; θ ) < 1 for an arbitrary n . In other words, even if the testing time tends to be infinite, there still exists a positive probability of the n -th failure not occurring. It is obvious that the CDF of T n is defective. Similarly, for realizations of T i   ( i = 1 , 2 , , n ) ,     t 1 ,     t 2 , , t n , we can obtain the CDF of the inter-failure time X n in the time interval ( t n 1 ,     t n 1 + x ) as follows.
F n ( x ;   θ ) = 1 Pr {   N ( t n 1 + x ) N ( t n 1 ) = 0 | N ( t n 1 ) = n 1 } = 1 exp ( ( M ( t n 1 + x ; θ ) M ( t n 1 ; θ ) ) ) ,
where Pr {   N ( t n 1 + x ) N ( t n 1 ) = 0 | N ( t n 1 ) = n 1 } denotes the probability that no failure occurs in time interval ( t n 1 , t n 1 + x ) . Since the mean value function is bounded, i.e., lim t M ( t ; θ ) = μ 0 , when x is infinite, Equation (7) can be reduced to 1 e ( μ 0 M ( t n 1 ; θ ) ) < 1 . It means that regardless of the number of previous failures, the probability that the software fails over an infinite time horizon is always non-zero. Hence, the inter-failure time CDF of type-I NHPP is also defective. For the type-I NHPP-based SRMs, it is not meaningful to discuss some reliability metrics, such as mean time between failures (MTBF), because the finite moments of T n and X n always diverge.

2.2.2. Infinite-Failure (Type-II) NHPP-Based SRMs

Type-II NHPP assumes that a new software fault is not inserted at each software debugging. However, this assumption may be somewhat specific, because so-called imperfect debugging may occur in the actual software testing phases. When the possibility of imperfect debugging is considered, the assumption of finiteness in the type-I NHPP-based SRMs seems to be rather strong. Similarly to the classical preventive maintenance modeling [21], if each software failure is minimally repaired through the debugging, the mean value function of software fault-detection process is unbounded and is given by
M ( t ; α ) = ln ( 1 F ( t ; α ) ) ,
where lim t M ( t ; α ) . It is obvious that the CDFs, G n ( t ; θ ) and F n ( x ; θ ) in Equations (6) and (7) are not defective; for instance, lim t G n ( t ; θ ) = 1 and lim x F X i ( x ; θ ) = 1 . Hence, it becomes significant to consider important metrics, such as MTBF. In this modeling framework, investigating the residual number of software faults before testing has no significant meaning, because it may increase by imperfect debugging through the software testing.
As far as we know, the Cox-Lewis process [22] is one of earliest type-II NHPPs. The unbounded mean value function of this model is given by M ( t ; α ) = ( exp ( μ 1 + μ 2 t ) exp ( μ 1 ) ) / μ 2 with the extreme-value distribution F ( t ; α ) = 1 exp ( exp ( μ 1 + μ 2 t ) exp ( μ 1 ) ) / μ 2 . This distribution is also referred to as truncated extreme-value minimum distribution in [17]. Another well-known type-II NHPP-based SRM is referred to as a power-law process model [12,13,14], where mean value function and CDF are given by M ( t ; α ) = μ 2 / μ 1 t ( 1 / μ 1 ) and F ( t ; α ) = 1 exp ( exp ( μ 2 + ln ( t ) μ 1 ) ) , respectively. The latter is also recognized as the log-extreme-value minimum distribution in [17]. Besides the above two representative NHPPs, the well-known logarithmic Poisson execution time SRM [15,16] belongs to the type-II category, too. The mean value function of this model is given by M ( t ; α ) = μ 2 ln ( ( 1 + t ) / μ 1 ) with the Pareto distribution F ( t ; α ) = 1 ( μ 1 / ( t + μ 1 ) ) μ 2 in [17]. In Table 1, it is easy to see that the same CDFs are used for type-I Txvmin SRM v.s. type-II Cox-Lewis SRM, type-I Lxvmin SRM v.s. type-II Power-law SRM, and type-I Pareto SRM v.s. type-II Musa-Okumoto SRM, respectively. Hence, by substituting 11 software fault-detection time CDFs in Table 1 into Equation (8), we can derive the corresponding type-II NHPP-based SRMs in Table 2.

2.3. Parameter Estimation

For the existing type-I and type-II NHPP-based SRMs, the maximum likelihood (ML) estimation is a typical technique that is widely applied in software reliability modeling. The parameters of the maximized log-likelihood function (LLF) provide the ML estimates. In addition, the LLF depends on the observed fault-count data, the intensity function and/or the mean value function in the type-I and type-II NHPP-based SRMs. Next, we give the likelihood functions for the software fault-count time-domain data and the software fault-count time-interval data (group data).

2.3.1. Software Fault-Count Time-Domain Data

Consider that the total number of faults observed in the testing phase is m D before the time observation point t m D , where the time sequence consisting of the time points at which each fault is detected is given by D = { t 1 , t 2 , , t m D } . This kind of time series is called software fault-count time-domain data. Generally, CPU time is used to measure the time-domain data in software testing. Then, for the time-domain data, the likelihood function of NHPP is as follows.
L ( θ   or   α ; D ) = exp ( M ( t m D ; θ   or   α ) ) i = 1 m D λ ( t i ; θ   or   α ) .
Taking logarithm of both sides in Equation (9), the log-likelihood function is obtained as
ln L ( θ   or   α ; D ) = i = 1 m D ln λ ( t i ; θ   or   α ) M ( t m D ; θ   or   α ) .
The ML estimate, θ ^ or α ^ , is given by argmax θ ln L ( θ ; D ) or argmax α ln L ( α ; D ) .

2.3.2. Software Fault-Count Time-Interval Data (Group Data)

Software fault-count time-interval data (group data) consist of the number of detected faults in a set of calendar-time-based time intervals ( t i 1 , t i ]   ( i = 1 , 2 , , m I ) . Each record in these types of data is composed of the observation time point t i and n i , the cumulative number of faults detected in the time interval ( 0 ,     t i ] , written as ( t i , n i ) . Calendar time is usually measured as testing days or weeks. When the group data I = { ( t i , n i ) , i = 1 , 2 , , m I } are available, the likelihood function and LLF are given by
L ( θ   or   α ; I ) = i = 1 m I exp [ M ( t i ; θ   or   α ) M ( t i 1 ; θ   or   α ) ] [ [ M ( t i ; θ   or   α ) M ( t i 1 ; θ   or   α ) ] n i n i 1 ( n i n i 1 ) ! ]
and
ln L ( θ   or   α ; I ) = i = 1 m I ( n i n i 1 ) ln { M ( t i ; θ   or   α ) M ( t i 1 ; θ   or   α ) } i = 1 m I ln { ( n i n i 1 ) ! } M ( t m I ; θ   or   α ) ,
respectively. The ML estimate, θ ^ or α ^ , is given by argmax θ ln L ( θ ; I ) or argmax α ln L ( α ; I ) .

3. Performance Comparison

3.1. Datasets

In the selection of data sources for numerical experiments, we selected the well-known benchmark software fault-count datasets in software reliability engineering, which are observed in mission-critical systems. Although the evolution of these systems may be slower than that of business-oriented systems, the effects of a failure are much greater. Hence, reliability is particularly important for the developers of these mission-critical systems. In the industry, the software fault-count data observed in the distributed test environment for mission-critical systems can be divided into two categories: software fault-count time-domain data and software fault-count time-interval data (group data). We selected eight sets of each type of data, which have been widely utilized as fault-count data in software reliability engineering. The details of these datasets are shown in Table 3 and Table 4, respectively.

3.2. Goodness-of-Fit Performance

Assuming that the parameters of the SRMs were estimated by maximum likelihood estimation, in the first experiment, we employed two criteria for evaluating the goodness-of-fit performance of the 11 type-I and type-II NHPP-based SRMs, for instance, Akaike information criterion (AIC):
AIC ( θ ^   or   α ^ ) = 2 × ( number   of   free   parameters ) 2   ln L ( θ ^   or   α ^ )
and the mean squared error (MSE);
MSE ( θ ^   or   α ^ ; D ) = i = 1 m D ( i M ( t i ; θ ^   or   α ^ ) ) 2 m D
or
MSE ( θ ^   or   α ^ ; I ) = i = 1 m I ( n i M ( t i ; θ ^   or   α ^ ) ) 2 m I ,
respectively. In Equations (14) and (15), n i is defined as the count of detected faults in the time interval ( 0 ,     t i ] , m D and m I are the lengths of time-domain and group data, and θ ^ and α ^ are the ML estimates by maximizing ln L ( θ   or   α ; D ) and ln L ( θ   or   α ; I ) . The AIC with ML estimates generally represented an approximation of the Kullback–Leibler divergence between our proposed SRM and the empirical stochastic process behind the fault-count data, while direct MSE exhibited a vertical distance between the estimated mean value function and the fault-count data. A smaller AIC/MSE indicated that the SRM had a better goodness-of-fit performance (showing a better fit to the underlying data).
Figure 1 and Figure 2 plot the behavior of the mean value functions of type-I and type-II SRMs in the time-domain data, TDDS1, and the group data, TIDS7. The red curve and the orange curve are plotted as the best SRMs selected from 11 type-II SRMs and 11 type-I SRMs based on their AIC, respectively. Not surprisingly, the two modeling frameworks showed slightly different growth trends. More specifically, the type-I (orange curve) always fitted better to the actual data in the tail segment, in both the time-domain and the group data. However, we still were not able to make a comprehensive assessment regarding which SRM exhibited a better fitting ability over the whole dataset. It was therefore necessary to think about AIC as well as MSE as such criteria. First, in Table 5, we make a more precise comparison between our proposed type-II and the existing type-I on AIC and MSE. Without comparing them with each other, it is evident that in the vast majority of cases, the best models among the type-I SRMs were given by the extreme-value distributions. By contrast, the type-II Pareto (Musa-Okumoto) SRM performed better than the other SRMs. In the next step, by comparing the best type-I and type-II SRMs for each dataset, it is not difficult to observe that in three cases (TDDS1, TDDS3, and TDDS6), the type-II SRMs provided a smaller AIC than the type-I SRMs. However, in all the datasets, the type-I SRMs provided a smaller MSE than the type-II SRMs.
In Table 6, we compared the SRMs of our type-II NHPP with the existing type-I NHPP-based SRMs in eight group datasets. It can be seen that our type-II SRMs could guarantee a smaller AIC than the existing type-I in three cases (TIDS2, TIDS3, and TIDS7), but at the same time, it still could not outperform the type-I from the viewpoint of MSE for any group dataset. We can therefore draw the conclusion that the type-II NHPP-based SRMs could not consistently outperform the existing type-I NHPP-based SRMs in terms of goodness-of-fit performance, but in some cases, especially in time-domain data, the three existing type-II NHPP-based SRMs, Musa-Okumoto, Cox-Lewis, and power-law SRMs, could indicate the better experimental results.

3.3. Predictive Performance

Notably, according to the previous studies, SRMs with better goodness-of-fit do not necessarily provide an excellent predictive performance. In other words, investigating the predictive performance of the type-I and type-II NHPP-based SRMs is of significant importance. Hence, in our second experiment, we employed the prediction mean squared error (PMSE) to measure the predictive performance of our type-II SRMs, where
PMSE ( θ ^   or   α ^ ; D ) = i = m + 1 m + l ( i M ( t i ; θ ^   or   α ^ ) ) 2 l ,
and
PMSE ( θ ^   or   α ^ ; I ) = i = m + 1 m + l ( n i M ( t i ; θ ^   or   α ^ ) ) 2 l ,
for the time-domain and group data, respectively, where m or n m software faults were observed in ( 0 ,     t m ] , and the prediction length is given by l   ( = 1 , 2 , ) . θ ^ and α ^ are the ML estimates at observation time t m for the type-I and type-II NHPP-based SRMs, respectively. Similarly to MSE, PMSE is also a metric that evaluates the mean squared distance between the predicted number of detected faults and its (unknown) realization for each prediction length.
For a comprehensive investigation of the predictive performance of SRMs at different software testing phases, three observation points were set at 20%, 50%, and 80% of the total length of each dataset to represent the early, middle, and late phases of software testing and to predict the total number of software faults at the remaining 80%, 50% and 20% of the time periods. Then, we calculated the PMSE for the type-I and type-II NHPP-based SRMs. It was immediately evident that a larger observation point corresponded to a shorter prediction length. In Figure 3, Figure 4 and Figure 5, we plot the predictive behavior of the best existing type-I and the best type-II NHPP-based SRMs in time-domain data, TDDS1, at three different observation points. The red curve in each figure represents our best type-II NHPP, while the orange curve denotes the best type-I NHPP. All the best SRMs were taken from the type-I NHPP-based SRMs and the type-II NHPP-based SRMs with their smaller PMSEs in TDDS1. It can be seen that both type-I and type-II tended to give almost the same number of predicted software faults in the early and late testing phases. However, after the mid-term of testing, the type-I NHPP-based SRM tended to make more optimistic software fault predictions. In Figure 6, Figure 7 and Figure 8, we also plot the predictive behavior of the best existing type-I and the best type-II NHPP-based SRMs in group data, TIDS7. It can be seen that the type-I still tended to falsely predict the number of software faults in the early and middle testing phases. More specifically, in Figure 6 and Figure 7, the type-II NHPP-based SRMs showed an increasing trend, the opposite was true for the type-I, whose predictive trend for future phases becomes very flat. However, in Figure 8, the type-I and type-II showed more similar predictive trends. In general, prediction of unknown trend changes over longer periods of time in the future is essentially difficult for both the type-I NHPP and the type-II NHPP. In contrast, prediction of trend changes over a short period of time is relatively easy, but absolute accuracy cannot be guaranteed.
In Table 7, we present the PMSEs of the best type-I SRM compared to the best type-II SRM in each set of time-domain data. We compared the PMSEs in 11 type-I SRMs and 11 type-II SRMs by selecting the models with the smaller PMSEs as the best SRMs at each observation point. It can be seen that at the 20% observation point, our type-II SRMs provided smaller PMSEs than the existing type-I SRMs in three cases (TDDS2, TDDS6, and TDDS7). During the middle testing phase (at the 50% observation point), we observed that our type-II SRMs outperformed the type-I SRMs in four datasets (TDDS4, TDDS6~TDDS8). As the test proceeded to the late phases (at the 80% observation point), type-II SRMs were able to guarantee smaller PMSEs in TDDS1, TDDS4, and TDDS7. On the other hand, it was found that the best type-II SRMs with better predictive performance than the type-I were all provided by logistic distribution, Musa-Okumoto SRM, and power-law SRM. Upon comparing PMSEs in time-domain data, we believe that the type-II SRMs could become a good alternative to the type-I SRMs.
As shown in Table 8, when testing in the early phase (at the 20% observation point), it was immediately noticed that our type-II SRMs showed smaller PMSEs than the type-I SRMs in seven out of eight group datasets (except in TIDS3). In addition to logistic-based SRM, Musa-Okumoto SRM, and power-law SRM, which were proven to perform better in Table 7, we observed that Cox-Lewis SRM was also appropriate in some cases of group data (TIDS4 and TIDS5) in terms of predictive performance. At the 50% observation point, we found that the type-II SRMs could guarantee better predictive performance than the type-I SRMs in three cases (TIDS3, TIDS6 and TIDS7). In the late testing phase (at the 80% observation point), only in TIDS2 did our Tlogist type-II SRMs give the smallest PMSE in the future prediction phase. In the group data, the predictive performance of the type-II SRMs decreased as the software testing proceeded. Hence, it is possible to conclude that the infinite-failure NHPP-based SRMs outperformed the existing finite-failure NHPP-based SRMs for software fault-detection prediction in the early testing phase when group data were available.

3.4. Software Reliability Assessment

Our final research question for NHPP-based Type-II SRMs is how to utilize them to quantitatively assess the software reliability. In general, in NHPP software reliability modeling, the reliability of software at a given time point t r can be calculated by R ( t r ;   θ   or   α ) ; that is, the probability that the software will be failure-free during time interval t r t m , which can be written as
R ( t r ; θ   or   α ) = Pr { N ( t r ) N ( t m ) = 0 }       = exp ( [ M ( t + x ; θ   or   α ) M ( t ; θ   or   α ) ] ) ,
where for time data and group data, t m is defined as the time point of the last fault detected during the software test and the calendar time when the test was stopped, respectively. m is the total number of detected faults before the time point t m . In this numerical experiment, we assumed that t r = t m . The software reliability in each software development project was predicted quantitatively by importing the mean value functions of type-I NHPP and type-II NHPP into Equation (18), respectively.
In Table 9 and Table 10, we compare the quantitative software reliability of the best type-I SRMs and the best type-II SRMs in the time-domain data and group data, respectively. We selected the type-I SRM and the type-II SRM with the smaller AIC in each time domain and group dataset as the best SRMs. We can see that in almost all datasets (except in TDDS1 and TIDS7), the type-I SRMs tended to predict software reliability than our type-II SRMs. In other words, during the time period t r t m , the probability of software failure predicted by the type-II NHPP was much higher than the type-I NHPP. This observation indicates that our type-II SRMs tended to make more conservative decisions than the type-I SRMs in software reliability assessment. It is important to note that optimistic reliability estimates are often undesirable. This is because software faults are additionally detected as the ex-post results after each observation point in all the datasets.

4. Conclusions

Under the infinite-failure assumption, in addition to the well-known Musa-Okumoto model, Cox-Lewis model, and power-law model, in this work we proposed another eight type-II NHPP-based SRMs with eight different software fault-detection time CDFs. By analyzing eight software fault-count time-domain datasets and eight software fault-count time-interval datasets (group data), we investigated the goodness-of-fit performance and predictive performance of our SRMs. We also compared these SRMs with 11 existing type-I NHPP-based SRMs under the finite-failure assumption. The important point to note is that the type-I and type-II NHPP-based SRMs considered in this paper had the same software fault-detection CDFs, which has never been addressed in the past literature.
The experimental results confirmed that our type-II NHPP-based SRMs showed better goodness-of-fit performance in some cases. On the other hand, for the group data, the type-II NHPP-based SRMs exhibited rather better predictive ability than the existing type-I NHPP-based SRMs in the early testing phase. However, as software testing progressed, it was found that the advantages of type-II NHPP in terms of predictive performance were diminished. Hence, we can conclude that the type-II NHPP-based SRMs are a good complement to the type-I NHPP-based SRMs for describing the fault-detection process of software systems, while at the same time, they have greater potential in the early software testing phase. On the other hand, we also confirmed that the type-II NHPP tended to make more conservative predictions than the type-I NHPP in software reliability assessment.
We do not believe that there are any significant limitations to the validity in this work. The datasets used for the numerical experiments were collected during the software/system testing phase under careful supervision and with specific objectives. They have been proven to be of high quality [18,23,24,25,26]. Both the finite-failure software reliability modeling assumptions and the infinite-failure software reliability modeling assumptions have been shown to be well-founded. The experimental results exhibited by type-I NHPP-based SRMs and type-II NHPP-based SRMs, such as software reliability, also match the actual situation. There is no evidence so far that our proposed SRMs are inapplicable in any type of software.
In the future, we will introduce virtual testing time in the type-II NHPP-based SRMs, which will be beneficial as we continue to explore the potential of the type-II modeling hypothesis.

Author Contributions

Conceptualization, S.L., T.D. and H.O.; methodology, S.L., T.D. and H.O.; validation, S.L., T.D. and H.O.; writing—original draft preparation, S.L.; writing—review and editing, T.D. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

This work was supported by JST SPRING, Grant Number JPMJSP2132.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Kuo, L.; Yang, T.Y. Bayesian computation for nonhomogeneous Poisson processes in software reliability. J. Am. Stat. Associ. 1996, 91, 763–773. [Google Scholar] [CrossRef]
  2. Goel, A.L.; Okumoto, K. Time-Dependent Error-Detection Rate Model for Software Reliability and Other Performance Measures. IEEE Trans. Reliab. 1979, R-28, 206–211. [Google Scholar] [CrossRef]
  3. Okamura, H.; Dohi, T.; Osaki, S. Software reliability growth models with normal failure time distributions. Reliab. Eng. Syst. Saf. 2013, 116, 135–141. [Google Scholar] [CrossRef]
  4. Achcar, J.A.; Dey, D.K.; Niverthi, M. A Bayesian approach using nonhomogeneous Poisson processes for software reliability models. In Frontiers in Reliability; World Scientific: Hackensack, NJ, USA, 1998; pp. 1–18. [Google Scholar] [CrossRef] [Green Version]
  5. Ohba, M. Inflection S-Shaped Software Reliability Growth Model. In Stochastic Models in Reliability Theory; Springer: Berlin/Heidelberg, Germany, 1984; pp. 144–162. [Google Scholar] [CrossRef]
  6. Gokhale, S.S.; Trivedi, K.S. Log-logistic software reliability growth model. In Proceedings of the Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No. 98EX231), Washington, DC, USA, 13–14 November 1998; pp. 34–41. [Google Scholar]
  7. Goel, A.L. Software reliability models: Assumptions, limitations, and applicability. IEEE Trans. Softw. Eng. 1985, SE-11, 1411–1423. [Google Scholar] [CrossRef]
  8. Ohishi, K.; Okamura, H.; Dohi, T. Gompertz software reliability model: Estimation algorithm and empirical validation. J. Syst. Softw. 2009, 82, 535–543. [Google Scholar] [CrossRef] [Green Version]
  9. Yamada, S.; Ohba, M.; Osaki, S. S-Shaped Reliability Growth Modeling for Software Error Detection. IEEE Trans. Reliab. 1983, R-32, 475–484. [Google Scholar] [CrossRef]
  10. Zhao, M.; Xie, M. On maximum likelihood estimation for a general non-homogeneous Poisson process. Scand. J. Stat. 1996, 23, 597–607. [Google Scholar]
  11. Abdel-Ghaly, A.A.; Chan, P.Y.; Littlewood, B. Evaluation of competing software reliability predictions. IEEE Trans. Softw. Eng. 1986, SE-12, 950–967. [Google Scholar] [CrossRef]
  12. Crétois, E.; Gaudoin, O. New Results on Goodness-of-Fit Tests for the Power-Law Process and Application to Software Reliability. Int. J. Reliab. Qual. Saf. Eng. 1998, 5, 249–267. [Google Scholar] [CrossRef]
  13. Duane, J.T. Learning Curve Approach to Reliability Monitoring. IEEE Trans. Aerosp. 1964, 2, 563–566. [Google Scholar] [CrossRef]
  14. Littlewood, B. Rationale for a modified Duane model. IEEE Trans. Reliab. 1984, R-33, 157–159. [Google Scholar] [CrossRef]
  15. Musa, J.D.; Okumoto, K. A logarithmic Poisson execution time model for software reliability measurement. In Proceedings of the 7th International Conference on Software Engineering, Orlando, FL, USA, 26–29 March 1984; pp. 230–238. [Google Scholar]
  16. Musa, J.D.; Iannino, A.; Okumoto, K. Software Reliability, Measurement, Prediction, Application; McGraw-Hill: New York, NY, USA, 1987. [Google Scholar]
  17. Okamura, H.; Dohi, T. SRATS: Software reliability assessment tool on spreadsheet (Experience report). In Proceedings of the 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE), Pasadena, CA, USA, 4–7 November 2013; pp. 100–107. [Google Scholar]
  18. Lyu, M.R. Handbook of Software Reliability Engineering; IEEE Computer Society Press: Los Alamitos, CA, USA, 1996; Volume 222. [Google Scholar]
  19. Min, X. Software Reliability Modeling; World Scientific: Singapore, 1991. [Google Scholar]
  20. Jun, H.; Shigeru, Y.; Shunji, O. Reliability assessment measures based on software reliability growth model with normalized method. J. Inf. Process. 1991, 14, 178–183. [Google Scholar]
  21. Barlow, R.E.; Proschan, F. Mathematical Theory of Reliability, 1965; SIAM: Philadelphia, PA, USA, 1996. [Google Scholar]
  22. Cox, D.; Lewis, P.A.W. The Statistical Analysis of Series of Events; Springer: Dordrecht, The Netherlands, 1966. [Google Scholar]
  23. Musa, J.D. Software Reliability Data; Technical Report in Rome Air Development Center; 1979. [Google Scholar]
  24. Vouk, M.A. Using reliability models during testing with non-operational profiles. In Proceedings of the 2nd Bellcore/Purdue Workshop on Issues in Software Reliability Estimation; IEEE: Manhattan, NY, USA, 1992; pp. 103–111. [Google Scholar]
  25. Okamura, H.; Etani, Y.; Dohi, T. Quantifying the effectiveness of testing efforts on software fault detection with a logit software reliability growth model. In Proceedings of the 2011 Joint Conference of the 21st International Workshop on Software Measurement and the 6th International Conference on Software Process and Product Measurement, Nara, Japan, 3–4 November 2011; pp. 62–68. [Google Scholar]
  26. Wood, A. Predicting software reliability. Computer 1996, 29, 69–77. [Google Scholar] [CrossRef]
Figure 1. Behavior of mean value functions in TDDS1.
Figure 1. Behavior of mean value functions in TDDS1.
Software 02 00001 g001
Figure 2. Behavior of mean value functions in TIDS7.
Figure 2. Behavior of mean value functions in TIDS7.
Software 02 00001 g002
Figure 3. Behavior of fault prediction in TDDS1 (20% observation point).
Figure 3. Behavior of fault prediction in TDDS1 (20% observation point).
Software 02 00001 g003
Figure 4. Behavior of fault prediction in TDDS1 (50% observation point).
Figure 4. Behavior of fault prediction in TDDS1 (50% observation point).
Software 02 00001 g004
Figure 5. Behavior of fault prediction in TDDS1 (80% observation point).
Figure 5. Behavior of fault prediction in TDDS1 (80% observation point).
Software 02 00001 g005
Figure 6. Behavior of fault prediction in TIDS7 (20% observation point).
Figure 6. Behavior of fault prediction in TIDS7 (20% observation point).
Software 02 00001 g006
Figure 7. Behavior of fault prediction in TIDS7 (50% observation point).
Figure 7. Behavior of fault prediction in TIDS7 (50% observation point).
Software 02 00001 g007
Figure 8. Behavior of fault prediction in TIDS7 (80% observation point).
Figure 8. Behavior of fault prediction in TIDS7 (80% observation point).
Software 02 00001 g008
Table 1. The representative existing finite-failure (type-I) NHPP-based SRMs.
Table 1. The representative existing finite-failure (type-I) NHPP-based SRMs.
SRM & Time Distribution F ( t ; α ) M ( t ; θ )
Exp [2]
(Exponential distribution)
1 exp ( μ 1 t ) μ 0 F ( t ; α )
Gamma [9,10]
(Gamma distribution)
0 t μ 2 μ 1 s μ 2 1 exp ( μ 2 s ) Γ ( μ 1 ) d s μ 0 F ( t ; α )
Pareto [11]
(Pareto distribution)
1 ( μ 1 t + μ 1 ) μ 2 μ 0 F ( t ; α )
Tnorm [3]
(Truncated normal distribution)
1 2 π μ 1 t exp ( ( s μ 2 ) 2 2 μ 1 2 ) d s μ 0 F ( t ; α ) F ( 0 ; α ) 1 F ( 0 ; α )
Tlogist [5]
(Truncated logistic distribution)
1 exp ( μ 1 t ) 1 + μ 2 exp ( μ 2 t ) μ 0 F ( t ; α ) F ( 0 ; α ) 1 F ( 0 ; α )
Txvmax [8]
(Truncated extreme-value maximum distribution)
exp ( exp ( t μ 2 μ 1 ) ) μ 0 F ( t ; α ) F ( 0 ; α ) 1 F ( 0 ; α )
Txvmin [8]
(Truncated extreme-value minimum distribution)
exp ( exp ( t μ 2 μ 1 ) ) μ 0 F ( 0 ; α ) F ( t ; α ) F ( 0 ; α )
Lnorm [3,4]
(Log-normal distribution)
1 2 π μ 1 t exp ( ( s μ 2 ) 2 2 μ 1 2 ) d s μ 0 F ( ln   t ; α )
Llogist [6]
(Log-logistic distribution)
1 exp ( μ 1 t ) 1 + μ 2 exp ( μ 2 t ) μ 0 F ( ln   t ; α )
Lxvmax [8]
(Log-extreme-value maximum distribution)
exp ( exp ( t μ 2 μ 1 ) ) μ 0 F ( ln   t ; α )
Lxvmin [7]
(Log-extreme-value minimum distribution)
exp ( exp ( t μ 2 μ 1 ) ) μ 0 ( 1 F ( ln t ; α ) )
( μ 0 > 0 , μ 1 > 0 , μ 2 > 0 ) ; Γ ( ) :   standard   gamma   function ; erfc ( ) :   complementary   error   function ;   ln ( ) :   natural   logarithmic   function .
Table 2. Infinite-failure (type-II) NHPP-based SRMs.
Table 2. Infinite-failure (type-II) NHPP-based SRMs.
SRM & Time Distribution F ( t ; α ) M ( t ; θ )
Exp (HPP)
(Exponential distribution)
1 exp ( μ 1 t ) μ 1 t
Gamma
(Gamma distribution)
0 t μ 2 μ 1 s μ 2 1 exp ( μ 2 s ) Γ ( μ 1 ) d s ln ( Γ ( μ 1 ) ) ln ( Γ ( μ 1 , t μ 2 ) )
Pareto (Musa-Okumoto) [15,16]
(Pareto distribution)
1 ( μ 1 t + μ 1 ) μ 2 μ 2 ( ln ( μ 1 ) ln ( μ 1 + t ) )
Tnorm
(Truncated normal distribution)
1 2 π μ 1 t exp ( ( s μ 2 ) 2 2 μ 1 2 ) d s ln ( erf ( μ 2 2 μ 1 ) + 1 ) ln ( erf ( μ 2 t 2 μ 1 ) + 1 )
Tlogist
(Truncated logistic distribution)
1 exp ( μ 1 t ) 1 + μ 2 exp ( μ 2 t ) ln ( exp ( μ 2 / μ 1 ) + exp ( t / μ 1 ) ) ln ( exp ( μ 2 / μ 1 ) + 1 )
Txvmax
(Truncated extreme-value maximum distribution)
exp ( exp ( t μ 2 μ 1 ) ) ln ( 1 exp ( exp ( μ 2 / μ 1 ) ) ) ln ( 1 exp ( exp ( μ 2 t μ 1 ) ) )
Cox-Lewis [22]
(Truncated extreme-value minimum distribution)
exp ( exp ( t μ 2 μ 1 ) ) ln ( exp ( exp ( μ 2 / μ 1 ) ( exp ( t / μ 1 ) 1 ) ) )
Lnorm
(Log-normal distribution)
1 2 π μ 1 t exp ( ( s μ 2 ) 2 2 μ 1 2 ) d s ln ( 2 ) ln ( erf ( μ 2 ln ( t ) 2 μ 1 ) + 1 )
Llogist
(Log-logistic distribution)
1 exp ( μ 1 t ) 1 + μ 2 exp ( μ 2 t ) ln ( exp ( μ 2 / μ 1 ) + t 1 / μ 1 ) μ 2 / μ 1
Lxvmax
(Log-extreme-value maximum distribution)
exp ( exp ( t μ 2 μ 1 ) ) ln ( 1 exp ( exp ( μ 2 ln ( t ) μ 1 ) ) )
Power-law [12,13,14]
(Log-extreme-value minimum distribution)
exp ( exp ( t μ 2 μ 1 ) ) μ 2 / μ 1 t 1 / μ 1  
Table 3. Software fault-count time-domain datasets.
Table 3. Software fault-count time-domain datasets.
Data SourceNature of SystemTesting Length
(CPU Time)
Numbers of Detected Faults
TDDS1SYS2 [23]Real-time command and control system10870854
TDDS2S10 [23]Real-time command and control system23370038
TDDS3SYS3 [23]Military application6736238
TDDS4S27 [23]Single-user workstation431259841
TDDS5SYS4 [23]Operating system5242253
TDDS6Project J5 [18]Real-time command and control system509073
TDDS7S17 [23]Single-user workstation19572126101
TDDS8SYS1 [23]Single-user workstation88682136
Table 4. Software fault-count time-interval datasets (group data).
Table 4. Software fault-count time-interval datasets (group data).
Data SourceNature of SystemTesting Length
(Week)
Numbers of Detected Faults
TIDS1SYS2 [23]Real-time command and control system1754
TIDS2NASA-supported project [24]Inertial navigating system149
TIDS3SYS3 [23]Military application1438
TIDS4DS3 [25]Embedded application for printer3052
TIDS5DS2 [25]Embedded application for printer3358
TIDS6Release 3 [26]Tandem software system1261
TIDS7DS1 [25]Embedded application for printer2066
TIDS8Release 2 [26]Tandem software system19120
Table 5. Goodness-of-fit results in time-domain data.
Table 5. Goodness-of-fit results in time-domain data.
Type-I NHPPType-II NHPP
Best SRMAICMSEBest SRMAICMSE
TDDS1Lxvmax896.6661.950Musa-Okumoto895.3052.315
TDDS2Lxvmax721.9281.442Cox-Lewis726.0522.803
TDDS3Lxvmax598.1311.705Musa-Okumoto596.5011.809
TDDS4Lxvmax1008.2205.970Musa-Okumoto1007.1007.039
TDDS5Txvmin759.5793.747Cox-Lewis759.9485.509
TDDS6Exp757.86918.985Power-law757.03119.315
TDDS7Pareto2504.17047.404Musa-Okumoto2503.37063.699
TDDS8Lxvmin1938.1606.570Musa-Okumoto1939.6008.052
Table 6. Goodness-of-fit results in group data.
Table 6. Goodness-of-fit results in group data.
Type-I NHPPType-II NHPP
Best SRMAICMSEBest SRMAICMSE
TIDS1Llogist73.0534.115Tlogist85.33948.269
TIDS2Exp29.9110.118Exp27.7530.186
TIDS3Lxvmax61.6943.239Llogist60.6743.557
TIDS4Llogist117.4709.408Llogist148.43845.178
TIDS5Txvmin123.2652.122Tlogist138.02924.847
TIDS6Tlogist51.0521.968Cox-Lewis63.55627.199
TIDS7Lxvmax108.83122.514Llogist107.21124.394
TIDS8Tnorm87.2676.151Cox-Lewis91.91931.232
Table 7. Prediction results in time-domain data.
Table 7. Prediction results in time-domain data.
20% Observation Point
Type-I NHPP Type-II NHPP
Best SRMPMSEBest SRMPMSE
TDDS1Lxvmax5.073Musa-Okumoto6.420
TDDS2Txvmin83.964Llogist79.614
TDDS3Tnorm42.104Musa-Okumoto145.648
TDDS4Lxvmax32.217Llogist207.592
TDDS5Lnorm56.477Musa-Okumoto198.490
TDDS6Exp9177.670Tlogist467.320
TDDS7Lxvmax1852.520Lnorm1474.020
TDDS8Lxvmax32.131Power-law1417.110
50% Observation Point
Type-I NHPPType-II NHPP
Best SRMPMSEBest SRMPMSE
TDDS1Pareto6.118Musa-Okumoto6.420
TDDS2Lxvmax10.493Llogist30.944
TDDS3Txvmin5.874Llogist11.747
TDDS4Exp4480.620Llogist18.425
TDDS5Tlogist103.504Cox-Lewis106.282
TDDS6Llogist193.903Tlogist77.498
TDDS7Txvmin3569.230Musa-Okumoto45.344
TDDS8Pareto11.712Musa-Okumoto10.283
80% Observation Point
Type-I NHPPType-II NHPP
Best SRMPMSEBest SRMPMSE
TDDS1Lxvmax5.772Power-law3.432
TDDS2Lxvmax2.041Lxvmax3.697
TDDS3Lxvmax0.588Musa-Okumoto0.819
TDDS4Txvmin6.875Power-law4.291
TDDS5Txvmin4.253Cox-Lewis4.258
TDDS6Lxvmax21.715Power-law51.677
TDDS7Lxvmax57.901Power-law9.268
TDDS8Lxvmax9.419Power-law819.992
Table 8. Prediction results in group data.
Table 8. Prediction results in group data.
20% Observation Point
Type-I NHPP Type-II NHPP
Best SRMPMSEBest SRMPMSE
TIDS1Gamma220.732Power-law218.763
TIDS2Pareto2.628Musa-Okumoto2.625
TIDS3Lxvmax29.244Llogist47.377
TIDS4Txvmin448.935Cox-Lewis423.360
TIDS5Exp387.694Cox-Lewis67.730
TIDS6Exp142.854Tlogist86.083
TIDS7Tlogist98.903Llogist25.613
TIDS8Gamma820.049Gamma171.702
50% Observation Point
Type-I NHPPType-II NHPP
Best SRMPMSEBest SRMPMSE
TIDS1Txvmin96.992Musa-Okumoto159.545
TIDS2Exp0.344Musa-Okumoto0.347
TIDS3Txvmin30.786Power-law3.722
TIDS4Txvmin29.097Llogist156.329
TIDS5Lxvmax22.894Gamma27.045
TIDS6Exp101.303Musa-Okumoto101.258
TIDS7Pareto365.493Gamma18.825
TIDS8Lxvmax564.782Gamma849.736
80% Observation Point
Type-I NHPPType-II NHPP
Best SRMPMSEBest SRMPMSE
TIDS1Lnorm1.762Llogist8.736
TIDS2Tnorm0.224Lxvmax0.090
TIDS3Exp0.464Cox-Lewis0.464
TIDS4Tnorm0.864Llogist6.333
TIDS5Txvmin6.118Llogist17.300
TIDS6Lxvmax1.850Llogist18.985
TIDS7Lnorm3.432Llogist6.144
TIDS8Tnorm0.331Cox-Lewis41.228
Table 9. Software reliability assessment in time-domain data.
Table 9. Software reliability assessment in time-domain data.
Type-I NHPP Type-II NHPP
Best SRMReliabilityBest SRMReliability
TDDS1Lxvmax 2.631 × 10 6 Musa-Okumoto 2.674 × 10 6
TDDS2Lxvmax 3.283 × 10 4 Cox-Lewis 4.694 × 10 8
TDDS3Lxvmax 3.687 × 10 3 Musa-Okumoto 3.751 × 10 7
TDDS4Lxvmax 2.453 × 10 4 Musa-Okumoto 2.398 × 10 4
TDDS5Txvmin 4.573 × 10 1 Cox-Lewis 3.231 × 10 3
TDDS6Exp 1.035 × 10 5 Power-law 2.596 × 10 8
TDDS7Pareto 8.971 × 10 6 Musa-Okumoto 7.736 × 10 6
TDDS8Lxvmin 4.592 × 10 5 Musa-Okumoto 2.516 × 10 10
Table 10. Software reliability assessment in group data.
Table 10. Software reliability assessment in group data.
Type-I NHPP Type-II NHPP
Best SRMReliabilityBest SRMReliability
TIDS1Llogist 4.152 × 10 3 Tlogist 2.217 × 10 25
TIDS2Exp 9.832 × 10 4 Exp 1.234 × 10 4
TIDS3Lxvmax 7.236 × 10 5 Llogist 6.264 × 10 5
TIDS4Llogist 6.373 × 10 1 Llogist 4.052 × 10 10
TIDS5Txvmin 9.633 × 10 1 Tlogist 1.280 × 10 27
TIDS6Tlogist 2.816 × 10 1 Cox-Lewis 3.221 × 10 27
TIDS7Lxvmax 1.939 × 10 7 Llogist 3.892 × 10 7
TIDS8Tnorm 3.865 × 10 2 Cox-Lewis 2.203 × 10 23
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Li, S.; Dohi, T.; Okamura, H. Are Infinite-Failure NHPP-Based Software Reliability Models Useful? Software 2023, 2, 1-18. https://doi.org/10.3390/software2010001

AMA Style

Li S, Dohi T, Okamura H. Are Infinite-Failure NHPP-Based Software Reliability Models Useful? Software. 2023; 2(1):1-18. https://doi.org/10.3390/software2010001

Chicago/Turabian Style

Li, Siqiao, Tadashi Dohi, and Hiroyuki Okamura. 2023. "Are Infinite-Failure NHPP-Based Software Reliability Models Useful?" Software 2, no. 1: 1-18. https://doi.org/10.3390/software2010001

Article Metrics

Back to TopTop