Using Statistical Test Method to Establish a Decision Model of Performance Evaluation Matrix

Abstract

Many studies have pointed out that the Performance Evaluation Matrix (PEM) is a convenient and useful tool for the evaluation, analysis, and improvement of service operating systems. All service items of the operating system can collect customer satisfaction and importance through questionnaires and then convert them into satisfaction indices and importance indices to establish PEM and its evaluation rules. Since the indices have unknown parameters, if the evaluation is performed directly by the point estimates of the indices, there will be a risk of misjudgment due to sampling error. In addition, most of the studies only determine the critical-to-quality (CTQ) that needs to be improved, and do not discuss the treatment rules in the case of limited resources nor perform the confirmation after improvement. Therefore, to address similar research gaps, this paper proposed the unbiased estimators of these two indices and determined the critical-to-quality (CTQ) service items which need to be improved through the one-tailed statistical hypothesis test by building a PEM method of the satisfaction index. In addition, through the one-tailed statistical hypothesis test method of the importance index, the improvement priority of service items was determined under the condition of limited resources. Confirmation of the effect on improvement is an important step in management. Thus, this paper adopted a statistical two-tailed hypothesis test to verify whether the satisfaction of all the CTQ service items that need to be improved was enhanced. Since the method proposed in this paper was established through statistical hypothesis tests, the risk of misjudgment due to sampling error could be reduced. Obviously, reducing the misjudgment risk is the advantage of the method in this paper. Based on the precondition, utilizing the model in this study may assist the industries to determine CTQ rapidly, implement the most efficient improvement under the condition of limited resources and also confirm the improvement effect at the same time. Finally, a case study of computer-assisted language learning system (CALL System) was used to illustrate a way to apply the model proposed in this paper.

1. Introduction

In the face of global warming and the impact of the COVID-19 pandemic in recent years, human coexistence with the natural environment has become a relatively important issue. The awareness of sustainable development that enterprises and environmental protection have in common has also become the major concern for governments and enterprises in various countries. According to many studies, as the Internet of Things (IoT) has gradually become popular and mature, various smart APP platforms have also sprung up [1,2,3,4]. Not only can efficiency be leveled up, but the impact on the environment can also be reduced. For example, by means of the digital learning model of the teaching APP, students can eliminate the inconvenience of going out as well as achieve the effect of learning at home. By doing so, not only can the amount of traffic jams be reduced, but the time of going out can also be saved, so that the effect of energy saving and carbon reduction can be achieved. Based on the above scenario, constructing a complete evaluation and improvement model of the digital learning effectiveness of the teaching APP will help attract more users to achieve digital learning effect by means of the teaching APP in order to achieve the abov-mentioned effect of alleviating the congestion problem, saving energy, and reducing carbon emissions.

Some methods such as Quality Function Deployment (QFD) and Balanced Scorecard (BSC) are employed to weigh performance evaluations or product designs, and some apply Fuzzy inference system (FIS) to increase the competitive capability and customer satisfaction [5,6,7]. According to numerous studies, the Performance Evaluation Matrix (PEM) is a convenient and useful evaluation tool for the service operating systems. Thus, plenty of dissertations have engaged in the research on PEM [8,9,10,11]. In order not to lose generality, this paper assumes that there are q key service items of digital learning provided by the teaching APP, so q questions are designed to investigate the importance and satisfaction of the q service items for learners. In addition, many studies assert that since Beta assigns values between 0 and 1, it represents satisfaction ranging from 0, which is completely dissatisfied, to 1, which is 100% satisfied. Similarly, the importance ranges from 0, which means total ignorance, to 1, which refers to great importance. Consequently, satisfaction and importance are regarded as random variables subject to the Beta distribution [12,13]. Hung, Huang, and Chen [14] proposed a standardized and significant importance index and satisfaction index based on the two parameters assigned by Beta distribution. If random variable $X_i$ represents the satisfaction of the ith service item, then $X_i$ is distributed as the Beta distribution denoted by $X_i~Beta\left(a_i^S,b_i^S\right)$, where $a_i^S$ and $b_i^S$ are two parameters. Similarly, if random variable $Y_i$ indicates the importance attached to the ith service item, then $Y_i$ is distributed as the Beta distribution denoted by $Y_i~Beta\left(a_i^I,b_i^I\right)$, where $a_i^I$ and $b_i^I$ are two parameters. Thus, the satisfaction index and the importance index can be expressed, as follows, for satisfaction:

$${\theta }_i^S=\frac{a_i^S}{a_i^S+b_i^S} \left(\mathrm{satisfaction}\mathrm{index}\right),$$

(1)

$${\theta }_i^I=\frac{a_i^I}{a_i^I+b_i^I} \left(\mathrm{importance}\mathrm{index}\right).$$

(2)

According to the abovementioned data, the values of the two indices are between 0 and 1. In this paper, we set the satisfaction index as the x-axis and set the importance index as the y-axis to form the performance evaluation matrix. Since the indices have unknown parameters, if the evaluation is performed directly by the point estimates of the indices, there will be a risk of misjudgment due to sampling error [15,16,17]. In addition, most studies just determine the critical-to-quality (CTQ) that needs to be improved, without discussing the processing rules in resource-limited settings, and without the confirmation of the improvements. In order to make up for such research gaps, this paper adopts the one-tailed statistical hypothesis testing method of satisfaction index, determines the CTQ service items that need to be improved, and then defines their improvement order. Then, this paper uses a statistical two-tailed hypothesis test to verify whether the satisfaction of all CTQ service items that need to be improved is improved. First, this paper detects the unbiased estimators of the satisfaction index and the importance index for all service items. However, according to numerous studies, if the PEM locations of the point estimates are directly used to determine whether the service items need to be improved, then the risk of misjudgment may be caused by sampling error [18,19]. Therefore, this paper uses the average value ${\theta }_0^S$ of all satisfaction indices as the testing standard in accordance with the spirit of continuous improvement on the comprehensive quality management and employs the statistical one-tailed test to determine the critical value $C_0^S$. At the same time, the average value ${\theta }_0^I$ of all importance indices is adopted as the testing standard, and the statistical one-tailed test is employed to determine the critical value. Next, this paper establishes the evaluation blocks and evaluation rules of PEM based on the critical value derived from the statistical test and judges whether the service item needs to be improved according to the location of the evaluation coordinate point $\left(x_i,y_i\right)=\left({\widehat{\theta }}_i^S,{\widehat{\theta }}_i^I\right)$ of each service item, where ${\widehat{\theta }}_i^S$ is the estimator of ${\theta }_i^S$ and ${\widehat{\theta }}_i^I$ is the estimator of ${\theta }_i^I$. Meanwhile, according to the locations of the evaluation coordinate points of the service items that need to be improved in the evaluation quadrant of PEM, the priority of improvement is determined under the condition of limited resources. Because this method is established by statistical tests, the risk of misjudgment caused by sampling error can be lowered. After the task of improvement is completed, the effectiveness of the improvement must be confirmed. Therefore, this paper then adopts the statistical two-tailed test to verify whether the satisfaction of all service items that need to be improved is leveled up. Finally, a case study of the computer-assisted language learning system (CALL System) is used in this paper to illustrate the application of the abovementioned statistical test rules to define the key service items that need to be improved and to determine the priority of improvement in the situation of limited resources.

The remainder of this paper Is arranged as follows. In Section 2, we propose the unbiased estimators and the statistical hypothesis test method for the satisfaction index and the importance index, respectively. According to the statistical test results, we procure the service items that need to be improved as well as determine the order of improvement for all service items. In Section 3, we define the evaluation block of the Performance Evaluation Matrix based on the two critical values derived from the statistical test conducted in Section 2; at the same time, we also define evaluation rules. In Section 4, we adopt a statistical two-tailed test to verify the improvement effects on all key service items that need to be improved. In Section 5, we design a case study of the computer-assisted language learning system (CALL System), illustrating the application of the method proposed in this paper. In Section 6, we offer conclusions.

2. Unbiased Estimators and Statistical Hypothesis Test

As mentioned earlier, since the indices have unknown parameters, they must be estimated by sample data of the interviewed customers. Due to the application of the Central Limit Theorem in this paper to carry out the statistical test, the sample size must be sufficiently large. The sample size of the cases applied to the method in this study are usually larger than 100. They are much larger than the minimum sample size required in the Central Limit Theorem (the minimum sample size for requirements is 30). In this section, we propose the unbiased estimators and the statistical hypothesis test method for the satisfaction index and the importance index, respectively.

2.1. Satisfaction Index

First, we assume that the satisfaction sample data of the customer’s ith service item is $\left(X_{i,1},\cdots , X_{i,j},\cdots , X_{i,n}\right)$, $i=1,2,\dots ,q$. Then, the estimator of the satisfaction index for service item i is expressed as follows:

$${\widehat{\theta }}_i^S=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{X}_{i,j}}.$$

(3)

The expected value of ${\widehat{\theta }}_i^S$ is

$$\begin{array}{cc}E\left[{\widehat{\theta }}_i^S\right]& =\frac{1}{n}\times {\displaystyle \sum _{j=1}^{n}E\left[X_{i,j}\right]}\hfill \\ & =\frac{1}{n}\times {\displaystyle \sum _{i=1}^n{\displaystyle \underset{0}{\overset{1}{\int }}x\frac{\Gamma \left(a_i^S+b_i^S\right)}{\Gamma \left(a_i^S\right)\Gamma \left(b_i^S\right)}{x}^{a_i^S}{\left(1-x\right)}^{b_i^S}{d}_x}}\hfill \\ & =\frac{1}{n}\times {\displaystyle \sum _{i=1}^n\left(\frac{a_i^S}{a_i^S+b_i^S}\right)}\hfill \\ & ={\theta }_i^S. \hfill \end{array}$$

(4)

Thus, ${\widehat{\theta }}_i^S$ is an unbiased estimator of ${\theta }_i^S$.

$${\theta }_{\mathrm{S}}=\frac{1}{q}{\displaystyle \sum _{i=1}^q{\widehat{\theta }}_i^S}.$$

(5)

Then, based on the concept of total quality management, when the satisfaction index value of service item i is greater than or equal to the average value ${\theta }_{\mathrm{S}}$, it is fine to maintain the current situation. When the satisfaction index value of service item i is smaller than the average value ${\theta }_{\mathrm{S}}$, then the service item i needs to be improved [20,21]. Based on the stated above, the hypotheses of the statistical test for satisfaction index i are as follows:

$$\mathrm{null} \mathrm{hypothesis} H_0: {\theta }_i^S\ge {\theta }_{\mathrm{S}};$$

(6)

$$\mathrm{alternative} \mathrm{hypothesis} H_1: {\theta }_i^S<{\theta }_{\mathrm{S}}.$$

(7)

We assume that the test statistic is the unbiased estimator ${\widehat{\theta }}_i^S$ with significance level $\alpha$; then, the critical region for satisfaction index i is

$$C{R}_{Xi}=\left\{\left.{\widehat{\theta }}_i^S\right|{\widehat{\theta }}_i^S<C_{X0i}\right\}.$$

(8)

Thus, we have

$$p\left(\left.{\widehat{\theta }}_i^S<C_{X0i}\right|{\theta }_i^S={\theta }_{\mathrm{S}}\right)=\alpha .$$

(9)

Equivalently,

$$p\left({\mathrm{Z}}_{Xi}<\frac{\sqrt{n}\left(C_{X0i}-{\theta }_{\mathrm{S}}\right)}{S_{Xi}}\right)=\alpha ,$$

(10)

where

$$S_{Xi}=\sqrt{\frac{1}{n-1}\times {\displaystyle \sum _{j=1}^n{\left(X_{i,j}-{\overline{X}}_i\right)}^2}}$$

(11)

and

$${\mathrm{Z}}_{Xi}=\frac{\sqrt{n}\left({\theta }_i^S-{\theta }_{\mathrm{S}}\right)}{S_{Xi}}$$

(12)

are approximate values, and they are distributed as standard normal distributions when sample size n is large. Then, we have

$$\frac{\sqrt{n}\left(C_{X0i}-{\theta }_S\right)}{S_{Xi}}=-z_{\alpha },$$

where the critical value $C_{X0i}$ is displayed below:

$$C_{X0i}={\theta }_{\mathrm{S}}-z_{\alpha }\frac{S_{Xi}}{\sqrt{n}}.$$

(13)

2.2. Importance Index

We assume the importance sample data of the customer’s ith service item is $\left(Y_{i,1},\cdots , Y_{i,j},\cdots , Y_{i,n}\right)$, $i=1,2,\dots ,q$; then, the estimator of importance index for service item i is expressed as follows:

$${\widehat{\theta }}_i^I=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{Y}_{i,j}}.$$

(14)

The expected value of ${\widehat{\theta }}_i^I$ is

$$\begin{array}{cc}E\left[{\widehat{\theta }}_i^I\right]& =\frac{1}{n}\times {\displaystyle \sum _{j=1}^{n}E\left[Y_{i,j}\right]}\hfill \\ & =\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{\displaystyle \underset{0}{\overset{1}{\int }}y\frac{\Gamma \left(a_i^I+b_i^I\right)}{\Gamma \left(a_i^I\right)\Gamma \left(b_i^I\right)}{y}^{a_i^I}{\left(1-y\right)}^{b_i^I}{d}_y}}\hfill \\ & =\frac{1}{n}\times {\displaystyle \sum _{i=1}^n\frac{\Gamma \left(a_i^I+b_i^I\right)\Gamma \left(a_i^I+1\right)}{\Gamma \left(a_i^I+b_i^I+1\right)}\times {\displaystyle \underset{0}{\overset{1}{\int }}Beta\left(a_i^I+1,b_i^I\right)d_y}} \hfill \\ & =\frac{1}{n}\times {\displaystyle \sum _{i=1}^n\left(\frac{a_i^I}{a_i^I+b_i^I}\right)}\hfill \\ & ={\theta }_i^I. \hfill \end{array}$$

(15)

Thus, ${\widehat{\theta }}_i^I$ is an unbiased estimator of ${\theta }_i^I$.

$${\theta }_I=\frac{1}{q}{\displaystyle \sum _{i=1}^q{\widehat{\theta }}_i^I}.$$

(16)

Then, when the importance index value of service item i is greater than or equal to the average value ${\theta }_I$, the priority of improvement is high. When the importance index value of service item i is less than the average value ${\theta }_I$, the priority of improvement is low. Based on the above statements, the hypotheses of the statistical test for importance index i are defined as follows:

$$\mathrm{null} \mathrm{hypothesis} H_0: {\theta }_i^I\ge {\theta }_I;$$

(17)

$$\mathrm{alternative} \mathrm{hypothesis} H_1: {\theta }_i^I<{\theta }_I.$$

(18)

We assume the test statistic is the unbiased estimator ${\widehat{\theta }}_i^I$ with significance level $\alpha$; then, the critical region for importance index i is

$$C{R}_{Yi}=\left\{\left.{\widehat{\theta }}_i^I\right|{\widehat{\theta }}_i^I<C_{Y0i}\right\}.$$

(19)

Thus, we have

$$p\left(\left.{\widehat{\theta }}_i^I<C_{Y0i}\right|{\theta }_i^I={\theta }_I\right)=\alpha .$$

(20)

Equivalently,

$$p\left(Z_{Yi}<\frac{\sqrt{n}\left(C_{Y0i}-{\theta }_I\right)}{S_{Yi}}\right)=\alpha ,$$

(21)

where

$$S_{Yi}=\sqrt{\frac{1}{n-1}\times {\displaystyle \sum _{j=1}^n{\left(Y_{i,j}-{\overline{Y}}_i\right)}^2}}$$

(22)

and

$$Z_{Yi}=\frac{\sqrt{n}\left({\theta }_i^I-{\theta }_I\right)}{S_{Yi}}$$

(23)

are approximate values, and they are distributed as standard normal distributions when sample size n is large. Then,

$$C_{Y0i}={\theta }_I-z_{\alpha }\frac{S_{Yi}}{\sqrt{n}}.$$

(24)

3. Performance Evaluation Matrix

As mentioned earlier, the performance evaluation matrix is widely used to evaluate and improve the performance levels of q service items for various service systems. In this paper, the satisfaction index is set as x-axis and the importance index is set as y-axis to form the performance evaluation matrix. Then, the horizontal line $y=C_{Y0}$ and the vertical line $x=C_{X0}$ are used to divide the performance evaluation matrix into four quadrants, where $C_{X0}=Min\left\{C_{X01},C_{X02},\dots ,C_{X0q}\right\}$ and $C_{Y0}=Min\left\{C_{Y01},C_{Y02},\dots ,C_{Y0q}\right\}$. These four quadrants can be defined as follows:

$$\mathrm{Quadrant} 1: Q_1=\left\{\left(x,y\right)|C_{X0}\le x\le 1,C_{Y0}\le y\le 1\right\},$$

(25)

$$\mathrm{Quadrant} 2: Q_2=\left\{\left(x,y\right)|0\le x<C_{X0},C_{Y0}\le y\le 1\right\},$$

(26)

$$\mathrm{Quadrant} 3: Q_3=\left\{\left(x,y\right)|0\le x<C_{X0},0\le y\le C_{Y0}\right\},$$

(27)

$$\mathrm{Quadrant} 4: Q_4=\left\{\left(x,y\right)|C_{X0}\le x\le 1,0\le y\le C_{Y0}\right\}.$$

(28)

Then, the performance evaluation matrix and four quadrants can be displayed in Figure 1 as follows:

Obviously, when the evaluation coordinate points fall into Quadrant 2 and Quadrant 3, it means that the satisfaction of the service items is lower than the average satisfaction, so that the service items definitely need to be improved. Since the evaluation coordinate points falling into Quadrant 2 are more important than the ones falling into Quadrant 3, the priority of their improvement is higher. In addition, when the evaluation coordinate points fall into Quadrant 1 and Quadrant 4, it means that the satisfaction of the service items is higher than the average satisfaction, so that there is no need to make any improvement, and the current state can remain unchanged. Based on the above concept, we can define the evaluation coordinates for the ith service item as follows:

$$\left(x_i,y_i\right)=\left({\widehat{\theta }}_i^S,{\widehat{\theta }}_i^I\right).$$

(29)

Then, the evaluation rules and improvement ranking rules are established as follows:

(1)

If $\left(x_i,y_i\right)\in Q_1\cup Q_4$, then service item i does not need to make any improvement.

(2)

If $\left(x_i,y_i\right)\in Q_2\cup Q_3$, then service item i needs to be improved.

(3)

If $\left(x_i,y_i\right)\in Q_2$, then the service item i is ranked the first priority of improvement.

(4)

If $\left(x_i,y_i\right)\in Q_3$, then the service item i is ranked the second priority of improvement.

4. Statistical Two-Tailed Test

As noted above, confirming the effect upon the improvement is an important step in management. Therefore, we use a statistical two-tailed test to judge whether the satisfaction of all the CTQ service items needs to be enhanced in this section. We let set A be the set formed by the service items that need to be enhanced; then,

$$A=\left\{i|\left(x_i,y_i\right)\in Q_2\cap Q_3\right\}.$$

(30)

${\theta }_{bi}^S$ is the value of satisfaction index before improvement, ${\theta }_{ai}^S$ is the value of satisfaction index after improvement, and $i\in A$. If the satisfaction sample data are $\left({X^{\prime }}_{i,1},\cdots , {X^{\prime }}_{i,j},\cdots , {X^{\prime }}_{i,n}\right)$ for $i\in A$, then the estimator of ${\theta }_{ai}^S$ can be shown as follows:

$${\widehat{\theta }}_{ai}^S=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{X^{\prime }}_{i,j}}.$$

(31)

Similarly, the estimator of ${\theta }_{bi}^S$ for $i\in A$ as follows:

$${\widehat{\theta }}_{bi}^S=\frac{1}{n}\times {\displaystyle \sum _{j=1}^n{X}_{i,j}}.$$

(32)

Then,

$${\mathrm{Z}}_{hi}=\frac{\sqrt{n}\left({\widehat{\theta }}_{hi}^S-{\theta }_{hi}^{\mathrm{S}}\right)}{S_{Xhi}}, h=a,b \mathrm{and} i\in A,$$

(33)

where

$$S_{Xai}=\sqrt{\frac{1}{n-1}\times {\displaystyle \sum _{j=1}^n{\left(X_{i,j}^{\prime }-{\overline{X}}_i^{\prime }\right)}^2}} \mathrm{and} S_{Xbi}=\sqrt{\frac{1}{n-1}\times {\displaystyle \sum _{j=1}^n{\left(X_{i,j}-{\overline{X}}_i\right)}^2}}.$$

(34)

Then, ${\mathrm{Z}}_{hi}$ is an approximate value, and it is distributed as a standard normal distribution when sample size n is large. Thus,

$$\begin{array}{cc}1-\alpha & =p\left(-Z_{\alpha /2}\le {\mathrm{Z}}_{hi}\le Z_{\alpha /2}\right)\hfill \\ & =p\left(-Z_{\alpha /2}\le \frac{\sqrt{n}\left({\widehat{\theta }}_{hi}^S-{\theta }_{hi}^{\mathrm{S}}\right)}{S_{Xhi}}\le Z_{\alpha /2}\right)\hfill \\ & =p\left({\widehat{\theta }}_{hi}^S-Z_{\alpha /2}\frac{S_{Xhi}}{\sqrt{n}}\le {\theta }_{hi}^{\mathrm{S}}\le {\widehat{\theta }}_{hi}^S+Z_{\alpha /2}\frac{S_{Xhi}}{\sqrt{n}}\right).\hfill \end{array}$$

(35)

Then, the $100\left(1-\alpha \right)\%$ confidence interval of ${\theta }_{bi}^S$ and ${\theta }_{ai}^S$ can be demonstrated as follows:

$$\mathrm{before} \mathrm{improvement}: \left[L{\theta }_{bi},U{\theta }_{bi}\right]=\left[{\widehat{\theta }}_{bi}^S-Z_{\alpha /2}\frac{S_{Xbi}}{\sqrt{n}},{\widehat{\theta }}_{bi}^S+Z_{\alpha /2}\frac{S_{Xbi}}{\sqrt{n}}\right];$$

(36)

$$\mathrm{after} \mathrm{improvement}: \left[L{\theta }_{ai},U{\theta }_{ai}\right]=\left[{\widehat{\theta }}_{ai}^S-Z_{\alpha /2}\frac{S_{Xai}}{\sqrt{n}},{\widehat{\theta }}_{ai}^S+Z_{\alpha /2}\frac{S_{Xai}}{\sqrt{n}}\right].$$

(37)

Obviously, when the value of ${\theta }_{ai}^S$ is bigger than that of ${\theta }_{bi}^S$, it indicates that the improvement is effective. When ${\theta }_{ai}^S$ is equal to ${\theta }_{bi}^S$, it means that the improvement has no effect. When the value of ${\theta }_{ai}^S$ is smaller than that of ${\theta }_{bi}^S$, improvement does not occur; in fact, it is made worse. Accordingly, the statistical hypothesis test for validating the effectiveness of improvement is displayed as follows:

$$\mathrm{null} \mathrm{hypothesis} H_0: {\theta }_{bi}^S={\theta }_{ai}^S;$$

(38)

$$\mathrm{alternative} \mathrm{hypothesis} H_1: {\theta }_{bi}^S\ne {\theta }_{ai}^S.$$

(39)

Based on Equations (36) and (37), we perform the two-tailed statistical test using two confidence intervals. The test rules are defined as follows:

(1)

If $U{\theta }_{bi}<L{\theta }_{ai}$, then the improvement of service item i works well.

(2)

If $\left[L{\theta }_{bi},U{\theta }_{bi}\right]\cap \left[L{\theta }_{ai},U{\theta }_{ai}\right]\ne \varphi$, then service item i does not improve significantly.

(3)

If $U{\theta }_{ai}<L{\theta }_{bi}$, then it means that improvement does not occur; in fact, it is made worse.

Based on the rules of the two-tailed statistical test, we can assist the system administrator of the teaching APP to verify the improvement effect upon all the service items that need to be improved and determine whether the improvements should continue so as to ensure the system quality of the teaching APP.

5. Case Study

As mentioned above, with the performance evaluation matrix, we can help the system administrator of the teaching APP quickly figure out the key service items that need to be improved and implement some improvements. In the field of English digital learning, many scholars have been involved in the research on the computer-assisted language learning system (CALL system) to help students use the CALL system to improve their foreign language [22,23,24,25,26]. The web-based e-learning system (WELS) satisfaction questionnaire designed by Shee and Wang [27], including four dimensions—1. Learner Interface, 2. Learning Community, 3. System Content, and 4. Personalization and Contains—and 13 questions can be said to be simple and complete. Therefore, it has been adopted by plenty of studies [28,29,30,31], for example, on the perceptions of students’ e-learning experiences in English for specific purpose (ESP) classes [28], examining and categorizing proper conditions for mathematics teaching as a real-life utilization with a multi-criteria decision analysis and fuzzy-decision-making trial and evaluation laboratory (MCDA and F-DEMATEL) method [29], and examining 98 articles to confirm the satisfaction as a critical aspect of student success in online education [30]. This paper also used this questionnaire to collect users’ satisfaction and importance and adopted the Call system as a case study to illustrate the model proposed in this study.

This questionnaire survey recruited students from a university in Taiwan as a sample. A total of 398 questionnaires were sent out, and 355 questionnaires were received (the response rate was 89.2%), of which 11 were invalid and 344 were effective (the effective response rate was 86.4%). The following is a data analysis based on the received 344 valid questionnaires (N = 344).

First, according to Equations (3) and (14), we calculated the estimator (${\widehat{\theta }}_i^S$) of the satisfaction index and the estimator (${\widehat{\theta }}_i^I$) of the importance index of each question, respectively. At the same time, according to Equation (5), we also calculated the average of the estimators of all satisfaction indices, ${\theta }_S=$ 0.52. Then, the hypotheses of the statistical test for satisfaction index i are displayed as follows:

$$\mathrm{null} \mathrm{hypothesis} H_0: {\theta }_i^S\ge 0.52;$$

(40)

$$\mathrm{alternative} \mathrm{hypothesis} H_1: {\theta }_i^S<0.52.$$

(41)

Subsequently, based on Equation (13) and ${\theta }_S=$0.52, we calculated the critical value ($C_{X0i}$) of the danger zone of the satisfaction index of each question and filled it in

Table 1. Satisfaction and Importance survey for CALL System.

Dimensions	Items	${\widehat{\mathsf{\theta }}}_{\mathit{i}}^{\mathit{S}}$	${\widehat{\mathsf{\theta }}}_{\mathit{i}}^{\mathit{I}}$	${\mathit{C}}_{\mathit{X}0\mathit{i}}$	${\mathit{C}}_{\mathit{Y}0\mathit{i}}$	Quadrant
Learner Interface	1. Ease of use	0.65	0.77	0.49	0.69	Q1
	2. User-friendliness	0.70	0.77	0.50	0.69	Q1
	3. Ease of understanding	0.68	0.76	0.50	0.69	Q1
	4. Operational stability	0.59	0.75	0.49	0.69	Q1
Learning Community	5. Ease of discussion with other learners	0.41	0.77	0.49	0.69	Q2 **
	6. Ease of discussion with teachers	0.39	0.77	0.49	0.69	Q2 **
	7. Ease of accessing shared data	0.37	0.63	0.49	0.68	Q3 *
	8. Ease of exchanging learning with the others	0.34	0.57	0.49	0.68	Q3 *
System Content	9. Up-to-date content	0.52	0.67	0.49	0.68	Q4
	10. Sufficient content	0.52	0.67	0.49	0.68	Q4
	11. Useful content	0.53	0.66	0.49	0.68	Q4
Personalization	12. Capability of controlling learning progress	0.51	0.65	0.49	0.68	Q4
	13. Capability of recording learning performance	0.51	0.67	0.49	0.68	Q1

Note: * indicates the need for improvement; ** indicates the need for the top priority of improvement.

.

Similarly, according to Equation (16), the average of estimators of all the importance indices, ${\theta }_I=$0.70, was calculated. The hypotheses of the statistical test for importance index i were defined as follows:

$$\mathrm{null} \mathrm{hypothesis} H_0: {\theta }_i^I\ge 0.70;$$

(42)

$$\mathrm{alternative} \mathrm{hypothesis} H_1: {\theta }_i^I<0.70.$$

(43)

Next, according to Equation (24) and ${\theta }_I=$0.70, we determined the critical value ($C_{Y0i}$) of the danger zone of the importance index of each question and filled it in Table 1.

According to Table 1, we used horizontal line $y=C_{Y0}$ and vertical line $x=C_{X0}$ to divide the performance evaluation matrix into four quadrants, where $C_{X0}=Min\left\{C_{X01},C_{X02},\dots ,C_{X0q}\right\}$ = 0.49 and $C_{Y0}=Min\left\{C_{Y01},C_{Y02},\dots ,C_{Y0q}\right\}$ = 0.68. Next, according to the evaluation coordinate points of $\left(x_i,y_i\right)=\left({\widehat{\theta }}_i^S,{\widehat{\theta }}_i^I\right)$ for all service items in Table 1, we plotted them in Figure 2, as shown below.

According to Figure 2, the evaluation coordinate points of these 13 service items fall into the four evaluation quadrants as follows:

Quadrant 1: items 1–4;

Quadrant 2: item 5, item 6;

Quadrant 3: item 7, item 8;

Quadrant 4: items 9–13.

According to the evaluation rules, service items 5–8 need to improve. Based on the improvement ranking rules, since evaluation coordinates $\left(x_5,y_5\right)\in Q_2$ and $\left(x_6,y_6\right)\in Q_2$, service item 5 and service item 6 were ranked to be of the first priority of improvement. Similarly, evaluation coordinates $\left(x_7,y_7\right)\in Q_3$ and $\left(x_8,y_8\right)\in Q_3$, then service item 7 and service item 8 were ranked to be of the second priority of improvement.

Next, managers would be able to refer to the recommendations of these rules to carry out improvement procedures. After the improvement concludes, a test of the improvement can be carried out. Based on Equations (38) and (39), the assumptions of its statistical test are as follows:

$$\mathrm{null} \mathrm{hypothesis} H_0: {\theta }_{bi}^S={\theta }_{ai}^S;$$

(44)

$$\mathrm{alternative} \mathrm{hypothesis} H_1: {\theta }_{bi}^S\ne {\theta }_{ai}^S.$$

(45)

Based on Equations (36) and (37), the 95% confidence intervals with sample size n = 258 of the satisfaction indices of the four service items that need to implement improvements are calculated.

• Service Item 5

  $$\mathrm{before} \mathrm{improvement}: \left[L{\theta }_{b5},U{\theta }_{b5}\right]=\left[0.41-1.96\frac{0.28}{\sqrt{344}},0.41+1.96\frac{0.28}{\sqrt{344}}\right]=[0.38,0.44];$$

  (46)

  $$\mathrm{after} \mathrm{improvement}: \left[L{\theta }_{a5},U{\theta }_{a5}\right]=\left[0.48-1.96\frac{0.27}{\sqrt{258}},0.48+1.96\frac{0.27}{\sqrt{258}}\right]=[0.45,0.51].$$

  (47)
• Service Item 6

  $$\mathrm{before} \mathrm{improvement}: \left[L{\theta }_{b6},U{\theta }_{b6}\right]=\left[0.39-1.96\frac{0.29}{\sqrt{344}},0.39+1.96\frac{0.29}{\sqrt{344}}\right]= [0.36,0.42];$$

  (48)

  $$\mathrm{after} \mathrm{improvement}: \left[L{\theta }_{a6},U{\theta }_{a6}\right]=\left[0.47-1.96\frac{0.28}{\sqrt{258}},0.47+1.96\frac{0.28}{\sqrt{258}}\right]= [0.43,0.50].$$

  (49)
• Service Item 7

  $$\mathrm{before} \mathrm{improvement}: \left[L{\theta }_{b7},U{\theta }_{b7}\right]=\left[0.37-1.96\frac{0.29}{\sqrt{344}},0.37+1.96\frac{0.29}{\sqrt{344}}\right]= [0.34,0.40];$$

  (50)

  $$\mathrm{after} \mathrm{improvement}: \left[L{\theta }_{a7},U{\theta }_{a7}\right]=\left[0.42-1.96\frac{0.27}{\sqrt{258}},0.42+1.96\frac{0.27}{\sqrt{258}}\right]= [0.39,0.46].$$

  (51)
• Service Item 8

  $$\mathrm{before} \mathrm{improvement}: \left[L{\theta }_{b8},U{\theta }_{b8}\right]=\left[0.34-1.96\frac{0.28}{\sqrt{344}},0.34+1.96\frac{0.28}{\sqrt{344}}\right]= [0.31,0.37];$$

  (52)

  $$\mathrm{after} \mathrm{improvement}: \left[L{\theta }_{a8},U{\theta }_{a8}\right]=\left[0.27-1.96\frac{0.24}{\sqrt{258}},0.27+1.96\frac{0.24}{\sqrt{258}}\right]= [0.24,0.30].$$

  (53)

In summary, the aforementioned test rules are listed as follows:

(1)

For service items 5 and 6, their upper limits before improvement are smaller than their lower limits after improvement ($U{\theta }_{b5}=0.44<L{\theta }_{a5}=0.45$; $U{\theta }_{b6}=0.42<L{\theta }_{a6}=0.43$), indicating that the improvements of service items 5 and 6 are effective.

(2)

For service item 7, $\left[L{\theta }_{b7},U{\theta }_{b7}\right]=\left[0.34,0.40\right]$ and $\left[L{\theta }_{a7},U{\theta }_{a7}\right]=\left[0.39,0.46\right]$, then $\left[L{\theta }_{b7},U{\theta }_{b7}\right]\cap \left[L{\theta }_{a7},U{\theta }_{a7}\right]\ne \varphi$. This shows that the improvement effect of service item 7 is not significant, so continuous improvement is necessary.

(3)

For service item 8, $U{\theta }_{a8}=0.30<L{\theta }_{b8}=0.31$; it means that improvement does not occur; in fact, it is made worse. It also indicates that the improvement strategy needs to be re-examined so as to make effective improvements.

6. Conclusions

According to the viewpoints of Hung, Huang, and Chen [14], we proposed standardized and significant indices of importance and satisfaction on the premise that satisfaction and importance were assigned by Beta in this paper, and the values of these two indices were both between 0 and 1. Next, we set the satisfaction index as x-axis and the importance index as y-axis to form a performance evaluation matrix. In order not to lose generality, we assumed that the digital learning service items provided by the teaching APP in this paper had a total of q service items, so there were q satisfaction indices and q importance indices in total. First, in this paper, we determined the unbiased estimators of the satisfaction index and the importance index for all service items since the indices had unknown parameters. Second, we used the average value ${\theta }_0^S$ of all satisfaction indices as the testing standard in accordance with the spirit of continuous improvement on the comprehensive product management as well as adopted the statistical one-tailed test to determine the critical value $C_0^S$. Third, we established testing rules to facilitate the management to procure the service items that need to be improved. At the same time, the average value ${\theta }_0^I$ of all the importance indices was adopted as the testing standard, and the ranking of all service items that need to improve was determined by the statistical one-tailed test. Next, the above two statistical testing rules were applied as the basis of defining the PEM evaluation blocks and establishing the PEM evaluation rules. Obviously, since the method in this paper was established via statistical tests, the risk of wrong judgment caused by sampling error can be diminished. After the task of improvement was completed, we employed the statistical two-tailed test to verify whether the satisfaction of all key service items that need to make improvements was leveled up. Finally, a case study of the computer-assisted language learning system (CALL System) was used to explain the application of the abovementioned statistical testing rules.

As noted above, most studies just identify critical qualities (CTQs) that need improvement, without discussing processing rules in resource-limited settings, and without confirmation of improvement. The method of this paper just makes up for the gap in similar research. This paper adopts the one-tailed statistical hypothesis testing method of satisfaction index, which can help the industry to detect the CTQ service items that need to be improved and determine the order of improvement with a low risk of misjudgment. Finally, the two-tailed statistical hypothesis testing method is used to assist the industry to confirm its improvement effect. In addition, the digital learning effect evaluation and improvement model of the teaching APP constructed in this paper will contribute to attracting more users to apply the teaching APP to achieve the effectiveness of digital learning. The APP enables more students to avoid the inconvenience of going out and obtain good learning outcomes at home. Not only can it alleviate the problem of traffic congestion, but it can also achieve the effect of energy saving and carbon reduction by decreasing the need of going out.

Clearly, this paper refers to the literature, closely examines the needs of the customers, and assumes that the distribution of customer satisfaction and importance lies with Beta distribution since Beta distribution belongs to continuous probability distribution; however, data would have to be collected from the highest score of 100 and the lowest score of 0 and then divided by 100 to produce a value between 0 and 1. In addition, the statistical test must be conducted by the Central Limit Theorem. These are the limitations of this study. Therefore, future research can focus on the data collection method of the Beta distribution, and explore the fitting quantity of sample size in the application of the Central Limit Theorem.