# Reasoning about Confidence in Goal Satisfaction

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Background

`C[x]`and

`DQ[x]`labels are not part of the standard language but are new types of labels proposed in this paper.

- AND-Decomposition: the minimum satisfaction value of sub-goals is propagated (see Figure 3), as expressed in Equation (1);$$v\left(S\right)=Min(v\left({D}_{1}\right),v\left({D}_{2}\right),...,v\left({D}_{n}\right)),$$
- OR-Decomposition: using the same notation conventions, here the maximum satisfaction value among the decomposing elements is propagated, as shown in Equation (2) and Figure 4. GRL’s XOR-decomposition is computed in the same way, with a warning if more than one decomposing element has a satisfaction value different from 0.$$v\left(S\right)=Max(v\left({D}_{1}\right),v\left({D}_{2}\right),...,v\left({D}_{n}\right)),$$
- Contribution: in GRL, contributions can have a type/weight that indicates whether it is negative, neural (0), or positive, with a weight between 0 and 100. Hence, the contribution scale goes from $-100$ to $+100$. Here, the product of the weighted contribution values (${C}_{x}$ and the satisfaction values ($v\left({D}_{x}\right)$) of the n contributing elements are propagated to the targeted GRL element, as illustrated in Figure 5 and Equation (3). Note that the final satisfaction is also truncated to a value between 0 and 100, inclusively:$$v\left(S\right)=Max(0,Min(100,\frac{{\sum}_{x=1}^{n}(v\left({D}_{x}\right)\times {C}_{x})}{100}))$$
- Dependency: the satisfaction of the dependent element ($v\left(S\right)$) is truncated to the minimum satisfaction value of its depending elements (see Figure 6, where $v\left(S\right)\ge 25$ initially, and Equation (4)). This means that the GRL element that is the source of dependencies cannot be better satisfied than the other elements it depends on.$$v\left(S\right)=Min(v\left(S\right),v\left({D}_{1}\right),v\left({D}_{2}\right),...,v\left({D}_{n}\right))$$

## 3. Related Work

## 4. Data Quality Tagging and Propagation Mechanism

#### 4.1. Data Quality Tagging

#### 4.2. Propagation Method

`CalculateEvaluation`algorithm [4]. Confidence is computed in an integrated manner from the decomposition, contribution, and dependency relationships, in that order. It is important to note that confidence can be computed independently from the satisfaction values of goals; for example, a goal could have a high satisfaction with a low confidence or a low satisfaction with a high confidence.

**AND-decomposition:**the confidence of the parent goal satisfaction is equal to the average of its sub-goals’ confidence values. Unlike satisfaction propagation (where the minimum satisfaction is propagated to the parent goal), the average is used for confidence because:

- The confidence of the sub-goal with minimum satisfaction value might be different from the minimum confidence among all sub-goals of the AND-decomposition.
- The confidence of the sub-goal with minimum satisfaction value might be much higher or lower than the confidence of the other sub-goals. Yet, all sub-goals are taken into consideration during the propagation decision.

`DQ`) and confidence (

`C`) values are highlighted in red in the figure. Note that for a model element initially tagged with

`DQ[x]`, the confidence is also that value (

`C[x] = DQ[x]`).

**XOR/OR-decomposition:**the confidence of a parent goal satisfaction value is equal to the confidence value of its maximally satisfied sub-goals. In case there are more than one sub-goal sharing the maximum satisfaction, the parent’s confidence level becomes the average confidence of the maximally satisfied sub-goals for the OR-decomposition (see Figure 8 and Equation (6)). For the XOR-decomposition, the maximum confidence among the maximally satisfied sub-goals is selected (as $i=1$ in the equation). As the OR-decomposition is about selecting one or many alternatives and the XOR-decomposition only one alternative, only the confidence levels of the selected alternatives are considered by the confidence propagation mechanism. This is different from the AND-decomposition, where the average of all confidence values was considered, because the XOR/OR-decomposition is an optimistic decomposition operator by nature, and is concerned with selecting some options while ignoring others.

**Contribution:**the confidence of the satisfaction of a goal that is the target of contributions is the sum of the product of the contribution values of sub-goals and their confidence values, over 100 (and truncated to an integer value between 0 and 100), as formalized in Equation (7). ${C}_{x}$ is a contribution level between $-100$ and 100. In the example of Figure 9, $(50\times 75+75\times 25)/100=56$. If the sum of the contribution weights is larger than 100 (overcontribution), in order to avoid ‘confidence building’, the computed confidence level is normalized. This is a mechanism similar to the propagation of satisfaction values in GRL, where all the contributions and their weights are considered and truncated whenever necessary.

**Dependency:**the confidence of a dependent element is its current confidence level (if any) when the depending elements all have higher (or equal) satisfaction levels than that dependent element’s satisfaction (see Equation (8)). However, if some depending elements have lower satisfaction levels, then the confidence is computed as the minimum between the current confidence level (if any) and the confidence levels of the depending elements with the lowest satisfaction level (see Figure 10). This is a conservative propagation of confidence, and this choice is again dictated by the nature of the dependency link in GRL, which aims to identify locations for conservative evaluations in GRL models.

**Alternative Selection:**in a goal model, there could be several design alternatives that could be used in combination to evaluate the satisfaction of a goal. The confidence propagated to top-level goals and other elements reflects only the confidence of the chosen design alternative where the other alternatives’ impact is considered to be absent if no confidence value is assigned to them. For the non-selected alternatives, an A letter (meaning: absent) is propagated by default to indicate that there could be some impact of other alternatives, which are linked to the top-level model elements, on the propagated confidence levels, but that impact is not considered in a particular evaluation strategy (see Figure 11).

## 5. Implementation

`DQ[x]`). The algorithm for propagating confidence values (CalculateEvaluationAndConfidence) extends the CalculateEvaluation algorithm of standard GRL [4]. The algorithm, and the non-primitive data types are classes from the URN metamodel. The CalculateEvaluationAndConfidence algorithm generates a new confidence value (between 0 and 100) that can then be also stored as a metadata for the intentional element being evaluated. The algorithm invokes three sub-algorithms (CalculateDecompositions, CalculateContributions, and CalculateDependencies), which are also modified. Once CalculateEvaluationAndConfidence has completed, the ActorSatisfaction algorithm (also modified) can be invoked to compute the satisfaction and confidence of an actor, and then the confidence can be stored again as metadata attached to that actor. The details of the algorithm can be found in Section 4.4 of Baslyman’s thesis (https://ruor.uottawa.ca/bitstream/10393/38104/3/Baslyman_Malak_2018_thesis.pdf, accessed on 2 June 2022) and the corresponding Java module for jUCMNav is also available online (https://www.site.uottawa.ca/~damyot/pub/Baslyman/QuantitativeGRLStrategyAlgorithm.java, accessed on 2 June 2022).

## 6. Case Study

- having patient information entered into the RTTS by a nurse, or
- having the RTTS pull patient information automatically from the MHR.

#### 6.1. Application of Data Quality Tagging and Propagation Mechanism

#### 6.2. Confidence Propagation Interpretation

## 7. Discussion

- One major limitation is that data quality types (Section 4.1) could be improved based on other dimensions. The quality types presented in this paper are generic and function well, at least in the context of this paper and its case study. However, more precise types may be needed in other contexts. For example, the classification of data gathered through sensors in real time is not yet supported in the proposed approach.
- It is also worth mentioning that assigning quality types to data is not trivial. Currently, this is done based on an assessment of analysts and stakeholders involved in the context where, probably, many disagreements and conflicts arise. Therefore, it is important to systematize and more formally describe the process of assigning quality types to data.
- The functions used to propagate confidence levels (Section 4.2), although they did not generate complaints from the case study participants, are currently only justified through arguments. They could be further validated empirically, especially against alternative propagation functions.
- The illustrative example may not reflect the complexity of other real-world cases and contexts, especially outside the healthcare domain. Additional cases studies in other domains would help raise our confidence in the suitability and generalizability of the mechanism proposed here. More formal experiments could also provide more reliable empirical evidence, for instance by comparing the outputs of the uncertainty reasoning proposal with those of domain experts, or by studying the scalability and usability of the reasoning as goal models get larger.
- The approach was currently implemented for GRL models. Although we do not see major issues in porting it to other goal-oriented modeling languages, whether specific semantics of these languages will require major adaptations to some of the confidence propagation functions remains a research topic.
- As the creators of the Data Quality Tagging and Propagation Mechanism also led the development and analysis of the case study, real or perceived biases represent another potential threat to the internal validity of our work. One potential mitigation would be to have people other than us lead experiments and case studies about the usefulness of the approach.

## 8. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## Abbreviations

AHP | Analytical Hierarchy Process |

BBN | Bayesian Belief Networks |

ER | Emergency Room |

GA | Genetic Algorithm |

GRL | Goal-oriented Requirement Language |

KPI | Key Performance Indicator |

MHR | Medical Health Record |

RTTS | Real-Time Tracking Sample system |

SAR | Saudi Riyals |

URN | User Requirements Notation |

## References

- Letier, E.; Stefan, D.; Barr, E.T. Uncertainty, Risk, and Information Value in Software Requirements and Architecture. In Proceedings of the 36th International Conference on Software Engineering (ICSE), Hyderabad, India, 31 May–7 June 2014; ACM: New York, NY, USA, 2014; pp. 883–894. [Google Scholar] [CrossRef]
- Frank, U. Domain-Specific Modeling Languages: Requirements Analysis and Design Guidelines. In Domain Engineering: Product Lines, Languages, and Conceptual Models; Reinhartz-Berger, I., Sturm, A., Clark, T., Cohen, S., Bettin, J., Eds.; Springer: Berlin/Heidelberg, Germany, 2013; pp. 133–157. [Google Scholar] [CrossRef]
- Mussbacher, G.; Amyot, D. Goal and scenario modeling, analysis, and transformation with jUCMNav. In Proceedings of the 31st International Conference on Software Engineering—Companion Volume, Vancouver, BC, Canada, 16–24 May 2009; pp. 431–432. [Google Scholar] [CrossRef]
- ITU-T. Recommendation Z.151 (10/18) User Requirements Notation (URN)—Language definition. 2018. Available online: https://www.itu.int/rec/T-REC-Z.151/en (accessed on 2 June 2022).
- Fan, Y.; Anda, A.A.; Amyot, D. An Arithmetic Semantics for GRL Goal Models with Function Generation. In Proceedings of the International Conference on System Analysis and Modeling, Copenhagen, Denmark, 15–16 October 2018; Springer: Berlin/Heidelberg, Germany, 2018; pp. 144–162. [Google Scholar] [CrossRef]
- Busari, S.A. Modelling and Analysing Software Requirements and Architecture Decisions under Uncertainty. Ph.D. Thesis, UCL (University College London), London, UK, 2019. Available online: https://discovery.ucl.ac.uk/id/eprint/10067421 (accessed on 2 June 2022).
- Busari, S.A.; Letier, E. Radar: A lightweight tool for requirements and architecture decision analysis. In Proceedings of the 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), Buenos Aires, Argentina, 20–28 May 2017; pp. 552–562. [Google Scholar] [CrossRef]
- Cailliau, A.; van Lamsweerde, A. Handling knowledge uncertainty in risk-based requirements engineering. In Proceedings of the IEEE 23rd International Requirements Engineering Conference (RE), Ottawa, ON, Canada, 24–28 August 2015; pp. 106–115. [Google Scholar] [CrossRef]
- Sabetzadeh, M.; Falessi, D.; Briand, L.; Alesio, S.D.; McGeorge, D.; Åhjem, V.; Borg, J. Combining Goal Models, Expert Elicitation, and Probabilistic Simulation for Qualification of New Technology. In Proceedings of the IEEE 13th International Symposium on High-Assurance Systems Engineering, Boca Raton, FL, USA, 10–12 November 2011; pp. 63–72. [Google Scholar] [CrossRef]
- Chen, B.; Peng, X.; Yu, Y.; Zhao, W. Uncertainty handling in goal-driven self-optimization–limiting the negative effect on adaptation. J. Syst. Softw.
**2014**, 90, 114–127. [Google Scholar] [CrossRef] - Liaskos, S.; Khan, S.M.; Mylopoulos, J. Modeling and reasoning about uncertainty in goal models: A decision-theoretic approach. Softw. Syst. Model.
**2022**, 1–24. [Google Scholar] [CrossRef] - Salay, R.; Famelis, M.; Chechik, M. Language independent refinement using partial modeling. In Proceedings of the 15th International Conference on Fundamental Approaches to Software Engineering, Tallinn, Estonia, 24 March–1 April 2012; Springer: Berlin/Heidelberg, Germany, 2012; pp. 224–239. [Google Scholar] [CrossRef] [Green Version]
- Bowers, K.M.; Fredericks, E.M.; Cheng, B.H. Automated Optimization of Weighted Non-functional Objectives in Self-adaptive Systems. In Proceedings of the 10th International Symposium on Search Based Software Engineering, Montpellier, France, 8–9 September 2018; Springer: Berlin/Heidelberg, Germany, 2018; pp. 182–197. [Google Scholar] [CrossRef]
- Fredericks, E.M.; DeVries, B.; Cheng, B.H. AutoRELAX: Automatically RELAXing a goal model to address uncertainty. Empir. Softw. Eng.
**2014**, 19, 1466–1501. [Google Scholar] [CrossRef] - Zawawy, H.; Mankovskii, S.; Kontogiannis, K.; Mylopoulos, J. Mining software logs for goal-driven root cause analysis. In The Art and Science of Analyzing Software Data; Elsevier: Amsterdam, The Netherlands, 2015; pp. 519–554. [Google Scholar] [CrossRef]
- Dąbrowski, J. Towards an adaptive framework for goal-oriented strategic decision-making. In Proceedings of the 2017 IEEE 25th International Requirements Engineering Conference (RE), Lisbon, Portugal, 4–8 September 2017; pp. 538–543. [Google Scholar] [CrossRef]
- Pasquale, L.; Spoletini, P.; Salehie, M.; Cavallaro, L.; Nuseibeh, B. Automating trade-off analysis of security requirements. Requir. Eng.
**2016**, 21, 481–504. [Google Scholar] [CrossRef] - Hobbs, C.; Lloyd, M. The Application of Bayesian Belief Networks to Assurance Case Preparation. In Achieving Systems Safety, Proceedings of the Twentieth Safety-Critical Systems Symposium, Bristol, UK, 7–9 February 2012; Dale, C., Anderson, T., Eds.; Springer: London, UK, 2012; pp. 159–176. [Google Scholar] [CrossRef]
- Guiochet, J.; Do Hoang, Q.A.; Kaaniche, M. A Model for Safety Case Confidence Assessment. In Proceedings of the 34th International Conference, Computer Safety, Reliability, and Security, Delft, The Netherlands, 23–25 September 2015; Springer: Cham, Switzerland, 2015; pp. 313–327. [Google Scholar] [CrossRef]
- Hall, R.J. Forensic System Verification. In Proceedings of the 17th IEEE International Requirements Engineering Conference, Atlanta, GA, USA, 31 August–4 September 2009; pp. 111–120. [Google Scholar] [CrossRef]
- Dhaouadi, M.; Spencer, K.; Varnum, M.H.; Grubb, A.M.; Famelis, M. Towards a generic method for articulating design uncertainty. J. Object Technol.
**2021**, 12, 3. [Google Scholar] [CrossRef] - Bencomo, N.; Götz, S.; Song, H. Models@run.time: A guided tour of the state of the art and research challenges. Softw. Syst. Model.
**2019**, 18, 3049–3082. [Google Scholar] [CrossRef] - Alwidian, S.; Dhaouadi, M.; Famelis, M. A Vision towards a Conceptual Basis for the Systematic Treatment of Uncertainty in Goal Modelling. In Proceedings of the 12th System Analysis and Modelling Conference, Montreal, QC, Canada, 19–20 October 2020; Association for Computing Machinery: New York, NY, USA, 2020; pp. 139–142. [Google Scholar] [CrossRef]
- Velasquez, M.; Hester, P.T. An analysis of multi-criteria decision making methods. Int. J. Oper. Res.
**2013**, 10, 56–66. Available online: http://www.orstw.org.tw/ijor/vol10no2/ijor_vol10_no2_p56_p66.pdf (accessed on 2 June 2022). - Lilja, K.K.; Laakso, K.; Palomäki, J. Using the Delphi method. In Proceedings of the 2011 Proceedings of PICMET’11: Technology Management in the Energy Smart World (PICMET), Portland, OR, USA, 31 July–4 August 2011; pp. 1–10. Available online: https://ieeexplore.ieee.org/document/6017716 (accessed on 2 June 2022).
- Akhigbe, O.; Alhaj, M.; Amyot, D.; Badreddin, O.; Braun, E.; Cartwright, N.; Richards, G.; Mussbacher, G. Creating quantitative goal models: Governmental experience. In Proceedings of the International Conference on Conceptual Modeling, Atlanta, GA, USA, 27–29 October 2014; Springer: Berlin/Heidelberg, Germany, 2014; pp. 466–473. [Google Scholar] [CrossRef]
- Liaskos, S.; Jalman, R.; Aranda, J. On eliciting contribution measures in goal models. In Proceedings of the 2012 20th IEEE International Requirements Engineering Conference (RE), Chicago, IL, USA, 24–28 September 2012; pp. 221–230. [Google Scholar] [CrossRef]
- Vinay, S.; Aithal, S.; Sudhakara, G. Effect of Contribution Links on Choosing Hard Goals in GORE Using AHP and TOPSIS. In Emerging Research in Electronics, Computer Science and Technology: Proceedings of International Conference, ICERECT 2012; Springer: New Delhi, India, 2014; pp. 737–745. [Google Scholar] [CrossRef]
- Kassab, M. An integrated approach of AHP and NFRs framework. In Proceedings of the IEEE 7th International Conference on Research Challenges in Information Science (RCIS), Paris, France, 29–31 May 2013; pp. 1–8. [Google Scholar] [CrossRef]
- Zhao, T.; Zhao, H.; Zhang, W.; Jin, Z. Goal model driven alternative selection: A quantitative approach. In Proceedings of the 2015 IEEE International Model-Driven Requirements Engineering Workshop (MoDRE), Ottawa, ON, Canada, 24 August 2015; pp. 1–10. [Google Scholar] [CrossRef]
- Sumesh, S.; Krishna, A. Hybrid analytic hierarchy process-based quantitative satisfaction propagation in goal-oriented requirements engineering through sensitivity analysis. Multiagent Grid Syst.
**2020**, 16, 433–462. [Google Scholar] [CrossRef] - Baslyman, M.; Almoaber, B.; Amyot, D.; Bouattane, E.M. Activity-based Process Integration in Healthcare with the User Requirements Notation. In Proceedings of the 7th International Conference on E-Technologies, Ottawa, ON, Canada, 17–19 May 2017; Springer: Berlin/Heidelberg, Germany, 2017; pp. 151–169. [Google Scholar] [CrossRef]
- Gonçalves, E.; Araujo, J.; Castro, J. PRISE: A process to support iStar extensions. J. Syst. Softw.
**2020**, 168, 110649. [Google Scholar] [CrossRef] - Singh, H.; Khalajzadeh, H.; Paktinat, S.; Graetsch, U.M.; Grundy, J. Modelling human-centric aspects of end-users with iStar. J. Comput. Lang.
**2022**, 68, 101091. [Google Scholar] [CrossRef] - Runeson, P.; Höst, M. Guidelines for conducting and reporting case study research in software engineering. Empir. Softw. Eng.
**2009**, 14, 131–164. [Google Scholar] [CrossRef] [Green Version]

**Figure 8.**Confidence of element satisfactions in the case of an OR-decomposition. (

**Left**): confidence of maximally satisfied sub-goal propagated to the parent goal. (

**Right**): average confidence of maximally satisfied sub-goals propagated to the parent goal.

**Figure 11.**Propagation of “

`A`” labels with confidence values, which are used to highlight the absent impact of non-selected alternatives (e.g.,

`Option Two here`).

**Figure 13.**GRL model evaluation corresponding to the RTTS by nurse alternative in a critical situation.

**Table 1.**Data quality types and corresponding confidence values, in a decision-making context related to design alternatives.

Quality Type | Confidence Value | Definition |
---|---|---|

Valid | 100 | Data already measured and available for the design alternative, in the same context as the one under evaluation |

Borrowed-Valid | 100 | Data already measured and available for the design alternative, in a context similar to the one under evaluation |

Borrowed | 75 | Data already measured and available for the design alternative, but in a different context |

Estimated-Context | 50 | No data available, but it was estimated according to a similar design alternatives in a different context |

Estimated-Literature | 25 | No data available, but it was estimated based on the literature or previous studies |

Unknown | 0 | No data-driven evidence will be used in the evaluation |

Indicator | Definition | Unit | Data Quality |
---|---|---|---|

Total yearly cost | Summation of four other indicators: software installation cost, software acquisition cost, software maintenance cost, and the hardware cost per sample | SAR | Estimated-Literature 25 |

Installation cost | Software installation cost | SAR | Estimated-Literature 25 |

Acquisition cost | Software acquisition cost | SAR | Estimated-Literature 25 |

Maintenance and operation cost | Software maintenance cost | SAR | Estimated-Literature 25 |

Hardware cost per sample | Cost of the tracking chip | SAR | Estimated-Literature 25 |

Number of interactions with patients per instance | Number of interactions between a nurse and a patient inquiring about the lab sample result | Number | Valid 100 |

Number of duplicated tasks per instance | Number of duplicated tasks per instance in an alternative | Number | Valid 100 |

Time spent per instance | Time from collecting the lab sample to its delivery to the lab unit | Second | Valid 100 |

**Table 3.**Propagated confidence levels of top-level goals in the critical context.

**Sat**is the satisfaction value and

**Conf**is the propagated confidence value.

By Nurse | Automated RTTS | Current Method | ||||
---|---|---|---|---|---|---|

Goals | Sat | Conf | Sat | Conf | Sat | Conf |

Increase process efficiency | 70 | 66 | 76 | 66 | 23 | 30 |

Reduce turn around time | 84 | 94 | 84 | 94 | 59 | 20 |

Identify process break points | 56 | 56 | 56 | 56 | 7 | 8 |

Reduce risk | 100 | 57 | 100 | 57 | 7 | 8 |

Monitor collecting samples till delivering results | 75 | 75 | 75 | 75 | 10 | 10 |

Have low cost | 52 | 20 | 46 | 20 | 100 | 20 |

Get the lab results within the allowed timeframe | 30 | 30 | 30 | 30 | 0 | 0 |

Reduce number of interactions with patients | 12 | 25 | 12 | 25 | 0 | 25 |

Reduce number of duplicated tasks | 30 | 75 | 75 | 75 | 22 | 75 |

Stay updated about the sample status in real time | 75 | 75 | 75 | 75 | 0 | 100 |

Stay informed/updated about the arrival of samples in real time | 100 | 100 | 100 | 100 | 0 | 100 |

Track sample position in real time | 100 | 100 | 100 | 100 | 0 | 0 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Baslyman, M.; Amyot, D.; Mylopoulos, J.
Reasoning about Confidence in Goal Satisfaction. *Algorithms* **2022**, *15*, 343.
https://doi.org/10.3390/a15100343

**AMA Style**

Baslyman M, Amyot D, Mylopoulos J.
Reasoning about Confidence in Goal Satisfaction. *Algorithms*. 2022; 15(10):343.
https://doi.org/10.3390/a15100343

**Chicago/Turabian Style**

Baslyman, Malak, Daniel Amyot, and John Mylopoulos.
2022. "Reasoning about Confidence in Goal Satisfaction" *Algorithms* 15, no. 10: 343.
https://doi.org/10.3390/a15100343