A Review to Find Elicitation Methods for Business Process Automation Software

Round 1

Reviewer 1 Report

Abstract: fine but add a hint at what the challenges/opportunities might be.

Introduction: excellent introduction and description of a business process. The chart on investment into BPA is good.

RQ3: How can we elicit… or What is the most appropriate way to elicit requirements…

Really good RQs overall.

Section 2

P2 l63. I don’t agree a business process is only a digital process. What about warehouse processes that require physical, manual work to be done such as loading items from a delivery truck into the warehouse onto shelves? It isn’t possible to automate every single step of a business process such as fulfilment, supply chain, value chain and so on. Humans are still required to provide necessary triggers to maintain the flow of data and information (and sometimes product or component).

A comment on this is employers in my industry are beginning to replace admin staff with “help” screens. The effect is efficiency of cost for the employer but disengages people who need human-to-human contact (everyone, then). Also, lots of people lose their jobs even though they are very valuable employees to those who need to communicate with them. My experience is that industry is only interested in cutting costs and the best way to do this is reduce staff numbers. There is a point at which such digitization of business processes fails because not enough people are there solve the inevitable problems that arise. Such as, as you indicate, having to change how your business operates to suit the inflexibility and myopia of the software (hyperautomation cannot manage the outliers called the vagaries of everyday life that people find a way to deal with—it simply does not compute). What do you think?

2.1.2. last sentence page 3. Very important point. Thank you for seeing this. Whole of 2.1.2 is an excellent section.

2.2 l135 Requirements engineering is not a science. It isn’t even engineering. It is perhaps a social science in the form of communication and understanding (a bit like psychology). Definitely not a science and no requirements engineer (researcher or practitioner-business analyst) would call it a science with a straight face. It has no theories, no proofs, no repeatability that a guaranteed outcome will always appear and the same outcome if performed by multiple people at the same time. RE contains a very flexible sort-of process, has techniques for finding out information associated with it, and hopes for some form of documentation of that information. It is much more an art, in all honesty, that relies heavily upon the insight and ability of the person conducting the work, rather than the mechanical, robotic use of the techniques themselves.

2.2.2. I wonder about whether your definition of requirements specification is correct. There’s plenty to suggest otherwise. Is specification really necessary? Would XP want to have a formalised specification document? Should specification include business goals? I don’t think so. A specification should be a description of the external behaviour of the software-system. You reference Michael Jackson’s Problem Frames; Jackson would not view goals as specification. The notion of your 4 phases is also a little bit too much compartmentalisation. Project management ensures throughout the project and cannot begin without any idea of the requirements. So your requirements specification would commence in parallel or before project management. I look at most organisations today and they are deploying a hybrid / agile approach to development (your SDC). Not many stick rigidly to the process implied by you. Perhaps those whose product is mostly physical with a little bit of software e.g. a white goods manufacturer.

2.2.3. Elicitation vs gathering is over-picky. Basically they are the same thing. If you interview someone to work out their requirements, you are writing down what they say because you are eliciting them. Perhaps you are referring to when you are handed a list of requirements and told to implement these as they are?

Section 3. Excellent description of the systematic review process. Well done in depicting the criteria, scope and outcomes. Why were 69% of relevant studies are not found in your 3 key online catelogues? You may not be able to answer this beyond, they just were ? The key thing here is that you looked.

Fig. 4. A very long list! My personal view is that some of the list are not elicitation techniques but modelling tools such as i* and KAOS, use cases also. Scenarios are ways to describe needs, user needs and requirements. Are they really techniques for elicitation which I would list as: interviews, surveys, observation (ethnography), workshops, focus groups, document analysis, review of existing product, role play. I do find it curious misunderstanding in the research community between a modelling technique and elicitation. Perhaps showing a picture to a stakeholder can draw from them a new requirement? I could show a picture of a car and ask the stakeholder, what other requirement would you want from that car? Well, this is document analysis, I suppose. I could go on—problem frames are a brilliant way to modularise a system to be built into a context. You could use the diagram to elicit a requirement. The diagram is a tool but problem frames has no unique approach to eliciting requirements. Stakeholders look at a model (and assuming they understand it) might think about what they want to be able to do. I find problem frames are entirely the opposite really: they are a way to organise requirements into a context to help separate concerns in order to make development easier through modularisation. I would review this figure and examine what is the actual elicitation (so-called) occurring here. Do you need the model to extract the requirement? OK, this is document analysis. Is there something within each technique that says follow steps 1, 2, 3 to discover information from a stakeholder in order to create the model? Your table 3 is claiming this but I don’t think all of the approaches are really elicitation tools. Ref 105 is an example of what I mean (I conducted that particular study in an industrial context (I was working in a financial company whilst doing this study but its goal was to identify where IT could support processes rather than find automation requirements). I understand, of course, you are searching on key words rather than conducting a detailed analysis each technique / tool / approach listed.

4. Discussion. I like the Limitations. I wonder how old some of the studies are that involve elicitation of requirements with business processes? I know 105 was conducted in 2001. At that time, the main concern was the regrowth of the .dot com world after the bubble had burst. Perhaps automation was not on the visible horizon? Now, though, maybe there is a slow waking up to this fact. The key part of your excellent paper is figure 1: the money invested in business process automation is very large. There should be scope, therefore, to study this further.

Does process mining imply a degree of automation needs to be in place already in order to conduct the mining?

Well done—an excellent systematic review.

Author Response

I'd like to thank you for your time and feedback. They opened my mind to new future works. I really appreciate them.

POINT1: Abstract: fine but add a hint at what the challenges/opportunities might be.

RESPONSE: I will add the following text: The challenges of developing a business process automation software include dealing with business processes, meeting the needs of the organization, choosing the right approach, and adapting to changes in the process during the development. These challenges open opportunities for proposing specific approaches to elicit requirements in this context. 

 

POINT2: RQ3: How can we elicit… or What is the most appropriate way to elicit requirements…

RESPONSE: I will change to: What is the most appropriate way to elicit requirements for business process automation software?

 

POINT3: P2 l63. I don’t agree a business process is only a digital process. What about warehouse processes that require physical, manual work to be done such as loading items from a delivery truck into the warehouse onto shelves? It isn’t possible to automate every single step of a business process such as fulfilment, supply chain, value chain and so on. Humans are still required to provide necessary triggers to maintain the flow of data and information (and sometimes product or component).

RESPONSE: I agree with you and I will remove the following text: In other words, a business process is an organizational process.

POINT4: 2.2 l135 Requirements engineering is not a science. It isn’t even engineering. It is perhaps a social science in the form of communication and understanding (a bit like psychology). Definitely not a science and no requirements engineer (researcher or practitioner-business analyst) would call it a science with a straight face. It has no theories, no proofs, no repeatability that a guaranteed outcome will always appear and the same outcome if performed by multiple people at the same time. RE contains a very flexible sort-of process, has techniques for finding out information associated with it, and hopes for some form of documentation of that information. It is much more an art, in all honesty, that relies heavily upon the insight and ability of the person conducting the work, rather than the mechanical, robotic use of the techniques themselves.

RESPONSE: I will change to: Requirements engineering is a discipline

POINT5: 2.2.2. I wonder about whether your definition of requirements specification is correct. There’s plenty to suggest otherwise. Is specification really necessary? Would XP want to have a formalised specification document? Should specification include business goals? I don’t think so. A specification should be a description of the external behaviour of the software-system. You reference Michael Jackson’s Problem Frames; Jackson would not view goals as specification. The notion of your 4 phases is also a little bit too much compartmentalisation. Project management ensures throughout the project and cannot begin without any idea of the requirements. So your requirements specification would commence in parallel or before project management. I look at most organisations today and they are deploying a hybrid / agile approach to development (your SDC). Not many stick rigidly to the process implied by you. Perhaps those whose product is mostly physical with a little bit of software e.g. a white goods manufacturer.

RESPONSE: I verified that in English the Requeriment Specification is a document and not a phase. To avoid the misunderstanding I will change the concept about Requirement Spec and introduce the PLANNING PHASE.

POINT6: 2.2.3. Elicitation vs gathering is over-picky. Basically they are the same thing. If you interview someone to work out their requirements, you are writing down what they say because you are eliciting them. Perhaps you are referring to when you are handed a list of requirements and told to implement these as they are?

RESPONSE: I agree with you and I will remove the text that refers to your point.

POINT7: Fig. 4. A very long list! My personal view is that some of the list are not elicitation techniques but modelling tools such as i* and KAOS, use cases also. Scenarios are ways to describe needs, user needs and requirements. Are they really techniques for elicitation which I would list as: interviews, surveys, observation (ethnography), workshops, focus groups, document analysis, review of existing product, role play. I do find it curious misunderstanding in the research community between a modelling technique and elicitation. Perhaps showing a picture to a stakeholder can draw from them a new requirement? I could show a picture of a car and ask the stakeholder, what other requirement would you want from that car? Well, this is document analysis, I suppose. I could go on—problem frames are a brilliant way to modularise a system to be built into a context. You could use the diagram to elicit a requirement. The diagram is a tool but problem frames has no unique approach to eliciting requirements. Stakeholders look at a model (and assuming they understand it) might think about what they want to be able to do. I find problem frames are entirely the opposite really: they are a way to organise requirements into a context to help separate concerns in order to make development easier through modularisation. I would review this figure and examine what is the actual elicitation (so-called) occurring here. Do you need the model to extract the requirement? OK, this is document analysis. Is there something within each technique that says follow steps 1, 2, 3 to discover information from a stakeholder in order to create the model? Your table 3 is claiming this but I don’t think all of the approaches are really elicitation tools. Ref 105 is an example of what I mean (I conducted that particular study in an industrial context (I was working in a financial company whilst doing this study but its goal was to identify where IT could support processes rather than find automation requirements). I understand, of course, you are searching on key words rather than conducting a detailed analysis each technique / tool / approach listed.

RESPONSE: I prefer to keep the long list. I really agree with your point about how community classifies the elicitation methods. But my objective was to find a set of methods and from it, filtering them to find the proper ones for automation, even if some methods do not seem elicitation ones.

POINT8: 4. Discussion. I like the Limitations. I wonder how old some of the studies are that involve elicitation of requirements with business processes? I know 105 was conducted in 2001. At that time, the main concern was the regrowth of the .dot com world after the bubble had burst. Perhaps automation was not on the visible horizon? Now, though, maybe there is a slow waking up to this fact. The key part of your excellent paper is figure 1: the money invested in business process automation is very large. There should be scope, therefore, to study this further.

RESPONSE: You have opened my mind to your viewpoint. Very interesting topic. I believe I can explore it in future work as you proposed.

POINT9: Does process mining imply a degree of automation needs to be in place already in order to conduct the mining?

RESPONSE: No. Process mining just needs event logs from the systems in the digital ecosystem of an organization.

Reviewer 2 Report

This paper is a systematic literature review on requirement elicitation methods for BPAS. I think this topic is useful for the future. However, there are some issues in this paper at present.  

1. Insufficient explanation of snowball sampling.

It is necessary to explain what snowball sampling is and how it is performed.  

2. Insufficient explanation on how to select papers.

Related to point 1, 75 of the 108 articles found were selected by snowball sampling. This may be due to the lack of keyword selection described in section 3.1.1. It is necessary to explain why the papers found by snowball sampling could not be found by keyword searches in the ACM Digital Library, IEEE Xplore, and Science Direct.

Author Response

Thank you for your feedback. It follows my responses.

POINT 1: Insufficient explanation of snowball sampling.

RESPONSE: I will add a footnote to briefly explain snowball sampling. I will add a detailed description of how I use the technique in the review: the eligibility criteria and the number of rounds utilized to get the sampling.

POINT 2: Insufficient explanation on how to select papers.

RESPONSE: I will add the reason into Section 3.2.

Round 2

Reviewer 2 Report

The manuscript has been revised well.