In Search of Complementarity: Insights from an Exercise in Quantifying Qualitative Energy Futures

Abstract

In this study, we considered a bridging strategy between qualitative and quantitative research with the aim of achieving complementarity. A pilot case study using the Sheffield Elicitation Framework “SHELF” to estimate appropriate inputs for a quantitative energy systems model (based on a qualitative energy future scenario) was used to gain insights. Of novelty are the ethnographic insights of an example translation procedure as well as the methodological approach of the translation procedure itself. This paper reports the findings from this exercise concerning the practicalities of applying such a technique and the observations from the expert elicitation process itself. Based on this pilot, we make two recommendations. The first is the importance of devising a strategy in projects, and research programmes, where bridging between qualitative and quantitative research activities would be most effective. The second is that observations of discussions during the expert elicitation process provide value in the provenance of the estimates for quantitative modelling purposes and provide considerations for further development of qualitative future scenarios.

1. Introduction

The urgent need to eradicate unabated greenhouse gas emissions arising from human activities has led to shifts in our thinking around the appraisal, planning, and decision making for energy systems [1]. The properties of fossil fuels (coal, oil, and natural gas), such as energy density and portability among other factors, have enabled large centralised energy systems where power plants and stores are connected to grid networks of electricity and gas and networks of petrol stations supplying businesses, industry, and households [2]. When combined with reasonably predictable energy demand profiles and available computing power, this has led to the dominance of quantitative energy system models as tools for planning and decision making [3].

The properties of alternative renewable resources increase the complexity for both energy supply, due to the spatial distribution and intermittency of resources such as wind and solar, and energy demand, with solutions such as energy efficiency, time of use tariffs, and prosumption (producing and consuming) emerging as key considerations in future energy provision [4]. This has led to increased interest in understanding, and even modelling, the wider social aspects of energy systems rather than just the traditional security and cost issues, which in turn has led to increased interest in qualitative approaches and methods to understanding energy systems [5].

Planning for the transition of the energy system further requires approaches that can interface not only with the energy trilemma (energy security, sustainability, and affordability), but also the recent trends in decarbonisation, decentralisation and digitisation, sometimes referred to as the three Ds of energy policy [6]. The future energy system is increasingly being seen as undergoing a socio-technical transition rather than a wholly technological one [7]. This necessitates more interdisciplinarity across academic research generally, and in energy research specifically [8]. In this paper, we contribute to this need by exploring a novel method for incorporating insights from qualitative and quantitative research methods.

Our research sought to address the following questions:

• What are the challenges and benefits in achieving complementarity of qualitative and quantitative research?
• How suitable are tools from Bayesian statistics as a bridging strategy, i.e., for conducting the translation, and what useful qualitative information can be learned in the process?
• What are the recommendations for similar studies?

This paper is structured as follows. In the next section, we discuss a review of the literature on integrating qualitative and quantitative research with a focus on the energy sector. We then outline our case study in Section 3. In Section 4, we discuss the case study findings, and conclusions are drawn in Section 5.

2. Integrating Qualitative and Quantitative Energy Research

2.1. Integration Approaches

To date, most quantitative energy models have limited their scope to include mostly techno-economic factors, leaving political, social, and behavioural phenomena outside the scope of what is being modelled. This has typically been the way in which modellers have drawn boundaries around what their models can and cannot process. The need to consider broader societal concerns and behaviour have led to the emergence of new research approaches that seek to combine different kinds of technical and societal knowledge by using both quantitative and qualitative research methods. Combining, or integrating, both research methods is viewed as more comprehensive in providing information and support for addressing sustainability in decision making [9]. The ways in which qualitative and quantitative research practices can inform one another vary greatly in form and extent. Additionally, not all qualitative research can be integrated with quantitative modelling work, e.g., how changes in a system may be driven by the behaviour and interactions over time of the various actors in that system. This also applies vice versa; e.g., what behaviours and interactions of the various actors in the system are needed in order to realise particular levels and types of energy demand and supply. After reviewing the literature on approaches to qualitative-quantitative integration, this paper uses a case study to highlight the considerations, possibilities and challenges faced when undertaking such integration.

One of the ways that energy system research is seeking to integrate social and technical insights is by combining different kinds of disciplinary models to build Whole Systems perspectives. An example of this is the Whole Systems Energy Modelling Consortium (WholeSEM) project, which “employs extensive integration mechanisms to link and apply interdisciplinary models to key energy policy problems” (http://www.wholesem.ac.uk/about-the-project.html, accessed on 29 April 2019). This approach of formally integrating disciplinary insights through different mechanisms for soft or hard coupling of models is also used in the field of Socio-Technical Energy Transitions (STET) modelling [4], which aims to address the ‘social deficit’ in energy systems models by seeking to “develop formal quantitative energy models that also capture the elements of socio-technical transitions including societal actors and the co-evolutionary nature of policy, technology and behaviour” [10]. In an important positioning paper for this emerging field of research, Holtz et al. [4] proposed that this type of socio-technical modelling framework can provide significant contributions to the wider field of societal transitions research.

In a commentary on Holtz et al. [4], McDowall and Geels [7] argued for caution in over-emphasising the strengths of quantitative modelling for societal transitions. There are several challenges in doing so that relate to handling and down-playing uncertainties and hidden assumptions, and the tendency to implicitly adopt a positivist epistemology in modelling with a worldview that is more objective than subjective. The authors conclude that in complex processes such as societal transitions, pluralistic “bridging strategies” and dialogue between disciplines are preferable over formal integration of the (often) qualitative Socio-Technical Systems analysis with quantitative models [7].

Projects such as WholeSEM and the field of STET combine interdisciplinary social and technical knowledge, but they operate almost exclusively in the methodological realm of quantitative modelling. By seeking to extend this to also incorporate qualitative knowledge, they encounter significant difficulties, as highlighted by McDowall and Geels. In addition to the challenges outlined by McDowall and Geels [7], qualitative-quantitative integrations also bring together different aims and approaches to knowledge creation and (de)construction as well as differences in reasoning. In contrasting qualitative and quantitative research, the former tends to infer via induction and the latter via deduction modes of reasoning [11]. In short, many qualitative researchers in the social and socio-technical disciplines see themselves and their work very differently than many quantitative researchers. Therefore, translating between qualitative and quantitative types of knowledge is a very different challenge than bringing together quantitative types of knowledge from different disciplines.

To further develop this account of the challenges of integration, there can be an interdisciplinary hierarchy that shapes whether quantitative or qualitative research leads or dominates a collaborative project. Barry et al. [12] identified three different modes of interdisciplinarity in which academic researchers collaborate between disciplines: (1) integrative–synthesis; (2) subordination–service; and (3) agonistic–antagonistic. Projects involving engineering and natural sciences or social sciences have tended to adopt the subordination–service mode. This is where research questions defined by engineering and natural sciences have drawn in social scientists to help understand “social factors” [12].

Seen through the subordination-service mode of working, the social sciences become service providers to clients in the engineering disciplines in a common but problematic configuration which sets up unequal power relations that work against genuine collaborative enquiry. A critical social science literature exists which has documented this problematic relationship, and the effect it has had both on energy research since the 1970s and on the declining willingness of social scientists to engage in interdisciplinary energy research [13,14] as a result.

Understanding how qualitative research has been utilised in a predominantly quantitative research project, or indeed vice versa, is important in appraising the outcomes for decision and policy making. Research shows that energy researchers are driven by the underlying objective that their findings will be “useful” [15]; therefore, there is an argument that such interdisciplinary projects ought to be transparent as to the degree of subordination of different disciplines. The next section provides an overview of two notable cases that have taken a “hybrid” approach to energy systems modelling research in recent years. One is from the UK (collectively “Transition Pathways” and “Realising Transition Pathways”) and the other from the European Union (“Transition Pathways to Sustainable Low-Carbon Societies Project”).

2.2. Hybrid Research Projects

There have been attempts by qualitative and quantitative energy researchers to collaborate more closely with one another in recent years for the reasons outlined above. Two recent UK research projects are “Transitions Pathways” [16], its follow-up “Realising Transitions Pathways” [17] (TP and RTP, respectively, which for the purpose of this review are treated as one project), and the European Commission funded European Union “Transition Pathways to Sustainable Low-Carbon Societies Project” (referred to as Pathways) [18]. There are some key methodological differences between TP/RTP and Pathways that are discussed below.

The TP and RTP projects were conducted between 2008 and 2016 and together took the form of an interdisciplinary study of possible future pathways for the UK electricity system. The study’s primary objective was to develop “a set of interacting and complementary tools to analyse electricity network infrastructure investment and operational decisions, in order to model decisions to invest in the range of fossil and low carbon generation” (https://www.ucl.ac.uk/bartlett/energy/realising-transition-pathways, accessed on 29 April 2019). Hence, a key aim was to improve relevance for policy and decision making, and this necessitated engagement with stakeholders in the UK electricity system.

The sequential steps in the RTP process began with (qualitative) stakeholder dialogue, and this informed the following step of creating qualitative futures in “transition pathway narratives”. Next, there was a translation process whereby the narrative pathways underwent a quantification process, and these were tested for feasibility under a “technical elaboration process”. The translation step used here involved a range of methods, including expert elicitation and engineering models, along with a feedback loop to allow the modification of the narrative pathways to incorporate the insights generated. The final step in the RTP process moved beyond the technical feasibility of the pathways for the electricity system and considered cross sector interactions with the wider energy system (quantitative led) and spatial impacts on regional actors (qualitative led). An important aspect was that the project outcome was a process based on the complementarity of research methods for insights rather than an improved quantitative energy model per se [19].

This methodological framework for RTP provides a clear consideration of different types of knowledge (qualitative and quantitative) and their roles in generating different types of insight. The sequential framework and allowing for feedback and iteration to previous stages permits the complementary stages in the research to build on insights for more holistic information, and in turn improved decision and policy making. The framework focuses particularly on placing the modelling within wider strategy frameworks, which enable a model-informed, rather than a model-led, decision-making process [19]. The bridging process in TP/RTP between the scenario narratives and the modelling applied the “Story and Simulation” approach developed by Alcamo [20]. This draws on fuzzy set theory, where descriptions such as low, medium, and high, are translated into numerical ranges for quantities, and used for inputs into quantitative models. This technique is discussed in more detail in Section 2.3 below.

Turning to the Pathways project, the focus here was on the notion of “bridging” between existing tools and approaches, and this was a notable recommendation by McDowall and Geels referred to above [7]. Pathways “looked at ways to integrate alternative approaches for analysing sustainability transitions… [and implement] methods to bridge across scenario-modelling, socio-technical transition and practice-based action research approaches” ([21], p. 1).

Turnheim et al. [18] proposed a bridging method for linking three approaches: quantitative modelling, socio-technical analysis, and initiative-based learning. Rather than specifying a particular mathematical tool for translation of qualitative information into quantitative information, Turnheim and co-authors described an integration process for more robust “sustainable transition pathways”.

In the Pathways project, activities in and between three different research methods (quantitative systems modelling, socio-technical analysis, and initiative-based learning) can be used to appraise the energy system historically (“past”), currently (“present”), and in the future. The qualitative research, socio-technical analysis, and initiative-based learning informs, and is informed by, the quantitative systems modelling at various points in the timeframe of the energy system appraisal.

In comparison, in the TP/RTP project, the future (qualitative) narratives are checked for feasibility by the quantitative systems modelling. This means that while there are clear collaboration attempts between qualitative and quantitative research, qualitative futures are, in both, being constrained by the quantitative modelling for “sense-checking” and “feasibility” testing. While this does mean that the quantitative research can set limits on the qualitative input, and as such raises the issue of power dynamics between the teams and approaches, this does not amount to the subordination–service mode of working outlined earlier. Instead, there are attempts in the frameworks to have points in the process to open dialogue between qualitative and quantitative research in the development of these transition pathways for the energy system.

These dialogue openings are to some extent conducted in an iterative fashion. This provides some degree of integration: “There will be shallower and deeper ways of achieving bridging, and the specific approach taken in any given case will be determined by the resources available and the benefits expected from deeper integration” [21]. Importantly, these “dialogue points” sit in particular places, indicating that careful consideration must inform where bridging can be achieved and where the insights from a particular research method can be most usefully deployed in achieving overall objectives. In reporting on findings from the Pathways project, Hof and colleagues recommended that bridging should be considered as part of a “family of analytical procedures”, rather than as a singular formal methodology [21].

The growing practice in energy research of creating large and long-lasting centres to study aspects of transitions should be recognised as an opportunity to create and strategically enable moments of bridging between qualitative and quantitative research. This can be framed as dialogue steps, as in Hof et. al. [21], drawing on Turnheim [18]. Successful collaboration hinges as much on research programme timing and resources, including the skills of the personnel, as it does on the quality of ideas and models.

Methodological lessons from the Pathways project highlight the need to carve out a shared conceptual and methodological space and framework. The authors emphasise in particular the need to “open the black box” on the assumptions, analytical procedures, and data intrinsic to each approach [21]. Doing so forces researchers to adopt “non-optimal approaches” (seen from the perspective of individual approaches), also sometimes referred to as “second best” approaches, which have consequences in terms of complexity, but which add to their reliability and usefulness for decision-makers [22]. This amounts to pragmatism and contextual compromise, through which all parties must be open to—and indeed see a real benefit in—departing from their usual research practice to enable meaningful collaboration.

Table 1. Summary features of the TP/RTP and Pathways Projects.

Project	Scope	Process Steps	Qualitative & Quantitative Roles	Translation or Bridging Approach
TP/RTP [17]	UK electricity system (TP) then UK whole energy system (RTP)	Iterative	Technological aspects led by quantitative models. Spatial and actor behaviour aspects led by qualitative research.	Expert elicitation and translation following the “Story and Simulation” approach [19,20]
Pathways [18]	Selected EU cities and regions whole energy system	Linear with dialogue openings	Feasibility checks by quantitative models. Qualitative research considerations at dialogue openings.	Qualitative research informs quantitative modelling that conducts feasibility and sense checking thereby informing qualitative research.

below provides a summary of the main features of these two integrative qualitative and quantitative research projects.

The relationship between quantitative and qualitative methodologies and their proponents, as outlined above, is not neutral. Further, there is no agreed-upon methodology that enables perfect translation of information and knowledge between them. Developing an understanding and appreciation of different epistemological underpinnings of research methodologies and types of inferences through methods of reasoning (deduction or induction) is an important step in developing a shared methodological framework that must involve both qualitative and quantitative researchers in pragmatic but productive compromise.

2.3. Translation Methods

Beyond these two recent cases of hybridity and bridging, there are several translation methods that draw on tools and techniques from the fields of mathematics and statistics. What all translation methods have in common is the elicitation of expert judgments as to what plausible values the qualitative items may take. However, there are then different schools of thought, broadly fuzzy and Bayesian, that use theory to develop mathematical or statistical models for representing this information. In this section, we outline four further examples of such approaches to provide an illustration of the range of approaches that have been taken by researchers seeking to translate qualitative and quantitative data.

A mathematical approach that has been used for translating qualitative information to quantitative data, and was used in the RTP project, is fuzzy set theory within Alcamo’s “Story and Simulation Approach” [20] (referred to here as SAS). The goal in the SAS approach was to harmonise qualitative and quantitative scenarios. The theory allows for the “fuzziness”, or lack of clear boundary conditions, that characterizes much of the qualitative information and assumptions. For example, what does low, medium, or high mean? In deploying this theory, stakeholders or experts are asked to provide what they believe to be a plausible range of values for, say, low, medium, or high qualities of an item in the different scenarios. Membership functions are then used to determine the boundary conditions and estimates are derived to be used in quantitative modelling. The translation back from quantitative outputs to qualitative can then be put through the same functions in reverse to “fuzzify” the quantitative outputs and compare with the original narratives for consistency. The advantage of this method is its reproducibility and ease in communication since a detailed understanding of statistics is not a prerequisite. However, fuzzy sets are criticized as to whether they adequately model causal relations [23], and due to their lack of operational definitions (e.g., what is a “plausible range” of values?) [24].

Statistical tools have also been used for translating qualitative and quantitative information in futures studies. For example, in “Q2” a Delphi method of expert elicitation was used followed by cluster analysis [25], and in another study morphological analysis was adopted [26].

In the Q2 method, it is argued that the adoption of cluster analysis allows for a diverse range of expert views, particularly when a system is dynamically changing and mathematical modelling may not be appropriate [25]. The cluster analysis in Q2 was used to aggregate the individual elicited judgements from the experts, elicited virtually via a questionnaire, into a smaller number of “group estimates”. The result was a number of aggregated estimates, one for each cluster. A criticism of such an approach is that the resulting estimates do not represent the beliefs of any of the experts.

Adopting morphological analysis allows for multidimensional problems and seeks to identify parameters that co-exist with each other. This is to identify interdependent relationships that form an inference model [27,28] and is closely related to Bayesian analysis [29]. Weaknesses of this method are that it is time consuming and can create trivial co-existing relationships; strong facilitation in expert workshops is required [27].

The Intergovernmental Panel on Climate Change (IPCC), in development of their latest set of futures, have also taken the route of the development of narratives of Shared Socio-economic Pathways, and matching and translating these into quantities to be used in a range of Integrated Assessment Models (IAMs) [30,31]. There does not appear to be a specific translation method reported as being used in the literature; rather, they have been used as “guiding assumptions” for socioeconomic projections [32].

Another statistical technique that can be used in the translation of qualitative to quantitative information is the SHeffield ELicition Framework (SHELF) [33], a tool to enable the aggregation of expert judgement. SHELF is founded on Bayesian probability theory and elicits expert judgments of uncertainty to determine probability distributions for each unknown quantity of interest. The result is a probability distribution on each quantity, representing a consensus view of a group of experts.

This range of approaches shows the diversity of practice in this area of research. Within the statistics and uncertainty quantification fields, there is debate about which tools and techniques are more truly representative of observed data and causal relations. However, as Lavine argues, considering different models that describe the data “reasonably” well can give greater confidence on the inferences made, rather than a focus on determination of true values [34]. In statistical approaches (e.g., frequentist and Bayesian models), inferences can converge provided there is sufficient data. In situations with limited observed data, as is often the case, this choice becomes more important. In addition, there is the question as to what would be relevant data in complicated processes such as the societal transition. However, the purpose of this project is not to answer this statistical debate, but to trial a single translation method as a means for illustrating the challenges in converting qualitative to quantitative information.

In this research, we pilot a particular bridging strategy for translation of qualitative narrative energy scenarios to quantitative information for energy modelling and report on findings and observations. The novel approach in this test case applies Bayesian statistics via SHELF as a bridging approach to conduct the translation from qualitative to quantitative information and provide qualitative information about the process itself. The authors are not aware of any similar studies that have produced qualitative data about the translation process.

In this pilot project, we selected the SHELF method with its Bayesian approach. The expert elicitation process generates continuous probability distributions to represent the expert’s uncertainty about each of the quantities of interest. The key feature of SHELF is that it is a group elicitation method in which the experts provide initial individual estimates for, discuss, and then come to consensus on, each quantity of interest. This facilitates more reflexive thinking in the expert elicitation process. This in turn enables the qualitative observation of the elicitation process, providing further insights that can aid strengthening complementarity between qualitative and quantitative research.

3. Materials and Methods

3.1. Expert Elicitation

Before outlining our case study, we first discuss expert elicitation, a tool adopted in our translation method between qualitative and quantitative research.

The elicitation of expert judgements typically refers to the process of obtaining quantitative information about unknowns from an expert or multiple experts, based on their beliefs and the evidence available to them, for use in modelling [35,36]. This information is used to specify a probability distribution over the unknown quantities, which represents the expert’s uncertainty about the values of unknowns. In Bayesian statistics [37], this probability distribution is known as a prior distribution, but more generally can be referred to as an uncertainty distribution.

The unknown ‘quantities of interest’ are typically input values (parameters) to mathematical models. Input parameters are often rather obscure in nature; asking an expert about their likely values explicitly is not useful (for example, a slope parameter in a linear regression model). It is regarded as good practice to only ask experts about quantities which could be observed in practice. Therefore, part of the elicitation process is to convert the information expressed by the expert into a probability distribution for the model input parameters.

Consider as an example that we are interested in an expert’s beliefs about the energy generation from a particular wind farm next year. We could ask the expert to provide us with the value above and below which they think it is equally likely the true generation of the farm will be. This is the expert’s median (the 50% point of their uncertainty distribution). Suppose that this is 5000 MWh. We could then ask the expert for the value which they think it is equally likely that the generation is between 0 MWh and this value and this value and 5000 MWh. This is the expert’s lower quartile (25% point of their uncertainty distribution). A similar process would provide their upper quartile. A probability distribution can then be fitted to these three values. There are several ways to do this. Typically, a particular parametric distribution is assumed, such as a Normal distribution, and then parameters, e.g., mean and variance of the distribution, are chosen to match the three elicited values as closely as possible [36].

Other approaches are available, which are based on asking about other quantiles of an expert’s uncertainty distribution, or directly getting the expert to “draw” their uncertainty distribution by giving them a number of “chips” and asking them to place them into “bins” representing a range of values. This latter approach is called “the roulette method” [36].

The simplest form of expert judgement elicitation will repeat this process for each unknown over which we would like to quantify the expert’s beliefs. This would be very time consuming if there are many input parameters needed for a model. For complex models, a smaller subset of the input parameters could be chosen to elicit the expert’s uncertainty on. These parameters would be chosen to be those which the model outputs are most sensitive to, or those which are otherwise considered the “most important” inputs in the model. For example, they may be the model inputs with the greatest uncertainty.

This technical part of the elicitation, obtaining the quantitative estimates of the expert’s uncertainty and translating them to a probability distribution, only makes up a small element in an expert judgement elicitation. The complete steps in the elicitation depend on the circumstances and the approach to be used. In this pilot exercise, we undertook an elicitation using the SHELF approach. We detail the full set of steps required in an expert elicitation below.

The above procedure will work well when eliciting the beliefs of a single expert to quantify their uncertainty. However, we will typically want to ask a range of experts for their beliefs if the question to be answered is of wider scientific importance. This begs the question: who is an expert? In this exercise, we identified a range of people with expertise in the energy sector that we invited to an elicitation workshop. When conducting studies of systems undergoing complex transitions such as the energy system, it is of course challenging to identify those with the appropriate expertise. As we have acknowledged, there can also be questions about inclusivity and diversity, but for the purposes of this project, we adopted a more general definition of ‘expert’.

Different elicitation methods have techniques for dealing with uncertainty about the quantity of interest’s true value based on the opinions elicited. In the literature, it is observed that these tend to be one of the following three types.

• Uncertainty distributions are created for each expert individually and these are then combined using some mathematical rule to obtain a single uncertainty distribution.
• Experts are brought together to agree on some “consensus” uncertainty distribution.
• Combination or “hybrid” of the above two approaches, where there are some elements of the mathematical rule and group consensus approaches.

We outline the three most widely used methods of expert judgement aggregation: the Classical Method [38], SHELF [39], and the modified Delphi method [40]. These reflect mathematical, consensus, and hybrid approaches, respectively.

In the Classical method, experts are first asked to provide their uncertainty (e.g., quartiles) about some unknowns to which they do not know the true values while the person carrying out the elicitation (the facilitator) does. The answers are then scored on how well-calibrated they are and how much information they provide for these “seed” questions, and these two scores are used to weight the experts, with those achieving higher scores being given higher weight. For a well-calibrated expert, 25% of the true values of the seed questions would lie below the expert’s lower quartile, 25% would lie between the lower quartile and the median, 25% would lie between the median and the upper quartile, and 25% would lie above the upper quartile. A more informed expert would be less uncertain about the value of the quantity of interest and therefore provide narrower uncertainty bounds (quartiles) than a less informed expert. The aggregated uncertainty distributions for the unknowns of interest are then weighted averages of the experts’ distributions using these weights. There is software available to carry out the aggregation. Since the experts will not all have the same degree of expertise, or expertise in the same topic, the attraction of this method is that allowance is made for this variability. A weakness of this method is the challenge in devising seed questions that will reflect the degree of expertise for the unknowns of interest and therefore could introduce unintended bias.

In the SHELF approach, the experts are brought together into the same room (or virtually) and an elicitation workshop is conducted. For each unknown quantity to be elicited, each expert initially specifies their uncertainty distribution using one of the methods described earlier. All the uncertainty distributions are then displayed to the experts, and the experts discuss these to understand the differences between them. Following this, a group elicitation takes place, in which the experts are asked to come to a consensus about an uncertainty distribution for the unknown quantity of interest. The consensus reached does not represent the actual beliefs of any of the experts individually but aims to represent the beliefs of a Rational Impartial Observer (RIO), someone who has listened to all the discussions and has understood all the arguments made. A full set of resources for the SHELF method is available online on the developer’s webpage (http://www.jeremy-oakley.staff.shef.ac.uk/shelf/, accessed on 1 April 2019). This method has an advantage, as it focuses on the degree of uncertainty in expert’s estimates, and avoids the weakness in the Classic method by taking a group consensus approach. However, there are criticisms of consensus approaches due to interpersonal and power dynamics [41], but we have attempted to mitigate those effects through incorporating ethnographic approaches.

In the Delphi method, each expert makes decisions about the quantity of interest independently of the other experts. The facilitator, who acts as a gatekeeper, shares anonymised information between them. There would normally be a number of rounds where experts assess the quantity of interest and anonymised answers are shared among the experts. In each round, an expert provides their uncertainty distribution for the unknown quantity of interest. In between each round, time is provided for the experts to reflect on their answers using the information shared by the facilitator. By performing several rounds, an attempt is made to converge towards consensus. If consensus is not reached, then a mathematical rule to create an aggregated uncertainty distribution is adopted. In the Q2 method, clustering analysis was used. Another recently developed hybrid approach is the IDEA protocol [42]. This method has the advantage of avoiding interpersonal dynamics, as the experts do not interact; however, there are criticisms about the authenticity of any convergence in opinion, and of possible biases introduced by the gatekeeper [43].

In each of the three methods above, there are several process steps in addition to the elicitation itself. A generic list of these is given below, but note that tasks can vary for each type of method:

• Selection of experts: selected to span all areas of expertise relevant to the elicitation.
• Arranging the workshop: this could be in person and all together (e.g., SHELF) or one-on-one (e.g., Classical method), virtually or via email or online questionnaire (e.g., Delphi).
• Putting together an evidence dossier that contains the quantitative information available about the unknowns of interest. Much of this information may be collected from the experts.
• The experts are given time to review the evidence dossier.
• Statistical training for the experts: they have expertise in the quantities of interest but may not have much knowledge about probability and statistics. Some training is then provided so that the experts are clear about what they are being asked to do.
• Feedback and revision: the fitted uncertainty distributions are fed back to the experts to check whether these represent their beliefs and uncertainty with subsequent rounds until agreement is reached.
• Documenting: the arguments used to justify the choices of the experts need to be documented in an anonymous fashion (to encourage honesty). Good practices in recording who attended and in what capacity, etc. should be followed.

The elicitation will therefore provide a set of uncertainty distributions, one for each unknown input parameter (“quantity of interest”), in a mathematical model, such as an energy systems model. These can in turn be used to perform an uncertainty analysis.

An uncertainty analysis is performed by first obtaining a single value of each input parameter by sampling once from each of the uncertainty distributions (assuming the parameters are independent). This single set of input parameters is then run through the mathematical model a single time. This will provide a single value for the model output (or one value for each model output if the model has multiple outputs). By repeating this process several times, we will obtain different values of the model outputs, one for each set of sampled model inputs. By plotting the distribution of these model output values, for example as a histogram, we have an estimate of our uncertainty distributions on the model outputs. This is known as Monte Carlo sampling or Monte Carlo simulation [44].

It is worth noting here that the mathematical model we will be running our samples through in the case study below has two different modes: deterministic and stochastic. Monte Carlo sampling can be used with either, although more samples are necessary to obtain a good estimate of the uncertainty distribution on the outputs with stochastic models, due to the added uncertainty in the models.

We now turn to our case study itself, in which we used the SHELF method to elicit quantitative estimates of future electricity demand in a qualitative future energy scenario.

3.2. Case Study

3.2.1. Overview

This pilot was conducted over a limited timeframe and at a particular point in a longer and larger research programme: the UK Centre for Energy Systems Integration (CESI) (https://www.ncl.ac.uk/cesi/, accessed on 1 April 2019), where a range of qualitative and quantitative research outputs and tools were at varying degrees of development and completion. Pragmatic decisions and choices were therefore taken regarding the “dialogue steps”, as described in Section 2.1, for the collaboration between the qualitative and quantitative research to proceed. This case study therefore contrasts somewhat to the ideally situated dialogue steps described above [18]. In practice, these dialogue points are better understood as “windows of opportunity” that may open or close at different times over the course of a project or programme.

In the future, an attempt to rebalance away from pragmatism toward ideal interactions would need to be planned over a longer timeframe, or at the outset in a research programme of scale. Here, rather than determining specific dialogue points where collaboration would be ideal from a methodological and analytical point of view, the project had to create dialogue steps where they were practically possible. In this case study, we sought to bring about translations between a specific energy model developed to a sufficient degree and qualitative energy futures that had been created. The authors acknowledge that a pragmatic approach was taken in this pilot study to understand the challenges and opportunities that adopting a translation or bridging between qualitative and quantitative research can provide. Therefore, we further acknowledge that the study here is of a shallower rather than a deeper nature, using the language of Hof et al. [21].

The window of opportunity identified within the CESI research programme was to apply a bridging strategy between qualitative energy futures developed focusing on the North of Tyne region [45] and the quantitative energy model for investment planning for the national electricity system [46]. The bridging method selected was to the (Bayesian) SHELF approach to gather information at an expert elicitation workshop hosted by the authors, and to use this information to estimate quantities of interest and the degree of uncertainty about them, for use in the investment planning model. More information about the respective qualitative and quantitative research in the case study is provided below.

3.2.2. Qualitative Energy Futures

The qualitative research used for the bridging exercise was a set of narrative future energy scenarios focusing on the North of Tyne region that were developed within the CESI research programme in 2019 [45]. The narrative energy scenarios were developed in collaboration with energy governance and civil society stakeholders in the North of Tyne through a hosted workshop. A participatory exploratory approach was adopted whereby stakeholders were encouraged to discuss and agree on the top two drivers for change in the North of Tyne energy system. These were used to create a 2 × 2 matrix of possible future energy scenarios, and stakeholders then discussed a possible pathway for each quadrant (high/low of each driver). This is a popular framework for future development and is used by National Grid (https://www.nationalgrideso.com/future-energy/future-energy-scenarios, accessed on 1 April 2019) in their annual future energy scenarios.

A key aspect of this methodological approach is that no probabilities are assigned to any of the resulting scenarios (in contrast to predictive and forecasting methods) [47]. The objective with this approach is to identity a broad range of possible scenarios to facilitate improved strategic thinking about future risks and opportunities [48]. The participatory explorative futures approach was selected, as it is well-suited to background conditions of fundamental systemic change and answering questions around what could happen as opposed to what will happen [49]. This approach is also suited to qualitative or narrative futures, and avoids what has been described as the “caged thinking” of models [50].

While there were a range of important drivers for change in the North of Tyne energy system, following discussions, the stakeholders agreed on the top two being decarbonisation and equity. Figure 1 below shows the North of Tyne energy futures 2 × 2 matrix. Further discussions in groups for each of the quadrants enabled a pathway for each to be developed and agreed with the participants.

Table 2. North of Tyne Narrative Energy Futures—Summary Features [45].

Fair Bare Minimum	Minimal Change	Just and Sustainable	Draconian Decarbonisation
High Equity, Low Decarbonisation	Low Equity, Low Decarbonisation	High Equity, High Decarbonisation	Low Equity, High Decarbonisation
Regional prosperity Carbon tax with waiver for poor Rewilding	Capitalism persists and monetary driven basis for any decision making Imports heavily relied on including for skills Increased fuel poverty and inequality	Fairness is prioritised Localised development autonomy including development of skills Strong circular economy	Heavy carbon taxation—disproportionately impacting on the poor Centralised energy—lots of nuclear power Carbon capture and storage essential

provides a summary of the features in each of these scenarios. See [45] for a more detailed discussion of this process.

For the purposes of this project, the expert elicitation focused on trialling the bridging strategy with the “Just and Sustainable” scenario.

3.2.3. Quantitative Energy Model

The quantitative aspect of this pilot was an energy model developed within CESI to assist with investment planning decisions for the national electricity system. The model is primarily a network planning model rather than an operation model; however, it does undergo a simplified operation feasibility check. The model also currently only considers electricity provision, although it is intended to include heating and transport in further developments. The model [46] is coded in Julia, uses the Gurobi optimizer, and requires a number of packages to be installed, including Suppressor, Printf, and XLSX.

For the purposes of the study, which focused on the bridging strategy, the model was run in deterministic mode, i.e., a single pathway to represent one of the future scenarios developed for the North of Tyne region. Running the model repeatedly in deterministic mode, with different values for the input variables sampled from a probability distribution, represents a means of introducing uncertainty in the values for the quantities of interest into the model.

To run the model in a way that makes use of the qualitative knowledge of the experts, certain inputs were needed. These are referred to as ‘quantities of interest’ in the context of expert elicitation. The purpose of the expert elicitation workshop was to draw on expert opinions as to likely future electricity demand levels between now and 2050 for the “Just and Sustainable” scenario, in which the net zero target is reached by 2050. Great Britain’s annual electricity demand in this scenario was the quantity of interest to our modelling colleagues and what the elicitation event focused on.

Future electricity demand is of course uncertain, and our aim was to elicit expert views to develop a probability distribution of possible values for the future years 2030 and 2040 to provide two interim points between now and 2050, the target year for many CO₂ policies and laws in the UK. Samples from these distributions could then be run through the model in a Monte Carlo simulation, and the resulting probability distributions on the key model outputs compared with the narrative scenarios to further enrich the scenarios with quantitative characteristics. The qualitative aspects of conducting the exercise also help inform quantitative modelling as to the experts’ thinking and their views of the uncertainty for this quantity.

4. Results

4.1. Elicitation Event

The authors hosted the elicitation event with support from CESI colleagues. A total of eight experts were recruited for the event that was held on 27 August 2019 over a half day. The experts were recruited from suggestions made by the participants in the North of Tyne energy futures development workshop in 2019 and further contacts through CESI colleagues. The aim was to have representation of a range of expertise across the energy system. The affiliations of the attendees were as follows: national grid, an electricity distribution network operator (DNO), a gas network operator, a local enterprise partnership (LEP), a national fuel poverty charity, an energy storage business, and academia.

The elicitation was conducted using SHELF. Prior to the workshop, the experts were sent an evidence dossier which contained all the quantitative evidence available on energy demand. This is provided as Supplementary Material File S1. A briefing document was also provided to the experts, explaining how the elicitation would work. This is provided as Supplementary Material File S2. In the workshop, a summary of the narrative scenario was provided, followed by a training exercise to train the experts in expressing their knowledge as probabilities. This short training exercise was undertaken to illustrate the process of providing the uncertainty we wanted to elicit. Attention then turned to the two quantities of interest. The experts were asked to estimate annual electricity demand in Terawatt hours (TWh) for GB in the years 2030 and 2040.

Each round of elicitation was undertaken under the assumption that the future was as described in the “Just and Sustainable” narrative scenario. In the initial phase of each elicitation, the experts provided their individual judgements about the quantities of interest using the median and quartiles approach described earlier. The eight anonymised probability distributions were then shown to all the experts, and the experts discussed their rationales and evidence for giving the judgements they had. In the following phase the experts were invited to achieve a consensus on the judgements of the Rational Impartial Observer (RIO) for that quantity. This group elicitation used the roulette method, that is, the experts collectively placed roulette chips to form a histogram. The use of different approaches to elicitation in the individual and group stages is regarded as good practice.

The authors were time- and resource-constrained, resulting in it only being feasible to elicit limited quantities in the workshop. However, it was important to give the experts time for discussion, and it was felt to be more important to elicit a small number of quantities well rather than a larger number in a hurried fashion. This too is because the focus of the study was on the process of the bridging strategy.

A further adjustment was made to accommodate a greater interest in qualitative and methodological data than is the norm for SHELF elicitations. This was achieved by audio recording the session and enabling content analysis of the process.

4.2. Quantitative Results

For the two elicitation quantities, the initial stage of the elicitation produced eight sets of lower quartiles, medians, and upper quartiles (one for each expert). The elicited lower quartile (LQ), median, and upper quartile (UQ) for the total annual electricity demand (TWh) for GB in 2030 and 2040 are provided in

Table 3. Elicited quartiles for total GB electricity demand in 2030.

Quantiles	Expert A	Expert B	Expert C	Expert D	Expert E	Expert F	Expert G	Expert H
LQ	264	265	275	210	245	215	250	265
Median	290	285	290	230	255	220	300	300
UQ	320	295	330	245	290	235	320	400

and

Table 4. Elicited quartiles for total GB electricity demand in 2040.

Quantiles	Expert A	Expert B	Expert C	Expert D	Expert E	Expert F	Expert G	Expert H
LQ	220	260	310	200	360	200	320	320
Median	320	310	345	220	390	210	350	350
UQ	340	340	375	265	420	220	380	400

below.

These quantities were used to create a graphical representation and presented to the experts. The graphs are shown in Figure 2 and Figure 3 below. Each expert is represented by a horizontal line, with the dot representing the expert’s median and the length of the line representing the interquartile range, from the lower quartile to the upper quartile. The longer the line, the greater the uncertainty around the central estimate.

There are clear differences between the experts, with some giving very narrow uncertainty intervals and some much wider. There are also groupings of experts, with some predicting an increase in electricity demand from today’s figures (330 TWh in 2021; [51]) and some predicting a decrease. Following a group discussion, consensus distributions representing RIO were created using the roulette method. Here, the experts collectively made decisions, placing 20 “chips”, each worth a probability of 0.05, into pre-defined “bins”. Histograms representing the number of chips placed in the bins for certain levels of future electricity demand are shown in Figure 4 for the years 2030 and 2040. The ranges of values were taken from the individual exercise outlined above. The same number of bins was offered for 2030 and 2040; however, note that in 2040 the range is twice that of 2030, and therefore each bin represented 20 TWh increments compared with 10 TWh increments in 2030.

SHELF was then used to fit a parametric probability distribution to each of these two empirical distributions. The best-fitting distribution to the 2030 electricity demand was a gamma distribution, with shape parameter 4.19 and rate parameter 0.0647. This probability density function is provided in the left-hand side of Figure 5 below. The best-fitting distribution for the total demand in 2040 was a beta distribution with parameters 4.77 and 3.13, respectively. This probability density function is provided in the right-hand side of Figure 5 below.

We can see that the consensus distribution for 2030 indicates that the experts feel that total electricity demand will almost certainly be between 200 and 300 TWh, with a most-likely value around 260 TWh. The distribution shows positive skew, with a larger tail on the right. In 2040, the experts feel that the total electricity demand will be between 150 and 450 TWh, with a most-likely value around 320 TWh. This distribution shows a negative skew, with a larger tail on the left.

4.3. Qualitative Observations

A qualitative analysis of the expert elicitation event was carried out using methods of participant observation and audio recording with subsequent analysis. It focused on two areas:

• A content analysis of audio recordings of the elicitation event
• A methodological evaluation of the expert elicitation process

The content analysis provided information about the way the experts engaged with the questions asked in the elicitation process. It helped the authors understand what information the experts found to be important but had not directly been asked about (which pieces of contextual information were missing that would have better shaped their judgements) and enabled the research team to use that information to better understand how they had arrived at the answers they did give. The authors are not aware of such analyses taking place in elicitation studies, and therefore this represents uniqueness to this exercise.

Two types of data came out in the analysis; insight into what kinds of uncertainty the energy systems experts themselves were grappling with in their own work and understanding of how these uncertainties might have affected the elicited future energy demand probabilities.

Data on the kinds of uncertainties showed that, first of all, there was a mismatch between the questions the experts had been asked to address and the questions some of experts argued they should have been asked or would have been more important. This suggests a disconnect between modelling activity and the most important questions and uncertainties as seen from the perspective of experts.

The experts had been invited to answer questions on annual electricity demand; however, experts said that uncertainties around other key quantities were also important and interconnected. These were peak rather than total annual electricity demand, the impact from the take up of electric vehicles, and availability or not of hydrogen for heating.

“For us, we are worrying about peak demand, because that is about how much we’re going to have to invest in infrastructure to be able to support electric vehicles and heat pumps and so on, and you have chosen total (annual) electricity… if we, as a society, can bring the peak down, we can reduce investment. So, I am wondering why you have left peak demand out?”

“EVs is a big story for us, and it is the area of most uncertainty for us because that is when you start to get into behavioural shifts and that is really hard and getting it wrong, the consequences become big”

The process of making assumptions about future demand further shows the entanglement of uncertainties that are both technical, social, and political and cut across areas of expertise, such as generation-consumption and energy-transport.

There were three main areas of uncertainty about the future electricity demand that experts discussed:

• Changes to transport and heating demand, notably hydrogen availability, will significantly impact electricity demand
• Demand reduction through technological change is unlikely to continue at the rate it has in recent years (this is important, as it shows how problematic it is to use historical data for future scenarios)
• Increased decentralisation will lead to increased electricity demand

The following anonymised quotations are illustrative of the data and provide insight into which uncertainties were seen as important when making estimates of future energy demand and what the experts were processing mentally when quantifying their estimates:

“I was trying to balance between reduction of demand through energy efficiency and an increase in demand from EV transport”

“I was thinking about how the demand for electricity might be influenced by the shift to decarbonised generation”

“We have kind of gone as far as we can on energy efficiency and we’re unlikely to get much more from solar because that kind of reached its Zenith and tailed off”

“There probably isn’t going to be much change in electricity until EVs become bigger—diesel ban is not till 2030 so until then there might not be much change”

“I was thinking about localism [outlined as part of the narrative scenario] and producing things locally thinking does that create a boom in the industrial sector? If you don’t want to be shipping stuff in from China and creating pollution elsewhere then you see a return to more being produced locally… Localism is very carbon intensive”

These quotations reveal the range of uncertain issues that the experts wrestled with before settling on an estimate for a particular quantity. The diverse uncertainties are representative of the multi-faceted challenge and complexity in achieving a future decarbonised energy system [21]. In this sense, the consensus estimates are perhaps most accurately thought of as hybrid estimates, as they emerge from the interaction and discussion of the different mental models of the experts, rather than being based on common consensual understandings, assumptions, and definitions.

5. Discussion

The main purpose of this pilot translation study was to critically assess the suitability of the method. Therefore, we did not ask if SHELF works, but rather if SHELF was a useful tool for the intended purpose. In evaluating this, the authors paid attention to three questions in particular:

• What were the final numbers a representation of?
• What uses do these numbers have?
• Did the experts have expertise in the questions they were asked?

Providing a quantity for future electricity demand, and for a particular future scenario, is challenging. During the process, the experts drew on the information provided to them by the background dossier they had been provided with as well as information and assumptions about ‘the future’ which they themselves held. This made it difficult to distinguish the extent to which they were providing probable future demand levels for the narrative scenarios provided, or for futures determined by their own worldviews. The balance between scenario futures and personally predicted futures was most likely not the same for all participants. The scenarios were loosely specified, so the experts would have all imagined them differently.

To deliver a qualitative picture of the kind of future demand anticipated by the experts and the kinds of defined and undefined uncertainties surrounding this is not problematic as such. However, a potential problem does become clear at the point of applying the consensus process based on a “Rational Impartial Observer” (RIO) and arriving at a single probability distribution. At this point, several potentially quite different personal future scenarios are conflated, and thus the achievement of consensus, and understanding the uncertainty about the quantity of interest, is problematic. The process creates ‘black boxes’ in the form of new artificial uncertainty specifications that cannot be decomposed to their constituent parts, as this is not a mathematical derivation of the prior distribution, it is a negotiated distribution.

Use of the outcome distributions in future scenario development and quantitative modelling processes need to proceed carefully, always ensuring that the use of the outcomes of the SHELF process are informed by a detailed appreciation of what they represent. Further work is required to fully consider the effect of using a method such as SHELF on questions that have such a high level of complexity and indeterminacy. In contrast, for much more tightly defined and bounded problems, the method would result in a probability specification with a clearer interpretation.

A further area for critical reflection surrounds the appropriateness of expertise. Here it is important to note the way the experts, all selected for their expertise in energy (but each with their different expertise within the energy sector), can be said to represent the required expertise to respond to the question. While there is no doubt about their technical expertise when it comes to energy systems, the uncertainties that they were grappling with were not exclusively (not even primarily) to do with technical energy expertise; rather they were socio-technical that included questions around transport, governance, and policy as well as social and cultural change. It is no surprise that the areas of greatest uncertainty for the experts were areas outside of their personal expertise. This represents an area where further work is needed to develop the appropriateness of the SHELF method for estimation of quantities embedded in future scenarios which are inherently much more open to differences in imagination and that must engage expertise in areas of socio-technical, cultural, and political change.

The outputs from the expert elicitation, namely the two sets of simulated electricity demand in the years 2030 and 2040, can be used to perform a Monte Carlo simulation to assess the impact of this uncertainty on the outputs of the planning model outlined in Section 3.2. However, this would also require a review of the other inputs the model requires, to ensure consistency with the Just and Sustainable narrative scenario.

Taking 1000 samples from the Monte Carlo simulation will result in 1000 runs of the model: one for each pair of simulated values of electricity demand in 2030 and 2040. This will produce 1000 sets of potential investment decisions, and therefore 1000 sets of installed capacity of the different electricity technologies considered in the model. That is, these runs will provide 1000 potential optimal investment decisions to be made in a Just and Sustainable future energy scenario, given our current uncertainty on the electricity demand for this scenario. Further research is needed into how such information can support decision making.

6. Conclusions

We return to the questions raised in Section 1:

• What are the challenges and benefits in achieving complementarity of qualitative and quantitative research?
• How suitable are tools from Bayesian statistics as a bridging strategy, i.e., for conducting the translation, and what useful qualitative information can be learned in the process?
• What are the recommendations for similar studies?

While this pilot case study was limited in terms of time and resources, and in the timely availability of qualitative and quantitative research outputs, we were able to draw some general conclusions that enable these questions to be addressed.

6.1. Achieving Complementarity of Qualitative and Quantitative Research

Insight provided by a study into the Control Rooms of the Future highlighted the difficulty faced by modellers in understanding the needs and interests of external stakeholders in policy and industry, and in finding ways to make models as useful as possible [52]. In this paper, we have focused on complementarity between qualitative and quantitative approaches to energy systems research and reporting on and discussing a reflective case study of qualitative–quantitative translation and bridging. The findings here would also apply in other contexts where an interchange of qualitative and quantitative information could lead to improved outcomes.

A key aspect of complementarity in qualitative and quantitative research found in this research was that engagement with stakeholders and experts in workshops or other activities provide opportunities for a range of information to be generated on a range of topics that matter to stakeholders. In our case study, discussions highlighted key uncertainties existing around future energy and climate policy, and around how to make future energy systems fair and equitable on different scales. Future research would need to ensure that such insights gathered from stakeholders are ‘made to matter’ in shaping modelling research at early ‘dialogue steps’ rather than once modelling is well-established. Steps must therefore be taken to ensure that qualitative information gathered in this manner is incorporated into the research process and allowed to significantly shape research questions and research outputs.

6.2. Suitability of Bayesian Tools for the Translation Process, and the Value of Qualitative Data

While the SHELF process was found to be a useful tool for expert elicitation and for generating data about the process of quantifying uncertainty, the process was time-consuming, which was problematic in several ways. As the main purpose of the project was to trial the process, it was decided to focus on fewer questions more thoroughly, rather than attempt to rush through more ‘quantities of interest’. Having only a small number of questions to answer enabled the experts to have time to debate their decisions and provide reasons for their answers, which made the qualitative data richer. In terms of future research using the SHELF process, this is a question (and a trade-off) that needs further consideration. The much longer elicitation sessions that would be required to create estimates properly on many more quantities of interest would very likely run up against problems of participant fatigue, reluctance to commit to attendance, and perhaps even drop-out. This would be especially true in contexts whether experts are busy professionals and are not being rewarded, financially or otherwise, for their participation.

Whereas the potential of the SHELF method to generate quantitative data has been well-established, the capacity for it to provide insights from qualitative data has not to our knowledge been explored. The potential for the process to generate both quantitative and qualitative data is therefore a particularly interesting area for further research. Indeed, the qualitative data about quantification generated at least as much valuable insight about the problem of future energy demand, the scenarios, and the broader energy system as can be gleaned from the probability density functions, which are normally the sole intended output of the SHELF method.

6.3. Recommendations

We have two clear recommendations for similar research in the future. First, while not all qualitative research can be integrated into quantitative modelling (and vice versa), it is important that in achieving complementarity, coordination of activities in different disciplines is established early on in a programme of research. This is so that strategic opportunities in programmes of research for dialogue steps between qualitative and quantitative disciplines, and exchanges of insights, can be identified early. Ideally, this should include specific steps that identify the opportunities, activities, and resources for deep rather than shallow qualitative input into, and shaping of, quantitative modelling activities.

Secondly, generating qualitative data during translation and bridging activities is an extremely effective way to understand the processes of the experts themselves concerning their estimates of the quantities of interest. It provides quantitative modellers with the data they need to understand the provenance of the estimates produced and some critical aspects that could be missing from their models and further aspects to consider in qualitative scenario building for qualitative researchers. This contrasts with the way in which these quantifications are conventionally produced, as ‘black boxes’ with limited rationales reported, taken from notes made during the workshop. We see qualitative data created during and about the translation process as important reference material for all future users’ hybrid research.

Further research is needed in including ethnographic data to improve the quality of the of translation methods between qualitative and quantitative information.