Modern Methods of Prediction

Definition

Humans have always wanted to know what the future holds in store for them. In earlier centuries, people often sought clues to the future from sacred texts. Today, more secular approaches are increasingly used, although the older approaches to the future persist. Modern methods for prediction include trend extrapolation, the Delphi method, mathematical modeling, and scenario analysis, including backcasting. Extrapolation was only possible when reliable past data became available. The Delphi method relies on the judgement of experts in the subject matter. Mathematical modeling has been very successful in the physical sciences, and, in the form of integrated assessment models (IAMs), has been applied to problems such as assessing future energy use. Scenario analysis looks at a number of possible futures and develops internally consistent story lines around each. It is often used in conjunction with IAMs. Each of the four methods, including both their strengths and weaknesses, are discussed in turn. Finally, this entry looks at the future of prediction, and concludes that despite progress in each of the four approaches treated, predicting the future, never easy, is now harder than ever.

1. Introduction: History of Forecasting and Prediction

There has never been any shortage of predictions about what the future will hold. Such forecasts have always been popular because humans have an urgent need to know what is in store for them, and because our actions today are guided to a large extent by our ‘image of the future’, to use the term introduced many decades ago by Dutch writer Fred Polak [1]. Two books on the future from the US, Alvin Toffler’s Future Shock [2] and John Naisbitt’s Megatrends [3]; each sold many millions of copies.

For many situations, we do not need to forecast, but can simply plan, such as what shows we will watch on TV this evening. We can even plan further ahead, as shown, for example, by publication of the dates for football matches for the coming year. Although, as the COVID-19 pandemic and ensuing lockdowns with the cancellations or postponement of sporting events (including the 2020 Tokyo Olympic Games, postponed until 2021) showed, these attempts to ‘fix’ the future, even the near future, can come unstuck.

Many pre-modern predictions are religion-based and often concern the date for the world’s end or the second coming [4]. Nevertheless, many of these types of forecasts were still being made in the 20th and 21st centuries. These include those made by the late Howard Camping, who used the bible to calculate the exact date for the end of the world, or Judgement Day, with (fortunately, all incorrect!) forecasts dating back to 1994. His last forecast, made shortly before his death, prophesied that the end would occur on 16 October 2011, following the rapture on the 21 May 2011, predictions no more accurate than his earlier ones. For Christians, life on Earth was brief, but was followed by an eternal future. Depending on their ethical conduct while living, this future would be spent in either heaven or hell. Thus, the image of their personal future strongly affected their life on Earth.

One famous early set of prophecies is that by Nostradamus, a 16th century French astrologer and seer. Even today, millions follow his predictions. Many have seen his prophecies as being validated by subsequent events [5]. Given how vague his ‘predictions’ are, it is easy to make them fit after the event. There will always be disasters—and assassinations. Further, as Leoni has pointed out, when used to try and predict future events, they have proved less than useful [6].

Yet even more recent predictions supposedly based on science and reason can get it badly wrong [7]. Very few economic forecasters foresaw the 2008 global financial crisis, and the 2019 COVID pandemic was similarly unforeseen, although pandemic specialists had warned years before that such an event could occur [8]. Where are the undersea settlements, the human settlement colonies on the moon, or flying cars?

Because of the poor accuracy record of earlier forecasts, modern methods of prediction owe little to these earlier approaches, where the terms prophecy and prediction were used interchangeably. For modern forecasting, the passing of forecast date enables predictions made a decade or more ago (especially if numerical) to be evaluated for accuracy. Many papers are now published on evaluations of past forecasts, facilitating a check on the efficacy of the various approaches, helping to improve prediction (see, e.g., [9,10,11,12,13]). (One recent set of forecasts which cannot be assessed for accuracy is by the well-known late futurologist Graham Molitor [14], who published forecasts for the entire third millennium. The leisure era was supposed to commence in 2015, and a new space age ‘commencing around 2500–3000’).

Figure 1 shows annual papers in the Scopus database with each of the terms ‘Delphi method’, ‘Integrated Assessment Models’, and ‘scenario analysis’ in the title, abstract or keywords, for the years 1995 to 2022. Extrapolation was not included, because the terms ‘extrapolation’, ‘extrapolation AND trend’, and ‘trend extrapolation’ on inspection showed that many papers were not concerned with forecasting.

The figures show far more annual papers on scenarios and Delphi approaches than for IAMs in recent years. The growth of the first two is roughly exponential, while that for IAMs is only roughly linear. The main reason is the far greater range of applications for these two methods. As Figure 1 also makes clear, it is not possible to include more than a small fraction of even post-1995 papers for discussion in this Entry.

Papers that give a non-mathematical description of each of the methods are favored, as well as papers that assess their strengths and weaknesses. The main findings are that improvements have been made in each of the four approaches discussed. At the same time, the rapid changes in both the biophysical and the socioeconomic worlds have made forecasting the future far more difficult. Nevertheless, despite these difficulties, we must attempt to forecast the future, even if we must abandon long-term forecasts; we have no alternative.

2. Modern Approaches to Future Prediction

Many companies exist to provide forecasting services, mainly to businesses. Businesses are prepared to pay for short-term forecasts of economic growth, inflation rates, interest rates, and so on. Forecasting is also an important academic topic, with many journals published. Among the best-known are the following:

• Foresight
• Futures
• International Journal of Forecasting
• Journal of Forecasting
• Long Range Planning
• Technological Forecasting & Social Change
• The Futurist

Nevertheless, the majority of articles on forecasting are published in specialist subject matter journals, not in journals dedicated to forecasting. For example, many forecasts for future energy use are published in various specialist energy journals. Also, in the energy field, a number of international energy organizations annually publish their predictions on future energy use by energy type and geographical region for decades ahead. The list includes the Shell, BP, and ExxonMobil oil companies, the World Energy Council, and the International Energy Agency (IEA).

2.1. General Comments

In recent decades, attempts have been made to put forecasting on a more solid foundation. In this Entry, several important methods are discussed, from the oldest and simplest, extrapolation, to the more recent ones of the Delphi method and Integrated Assessment Models (IAMs). Finally, scenarios in the form of ‘what if’ exercises have a long history but have been subject to much refinement in recent years. Although these are treated separately for convenience, all modern methods of prediction involve the use of mathematics, including statistics, and IAMs usually incorporate scenarios. Even the Delphi method can be incorporated into scenario construction [15].

An obstacle for forecasting by any approach, particularly in the social sciences, is the existence of self-fulfilling and self-negating forecasts [16]. The classic example of the former is if the likelihood of a bank failure becomes general knowledge. A subsequent run on the bank will make failure more probable. A modern example comes from the UK. Under the ‘predict and provide’ model, road interests forecast high levels of future traffic, which were used to justify ambitious road construction programs. The additional road space induced more traffic [17]. An example of a self-negating prediction is when warning of a disease outbreak causes the public to take precautionary measures such as vaccination, which can greatly lessen infection rates. Today, there is such a plethora of scenarios to choose from, especially on the topics of future energy and climate change, that it is doubtful if they will lead to either self-fulfilling or self-negating forecasts. Yet nearly all but a few scenarios forecast continued economic growth, and both the public and policy makers act as if economic growth can continue indefinitely.

Many past forecasts have had their prediction year elapse, allowing analysis of what accounts for their prediction successes or failures. Albright [18] performed such an analysis on the well-known 1967 study by Kahn and Wiener [19] with the title ‘The Year 2000’, which contained forecasts for ‘One hundred technical innovations very likely in the last third of the twentieth century’. He found that although less than half the predictions could be regarded as accurate, there were large variations in the success rate for different technological areas. Forecasts for computers and communications were 80% accurate: those for all other areas had less than 50% accuracy. An important question: has continued development of forecasting techniques led to more accurate forecasts?

2.2. Trend Extrapolation: Projecting the Future from the Past

Trend extrapolation as a prediction method—projecting past and present trends into the future—is the simplest way of forecasting and is still widely used. The method can handle known cyclic variations, such as the annual rise in sales of sunscreen products during summer. To be successful, extrapolation had to wait until accurate knowledge of both the past and present situation were obtained. Over the past century at least, and for some countries over several centuries, countries have recorded births, deaths and marriages, and household size and composition, often at a detailed spatial level. Data series in many cases are also available for energy production, transport levels, and other relevant parameters. Especially since the 1950s, detailed statistics by mode are available—often at state and urban level—for vehicle fleet sizes, air travel, electricity consumption, national, and global GDP, and energy production by type.

Global population size and growth is so important for future human prospects, therefore, it is used here to illustrate extrapolation as a prediction method. When accurate statistics for the past are not available, forecasts are necessarily in error, as illustrated by historical population forecasts [20]. Although population forecasts for high- and middle-income countries made decades ahead had acceptable accuracy, such was not the case for many low-income countries. Thus, the 1951 global population forecast made for 1980 was found to be badly underestimated: 3.3–3.6 billion versus the actual 4.4 billion [21]. Improved data from low-income countries enabled better forecasting. Extrapolating global population mainly involves assessing the fertility rate trends for each country and projecting them into the future, rather than simple extrapolation of gross population.

The UN expect this rate of global population growth to decline in the coming decades, with the latest UN median forecast showing a peak in year 2086 at 10.43 billion and declining slowly thereafter, but with the 2100 population still at 10.35 billion [22]. This figure can be compared with the world population at the beginning of the current era, and in 1900, estimated at 226 and 1563 million respectively [23].

The decline in the UN forecast after 2086 is attributed to fertility decline, not to famine or pandemics. Although global fertility decline has been slower than successive UN population forecasts have predicted, for some countries, fertility decline has been faster than forecast. In the important case of China, as recently as 2019, the UN projected China’s population to peak in 2031–2032, but it peaked almost a decade earlier in early 2023 [24]. Nevertheless, despite these shortcomings, extrapolation has generally worked well for estimates of global and national populations over the next five years or so. The percentage errors are far less than similar forecasts for global oil prices, where extrapolation has proven unsuccessful.

A good example of the successful application of trend extrapolation is Moore’s Law: ‘the number of transistors on a microprocessor chip will double every two years or so—which has generally meant that the chip’s performance will, too’ [25]. This law has guided microprocessor development since the 1960’s. Even so, as Waldrop pointed out, the doubling every two years has already begun to falter.

The growth of air travel (in revenue passenger-km (RPK) when plotted against global Gross Domestic Product (GDP) (measured in purchase parity pricing (PPP) constant 2017 US dollars (USD)) illustrates well the strengths and weaknesses of this method. Figure 2 plots the data from 1990 to 2019 and shows the very high correlation (R² = 0.984) between the two variables. Given this high correlation, it might be expected that the 2020 data would also fall on this curve. Yet COVID-19 and subsequent air travel restrictions saw RPK to only 2.3 trillion, whereas GDP only fell a few percent. COVID-19—at least for a few years—has broken the long-established link between the two.

Trend extrapolation does not always involve using a curve fitted to past data to predict the future. Marchetti [26] has argued that new technologies such as new transport modes or energy sources follow each other in a regular manner, similar to the way new biological species displace existing species in an ecological niche. He used this approach to predict the share of a new energy source, such as nuclear power, and when its share would peak. However, while providing useful insights, recent energy use patterns do not follow Marchetti’s predictions [27].

Figure 2. Global RPK vs. global GDP (trillion 2017 PPP USD), 1990–2019. Data Sources: Airbus [28], World Bank [29].

Figure 2. Global RPK vs. global GDP (trillion 2017 PPP USD), 1990–2019. Data Sources: Airbus [28], World Bank [29].

2.3. Delphi Method: The Judgement of Experts

The Delphi method is named from the ancient Greek oracles at Delphi, who were consulted about important decisions. According to Humphrey-Murto et al. [30], this approach has four essential features: anonymity of participants, iteration, statistical group response, and controlled feedback. As mentioned, the process is iterative: all responses are shared and respondents are free to modify their responses in light of the answers of other participants. It is thus a structured approach, ending when responses converge. The rationale is explained by Grime and Wright [31] as follows: ‘With repolling and feedback, it is assumed that the median response of the group shifts nearer to the true value of the outcome to be predicted’. A related assumption is that a group will together produce a more accurate response than an individual.

Delphi relies on expert judgements. One difficulty with experts is that while they clearly know more about the topic than non-experts, they may be over-optimistic as to the rate of progress in their field. This optimism can be shown by the study of Valette et al. [32]. They performed a Delphi Analysis study in 1974–1975 on the future for hydrogen (H2) as a fuel, drawing on an international panel of 86 experts from fields such as research and development, technology forecasting, and hydrogen. For the future use of H2 in transport, the median forecasts for the year 2000 share of H2 use in private road vehicles, public road vehicles, aircraft, and ships were 10%, 20%, 10%, and 2% respectively. The actual 2000 share was zero in all cases.

Another core assumption is that ‘group judgments are more valid than individual judgments’ [31]. Buckee and Johansson [33] have also argued in epidemiology that ‘Individual model forecasts can be misleading, but together they are useful’. Although not addressing the Delphi method, they showed that in the early days of the COVID-19 epidemic an ensemble of model results gave a better forecast of how the epidemic would evolve.

The Delphi method has proved a very popular approach to forecasting, especially in the health sciences. Although the overall success rate of the method for prediction (for instance, when a given technology breakthrough will occur) is mixed, some researchers feel that this is because the method had been either been incorrectly applied or needs refinement. Beiderbeck et al. [34] have reviewed a number of refinements to the method aimed at improving its forecast accuracy.

2.4. Mathematical Modeling: Integrated Assessment Models (IAMs)

Mathematical modeling has proved to be very useful in the physical sciences. In the case of astronomy, it is possible to predict very accurately where the moon or other planets in our solar system will be at any time in the future or when Halley’s comet will return. The French polymath Pierre-Simon Laplace (1749–1827) even thought that mathematical physics could be applied far more generally, in his deterministic universe we could predict ‘everything that happens over its entire history’ [35]. Yet modeling human behavior is difficult—or perhaps impossible.

Today, mathematics is mainly used in model development to fit experimental results or other data series, it is even used for understanding embodied intelligence [36]. As Turbé et al. [37] wrote that time-series are ubiquitous and ‘can describe physical systems such as the state of the atmosphere and its evolution, social and economic systems such as the financial market, and biological systems such as the heart and the brain via electrocardiogram (ECG) and electroencephalogram signals, respectively’. They further pointed out that Artificial Intelligence and neural networks can be used to improve predictability.

Gambhir et al. [38] analysed the various criticisms that have been made of IAMs, using bioenergy with carbon capture and storage (BECCS) as a case study. They found that ‘Critics have asserted that there is a lack of transparency around model structures and input assumptions, a lack of credibility in those input assumptions that are made visible, an over-reliance on particular technologies and an inadequate representation of real-world policies and processes such as innovation and behaviour change’.

Rosen and Guenther [39] argued that the economic results generated by IAMs were subject to much uncertainty and consequently ‘raise serious questions about whether or not the net costs and benefits of mitigating climate change over periods as long as 50 to 100 years can be known to such a level of accuracy that they should be reported to policymakers and the public’. They concluded that ‘policymakers should not base climate change mitigation policy on the estimated net economic impacts computed by integrated assessment models’. Uncertainties include the future costs and availability of various technologies, especially renewable energy (RE) sources [40,41], and whether the minimum cost will be the deciding factor in selection. Increasingly, energy independence considerations have been seen as important in technology selection.

Opinions are divided on the use of IAMs in climate mitigation modeling. In an exchange of views in the Nature journal, climate scientist Kevin Anderson argued that IAMs are simply the ‘wrong tool for the job’ [42]. Anderson believed that the technocratic solutions they offer could have worked in 1990, but the rate of carbon emissions reductions because of inactivity on CC has made them irrelevant. They are forced to rely on unproven technologies such BECCS. Lewis and Nemet [43] stressed the need for context-dependent approaches, rather than global assumptions.

Jessica Jewel defended the use of IAMs. She argued that they have played a central role in the CC debate, showing that the IAM-based scenarios would give dangerous levels of global warming in the present century if emissions continued on their present path. Yet even she sounded a note of caution, stating that ‘Although IAMs identify plausible combinations of potential solutions to mitigate climate change, they do not indicate whether these solutions are feasible at the required scale’ [42]. She also added that IAMs focus more on technocratic rather than social change.

2.5. Scenarios: Examining the Possible, Probable and Preferred Futures

One response to the problem of change in underlying conditions, which render extrapolation techniques of limited value, is the use of scenarios in forecasting. According to Durance and Godet [44]: ‘A scenario is not a future reality but rather a means to represent it with the aim of clarifying present action in light of possible and desirable futures. Foresight must master the constraints of the present’. They added five conditions scenarios must meet for scenarios ‘to be both credible and useful’: ‘pertinence, coherency, likelihood, importance, and transparency’.

Futurists often use the terms possible, probable, and preferred to classify futures [45]. Clearly, the future that does come to pass must be drawn from the list of possible futures. Possible futures only rule out those that violate well-established physical laws, such as energy conservation laws, or speed of light constancy. Preferred futures are also called backcasting, and unlike possible or probable futures, are inescapably value laden.

This begs the vital question: preferred by whom? FF producing countries want to continue producing FFs—for OPEC countries loss of oil or NG export revenue would in some cases be catastrophic for their economies. High-income countries (and high-income households in all countries) do not want to share incomes or wealth more equitably. The result is that, more than three decades after the first IPCC report was published in 1990, FF consumption is higher than ever and global energy-and industry-related CO₂ emissions have risen from 21.31 Gt (Gt = 10¹⁸ tonne) in 1990 to 33.88 Gt in 2021 [46,47].

Modern forecasts increasingly use scenario analysis, rather than single numerical predictions. Thus, it might appear difficult to assess their accuracy. Nevertheless, conditional scenarios are useful. An example would be in CC, where the uncertainty regarding human actions for CC mitigation are separated out from the biophysical aspects. Thus, it is useful to ask: if atmospheric GHG levels reach a certain value, what will this mean for global average temperature and for extreme weather events?

Backcasting scenarios are explicitly value laden—they deal with preferred futures. For example, Supapo et al. [48] examined various approaches that would produce a desired good—100% RE by 2040 for off-grid islands in the Philippines. Similarly, the IPCC [49] have modeled various scenarios that would keep global temperature increase below 1.5 or 2.0 °C.

3. Conclusions and Prospects

Is attempting to predict the future a waste of effort? As early as 1984, Moyer [50] entitled his article ‘The futility of forecasting’. Vaclav Smil [51] has stressed ‘the perils of long-range energy forecasting’, and later wrote an even stronger critical paper with the title: ‘Long range energy forecasts are no more than fairy tales’ [52].

Earth faces multiple serious environmental crises [53,54,55]. Since these can all interact in complex ways [56], it is no surprise that forecasting the future, even the next few years, is becoming more difficult. The uncertainty surrounding the interactions between these various sustainability challenges would seem to strengthen the position of Moyer and Smil.

Consider what are regarded as the most reliable forecasts, those for items like housing stock, where the large existing stock should enable reliable forecasts to be made for at least the following few years. Yet, as the ongoing devastation to infrastructure and housing caused by war in the Ukraine or by the February 2023 earthquake in Turkey and Syria showed, any projections made in (say) 2021 for 2023 will have to be discarded. None of the modern methods of forecasting discussed in earlier sections could have predicted the timing or severity of these catastrophes, although for earthquakes, Hall [57] points out improvements are occurring. (Hall also points out that two decades ago, earthquake researchers correctly identified the precise location of the next major earthquake but were unable to predict when it would occur. Nevertheless, the earthquake in Turkey cannot be even partly ascribed to human action; so even our predictions in the physical sciences need improvement.)

One response to this rising uncertainty is to broaden the range of what is considered possible when selecting scenarios. Yet how useful is this? Much controversy still surrounds the value for climate sensitivity, with the IPCC [58] giving only a ‘very likely’ range of 2–5 °C; others give a narrower (and lower) range [59]. Similarly, the recent spate of extreme weather events around the world has left climate scientists questioning the ability of their models to deal with such extreme events [60], which further complicates predictions [61].

Despite all the problems facing attempts to predict the future, we must keep trying. Consider a public organisation that is planning to invest in a large project such as the construction of a new hydro scheme or airport. Most costs occur in the short term, as revenues only accrue in the years following project completion. The organisation will have to estimate not only the project costs, but also the annual revenue stream, perhaps for decades into the future [62]. Increasingly, we will need to ask the question: should we invest in such projects, or should we instead only invest in projects with a more immediate payoff?

In conclusion we need to recognize that, just as was seen in Section 2.3 the pooled views of a number of experts can give more accurate forecasts than a single expert. Therefore, using a number of prediction methods—including the four central approaches discussed in this entry—can improve the accuracy of our necessary attempts to predict the future. Nevertheless, major improvements in prediction methods will be needed, given the increasingly uncertain futures the world will face.