# Information Theory for Correlation Analysis and Estimation of Uncertainty Reduction in Maps and Models

## Abstract

**:**

## 1. Introduction

**Figure 1.**Principle of the evaluation of spatial uncertainty reduction through additional information, investigated in this work. (

**a**) Map of three regions with uncertain boundaries (dashed lines) and cells in a regular grid used for subsequent uncertainty analysis. (

**b**) Uncertainty estimation based on probabilities of discrete outcomes in each cell. (

**c**) Estimated reduction of uncertainties; given the information in one cell (black outline), the remaining uncertainty within this cell is 0 and uncertainties in the surrounding cells are reduced. Adapted from Figure 2 in [2].

**Figure 2.**Example model of a geological layer at depth with two types of uncertainty: (

**i**) the depth to the top surface of the layer is uncertain; and (

**ii**) the thickness of the layer is uncertain.

## 2. Analysis of Spatial Uncertainties with Measures from Information Theory

#### 2.1. Uncertainty of a Single Random Variable

- If no uncertainty exists at a specific location, then the measure is zero;
- The measure is strictly greater than zero when uncertainty exists;
- If several outcomes at a location are probable, and all are equally likely, then the uncertainty is maximal;
- If an additional outcome is considered, the uncertainty cannot be lower than without this outcome;

#### 2.2. Interpretation in a Spatial Context: Uncertainty at a Single Location

#### 2.3. Correlations of Uncertainty between Two Variables or Locations

#### 2.3.1. Joint Entropy

#### 2.3.2. Conditional Entropy

#### 2.3.3. Mutual Information

#### 2.4. Correlations of Uncertainty between Multiple Variables

## 3. Estimation of Uncertainty Correlation and Reduction in a Geological Model

#### 3.1. Analysis of Uncertainties at a Potential Drilling Location

**Figure 3.**Probability distribution of geological units (${p}_{C}(\overrightarrow{r})$ for the cover layer, ${p}_{L}(\overrightarrow{r})$ for the layer of interest and ${p}_{B}(\overrightarrow{r})$ for the base) at depth in drill-hole example, and the corresponding information entropy $H({X}_{r})$; the dashed gray lines and the labels indicate important positions in the graph: A: ${p}_{C}=1$, $H=0$: no uncertainty; at B and D, H = 1, two outcomes equally probable (${p}_{C}={p}_{L}=0.5$ at B, ${p}_{L}={p}_{B}=0.5$ at D); C: minimum of uncertainty for layer; see text for further description.

#### 3.2. Uncertainty Correlation between Two Locations at Depth

- In each of the areas of highest uncertainty (label B and D in Figure 3), knowing the outcome at one point will necessarily reduce the uncertainty in the surrounding areas. This is due to the set-up of the example simulation (representing here, for example, geological expert knowledge) where the boundary position in the subsurface is simulated as normally distributed around an expected value.
- Uncertain areas about the top and the base of the layer of interest should be correlated and knowing the outcome in the uncertain area about the top of the layer should influence uncertainties about the base, and vice versa. Although this correlation might be counter-intuitive, it follows from the set-up of the model with the top of the layer of interest defined, and the thickness considered uncertain (and not the base of the layer).

**Figure 4.**Joint entropy between two variables at different z-values (depth in drill-hole). Labels correspond to important features, explained in detail in the text. The colour bar is set to reflect increments of ${log}_{2}(n)$, with an additional subdivision between two steps for visualisation purposes. Joint entropy is a symmetrical measure, the dashed line represents ${z}_{1}={z}_{2}$.

**Figure 5.**Conditional entropy of one variable ${X}_{2}$ at depth, given the information of another variable ${X}_{1}$. It is interesting to note that the entropies around ${z}_{2}=60\phantom{\rule{0.166667em}{0ex}}\mathrm{m}$ are reduced when information around a depth of ${z}_{1}=30\phantom{\rule{0.166667em}{0ex}}\mathrm{m}$ is obtained (label C) and vice versa (label D). The figure also clearly shows that conditional entropy is not symmetrical. The colour bar is set to reflect increments of ${log}_{2}(0.5)$.

**Figure 6.**Mutual information between two random variables ${X}_{1}$ and ${X}_{2}$ at different z-positions in the drill-hole example. Variables close to each other with high entropy share a large amount of information (labels A and B). However, it is also interesting to note that variables at locations close to the top share information with variables around the base of the layer (label C). Mutual information is a symmetrical measure, the dashed line represents the symmetry axis, ${z}_{1}={z}_{2}$.

#### 3.3. Interpretation of the Relationship between all Measures

**Figure 7.**Graphical representation of the relationships between information entropy, joint entropy, conditional entropy, and mutual information (after [17]). The z values correspond to the actual depth values in the drill-hole example and the corresponding pair-wise entropy measures, see Figure 4, Figure 5 and Figure 6. (

**a**) ${X}_{1}$ at ${z}_{1}$ = 30$\phantom{\rule{0.166667em}{0ex}}\mathrm{m}$, ${X}_{2}$ at ${z}_{2}$ = 31$\phantom{\rule{0.166667em}{0ex}}\mathrm{m}$; (

**b**) ${X}_{1}$ at ${z}_{1}$ = 30$\phantom{\rule{0.166667em}{0ex}}\mathrm{m}$, ${X}_{2}$ at ${z}_{2}$ = 60$\phantom{\rule{0.166667em}{0ex}}\mathrm{m}$.

#### 3.4. Application of Multivariate Conditional Entropy to Determine Entropy Reduction during Drilling

**Figure 8.**Reduction of conditional entropy during information gain; the matrix visualisation is comparable with Figure 5 but shows here the remaining uncertainty of a variable ${X}_{n}$ at a position ${z}_{n}$, given the information of all variables ${X}_{1},{X}_{2},\cdots ,{X}_{n-1}$ at shallower positions ${z}_{1},{z}_{2},\cdots ,{z}_{n-1}$; the bottom figure shows cuts in x-direction through the matrix, at the positions of highest uncertainty (30$\phantom{\rule{0.166667em}{0ex}}\mathrm{m}$ and 60$\phantom{\rule{0.166667em}{0ex}}\mathrm{m}$); the right figure shows conditional entropy profiles with remaining uncertainties after drilling to 30$\phantom{\rule{0.166667em}{0ex}}\mathrm{m}$, 43$\phantom{\rule{0.166667em}{0ex}}\mathrm{m}$, and 60$\phantom{\rule{0.166667em}{0ex}}\mathrm{m}$.

#### 3.5. Determination of Structural Correlations of Uncertainty in a Higher Dimension

**Figure 9.**Extension of the example model to simulate folded top and base surfaces of the central layer.

**Table 1.**Application of information theoretic measures for uncertainty estimation, correlation analysis, and estimations of uncertainty reduction.

Step | Measure | Use | Variables | Spatial Interpretation |
---|---|---|---|---|

I | Information entropy | Uncertainty quantification | Single variable | Analysis of uncertainty at one location |

Joint entropy | Uncertainty quantification | Two variables | Analysis of combined uncertainty at two locations | |

II | Mutual information | Correlation analysis | Two variables | Estimate of information shared between two different locations in space |

III | Conditional entropy | Uncertainty reduction | Two variables | Estimate of how information at one location would reduce uncertainty in space |

Multivariate conditional entropy | Uncertainty reduction | Multiple variables | Estimate of how information at multiple locations would reduce uncertainty in space |

## 4. Discussion and Conclusions

## Acknowledgements

## Appendix

## A. Information Entropy of a Coin Flip

**Figure A1.**Information entropy of a binary system: in the case of the fair coin with P(head) = P(tail) = 0.5, the information entropy is maximal with a value of $H(0.5)=1$ (green dot); in the case of the bent coin with P(head) = 0.7, the uncertainty of the system is reduced, and the information entropy is accordingly lower $H(0.7)\approx 0.88$ (red dot). In the case of a double headed coin with P(head) = 1, no uncertainty remains because the outcome is known, and $H(1.0)=0$ (black dot).

## References

- Potter, K.; Gerber, S.; Anderson, E.W. Visualization of uncertainty without a mean. IEEE Comput. Graph. Appl.
**2013**, 33, 75–79. [Google Scholar] [CrossRef] [PubMed] - Wellmann, J.F.; Regenauer-Lieb, K. Uncertainties have a meaning: Information entropy as a quality measure for 3-D geological models. Tectonophysics
**2012**, 526–529, 207–216. [Google Scholar] [CrossRef] - Zhang, J.; Goodchild, M.F. Uncertainty in Geographical Information; Taylor & Francis: New York, NY, USA, 2002. [Google Scholar]
- Devillers, R.; Jeansoulin, R. Fundamentals of Spatial Data Quality; ISTE: London, UK, 2006. [Google Scholar]
- Atkinson, P.M.; Foody, G.M. Uncertainty in remote sensing and GIS: Fundamentals. In Uncertainty in Remote Sensing and GIS; John Wiley & Sons, Ltd: Chichester, UK, 2006; pp. 1–18. [Google Scholar]
- Jeansoulin, R.; Papini, O.; Prade, H.; Schockaert, S. Introduction: Uncertainty issues in spatial information. In Studies in Fuzziness and Soft Computing; Jeansoulin, R., Papini, O., Prade, H., Schockaert, S., Eds.; Springer: Heidelberg, Germany, 2010; pp. 1–11. [Google Scholar]
- Fisher, P.; Comber, A.; Wadsworth, R. Approaches to Uncertainty in Spatial Data. In Fundamentals of Spatial Data Quality; ISTE: London, UK, 2006; pp. 43–59. [Google Scholar]
- Chilès, J.P.; Delfiner, P. Geostatistics: Modeling Spatial Uncertainty; Wiley: New York, NY, USA, 1999. [Google Scholar]
- Deutsch, V.C. Geostatistical Reservoir Modeling (Applied Geostatistics Series); Oxford University Press: New York, NY, USA, 2002. [Google Scholar]
- Turner, A. Challenges and trends for geological modelling and visualisation. Bull. Eng. Geol. Environ.
**2006**, 65, 109–127. [Google Scholar] [CrossRef] - Caers, J. Modeling Uncertainty in the Earth Sciences; John Wiley & Sons, Ltd: Chichester, UK, 2011. [Google Scholar]
- Wellmann, J.F.; Horowitz, F.G.; Schill, E.; Regenauer-Lieb, K. Towards incorporating uncertainty of structural data in 3D geological inversion. Tectonophysics
**2010**, 490, 141–151. [Google Scholar] [CrossRef] - Lindsay, M.; Aillères, L.; Jessell, M.; de Kemp, E.; Betts, P.G. Locating and quantifying geological uncertainty in three-dimensional models: Analysis of the Gippsland Basin, southeastern Australia. Tectonophysics
**2012**, 546–547, 1–44. [Google Scholar] [CrossRef] - Goodchild, F.M.; Buttenfield, B.; Wood, J. Introduction to visualizing data validity. In Visualization in Geographical Information Systems; Hearnshaw, M.H., Unwin, J.D., Eds.; John Wiley & Sons: New York, NY, USA, 1994; pp. 141–149. [Google Scholar]
- Cover, T.M.; Thomas, J.A. Elements of Information Theory, 2nd ed.; John Wiley & Sons, Inc.: Hoboken, NJ, USA, 2005. [Google Scholar]
- MacKay, D.J. Information Theory, Inference, and Learning Algorithms, 4th ed.; Cambridge University Press: Cambridge, UK, 2003. [Google Scholar]
- Ben-Naim, A. A Farewell to Entropy; World Scientific: Singapore, 2008. [Google Scholar]
- Shannon, E.C. A mathematical theory of communication. Bell Syst. Tech. J.
**1948**, 27, 379–423. [Google Scholar] [CrossRef] - Metropolis, N.; Ulam, S. The monte carlo method. J. Am. Stat. Assoc.
**1949**, 44, 335–341. [Google Scholar] [CrossRef] [PubMed] - Tacher, L.; Pomian-Srzednicki, I.; Parriaux, A. Geological uncertainties associated with 3-D subsurface models. Comput. Geosci.
**2006**, 32, 212–221. [Google Scholar] [CrossRef] - Thore, P.; Shtuka, A.; Lecour, M.; Ait-Ettajer, T.; Cognot, R. Structural uncertainties: Determination, management, and applications. Geophysics
**2002**, 67, 840–852. [Google Scholar] [CrossRef] - Bardossy, G.; Fodor, J. Evaluation of Uncertainties and Risks in Geology: New Mathematical Approaches for their Handling; Springer: Berlin, Germany, 2004. [Google Scholar]
- Suzuki, S.; Caumon, G.; Caers, J. Dynamic data integration for structural modeling: Model screening approach using a distance-based model parameterization. Comput. Geosci.
**2008**, 12, 105–119. [Google Scholar] [CrossRef] - Lindsay, M.; Aillères, L.; Jessell, M. Integrating geological uncertainty into combined geological and potential field inversions. In Proceedings of the GeoMod 2010 Conference, Lisbon, Portugal, 27–29 September 2010.
- Refsgaard, C.J.; Christensen, S.; Sonnenborg, O.T.; Seifert, D.; Højberg, L.A.; Troldborg, L. Review of strategies for handling geological uncertainty in groundwater flow and transport modelling. Adv. Water Resour.
**2011**, 36, 36–50. [Google Scholar] [CrossRef] - Bistacchi, A.; Massironi, M.; Dal Piaz, V.G.; Monopoli, B.; Schiavo, A.; Toffolon, G. 3D fold and fault reconstruction with an uncertainty model: An example from an Alpine tunnel case study. Comput. Geosci.
**2008**, 34, 351–372. [Google Scholar] [CrossRef] - Cherpeau, N.; Caumon, G.; Lévy, B. Stochastic simulations of fault networks in 3D structural modeling. Compt. Rendus Geosci.
**2010**, 342, 687–694. [Google Scholar] [CrossRef] - Judge, P.A.; Allmendinger, R.W. Assessing uncertainties in balanced cross sections. J. Struct. Geol.
**2011**, 33, 458–467. [Google Scholar] [CrossRef] - Cherpeau, N.; Caumon, G.; Caers, J.; Lévy, B. Method for stochastic inverse modeling of fault geometry and connectivity using flow data. Math. Geosci.
**2012**, 44, 147–168. [Google Scholar] [CrossRef]

© 2013 by the author; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

## Share and Cite

**MDPI and ACS Style**

Wellmann, J.F.
Information Theory for Correlation Analysis and Estimation of Uncertainty Reduction in Maps and Models. *Entropy* **2013**, *15*, 1464-1485.
https://doi.org/10.3390/e15041464

**AMA Style**

Wellmann JF.
Information Theory for Correlation Analysis and Estimation of Uncertainty Reduction in Maps and Models. *Entropy*. 2013; 15(4):1464-1485.
https://doi.org/10.3390/e15041464

**Chicago/Turabian Style**

Wellmann, J. Florian.
2013. "Information Theory for Correlation Analysis and Estimation of Uncertainty Reduction in Maps and Models" *Entropy* 15, no. 4: 1464-1485.
https://doi.org/10.3390/e15041464