# Uncertainty Assessment of Hyperspectral Image Classification: Deep Learning vs. Random Forest

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Methods and Dataset

#### 2.1. Method

#### 2.1.1. Supervised Uncertainty Assessment Approach

_{i}is the probability of class membership for h class labels. Further, the selection of the logarithm base is unimportant, as it only affects the units of entropy [25].

#### 2.1.2. Deep Neural Network (DNN)

_{j}total input from x

_{j}:

_{j}, into a class probability, P

_{j}, by using a normalised exponential function named “softmax”:

#### 2.1.3. Random Forests as a Benchmark

#### 2.1.4. RMSE of Uncertainty Assessment

#### 2.2. Datasets

## 3. Results

#### 3.1. Salinas Simulation Experiments

#### 3.2. Indian Pines Simulation Experiments

## 4. Discussion

#### 4.1. Comparing the Quality of Uncertainty Assessment Based on RMSE

#### 4.2. Quality of Uncertainty Assessment for Different Sample Sizes

#### 4.3. Uncertainty vs. Accuracy

## 5. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Xie, Y.; Sha, Z.; Yu, M. Remote sensing imagery in vegetation mapping: A review. J. Plant Ecol.
**2008**, 1, 9–23. [Google Scholar] [CrossRef] - Roodposhti, M.S.; Safarrad, T.; Shahabi, H. Drought sensitivity mapping using two one-class support vector machine algorithms. Atmos. Res.
**2017**, 193, 73–82. [Google Scholar] [CrossRef] - Dutta, R.; Das, A.; Aryal, J. Big data integration shows australian bush-fire frequency is increasing significantly. R. Soc. Open Sci.
**2016**, 3, 150241. [Google Scholar] [CrossRef] - Xiao, J.; Shen, Y.; Ge, J.; Tateishi, R.; Tang, C.; Liang, Y.; Huang, Z. Evaluating urban expansion and land use change in shijiazhuang, china, by using gis and remote sensing. Landsc. Urban Plan.
**2006**, 75, 69–80. [Google Scholar] [CrossRef] - Weng, Q.; Quattrochi, D.; Gamba, P.E. Urban Remote Sensing; CRC Press: Boca Raton, FL, USA, 2018. [Google Scholar]
- Congalton, R.G.; Green, K. Assessing the Accuracy of Remotely Sensed Data: Principles and Practices; CRC Press: Boca Raton, FL, USA, 2008. [Google Scholar]
- Congalton, R.G. A review of assessing the accuracy of classifications of remotely sensed data. Remote Sens. Environ.
**1991**, 37, 35–46. [Google Scholar] [CrossRef] - Khatami, R.; Mountrakis, G.; Stehman, S.V. Mapping per-pixel predicted accuracy of classified remote sensing images. Remote Sens. Environ.
**2017**, 191, 156–167. [Google Scholar] [CrossRef] - Cohen, J. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychol. Bull.
**1968**, 70, 213. [Google Scholar] [CrossRef] - Richards, J.A.; Jia, X. Remote Sensing Digital Image Analysis: An Introduction; Springer Inc.: New York, NY, USA, 1999; p. 363. [Google Scholar]
- Ye, S.; Pontius, R.G.; Rakshit, R. A review of accuracy assessment for object-based image analysis: From per-pixel to per-polygon approaches. ISPRS J. Photogramm. Remote Sens.
**2018**, 141, 137–147. [Google Scholar] [CrossRef] - Foody, G.M. Status of land cover classification accuracy assessment. Remote Sens. Environ.
**2002**, 80, 185–201. [Google Scholar] [CrossRef] - Comber, A.; Fisher, P.; Brunsdon, C.; Khmag, A. Spatial analysis of remote sensing image classification accuracy. Remote Sens. Environ.
**2012**, 127, 237–246. [Google Scholar] [CrossRef] [Green Version] - Yu, Q.; Gong, P.; Tian, Y.Q.; Pu, R.; Yang, J. Factors affecting spatial variation of classification uncertainty in an image object-based vegetation mapping. Photogramm. Eng. Remote Sens.
**2008**, 74, 1007–1018. [Google Scholar] [CrossRef] - Burnicki, A.C. Modeling the probability of misclassification in a map of land cover change. Photogramm. Eng. Remote Sens.
**2011**, 77, 39–49. [Google Scholar] [CrossRef] - Tsutsumida, N.; Comber, A.J. Measures of spatio-temporal accuracy for time series land cover data. Int. J. Appl. Earth Obs. Geoinf.
**2015**, 41, 46–55. [Google Scholar] [CrossRef] - Stehman, S.V. Selecting and interpreting measures of thematic classification accuracy. Remote Sens. Environ.
**1997**, 62, 77–89. [Google Scholar] [CrossRef] - Legleiter, C.J.; Goodchild, M.F. Alternative representations of in-stream habitat: Classification using remote sensing, hydraulic modeling, and fuzzy logic. Int. J. Geogr. Inf. Sci.
**2005**, 19, 29–50. [Google Scholar] [CrossRef] - Loosvelt, L.; Peters, J.; Skriver, H.; Lievens, H.; Van Coillie, F.M.; De Baets, B.; Verhoest, N.E. Random forests as a tool for estimating uncertainty at pixel-level in sar image classification. Int. J. Appl. Earth Obs. Geoinf.
**2012**, 19, 173–184. [Google Scholar] [CrossRef] - Dehghan, H.; Ghassemian, H. Measurement of uncertainty by the entropy: Application to the classification of mss data. Int. J. Remote Sens.
**2006**, 27, 4005–4014. [Google Scholar] [CrossRef] - Giacco, F.; Thiel, C.; Pugliese, L.; Scarpetta, S.; Marinaro, M. Uncertainty analysis for the classification of multispectral satellite images using svms and soms. IEEE Trans. Geosci. Remote Sens.
**2010**, 48, 3769–3779. [Google Scholar] [CrossRef] - Prasad, M.G.; Arora, M.K. A simple measure of confidence for fuzzy land-cover classification from remote-sensing data. Int. J. Remote Sens.
**2014**, 35, 8122–8137. [Google Scholar] [CrossRef] - Wang, Q.; Shi, W. Unsupervised classification based on fuzzy c-means with uncertainty analysis. Remote Sens. Lett.
**2013**, 4, 1087–1096. [Google Scholar] [CrossRef] - Foody, G.M.; Campbell, N.; Trodd, N.; Wood, T. Derivation and applications of probabilistic measures of class membership from the maximum-likelihood classification. Photogramm. Eng. Remote Sens.
**1992**, 58, 1335–1341. [Google Scholar] - Brown, K.; Foody, G.; Atkinson, P. Estimating per-pixel thematic uncertainty in remote sensing classifications. Int. J. Remote Sens.
**2009**, 30, 209–229. [Google Scholar] [CrossRef] - McIver, D.K.; Friedl, M.A. Estimating pixel-scale land cover classification confidence using nonparametric machine learning methods. IEEE Trans. Geosci. Remote Sens.
**2001**, 39, 1959–1968. [Google Scholar] [CrossRef] - Breiman, L. Random forests. Mach. Learn.
**2001**, 45, 5–32. [Google Scholar] [CrossRef] - Breiman, L. Bagging predictors. Mach. Learn.
**1996**, 24, 123–140. [Google Scholar] [CrossRef] [Green Version] - Naidoo, L.; Cho, M.A.; Mathieu, R.; Asner, G. Classification of savanna tree species, in the greater kruger national park region, by integrating hyperspectral and lidar data in a random forest data mining environment. ISPRS J. Photogramm. Remote Sens.
**2012**, 69, 167–179. [Google Scholar] [CrossRef] - Chan, J.C.-W.; Paelinckx, D. Evaluation of random forest and adaboost tree-based ensemble classification and spectral band selection for ecotope mapping using airborne hyperspectral imagery. Remote Sens. Environ.
**2008**, 112, 2999–3011. [Google Scholar] [CrossRef] - Crawford, M.M.; Ham, J.; Chen, Y.; Ghosh, J. Random forests of binary hierarchical classifiers for analysis of hyperspectral data. In Proceedings of the 2003 IEEE Workshop on Advances in Techniques for Analysis of Remotely Sensed Data, Greenbelt, MD, USA, 27–28 October 2003; pp. 337–345. [Google Scholar] [Green Version]
- Lawrence, R.L.; Wood, S.D.; Sheley, R.L. Mapping invasive plants using hyperspectral imagery and breiman cutler classifications (randomforest). Remote Sens. Environ.
**2006**, 100, 356–362. [Google Scholar] [CrossRef] - Abdel-Rahman, E.M.; Makori, D.M.; Landmann, T.; Piiroinen, R.; Gasim, S.; Pellikka, P.; Raina, S.K. The utility of aisa eagle hyperspectral data and random forest classifier for flower mapping. Remote Sens.
**2015**, 7, 13298–13318. [Google Scholar] [CrossRef] - Coulston, J.W.; Blinn, C.E.; Thomas, V.A.; Wynne, R.H. Approximating prediction uncertainty for random forest regression models. Photogramm. Eng. Remote Sens.
**2016**, 82, 189–197. [Google Scholar] [CrossRef] - Ließ, M.; Glaser, B.; Huwe, B. Uncertainty in the spatial prediction of soil texture: Comparison of regression tree and random forest models. Geoderma
**2012**, 170, 70–79. [Google Scholar] [CrossRef] - Loosvelt, L.; De Baets, B.; Pauwels, V.R.; Verhoest, N.E. Assessing hydrologic prediction uncertainty resulting from soft land cover classification. J. Hydrol.
**2014**, 517, 411–424. [Google Scholar] [CrossRef] - Caruana, R.; Karampatziakis, N.; Yessenalina, A. An empirical evaluation of supervised learning in high dimensions. In Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland, 5–9 July 2008; ACM: New York, NY, USA, 2008; pp. 96–103. [Google Scholar] [Green Version]
- Krizhevsky, A.; Sutskever, I.; Hinton, G.E. Imagenet classification with deep convolutional neural networks. In Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA, 3–8 December 2012; pp. 1097–1105. [Google Scholar]
- Hinton, G.E.; Salakhutdinov, R.R. Reducing the dimensionality of data with neural networks. Science
**2006**, 313, 504–507. [Google Scholar] [CrossRef] [PubMed] - Chen, Y.; Lin, Z.; Zhao, X.; Wang, G.; Gu, Y. Deep learning-based classification of hyperspectral data. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.
**2014**, 7, 2094–2107. [Google Scholar] [CrossRef] - Adep, R.N.; Vijayan, A.P.; Shetty, A.; Ramesh, H. Performance evaluation of hyperspectral classification algorithms on aviris mineral data. Perspect. Sci.
**2016**, 8, 722–726. [Google Scholar] [CrossRef] - Acquarelli, J.; Marchiori, E.; Buydens, L.; Tran, T.; van Laarhoven, T. Convolutional neural networks and data augmentation for spectral-spatial classification of hyperspectral images. arXiv, 2017; arXiv:1711.05512. [Google Scholar]
- Pan, B.; Shi, Z.; Xu, X. Mugnet: Deep learning for hyperspectral image classification using limited samples. ISPRS J. Photogramm. Remote Sens.
**2018**, 145, 108–119. [Google Scholar] [CrossRef] - Liu, P.; Zhang, H.; Eom, K.B. Active deep learning for classification of hyperspectral images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.
**2017**, 10, 712–724. [Google Scholar] [CrossRef] - Wei, W.; Zhang, J.; Zhang, L.; Tian, C.; Zhang, Y. Deep cube-pair network for hyperspectral imagery classification. Remote Sens.
**2018**, 10, 783. [Google Scholar] [CrossRef] - Tao, Y.; Xu, M.; Lu, Z.; Zhong, Y. Densenet-based depth-width double reinforced deep learning neural network for high-resolution remote sensing image per-pixel classification. Remote Sens.
**2018**, 10, 779. [Google Scholar] [CrossRef] - R Core Team. R: A Language and Environment for Statistical Computing. Available online: https://www.scirp.org/(S(351jmbntvnsjt1aadkposzje))/reference/ReferencesPapers.aspx?ReferenceID=2144573 (accessed on 31 October 2017).
- Candel, A.; Parmar, V.; LeDell, E.; Arora, A. Deep Learning with H2O; H2O.ai Inc.: Mountain View, CA, USA, 2016. [Google Scholar]
- Liaw, A.; Wiener, M. Classification and regression by randomforest. R News
**2002**, 2, 18–22. [Google Scholar] - Hausser, J.; Strimmer, K.; Strimmer, M.K. Package ‘Entropy’. Available online: http://strimmerlab.org/software/entropy/ (accessed on 19 February 2015).
- Shannon, C.E. A mathematical theory of communication. ACM SIGMOBILE Mob. Comput. Commun. Rev.
**2001**, 5, 3–55. [Google Scholar] [CrossRef] - Hinton, G.; Deng, L.; Yu, D.; Dahl, G.E.; Mohamed, A.-R.; Jaitly, N.; Senior, A.; Vanhoucke, V.; Nguyen, P.; Sainath, T.N. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Process. Mag.
**2012**, 29, 82–97. [Google Scholar] [CrossRef] - Li, L.; Zhao, Y.; Jiang, D.; Zhang, Y.; Wang, F.; Gonzalez, I.; Valentin, E.; Sahli, H. Hybrid deep neural network--hidden markov model (dnn-hmm) based speech emotion recognition. In Proceedings of the 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), Geneva, Switzerland, 2–5 September 2013; pp. 312–317. [Google Scholar]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature
**2015**, 521, 436–444. [Google Scholar] [CrossRef] - Rumelhart, D.E.; Hinton, G.E.; Williams, R.J. Learning representations by back-propagating errors. Cogn. Model.
**1988**, 5, 1. [Google Scholar] [CrossRef] - Ham, J.; Chen, Y.; Crawford, M.M.; Ghosh, J. Investigation of the random forest framework for classification of hyperspectral data. IEEE Trans. Geosci. Remote Sens.
**2005**, 43, 492–501. [Google Scholar] [CrossRef] [Green Version] - Belgiu, M.; Drăguţ, L. Random forest in remote sensing: A review of applications and future directions. ISPRS J. Photogramm. Remote Sens.
**2016**, 114, 24–31. [Google Scholar] [CrossRef] - Mahapatra, D. Analyzing training information from random forests for improved image segmentation. IEEE Trans. Image Process.
**2014**, 23, 1504–1512. [Google Scholar] [CrossRef] - Yu, S.; Jia, S.; Xu, C. Convolutional neural networks for hyperspectral image classification. Neurocomputing
**2017**, 219, 88–98. [Google Scholar] [CrossRef] - Kianisarkaleh, A.; Ghassemian, H. Nonparametric feature extraction for classification of hyperspectral images with limited training samples. ISPRS J. Photogramm. Remote Sens.
**2016**, 119, 64–78. [Google Scholar] [CrossRef] - Luo, F.; Huang, H.; Duan, Y.; Liu, J.; Liao, Y. Local geometric structure feature for dimensionality reduction of hyperspectral imagery. Remote Sens.
**2017**, 9, 790. [Google Scholar] [CrossRef] - Li, L.; Ge, H.; Gao, J. A spectral-spatial kernel-based method for hyperspectral imagery classification. Adv. Space Res.
**2017**, 59, 954–967. [Google Scholar] [CrossRef] - Yang, C.; Tan, Y.; Bruzzone, L.; Lu, L.; Guan, R. Discriminative feature metric learning in the affinity propagation model for band selection in hyperspectral images. Remote Sens.
**2017**, 9, 782. [Google Scholar] [CrossRef] - Guo, C.; Pleiss, G.; Sun, Y.; Weinberger, K.Q. On calibration of modern neural networks. arXiv, 2017; arXiv:1706.04599. [Google Scholar]
- Zhu, D.; Yao, H.; Jiang, B.; Yu, P. Negative log likelihood ratio loss for deep neural network classification. arXiv, 2018; arXiv:1804.10690. [Google Scholar]
- Lu, D.; Weng, Q. A survey of image classification methods and techniques for improving classification performance. Int. J. Remote Sens.
**2007**, 28, 823–870. [Google Scholar] [CrossRef] [Green Version] - Foody, G.M.; Mathur, A. The use of small training sets containing mixed pixels for accurate hard image classification: Training on mixed spectral responses for classification by a svm. Remote Sens. Environ.
**2006**, 103, 179–189. [Google Scholar] [CrossRef] - Pal, M.; Foody, G.M. Feature selection for classification of hyperspectral data by svm. IEEE Trans. Geosci. Remote Sens.
**2010**, 48, 2297–2307. [Google Scholar] [CrossRef] - Li, C.; Wang, J.; Wang, L.; Hu, L.; Gong, P. Comparison of classification algorithms and training sample sizes in urban land classification with landsat thematic mapper imagery. Remote Sens.
**2014**, 6, 964–983. [Google Scholar] [CrossRef] - Heydari, S.S.; Mountrakis, G. Effect of classifier selection, reference sample size, reference class distribution and scene heterogeneity in per-pixel classification accuracy using 26 landsat sites. Remote Sens. Environ.
**2018**, 204, 648–658. [Google Scholar] [CrossRef] - Zhang, L.; Zhang, L.; Du, B. Deep learning for remote sensing data: A technical tutorial on the state of the art. IEEE Geosci. Remote Sens. Mag.
**2016**, 4, 22–40. [Google Scholar] [CrossRef] - Canters, F. Evaluating the uncertainty of area estimates derived from fuuy land-cover classification. Photogramm. Eng. Remote Sens.
**1997**, 63, 403–414. [Google Scholar]

**Figure 2.**The best-case scenarios for every pixel representing low uncertainty (

**a**) versus the worst-case scenario denoting high uncertainty (

**b**). The other instances would be intermediate states of these two.

**Figure 3.**Ground truth data of two datasets including the Salinas (

**a**) and the Indian Pines (

**b**). The bottom images represent the location of the train and test data for the Salinas (

**c**) and the Indian Pines (

**d**).

**Figure 4.**Results of uncertainty assessment for DNN (

**a**) and RF (

**b**) using different portions of training sample (S, in %) and mode of correct/incorrect classified test data for the Salinas dataset. The estimated overall accuracy (OA, in %) of the whole classification scheme is also demonstrated for each training sample.

**Figure 5.**The estimated RMSE values of uncertainty assessment for test datasets (y-axis) where the algorithm is trained with different portions of the training sample (x-axis) of Salinas dataset. Dashed lines represent the minimum and maximum RMSE values for each sample size achieved in five consecutive simulation runs.

**Figure 6.**Class entropy/uncertainty (x-axis) versus class accuracy (y-axis) plots of Salinas dataset using DNN (

**a**, left) and RF (

**b**, right) algorithms observed by applying 50% of training data. The bubble sizes represent the frequency of land use class labels while bigger bubbles indicate the higher frequency and vice versa.

**Figure 7.**Results of uncertainty assessment for DNN (

**a**) and RF (

**b**) using different portions of training sample (S, in %) and mode of correct/incorrect classified test data for the Indian Pines dataset. The estimated overall accuracy (OA, in %) of the whole classification scheme is also demonstrated for each training sample.

**Figure 8.**The estimated RMSE values of uncertainty assessment for test datasets (y-axis) where the algorithm is trained with different portions of training sample (x-axis) of Indian Pines dataset. Dashed lines represent the minimum and maximum RMSE values for each sample size achieved in five consecutive simulation runs.

**Figure 9.**Class entropy/uncertainty (x-axis) versus class accuracy (y-axis) plots of Indian Pines dataset using DNN (

**a**, left) and RF (

**b**, right) algorithms observed by applying 50% of training sample size. The bubble sizes represent the frequency of land use class labels while bigger bubbles indicate the higher frequency and vice versa.

**Table 1.**The optimised hyper-parameters of DNN and RF using 5-fold cross-validation data for uncertainty assessment.

Algorithm | Hyper-Parameter | Description | Salinas | Indian Pines |
---|---|---|---|---|

DNN | hidden | Hidden layer sizes | (100, 100) | (200, 200) |

DNN | epoch | How many times the dataset should be iterated (streamed) | 300 | 300 |

DNN | activation | Activation function for non-linear transformation. | “Maxout” | “Maxout” |

DNN | stopping metric | A metric that is used as a stopping criterion | “RMSE” | “RMSE” |

DNN | l1 | Only allows strong values to survives | 0.0001 | 0.0001 |

DNN | l2 | Prevents any single weight from getting too big | 0.001 | 0.001 |

DNN | epsilon | Prevents getting stuck in local optima | 1 × e^{−10} | 1 × e^{−10} |

RF | ntree | Number of trees to grow | 100 | 100 |

RF | mtry | Number of variables available for splitting at each tree node | 14 | 15 |

Best-Case Scenarios | e | o | RMSE | Worst-Case Scenarios | e | o | RMSE |
---|---|---|---|---|---|---|---|

Positive | 0 | 0 | 0 | Positive | 0 | 1 | 1 |

Negative | 1 | 1 | 0 | Negative | 1 | 0 | 1 |

Dataset | Sensor | Total Bands | Excluded Bands | Number of Classes | Dimension | Resolution |
---|---|---|---|---|---|---|

Salinas | AVIRIS | 224 | 20 | 16 | 512 × 217 | 20 metre |

Indian Pines | AVIRIS | 224 | 24 | 16 | 145 × 145 | 20 metre |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Shadman Roodposhti, M.; Aryal, J.; Lucieer, A.; Bryan, B.A.
Uncertainty Assessment of Hyperspectral Image Classification: Deep Learning vs. Random Forest. *Entropy* **2019**, *21*, 78.
https://doi.org/10.3390/e21010078

**AMA Style**

Shadman Roodposhti M, Aryal J, Lucieer A, Bryan BA.
Uncertainty Assessment of Hyperspectral Image Classification: Deep Learning vs. Random Forest. *Entropy*. 2019; 21(1):78.
https://doi.org/10.3390/e21010078

**Chicago/Turabian Style**

Shadman Roodposhti, Majid, Jagannath Aryal, Arko Lucieer, and Brett A. Bryan.
2019. "Uncertainty Assessment of Hyperspectral Image Classification: Deep Learning vs. Random Forest" *Entropy* 21, no. 1: 78.
https://doi.org/10.3390/e21010078