Next Article in Journal
Development and Application of a Rapid Scan Technique for Emissivity Measurements of Cooled-Down Molten Materials
Previous Article in Journal
Design of an Electrochemical Genosensor for BDNF Gene Polymorphism Sequence Detection Using an Enzymatically Labeled DNA Probe
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Proceeding Paper

Enhancing the Thermal Inspection of Buildings Using Texture Analysis †

by
Setayesh Hesam
*,
Reza Khoshkbary Rezayiye
,
Clemente Ibarra-Castanedo
and
Xavier Maldague
Department of Electrical and Computer Engineering, Universite Laval, 2325 Rue de l’Universite, Quebec, QC G1V 0A6, Canada
*
Author to whom correspondence should be addressed.
Presented at the 17th International Workshop on Advanced Infrared Technology and Applications, Venice, Italy, 10–13 September 2023.
Eng. Proc. 2023, 51(1), 9; https://doi.org/10.3390/engproc2023051009
Published: 26 October 2023

Abstract

:
The thermographic inspection of buildings is a powerful and non-invasive method for monitoring and diagnosing building performance and structural integrity. It can effectively detect moisture, evaluate heat loss, and assess building’s roofs. The early detection of problems in a building allows owners to fix issues before they become more severe and costly. One of the challenges in automating thermal analysis for building inspections comes in the form of distinguishing between different surfaces. This paper presents an automated process pipeline using coupled thermal and visible images for building inspections to assist inspectors in adapting strategies and methods to discriminate the thermal signatures between different kinds of surfaces. A deep learning method is employed to segment visible images texturally. Thermal images are then analyzed based on the resulting segmentation. Moreover, a multi-modal dataset is introduced, presenting coupled thermal and visible images acquired during multiple building inspections.

1. Introduction

The building industry contributes significantly to global energy consumption and greenhouse gas emissions due to various inefficiencies in structures [1]. Regular inspections are critical but face scalability issues due to their time-consuming nature. The adoption of technologies like thermal cameras and laser scanners can help to automate inspections, but they present limitations [2]. Multi-modal data acquisition strategies that combine different forms of sensor data have been introduced to overcome these challenges [3]. Despite their potential, the construction industry’s adoption of these multi-modal platforms has been limited [4]. This paper proposes a solution by integrating multi-modal data acquisition strategies with texture analysis for energy management-focused building inspections. We hope that this paper can help to enhance building inspection processes and reduce the industry’s energy footprint [5].

2. Background

Thermographic inspection, a fast and non-invasive methodology, has been instrumental in monitoring and diagnosing building performance, significantly aiding the early detection and rectification of structural issues. However, automating thermal analysis for building inspections presents distinct challenges, primarily with respect to differentiating between various surfaces. Research by Pozzer et al. and Mirzabeigi et al. has advanced the field, offering promising methods for enhancing our understanding of building energy performance and detecting structural anomalies using integrated data, infrared thermographic images, and drone thermography [6]. Further expanding the spectrum of techniques, Singh et al. provided an in-depth discussion on multimodal inspection in industrial settings, underscoring the importance of such techniques in ensuring equipment safety and reliability. Novel contributions like the multi-modal mapping unit by Mascarich et al. for underground tunnel exploration and the thermal–optical imaging and texture analysis algorithm by Nooralishahi et al. exemplify the ongoing efforts to innovate within the field [7,8,9]. In alignment with these innovative approaches, this paper presents an automated process pipeline that employs deep learning for the textural segmentation of visible images, followed by thermal analysis of the segmented regions. Furthermore, this paper introduces a novel multi-modal dataset composed of coupled thermal and visible images collected over several building inspections, marking a significant contribution to the automation and optimization of building inspections.

3. Methodology

3.1. Image Registration

This study employs a two-stage approach, with image registration forming the crucial first stage. Image registration aligns two or more images of the same scene, facilitating an effective analysis of the corresponding regions across these images. This section explores our usage of both manual and automatic image registration techniques.

3.1.1. Manual Image Registration

Manual image registration, or point-to-point registration, necessitates the manual identification of the corresponding points across the images that need alignment. A geometric transformation is applied to one image to align it with the other. Utilized as an initial benchmark for the automatic methods, this approach provides flexibility in handling diverse image scenarios despite being time-consuming and error-prone.

3.1.2. Automatic Image Registration

To address the limitations of the manual method, automatic image registration is preposed herein and specifically leverages homography matrices. Homography involves transformation mapping the points in one image to corresponding points in another while accommodating for changes in scale, rotation, translation, and perspective. A homography matrix based on the feature matching between images was employed to warp and align the images. This method provided a rapid, accurate alignment process, essential for efficiently processing large multi-modal datasets. In the subsequent section, the second stage of the methodology, which involved the application of deep learning techniques to the registered images for segmentation and analysis, is discussed.

3.2. Image Segmentation

3.2.1. Segmentation of Visible Images

For visible image segmentation, a deep learning approach was used. Specifically, the UNET++ architecture, an advanced variant of the classic UNET, was employed, along with the Resnet 152 as the backbone. This combination delivered an effective model capable of predicting and outputting a mask that highlighted different building components. Each segment of the mask corresponded to a different part of the building, such as windows, walls, or roofs. This mask served as an invaluable guide to the exact location and context of various building components, paving the way for an efficient analysis of the corresponding thermal images.

3.2.2. Segmentation of Thermal Images

The segmentation of thermal images leveraged an unsupervised image segmentation approach based on the method developed by Kanezaki [10]. As thermal images do not readily lend themselves to traditional segmentation techniques due to the nature of the data they contain, this unsupervised method proved particularly useful. It allowed for the segmentation of thermal images without prior training on specific building components. The segmented thermal images, when combined with the segmented visible images, provided a comprehensive understanding of the building’s thermal properties, thus enabling effective defect detection and energy performance analysis.

4. Results and Discussion

4.1. Image Registration

The results confirmed that automatic image registration outperforms the manual approach. Its capacity to quickly and accurately align thermal and visible images was integral to establishing an efficient and scalable automated pipeline for building inspections.

4.2. Segmentation of Visible Images

The effectiveness of UNet ++ with Resnet 152 for visible image segmentation was validated by the obtained results. As displayed in Figure 1, the deep learning model predicted a mask that closely mirrored the ground truth, effectively pinpointing the location of various building components. Whether for cement walls, windows, window frames, or metal parts, the method showed the potential to segment and identify different building parts accurately.

4.3. Segmentation of Thermal Images

Similarly, the unsupervised image segmentation method produced promising results on thermal images. It effectively highlighted regions with elevated thermal leakage, underscoring the efficacy of this approach for analyzing thermal properties. Figure 2b shows the results of unsupervised image segmentation on the thermal images. The averages for the IoU scores and F1-scores achieved by our model across various building components were 0.83 and 0.89, respectively.

4.4. Fusion of Results

Figure 2c shows the fusion results regarding thermal image segmentation and visible image segmentation. Our methodology’s core strength lies in merging segmented visible and thermal images, leading to the superior prediction of thermal leakage locations across different building parts. This integrated approach, which involves utilizing various image processing and machine learning methods, offers an efficient, automated pipeline for enhanced building inspections. Our promising results suggest the technique’s vast potential for broader applications in support of energy-efficient building management.

5. Conclusions

This paper introduced a pioneering method that combines multi-modal data acquisition, deep learning, and unsupervised image segmentation for enhanced building inspection, focusing on thermal leakage detection. The implemented methodology yielded promising results, displaying its capacity to automate and optimize building inspections. Key advancements include efficient automatic image registration and the successful segmentation of visible and thermal images, leading to accurate thermal leakage localization on specific building components. Despite its achievements, further research is required to refine this methodology and explore its potential limitations. Our future studies will incorporate varied deep learning architectures and segmentation methods, along with diverse building types and environmental conditions, to bolster the model’s robustness. In conclusion, this study signifies a substantial stride towards an energy-efficient future by revolutionizing building inspections through precise, automated, and data-driven techniques, holding extensive implications for both the building industry and broader global energy framework.

Author Contributions

Conceptualization, S.H., X.M. and C.I.-C.; method-ology, S.H.; software, S.H.; validation, S.H. and R.K.R.; formal analysis, S.H. and C.I.-C.; investigation, S.H.; resources, X.M. and C.I.-C.; data curation, X.M.; writing—original draft preparation, S.H.; writing—review and editing, S.H. and R.K.R.; visualization, S.H. and R.K.R.; supervision, X.M. and C.I.-C.; project administration, X.M.; funding acquisition, X.M. and C.I.-C. All authors have read and agreed to the published version of the manuscript.

Funding

Our research was supported by the Natural Sciences and Engineering Council of Canada (NSERC), CREATE oNDuTy Program (funding ref. number 496439-2017), the Canada Research Chair in Multipolar Infrared Vision (MIVIM), Department of Civil Engineering from Université Laval, and the MITACS Accelerate Program.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Shariq, M.H.; Hughes, B.R. Revolutionising building inspection techniques to meet large-scale energy demands: A review of the state-of-the-art. Renew. Sustain. Energy Rev. 2020, 130, 109979. [Google Scholar] [CrossRef]
  2. Kula, J.; Tunák, M.; Tunak, M.; Linka, A. Inspection System of Fabric Based on Texture Segmentation Utilizing Gabor Filters. 2014. Available online: https://www.researchgate.net/publication/260487866 (accessed on 1 September 2010).
  3. Preda, A.; Scurtu, I.C. Thermal image building inspection for heat loss diagnosis. J. Phys. Conf. Ser. 2019, 1297, 012004. [Google Scholar] [CrossRef]
  4. Jordanov, I.N.; Petrov, N.; Jordanov, I. Unsupervised Texture Image Classification Using Self-Organizing Maps. 2012. Available online: https://www.researchgate.net/publication/268434714 (accessed on 1 April 2012).
  5. Vidal-Calleja, T.; Miro, J.V.; Martin, F.; Lingnau, D.C.; Russell, D.E. Automatic detection and verification of pipeline construction features with multi-modal data. In Proceedings of the IEEE International Conference on Intelligent Robots and Systems, Chicago, IL, USA, 14–18 September 2014; pp. 3116–3122. [Google Scholar] [CrossRef]
  6. Hoang, N.D.; Tran, V.D. Image Processing-Based Detection of Pipe Corrosion Using Texture Analysis and Metaheuristic-Optimized Machine Learning Approach. Comput. Intell. Neurosci. 2019, 2019, 809721. [Google Scholar] [CrossRef] [PubMed]
  7. Chow, J.K.; Liu, K.-F.; Tan, P.S.; Su, Z.; Wu, J.; Li, Z.; Wang, Y.-H. Automated defect inspection of concrete structures. Autom. Constr. 2021, 132, 103959. [Google Scholar] [CrossRef]
  8. Hoang, N.D.; Nguyen, Q.L.; Tran, X.L.; Andrea, M. Automatic Detection of Concrete Spalling Using Piecewise Linear Stochastic Gradient Descent Logistic Regression and Image Texture Analysis. Complexity 2019, 2019, 5910625. [Google Scholar] [CrossRef]
  9. Zheng, H.; Gao, G.; Zhong, X.; Zhao, L. Monitoring and diagnostics of buildings’ heat loss based on 3D IR model of multiple buildings. Energy Build 2022, 259, 111889. [Google Scholar] [CrossRef]
  10. Kanezaki, A. Unsupervised Image Segmentation by Backpropagation; IEEE: Piscataway, NJ, USA, 2018; p. 7020. [Google Scholar] [CrossRef]
Figure 1. Prediction results of UNet ++ network. (a) Input image; (b) ground truth; (c) predicted mask.
Figure 1. Prediction results of UNet ++ network. (a) Input image; (b) ground truth; (c) predicted mask.
Engproc 51 00009 g001
Figure 2. Unsupervised image segmentation results on the thermal images. (a) Thermal image; (b) segmentation results; (c) fusion results.
Figure 2. Unsupervised image segmentation results on the thermal images. (a) Thermal image; (b) segmentation results; (c) fusion results.
Engproc 51 00009 g002
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Hesam, S.; Rezayiye, R.K.; Ibarra-Castanedo, C.; Maldague, X. Enhancing the Thermal Inspection of Buildings Using Texture Analysis. Eng. Proc. 2023, 51, 9. https://doi.org/10.3390/engproc2023051009

AMA Style

Hesam S, Rezayiye RK, Ibarra-Castanedo C, Maldague X. Enhancing the Thermal Inspection of Buildings Using Texture Analysis. Engineering Proceedings. 2023; 51(1):9. https://doi.org/10.3390/engproc2023051009

Chicago/Turabian Style

Hesam, Setayesh, Reza Khoshkbary Rezayiye, Clemente Ibarra-Castanedo, and Xavier Maldague. 2023. "Enhancing the Thermal Inspection of Buildings Using Texture Analysis" Engineering Proceedings 51, no. 1: 9. https://doi.org/10.3390/engproc2023051009

Article Metrics

Back to TopTop