A Contactless Measuring Method of Skin Temperature based on the Skin Sensitivity Index and Deep Learning

Featured Application

The NISDL method proposed in this paper can be used for real time contactless measuring of human skin temperature, which reflects human body thermal comfort status and can be used for control HVAC devices.

Abstract

In human-centered intelligent building, real-time measurements of human thermal comfort play critical roles and supply feedback control signals for building heating, ventilation, and air conditioning (HVAC) systems. Due to the challenges of intra- and inter-individual differences and skin subtleness variations, there has not been any satisfactory solution for thermal comfort measurements until now. In this paper, a contactless measuring method based on a skin sensitivity index and deep learning (NISDL) was proposed to measure real-time skin temperature. A new evaluating index, named the skin sensitivity index (SSI), was defined to overcome individual differences and skin subtleness variations. To illustrate the effectiveness of SSI proposed, a two multi-layers deep learning framework (NISDL method I and II) was designed and the DenseNet201 was used for extracting features from skin images. The partly personal saturation temperature (NIPST) algorithm was use for algorithm comparisons. Another deep learning algorithm without SSI (DL) was also generated for algorithm comparisons. Finally, a total of 1.44 million image data was used for algorithm validation. The results show that 55.62% and 52.25% error values (NISDL method I, II) are scattered at (0 °C, 0.25 °C), and the same error intervals distribution of NIPST is 35.39%.

1. Introduction

Higher economic growth drives increasing energy consumption, and 50% of housing consumption is generated by heating, ventilation and air conditioning (HVAC) systems [1,2]. Furthermore, one of the most important reasons for energy waste is that the actual thermal requirements of indoor occupants are ignored, with the result that overheating and overcooling occur often. Fortunately, real-time thermal comfort perception can provide useful signals to HVAC systems for achieving energy saving and human-centered intelligent control. Therefore, many researchers have been studying thermal comfort measurements for indoor environments in recent decades. Many methods were generated, including the questionnaire survey method [3,4,5], environmental measurement method [6,7] and contact measuring method of human body physiological parameters [8,9,10,11,12,13,14,15,16,17,18]. In recent years, the semi-contact measuring method [19,20] and contactless measuring method [21,22,23] for human body physiological parameters were also generated. For example, in references [19,20], an infrared sensor was fixed on the frame of eyeglasses in order to measure skin temperature. In reference [21], a normal vision sensor was also used for measuring skin temperature and two non-linear models were trained. In references [22,23], Kinect was used for recognizing human poses or indoor locations, and then human thermal comfort and dynamic metabolism were estimated, respectively. All these methods are meaningful attempts. However, due to the challenges of measuring thermal comfort which are (1) skin subtleness variation [24], (2) inter-individual differences [14,25] and (3) temporal intra-individual differences [14,26], there is still no satisfactory method for perceiving human thermal comfort.

To overcome the aforementioned challenges, the skin sensitivity index (SSI) was defined in this paper. The SSI is strongly related to skin temperature. A contactless measuring method of skin temperature based on SSI and Deep Learning was proposed, hereinafter referred to as NISDL. Two different deep learning methods of NISDL have been designed and trained, respectively, which are NISDL methods I and II. The main difference between them is that the location of SSI participation in the neural network training is not the same. A total of 1.44 million images were collected for 16 Asian female subjects, and this ‘big data’ was used for algorithm validation.

The main contributions of this paper are:

(1)

The skin sensitivity index (SSI) was proposed for describing individual sensitivity of thermal comfort, and the index was combined with skin images for deep learning network training.

(2)

A novel contactless measuring algorithm (NISDL) based on SSI was proposed, with two different frameworks of NISDL having been designed for real-time thermal comfort measurement.

(3)

A deep learning algorithm without SSI was also generated and trained. Two comparisons were made: (1) comparison between data-driven methods (deep learning) and model-driven methods (linear models); (2) comparison of measuring effects in the case of SSI participation in training and non-participation in training.

The rest of this paper is organized as follows. Section 2 introduces the related work about thermal comfort. In Section 3, the research methods, including SSI computation, subjective experiments and NISDL methods, are introduced. The results and discussion are shown in Section 4 and Section 5. Finally, Section 6 gives the conclusion.

2. Related Work

Since the 1970s, Fanger has explored human thermal comfort and conducted many kinds of subjective experiments. Based on this, he eventually established what is known as Fanger’s theory [27]. From then on, many studies about thermal comfort were carried out.

Questionnaire surveys are good as a method to understand the inner feeling of an occupant. With the development of the internet, online questionnaire surveys can also be generated [3,4]. However, it is inconvenient and also difficult to guarantee that occupants will continue to give feedback based on their personal thermal feelings [5]. Therefore, the environment measurement method was also adopted in the building industry [6]. With this kind of method, some objective parameters, such as indoor temperature, airflow and humidity, are often measured. Unfortunately, the goal of the environment measurement method is to meet the thermal comfort needs of a majority of indoor occupants. Therefore, the thermal feelings of a minority were ignored. To overcome this drawback, a kind of nonlinear autoregressive network, still belonging to the environment measurement method, was generated to predict indoor temperature [7]. In fact, human thermal comfort is complicated, and with constant indoor parameters it is difficult to meet each individual’s requirements for thermal comfort. As such, some researchers study physiological measurement methods, including the contact measuring method, semi-contact measuring method and contactless measuring method.

For the contact measuring method, skin temperature and heart rate are usually the measured parameters. Wang and Nakayama [8,9] made early attempts at measuring the skin temperature around the human body. Liu [10] conducted subjective experiments and a total of 22 subjects were invited. The data, being local skin temperatures and electrocardiograms, were collected. Based on these collected data, 26 measuring methods of mean skin temperature were assessed. Takada [11] presented a multiple regression equation to predict skin temperature in non-steady state. The multiple regression equation was considered as a function of mean skin temperature. Wrist skin temperatures and upper extremity skin temperatures were also adopted to estimate human thermal sensation, respectively [12,13]. Chaudhuri [14] presented a predicted thermal state (PTS) model, and the capture of peripheral skin temperature. Furthermore, body surface area and clothing insulation were used for analyzing inter- and intra-individual differences. As for the thermal comfort study using heart rate, based on physiological experimentation, Yao [15] investigated the relationship between heart rate variation (HRV) and electroencephalograph (EGG). The results show that HRV and EEG are useful for thermal comfort studies, but further data validation is needed. Moreover, Dai, Chaudhuri and Kim [16,17,18] combined machine learning with contact measurement, and they are all meaningful attempts. However, as the linear kernel of SVM was used for predicting human thermal sensation in different experiments, sometimes overfitting can happen.

For the semi-contact measuring method, Ghahramani [19] used an infrared sensor to estimate skin temperature of different face points. The infrared sensor was mounted on the frame of eyeglasses. Based on this, a hidden Markov model was constructed to capture personal thermal sensation [20].

In practical application, contact and semi-contact measurement are both difficult to apply widely. The reason is that an occupant needs to wear a sensor, which is uncomfortable and is also not in line with the goal of human-oriented intelligent buildings. For this reason, a kind of contactless measuring method was studied in reference [21]. Based on vision sensors, Cheng [21] extracted the saturation (S) channel from skin images and constructed two saturation-temperature models to estimate skin temperature. The two models are the contactless measuring method of thermal comfort based on saturation-temperature (NIST) and the contactless measuring method of thermal comfort based on partly saturation-temperature (NIPST). Alan [22] proposed a contactless measuring method based on human poses. A total of 12 poses of thermal comfort were defined and Kinect was adopted to estimate human skeleton and poses. Further, Dziedzic [23] also used Kinect to predict human thermal sensation and dynamic metabolic rates.

With the development of machine learning (ML) and computer vision (CV), some thermal comfort perception methods based on ML and CV were proposed. Support Vector Machine (SVM) are often used for analyzing existing databases (RP-884) and captured environmental parameters [24,28,29]. Further, Peng [26] use unsupervised and supervised learning to predict occupants’ behavior, applied to three types of offices which are single person offices, multi-person offices, and meeting rooms. The results show that the average energy savings for the entire space is 21% in experimental condition. Li [30] proposed a fuzzy model to predict thermal sensation, skin temperature and heart rate considered as objective parameters. For avoiding overheating, Cosma [31] extracted data from multiple local body parts and analyzed them with four kinds of machine learning algorithms, including SVM, Gaussian process classifier (GPC), k-neighbors classifier (KNC) and random forest classifier (RFC).

The kinds of machine learning adopted in references [24,26,28,29,30,31] are traditional algorithms. In recent years, the use of deep neural networks is on the rise [32,33]. In addition, a kind of subtleness magnification technology was presented [34,35]. These provide new directions and opportunities for the measurement of human thermal comfort. Based on this technology, a novel contactless measuring method was generated which will be introduced as follows.

3. Research Methods

3.1. Subjective Physiological Experiments

3.1.1. Subjects Data and Chamber Environments

16 human subjects were invited for experiments and the resulting data volume is 1.44 million images. The experiments were conducted in a chamber with controllable indoor air temperature and relative humidity. The corresponding dry-bulb air temperature is 22.2 ± 0.2 °C and the relative humidity is 36.9 ± 2.5%. The resolution of vision sensor used for capturing video is 1280 × 720. The iButton, model DS192H with uncertainty ±0.125 °C, was used for measuring skin temperature from the back of subject’s hand. All the subjects are Asian females with an average age of 23.9 ± 3.9 years, average weight of 52.2 ± 6.5 kg, and body mass index (BMI) 19.9 ± 2.2 kg/m².

3.1.2. Experimental Procedures

The experiment was conducted during winter in Sweden. There are generally three steps in subjective physiological experiments. (1) Preparation stage: The indoor environment parameters were measured and controlled to a suitable level. When the subjects came into the chamber, they should rest for 10 min for adaptation. At the same time, warm water with constant temperature (45 °C) was prepared. (2) Thermal stimulus: After 10 min adaptation, subjects were asked to immerse hands into the water with 45 °C. The whole thermal stimulus process lasted for 10 min. (3) Big data collection: After 10 min of stimulus, subjects were asked to sit next to the data collection desk and put her pairs of hands under the vision sensor. The back of the hand is faced up and the data is collected for 50 min. At the same time, skin temperature sensor (iButton) was attached to the back of one hand. The corresponding sampling interval is 1 min. It should be noted that, based on piecewise stationary time series analysis [36], linear interpolation was adopted in this paper, and 11 points were interpolated into 1 min for real skin temperature captured by iButton.

3.2. Skin Sensitivity Index

3.2.1. SSI Definition

When human body encounters thermal stimuli, blood circulation will change which will also be reflected in skin’s color and texture. In reference [21], based on the HSV (hue, saturation, value) color space, the S channel was extracted and a linear ST (saturation-temperature) model was established.

T = k_i × S + b_i

(1)

where i denotes subject number, the k reacts to the change rate of skin temperature, and b denotes the intercept. S and T are skin saturation and temperature respectively. In this paper, k is defined as the skin sensitivity index (SSI). SSI is a high weight coefficient in skin temperature changes and SSI reflects the skin sensitivity level to external thermal stimuli.

3.2.2. SSI Computing

Based on subjective physiological experiments, real skin temperature can be obtained by iButton. The images were also collected from subjects’ hands. Therefore, the SSI can be calculated. The steps are as follows: (1) Extracting each frame from captured video; (2) Segment region of interest (ROI); (3) Extracting S channel from ROI images and computing mean values of S for each ROI image; (4) Search SSI value based on real skin temperature and S for each subject.

3.3. NISDL Algorithm

In this paper, considering that SSI is a high weight coefficient for contactless thermal comfort measurement, it will improve the prediction accuracy of skin temperature. Based on SSI, the NISDL algorithm was introduced in this paper. Furthermore, to validate the effectiveness of SSI, two kinds of deep learning frameworks (NISDL method I, II) have been constructed. The main difference between NISDL methods I and II is that the location where the network invokes SSI to participate in the model training is different. The NISDL algorithm constructed is introduced as follows.

3.3.1. Video Pre-Processing

In fact, skin texture variation is subtle and is difficult to perceive. In this paper, for magnifying this kind of subtleness variation, an image subtleness magnification technology known as Euler Video Magnification (EVM) is adopted [34,35]. Based on EVM, let c (x, t) denote skin images which are subtly varied with time t. Suppose that the variation function is formula (2) [34,35].

C (x, t) = F (x + h (t))

(2)

where, h (t) is variation degree, F is a function which constructs the relationship between C (x, t) and h (t). If the skin image C (x, t) is magnified, and the first-order Taylor series expansion can be handed to F (x + h (t)).

C (x, t) = F(x + (1 + ξ) * h(t))

(3)

where ξ is the magnification coefficient which can be set based on practical application. According to Formula (3), only the variation part was magnified to a magnitude of 1 + ξ, while the other part of skin texture is not magnified. Therefore, the invisible texture variation is made to be visible.

It should be noted that de-noise processing should be handled before EMV processing. Further, after the video is magnified, ROI is selected and cropped from each frame. The ROI images are imported into NISDL method I and II for model training.

3.3.2. NISDL Method I

As shown in Figure 1, SSI values are used as input data and imported into the deep learning network at the very beginning. The ROI images were also combined with SSI values in the first step. According to the size of ROI images, the SSI value of each ROI image was expanded into a matrix. The matrix is considered as a channel and combined with the 3 channels of ROI images.

The merged data between ROI images and SSI values above is inputted into four convolution layers, which are used for dimensionality reduction. In NISDL method I, the DenseNet201 [33] is adopted for features extraction. The last two layers of DenseNet201 are not suitable for skin temperature measurement, hence the two layers are removed. The reason for this is that the activation function of the last layers is softmax. Instead of these two layers, an average pooling layer and a fully connected layer are added behind denseNet201. Based on the deep learning networks designed above, n ROI images and n SSI values were inputted into NISDL method I. Therefore, n skin temperatures can be obtained.

3.3.3. NISDL Method II

Figure 2 is the deep learning framework of NISDL method II. In this method, the ROI images and SSI values were processed for features extraction. Subsequently, the two kinds of features were combined in the second half of the whole framework. An average pooling layer and the DenseNet201 (excluding the last two layers) were also used for features extraction of ROI images. For SSI values, a convolution layer and an average pooling layer were adopted for feature extraction and dimensionality reduction. After features combination, three fully connected layers were constructed in NISDL method II. Therefore, the skin temperature can be obtained. The algorithm in detail, including NISDL method I and II, is shown in

Table 1. Contactless measuring method of skin temperature based on skin sensitivity index.

Algorithm: the NISDL algorithm

Output: NISDL model (*.h5), skin temperature (℃)

Step:

Video pre-processing (1) De-noise and handle subtleness magnification for captured video. (2) Magnification coefficient ξ is 10 (formula (3)). (3) Extracting region of interest (ROI) from each frame of video, the size is 150×150. Making label (1) Making numerical interpolation for skin temperatures captured by iButton. (2) Uniform interpolation is adopted, plus 11 points/min. (3) Establish a correspondence table between ROI images and skin temperatures (after interpolation). Algorithm training (1) Commonality between NISDL method I and II 1) Training set and test set ratio: 12: 4. 2) Validation set: 500 images. 3) During network training, 32 images/batch, epoch is 8. 4) Training ~30000 images, validate once. 5) Activation function: ReLU (2) NISDL method I 1) SSI values were concatenated with ROI images in the first steps. 2) Convolutional kernel: 1 × 1. (3) NISDL method II 1) Features are extracted from SSI values and ROI images, respectively. 2) Concatenating the two kind of features in the second half of network. 3) Convolutional kernel: 3 × 1. Optimizing model parameters

.

3.3.4. Evaluation Metric

For assessing NISDL algorithm constructed in this paper, the absolute error is adopted.

Error = |T_p(i) – T_r(i)| i = 1, 2, 3, …

(4)

where T_p (i) is the prediction values of skin temperature and obtained from the proposed NISDL algorithm. T_r (i) is the real value of skin temperature and captured by iButton. The parameter i denotes the particular ROI image.

3.3.5. Algorithms for Comparison

Two algorithms are used for comparison in this paper: (1) DL algorithm. The commonality between NISDL method I and II is that they all use SSI for model training. For validating the effectiveness of NISDL algorithm (with SSI), we remove the SSI and corresponding hidden layers for SSI features extraction from NISDL method II, so that it will be another deep learning network (without SSI) and is named the DL algorithm hereinafter. (2) NIPST algorithm. DL algorithm, NISDL method I and II are all nonlinear methods and data driven methods with deep learning networks. For further validating NISDL method I and II, the NIPST algorithm is also used for algorithm comparison, which is a linear and model driven method.

4. Results

16 subjects were invited for subjective physiological experiments and a total of 1.44 million images were captured. Based on this, the NISDL algorithm was validated and compared with the NIPST algorithm and the DL algorithm.

4.1. Hardware Parameters

For this paper, a computer with a GPU was used for images processing and algorithm validation. The GPU is GeForce GTX TITAN X, the CPU is Intel core i5-4460 CPU@3.2Ghz X 4, the RAM is 16G and the word size is 64bit.

4.2. Training of NISDL method I

The size of ROI images are n × 150 × 150 × 3, and the size of expanded SSI vlues are n × 150 × 150 × 1. The SSI matrix was considered as a channel and concatenated with ROI images, so that the result is n × 150 × 150 × 4. The activation function of four convolution layers, shown in Figure 1, are Rectified Linear Units (ReLU) and the size of convolution kernel is 1 × 1. DenseNet201 was used for feature extraction and its output is a matrix with size of n × 4 × 4 × 1920. Based on this, two hidden layers are constructed and the size of the last layer, being a fully connected layer, is 1920 × n.

4.3. Training of NISDL Method II

The size of the expanded SSI matrix is n × 1920 × 1, which differs from that of NISDL method I. The corresponding convolution kernel is 3 × 1. The features of SSI were extracted by two hidden layers, and the features of ROI images were extracted by DenseNet201. As shown in Figure 2, in the second half of framework, the two kinds of features are concatenated. In order to ensure that the SSI features have a suitable influence on network training (moderate, not too big or too small), the size of ROI image features is set as n × 1920, and size of SSI features is set as n × 640 (triple relationship). Finally, the size of last three hidden layers is 2560 × 1024, 1024 × 512 and 512 × 1, respectively.

4.4. Commonality between NISDL Method I and II

During network training, the same parameters of NISDL method I and II are shown as follows. Based on data of 1.44 million images, the ratio of training set and test set is 12:4, the number of validation set is 500. The epoch is 8, which means that the training set was trained 8 times. The input data batch is 32. When the error of validation set is less than 0.46 °C, the corresponding model (*.h5) will be saved. Further, when 30,0000 images of training set were trained, the corresponding model (*.h5) will also be saved. After the generation of the model, the test set images were inputted into generated model, so that the prediction values of skin temperatures could be obtained.

4.5. Quantitative Comparison

The prediction values of skin temperature are shown in Figure 3. The set of values obtained from iButton is ground truth. The corresponding error statistics, including mean, median, are shown in Figure 4, which is a box-whisker plot. The mean values of NISPT, DL, NISDL method I and II in °C are 0.579, 0.359, 0.335 and 0.265, respectively. In addition, the median values of them in °C are 0.343, 0.309, 0.238 and 0.228, respectively. It was shown that deep learning methods (DL, NISDL method I and II) are all better than the nonlinear model (NIPST) and further that the method with SSI (NISDL method I and II) is better than the method without SSI (DL).

In this paper, the error distributions are given in Figure 5 and

Table 2. Absolute error distribution.

Absolute Error (℃)	NIPST (Baseline, %)	DL (Baseline, %)	NISDL Method I (Figure 1, %)	NISDL Method II (Figure 2, %)
[0, 0.25)	35.39	37.64	52.25	55.62
[0.25, 0.5)	24.16	35.96	28.09	30.34
[0.5, 0.75)	5.06	19.10	10.11	11.80
[0.75, 1.0)	11.24	5.62	3.37	2.250
[>1.0)	24.16	1.69	6.18	0

. The errors of DL, NISDL method I and II are mainly concentrated in the range of 0 °C and 0.75 °C. NISDL is better than DL, because two error percentages of NISDL corresponding to [0, 0.25) are 52.25% and 55.62%. In addition, the error percentages of NISDL corresponding to [0.25, 0.5) and [0.5, 0.75) are less than that of DL. The error percentage of NIPST is increased from the interval of [0.75, 1), meaning that the performance of NIPST is worse than DL and NISDL methods I and II.

5. Discussion

5.1. Situation of Overcoming Challenges

NISDL has overcome the three challenges mentioned in Section 1 to some extent. A kind of subtleness magnification technology, which is Euler Video Magnification (EVM), was used for magnifying the skin texture variation, so that the challenges ‘(1)’ given in section 1 can be overcome. For overcoming challenges ‘(2)’ which is inter-individual difference, the skin sensitivity index (SSI) was proposed and SSI is related with skin saturation. Figure 4 and Figure 5 shows that the performance of NISDL algorithm with SSI is better than that of DL without SSI. In practical application, skin images will be captured in real-time (30 frames/s), the skin temperature variation can always be obtained. Therefore, challenges ‘(3)’ proposed in Section 1 can be overcome. Furthermore, piecewise stationary time series analysis was adopted in this paper for overcoming challenges ‘(3)’. Considering operability, the breakpoint interval of piecewise stationary signal is set to 5 s, supposing, e.g., that the skin temperature has a constant value during 5 s.

5.2. The Deep Learning Framework

In this paper, NISDL method I and II all belong to the deep learning method. In addition, DL generated for algorithm comparison is also a deep learning method. The error distribution comparison is shown in Figure 4 and Figure 5 and the performance of NISDL II is encouraged. From the perspective of deep learning, the main reason for this is that big data is adopted.

5.3. The Proposed SSI

Although NISDL and DL are all better than NIPST, there are still a big gap between NISDL and DL. Figure 4 and Figure 5 and Table 2 also show that NISDL is better than DL. The main reason is that SSI is used in NISDL. The NISDL method II is also shown to be better than NISDL method I. Further, when SSI features are extracted and concatenated with ROI images features in the second half of network, the performance will be better.

5.4. Reasons of Designing Two Frameworks for NISDL

Some researchers may ask, why do we design NISDL methods I and II together? The main reason is that we want to extensively confirm the effectiveness of SSI. In this paper, SSI participates in network training from different locations, and the results are all good. When the SSI values are removed from network (DL), the corresponding performance decreased significantly. Based on this, we can know that SSI is helpful for predicting skin temperature through deep learning networks.

5.5. Practical Application

Some researchers may argue that the method proposed in this paper still cannot be applied in practice right now. In fact, a method is always being gradually improved. For example, the NISDL proposed in this paper is better than NIPST which was proposed in 2017. In addition, when more diverse data is captured and used for model training, the performance of NISDL will be better.

Some researchers may argue that the infrared sensor can also be used for measuring skin temperature, so why do we use a vision-based method? In fact, the study [19,20] focused on thermal comfort measurement with an infrared sensor. However, the measurement accuracy is limited. Beyond this, there are other drawbacks in the infrared based measuring method: (1) Distance. The infrared sensor should be placed close to occupant. (2) Cost. The infrared sensor with high accuracy is expensive and the accuracy of an infrared sensor with low cost is also low. (3) Information is limited. From the perspective of the human sensory system, the infrared sensor is ‘touch’ and vision-based method is ‘sight’. The data captured by vision-based methods is much more than that collected by infrared sensor. e.g., human poses can be captured by vision sensor rather than infrared sensor for analyzing human thermal comfort. Based on these three drawbacks, the infrared-based method is difficult to widely apply in practice.

Furthermore, some other researchers may say that vison-based contactless measuring method may have concerns related to personal privacy. In fact, there are at least two options to protect personal privacy issues: (1) Switch button. Based on this switch button, any customer can choose to accept or reject the implementation of real-time personal service of thermal comfort. 92) Information selection. In future practical applications, only the information about human thermal comfort be processed and saved, while other information will be discarded in real-time. (3) Data protection. In order to avoid data protection issues, processed data related to thermal comfort can be directly transferred to the HVAC system instead of being saved. Therefore, from the perspective of human-centeredness, the NISDL algorithm proposed in this paper is helpful.

In future practical applications, the processing capacity of HVAC system should be considered. The HVAC system is usually equipped with computer server. The framework proposed in this paper can be embedded on the computer server directly and GPU is required in computer server. Based on occupant number in the building, it is necessary to prepare one or more GPUs. Further, the best ratio between GPU number and occupant number should be validated and tuned in practical application and will not be considered in this paper.

5.6. Exceptions

While human physiology and human thermal comfort remains a complex issue, the possibility to measure temperature distributions on the body’s surface can provide valuable indicators. Therefore, in this paper, we just focus on the prediction of skin temperature. However, some exceptions still should be mentioned. (1) There are some exceptions to the close relationship between skin temperature and thermal comfort. e.g., while sweating occurs, as the reason of sweat evaporates and heat absorption, the skin surface temperature will drop. However, the human perception could be hot. (2) There are some special cases between thermal sensation and thermal comfort. e.g., Occupant sometimes has a warm sensation, but he or she is very comfortable.

5.7. Others

Some potential limitations should be noted. (1) In this paper, all subjects are Asian females. Therefore, maybe we only can say that the NISDL is applicable to Asian women at the present time. Further, more data validation is required. (2) The subject acclimatization time is 10 min in this paper. The result of a longer time, e.g., more than 30 min, will be better than that of 10 min. When the acclimatization time is 30 min, the subjects are more likely to reach a stable starting state. (3) Relative constant parameters were set in an experiment chamber. This means that only one thermal condition was tested. If we handle the physiological experiment in different indoor parameters (e.g., indoor temperature), more valuable data and conclusions can be obtained.

6. Conclusions

In this paper, a kind of contactless measuring method based on skin sensitivity index for thermal comfort (NISDL) is proposed. For validating the effectiveness of SSI, two different deep learning frameworks with SSI were designed. A total of 1.44 million images were used for algorithm validation. The conclusions can be summarized as follows.

(1)

SSI is a good and high weight parameter in contactless measurement of skin temperature based on a deep learning network.

(2)

The location of SSI participation in NISDL network training has little impact on measuring the performance of skin temperature. Of course, if the SSI features are extracted firstly, and then merged with the features of ROI images, the corresponding effect is slightly better.

(3)

The NISDL method proposed in this paper can be used for measuring thermal comfort and more diverse data can help it to improve the measuring accuracy.

In practical application, the inter-difference is very large. How to define and calculate suitable SSI will affect the measuring results. Further, more diverse data comparison is required to improve the algorithm robustness. These areas will be our research directions in the near future.