A Contactless Measuring Method of Skin Temperature based on the Skin Sensitivity Index and Deep Learning

Round  1

Reviewer 1 Report

The paper could be an interesting insight on skin temperature measurement, but its practical application to HVAC control is not feasible because of several barriers, partially presented by the authors. The main question to be replied by the authors is: why a vision system has to be used when a simple wearable device (eg. Smartbands) can provide more information on human thermal response? Wearable solutions would provide better performances without any privacy and accuracy problem.

Introduction

· Ref 1 is old: more recent literature should be considered

·  It is not clear what is the advancement with respect to the state of the art for what concerns the measurement approach

Related work

· This section seems to be a detailed replication of the introduction. This would make the reading longer and heavy. I would suggest incorporating this part into the introduction.

Research methods

· The season is not described, this could impact the adaptation process

· The equation 1 is not clearly described (each coefficient should be explained)

·  Given the impact of several factors, the subjective sensation should had been acquired with questionnaires so to investigate the correlation with real subjects’ sensation

Results

·  The error is presented with too many digits considered the uncertainty of the temperature sensor used for the test

Discussion and conclusion

· The argumentation about the results and applicability is poor. The practical application of the proposed methodology presents a lot of barriers that are not faced by the paper and discussion of results. The idea of proposing the SSI as a new comfort index is too ambitious considering the results presented and the reduced number of subjects involved, especially without considering the subjective sensation.

Author Response

Peer reviewer A

1. The paper could be an interesting insight on skin temperature measurement, but its practical application to HVAC control is not feasible because of several barriers, partially presented by the authors. The main question to be replied by the authors is: why a vision system has to be used when a simple wearable device (eg. Smartbands) can provide more information on human thermal response? Wearable solutions would provide better performances without any privacy and accuracy problem.

Answer: Based on current technology status, from the perspective of practical application, it is indeed that the privacy and accuracy of wearable device is better than the vision-based method. But, why do we still study vision-based method for skin temperature detection and thermal comfort perception? The motivation is shown as follows.

(1)   One of the reasons for the outbreak of Cambrian species is that the creature in the earth have visual capabilities. Using artificial intelligence, in the three capabilities of ‘vision’, ‘touch’ and ‘hearing’, we believe that when the machine has visual capabilities, it can better serve humans.

(2)   From the current technical point of view, the wearable equipment is indeed better. However, each kind of product has a corresponding customer, and not everyone is willing to wear wearable device indoors (e.g. Wristband, Smartbands). In addition, the wearable device belongs to ‘touch’ capability. The expand ability of wearable device is weaker than that of the vision-based method.

(3)   For privacy and ethics, two solutions can be used in practical application.

[1]   The user has permission to turn on/off the skin temperature and thermal comfort measurement function freely.

[2]   Only the information related human thermal comfort will be saved, others will be discarded.

As to a kind of new technology, we just focus on the technical breakthrough itself which can serve for a comfortable environment and energy saving. In practical application, we also believe that related company (using our technology) will consider the privacy and ethics in future.

(4)   The technology proposed in this paper is still a long way behind the practical application. However, technology is always improving gradually. Further, our philosophy on this technology is human-centered, and the direction of artificial intelligence (AI) should also be human-centered. Therefore, contactless perception of thermal comfort is worth exploring.

Introduction

2.  Ref 1 is old: more recent literature should be considered

Answer: Reference 1 has been replaced by another related paper, see line 37-38, 424-426. We also check all the other references.

3.  It is not clear what is the advancement with respect to the state of the art for what concerns the measurement approach

Answer: Could I answer the comments in this way?

From the perspective of human thermal comfort perception, there are three kinds of methods, they are 1) contact method 2) semi-contact method 3) contactless method.

For contact method, the state-of-the-art sensor is iButton and the corresponding accuracy is 0.125 ºC. In this paper, the ground truth of skin temperature is collected by iButton.

For semi-contact method, infrared based- method (Reference 19, 20) is the state-of-the-art method, and the sensor was fixed on the frame of eyeglasses. However, the accuracy of infrared sensor is more than 1 ºC. Further, not everyone wears glasses, and not everyone is willing to fix infrared sensors on their glasses. Therefore, the practical application possibility is weak.

Thermal camera also is a kind of contactless method, however, in practical application of intelligent buildings, the price and convenience are very important. Therefore, thermal camera is not considered for comparison in this paper.

For contactless method, NIPST and NIST were proposed in 2017 (Reference 21) and normal computer or cell phone camera was adopted for skin temperature estimation. This paper is the first work in contactless method of thermal comfort perception. Therefore, NIPST was defined as baseline in this paper.

Related work

4. This section seems to be a detailed replication of the introduction. This would make the reading longer and heavy. I would suggest incorporating this part into the introduction.

Answer: Thanks for peer reviewer’s suggestion. Could we still keep it into two parts (introduction, related work)? We think that it will be more concise. Thanks so much!

Research methods

5. The season is not described, this could impact the adaptation process

Answer: The experiment was handled in the winter of Sweden. I have add this information into section 3.1.2.

6. The equation 1 is not clearly described (each coefficient should be explained)

Answer: The meaning of parameters in equation 1 has been added in the paper (line 177-180, clean version of paper).

7.  Given the impact of several factors, the subjective sensation should had been acquired with questionnaires so to investigate the correlation with real subjects’ sensation

Answer: It is true that questionnaire is a good way to know the subjects’ thermal sensation. In this paper, we focus on skin temperature and construct a new method to estimate skin temperature based on deep learning. But, we will ask more subjects (>64 subjects) to participate into the experiment in next step (in 2019). Then, subjects’ sensation will be adopted in experiment. Thanks so much1!

Results

8. The error is presented with too many digits considered the uncertainty of the temperature sensor used for the test.

Answer: In this paper, when the skin temperature is outputted from deep learning model (*.h5), there are many decimals digits. Based on the accuracy of iButton (Rounded to 3 decimals digits), one decimals digit has been rounded in this paper. Also, other percentage values are accurate to 2 digits. The corresponding data are shown in table 2 and ‘results’.

Discussion and conclusion

9. The argumentation about the results and applicability is poor. The practical application of the proposed methodology presents a lot of barriers that are not faced by the paper and discussion of results. The idea of proposing the SSI as a new comfort index is too ambitious considering the results presented and the reduced number of subjects involved, especially without considering the subjective sensation.

Answer: From the perspective of practical application, there are still a lot of work to be done. More validation will be handled for improving the robustness of framework we proposed. The motivation of proposing SSI is overcoming inter-individual difference. In next step, we will handle more experiment with more human subjects, also the subjects’ sensation will be included. Thanks for reviewers’ suggestion.

Reviewer 2 Report

Clarity and Length: 

The manuscript has multiple sentences that are confusing. For example: "The infrared sensor is similar with the ‘human tactile organ’ which only can get limited thermal comfort information. But the vision-based methods are similar to the ‘human eye’. Some information, such as poses of thermal comfort, can be obtained and analyzed by a vision-based method, but not by an infrared-based method."

Also, the authors repeat ideas over multiple chapters. For example, the section 5.1 and 5.2 both describe how the proposed method (NISDL) is better than DL, and the advantage comes from using skin sensitivity index (SSI).

Originality and contributions:

Dataset and SSI concepts were introduced in their previous paper (reference [21]). The novelty of this paper is the convolutional neural network trained on the color images and SSI information to predict skin temperature. The authors can work with the editor to see if the length can be reduced accordingly.

Comments:

1. The manuscript focuses on the fact that "no satisfactory method for perceiving human thermal comfort" exists. However, the thermal comfort was neither measured, neither predicted by the proposed method. Thus, one question that is unaddressed is: how does this method improve the state-of-the-art in thermal comfort modeling and prediction?

2. Only 10 minutes for test subjects acclimatization time is not optimal. Authors should consider a larger acclimatization time in the future (> 30 min).

3. The iButton was used to measure the subjects' temperature from the back of the hand. However, this device is reporting the skin temperature from a small area of the hand. How are the variations in temperature between different parts of the hand treated by the proposed model ?

4. The process of evaluating a new frame and predicting the skin temperature was not explained. Given a new subject, does the proposed model needs skin temperature and skin saturation to compute SSI ? How is the SSI computed for each pixel in the ROI image ?

5. Discuss the the limitations of the proposed model. Would this apply under different lighting conditions ? How about different skin tones ?

Author Response

Peer reviewer B

Clarity and Length:

1.  The manuscript has multiple sentences that are confusing. For example: "The infrared sensor is similar with the ‘human tactile organ’ which only can get limited thermal comfort information. But the vision-based methods are similar to the ‘human eye’. Some information, such as poses of thermal comfort, can be obtained and analyzed by a vision-based method, but not by an infrared-based method."

Also, the authors repeat ideas over multiple chapters. For example, the section 5.1 and 5.2 both describe how the proposed method (NISDL) is better than DL, and the advantage comes from using skin sensitivity index (SSI).

Answer: The corresponding contents have been revised in the paper (line 357-362, section 5.1 and section 5.2, clean version of paper). Thank you so much!

Originality and contributions:

2. Dataset and SSI concepts were introduced in their previous paper (reference [21]). The novelty of this paper is the convolutional neural network trained on the color images and SSI information to predict skin temperature. The authors can work with the editor to see if the length can be reduced accordingly.

Answer: Yes, we will ask editor and try to reduce the length of this paper.

Comments:

3. The manuscript focuses on the fact that "no satisfactory method for perceiving human thermal comfort" exists. However, the thermal comfort was neither measured, neither predicted by the proposed method. Thus, one question that is unaddressed is: how does this method improve the state-of-the-art in thermal comfort modeling and prediction?

Answer: While human physiology and human thermal comfort remains a complex issue, the possibility to measure temperature distributions on the body's surface can provide valuable indicators. In fact, if special cases are not considered, there are dense relationship between human thermal comfort and skin temperature. So that many researchers study thermal comfort from the perspective of estimating skin temperature (e.g. reference 8, 9, 11-13). Further, heart rate is estimated for thermal comfort perception (e.g. reference 15). From this point of view, in this paper, we just studied skin temperature estimation based on contactless method. This is the reason that why our paper title is “contactless measuring method of skin temperature based on skin sensitivity index and deep learning”.

However, the human sensation is very important. So that we will ask more subjects (more than 64 subjects) to participate in a new experiment in the next step (in 2019). Also, subjects’ sensation will be adopted in experiment.

4.  Only 10 minutes for test subjects acclimatization time is not optimal. Authors should consider a larger acclimatization time in the future (> 30 min).

Answer: Yes, we agree with peer reviewers’ comment. In the next experiment, the subjects acclimatization time (in chamber) will be more than 30 min.

5.  The iButton was used to measure the subjects' temperature from the back of the hand. However, this device is reporting the skin temperature from a small area of the hand. How are the variations in temperature between different parts of the hand treated by the proposed model?

Answer: In this paper, the middle area of the back of the right hand was covered by iButton when the iButton is collecting temperature. Therefore, we cannot capture good images from the back of the right hand. Therefore, we assume that the temperature of the back of left hand and that of the back of right hand are the same. Also, assuming the temperature of the whole back of one hand is the same.

In the experiment of this paper, we have been taken some measures to reduce the error. e.g. iButton is placed in the middle of the back of the right hand, video is captured from the back of left hand, we have made sure that the ROI (region of interest) image covers the middle area of back of the left hand.

In the next step, we will consider the skin temperature difference between different parts of the hand and body.

6.  The process of evaluating a new frame and predicting the skin temperature was not explained. Given a new subject, does the proposed model needs skin temperature and skin saturation to compute SSI? How is the SSI computed for each pixel in the ROI image?

Answer: Given a new subject, the skin temperature and skin saturation are needed for computing SSI. This information will be a kind of prior knowledge for the model proposed in this paper. However, only a small amount of skin temperature and skin saturation data is needed to calculate SSI.

For ROI image (150×150×3), the saturation channel is extracted which is a matrix (150×150). Based on this saturation matrix, the mean value is adopted in this paper and one saturation mean was outputted. The corresponding skin temperature is collected by iButton. Two minutes data can be used for obtaining two number pairs (S, T). Then the SSI of indoor occupant can be computed based on the equation 1 (T = k_i × S + b_i). For improving the accuracy, many more minute data can be adopted for computing SSI.

In this paper, the SSI is defined as a constant parameter for one indoor occupant. In next step, SSI of one occupant will be extended to a math function, so that many kind of situation can be covered, e.g. different seasons, different time in one day, different skin tones, etc.

7.  Discuss the limitations of the proposed model. Would this apply under different lighting conditions? How about different skin tones?

Answer: The practical application situation is very complicated, many factors should be considered, including different skin tones, camera distance from indoor occupant, camera angle, indoor lighting, sunlight coming in from the window or not, day or night, season, and so on.

From the research point of view, we set relative constant indoor parameters in experiment chamber, including lighting, dry-bulb air temperature, humidity. Further, the camera angle and distance are constant.

Therefore, we have not validation data in different lighting conditions and skin tones now.

But, as for condition of different lighting and that of different skin tones, we think that NISDL will be effective if the data is sufficient. The reasons are shown as follows.

(1) NISDL is a deep learning framework composed of different components, including convolutional layers, pooling layers, and fully connected layers. If the data with condition of different lighting and that of different skin tones are contained in training data, a new model (*.h5) can be generated, and the skin temperature of occupant can be outputted by model (*.h5).

(2) Why do we design SSI? The motivation is to overcoming inter-individual difference, including skin color difference. When the skin tones is changed, the corresponding SSI is changed. The SSI will be useful in data training and skin temperature prediction. 

Anyway, whether our method is effective or not, it requires verification. We will consider this situation in the next step of study.

Reviewer 3 Report

The authors describe a method used to predict skin temperature based on a vision based method. While the methods and results are rather clear, the following points need to be addressed in a major revision before publication.

1)      First of all, the title and in general the usage of invasive/non-invasive is wrong. Invasive methods by definition enter the human body. Non-invasive methods are all methods when no break in the skin is created and there is no contact with the mucosa (even written in Wikipedia though not a scientific source). None of the methods used in the papers referenced by the authors are invasive methods, they are all non-invasive. Therefore, I urge the authors to change title and all occurrences e.g. to contactless methods instead of non-invasive methods in order to avoid this error.

2)      The introduction is focussed on thermal comfort; however the study itself and the results are far away from thermal comfort measurements – they simply describe a method to predict skin temperature. It is commonly know in the field, that skin temperature is related to thermal perception, but also that there are boundaries, when such relationship fails, e.g. when sweating occurs, which reduces skin temperature due to evaporation, while thermal perception is still warm to hot. In addition, authors should be clear about the distinction between thermal sensation and thermal comfort – I can feel perfectly comfort at a warm sensation, when this is my preferred state for a given activity. As such thoughts are not relevant for the content and focus of the paper, I suggest removing large parts of the introduction related to thermal comfort and instead extend the review on earlier attempts to predict skin temperature together with their results. If the authors decide to keep some of the notes related thermal comfort, they are clearly advised to mentioned the limitations concerning skin temperature, thermal sensation, and thermal comfort as briefly sketched above.

3)      Introduction in general: many references are listed and mentioned, but their main findings are not described or evaluated – please describe for each study referenced, their main contribution/results and how this related to your study. For examples line 104 “Yao investigated the relationship …” – so besides investigating such relationship – what was the outcome? Was this method successful? Please revise the introduction accordingly.

4)      Line 96: “Liu proposed” This might be correct, but this is not the first method proposed to estimate mean skin temperature. See e-g- ISO 9886, which exists since much longer, together with related research papers.

5)      Line 123 “Taleghani, de Dear, Rupp and Djamila also reviewed the study of thermal comfort” Besides my general note above, that the focus on thermal comfort should be reduced, I wonder, why these studies are mentioned here and not other – more recent – reviews. In addition, what is the relation of this and the following sentence to the section starting afterwards “With the development” – Both parts are referring to very difference aspects of thermal comfort research. Therefore, I suggest deleting lines 122 to 125. In addition, there is a new database, database II of thermal comfort votes, and ASHRAE 55-2013 is not the latest version.

6)      Line 143, the number of subjects is rather small and it is not clear why you need 1.44 million images from so few subjects, and how you reached this number of images – what was the sampling rate?

7)      Line 145, only one thermal conditions was tested, which does not allow any generalizable conclusions on the method applied and should be clearly mentioned in the limitations. Preferably, more experiments will be added before publication.

8)      Line 148, only young Asian females were tested, which again does not allow any generalizable conclusions and should be clearly mentioned in the limitations. Preferably a wider variation of subjects will be tested before publication.

9)      Line 153, where did subjects come from? What were the thermal conditions before entering the chamber? In general 10 minutes is not sufficient to reach steady-state, so that you are looking at transient effect, which needs to be discussed.

10)   Line 167, “linear ST model” – please explain each abbreviation before its first usage

11)   Line 168, please explain each variable in the equation, bi is not defined.

12)   Line 355, A third option is processing without saving in order to avoid data protection issues

13)   Discussion, another critic is the computing power required for your method, which reduces application potentials –please discuss.

Author Response

Peer reviewer C

Comments and Suggestions for Authors

The authors describe a method used to predict skin temperature based on a vision based method. While the methods and results are rather clear, the following points need to be addressed in a major revision before publication.

1. First of all, the title and in general the usage of invasive/non-invasive is wrong. Invasive methods by definition enter the human body. Non-invasive methods are all methods when no break in the skin is created and there is no contact with the mucosa (even written in Wikipedia though not a scientific source). None of the methods used in the papers referenced by the authors are invasive methods, they are all non-invasive. Therefore, I urge the authors to change title and all occurrences e.g. to contactless methods instead of non-invasive methods in order to avoid this error.

Answer: We have changed non-invasive to contactless in this paper. Thank you so much!

2. The introduction is focused on thermal comfort; however the study itself and the results are far away from thermal comfort measurements – they simply describe a method to predict skin temperature. It is commonly know in the field, that skin temperature is related to thermal perception, but also that there are boundaries, when such relationship fails, e.g. when sweating occurs, which reduces skin temperature due to evaporation, while thermal perception is still warm to hot. In addition, authors should be clear about the distinction between thermal sensation and thermal comfort – I can feel perfectly comfort at a warm sensation, when this is my preferred state for a given activity. As such thoughts are not relevant for the content and focus of the paper, I suggest removing large parts of the introduction related to thermal comfort and instead extend the review on earlier attempts to predict skin temperature together with their results. If the authors decide to keep some of the notes related thermal comfort, they are clearly advised to mentioned the limitations concerning skin temperature, thermal sensation, and thermal comfort as briefly sketched above.

Answer: This comment is very valuable, and the thermal comfort is discussed from a high level perspective. I think this comment has broadened our horizons. Instead of removing the contents related to thermal comfort in ‘introduction’ and ‘related work’, we have been added the limitations mentioned above in discussion parts of this paper (5.6 Exceptions). Could we arrange the content in this way? Thank you so much!

3.  Introduction in general: many references are listed and mentioned, but their main findings are not described or evaluated – please describe for each study referenced, their main contribution/results and how this related to your study. For examples line 104 “Yao investigated the relationship …” – so besides investigating such relationship – what was the outcome? Was this method successful? Please revise the introduction accordingly.

Answer: Thanks so much for this comments. We have been revised the corresponding contents in introduction and related work, e.g. line 105-109, line 129-130. (clean version of paper)

4. Line 96: “Liu proposed” This might be correct, but this is not the first method proposed to estimate mean skin temperature. See e-g- ISO 9886, which exists since much longer, together with related research papers.

Answer: We re-constructed the sentence in this paper, Could you see line 95 (clean version of paper).

We also read ISO 9886. Thank you so much!

5.  Line 123 “Taleghani, de Dear, Rupp and Djamila also reviewed the study of thermal comfort” Besides my general note above, that the focus on thermal comfort should be reduced, I wonder, why these studies are mentioned here and not other – more recent – reviews. In addition, what is the relation of this and the following sentence to the section starting afterwards “With the development” – Both parts are referring to very difference aspects of thermal comfort research. Therefore, I suggest deleting lines 122 to 125. In addition, there is a new database, database II of thermal comfort votes, and ASHRAE 55-2013 is not the latest version.

Answer: Thanks so much for this comment. We have been removed the corresponding contents.

6. Line 143, the number of subjects is rather small and it is not clear why you need 1.44 million images from so few subjects, and how you reached this number of images – what was the sampling rate?

Answer: The sampling rate is 30 images / s. Why do we collect 1.44 million images? The reason is that big data helps NISDL to learn more features of skin texture. In this paper, we designed the NISDL framework based on deep learning. For deep learning, the bigger the data is, the easier it is to get a more robust model. The data of 12 subjects was defined as training set and that of 4 subjects was defined as test set. It means that 360,000 images (test set) were not used in the process of model generated. Fig 5 show that NISDL is effective to some extent.

However, it is truth that the subject number is limited in this paper. We will ask more subjects to participate into our experiment in the next step.

7. Line 145, only one thermal conditions was tested, which does not allow any generalizable conclusions on the method applied and should be clearly mentioned in the limitations. Preferably, more experiments will be added before publication.

Answer: In this paper, in order to observe the skin texture variation of the back of the hand, the indoor parameters, such as temperature, were controlled to a relatively constant number.

The peer reviewers' opinion is right. Following this comments, more data can be obtained. We have been added the limitations in discussion, see line 393-395 (clean version of paper).

8. Line 148, only young Asian females were tested, which again does not allow any generalizable conclusions and should be clearly mentioned in the limitations. Preferably a wider variation of subjects will be tested before publication.

Answer: We have been mentioned the limitation in discussion. See line 389-390 (clean version of paper). Thank you so much! 

9. Line 153, where did subjects come from? What were the thermal conditions before entering the chamber? In general 10 minutes is not sufficient to reach steady-state, so that you are looking at transient effect, which needs to be discussed.

Answer: The subjects are all Asian female and studied or worked in Sweden. The experiment was handled in the winter of Sweden. It was very cold outside. Unfortunately, we did not record outdoor information, e. g. outdoor temperature.

In fact, we plan to adjust the ‘acclimatization time’ in next experiment. We also have been added related information in ‘Discussion’, see line 391-393 (clean version of paper).

10. Line 167, “linear ST model” – please explain each abbreviation before its first usage

Answer: It has been done, see line 175? (clean version of paper)

11. Line 168, please explain each variable in the equation, bi is not defined.

Answer: The meaning of parameters in equations has been explained now. Thank you so much!

12. Line 355, A third option is processing without saving in order to avoid data protection issues

Answer: This comments has been added in the paper (5.5. Practical Application, from line 364 to line 372, clean version of paper).

13. Discussion, another critic is the computing power required for your method, which reduces application potentials –please discuss.

Answer: It has been added in the paper (5.5 practical application, from line 372 to line 377, clean version of paper).

Round  2

Reviewer 1 Report

Dear Author,

thanks for reviewing the paper. However I still have concerns about the applicability. It is true that not everybody would use wearables, but please consider also that most of the users would not accept to be under a vision system. So, if you argument that the barrier can be overcome by giving the possibility to switch off the camera, my idea is that most of the users would kept it off. Section 5.7 is empty.

Reviewer 3 Report

Please find my comments in attached word file.

Comments for author File: Comments.pdf