Vision Based Dynamic Thermal Comfort Control Using Fuzzy Logic and Deep Learning

Abstract

A wide range of techniques exist to help control the thermal comfort of an occupant in indoor environments. A novel technique is presented here to adaptively estimate the occupant’s metabolic rate. This is performed by utilising occupant’s actions using computer vision system to identify the activity of an occupant. Recognized actions are then translated into metabolic rates. The widely used Predicted Mean Vote (PMV) thermal comfort index is computed using the adaptivey estimated metabolic rate value. The PMV is then used as an input to a fuzzy control system. The performance of the proposed system is evaluated using simulations of various activities. The integration of PMV thermal comfort index and action recognition system gives the opportunity to adaptively control occupant’s thermal comfort without the need to attach a sensor on an occupant all the time. The obtained results are compared with the results for the case of using one or two fixed metabolic rates. The included results appear to show improved performance, even in the presence of errors in the action recognition system.

1. Introduction

A large number of intelligent buildings have recently been developed with automated control systems [1,2]. Different independent automation systems can be integrated in such buildings for resource sharing and efficient services [2]. The control of the thermal comfort in indoor environments is an important field in intelligent buildings. Different environmental parameters can be considered, such as occupancy, temperature, carbon dioxide levels, and relative humidity. These can be used to help control the thermal comfort level inside buildings. Some of the research studies have incorporated occupant preferences into thermal control systems. This can be employed either using a fixed set point regarding every single zone in the building or through occupants’ feedback [3,4,5]. In other cases, occupants’ behaviour information has been used to predict thermal comfort levels [6,7]. However, this only happens over long periods, while an effective system must adapt to both long and short-term changes. An adaptive thermal heat balance model (ATHB) was proposed by [8]. This model integrated the adaptive approach with Predicted Mean Vote (PMV). The ATHB computes the predicted sensation votes when considering the variations of indoor environmental parameters and the running mean outdoor temperature. The metabolic rate was directly extracted from a pre-made table. The metabolic rate and clothes insulation were modified in relation to the outdoor temperature. The evaluation showed that the ATHB model performed well for naturally ventilated and air conditioned buildings.

In fact, an occupant’s preferences could also change, depending on their different daily activities due to the change of it’s metabolic rate. Therefore, the adaptation to human actions is considered in this work. Importantly, the computer vision-based action recognition is used to associate an occupant’s action to it’s metabolic rate and then compute it’s thermal comfort level. This gives the opportunity to avoid the need of using a sensor that is attached to an occupant all the time, such as in [9].

Different variables should be considered in the computation of thermal comfort levels. These include environmental and subjective variables, such as temperature, humidity, clothes, and actions. The PMV model is one of the most important thermal comfort models [10]. The metabolic rate of the occupant is considered to be a main factor in the PMV calculation. Many research studies, such as [11,12,13], assume fixed metabolic rates, often set to 1 MET (for sitting, 4184 J/kg/h). However, the resulting PMV calculation is likely to produce an inaccurate estimate of the thermal comfort level due to the change of metabolic rate in the change of occupant activities.

This work provides a novel adaptive methodology for thermal control that is based on occupant’s actions using computer vision for action recognition in conjunction with a thermal comfort fuzzy control system. Deep learning and fuzzy logic techniques are selected in this work because they can simultaneously provide great performance and explainability for the intended application. Furthermore, these popular techniques is an area of on-going research, as can be seen in a number of modeling cases e.g., [14,15,16,17].

An occupant’s actions are automatically associated with a metabolic rate. This is used in the PMV model calculation to predict the occupant thermal comfort levels. Finally, an adaptive context-aware fuzzy temperature control system is proposed to adjust thermal settings according to an occupant’s actions. Our goal is to improve the traditional thermal model by presenting an adaptive method that takes detailed activity information into account. Figure 1 illustrates an overview of the proposed system.

2. Related Work

The indoor temperature of a building is often considered to be the main parameter in air-conditioning systems. It has a big impact on energy consumption. An increase or decrease by just $1{}^{\circ }$C can mean that the energy consumption from the cooling and heating can be changed by 5–10% [18].

The simplest control system that can be used for energy conservation is based on a scheduled time strategy for switching on or off a heating system [19,20,21,22].

An occupant’s thermal comfort should ideally be adaptively maintained by changing the temperature depending on the activity of the occupant(s). Many researchers have shown that PMV can recognise and help to maintain thermal comfort levels. A further advantage is that there can be improvements in energy conservation when compared with conventional control systems [23,24].

A study that used the PMV model to quantitatively describe the thermal comfort levels of occupants can be found in [18]. A PMV model was used in a fuzzy control strategy to maintain occupants’ thermal comfort. The PMV was considered as the set point of the controller, whilst feedback was obtained from the PMV calculation based on temperature and humidity variables. Subsequently, the difference error was fed as Fuzzy controller input. The experimental results showed that the model achieved energy savings with the indoor temperature variation according to the ambient conditions whilst maintaining the thermal comfort level. The PMV model was also used by [25] to control and monitor the indoor thermal conditions. This work considered outdoor weather conditions to be explicitly included in the solution. A fuzzy logic controller was used to control the heating system. Two inputs were considered: the PMV index value and outdoor weather readings. The system was tested and compared with a classical Proportional-Integral-Derivative (PID) controller. The results showed that the proposed method achieved better performance than the PID controller.

An optimal balance between occupants’ thermal comfort and energy consumption was considered by [13]. Different parameters for an indoor room were measured, such as temperature, humidity, heat, and other subjective variables. PMV was used to calculate the thermal comfort level using measured parameters. A genetic algorithm was used to obtain optimum comfort temperatures to help reduce energy consumption. The genetic algorithm was shown to improve the system and reduce power consumption.

In [11], the PMV model was used for energy conservation and to help maintain the amenity index of indoor environments at a preferred level. A PMV prediction control method was proposed to maintain the PMV value within a preset range. This helped to reduce power consumption. The study also considered other parameters including room volume, window size and the number of people. The proposed method was evaluated in an office building in Japan. The experiments showed that the proposed method maintained the comfort level of the room at the desired level. A sensing system was proposed by [12] for thermal comfort and indoor air quality parameter monitoring. It also correlated the output of a PMV model with air quality factors, including the levels of carbon monoxide and carbon dioxide. Tests were performed using 50 subjects over a period of 10 days. The results showed good performance of the proposed system; this was confirmed with the use of Thermal Sensation Vote (TSV). All of these methods did not vary the metabolic rate of the occupants. The metabolic rate was assumed to be fixed, despite it typically varying as a function of a person’s activity.

In [26,27], a heating control system was proposed to improve the trade-off between environmental comfort and energy conservation. Mobile phone devices were proposed to be used to measure occupants’ metabolic rates. This was then used in the calculation of a PMV thermal comfort level. Experiments were performed and demonstrated that the PMV values best matched occupants’ responses to a questionnaire when measured metabolic values were used. The proposed method was also shown to improve energy conservation. However, this method requires that a mobile phone device be continuously carried to measure the metabolic equivalent values. Furthermore, a number of common mobile phone usage scenarios could lead to potential problems. In addition, it could be argued that the system is not fully automatic.

A different approach was proposed by [28] using a thermal camera for human thermal comfort measurement. PMV was included to quantify the thermal comfort. The proposed system estimated clothing insulation values that were based on thermal measurements. Different variables were considered in the calculation of the clothing insulation value. This included the heat transfer coefficient of the human body, operating temperature, skin surface temperature, and clothing surface temperature. The level of clothing insulation was factored into the PMV calculations. The evaluation results demonstrated a reasonable level of accuracy that could potentially be used for thermal comfort monitoring. A simple temperature difference between the clothing and the skin surface was used for the clothes insulation. The authors also considered a range of poses of the occupant. However, the pose estimates were not directly linked to the metabolic rate calculations in the process of achieving an accurate comfort level. The authors also did not consider different combinations of clothing that could potentially further impact the accuracy of the model. Furthermore, the use of a thermal camera might prove to not be cost-effective for many potential applications.

In contrast to existing works, automated action recognition using computer vision is used here to estimate metabolic rates. To the best of the authors’ knowledge, a new thermal comfort system is proposed here. It automatically adjusts thermal settings using time adaptive metabolic rates. Uniquely, these are estimated from actions that are recognised from RGB-D camera video sequences. An occupant’s recognised actions are used alongside different preferences and comfort levels. A virtue of the included action recognition is the ability to adapt to actions being performed at different speeds. Thus, the system proposed here can be considered to be able to adapt to different age groups, making it highly applicable to assistive living applications. The system is also designed to benefit from the hierarchical approach of the action recognition. Consequently, a new adaptive context-aware environmental comfort management system is proposed here. The performance of the proposed system is demonstrated with the use of pre-recorded action sequences. The proposed system achieves significant improvements in terms of the results of the thermal system.

3. Methodology

An important consideration, although often over looked part of the calculation of thermal comfort, is the metabolic rate. The metabolic rate is highly dependent on the activity of the person (see e.g., [29]). It is proposed here to recognise actions using a deep learning trained computer vision system. This is illustrated in Figure 2.

The action recognition system uses both appearance information in the form of RGB from digital video together with depth information. This type of modality has become quite popular for computer vision applications, as it provides distance information as well as the appearance. Applications, such as activity recognition, have become considerably more viable since the advent of the RGB-D cameras, such as Microsoft’s Kinect and Asus Xtion. Video data are particularly important because they can provide detailed information regarding the movement of a person that is important in applications, such as activity recognition. Therefore, a 3D model of the appearance and depth information is used where the appearance information is taken over time to form a 3D stream of appearance information. Similarly for the depth information, the depth information is combined from multiple sequences into Depth Motion Maps (DMM)s [30]. The DMMs are extended to include multiple appearance information, similar to e.g., [31]. The important features in the 3D depth and appearance streams are then learned by a series of fine-tuned, pre-trained 3D Convolutional Neural Networks (CNN)s. More details on this action recognition technique can be found in [32].

The results of the action recognition system are used here as inputs for the simulation of different action sequences. Each recognised action is then mapped to metabolic rate. Following that, the metabolic rate forms an input variable to the PMV calculation. The resulting PMV value is then fed to a fuzzy controller. Figure 1 shows the proposed system.

3.1. Average Action Re-Labelling for Metabolic Rate Extraction

An average re-labelling process on the classes of the action recognition model is considered here. In this re-labelling process, each action is assigned a metabolic rate that is based on those provided by [29]. By doing this, a novel method can be used to compute the metabolic rate over a period of time. Here, in this work, an average metabolic rate is considered over a period of time for several reasons. One of them is the possibility of the misclassification of an action that would likely results in an incorrect relabelling of the metabolic rate. Such an error can be reduced by taking the average of the metabolic rate over a period of time. This gives an opportunity to control the thermal model over that period based on the metabolic rate of the occupant in addition to temperature, humidity, and other parameters. To formulate this relation, let t, $A_t$, and $m_t$ be time, actions and the metabolic rate variables respectively. In a period of time from $[t_1,t_2]$ a sequence of actions can be performed $A_{t1}$ to $A_{t2}$, which are associated with a sequence of metabolic rate. As a result, temperature T can be changed in different ways either at each single action per unit of time or as a function of the average of the actions over a period of time. In this work, the average of the metabolic rates for different actions over a period of time is considered. This will help to keep the thermal model as close as possible to achieving occupant comfort-ability. This can be formulated as:

$$M_{t_1,t_2}={\displaystyle \frac{1}{N}}\sum _{t=t_1}^{t_2}{A}_t(m_t)$$

(1)

where $M_{t1,t2}$ is the resulted metabolic rate over a specific period of time, N is the number of processed actions over period $[t_1,t_2]$.

Table 1. A number of exemplar actions and the corresponding metabolic rates whilst sitting and standing [29].

Action	Metabolic Rate (Sit)	Metabolic Rate (Stand)
Eat	90	120
Drink	90	120
Lay down on sofa	46	46
Play game	90	120
Play guitar	120	180
Read book	76	105
Sit down	58	58
Stand up	70	70
Use laptop	70	93
Write on a paper	105	133

includes some daily actions that are used in the action recognition model during sitting and standing poses, with its correlated metabolic rate based on [29]. These actions are considered in terms of fuzzy temperature control to show the differences that could result when each occupant has its own metabolic rate.

3.2. Calculation of the PMV Thermal Index

The thermal comfort level has been classified by the American Society of Heating, Refrigerating, and Air-Conditioning Engineers (ASHRAE) as a standard thermal sensation scale [25]. The PMV was used as the thermal comfort index. ASHRAE recommended the preferred range of PMV to be between −0.5 and +0.5 for thermal neutrality and comfort.

The PMV thermal comfort index has been used in many studies to quantify the thermal comfort. The occupant’s metabolic rate is one of the most effective variables in PMV. However, it is widely taken as a constant in most research. This can be accurately extracted, as shown in the previous section (related information can be seen in Section 3.1).

Assigning metabolic rates to activities gives the opportunity to provide accurate thermal comfort levels based on the action of an occupant of a room. The PMV value can then be calculated from [10] based on Fanger’s experiments on the perception of thermal comfort [33], as follows:

$$PMV=(0.303e^{-0.036M}+0.028)L$$

(2)

where M is the metabolic rate and L is the thermal load on the body.

The thermal load on the body is defined as the difference between the internal heat production and the heat loss to the actual environment. This is for a person to be kept at a hypothetically thermally comfortable level. The thermal load can be calculated with:

$$\begin{array}{cc}\hfill L=(M-W)-3.96\times {10}^{-8}& f_{\mathrm{cl}}[{(T_{\mathrm{cl}}+273)}^4-{(T_{\mathrm{mr}}+273)}^4]+f_{\mathrm{cl}}{h}_{\mathrm{c}}(T_{\mathrm{cl}}-T_{\mathrm{a}})\hfill \\ \hfill +3.05[5.73-& 0.007(M-W)-{\rho }_{\mathrm{a}}]+0.42[(M-W)-58.15]\hfill \\ \hfill & +0.0173M(5.87-{\rho }_{\mathrm{a}})+0.0014M(34-T_{\mathrm{a}})\hfill \end{array}$$

(3)

where M is the metabolic rate, $(\frac{watt}{m^2})$. W is the external work, $(\frac{watt}{m^2})$, assumed zero in indoor environments. $T_{\mathrm{a}}$ is the air temperature, (${}^{\circ }$C). $T_{\mathrm{mr}}$ is the mean radiant temperature, (${}^{\circ }$C) assumed to be equal to the air temperature. ${\rho }_{\mathrm{a}}$ is partial water vapour pressure. $f_{\mathrm{cl}}$ is the ratio of a person’s body surface area when clothed to the person’s body surface area when nude. $T_{\mathrm{cl}}$ is the surface temperature of clothing, and $(C^{\circ })$. $h_{\mathrm{c}}$ is the convective heat transfer coefficient, $(\frac{W}{K.m^2})$.

The partial water vapour pressure ${\rho }_{\mathrm{a}}$ is given by:

$${\rho }_{\mathrm{a}}=6.11\times \frac{HR}{100}\times {10}^{\frac{7.5T_{\mathrm{a}}}{237.7+T_{\mathrm{a}}}}.$$

(4)

where $HR$ is the relative humidity. The surface temperature of clothing $T_{\mathrm{cl}}$ can be obtained by:

$$\begin{array}{c}\hfill T_{\mathrm{cl}}=35.7-0.028·(M-W)-I_{\mathrm{cl}}·\{3.96·{10}^{-8}·f_{\mathrm{cl}}·\\ \hfill [(T_{\mathrm{cl}}{+273)}^4-{(T_{\mathrm{mr}}+273)}^4]+f_{\mathrm{cl}}·h_{\mathrm{c}}·(T_{\mathrm{cl}}-T_{\mathrm{a}})\}\end{array}$$

(5)

where $I_{\mathrm{cl}}$ is the thermal insulation of clothes and $(m^2\frac{K}{W})$ assumed to be 0.2.

$T_{\mathrm{cl}}$ can have a linear relationship with air temperature $T_{\mathrm{a}}$ in a range 16–26 degree as mentioned in [25]. This can simplify the calculation of $T_{\mathrm{cl}}$ to:

$$T_{\mathrm{cl}}=0.5760\times T_{\mathrm{a}}+14.1810$$

(6)

The convective heat transfer coefficient $h_{\mathrm{c}}$ is given by:

$$h_{\mathrm{c}}=\left\{\begin{array}{cc}2.38|T_{\mathrm{cl}}-T_{\mathrm{a}}{|}^{0.25},\hfill & \mathrm{if}2.38|T_{\mathrm{cl}}-T_{\mathrm{a}}{|}^{0.25}\ge 12.1\sqrt{V};\hfill \\ 12.1\sqrt{V},\hfill & \mathrm{if}2.38|T_{\mathrm{cl}}-T_{\mathrm{a}}{|}^{0.25}<12.1\sqrt{V}.\hfill \end{array}\right.$$

(7)

where V is the relative air velocity and $\frac{m}{s}$ that is assumed to be 0.15.

The clothes area coefficient $f_{\mathrm{cl}}$ is obtained by:

$$f_{\mathrm{cl}}=\left\{\begin{array}{cc}1.0+1.29·I_{\mathrm{cl}},\hfill & I_{\mathrm{cl}}\le 0.078\hfill \\ 1.05+0.645·I_{\mathrm{cl}},\hfill & I_{\mathrm{cl}}>0.078.\hfill \end{array}\right.$$

(8)

It would be extremely expensive to gather real-time measurements of all of the variables involved in the PMV formula. Therefore, values are taken from the assumptions used in e.g., [18,25,34]. However, the metabolic rate value is computed according to occupant activity.

Some of the relations between a number of exemplar actions with different PMV thermal indices and associated indoor temperatures can be seen in Figure 3a,b.

3.3. Fuzzy Method Based Thermal Control Systems

The controller is composed of three interconnected parts, including computer vision model, sensors, and decision part. Figure 1 shows the control block diagram of fuzzy temperature control system.

The fuzzy controller system consists of the data acquisition, PMV index computing, fuzzy inference mechanism, and output control (temperature change). Three environmental parameters, including temperature, relative humidity, and average radiant temperature, can be directly measured by the sensors and be sent into the controller. The PMV index can be obtained from Equation (2). It is employed as the current time value to the fuzzy controller. This value can be considered to be the main parameter to control the indoor temperature. It is compared with the previous value to produce PMV variation. The considered fuzzy sets for the input and output variables are included in

Table 2. Fuzzy sets of the fuzzy heating control in terms of input (PMV ranges from too cold to too warm, $\Delta$ PMV ranges from Big−negative to Big positive) and output (Change in Temp. ranges from negative ($N_4$) to positive ($P_4$)) variables.

Variables	Terms	Sets
	TC	−3, −3, −1.7, −1.4
	C	−1.7, −1.4, −1, −0.7
	SC	−1, −0.7, −0.3, 0
PMV	N	−0.2, 0, 0, 0.2
	SW	0, 0.3, 0.7, 1
	W	0.7, 1, 1.4, 1.7
	TW	1.4, 1.7, 3, 3
	B−Negative	−3, −3, −0.6, −0.4
	N	−0.6, −0.4, −0.3, −0.1
	S−N	−0.3, −0.2, −0.1, 0
$\Delta$ PMV	Z	−0.05, −0.01, 0.01, 0.05
	S−P	0 0.1, 0.2, 0.3
	P	0.1, 0.3, 0.4, 0.6
	B−Positive	0.4, 0.6, 3, 3
	$N_4$	−0.25, −0.2, −0.15
	$N_3$	−0.2, −0.15, −0.1
	$N_2$	−0.15, −0.1, −0.05
	$N_1$	−0.1, −0.05, 0
Change in Temp.	Z	−0.0005, 0, 0.0005
	$P_1$	0, 0.05, 0.1
	$P_2$	0.05, 0.1, 0.15
	$P_3$	0.1, 0.15, 0.2
	$P_4$	0.15, 0.2, 0.25

, Figure 4a–c.

The current and variation values are then fed to the fuzzy controller as inputs to process the temperature based on pre-set rules, as can be seen in

Table 3. Rules of fuzzy temperature control system based on PMV and $\Delta$PMV inputs.

	B-Negative	Negative	S-Negative	Zero	S-Positive	Positive	B-Positive
T-Warm	$N_1$	$N_4$	$N_4$	$N_4$	$N_4$	$N_4$	$N_4$
Warm	$N_1$	$N_3$	$N_2$	$N_4$	$N_4$	$N_4$	$N_4$
S-Warm	$P_1$	Z	Z	N1	$N_2$	$N_4$	$N_4$
Neutral	$P_4$	$P_3$	Z	Z	Z	$N_2$	$N_4$
S-Cold	$P_4$	$P_4$	$P_2$	$P_1$	Z	Z	$N_1$
Cold	$P_4$	$P_4$	$P_3$	$P_3$	$P_2$	$P_2$	$P_1$
T-Cold	$P_4$	$P_4$	$P_4$	$P_4$	$P_3$	$P_2$	$P_1$

.

If the PMV index is in a reasonable range, then the output of the controller will have an effect of no change. Otherwise, the controller will maintain the PMV index in the normal range by controlling the room temperature according to the thermal comfort of an occupant. This will be based on the person’s actions in addition to other parameters. The fuzzy control used $\pm 0.25$ degrees Celsius as maximum amount over which to change the temperature in order to achieve precise control of PMV index and thermal comfort. The actual value is set according to the occupant’s preferences, affecting the amount of the system takes to adapt to the occupant’s activities. This is considered next.

3.4. Comfort Time Constant

The comfort time constant can be defined as a time required to respond to a change in ambient temperature when the ambient temperature is changed from $T_1$ to $T_2$. The technical definition is: “the time required to change 63.2% of the total difference between initial and final temperature when subjected to a step function change in temperature” [35].

The time constant, $\tau$, is an appropriate mathematical way that can be used to describe the thermal properties of a room. Thermal comfort can be supported by considering the maximum and minimum indoor temperatures that the tenants accept. Therefore, a convenient environment can be provided by an adjustment of a comfort time constant, $\tau$ according to an occupant’s preferences. The relationship between the time elapsed during the temperature change and comfort temperature can be written, as follows [35]:

$$T=(T_2-T_1)(1-e^{(\frac{-t}{\tau })})+T_1$$

(9)

When t and $\tau$ are equal, then, the equation can be simplified, as follows:

$$T=(T_2-T_1)(1-e^{(-1)})+T_1$$

(10)

Because $(1-e^{-1})$ = 0.632, then T = 0.632 $T_2$ + 0.268 $T_1$. This shows that the environment reaches 63.2% of the total difference between initial and final temperatures within time constant $\tau$ (seconds).

In this work, different comfort time constants are used to support various occupants’ preferences, i.e. fast, medium, or slow. The sitting action is selected here as an instance in the fuzzy temperature control to show the difference between the various comfort time constants. The initial temperature used is $20{}^{\circ }$C. Subsequently, the temperature is controlled based on the fuzzy rules, as discussed previously. Figure 5 shows the results of fuzzy temperature control for this sitting action using different temperature change rates over (1, 1.25, 2.5) hours.

From Figure 5, it can be seen that different comfort time constant (fast, medium, and low) can be achieved based on the temperature change rate. This provides an opportunity for an occupant to select the option according to their comfort preferences. For instance, the occupant may choose the fast mode (big temperature change) that speeds up the system to reach 63.2% of preferred level within 16 minutes. This is shown in Figure 5a. This mode could also be utilised if there is a big gap between the preferred and current temperature levels. It could be then changed to another mode in order to stabilise the temperature. However, this may be preferred by some people, but not others. Therefore, in Figure 5b,c, other modes are considered, i.e., medium and low. These are provided to cover various preferences that have different comfort time constant. As can be seen, the medium mode reaches 63.2% within 30 min.; and, the slow mode reaches within 59 min. These are due to different temperature change rates that are used in the experiments.

4. Experiments

In this work, five actions are chosen from Table 1 as daily actions that are performed by an occupant. The selected actions are eating, laying down on a sofa, sitting down, standing up, and using a laptop.

Video sequences of these actions are selected from the Microsoft Research (MSR) 3D daily activity database [36]. Some of the frames from some exemplar sequences can be seen in Figure 6.

Figure 7 shows the confusion matrices of the action recognition system in terms of the relevant actions from the MSR 3D daily activity dataset [36]. The classification performance of these sequences are used to simulate the classification of actions of interest and then, consequently, the relabelling of those actions. The actions are a subset of the available actions in the MSR 3D daily activity dataset.

The selected actions (“Eating”, “Using laptop”, “Standing”, “Sitting”, and “Laying”) are simulated to run over different periods of time with: 1.2, 2, 0.8, 2, and 2 h, respectively (totalling 480 min).

Three cases (fixed, ground-truth, and classified or adaptive) are used to control the indoor temperature based on the proposed thermal system. A comparison between fixed, ground-truth, and classified states are provided by considering the following:

• Twelve action classifications assumed for every 2 min, distributed based on the selected actions and their periods (assumes a classification rate of 0.1 Hz).
• Metabolic rate averaging is performed over a window of 12 actions (i.e., 2 min). If the two minutes covers two actions (i.e. at the boundary), then the metabolic rate will be a mixture of the two.
• For the fixed state: a constant metabolic rate is used for the selected actions over the whole duration with M = 58 (sitting) and M = 70 (standing).
• For the ground-truth state: ground-truth data of the selected actions is used to extract the metabolic rate of each one within its respective period.
• Classified state: the results of the action recognition model for the selected actions are used to extract the metabolic rate. The classified state can include actions other than the five selected actions. Metabolic rates for the misclassified states are therefore selected.
• The start-point temperature is 20 degrees for all experiments.
• Two minutes period is considered here for temperature change rate.

By doing this, the average metabolic rate can be extracted every two minutes based on the number of actions (in this case 12) while using the aforementioned methodology.

5. Results

The experimental methodology outlined above was followed based on the pre-set rules of the fuzzy temperature control in terms of fixed, ground-truth, and classified states, respectively. Figure 8 illustrates the PMV thermal comfort index and the resulting controlled temperatures as functions of time. Included in the plots are the ground truth PMV values for the selected actions; the results of action recognition for the selected actions; and, the results of an assumption of a single fixed action of sitting (commonly assumed in the literature).

5.1. RMSE of Thermal Comfort Model

The Root Mean Square Error (RMSE) of the thermal comfort models is computed to provide another aspect of comparison between the traditional and improved thermal comfort models while taking into consideration two main effective values, including PMV and temperature values.

The PMV RMSE for N measurements is:

$$RMS{E}_{\mathrm{M}}^{PMV}=\sqrt{\frac{1}{N}\sum _i^N{(PM{V}_i^G-PM{V}_i^m)}^2}$$

(11)

where $PM{V}_{\mathrm{i}}^G$ is the PMV level measurement of the ground-truth model at time i and $PM{V}_{\mathrm{i}}^M$ is the PMV level measurement of the used model at time i.

The temperature measurements RMSE for N measurements is:

$$RMS{E}_{\mathrm{M}}^T=\sqrt{\frac{1}{N}\sum _i^N{(T_i^G-T_i^m)}^2}$$

(12)

where $T_i^G$ and $T_i^m$ are the temperatures at time i for the ground truth and model, respectively.

A number of different ways of comparing the different techniques are now considered in the RMSE calculation.

5.2. RMSE for an Extended Duration of a Number of Different Activities

First, RMSE is calculated every 10 minutes for the whole duration of a 480 min simulation. Figure 9 shows the RMSE values of the adaptive (dynamic) and fixed models with respect to PMV levels and temperature.

5.3. RMSE Calculated for Each Action

For the second scenario, RMSE is calculated based on each action (Eating, Using laptop, Standing, Sitting, Laying).

Table 4. PMV RMSE of the proposed and traditional models in terms of actions period and average scenario.

	PMV RMSE
Model	Actions					Overall
	A1	A2	A3	A4	A5
Fixed (sitting)	0.66	0.15	0.03	0.13	0.17	0.29
Dynamic (sitting)	0.02	0.19	0.07	0.04	0.00	0.08
Fixed (standing)	0.63	0.11	0.39	0.28	0.17	0.33
Dynamic (standing)	0.01	0.13	0.28	0.21	0.04	0.16

and

Table 5. The temperature RMSE of the proposed and traditional models in terms of actions period and average scenario.

	Temperature RMSE
Model	Actions					Overall
	A1	A2	A3	A4	A5
Fixed (sitting)	1.44	2.02	1.33	0.41	1.09	1.26
Dynamic (sitting)	0.03	0.37	0.28	0.06	0.00	0.15
Fixed (standing)	2.33	2.93	1.58	0.89	2.33	2.14
Dynamic (standing)	0.02	0.61	0.87	0.65	0.13	0.54

present comparisons between the fixed and adaptive (dynamic) models using the RMSE for each individual action.

The average RMSE for the whole process period is also calculated and included at the ends of Table 4 and Table 5. These values help to provide an overall idea regarding each model.

It can be seen from the results that are shown in Figure 9 and Table 5 that the adaptive (dynamic) control improves the temperature adjustment settings with respect to thermal comfort levels. The adaptive control produced RMSE values that are close to the ground-truth in comparison to the fixed models.

Other RMSE calculations are also considered under the same scenarios, but using PMV measurements to provide a comparison between the models based on the PMV thermal comfort levels. Table 4 includes a comparison between the fixed and adaptive models in terms of average RMSE calculation. These are based on each action period and the whole process duration using PMV measurements.

The adaptive control appears to improve the results of the PMV calculations. In turn, this also helps to obtain more accurate thermal comfort levels, based on an occupant’s actions. It has produced low RMSE readings that can be considered very close to the ground-truth. The fixed model, also shown in Table 4, appears to provide reduced performance in comparison to the adaptive model. A fixed model for sitting and standing has also been considered. Splitting the fixed model into these two gross poses also appears to not be as optimal as the time-dynamic approach of continuously recognising actions.

In Figure 9 it can be noticed that the adaptive model produces low RMSE in actions 2 and 3 in terms of temperature measurements, but more RMSE in terms of PMV level measurements in comparison with the fixed model. This is due to the big difference in temperature measurements. On the other hand, a small difference in the PMV level measurements. In addition, the effects of actions misclassification can be handled with an average process if they are not too much.

6. Conclusions

A novel context-aware adaptive environmental control system has been presented. The metabolic rate of an occupant is considered to be a very important factor in the calculation of the PMV index. The metabolic rate can have a great impact on the thermal comfort level. A deep learning-based action recognition system was proposed to recognise actions of a person and then identify the relevant metabolic rate for the recognised actions. This enables the extraction of a relatively accurate metabolic rate and, consequently, an accurate thermal comfort level. A novel adaptive methodology for thermal control based on occupant context has also been provided by integrating the computer vision for action recognition with the thermal comfort model using fuzzy logic. This integration provides a way to adaptively control occupant’s thermal comfort without a need to attach a sensor on an occupant. Pre-set rules of the fuzzy temperature control are designed a priori and included in the experiments that followed, i.e in terms of fixed, ground-truth, and classified states. The fuzzy temperature control was used to adjust the thermal settings according to the occupant’s actions and their personalised comfort for a number of different scenarios. A comparison between the fixed and adaptive (dynamic) models has been presented. Quantitative comparisons that are based on RMS error values for both PMV and temperature have been presented. The achieved results are compared with the results for the case of using one or two fixed metabolic rates. The considered actions in these assessments included eating, using a laptop, standing, sitting, and laying. The results have helped to demonstrate the good performance of the adaptive (proposed) model.