Energy-Saving Control Algorithm of Venlo Greenhouse Skylight and Wet Curtain Fan Based on Reinforcement Learning with Soft Action Mask

Abstract

Due to the complex coupling of greenhouse environments, a number of challenges have been encountered in the research of automatic control in Venlo greenhouses. Most algorithms are only concerned with accuracy, yet energy-saving control is of great importance for improving economic benefits. Reinforcement learning, as an unsupervised machine learning method with a framework similar to that of feedback control, is a powerful tool for autonomous decision making in complex environments. However, the loss of benefits and increased time cost in the exploration process make it difficult to apply it to practical scenarios. This work proposes an energy-saving control algorithm for Venlo greenhouse skylights and wet curtain fan based on Reinforcement Learning with Soft Action Mask (SAM), which establishes a trainable SAM network with artificial rules to achieve sub-optimal policy initiation, safe exploration, and efficient optimization. Experiments in a simulated Venlo greenhouse model show that the approach, which is a feasible solution encoding human knowledge to improve the reinforcement learning process, can start with a safe, sub-optimal level and effectively and efficiently achieve reductions in the energy consumption, providing a suitable environment for crops and preventing frequent operation of the facility during the control process.

1. Introduction

The Venlo-type greenhouse originates from the Netherlands. It is a kind of small-roofed glass greenhouse that has become the most widely used type of glass greenhouse in the world. It has the characteristics of small cross-sectional members, convenient installation, high light transmission rate, good sealing, and a large ventilation area. A typical Venlo-type greenhouse is shown in Figure 1. The equipment of a Venlo-type greenhouse generally includes skylights, wet curtain fans, sunshades, heating equipment, etc. Owing to the interaction of greenhouse microclimate factors, including light, temperature, humidity, carbon dioxide concentration, etc., the design of the control algorithm for a Venlo-type greenhouse faces some challenges. In practical application, the multi-factor coordinated control algorithm [1,2,3] effectively decouples control from the action of the actuator, but relies on experience and repeated experiments to set control parameters, and the parameters of the algorithm need to be adjusted manually, often with seasonal changes. Additionally, the energy consumption of greenhouse control facilities is seldom considered in the control algorithm. In the context of a greenhouse, higher control accuracy is bound up with greater energy consumption. Considering the comprehensive benefits of greenhouse environmental control, it is extremely important to reduce energy consumption on the premise of a comparatively suitable microclimate for crops. The energy consumption of heating equipment in the cold season and the electrical energy consumption of wet curtain fans for cooling in hot climates are the main components of consumption in this type of greenhouse, followed by the electrical energy consumption of the motor of the skylight and other facilities in the switching stroke.

This paper conducted in-depth research on the energy-saving optimization problem of the skylight and the wet curtain fan for cooling control in a hot climate. The skylight cools the indoor air through indoor and outdoor air heat exchange, while the wet curtain fan system forms a low-temperature airflow that speedily passes through the wet curtain wall and carries moisture from the curtain into the room through fan exhaust to achieve cooling; its cooling capacity is stronger than the skylight’s. Nonetheless, when the wet curtain fan is opened, the skylight needs to be closed to ensure the effect, and the actions between the two are mutually exclusive. As a result, our optimization goal is to ensure that the factors in the greenhouse are comparatively appropriate, to shorten the use time of the fan, and to avoid the frequent action of the sunroof resulting from the mutual exclusion of the actions of the two mechanisms.

Some researchers have applied multi-objective optimization [4,5,6] or model prediction methods [7,8] to optimize energy conservation, but the proposed algorithms usually have problems such as the high difficulty of solving and online optimization, as well as an over-reliance on model prediction accuracy [9]. In recent years, intelligent methods have gradually been adopted in the field of energy consumption model prediction and energy conservation control [10,11,12,13], where reinforcement learning is a new idea to explore optimization policies in complex environments. It continuously optimizes the multi-objective strategy through the rewards and punishments obtained by interacting with the unknown environment. In the field of greenhouse control, several researchers [14,15,16] optimized heating, fruit yield, and planting cost while maintaining effective control over greenhouse factors. Andrey et al. [17] and Afzali et al. [18] reduced the average light supplement duration on the basis of maintaining the performance of the leaf area. Nonetheless, most of the above research is on the basis of offline simulation experiments that are dissimilar from the actual complex environment. The direct use of the control strategies obtained in the actual environment often has poor benefits, but the reinforcement learning process that directly interacts with the actual environment online requires a lot of time and benefit costs for early training, and even unsafe actions may occur, causing damage to the facilities.

For the practical application of reinforcement learning, relevant theoretical research has also developed a multitude of branches. For instance, offline reinforcement learning aims to conduct pre-training through offline data obtained from existing policy interactions [19,20,21,22,23,24], and security reinforcement learning aims to ensure the security of the final policy and even the exploration process by defining the cost function [25], focusing on bad samples [26], or controlling stability theory [27]. Nevertheless, the offline learning effect is limited by the sample quality, and most of the safe learning does not guarantee the safety of the training process. Methods to explore the safety of the process [27] are mostly theoretical on account of the strong assumptions based on the object model.

As a model-less method, although its network structure has a strong learning ability for complex problems, the initialization and exploration of policies are random, which is the main reason for the low and unstable early benefits of this method. However, in practical terms, part of the knowledge about the environment is summarized from the past, which can be used to guide the policy initialization and even optimization process, avoiding low returns and dangerous actions from the initial stage of learning, reducing the time cost, and speedily improving the benefits. At present, it is feasible to reconstruct the policy into a specific space constrained by rules [28,29] or initialize the policy network based on a decision tree [30]. Nonetheless, the former rule space is determined before learning, while the second method’s learning effect depends on the design quality of the initial decision tree, which requires perfect rules and is difficult to apply to complex greenhouse environments.

Under the background of the above problems, this paper deals with the energy-saving control of a skylight and wet curtain fan over temperature and humidity in a Venlo-type greenhouse in summer by integrating some existing knowledge and rules into the reinforcement learning method to reduce energy consumption while ensuring environmental suitability. The control problem and the proposed approach are described in detail in the following Section 2. Section 2.1 starts with an introduction to the controlled microclimate factors in Venlo-type greenhouses, describing the summer control problem in Venlo-type greenhouses as well as the control objectives of the problem in a mathematical form. Section 2.2 presents the recommended approach, which is based on reinforcement learning combined with the structure of the Soft Action Mask (SAM) designed to enable knowledge-initiated reinforcement learning, a safe and efficient training process for reinforcement learning, and optimizable rules. Section 3 compares and analyzes the training results with several existing methods in terms of efficiency gains, control effectiveness, and safety. Section 4 summarizes the conclusions and implications of the work.

2. Method

2.1. Problem Description

Generally, in Venlo-type greenhouses, the mathematics of the change process can be described by Equation (1) [31], considering the three microclimate factors of the temperature, the humidity, and the concentration of CO${}_2$ in the greenhouse, where $\mathit{f}$ and $\mathit{G}$ are unknown functional relations characterizing the mechanism of the control object, respectively:

$$\begin{array}{c}\hfill \mathit{F}\left(\mathit{x}\right)=\frac{d\mathit{x}}{dt}=\mathit{f}(\mathit{x},\mathit{v})+\mathit{G}(\mathit{x},\mathit{v})\mathit{u}\end{array}$$

(1)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathit{x}& ={\left[T_{air},H_{air},C{O}_{2,air}\right]}^T\hfill \\ \hfill \mathit{v}& ={\left[I_{glob},T_{\mathrm{out}},H_{\mathrm{out}},C{O}_{2,out},V_{\mathrm{wind}}\right]}^T\hfill \\ \hfill \mathit{u}& ={\left[u_{\mathrm{vent}},u_{\mathrm{pad}},u_{\mathrm{scr}}\right]}^T\hfill \end{array}\end{array}$$

(2)

Variables in (2) are shown in

Table 1. Explanation of symbols of variables and parameters.

Symbol	Explanation	Unit
$\mathit{x}$	The indoor data
$T_{\mathrm{air}}$	The temperature of indoor air	${}^{\circ }\mathrm{C}$
$H_{\mathrm{air}}$	The absolute humidity of indoor air	$\frac{\mathrm{g}}{{\mathrm{m}}^3}$
${\mathrm{CO}}_{2,air}$	Indoor carbon dioxide concentration	$\frac{\mathrm{mg}}{{\mathrm{m}}^3}$
$\mathit{v}$	The outdoor data
$T_{\mathrm{out}}$	The temperature of outdoor air	${}^{\circ }\mathrm{C}$
$H_{\mathrm{out}}$	The absolute humidity of outdoor air	$\frac{\mathrm{g}}{{\mathrm{m}}^3}$
$I_{\mathrm{glob}}$	The light intensity	$\frac{\mathrm{W}}{{\mathrm{m}}^2}$
${\mathrm{CO}}_{2,\mathrm{out}}$	Outdoor carbon dioxide concentration	$\frac{\mathrm{mg}}{{\mathrm{m}}^3}$
$V_{\mathrm{wind}}$	Outdoor wind speed	$\frac{\mathrm{m}}{\mathrm{s}}$
$\mathit{u}$	The controller input vector
$u_{\mathrm{vent}}$	Skylight opening output
$u_{\mathrm{pad}}$	The wind speed scale of the wet curtain fan
$u_{scr}$	Opening output of the shading screen
$Q_{\mathrm{vent}}$	The heat exchange by the skylight	$\frac{\mathrm{W}}{{\mathrm{m}}^2}$
${\rho }_{\mathrm{air}}$	The air density	$\frac{\mathrm{kg}}{{\mathrm{m}}^3}$
$C_{p,\mathrm{air}}$	The heat capacity	$\frac{\mathrm{J}}{\mathrm{K}·\mathrm{kg}}$
${\phi }_{\mathrm{vent}}$	The ventilation flow rate	$\frac{{\mathrm{m}}^3}{{\mathrm{m}}^2·\mathrm{s}}$
$\mathit{s}$	State of observation
$\mathit{a}$	Action space
$\Delta T_{\mathrm{air}}$	The temperature change of indoor air	${}^{\circ }\mathrm{C}$
$\Delta T_{\mathrm{out}}$	the temperature change of outdoor air	${}^{\circ }\mathrm{C}$
$t_{\mathrm{vent}}$	Time steps the skylight keeps one state
$t_{\mathrm{pad}}$	Time steps the fan keeps one state

below.

The mechanism in a practical environment provides qualitative knowledge, but suffers from difficulties in decoupling and parameter determination. The skylight heat exchange $Q_{\mathrm{vent}}\left[\frac{\mathrm{W}}{{\mathrm{m}}^2}\right]$ described in (3) is taken as an example. The explanation of symbols is shown in Table 1.

$$\begin{array}{c}\hfill Q_{\mathrm{vent}}={\rho }_{\mathrm{air}}{C}_{p,\mathrm{air}}{\phi }_{\mathrm{vent}}\left(T_{\mathrm{air}}-T_{\mathrm{out}}\right)\end{array}$$

(3)

It is clear from this mechanism that a skylight can be opened to cool the room when the outdoor temperature is suitable, but it is difficult to quantify precisely the range of ‘suitable’ by summarizing this knowledge as a quantitative rule. In addition, the wet curtain fan forcibly draws out the indoor air by starting the fan to cause negative pressure, so that the curtain wall is wetted by cold water. This allows the airflow to pass quickly and carries moisture into the room in order to evaporate heat absorption to achieve cooling [32]. Generally, the cooling effect is related to the ventilation rate of the wet curtain fan, outdoor temperature and humidity, and the humidity difference between the wet curtain and the outdoor humidity. Therefore, high outdoor air humidity can cause poor cooling effects using wet curtain fans. The interaction of fans and skylights on the greenhouse environment easily gives rise to unnecessary frequent actions and energy consumption waste, making it difficult to formulate control parameters under multiple actuators and controlled factors. Considering that the main energy consumption of the wet curtain fan is generated by the fan that operates for a long time, while the skylight only generates a small amount of energy consumption in the opening and closing stroke, our goal is to reduce the fan’s operation time and the change in sunroof control quantity as much as possible within the reasonable range of temperature and humidity control errors. The specific optimization goal is as follows (4):

$$\begin{array}{c}\hfill J={\int }_0^T{u}_{\mathrm{pad}}\left(t\right)dt+\sum _{i=0}^T\Delta u_{\mathrm{vent}}\end{array}$$

(4)

The form of the problem to be optimized is shown in (5), where ${\mathit{c}}_{\mathit{j}}$ refers to the microclimate suitability and the actuator safety constraints.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill & \underset{u}{min}J\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \mathit{F}\left(\mathit{x}\right)-\mathit{f}(\mathit{x},\mathit{v})-\mathit{G}(\mathit{x},\mathit{v})\mathit{u}=0\hfill \\ \hfill & {\mathit{c}}_{\mathit{j}}(\mathit{x},\mathit{u},\mathit{v})\le 0, \mathrm{j}=1,2,\dots ,\mathrm{n}\hfill \end{array}\end{array}$$

(5)

2.2. Algorithm

In a practical application of reinforcement learning, if an invalid action can be predicted in the current state, the Action Mask can be used to prune it to accelerate learning [33,34,35]. However, this method sets the probability of some actions directly to zero under particular situations, so too many hard rules result in too many limitations on exploration. It is difficult to design a set of perfect rules and parameters for a greenhouse control problem with complexly coupled factors, but empirically, some fuzzy conditions from experience can be summarized to determine whether a particular facility is currently suitable for use. Our method proposes the design of a network of masking rules that can be manually initialized by partially empirical rules and that can be updated with parameters based on feedback by a policy gradient algorithm during reinforcement learning, which was referred to as SAM.

2.2.1. Overview

The overall algorithmic architecture of the proposed method is shown in Figure 2. The agent includes the Actor and the Critic, and the Actor includes two networks: one fully connected feedforward neural network, which is originally randomly initialized, and one soft-action-mask network (in Section 2.2.2), which is manually initialized according to the rules. The output of the SAM network, which is a bias layer, biases the probability of the logits layer of the original neural network before the softmax activation function, suppressing the sampling of low-benefit actions and obtaining the final policy ${\pi }_{\theta }(a\mid s)$, which is the probability of each action and where $\theta$ is the parameter of completed Actor-network.

The form of each variable in Figure 2 is shown in (6), where $\mathit{x},\mathit{v}$, and $\mathit{u}$ are the same as in Equation (2). $\Delta T_{\mathrm{air}}\left({}^{\circ }\mathrm{C}\right)$ and $\Delta T_{\mathrm{out}}\left({}^{\circ }\mathrm{C}\right)$, observing the trend and the intensity of indoor and outdoor temperature change, denote the difference between the sampled temperature at the moment $\mathit{t}$ and its previous moment. $t_{\mathrm{vent}},t_{\mathrm{pad}}$ are the time steps in which one state of the facilities lasts. $\mathit{a}$ is the output of the Actor with dimension $o_s$, i.e., the set of actions to be selected by RL agent policy, and the final selected action $a_i$ is sampled from the policy distribution.

$$\begin{array}{c}\hfill \left\{\begin{array}{c}\mathit{s}={\left[{\mathit{x}}^T,{\mathit{v}}^T,{\mathit{u}}^T,\Delta T_{\mathrm{air}},\Delta T_{\mathrm{out}},t_{\mathrm{vent}},t_{\mathrm{pad}}\right]}^{\mathit{T}}\\ \mathit{u}={\left[u_{\mathrm{pad}},u_{\mathrm{vent}}\right]}^{\mathit{T}}\\ \mathit{a}={\left\{a_i\right\}}_{i=0,1,\dots ,o_s-1}\end{array}\right.\end{array}$$

(6)

$$\begin{array}{c}\hfill \mathrm{where}\left\{\begin{array}{c}\Delta T_{\mathrm{out}}=T_{\mathrm{out},t}-T_{\mathrm{out},t-1}\hfill \\ \Delta T_{\mathrm{air}}=T_{\mathrm{air},t}-T_{\mathrm{air},t-1}\hfill \end{array}\right.\end{array}$$

After the action decoupling module, $\mathit{a}$ is converted into the facilities’ control input quantity $\mathit{u}$. The agent continuously selects the action according to the current policy Actor and observes the state of the environment to collect the reward information from the historical feedback, which is put into the experience recovery pool of the PPO Update Loss module in Figure 2. After several rounds, based on the PPO algorithm [36], the agent’s parameters are updated.

The PPO algorithm is based on the Actor–Critic framework [37] and invokes the concept of importance sampling to enhance efficiency. The loss function for policy optimization based on the advantage function is as shown in (7), where $\widehat{A}(s,a)$ is the estimated advantage. Forcing a constraint on the ratio of old to new probabilities in the update, the new loss function is described as in (8) and (9), where $r\left(\theta \right)$ is the ratio of the probability distribution of the updated policy ${\pi }_{\theta }$ to the old policy ${\pi }_{{\theta }_{\mathrm{old}}},\epsilon$ is a small positive constant close to zero, and the final loss function is as in (10), considering the error and entropy terms:

$$\begin{array}{c}\hfill L^{PG}\left(\theta \right)=E_{\pi }\left[ln{\pi }_{\theta }(a\mid s)\widehat{A}(s,a)\right]\end{array}$$

(7)

$$\begin{array}{c}\hfill L^{CLIP}\left(\theta \right)=E_{\pi }\left[min\left(r\left(\theta \right)A_{{\theta }_{\mathrm{old}}}(s,a),clip(r\left(\theta \right),1-\epsilon ,1+\epsilon )\right){\widehat{A}}_{{\theta }_{\mathrm{old}}}(s,a)\right]\end{array}$$

(8)

$$\begin{array}{c}\hfill r\left(\theta \right)=\frac{{\pi }_{\theta }(a\mid s)}{{\pi }_{{\theta }_{\mathrm{old}}}(a\mid s)}\end{array}$$

(9)

$$\begin{array}{c}\hfill L^{CLIP+VF+S}\left(\theta \right)=E_{\pi }\left[L^{CLIP}\left(\theta \right)-c_1{\left(V_{\theta }\left(S\right)-V_{\mathrm{target}}\right)}^2-c_{2}H\left(s,{\pi }_{\theta }\right)\right]\end{array}$$

(10)

When updating the parameters, the Actor estimates the advantage function by interactively collecting sample data over K time steps in (11), where $\lambda$ is the discount factor of advantage, $\gamma$ is the reward discount, ${\delta }_t$ is the TD error, $r_t$ is the immediate reward at t, and V is the state value calculated based on the Critic network. $V_{\mathrm{target}}$ is the actual long-term cumulative benefit, usually estimated as (12):

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\widehat{A}}_t(s,a)={\delta }_t+\left(\gamma \lambda \right){\delta }_{t+1}+\dots +{\left(\gamma \lambda \right)}^{K-t+1}{\delta }_{K-1}\\ {\delta }_t=r_t+\gamma V\left(s_{t+1}\right)-V\left(s_t\right)\end{array}\right.\end{array}$$

(11)

$$\begin{array}{c}\hfill V_{t\mathrm{arget}}={\widehat{A}}_t(s,a)+V\left(s_t\right)\end{array}$$

(12)

The PPO algorithm flow pseudo-code is shown in Algorithm 1. The symbols and parameters involved are explained in

Table 2. Explanation of symbols in formulas.

Symbol	Explanation
${\pi }_{\theta }$	The updated policy
${\pi }_{{\theta }_{\mathrm{old}}}$	The old policy
$\epsilon$	A small positive constant close to zero
$\mathrm{N}$	Total numbers of iterations
$\mathrm{K}$	Update every K time steps
$\mathrm{Z}$	Optimize the parameters for Z epochs every K steps
$\lambda$	The discount factor of advantage
$\gamma$	The reward discount
${\delta }_t$	The TD error
$r_t$	The immediate reward at t
V	The state value calculated based on the Critic network
$V_{\mathrm{target}}$	The actual long-term cumulative benefit
${\sigma }_i$	The output of the judgment nodes
$w_i$, $c_i$	The initialization weights and bias parameters of the judgment nodes, respectively
${\alpha }_i$	The degree of uncertainty of the discriminant conditions
${\mathit{P}}_{\mathit{n}\times \mathit{k}}$	The path matrix initialized by k discriminative paths composed of n different conditions in series according to k rules
${\mathit{M}}_{\mathit{o}\mathit{u}\mathit{t}}$	The final output of the bias layer after SAM
$I_s$	The dimension of observation input space
$O_s$	The dimension of action output space

.

Algorithm 1 PPO Algorithm

Input: total numbers of iterations N, max steps per round K, replay buffer pool of history experience $RB$, policy before update ${\pi }_{{\theta }_{old}}$, update epoch Z

1: for $episode=1,2,\cdots ,N$ do

2:       for $step=1,2,\cdots ,K$ do

3:             Get observation state s from environment

4:             Choose action a by policy ${\pi }_{{\theta }_{old}}$

5:             Run a in environment

6:             Observe reward r and new state $s^{{}^{\prime }}$

7:             Add experience information {s,a,r,$s^{{}^{\prime }}$} into $RB$

8:       end for

9:       Calculate advantage $\widehat{A_1}$, $\widehat{A_2}$,…, $\widehat{A_K}$, with $RB$ based on (11)

10:     Optimize loss function (8–10) and (12) with respect to $\theta$ for Z epochs

11:     Update ${\theta }_{old}$ to $\theta$

12: end for

2.2.2. Soft Action-Mask Net

In the study [34], it was mentioned that the use of action masks allows the pruning of invalid actions. It was demonstrated that this process can be regarded as a state-dependent differentiable function [35], which is in line with the assumptions of the Policy Gradient Algorithm. In the greenhouse, there are also several invalid actions that can be predicted. Significantly, exploration in a real greenhouse requires avoiding the execution of dangerous actions in particular states in order to ensure the safety of the crops and the facilities.

To distinguish it from the SAM proposed later, this type of action mask for dangerous actions in greenhouses is referred to as a Hard Action Mask (HAM). Take action $\mathit{a}=$$\left\{a_0,a_1,a_2,a_3\right\}$ for an example: if $a_3$ corresponds to the action of turning on the fan, it is not advisable to choose this action in low temperatures according to human experience. Assuming that state $s_0$ is in a low temperature, the logits output of the Actor is $l=\left[l_0,l_1,l_2,l_3\right]=[1.0,1.0,1.0,1.0]$ and the policy network output transformed into action sampling probabilities after softmax activation is ${\pi }_{\theta }\left(\mathit{a}\mid {\mathit{s}}_{\mathbf{0}}\right)$:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\pi }_{\theta }\left(\mathit{a}\mid {\mathit{s}}_{\mathbf{0}}\right)& =\left[{\pi }_{\theta }\left(a_0\mid {\mathit{s}}_{\mathbf{0}}\right),{\pi }_{\theta }\left(a_1\mid {\mathit{s}}_{\mathbf{0}}\right),{\pi }_{\theta }\left(a_2\mid {\mathit{s}}_{\mathbf{0}}\right),{\pi }_{\theta }\left(a_3\mid {\mathit{s}}_{\mathbf{0}}\right)\right]\hfill \\ \hfill & =softmax\left(\left[l_0,l_1,l_2,l_3\right]\right)\hfill \\ \hfill & =[0.25,0.25,0.25,0.25]\hfill \end{array}\end{array}$$

$$\begin{array}{c}\hfill \mathrm{where}{\pi }_{\theta }\left(a_i\mid {\mathit{s}}_{\mathbf{0}}\right)=\frac{e^{l_i}}{{\sum }_{i=0}^{o_s}{e}^{l_i}}\end{array}$$

(13)

The HAM sets the logits corresponding to action $a_3$ to an extremely small value, Min, and the probability of selecting $a_3$ will be set to zero as:

$$\begin{array}{c}\hfill {\pi }_{\theta }^{\prime }\left(\mathit{a}\mid {\mathit{s}}_{\mathbf{0}}\right)=softmax\left(\left[l_0,l_1,\mathrm{Min},l_3\right]\right)=[0.33,0.33,0,0.33]\end{array}$$

(14)

HAM ensures the safe process by completely setting the sampling probability of unsafe or invalid actions to zero. However, exploration in the safe region does not guarantee good gains for all action trajectories. Furthermore, in addition to dangerous actions, low benefits in the early stages of learning are also undesirable, and initialization of a high level can effectively shorten the learning period. Actually, in realistic scenarios, instructive human knowledge can be provided beforehand. To combine knowledge with reinforcement learning, ref. [30] proposed a method of initializing neural networks in the form of decision trees to implement a warm start. However, the human experience summarized in greenhouses is usually ambiguous, and it is hard to cover the complete state domain. In addition, the multi-factor coupling relationship results in a complex of conditional rules, leading to difficulty in applying the results of [30], where the initialization of the decision tree is too complicated and the over-regulated exploration restricts boosting capabilities of the agent. We propose a method transforming several ruled soft masks into a network form and incorporating partial experience to guide the agent’s exploration process, finally achieving reinforcement learning of better initialization for greenhouse control. The networked form is shown in Figure 3.

By manually initializing the network weights and comparators, the judicial process with combined first-order rules is transformed through the network into a bias layer that reduces the probability of sampling improper actions, which is referred to as SAM. Unlike the direct masking of HAM, SAM adds a bias layer to the output of the logits layer of the original policy network of the agent. The offset base value is initialized to a relatively small value, avoiding full suppression of the policy gradient backpropagation found in HAM. Secondly, the rules set by manual experience are sometimes vague and one-sided. The networked SAM allows the weights and offsets of the rule nodes in the network to be dynamically trained through the gradient, allowing the intelligent agent to explore better policy and rule parameters through the reward mechanism.

The network initialization requires manual pre-establishment of the masking rules, and the rules build the network in the form of weights and comparators. The initialization process is Algorithm 2. Assuming a total of $\mathrm{k}$ rules consisting of $\mathrm{n}$ first-order discriminant conditions, as in Figure 3, ${\sigma }_i$ is the output of the judgment nodes, $w_i$ and $c_i$ are the initialization weights and bias parameters of the judgment nodes, respectively, and ${\alpha }_i$ is the degree of uncertainty of the discriminant conditions, taken from 0 to 1. Equations (15)–(21) comprise the calculation process from the input to the bias layer, where ${\mathit{P}}_{\mathit{n}\times \mathit{k}}$ is the path matrix initialized by $\mathrm{k}$ discriminative paths composed of n different conditions in series according to k rules, and ${\mathit{M}}_{\mathit{o}\mathit{u}\mathit{t}}$ is the final output of the bias layer after SAM.

Algorithm 2 Initialize of SAM net

Input: Expert knowledge rules collection $R_{ini}$ = $\left\{r_j\right\}$, numbers of rules k, count of all conditions n

1: initialize collection of path $\mathit{P}$ = $\left\{\right\}$

2: initialize collection of weight $\mathit{W}$ = $\left\{\right\}$

3: initialize collection of comparison $\mathit{c}$ = $\left\{\right\}$

4: initialize collection of mask $\mathit{M}$ = $\left\{\right\}$

5: for $j=1,2,\cdots ,k$ do

6:      ${\mathit{p}}_{\mathit{j}}$ = $\left\{\right\}$

7:      ${\mathit{m}}_{\mathit{j}}$ = $\left\{\right\}$

8:      for $i=1,2,\cdots ,n$ do

9:            if condition $C_i\in r_j$ then

10:              $\mathit{W}=\mathit{W}\cup {\mathit{w}}_{\mathit{i}}$

11:              $\mathit{c}=\mathit{c}\cup c_i$

12:              initialize $p_{ij}$ by (17)

13:              ${\mathit{p}}_{\mathit{j}}={\mathit{p}}_{\mathit{j}}\cup p_{ij}$

14:          end if

15:      end for

16:      $\mathit{P}=\mathit{P}\cup {\mathit{p}}_{\mathit{j}}$

17:      initialize ${\mathit{m}}_{\mathit{j}}$ by (20-21)

18:      $\mathit{M}=\mathit{M}\cup {\mathit{m}}_{\mathit{j}}$

19: end for

$$\begin{array}{c}\hfill \sigma ={\left\{{\sigma }_i\right\}}_{i=1,..,n}={\left\{\mathbf{relu}\left[{\alpha }_i\left(w_i^{\mathbf{T}}\mathit{x}-c_i\right)\right]\right\}}_{i=1,\dots ,n}=\mathbf{relu}\left[{\alpha }^{\mathbf{T}}\left({\mathit{W}}^{\mathbf{T}}\mathit{x}-\mathit{c}\right)\right]\end{array}$$

(15)

$$\begin{array}{c}\hfill {\mathit{B}}_{\mathit{n}\times \mathit{k}}={\left\{b_{ij}\right\}}_{\mathit{n}\times \mathit{k}}=Filter\left({\left[{\sigma }_i\right]}_{i=1,\dots ,n}·{\mathit{P}}_{\mathit{n}\times \mathit{k}}\right)\end{array}$$

(16)

$$\begin{array}{c}\hfill \mathrm{where}\left\{\begin{array}{c}{\mathit{P}}_{\mathit{n}\times \mathit{k}}=\left\{p_{ij}\right\}=\left\{\begin{array}{c}1,\mathrm{condition}\mathrm{i}\mathrm{in}\mathrm{Rule}j\\ 0,\mathrm{otherwise}\end{array}\right.\\ \mathrm{Filter}\left(\mathbf{X}\right)=\left\{\begin{array}{c}1,x_{ij}=0\\ x_{ij},\mathrm{otherwise}\end{array}\right.\end{array}\right.\end{array}$$

(17)

$$\begin{array}{c}\hfill \mathit{B}{\mathit{M}}_{\mathbf{1}\times \mathit{k}}={\left[\mathit{B}{\mathit{M}}_{\mathit{j}}\right]}_{\mathbf{1}\times \mathit{k}}=\left[\mathbf{tanh}\left(\prod _i^n{b}_{ij}\right)\right]\end{array}$$

(18)

$$\begin{array}{c}\hfill {\mathit{M}}_{\mathit{out}}=\left[{\mathit{m}}_{\mathbf{1}},\dots ,{\mathit{m}}_{\mathit{j}},\dots ,{\mathit{m}}_{\mathit{k}}\right]·\mathit{B}{\mathit{M}}_{\mathbf{1}\times \mathit{k}}^{\mathbf{T}}\end{array}$$

(19)

$$\begin{array}{c}\hfill {\mathit{m}}_{\mathit{j}}={\left[m_{o_d}\right]}_{1\times o_s}=\left[m_0,m_1,\dots ,m_{o_s}\right]\end{array}$$

(20)

$$\begin{array}{c}\hfill \mathrm{where}{m}_{o_d}=\left\{\begin{array}{c}M,\mathrm{if}\mathrm{mask}\mathrm{action}\mathrm{o}\_\mathrm{d}\mathrm{in}\mathrm{Rule}j\\ 0,\mathrm{otherwise}\end{array}\right.\end{array}$$

(21)

Still considering the fan action, suppose there are rules as follows: when the indoor temperature is below 30 degrees Celsius and the outdoor temperature is below 28, try not to turn on the fan. State $\mathrm{s}=\left[T_i,T_o,H_i,H_o\right]$, action space $a=\left\{a_0,a_1,a_2,a_3\right\}$, where $a_3$ corresponds to turning on the fan. Initialize $C_1:T_i<30\to \left\{\begin{array}{c}{w}_1=[-1,0,0,0]\\ c_1=-30\end{array},C_2:T_o<28\to \right.$$\left\{\begin{array}{c}{w}_2=[0,-1,0,0]\\ c_2=-28\end{array}\right.$, and ${\mathit{m}}_{\mathbf{1}}:[0,0,0,\mathrm{M}]$. When the condition is not met, the offset is 0, otherwise the larger the absolute value of the offset, the higher the degree of inhibition of the action, with less inhibition near the critical value of the judgment condition.

2.2.3. Region Reward

The reward function consists of two main components: a control interval error penalty $R_{err}$ and a fan loss penalty $R_{fan\_on}$, as shown in Equations (22) and (23), respectively. $c_{\mathrm{err}},c_{fan\_on},c_{freq}$ are the weights for each, and ${\mathit{x}}_{set}$ indicates the target, in which $x_{set}$ is the target for each factor:

$$\begin{array}{c}\hfill R_{err}=c_{err}{\left|\mathit{x}-{\mathit{x}}_{set}\right|}^2\end{array}$$

(22)

The fan loss penalty $R_{fan\_on}$ is an indirect measure of fan energy consumption based on the cumulative hours of the fan running, T is the data sampling period, and the element value $x_{\mathrm{set}}$ is determined based on the current state and the upper and lower limits $x_{\mathrm{high}\_\mathrm{com}}$ and $x_{\mathrm{low}\_\mathrm{com}}$ of suitability, as in (24):

$$\begin{array}{c}\hfill R_{\mathrm{fan}\_\mathrm{on}}=c_{\mathrm{fan}\_\mathrm{on}}{\int }_0^T{u}_{\mathrm{fan},\mathrm{t}}dt\end{array}$$

(23)

$$\begin{array}{c}\hfill x_{set}=\left\{\begin{array}{cc}{x}_{\mathrm{high}\_\mathrm{com}},\hfill & x>x_{\mathrm{high}\_\mathrm{com}}\hfill \\ x_{\mathrm{low}\_\mathrm{com}},\hfill & x<x_{\mathrm{low}\_\mathrm{com}}\hfill \end{array}\right.\end{array}$$

(24)

Depending on the crop’s growth requirements and tolerance, three levels of condition areas are distinguished. ${\Re }_{\mathrm{comf}}$ is an excellent environmental condition for growth, ${\Re }_{\mathrm{toleran}}$ is a sub-optimal condition that can be tolerated for a long period, and ${\Re }_{\mathrm{extreme}}$ is an extreme condition that can only be tolerated for a short period. If the environment has difficulty in running at ${\Re }_{\mathrm{comf}}$ with the lack of precise control equipment, then running in ${\Re }_{\mathrm{toleran}}$ is expected and ${\Re }_{\mathrm{extreme}}$ should be avoided at all costs. The setting of the penalty term is therefore differentiated by the current state, as shown in Equation $\left(25\right)$:

$$\begin{array}{c}\hfill R=\left\{\begin{array}{ccc}{R}_{\mathrm{fan}\_\mathrm{on}}+R_{\mathrm{freq}}& ,\hfill & x\subset {\Re }_{\mathrm{comf}}\hfill \\ R_{\mathrm{err}}+R_{\mathrm{fan}\_\mathrm{on}}+R_{\mathrm{freq}}& ,\hfill & x\subset {\Re }_{\mathrm{toleran}}\hfill \\ R_{\mathrm{err}}+p& ,\hfill & x\subset {\Re }_{\mathrm{extreme}}\hfill \end{array}\right.\end{array}$$

(25)

The complete RL with SAM algorithm flow is shown in Algorithm 3.

Algorithm 3 RL with Soft Action-Mask

Input: Dimension of state observation $I_s$, dimension of action $O_s$, expert knowledge $R_{ini}$, total numbers of iterations N, max steps per round K, replay buffer pool of history experience $RB$, update epoch Z

1: randomly initialize actor and critic parameters in Agent

2: initialize mask net in Agent with $R_{ini}$ by Algorithm 2

3: for $episode=1,2,\dots ,N$ do

4:       for $step=1,2,\dots ,K$ do

5:             get observation state $\mathit{s}$ from environment

6:             get logits layer $\mathit{l}$ forward the actor net

7:             get bias ${\mathit{M}}_{\mathit{out}}$ forward the soft mask net by (15)–(21)

8:             ${\pi }_{{\theta }_{old}}\left(\mathit{a}\right|\mathit{s})=softmax(\mathit{l}+{\mathit{M}}_{\mathit{out}})$

9:             choose action a by policy ${\pi }_{{\theta }_{old}}\left(\mathit{a}\right|\mathit{s})$

10:           run a in environment

11:           observe new state ${\mathit{s}}^{{}^{\prime }}$ and calculate reward r by (22)–(25)

12:           add experience information {$\mathit{s}$,a,r,${\mathit{s}}^{{}^{\prime }}$} into $RB$

13:       end for

14:       update ${\theta }_{old}$ to $\theta$ for Z epochs by Algorithm 1

15: end for

3. Experimental Results and Discussion

A series of comparative experiments were conducted in a three-state simulation greenhouse model. The hyperparameters for the reinforcement learning process were selected as follows: $I_s=$$11,O_s=3,Z=10,N=1600,K=288$, and learning rate = 6 × 10${}^{-4}$.

3.1. RL Efficiency

Firstly, in terms of reward enhancement, the comparison methods included RL with a reward function alone, RL with HAM, and our proposed RL with SAM. Five learning sessions were performed for each learning method, and the results of recording their mean reward profile and standard deviation are shown in Figure 4.

SAM and HAM have significantly higher initial rewards compared to purely learning by reward, as both HAM and SAM suppress adverse actions in a given state based on existing experience. The SAM trajectory in the figure has a slightly higher initial reward than the HAM, due to the fact that the SAM rules can be optimized during the learning process, allowing for more leeway when initializing the masking rules. In addition, it can be seen from the late training period that the average reward level of HAM is surpassed by the pure reward learning method that is allowed to explore randomly. Although the pure reward method shows potential in the later stages, the the range shows that its efficiency and effectiveness are influenced by the initialized parameters and the randomness of sampling during training, which is difficult to accept in the early stages. SAM, on the other hand, is based on rule-based initialization and is expected to start sub-optimally with knowledge. Compared to learning from rewards alone, SAM achieves a significant improvement upfront and stabilizes at a superior level of performance. In addition, the exploration is based on a soft rule framework with a small standard deviation of rewards, which is more stable than the original approach.

3.2. Effect Comparison

The strategies obtained by the three reinforcement learning methods were validated on 30 days of test data and compared with the threshold control strategy. As the threshold control strategy has different control effects when different thresholds are selected, two sets of thresholds were implemented in the experiment for comparison. The results are shown in Figure 5, which compares the control effect in terms of the percentage of the total fan running time, the time percentage when the temperature and humidity are out of the appropriate range, and the percentage when the skylight is in frequent operation.

In the threshold control strategy with two sets of parameters, the more aggressive one for maintaining a suitable environment is less likely to result in frequent operation, but at the same time has a longer fan running time, accounting for 26.35% and 23.82% of the total control process time, respectively. The effect of adjusting the threshold parameters to reduce consumption is very limited and leads to frequent movements. Compared to the threshold control, the reinforcement learning method is effective in reducing the runtime, but the original reward RL cannot completely avoid frequent actions, which may be related to the reward weight setting as well as the limited generalization ability and low learning efficiency. With the addition of the Action Mask, frequent actions were completely avoided and unsuitable states were effectively reduced. The policy trained by SAM resulted in better temperature suitability. Although the frequency of out-of-range humidity was slightly higher than HAM, it remained at a lower level. What is more important is that the SAM fan runtime was only 16.56% of the total process, a reduction of 37.15% and 30.47% compared to the threshold control policy with two sets of parameters, and 22.07% compared to the HAM method with the same training rounds, significantly reducing the fan runtime while maintaining better environmental suitability.

3.3. Performance Comparison

3.3.1. Threshold Control vs. RL

Temperature and humidity controlled by threshold control, the HAM and SAM training policies, and the corresponding control output $u_{\mathrm{pad}},u_{\mathrm{vent}}$ over five days are shown and compared in Figure 6; the sampling period $T_s$ is 900 s.

It is clear that SAM and HAM effectively shorten the fan running time and that there are no frequent movements caused by improperly set threshold parameters, resulting in energy-saving optimization. In a few cases, the threshold control effect resulted in temperatures outside the appropriate range, due to large fluctuations in temperature caused by frequent actions, whereas the temperatures under both HAM and SAM policy control were maintained within the appropriate range. The study was conducted in a summer and autumn greenhouse. The data did not appear to be below the lower limit, so only the upper appropriate limit is plotted on the graphs. In fact, the temperatures under the HAM and SAM policies are sometimes slightly higher than those under the threshold control, since the regional reward is within the appropriate range for the factor, i.e., it penalizes mainly energy consumption, sacrificing some cooling effect in return for a reduction in fan operating hours. In a few cases, the RL method may exceed the upper limit of the suitable range because the temperature target in the reward is weighted higher than the humidity, while a short time spent out of the suitable range has less impact on the long-term benefits, and it quickly returns to the suitable area, which can be considered within the control target allowed.

3.3.2. HAM vs. SAM

SAM tends to have lower temperature peaks than HAM, slightly lower humidity curve exceedances than HAM, more stable control overall, and shorter fan run times compared to HAM. These benefits are considered to be mainly related to the trainable adjustment capability of SAM, where setting a masking rule in HAM prevents the exploration of policy under some condition state, whereas SAM can still learn a potentially better condition judgment than human experience. The second possible reason is SAM is more efficient than HAM, and it is more likely to learn better policies in the same amount of time.

In conclusion, through simulated environment experiments, it has been proven that the method provided has significant advantages over RL with pure rewards or HAM in terms of improving the levels of the initialization policy, learning efficiency, stability, and potential of optimization effect. Starting exploration from a safe and sub-optimal policy, our algorithm regulates the greenhouse environment with a lower initial cost, further reducing energy consumption and preventing facilities from running frequently due to environmental changes during the control process.

4. Conclusions

This work put forward a SAM net initialized on rules to start reinforcement learning to cope with the problem of energy conservation control in a Venlo-type greenhouse. It combines the interpretability of knowledge and rules with the strong learnability of a randomly initialized fully connected network, guiding exploration and ensuring the warm start of the policy, the safety of the training process, and the potential of policy exploration.

The advantages of the method proposed in this paper are summarized as follows:

Model-free dynamic optimization: Without model control objects, the agent dynamically optimizes policies through interaction with the environment and feedback on long-term benefits.

Safe Exploration: By adopting action masks, unsafe and unstable actions under specific conditions can be avoided. Moreover, energy consumption for the motor operation of the three-state actuator can be indirectly reduced.

Exploration guided by human knowledge: Random exploration in the early stage of learning results in a high cost of trial-and-error to the actual greenhouse application. The proposed approach embeds part of human knowledge into the randomly initialized policy network, ensuring that training starts from a higher level, and the learning process is constrained by the manually initialized rule framework, which effectively guides the stable and safe learning of agents.

Trainable networked rules of masks: Considering the difficulties in establishing complete rules in the greenhouse and determining rules that are disadvantageous for in-depth exploration, the proposed method constructs networked rules on the basis of fuzzy knowledge, and the initialized parameters can be updated by the policy gradient algorithm during the learning process, so as to intelligently adjust more appropriate rules. This approach combines the interpretability of rules and the learnability of neural networks.

The proposed method is one of generalization, especially in large, precisely controlled greenhouses, where energy-saving optimization has greater value for improving economic benefits. In addition, the rich data available can provide more feedback information, which greatly benefits control based on reinforcement learning. As a consequence, this method is worthy of further in-depth study and has a long-term significance in improving the economic benefits of modern agricultural facilities.