A New Fuzzy Reinforcement Learning Method for Effective Chemotherapy

Abstract

A key challenge for drug dosing schedules is the ability to learn an optimal control policy even when there is a paucity of accurate information about the systems. Artificial intelligence has great potential for shaping a smart control policy for the dosage of drugs for any treatment. Motivated by this issue, in the present research paper a Caputo–Fabrizio fractional-order model of cancer chemotherapy treatment was elaborated and analyzed. A fix-point theorem and an iterative method were implemented to prove the existence and uniqueness of the solutions of the proposed model. Afterward, in order to control cancer through chemotherapy treatment, a fuzzy-reinforcement learning-based control method that uses the State-Action-Reward-State-Action (SARSA) algorithm was proposed. Finally, so as to assess the performance of the proposed control method, the simulations were conducted for young and elderly patients and for ten simulated patients with different parameters. Then, the results of the proposed control method were compared with Watkins’s Q-learning control method for cancer chemotherapy drug dosing. The results of the simulations demonstrate the superiority of the proposed control method in terms of mean squared error, mean variance of the error, and the mean squared of the control action—in other words, in terms of the eradication of tumor cells, keeping normal cells, and the amount of usage of the drug during chemotherapy treatment.

1. Introduction

Cancer is one of the most hazardous and fatal diseases throughout the world [1]. This disease is caused by the abnormal division and spreading of cells that destroy the patient’s body and may lead to death. Though many research efforts are devoted to precisely understanding the interaction between the immune system and tumor cells, the treatment of cancer is still one of the most challenging issues in modern medicine [2,3].

Based on the patient’s conditions and type of cancer, there are several treatments, including radiotherapy, surgery, chemotherapy, immunotherapy, and so forth, to tackle cancer [4]. Cancer treatment can be challenging, as it can have a range of side effects, including fatigue, nausea, and hair loss. Managing these side effects is an important part of cancer care among various treatments. Chemotherapy is one of the most effective treatments for annihilating cancerous cells. For this reason, in this research the chemotherapy method was chosen to treat cancer. Nonetheless, in all chemotherapy treatments, not only do the applied drugs destroy the cancerous cells, but they also affect the healthy cells and kill some of those cells. Hence, it is crucial to make sure that the patient can tolerate the side effects of the treatment [5]. These sorts of side effects lead to a limitation in the dosage of the drugs [6]. Hence, for prescribing drugs, one aim is to decrease the number of cancerous cells as much as possible and reach the minimum side effects [7].

There are many factors which define the treatment schedule and drug dosing. Some of these factors are the weight of the patient, level of white blood cells, the patient’s age, the stage of the tumor, etc.; for this reason, certain established standards are followed by clinicians to define the therapy type and drug dosage for each patient. Howbeit, this approach has some limitations which have been approved by scientific and clinical communities [8,9]. Moreover, evaluating the effectiveness of the chemotherapy plan and its feasibility is of significant importance [9]. So as to evaluate the effectiveness of the chemotherapy plans, the clinical trials are a reliable choice, but they have some limitations such as high costs, long trial times, and they are difficult to be conducted. Keeping the above-mentioned limitations in mind, contriving an efficient chemotherapy treatment plan would be desired.

Mathematical models have been used as valuable tools in understanding the transmission dynamics of cancers [10]. Actually, mathematical modeling could play a pivotal role in the understanding of the disease’s dynamics [11,12,13,14,15]. One common approach to modeling cancer is to use ordinary differential equations (ODEs), which are used to describe the time evolution of a system. These models can represent the growth and spread of cancer cells and the response of the immune system to the cancer. Researchers can then use these models to study the long-term and short-term dynamics of cancer and to identify potential therapeutic strategies. Finding an accurate model of cancer dynamics could pave the way for the long and short-term prediction of the disease as well as the designing of therapies [16]. This field of study not only aims to anticipate the spread of cancerous cells but also to control the disease as effectively as possible [17,18]. A lot of research has been done using mathematical modeling to understand cancer, and various approaches have been used in this field [19,20,21].

Fractional calculus is an excellent tool for the description of hereditary properties, and memory of systems has been widely utilized in various fields of study including economics, biology, ecology, and engineering [22,23,24,25,26,27,28]. In this regard, the fractional-order model of cancers has started to attract some researchers’ attention [29,30]. Although early research studies on fractional-order calculus were based on the Caputo or Riemann–Liouville fractional-order derivative, it has been recently shown that these approaches possess some disadvantages, such as the singularity at the endpoint of an interval of definition [31,32,33]. In tackling this issue, recently other definitions of fractional derivatives have been introduced [34,35,36,37,38,39]. Caputo and Fabrizio offered a new derivative with nonsingular kernel [32], namely the Caputo–Fabrizio derivative. The Caputo and Caputo–Fabrizio derivatives are both types of fractional derivatives, which are a generalization of the classical derivative to functions with non-integer order. The Caputo derivative is based on a power law, which means that it is defined in terms of a power of the function’s argument. The Caputo–Fabrizio derivative, on the other hand, is based on an exponential decay law, which means that it is defined in terms of the exponential decay of the function over time. One key difference between the two derivatives is that the Caputo derivative is defined in terms of a power of the function’s argument, while the Caputo–Fabrizio derivative is defined in terms of an exponential decay law. This means that the two derivatives may behave differently when applied to different types of functions, and they may be used to model different types of physical phenomena.

Caputo and Fabrizio offered a new derivative with nonsingular kernel [32], namely the Caputo-Fabrizio derivative. The principal difference between the Caputo and Caputo–Fabrizio fractional derivative is that the Caputo derivative is based on a power law, while the Caputo–Fabrizio derivative is achieved with an exponential decay law [32,40]. There are several research studies in the literature that demonstrate the applications of the Caputo–Fabrizio derivative to different systems such as biological processes [41,42,43]. In Ref. [44], the underlying physical meaning of the nonsingular kernel as well as the applications of fractional differentiation operators with the non-singular kernel were presented.

During for the past several years, many control strategies have been applied to cancer chemotherapy drug dosing.

In [45], targeted chemotherapy was offered for tumor-immune interaction. As it has been studied in [46,47], in addition to the mathematical interpretation of tumor growth, scheduling appropriate treatment strategies has been an important issue, and control-designing strategies can be utilized to optimize drug dosing mathematically. To this end, up to now, many research studies have worked on these subjects. De Pillis and Radunskaya proposed a dynamical model for immunotherapy and chemotherapy schedules in 2001 [48]. In 2003, to control tumor growth, bang-bang type control was employed with chemotherapy [47]. In 2007, De Pillis et al. used linear and quadratic control to a cure tumor with the use of chemotherapy [49]. Also, through Pontryagin’s maximum principle, Ghaffari and Naserifar have designed optimal therapeutic protocols to schedule immunotherapy [50]. On the other hand, the cost-effectiveness of the treatment strategies has been investigated in [51]. Drug resistance, which is a vital phenomenon in cancerous tumors, has been examined in [52]. In addition, the conditions for an siRNA treatment were investigated to eradicate tumor burden in [53], while a combination of chemotherapy and anti-angiogenic agents was utilized to cure the disease in [54]. In [55], robust adaptive control was presented to adjust the drug dosage with an extended Kalman filter observer. An optimal control strategy based on a linear time-varying approximation technique was proposed in [56]. A model-free method for chemotherapy based on reinforcement learning was proposed in [57] using the closed-loop control. To be precise, they develop an optimal controller using the Q-learning algorithm for cancer chemotherapy treatment. In [47], the authors model the chemotherapy treatment based on optimal control theory, and the objective was to minimize the tumor cells while keeping the healthy cells above a fixed level.

The use of state-of-the-art automatic control methods can help to improve the effectiveness and efficiency of various processes and systems, leading to better outcomes and more sustainable solutions [58,59,60,61,62,63,64,65,66,67]. Among the stated control strategies, intelligent controllers have significant advantages [68,69,70,71,72,73]. Artificial intelligence does combine a wide variety of new technologies to give systems an ability to make decisions in new and unfamiliar conditions [74,75]. In some applications, due to the high value of their tasks and their remarkable risks, implementing a reliable controller is the main concern. Where the degree of uncertainty is high, classical methods of control may fail [76,77]. Hence, artificial intelligence-based controllers are rational choices for such systems. Moreover, optimal controls which could be provided based on artificial intelligence are helpful for drug dosage because of their ability to consider various aspects of the biological systems in the optimization function. For example, through intelligent approaches, we are able to design treatments which can take to account the number of healthy and infected cells as well as the cost and side effects of the drugs without having any mathematical model. Reinforcement learning is one of the most popular learning techniques, which explores the system’s response in possible actions; this way, it learns the optimal action by calculating how the last action pushes the system towards desired situations [78,79,80].

As mentioned, fractional calculus is very advantageous to the modeling of real-world processes [81,82]. Motivated by this background, in the present study, we investigated a Caputo–Fabrizio model of cancer, which is a new model; studies are quite rare in these systems. Also, few studies are dedicated to the evaluation of reinforcement learning methods in chemotherapy drug dosing; to the best of our knowledge, no studies have been done on cancer chemotherapy drug dosing using fuzzy-reinforcement learning-based optimal control. These issues motivated the current study. In this paper, a Caputo–Fabrizio fractional-order model of cancer chemotherapy treatment was analyzed. The existence and uniqueness of the solutions of the presented model were proven. Then, a fuzzy-reinforcement learning-based control method was proposed in order to control the cancer chemotherapy treatment, and the effective performance of the proposed method was illustrated.

The main contributions of this research paper are as follows: First, a Caputo-Fabrizio fractional-order model of cancer chemotherapy treatment was elaborated and analyzed, which is a novel model for this system. Afterwards, a fix-point theorem and an iterative method were implemented to prove the existence and uniqueness of the solutions of the proposed model. Finally, as the last contribution, a fuzzy-reinforcement learning-based control method that uses the SARSA algorithm was proposed to control cancer through chemotherapy treatment.

The rest of the paper has been organized as follows: In Section 2 the preliminaries for this work are given. Later, in Section 3, the Caputo–Fabrizio fractional model of cancer chemotherapy is elaborated, and then sensitivity analysis is done for the parameters of the system. The fuzzy-reinforcement-learning based controller is described in Section 4. Section 5 is devoted to the numerical simulations; finally, in Section 6, conclusions are made and discussed.

2. Preliminaries

As has been shown, the kernels of the Caputo fractional derivative are singular at the endpoint of the interval of integration [83]; therefore, the fractional derivative is not an appropriate kernel to describe the memory effect accurately in real systems. A new fractional derivative has been proposed that has no singularity in its kernels [32]. This section is a summary of the definitions and properties of the Caputo–Fabrizio fractional that have been used in this paper.

Consider $H^1\left(a,b\right)=\left\{x\right|x\in L^2(a,b)\}$ and $x^{\prime }\in L^2(a,b)$, where $L^2(a,b)$ is the space of square integrable functions on interval of $(a,b)$.

Definition 1.

If function$x$is$H^1\left(t_0,t\right)$and derivative order$\alpha \in \left(\mathrm{0,1}\right)$, then the Caputo–Fabrizio derivative is defined as[32]:

$${}_{t_0}{}^{CF}{D}_t^{\alpha }x\left(t\right)=\frac{A\left(\alpha \right)}{1-\alpha }{\int }_{t_0}^t{x}^{\prime }\left(z\right)e^{-\alpha \frac{t-z}{1-\alpha }}dz,$$

(1)

where $A\left(\alpha \right)$ is a normalization function that satisfies $A\left(0\right)=A\left(1\right)=1$. Furthermore, if $x\notin H^1\left(a,b\right)$, then the Caputo–Fabrizio derivative is defined as:

$${}_{t_0}{}^{CF}{D}_t^{\alpha }x\left(t\right)=\frac{\alpha A\left(\alpha \right)}{1-\alpha }{\int }_{t_0}^t\left(x\left(t\right)-x\left(z\right)\right)e^{-\alpha \frac{t-z}{1-\alpha }}dz,$$

(2)

Remark 1.

[32]. By considering $\beta =\alpha /(1-\alpha )$ and $\beta \in \left(0,\infty \right)$, we have $\alpha =1/(1+\beta )$ which $\alpha \in \left(0,1\right)$. Now, Equation (2) can be written in the following form:

$${}_{t_0}{}^{CF}{D}_t^{\alpha }x\left(t\right)=\frac{B\left(\beta \right)}{\beta }{\int }_{t_0}^t{x}^{\prime }\left(z\right)e^{-\frac{t-z}{\beta }}dz,$$

(3)

where $B\left(\beta \right)$ is a normalization function corresponding to $A\left(\alpha \right)$ which $B\left(0\right)=B\left(\infty \right)=1$.

Remark 2.

[32]. The following equation is true:

$$\underset{\beta \to 0}{\lim }\frac{1}{\beta }{e}^{-\frac{t-z}{\beta }}=\delta \left(z-t\right),$$

(4)

where $\delta (z-t)$ is the Dirac delta function.

The modified definition of Caputo-Fabrizio has been proposed by Nieto and Losada [31], and it has been defined as

$${}_{t_0}{}^{CF}{D}_t^{\alpha }x\left(t\right)=\frac{(2-\alpha )A\left(\alpha \right)}{2(1-\alpha )}{\int }_{t_0}^t{x}^{\prime }\left(z\right)e^{-\alpha \frac{t-z}{1-\alpha }}dz,$$

(5)

Definition 2.

The Caputo–Fabrizio fractional integral of order$\mathit{\alpha }$offunction $x\left(t\right)$ is defined as:

$${}_0{}^{CF}{I}_t^{\alpha }x\left(t\right)=\frac{2\left(1-\alpha \right)}{\left(2-\alpha \right)A\left(\alpha \right)}x\left(t\right)+\frac{2}{\left(2-\alpha \right)A\left(\alpha \right)}{\int }_0^{t}x\left(z\right)dz,t\ge 0$$

(6)

Definition 3.

[31]. The fractional Caputo–Fabrizio derivative of order $\alpha$ of function $x\left(t\right)$ is defined as

$${}_{t_0}{}^{CF}{D}_t^{\alpha }x\left(t\right)=\frac{1}{1-\alpha }{\int }_{t_0}^t{x}^{\prime }\left(z\right)e^{-\alpha \frac{t-z}{1-\alpha }}dz,t\ge 0.$$

(7)

Moreover, the Caputo–Fabrizio fractional integral of order $\mathit{\alpha }$ is given as

$${}_0{}^{CF}{I}_t^{\alpha }x\left(t\right)=\left(1-\alpha \right)x\left(t\right)+\alpha {\int }_0^{t}x\left(z\right)dz,t\ge 0.$$

(8)

In this definition, the normalization function has been considered as$A\left(\alpha \right)=2/(2-\alpha )$.

3. Caputo–Fabrizio Fractional Model of Cancer Chemotherapy

In this section, in order to derive the Caputo–Fabrizio fractional model of cancer chemotherapy, the nonlinear four-state model given in [47,84,85,86] has been considered. In this model, the first state is $N\left(t\right)$, which indicates the number of normal cells; the second variable in this model is $T\left(t\right),$ and it represents the number of tumor cells. The third state is $I\left(t\right)$, which represents the number of immune cells. Finally, the last state is the drug concentration $C\left(t\right)$. The original integer-order model [47,87] is given in Equation (9):

$$\left\{\begin{array}{c}\dot{N}\left(t\right)={\mathrm{r}}_{2}N\left(t\right)\left[1-b_{2}N\left(t\right)\right]-c_{4}N\left(t\right)T\left(t\right)-a_{3}N\left(t\right)C\left(t\right),\\ \dot{T}\left(t\right)=r_{1}T\left(t\right)\left[1-b_{1}T\left(t\right)\right]-c_{2}T\left(t\right)I\left(t\right)-c_{3}N\left(t\right)T\left(t\right)-a_{2}T\left(t\right)C\left(t\right),\\ \genfrac{}{}{0pt}{}{\dot{I}(t)=s+\frac{\lambda T\left(t\right)I\left(t\right)}{\gamma +T\left(t\right)}-c_{1}T\left(t\right)I\left(t\right)-d_{1}I\left(t\right)-a_{1}I\left(t\right)C\left(t\right),}{\dot{C}(t)=-d_{2}C\left(t\right)+u\left(t\right).}\end{array}\right.$$

(9)

The initial conditions for this model are assumed to be as follows:

$$N\left(0\right)=N_0,T\left(0\right)=T_0,I\left(0\right)=I_0,C\left(0\right)=C_0.$$

(10)

in which $r_1$ and $r_2$ are the growth rate of the tumor cells and normal cells, respectively; $b_1$ and $b_2$ denote the reciprocal carrying capacities of both cells; $d_1$ is cell death rate, $d_2$ stands for the decay rate of the injected drug; and the fractional cell kill rate of the immune cells, tumor cells, and normal cells are represented by $a_1$, $a_2$ and $a_3$ respectively; also in this model, $\lambda$ and $\gamma$ are positive constants which stand for the immune response rate and the immune threshold rate, respectively. The influx rate of the immune cells is represented by s, and $u\left(t\right)$ is the drug infusion rate.

Replacing the first-order derivatives on the left-hand side of Equation (9) with the Caputo–Fabrizio fractional derivative that has been defined in Equation (7) leads to our new Caputo–Fabrizio fractional model of cancer chemotherapy. The new model is written in the following equations:

$$\left\{\begin{array}{c}{}_0{}^{CF}{D}_{t_f}^{{\alpha }_1}N(t)={\mathrm{r}}_{2}N\left(t\right)\left[1-b_{2}N\left(t\right)\right]-c_{4}N\left(t\right)T\left(t\right)-a_{3}N\left(t\right)C\left(t\right),\\ {{}_0{}^{CF}{D}_{t_f}^{{\alpha }_2}T\left(t\right)=r}_{1}T\left(t\right)\left[1-b_{1}T\left(t\right)\right]-c_{2}T\left(t\right)I\left(t\right)-c_{3}N\left(t\right)T\left(t\right)-a_{2}T\left(t\right)C\left(t\right),\\ \begin{array}{c}{}_0{}^{CF}{D}_{t_f}^{{\alpha }_3}I(t)=s+\frac{\lambda T\left(t\right)I\left(t\right)}{\gamma +T\left(t\right)}-c_{1}T\left(t\right)I\left(t\right)-d_{1}I\left(t\right)-a_{1}I\left(t\right)C(t),\\ {}_0{}^{CF}{D}_{t_f}^{{\alpha }_4}C(t)=-d_{2}C\left(t\right)+u\left(t\right).\end{array}\end{array}\right.$$

(11)

Moreover, the initial condition for this system is:

$$N\left(0\right)=N_0,T\left(0\right)=T_0,I\left(0\right)=I_0,C\left(0\right)=C_0$$

(12)

In our new model of cancer chemotherapy, it has been assumed that the fraction-order of each of the state variables is theoretically different, and ${0<\alpha }_i<1,i=\mathrm{1,2},\dots ,4$.

3.1. Existence and Uniqueness of Solutions of the Cancer Chemotherapy Model

This section is devoted to the investigation of the uniqueness and existence of the solutions of our Caputo–Fabrizio fraction model of cancer chemotherapy in Equation (11) with the initial conditions mentioned in Equation (12). To seek this goal, the fix-point theory has been used [88,89].

In the light of the Caputo-Fabrizio fractional order integral operator, defined in Equation (8) and taking the initial conditions in Equation (12) into consideration, in using Equation (12) the following equation will be obtained:

$$\left\{\begin{array}{c}N\left(t\right)-N\left(0\right)={}_0{}^{CF}{\mathrm{I}}_{t_f}^{{\alpha }_1}{[\mathrm{r}}_{2}N\left(t\right)\left[1-b_{2}N\left(t\right)\right]-c_{4}N\left(t\right)T\left(t\right)-a_{3}N\left(t\right)C\left(t\right)],\\ T\left(t\right)-T\left(0\right)={}_0{}^{CF}{\mathrm{I}}_{t_f}^{{\alpha }_2}[r_{1}T\left(t\right)\left[1-b_{1}T\left(t\right)\right]-c_{2}T\left(t\right)I\left(t\right)-c_{3}N\left(t\right)T\left(t\right)-a_{2}T\left(t\right)C\left(t\right)],\\ \begin{array}{c}I\left(t\right)-I\left(0\right)={}_0{}^{CF}{\mathrm{I}}_{t_f}^{{\alpha }_3}[s+\frac{\lambda T\left(t\right)I\left(t\right)}{\gamma +T\left(t\right)}-c_{1}T\left(t\right)I\left(t\right)-d_{1}I\left(t\right)-a_{1}I\left(t\right)C\left(t\right)],\\ C\left(t\right)-C\left(0\right)={}_0{}^{CF}{\mathrm{I}}_{t_f}^{{\alpha }_4}[-d_{2}C\left(t\right)+u\left(t\right)].\end{array}\end{array}\right.$$

(13)

Then, the following kernels are defined:

$$\left\{\begin{array}{c}{K}_1(t,N(t\left)\right)={\mathrm{r}}_{2}N\left(t\right)\left[1-b_{2}N\left(t\right)\right]-c_{4}N\left(t\right)T\left(t\right)-a_{3}N\left(t\right)C\left(t\right)\\ K_2\left(t,T\left(t\right)\right)=r_{1}T\left(t\right)\left[1-b_{1}T\left(t\right)\right]-c_{2}T\left(t\right)I\left(t\right)-c_{3}N\left(t\right)T\left(t\right)-a_{2}T\left(t\right)C\left(t\right),\\ \begin{array}{c}{K}_3\left(t,I\left(t\right)\right)=s+\frac{\lambda T\left(t\right)I\left(t\right)}{\gamma +T\left(t\right)}-c_{1}T\left(t\right)I\left(t\right)-d_{1}I\left(t\right)-a_{1}I\left(t\right)C\left(t\right),\\ K_4\left(t,C\left(t\right)\right)=-d_{2}C\left(t\right)+u\left(t\right).\end{array}\end{array}\right.$$

(14)

By taking Equation (14) into consideration and then calculating the right-hand side of Equation (13) using the definition of the Caputo–Fabrizio fractional-order integration in Equation (6), the following equation is obtained:

$$\left\{\begin{array}{c}N\left(t\right)-N\left(0\right)=\Sigma \left({\alpha }_1\right)K_1\left(t,N\left(t\right)\right)+\sigma \left({\alpha }_1\right){\int }_0^t{K}_1\left(v,N\left(v\right)\right)dv,\\ T\left(t\right)-T\left(0\right)=\Sigma \left({\alpha }_2\right)K_2(t,T(t\left)\right)+\sigma \left({\alpha }_2\right){\int }_0^t{K}_2\left(v,T\left(v\right)\right)dv,\\ \begin{array}{c}I\left(t\right)-I\left(0\right)=\Sigma \left({\alpha }_3\right)K_3(t,I(t\left)\right)+\sigma \left({\alpha }_3\right){\int }_0^t{K}_3\left(v,I\left(v\right)\right)dv,\\ C\left(t\right)-C\left(0\right)=\Sigma \left({\alpha }_4\right)K_4\left(t,C\left(t\right)\right)+\sigma \left({\alpha }_4\right){\int }_0^t{K}_4\left(v,I\left(v\right)\right)dv\end{array}\end{array}\right.$$

(15)

where $\mathrm{\Sigma }\left(\mathrm{\alpha }\right)$ and $\sigma \left(\alpha \right)$ are defined as follows:

$$\mathrm{\Sigma }(\mathrm{\alpha })=\frac{2\left(1-\alpha \right)}{\left(2-\alpha \right)A\left(\alpha \right)}and\sigma (\alpha )=\frac{2\alpha }{\left(2-\alpha \right)A\left(\alpha \right)}.$$

(16)

Remark 3.

The above-mentioned kernels$K_1$,$K_2$,…,$K_4$satisfy the Lipschitz conditions and are contraction mapping if the following inequality satisfies

$$0\le \mathrm{L}=\max \left\{{\mathrm{\delta }}_1,{\mathrm{\delta }}_2,{\mathrm{\delta }}_3,{\mathrm{\delta }}_4\right\}<1$$

(17)

In the proof of Remark 3 the following assumption has been made:

$$‖N\left(t\right)‖\le {\theta }_1,‖T\left(t\right)‖\le {\theta }_2,‖I\left(t\right)‖\le {\theta }_3\mathrm{and}‖C\left(t\right)‖\le {\theta }_4.$$

(18)

Proof. 

Let $N$ and $N_1$ be two different functions; then, by taking the kernel $K_1$ into consideration, we have

$$\begin{array}{l}||K_1\left(t,N\right)-K_1\left(t,N_1\right)||\\ =||{\mathrm{r}}_2\left(N-N_1\right)\left[1-b_2\left(N-N_1\right)\right]-c_4\left(N-N_1\right)T\left(t\right)\\ -a_3\left(N-N_1\right)C\left(t\right)||\\ \le w_1‖\left(N-N_1\right)‖-w_2‖\left(N-N_1\right)‖-w_3‖\left(N-N_1\right)‖\\ \le (w_1-w_2-w_3)‖\left(N-N_1\right)‖\le {\delta }_1‖\left(N-N_1\right)‖\end{array}$$

(19)

For the second kernel we have:

$$\begin{array}{l}‖K_2\left(t,T\right)-K_2\left(t,T_1\right)‖\\ =||{\mathrm{r}}_1\left(T-T_1\right)\left[1-b_1\left(T-T_1\right)\right]-c_2\left(T-T_1\right)I\left(t\right)\\ -c_3\left(T-T_1\right)N\left(t\right)||\\ \le a_1‖\left(T-T_1\right)‖-a_2‖\left(T-T_1\right)‖-a_3‖\left(T-T_1\right)‖\\ \le (a_1-a_2-a_3)‖\left(T-T_1\right)‖\le {\delta }_2‖\left(T-T_1\right)‖\end{array}$$

(20)

Additionally, for the third and fourth kernels, $K_3$ and $K_4$, the following inequalities can be obtained:

$$\begin{array}{l}‖K_3\left(t,I\right)-K_3\left(t,I_1\right)‖\\ =‖\frac{\lambda T(I-I_1)}{\gamma +T\left(t\right)}-c_{1}T\left(t\right)(I-I_1)-d_1(I-I_1)-a_1(I-I_1)C\left(t\right)‖\\ \le \frac{\lambda {\theta }_2}{\gamma +{\theta }_2}‖\left(I-I_1\right)‖-c_1{\theta }_2‖\left(I-I_1\right)‖-d_1‖\left(I-I_1\right)‖\\ -a_1{\theta }_4‖\left(I-I_1\right)‖\le (m_1-m_2-m_3-m_4)‖\left(I-I_1\right)‖\\ \le {\delta }_3‖\left(T-T_1\right)‖\end{array}$$

(21)

and

$$‖K_4\left(t,C\right)-K_4\left(t,C_1\right)‖=‖-d_2(C-C_1)‖\le d_2‖(C-C_1)‖$$

(22)

Consequently, the Lipschitz conditions are satisfied for all of the kernels defined in Equation (14). Furthermore, because of $0\le \mathrm{L}=\max \left\{{\mathrm{\delta }}_1,{\mathrm{\delta }}_2,{\mathrm{\delta }}_3,{\mathrm{\delta }}_4\right\}<1$, the kernels are contractions.

By considering Equation (16), the following recursive formula can be obtained:

$$\left.\begin{array}{c}{N}_n\left(t\right)=\mathrm{\Sigma }\left({\alpha }_1\right)K_1\left(t,N_{n-1}\left(t\right)\right)+\sigma \left({\alpha }_1\right){\int }_0^t{K}_1\left(v,N_{n-1}\left(v\right)\right)dv,\\ T_n\left(t\right)=\mathrm{\Sigma }\left({\alpha }_2\right)K_2(t,T_{n-1}(t\left)\right)+\sigma \left({\alpha }_2\right){\int }_0^t{K}_2\left(v,T_{n-1}\left(v\right)\right)dv,\\ \begin{array}{c}{I}_n\left(t\right)=\mathrm{\Sigma }\left({\alpha }_3\right)K_3(t,I_{n-1}(t\left)\right)+\sigma \left({\alpha }_3\right){\int }_0^t{K}_3\left(v,I_{n-1}\left(v\right)\right)dv,\\ C_n\left(t\right)=\mathrm{\Sigma }\left({\alpha }_4\right)K_4(t,C_{n-1}(t\left)\right)+\sigma \left({\alpha }_4\right){\int }_0^t{K}_4\left(v,I_{n-1}\left(v\right)\right)dv.\end{array}\end{array}\right\}.$$

(23)

Moreover, the initial components for the above-mentioned recursive formals are considered to be as the following equation:

$$N_0\left(t\right)=N_0,T_0\left(t\right)=T_0,I_0\left(t\right)=I_0,C_0\left(t\right)=C_0$$

(24)

Equation (23) can be written in the following formation:

$$N_n\left(t\right)=\sum _{i=1}^n{\mu }_i\left(t\right),T_n\left(t\right)=\sum _{i=1}^n{\xi }_i\left(t\right),I_n\left(t\right)=\sum _{i=1}^n{\omega }_i\left(t\right),C_n\left(t\right)=\sum _{i=1}^n{\kappa }_i\left(t\right).$$

(25)

where ${\mu }_i\left(t\right)$, ${\xi }_i\left(t\right)$, ${\omega }_i\left(t\right)$ and ${\kappa }_i\left(t\right)$ are defined as follows:

$$\left.\begin{array}{c}{\mu }_i\left(t\right)=N_i\left(t\right)-N_{i-1}\left(t\right),{\xi }_i\left(t\right)=T_i\left(t\right)-T_{i-1}\left(t\right),\\ {\omega }_i\left(t\right)=I_i\left(t\right)-I_{i-1}\left(t\right),{\kappa }_i\left(t\right)=C_i\left(t\right)-C_{i-1}\left(t\right).\end{array}\right\}$$

(26)

Now, the following inequalities can be derived for the functions defined in Equation (26):

$$\begin{array}{l}‖{\mu }_n\left(t\right)‖=‖N_i\left(t\right)-N_{i-1}\left(t\right)‖\\ \le \mathrm{\Sigma }\left({\alpha }_1\right)‖\left(K_1\left(t,N_{n-1}\left(t\right)\right)-K_1\left(t,N_{n-2}\left(t\right)\right)\right)‖\\ +\sigma \left({\alpha }_1\right){\int }_0^t‖(K_1\left(v,N_{n-1}\left(v\right)\right)-K_1\left(v,N_{n-2}\left(v\right)\right))‖dv\end{array}$$

(27)

Equation (19) showed that the kernel $K_1$ satisfies the Lipschitz condition. Therefore, for Equation (27) we have:

$$\begin{gathered} ‖{\mu }_n\left(t\right)‖\le \Sigma \left({\alpha }_1\right){\delta }_1‖N_{n-1}\left(t\right)-N_{n-2}\left(t\right)‖+\sigma \left({\alpha }_1\right){\delta }_1{\int }_0^t‖N_{n-1}\left(v\right)-N_{n-2}\left(v\right)‖dv \\ \le \Sigma \left({\alpha }_1\right){\delta }_1‖{\mu }_{n-1}\left(t\right)‖+\sigma \left({\alpha }_1\right){\delta }_1{\int }_0^t‖{\mu }_{n-1}\left(v\right)‖dv \end{gathered}$$

(28)

The same is true for the other defined functions in Equation (26); for this reason, we can obtain the following inequalities readily:

$$\left.\begin{array}{c}‖{\xi }_n\left(t\right)‖\le \mathrm{\Sigma }\left({\alpha }_2\right){\delta }_2‖{\xi }_{n-1}\left(t\right)‖+\sigma \left({\alpha }_2\right){\delta }_2{\int }_0^t‖{\xi }_{n-1}\left(v\right)‖dv,\\ ‖{\omega }_n\left(t\right)‖\le \mathrm{\Sigma }\left({\alpha }_3\right){\delta }_3‖{\omega }_{n-1}\left(t\right)‖+\sigma \left({\alpha }_3\right){\delta }_3{\int }_0^t‖{\omega }_{n-1}\left(v\right)‖dv,\\ ‖{\kappa }_n\left(t\right)‖\le \mathrm{\Sigma }\left({\alpha }_4\right){\delta }_4‖{\kappa }_{n-1}\left(t\right)‖+\sigma \left({\alpha }_4\right){\delta }_4{\int }_0^t‖{\kappa }_{n-1}\left(v\right)‖dv.\end{array}\right\},$$

(29)

□

Remark 4.

The Caputo–Fabrizio fractional model of cancer chemotherapy (Equation (11)) has a system of solutions if following inequalities hold at a time such that$t_1>0$:

$$\mathrm{\Sigma }\left({\alpha }_i\right){\delta }_i+\sigma \left({\alpha }_i\right){\delta }_i{t}_1<1fori=1,2,3,4$$

(30)

Proof. 

In this part, we have proven the existence and smoothness of the defined functions in Equation (25), the existence of a system of solutions for the model in Equation (11), and Equation (12) has been illustrated.

It has been assumed that $N\left(t\right)$,$T\left(t\right)$,$I\left(t\right)$ and $C\left(t\right)$ are bounded $‖N\left(t\right)‖\le {\theta }_1$,$‖T\left(t\right)‖\le {\theta }_2$,$‖I\left(t\right)‖\le {\theta }_3$ and $‖C\left(t\right)‖\le {\theta }_4$. Moreover, we have proven that each of the defined kernels satisfies the Lipschitz condition. Consequently, we can obtain the following results using Equations (28) and (29):

$$\left.\begin{array}{c}‖{\mu }_n\left(t\right)‖\le ‖N\left(0\right)‖{\left[\mathrm{\Sigma }\left({\alpha }_1\right){\delta }_1+\sigma \left({\alpha }_1\right){\delta }_{1}t\right]}^n,\\ ‖{\xi }_n\left(t\right)‖\le ‖T\left(0\right)‖{\left[\mathrm{\Sigma }\left({\alpha }_2\right){\delta }_2+\sigma \left({\alpha }_1\right){\delta }_{2}t\right]}^n,\\ \begin{array}{c}‖{\omega }_n\left(t\right)‖\le ‖I\left(0\right)‖{\left[\mathrm{\Sigma }\left({\alpha }_3\right){\delta }_3+\sigma \left({\alpha }_3\right){\delta }_{3}t\right]}^n,\\ ‖{\kappa }_n\left(t\right)‖\le ‖C\left(0\right)‖{\left[\mathrm{\Sigma }\left({\alpha }_4\right){\delta }_4+\sigma \left({\alpha }_4\right){\delta }_{4}t\right]}^n.\end{array}\end{array}\right\}.$$

(31)

Now, it has been demonstrated that the functions $N_n\left(t\right)$,$T_n\left(t\right)$,$I_n\left(t\right)$ and $C_n\left(t\right)$ defined in Equation (25) converge to solutions of the model (Equation (11)) with the initial condition (Equation (12)), by defining remainder terms after n iteration as follows:

$$\left.\begin{array}{c}N\left(t\right)-N\left(0\right)=N_n\left(t\right)-X_n\left(t\right),\\ {T\left(t\right)-T\left(0\right)=T}_n\left(t\right)-Y_n\left(t\right),\\ \begin{array}{c}{I\left(t\right)-I\left(0\right)=I}_n\left(t\right)-Z_n\left(t\right)\\ C\left(t\right)-C\left(0\right)=C_n\left(t\right)-V_n\left(t\right).\end{array}\end{array}\right\}.$$

(32)

To prove that the functions $N_n\left(t\right)$,$T_n\left(t\right)$,$I_n\left(t\right)$ and $C_n\left(t\right)$ converge to the solutions of the model (Equation (11)), we must show that as $n\to \mathrm{\infty }$, then the reminder term converges to zero. Using the Lipschitz condition for the kernels defined in Equation (14) and triangle inequality, the following results will be obtained:

$$\begin{array}{l}‖X_n\left(t\right)‖=\le \mathrm{\Sigma }\left({\alpha }_1\right)‖\left(K_1\left(t,N\left(t\right)\right)-K_1\left(t,N_{n-1}\left(t\right)\right)\right)‖\\ +\sigma \left({\alpha }_1\right){\int }_0^t‖\left(K_1\left(v,N\left(v\right)\right)-K_1\left(v,N_{n-1}\left(v\right)\right)\right)‖\\ \le \mathrm{\Sigma }\left({\alpha }_1\right){\delta }_1‖N_{n-1}\left(t\right)-N_{n-2}\left(t\right)‖+\sigma \left({\alpha }_1\right){\delta }_1‖N_{n-1}\left(t\right)-N_{n-2}\left(t\right)‖t.\end{array}$$

(33)

Continuing the above process will lead us to the following inequality:

$$‖X_n\left(t\right)‖\le {\left[\mathrm{\Sigma }\left({\alpha }_1\right){\delta }_1+\sigma \left({\alpha }_1\right){\delta }_{1}t\right]}^{n+1}{\theta }_1$$

(34)

Now, by taking Equation (30) into account and when $n\to \mathrm{\infty }$ at time $t_1$, by taking a limit on both sides of Equation (34) the following result will be acquired:

$$\underset{n\to \infty }{\lim }‖X_n\left(t\right)‖\le \underset{n\to \infty }{\lim }{\left[\mathrm{\Sigma }\left({\alpha }_1\right){\delta }_1+\sigma \left({\alpha }_1\right){\delta }_{1}t\right]}^{n+1}{\theta }_1.$$

(35)

It can be seen that the right-hand side of Equation (35) converges to zero; for this reason, it can be concluded that $‖X_n\left(t\right)‖\to 0$ when $n\to \mathrm{\infty }$. Using the same manner for other remainder terms defined in Equation (32), the following results will be obtained:

$$\underset{n\to \infty }{\lim }‖Y_n\left(t\right)‖=0,\underset{n\to \infty }{\lim }‖Z_n\left(t\right)‖=0,\underset{n\to \infty }{\lim }‖V_n\left(t\right)‖=0$$

(36)

Equations (35) and (36) demonstrate the existence of a system of solutions for the model in Equations (11) and (12). □

Remark 5.

The system of solutions of the model in Equations (11) and (12) will be unique if the following inequality is satisfied:

$$\mathrm{\Sigma }\left({\alpha }_i\right){\delta }_i+\sigma \left({\alpha }_i\right){\delta }_{i}t<1fori=1,2,3,4.$$

(37)

Proof. 

To prove that the model described in Equation (11) with initial conditions in Equation (12) has a unique system of solutions, we have assumed that the first system of solutions of the model is {$N\left(t\right),T\left(t\right),I\left(t\right),C\left(t\right)$}. Furthermore, we have considered another system of solutions for the model, which is denoted by {$N_1\left(t\right),T_1\left(t\right),I_1\left(t\right),C_1\left(t\right)$}. Then, using Equation (16), we have

$$\begin{array}{l}N\left(t\right)-N_1\left(t\right)=\mathrm{\Sigma }\left({\alpha }_1\right)(K_1\left(t,N\left(t\right)\right)-K_1\left(t,N_1\left(t\right)\right))\\ +\sigma \left({\alpha }_1\right){\int }_0^t(K_1\left(v,N\left(v\right)\right)-K_1\left(v,N_1\left(v\right)\right))dv\end{array}$$

(38)

Then, we have

$$\begin{array}{l}‖N\left(t\right)-N_1\left(t\right)‖=\\ \le \mathrm{\Sigma }\left({\alpha }_1\right)‖\left(K_1\left(t,N\left(t\right)\right)-K_1\left(t,N_1\left(t\right)\right)\right)‖\\ +\sigma \left({\alpha }_1\right)‖{\int }_0^t\left(K_1\left(v,N\left(v\right)\right)-K_1\left(v,N_1\left(v\right)\right)\right)dv‖.\end{array}$$

(39)

As we proved, the kernels satisfy the Lipschitz conditions; as a result,

$$‖N\left(t\right)-N_1\left(t\right)‖\le \mathrm{\Sigma }\left({\alpha }_1\right){\delta }_1‖N\left(t\right)-N_1\left(t\right)‖+\sigma \left({\alpha }_1\right){\delta }_1‖N\left(t\right)-N_1\left(t\right)‖t$$

(40)

By subtracting the right-hand side of Equation (40) from both side of this inequality, the following inequality is given:

$$(1-\mathrm{\Sigma }\left({\alpha }_1\right){\delta }_1-\sigma ({\alpha }_1\left){\delta }_{1}t\right)‖N\left(t\right)-N_1\left(t\right)‖\le 0$$

(41)

Considering Equation (37) and the fact that the output of the norm function is non-negative, the following result will be obtained:

$$‖N\left(t\right)-N_1\left(t\right)‖=0$$

(42)

In the same way for $T\left(t\right)$,$I\left(t\right)$, $C\left(t\right)$, the following equations have been obtained:

$$\left.\begin{array}{c}‖T\left(t\right)-T_1\left(t\right)‖=0,\\ ‖I\left(t\right)-I_1\left(t\right)‖=0,\\ ‖C\left(t\right)-C_1\left(t\right)‖=0.\end{array}\right\},$$

(43)

Equations (42) and (43) imply

$$N\left(t\right)=N_1\left(t\right),T\left(t\right)=T_1\left(t\right),I\left(t\right)-I_1\left(t\right),C\left(t\right)=C_1\left(t\right)$$

(44)

Equation (44) shows that the system of solution for the model in Equation (11) with the initial conditions in Equation (12) is unique, and this is the end of the proof. □

3.2. Sensitivity Analysis

In this section, a sensitivity analysis for the chemotherapy drug-using system given in Equation (11) is conducted. The parameters of the system which have been used for analysis are given in

Table 1. The simulated patient’s parameters [47].

Parameter	Description (Unit)	Value
$a_1$	Fractional immune cell kill rate (mg−1 lday−1)	0.2
$a_2$	Fractional tumor cell kill rate (mg−1 lday−1)	0.3
$a_3$	Fractional normal cell kill rate (mg−1 lday−1)	0.1
$b_1$	Reciprocal carrying capacity of tumor cells (cell−1)	1
$b_2$	Reciprocal carrying capacity of normal cells (cell−1)	1
$c_1$	Immune cell competition term (competition between immune and tumor cells) (cell−1 day−1)	1
$c_2$	Tumor cell competition term (competition between immune and tumor cells) (cell−1 day−1)	0.5
$c_3$	Tumor cell competition term (competition between normal and tumor cells) (cell−1 day−1)	1
$c_4$	Normal cell competition term (competition between normal and tumor cells) (cell−1 day−1)	1
$d_1$	Immune cell death rate (day−1)	0.2
$d_2$	Decay rate of injected drug (day−1)	1
$r_1$	Per unit growth rate of tumor cells (day−1)	1.5
$r_2$	Per unit growth rate of normal cells (day−1)	1
$s$	Immune cell influx rate (cell day−1)	0.33
$\gamma$	Immune threshold rate (cell)	0.3
$\lambda$	Immune response rate (day−1)	0.01

. The results of the sensitivity analysis are given in Figure 1.

The relationship between the patient’s parameter variations is elucidated in Figure 1. Conspicuously, per-unit growth rate of tumor cells ($r_1)$, immune cell influx rate ($s$), and tumor cell competition term (competition between normal and tumor cells) ($c_3)$ are three parameters whose variations affect the number of normal cells the most. There are four parameters whose variations have the most impact on the number of tumor cells, of which three of them are the same as the ones for normal cells, and the other parameter is the reciprocal carrying capacity of normal cells ($b_2$). Moreover, the number of immune cells is highly affected by the variation of the per-unit growth rate of tumor cells ($r_1$), the reciprocal carrying capacity of normal cells ($b_2$), and the tumor cell competition term (competition between normal and tumor cells) ($c_3$).

4. Methodology

In this section, we have described the proposed fuzzy-reinforcement-learning based controller (FRLC) whose aim is to control the number of tumor cells; in other words, the aim of the proposed controller is to reach the desired number of tumor cells $T\left(t\right)=T_{desire}$ from a non-zero initial number of tumor cells $T\left(0\right)>0$.

Expected-SARSA is a variation of SARSA wherein the variance in its update rule is decreased in comparison with that of the SARSA algorithm, and it has better performance than SARSA for online applications that can be considered as an on-policy version of Q-Learning. On-policy methods have some advantages over off-policy learning algorithms, such as stronger convergence guarantees in cases in which it is combined with function approximation, while the off-policy methods can diverge in those cases [90,91,92]. They have outstanding performance in online applications because the policy that is estimated in on-policy ways will be improved iteratively, and the agent will behave based on this policy.

The advantage of data-driven controllers is that an accurate model of the system is not required, while selecting the best action for each rule in the fuzzy interface system demands accurate knowledge about the system. However, the proposed control method learns the best action for each rule; therefore, it has the advantages of data-driven controllers and fuzzy controllers simultaneously.

Fuzzy Controller Based on Expected SARSA Learning (FESL)

In this paper, in order to control the number of tumor cells, a fuzzy controller is proposed of which its learning process is based on the expected SARSA algorithm. The fuzzy controller uses Expected-SARSA algorithms in order to find the best map between state and actions. In other words, using the Expected-SARSA algorithm, the best action for each of the fuzzy system inference’s rules will be obtained. Consider the rules in a fuzzy inference system be as follows:

$$Ri:\mathrm{if}{x}_1\mathrm{is}{L}_{1i}\mathrm{and}\dots \mathrm{and}{x}_n\mathrm{is}{L}_{ni}\mathrm{then}(a_i\mathrm{is}{a}_{i1}\left)\mathrm{or}\right(a_i\mathrm{is}{a}_{i2})\mathrm{or}\dots \mathrm{or}(a_i\mathrm{is}{a}_{ik}),$$

in which Ri is the i-th rule of the fuzzy inference system, and $s$ is the vector of the n-dimensional input state and is defined as $s=x_1\times x_2\times \dots \times x_n$; $L_i$ is the n-dimensional strictly convex and normal fuzzy set of the i-th rule with a unique center and is defined as $L_i=L_{i1}\times L_{i2}\times \dots \times L_{in}$; $a_i$ is the consequent action for the i-th rule; $a_{i1}$ is the first candidate action for the i-th rule; $a_{i2}$ is the second candidate action; and finally, $a_{ik}$ is the kth candidate action for the i-th rule. The aim of the learning is to find the optimal action for each of the rules of the fuzzy inference system. So as to seek this aim, a value action matrix has been defined and is denoted by $Q$, of which its elements are the value action of each candidate action for each of the rules. The optimal action for each of the rules is the candidate action for that rule, of which its value action is the most among all the candidate actions of that rule.

As aforementioned, the best consequent action for each of the rules will be obtained using reinforcement learning methods. Figure 2 depicts the structure of reinforcement learning. In this paper, the reinforcement learning algorithm that has been implemented to update value actions is the Expected-SARSA method.

The algorithm used in this paper has been delineated in Algorithm 1. The function that is used for calculating the reward of the agent for a transition from state $s_k$ to state $s_{k+1}$ is as follows:

$$r_{k+1}=\left\{\begin{array}{c}\frac{e\left(kT\right)-e\left(\right(k+1\left)T\right)}{e\left(kT\right)},e\left(\left(k+1\right)T\right)<e\left(kT\right)\\ 0e\left(\left(k+1\right)T\right)<e\left(kT\right)\end{array}\right.$$

(45)

where $e\left(t\right),t\ge 0$ is defined as:

$$e\left(t\right)=\beta \left(T\left(t\right)-T_d\left(t\right)\right)+\left(1-\beta \right)\left(N\left(t\right)-N_d\left(t\right)\right).$$

(46)

The goal of the control agent in reinforcement learning (RL) is to find an optimal policy ${\pi }^{\mathrm{*}}$; using this policy, the expected discount reward would be maximum. The discount reward is defined as follows:

$$R_t=r_{t+1}+\delta r_{t+2}+{\delta }^2{r}_{t+3}+\dots =\sum _{k=0}^{\infty }{\delta }^k{r}_{t+k+1}$$

(47)

where $\delta$ is discount factor and is a non-negative constant, which satisfies $\delta \le 1$.

The output action of the fuzzy inference system is calculated as

$$a=\sum _{i=1}^n{\eta }_i{a}_{ip}/\sum _{i=1}^n{\eta }_i$$

(48)

where $n$ is the number of fuzzy rules in the fuzzy inference, ${\eta }_i$ is the firing strength of the i-th rule, and $a_{ip}$ is the selected action using the 𝜀-greedy method for the i-th rule among all possible actions. Updating the rule for FESL is considered to be as

$$Q_t[i,p]\leftarrow Q_{t-1}[i,p]+\alpha \frac{{\eta }_i}{\sum _{i=1}^n{\eta }_i}{\gamma }_t$$

(49)

where $\alpha$ denotes the learning rate of the algorithm. In Equation (49), ${\gamma }_t$ is defined as:

$${\gamma }_t=r_{t+1}+\delta \left[\sum _{k=1}^n\frac{{\eta }_k}{\sum _{t=1}^n{\eta }_t}\sum _{j=1}^m{\pi }_{s^{\prime }}[k,j]Q_t[k,j]\right]-Q_{t-1}\left[i,p\right].$$

(50)

in which $\delta$ is a discount factor, ${\pi }_{s^{\prime }}[k,j]$ is the probability of selecting the j-th action for the k-th rule that has been obtained using 𝜀-greedy method, and $Q_t[k,j]$ is an approximation of the value of the j-th action for the k-th rule.

Algorithm 1 Fuzzy Expected SARSA

1: Initialize Q-table 2: Loop {for all of episodes} 3:  Initialize state s 4:  repeat {for each step in episode} 5:   calculate firing strength of each rule (αi) at state s 6:   choose action (αi) for each of the rules at state s using policy π 7:   calculate action a at state s 8:   take action a, observe reward r and next state s′ 9:   calculate state value of state s′ 10:   update Q-table 11:  until s is terminal state 12: end loop

One of the key features of FRL-based controllers is their ability to handle uncertainty and imprecision in the system. Fuzzy logic allows the controller to make decisions based on approximate rather than precise values, which can be helpful in situations where the system is not well understood or where there is a high degree of variability. In addition to its ability to handle uncertainty, an FRL-based controller can also learn from its past experiences and adapt its behavior to better achieve the desired outcomes. This learning process is known as reinforcement learning, and it allows the controller to optimize its performance over time by adjusting its control inputs based on the consequences of those inputs. Overall, the combination of fuzzy logic and reinforcement learning in an FRL-based controller can make it well-suited for handling unexpected situations and adapting to a range of conditions.

5. Numerical Simulations

In this section, the performance of the FESL method for controlling the closed-loop system of cancer chemotherapy drug dosing, of which its input variable is considered to be a chemotherapy drug such as carboplatin, is assessed through simulations that are done in MATLAB. Oncologists consider different factors when they determine the drug dosage for a cancer patient. Age and gender are two examples of those factors. As has been shown in [87], the growth rate of normal cells and immune cells depends on age, and this rate will be larger for younger patients rather than elderly patients. Therefore, for young patients, oncologists prefer to eradicate cancerous cells as soon as possible without taking the damage to normal cells into consideration, which avoids cancer metastasis. In the current study, we assumed the value of fractional derivative as is mentioned in Table 1. However, estimating the fractional derivative parameter in the same way as the other parameters helps to ensure that the model is as accurate and reliable as possible. This can be done using a variety of techniques such as fitting the model to experimental data or using statistical methods to optimize the model parameters. By taking the time to accurately estimate all of the model parameters, researchers can be more confident in the results of their study and the conclusions they draw from the data.

For the cases in which the patient is elderly, the patient suffers from other diseases, or the cancer is in a vital organ such as the brain, the degeneration of the normal cells is not desirable, and the normal cells should be kept undamaged. The above-mentioned conditions have been taken into consideration by the proposed control method through the defining error in Equation (46).

For this reason, two scenarios have been considered to evaluate the performance of the proposed control method in the abovementioned cases. In scenario “A”, it has been considered that the patient is young. On the other hand, the simulation of scenario “B” has been conducted for an elderly patient. Finally, the results of the simulations for the proposed control method have been compared with those of Watkin’s Q-learning method for both scenarios.

The simulated patient’s parameters are considered to be the same as the ones given in Table 1 [87]; in [93], it was shown how the parameters that are given in Table 1 can be obtained. The number of episodes that have been considered for simulations is considered to be 500, of which each of the episodes is defined as a set of transitions from the initial state to the terminal state. For the simulations, 20 fuzzy-sets have been considered for the fuzzy system, and the membership functions used for the fuzzy sets are considered to be trapezoidal-shape and z-shape membership functions, which have been shown in Figure 3.

Table 2. Fuzzy-sets and their membership functions.

#	Type	[a, b, c, d]	#	Type	[a, b, c, d]
1	zmfl	[0.0055, 0.0068, ~, ~]	11	trapmf	[0.3495, 0.3505, 0.3995, 0.4005]
2	trapmf	[0.0058, 0.0068, 0.0120, 0.0130]	12	trapmf	[0.3995, 0.4005, 0.4495, 0.4505]
3	trapmf	[0.0120, 0.0130, 0.0245, 0.0255]	13	trapmf	[0.4495, 0.4505, 0.4995, 0.5005]
4	trapmf	[0.0245, 0.0255, 0.0395, 0.0405]	14	trapmf	[0.4995, 0.5005, 0.5495, 0.5505]
5	trapmf	[0.0395, 0.0405, 0.0495, 0.0505]	15	trapmf	[0.5495, 0.5505, 0.5995, 0.6005]
6	trapmf	[0.0495, 0.0505, 0.0995, 0.1005]	16	trapmf	[0.5995, 0.6005, 0.6495, 0.6505]
7	trapmf	[0.0995, 0.1005, 0.1995, 0.2005]	17	trapmf	[0.6495, 0.6505, 0.6995, 0.7005]
8	trapmf	[0.1995, 0.2005, 0.2495, 0.2505]	18	trapmf	[0.6995, 0.7005, 0.7995, 0.8005]
9	trapmf	[0.2495, 0.2505, 0.2995, 0.3005]	19	trapmf	[0.7995, 0.8005, 0.8995, 0.9005]
10	trapmf	[0.2995, 0.3005, 0.3495, 0.3505]	20	zmfh	[0.8995, 0.9005, ~, ~]

implicates the type of membership function used for each fuzzy-set and their features. Furthermore, in the simulations, we have set $\delta =0.9$, and the learning rate is considered to be $\alpha =0.2$ and should decrease as the number of iterations and episodes increase so as to guarantee the convergence of the learning algorithm.

In Table 2, zmfl denotes the z-shape membership function for the lower bound; trapmf and zmfh stand for trapezoidal-shape and z-shape membership function for the upper bound respectively. Variables “a”, “b”, “c”, and “d” are illustrated in Figure 3.

During simulations, learning algorithms should explore the best actions in order to find the best consequent action for each rule. Keeping this in mind, the 𝜀 value has been considered to be a small number; then, as the number of step-times and the number of episodes increases, the 𝜀 value in the 𝜀-greedy method increases in order to reach a greedy policy. The 𝜀 has been calculated using the following formula:

$$\epsilon =0.3\left(0.5+\frac{I}{MNI}\right)+0.25(1+3\frac{E}{MNE})$$

(51)

where $I$ is the iteration number or the time-step number, $MNI$ is the maximum number of iterations, $E$ is the episode number, and $MNE$ is the maximum number of episodes; in the simulations, the maximum number of iterations is chosen as 500. The candidate actions for each of the rules are considered to be $a_i\in \{0,0.1,0.2,0.3,0.4,0.5,0.6,0.8,1,1.5,2,2.5,3,3.5,4,5,6,7,8,9,10\}$. Moreover, the initial value of the state variables of the system model in Equation (11) are considered to be

$$N\left(0\right)=0.6,T\left(0\right)=0.8,I\left(0\right)=0.3,C\left(0\right)=0.2.$$

(52)

5.1. Scenario A

In this scenario, first, the simulations have been conducted for the patient system, which has no uncertainty. Furthermore, the aim is to reach $T=0$; to put it differently, the controller must take the system from the initial state to the first fuzzy set. Therefore, for the simulation, we have set $\beta =1$, which means in this scenario the error will be $\left(t\right)=\left(T\left(t\right)-T_d\left(t\right)\right)$.

The result of the simulation is given in Figure 4 and is comprised of the number of normal cells in (a), the number of tumor cells in (b), the number of immune cells in (c), the concentration of the chemotherapeutic drug in blood (d), and the amount of chemotherapeutic drug executed for scenario “A” (d). As is evident, the number of tumor cells decreased consistently until it reached zero, which means that the proposed controller was able to obliterate all of the tumor cells. However, due to injecting a chemotherapeutic drug into the body of the patient, the number of normal cells and immune cells decreased at the beginning of the chemotherapy. Nonetheless, later, the numbers of normal cells and immune cells increased.

The simulation confirms that the chemotherapy treatment was successful in eliminating all of the tumor cells, but it caused a decrease in the number of normal cells and immune cells at the beginning of treatment. This is a common occurrence with chemotherapy, as the drugs used to kill cancer cells can also damage healthy cells. However, it is also possible for the body to recover and for the number of normal cells and immune cells to increase again after treatment.

It is important to carefully consider the potential benefits and risks of chemotherapy treatment, as it can be an effective way to treat cancer; however, it can also cause significant side effects. It is generally recommended to undergo chemotherapy under the supervision of a medical professional who can help to monitor the patient’s condition and adjust the treatment plan as needed.

So as to prove the robustness of the proposed control method, after training the control agent using the reinforcement learning algorithm, the parameters of the patients were varied during simulation. The parameters of the modified patient system were considered to have a 10% variation from their original values, which are given in Table 1. The result of the simulation is depicted in Figure 5.

Figure 5 has the same sub-figures as Figure 4. As can be seen, the system has shown the same behavior as the system without uncertainty. Obviously, the controller has shown its ability to eradicate the tumor cells in the presence of the uncertainty in the parameters.

The obtained results are remarkable because the following main achievements:

• The effectiveness of the proposed controller in eliminating the tumor cells: The results show that the controller was able to successfully reduce the number of tumor cells to zero, which indicates that the treatment was effective in destroying the cancer cells.
• The temporary decrease in normal cells and immune cells: The chemotherapy used in the treatment caused a temporary decrease in the number of normal cells and immune cells in the body. This is a common side effect of chemotherapy, as the drugs used can also harm healthy cells.
• The recovery of normal cells and immune cells: Despite the initial decrease in normal cells and immune cells, the numbers of these cells eventually increased over time. This suggests that the body was able to recover and rebuild healthy cells after the chemotherapy treatment.
• The importance of monitoring the effects of chemotherapy: The results of the simulation highlight the importance of carefully monitoring the effects of chemotherapy to ensure that it is being administered effectively and safely. This includes monitoring the number of cancer cells, normal cells, and immune cells in the body, as well as the concentration of chemotherapy drugs in the blood.

5.2. Scenario B

This scenario has been considered to assess the performance of the proposed control method in control of the tumor-cell state in old patients. In this scenario, the aim is to control both tumor-cells and normal-cells. Consequently, in this scenario, it has been considered that $\beta =0.95$, which means the error in this scenario will be $e\left(t\right)=0.95\left(T\left(t\right)-T_d\left(t\right)\right)+0.05\left(N\left(t\right)-N_d\left(t\right)\right)$; as can be seen, the number of normal cells has been taken into consideration in this scenario. The result of the simulations for this scenario is illustrated in Figure 6.

By comparing Figure 4 and Figure 6, it can be confirmed that in the case in which the patient is elderly, the number of normal cells has reached its maximum faster. Moreover, the 2-norm of the amount of drug executed to the patient’s body in the case that the patient is elderly decreased by 5.5 percent.

It is remarkable that the proposed control method was able to reach the maximum number of normal cells and use a smaller amount of chemotherapy drugs more quickly in the case of an elderly patient compared to a non-elderly patient. This suggests that the control method was able to minimize the negative effects of the chemotherapy on healthy cells while still effectively treating the cancer. This is important because chemotherapy can have significant side effects on the body, and minimizing these effects is especially important for elderly patients who may be more sensitive to these effects.

In addition, in this scenario, in order to confirm the robustness of the proposed control method for elderly patients, the simulation was performed for the patient system with 10% uncertainty for each of the system parameters, and the results are given in Figure 7.

It is notable that the control method was able to maintain its effectiveness even when the system parameters were varied by 10%. This indicates that the control method is robust and can be applied effectively to elderly patients with a range of different characteristics. This is important because elderly patients may have a wide range of health conditions and characteristics that can affect their response to chemotherapy.

Figure 7 demonstrates the robustness of the proposed control method against the system’s parameter variation for an elderly patient. The results of the simulations for the elderly patient in both cases show the effectiveness of the proposed control method. Overall, these findings suggest that the proposed control method could be a valuable tool for improving the treatment of elderly patients with chemotherapy. They could help to minimize the negative effects of chemotherapy on healthy cells and provide a more effective treatment for cancer.

5.3. Scenario C

In this scenario, the proposed control method and Watkin’s Q-learning method were exerted to 10 different patients. So as to evaluate the performance of the proposed control method, the simulation results for the proposed control method and the Watkin’s Q-learning method are elucidated in

Table 3. Result of simulations.

	Young Patients			Old Patients
	${\mathbf{M}(‖\mathit{E}‖}_2)$	$\mathbf{M}\left(\mathbf{v}\mathbf{a}\mathbf{r}\right(\mathit{E}\left)\right)$	${\mathbf{M}(‖\mathit{U}‖}_2)$	${\mathit{M}(‖\mathit{E}‖}_2)$	$\mathbf{M}\left(\mathbf{v}\mathbf{a}\mathbf{r}\right(\mathit{E}\left)\right)$	${\mathbf{M}(‖\mathit{U}‖}_2)$
FESL	1.9633	0.0011	130.1827	3.0863	0.0011	123.0201
Q-Learning	3.0206	0.0079	131.7332	4.1083	0.0066	137.0524

where $M(.)$, $\mathrm{v}\mathrm{a}\mathrm{r}(.)$ and ${‖.‖}_2$ denote the mean function, variance function, and the norm function, respectively.

. The patient’s parameters are considered to be as follows: $a_i\in (0.1,0.5]$ for $i=1,2,3$, $a_i\in [0.3,1]$, $i=1,2,3,4$, $d_1\in [0.15,0.3]$, $r_1\in [1.2,1.6]$, $r_1\in \left[0.3,0.5\right],\gamma \in \left[0.3,0.5\right]$ and $\lambda \in \left[0.01,0.05\right]$; other parameters are the same as ones given in Table 1. Furthermore, the parameters must hold the following conditions [47]: $a_3\le a_1\le a_2$ and $b_1\ge b_2=1$.

Conspicuously, as can be seen in Table 3, the 2-norm of the input signal for the proposed control method is less than that of the Watkin’s Q-learning method; that means that the proposed control method used less drug than the conventional controller. In any case, the number of episodes for the proposed control method is less than 500 episodes, and it is significantly less than the required number of episodes for the Watkins Q-learning algorithm’s convergence, which is 50,000 [57]; therefore, the proposed control method is 100 times faster than the Watkins Q-learning algorithm in term of convergence rate.

Table 3 implies that the 2-norm of the error in the case in which the proposed control method was used decreased by 35 percent for scenario “A” and 24 percent for scenario “B”, in comparison with the case in which the Watkin’s Q-learning algorithm was used. In any case, the 2-norm of the variance of the error and amount of drug usage decreased by 86 percent and 1 percent, respectively, for scenario “A” and 83 percent and 10 percent for scenario “B”.

It is noteworthy to mention that the proposed control method can be exerted by clinicians to find the optimal amount of drug based on the patient’s state to eradicate tumor cells. To this aim, clinicians must first obtain the patients’ parameters by measuring the patient’s state and fit the logged data to the mathematical model of the patient given in Equation (11). Then, using the mathematical model, the control agent should be pre-trained. Afterwards, the control agent would be able to offer the optimal dosage of the drug for each day. Keeping that in mind, the training process of the control agent goes on during the chemotherapy process. For this reason, the control framework performance would not be affected by varying the patient’s parameters during chemotherapy.

6. Conclusions

This study was aimed at the investigation of a Caputo–Fabrizio fractional-order model of cancer chemotherapy treatment. At first, the existence and uniqueness of the solutions of the proposed model were proven through a fix-point theorem and an iterative method. After that, since applying optimal polices for drug dosage is crucial, we proposed to formulate the control problem of the chemotherapy treatment as an optimization problem and find optimal actions using a fuzzy reinforcement learning algorithm. The significant features of the designed fuzzy reinforcement learning-based method are its model-free approach as well as its optimal performance. Finally, three scenarios were considered to evaluate the performance of the proposed control technique. The results for the proposed control method demonstrate the effectiveness of the controller in the annihilation of tumor cells for young patients and elderly patients. The control method has the ability to bring the level of normal cells back to its normal range. In addition, the results implicate the robustness of the proposed control method against the uncertainties of the patients’ parameters. Moreover, because of being real-time, the performance of the proposed control method would not be affected even if the patients’ parameters had been changed during chemotherapy treatment. Also, the presented comparison with the Watkin’s Q-learning method conspicuously demonstrated the superiority of the proposed method in terms of the annihilation of the tumor cells in simulated patients and the drug usage during chemotherapy treatment. While in the current study it was shown that the fuzzy reinforcement learning algorithm is also able to achieve a good level of optimization in this task, deep reinforcement learning has the potential to achieve even better results by taking into account a greater amount of data and making more sophisticated decisions. However, it is important to note that this is an area of active research, and it is not yet clear how well deep reinforcement learning will perform in this specific application. Further research and development will be needed to determine whether this approach is practical and effective for scheduling cancer chemotherapy drug dosing. Furthermore, it would be helpful to run a statistical test using variance to determine whether there are statistically significant differences between two or more categorical groups.