A Novel Approach to Modeling Incommensurate Fractional Order Systems Using Fractional Neural Networks

Abstract

This research explores the application of the Riemann–Liouville fractional sigmoid, briefly ${}^{RL}F\sigma$, activation function in modeling the chaotic dynamics of Chua’s circuit through Multilayer Perceptron (MLP) architecture. Grounded in the context of chaotic systems, the study aims to address the limitations of conventional activation functions in capturing complex relationships within datasets. Employing a structured approach, the methods involve training MLP models with various activation functions, including ${}^{RL}F\sigma$, sigmoid, swish, and proportional Caputo derivative ${}^{PC}\sigma$, and subjecting them to rigorous comparative analyses. The main findings reveal that the proposed ${}^{RL}F\sigma$ consistently outperforms traditional counterparts, exhibiting superior accuracy, reduced Mean Squared Error, and faster convergence. Notably, the study extends its investigation to scenarios with reduced dataset sizes and network parameter reductions, demonstrating the robustness and adaptability of ${}^{RL}F\sigma$. The results, supported by convergence curves and CPU training times, underscore the efficiency and practical applicability of the proposed activation function. This research contributes a new perspective on enhancing neural network architectures for system modeling, showcasing the potential of ${}^{RL}F\sigma$ in real-world applications.

1. Introduction

Fractional calculus (FC) serves as an extension of classical calculus, expanding the concepts of integral and derivative. The increased interest in fractional-order integrals and derivatives is due to their unique non-locality and memory characteristics [1,2]. Recent research in practical systems modeling highlights a growing fascination among scholars with fractional-order calculus [3,4,5]. The use of fractional-order operators has significantly improved the precision of mathematical models in representing the behavior of various real dynamical systems. Examples include applications in Lithium-Ion battery State of Charge (SoC) [6], a flexible model for supercapacitors [7,8], the transient behavior of photovoltaic solar panels [9], understanding the mechanism of atrial fibrillation [10], thermal dynamics of buildings [11], analysis of viscoelasticity in asphalt mixtures [12], behavior of magneto-active elastomers in magnetic fields [13], and modeling macroeconomic systems [14].

Moving on, the domain of system identification has been a focal point of research for numerous decades. Historically, scholars relied on conventional methodologies such as linear systems theory and signal processing to model and scrutinize intricate systems [15,16,17], as well as advanced methods including the predictor-corrector compact difference scheme [18] and others [19,20]. However, the surge in data accessibility and computational capabilities has led to a rising inclination toward nonlinear techniques. Including additive noise in modeling is also broadening the scope of the problem [21]. The adoption of Artificial Neural Networks (ANNs) for system identification has gained prominence due to their adeptness in modeling intricate nonlinear systems. ANNs can approximate any continuous function with remarkable precision, rendering them an ideal instrument for system identification [22,23,24,25]. Nevertheless, the training of ANNs can be computationally demanding and necessitates substantial data volumes. Recently, Fractional Neural Networks (FNNs) have emerged as a promising realm for modeling complex systems that manifest non-local behaviors [26,27,28]. FC offers a potent tool for modeling non-local systems, a prevalent occurrence in various engineering and scientific applications. The amalgamation of ANNs and FC for system identification has garnered increasing interest. FNNs have been suggested as a means to synergize the strengths of both ANNs and FC in modeling complex systems characterized by non-local behaviors [29,30,31,32,33].

Additionally, fractional-order models can be categorized into two groups: commensurate and incommensurate models [34,35]. Each state variable may possess a distinct order in incommensurate state-space models, while in commensurate models, the derivative orders for all state variables are identical [36]. An incommensurate fractional-order system (IFOS) is a dynamic system or process exhibiting fractional-order behavior, which is not a rational number [37]. Some real-world systems and processes cannot be accurately described using integer-order models; instead, they exhibit fractional-order dynamics. Fractional-order systems (FOSs) are described by fractional differential equations (FDEs), where the differentiation order is a number, typically between 0 and 1 [38]. IFOSs with order values that are irrational can lead to particularly intricate and challenging behaviors in the system’s response, as the non-integer order introduces memory effects and long-range dependencies that are not present in integer-order systems [39,40]. The study of these systems is still an active and evolving area of research, and their applications range from modeling physical phenomena to improving control strategies for complex processes.

Finally, this paper proposes extending the applications of the developed fractional sigmoid activation function for behavior identification and prediction. For numerical validation, the incommensurate fractional-order Chua’s system is used, and a comparative analysis is presented. Additionally, this paper strives to maximize performance in system modeling, achieve faster convergence, and reduce CPU training time when using neural networks. Furthermore, Chua’s circuit is known for its complex and chaotic behavior. Its dynamics exhibit nonlinearity and sensitivity to initial conditions, making it a challenging and interesting system to study. The chaotic nature of the system provides a rigorous test for the predictive capabilities of the proposed fractional neural network. Chaotic systems are known for their sensitivity to perturbations, and successfully predicting their behavior can showcase the robustness of the proposed model. From the above comprehensive discussion, this study’s vital contributions are,

• To incorporate the newly developed fractional activation function into system modeling, more specifically for estimating the output of Chua’s model using simulation data.
• To compare the state-of-the-art method with available classical activation functions and existing fractional activation functions.
• To evaluate and verify the new FNN-based black-box modeling of Chua’s circuit based on data reduction and trainable parameter reduction.

2. Fractional Theory and Proposed Activation Function

The study of FC involves a variety of techniques and methods, such as the Grunwald–Letnikov fractional difference operator, the Riemann–Liouville fractional integral and derivative, the Caputo fractional derivative [41], and the newly developed definition known as the conformable fractional derivative (CFD) [42]. In this paper, the CFD defined below is used.

Given a function $f:R\to R$, the improved Riemann–Liouville-type conformable fractional derivative (${}^{RL}$CFD) [43] of f of order $\alpha$, is defined by:

$${}_a^{RL}{T}_{\alpha }\left(t\right)=\underset{\epsilon \to 0}{lim}\left[(1-\alpha )f\left(t\right)+\alpha \frac{f(t+\epsilon {(t-a)}^{1-\alpha }-f\left(t\right)}{\epsilon }\right]$$

(1)

Consider first the classical sigmoid function as

$$\sigma \left(t\right)=\frac{1}{1+e^{-t}},$$

(2)

Using (1) and taking $f\left(t\right)=e^{-t}$, one can obtain the function

$${}_0^{RL}{T}_{\alpha }\left(t\right)=(1-\alpha )\left(e^{-t}\right)-\alpha t^{1-\alpha }{e}^{-t},$$

(3)

Substituting (3) with the exponential function in (2), we obtain

$${}^{RL}F\sigma \left(t\right)=\frac{1}{1+{}_0^{RL}{T}_{\alpha }\left(t\right)}.$$

(4)

Table 1. Proposed fractional sigmoid function and its derivative.

Type	Function	Derivative
${}^{RL}F\sigma \left(t\right)$	$\frac{1}{1+{}_0^{RL}{T}_{\alpha }\left(t\right)}$	${}^{RL}F\sigma \left(t\right)(1-{}^{RL}F\sigma \left(t\right))$

portrays the equation of the proposed fractional sigmoid activation function and its derivative. Figure 1a shows the characteristics of the ${}^{RL}F\sigma \left(t\right)$ with different values of $\alpha$. For smaller values of $\alpha$, the slope gradually becomes steeper, and for $\alpha =1$, the traditional sigmoid activation function is achieved. Likewise, the derivative of ${}^{RL}F\sigma \left(t\right)$ in Figure 1b shows that the value of $\alpha$ determines the amplitude of the curve.

Table 2. Comparison of activation functions.

Activation Function	Diminishing Gradients	Better Generalization	Computational Efficiency	Faster Convergence
${}^{RL}F\sigma$	No	Yes	No	Yes
$\sigma$ [44]	Yes	No	No	No
Swish [45]	No	No	No	No
${}^{PC}\sigma$ [46]	No	Yes	No	No

provides a concise overview of different sigmoid-based activation functions commonly used in NNs, giving insight into their key characteristics. These characteristics include the presence of diminishing gradients, the ability to promote better generalization, computational efficiency, and the speed of convergence during training. The table clearly compares the properties of both standard and fractional activation functions, highlighting the advantages and limitations of each.

3. Problem Statement

Fractional-Order Chua’s Circuit

Chua’s circuit, also known as Chua’s oscillator, is a simple electronic circuit that exhibits nonlinear dynamical phenomena, such as bifurcation and chaos [47], as illustrated in Figure 2. Chua’s circuit comprises various components, including resistors, capacitors, and operational amplifiers, as shown in Figure 3. Its crucial element is Chua’s diode, a two-terminal nonlinear device with a piecewise-linear current-voltage characteristic. This nonlinearity is what allows the circuit to exhibit chaotic oscillations. Chaotic systems, like this one, display unpredictable and highly sensitive behaviors to initial conditions, making them useful in various applications, such as secure communications and random number generation [48,49].

FDEs provide a mathematical framework to describe systems with memory, complex dynamics, and power-law behaviors that traditional integer-order differential equations cannot adequately capture [50]. The incommensurate fractional-order Chua’s system with disturbance [51] is defined as:

$$\left\{\begin{array}{c}{D}^{0.98}{x}_1=a_1(x_2-x_1-m_1{x}_1-f\left(x_1\right))+D_1(x,t),\hfill \\ D^{0.98}{x}_2=x_1-x_2+x_3+D_2(x,t),\hfill \\ D^{0.94}{x}_3=-q_2{x}_2-a_3{x}_3+u+D_3(x,t),\hfill \\ y=x_1\hfill \end{array}\right.$$

(5)

where $D_1(x,t)=0.005sin\left(1.5t\right)cos\left(0.6x_1{x}_2\right)$, $D_2(x,t)=0.04cos\left(1.5t\right)cos\left(0.6x_2{x}_3\right)$, $D_3(x,t)$$=0.04sin\left(1.5t\right)$$sin\left(1.5{x_1}^2{x}_3\right)$ are the external disturbances, $f\left(x_1\right)=\frac{1}{2}(m_0-m_1)$$\left(\left|x_1+1\right|-\left|x_1-1\right|\right)$ is the nonlinear function, $a_1=10.725$, $a_2=10.593$, $a_3=0.268$ are the system parameters, $m_0=-0.7872$, $m_1=-1.1726$ and the initial value is $x\left(0\right)={(1.2,-0.1,-0.9)}^T$.

4. Dataset Description and Neural Model Design

The Chua’s circuit was modeled and simulated for 60 s using MATLAB. The simulated data consists of 12,000 samples taken at a sampling rate of 0.005 s. Figure 4 presents a visual representation of the input and output data of the Chua’s circuit.

The MLP architecture is utilized to test and validate the results using proposed functions compared to utilizing all activation functions presented in Table 2 to estimate the chaotic output of Chua’s circuit model. The architecture consists of an input layer with one node, two hidden layers, and an output layer with one output node. Figure 5 displays the general architecture used to map the input and output data of Chua’s circuit.

First, for each comparison, only the hidden layer activation functions are changed, while the output layer activation functions (linear) remain constant. The proposed function-based MLP models are trained for 25 epochs, whereas other MLP models are trained for 50 epochs. Also, based on initial results, in relation to the convergence curves, even though the conventional model reaches maximum accuracy within 25 epochs, it is not flexible enough to interpret the complex relationship between the dataset and suffers heavily when predicting the outputs. Therefore, other models are trained for 50 epochs for maximum training and prediction accuracy.

Table 3. Network hyper-parameters for training and validation.

Loss	$\mathrm{M}\mathrm{S}\mathrm{E}=\frac{1}{\mathit{N}}{\displaystyle \sum _{\mathit{i}=1}^{\mathit{N}}}({\mathit{y}}_{\mathit{i}}-\stackrel{\wedge }{{\mathit{y}}_{\mathit{i}}}{)}^2$
Hidden layers	$L=2$
Neurons	$n=16$ per layer
Learning rate	$\eta =0.001$
Batch size	$m=32$
Fractional order	$\alpha =0.3$

presents the fixed parameters of the MLP model used for training and validation.

5. Numerical Investigations

This section presents a comparative analysis of the MLP model utilizing the proposed ${}^{RL}F\sigma$, sigmoid [44], swish [45], and ${}^{PC}\sigma$ [46] functions. The evaluation is carried out by estimating the chaotic dynamics of the Chua’s circuit using 80% of the data for training and 20% for testing. The analysis focuses on two key metrics: Mean Squared Error (MSE) and the Coefficient of Determination ($R^2$). Three sets of analyses are performed to compare the performance of all models comprehensively. Firstly, the model is trained with all dataset samples and the complete MLP model. Secondly, the impact of dataset size is examined. Lastly, model parameter reduction is considered. The effectiveness of ${}^{RL}F\sigma$ is also evaluated in terms of its influence on the convergence rate, adaptability, and the CPU training time required to achieve maximum accuracy. The experiments were conducted on an Anaconda platform, utilizing the Jupyter Notebook environment, on a computer system featuring an 8th generation Intel i7 processor with a clock speed of 4.0 GHz and 16 GB of RAM.

5.1. Case 1: Full Model Analysis

All MLP models are trained and tested with full 12,000 sample data and complete network architecture with 609 trainable parameters.

Table 4. Performance comparison of ${}^{RL}F\sigma$ for case 1.

	${\mathit{R}}^2$ (Training)	${\mathit{R}}^2$ (Testing)	MSE ($\times {10}^{-3}$)
sigmoid	0.9604	0.9584	196.8
swish	0.9578	0.9558	207.8
${}^{PC}\sigma$	0.9005	0.9008	475.3
Proposed	0.9860	0.9853	69.4

presents the numerical comparison of the proposed sigmoid MLP model with other MLP models based on $R^2$ and MSE values for case 1. It can be observed that the proposed model outperforms all the other models in both the training and testing phases, achieving higher accuracy values. When comparing the error values, the proposed function reduces the error by 65% compared to the classical model, 67% compared to the enhanced classical model, and 85% compared to the fractional model. This significant reduction in error percentage contributes significantly to the prediction accuracy.

Figure 6 illustrates the training convergence curve of all the models. It can be observed that the proposed model has faster convergence as it achieves higher accuracy and a low loss value right from epoch 1. In comparison, the proposed model achieves 40% more accuracy than all other models at epoch 1 and converges quickly to maximum accuracy within 25 epochs. Moving on, Figure 7 below shows the estimated output of Chua’s circuit by all models and compares it with the real output. It can be observed that, due to the superior performance of the proposed model, it is able to estimate the output more accurately. The key difference that can be observed is that all other models suffer from accurately detecting the chaotic and abrupt peak changes in the circuit dynamics.

Table 5. Training time to achieve maximum accuracy for case 1.

Activation Function	CPU Time (Hrs:Mins:Secs)
sigmoid	00:02:05
swish	00:02:23
${}^{PC}\sigma$	00:02:54
Proposed	00:01:01

presents the CPU training time to achieve maximum accuracy for all the models in case 1. Due to the fast convergence properties of the proposed function, the CPU training time is reduced drastically. It can be observed that the proposed model trains approximately 50% faster than the classical model, 54% faster than the enhanced classical model, and 60% faster than the fractional model.

5.2. Case 2: Dataset Size Impact

For this analysis, the model is trained and tested with a 50% reduction in data samples, and the trainable network parameters remained the same as in case 1.

Table 6. Performance comparison of ${}^{RL}F\sigma$ for case 2.

	${\mathit{R}}^2$ (Training)	${\mathit{R}}^2$ (Testing)	MSE ($\times {10}^{-3}$)
sigmoid	0.9309	0.9292	346.7
swish	0.9256	0.9247	374.2
${}^{PC}\sigma$	0.8720	0.8729	849.1
Proposed	0.9703	0.9709	137.3

presents the numerical analysis of case 2. In regards to training and testing performance, it can be observed that the proposed model again outperforms all the other models even with a 50% reduced dataset size. In comparison to case 1, the proposed model accuracy only drops by 1% on average and other models accuracy drops by 3% on average. When comparing the error value, the proposed function reduces error by 60% when compared to the classical model, 63% when compared to the enhanced classical model, and 84% when compared to the fractional model.

Figure 8 illustrates the training convergence curve of case 2. Again, the proposed model displays faster convergence as it achieves 83% more accuracy on average compared to all other models at epoch 1. Figure 9 below shows the estimated output based on a reduced training dataset size. It can be observed that even with limited information about the system, the proposed model is able to estimate the output more accurately, whereas the other models suffer greatly when compared to the predictions of case 1.

Table 7. Training time to achieve maximum accuracy for case 2.

Activation Function	CPU Time (Hrs:Mins:Secs)
sigmoid	00:01:02
swish	00:01:12
${}^{PC}\sigma$	00:01:26
Proposed	00:00:34

presents the CPU training time to achieve maximum accuracy for case 2. It can be observed that the proposed model trains approximately 45% faster than the classical model, 53% faster than the enhanced classical model, and 60% faster than the fractional model.

5.3. Case 3: Network Parameter Reduction

For this analysis, instead, the network trainable parameters are reduced by 80%, which brings down the model with only 111 trainable parameters, and the model is trained and tested using full 12,000 data samples.

Furthermore,

Table 8. Performance comparison of ${}^{RL}F\sigma$ for case 3.

	${\mathit{R}}^2$ (Training)	${\mathit{R}}^2$ (Testing)	MSE ($\times {10}^{-3}$)
sigmoid	0.8912	0.8845	538.9
swish	0.9046	0.9022	492.4
${}^{PC}\sigma$	0.8584	0.8554	748.9
Proposed	0.9828	0.9822	83.8

presents the numerical analysis of case 3. Similar results are obtained, as seen in the previous two cases, whereby the proposed model outperforms all the other models in both the training and testing phases. When looking at the accuracies obtained in case 1, the proposed model still achieves similar accuracies, whereas the other models suffer greatly with an average of a 5% reduction in accuracies. When comparing the error values, the proposed function reduces error by 84% compared to the classical model, 83% compared to the enhanced classical model, and 89% compared to the fractional model.

Figure 10 illustrates the training convergence curve of case 3. Again, it can be observed that the proposed model has faster convergence. In comparison, the proposed model achieves 95% more accuracy compared to all other models at epoch 1, even with reduced trainable parameters. Figure 11 similarly shows the estimated output. Once again, even with a reduced network architecture, the proposed model is able to efficiently and accurately predict the dynamics of the circuit.

Table 9. Training time to achieve maximum accuracy case 3.

Activation Function	CPU Time (Hrs:Mins:Secs)
sigmoid	00:01:16
swish	00:01:32
${}^{PC}\sigma$	00:02:02
Proposed	00:00:41

presents the CPU training time to achieve maximum accuracy for case 3. It can be observed that the proposed model trains approximately 46% faster than the classical models, 55% faster than the enhanced classical models, and 66% faster than the fractional models.

6. Discussion

The study encompasses three distinct cases, each shedding light on different aspects of the proposed activation function’s performance. In Case 1, where the full model analysis is conducted with the complete dataset, ${}^{RL}F\sigma$ emerges as the standout performer. It consistently outperforms traditional activation functions such as sigmoid, swish, and ${}^{PC}\sigma$, achieving higher $R^2$ values and significantly reducing MSE. The faster convergence of the proposed model from the early epochs, coupled with notably lower CPU training time, underscores its efficiency in capturing the intricate dataset relationships.

Moving on to Case 2, which explores the impact of dataset size reduction, the proposed activation function continues to showcase resilience. Even with a 50% reduction in data samples, ${}^{RL}F\sigma$ maintains superiority in terms of $R^2$ values and MSE compared to other models. The convergence curve further illustrates its robust learning capability with limited data, suggesting adaptability to scenarios where obtaining comprehensive datasets may be challenging.

In Case 3, where network parameter reduction is implemented by 80%, ${}^{RL}F\sigma$ once again stands out, maintaining high accuracy and outperforming other models in terms of $R^2$ values and MSE. The model’s ability to generalize and accurately capture system dynamics, even with a simplified architecture, highlights the efficacy of the proposed activation function.

In conclusion, the comprehensive numerical analyses consistently support the superiority of the proposed function across various scenarios. Its efficiency in modeling Chua’s circuit dynamics, adaptability to reduced datasets, and resilience to network parameter reductions make it a promising candidate for real-world applications such as weather pattern prediction, fluid dynamics, and biological systems modeling. The study provides valuable insights into the specific application of the proposed activation function and prompts considerations for broader applications and avenues for future research. An essential direction for continued research involves the H1-norm error analysis of a robust ADI method applied to graded meshes for three-dimensional sub-diffusion problems. This exploration aims to enhance our understanding of the method’s accuracy and reliability in addressing complex scenarios. Additionally, investigating an efficient ADI difference scheme for the nonlocal evolution problem in three-dimensional space represents a crucial avenue for advancement. This scheme can contribute to developing effective numerical approaches for handling nonlocal phenomena, expanding the applicability of such methods in various scientific domains.