Selection of Appropriate Symbolic Regression Models Using Statistical and Dynamic System Criteria: Example of Waste Gasification

Abstract

In this paper, we analyze the interpretable models from real gasification datasets of the project “Centre for Energy and Environmental Technologies” (CEET) discovered by symbolic regression. To evaluate CEET models based on input data, two different statistical metrics to quantify their accuracy are usually used: Mean Square Error (MSE) and the Pearson Correlation Coefficient (PCC). However, if the testing points and the points used to construct the models are not chosen randomly from the continuum of the input variable, but instead from the limited number of discrete input points, the behavior of the model between such points very possibly will not fit well the physical essence of the modelled phenomenon. For example, the developed model can have unexpected oscillatory tendencies between the used points, while the usually used statistical metrics cannot detect these anomalies. However, using dynamic system criteria in addition to statistical metrics, such suspicious models that do fit well-expected behavior can be automatically detected and abandoned. This communication will show the universal method based on dynamic system criteria which can detect suitable models among all those which have good properties following statistical metrics. The dynamic system criteria measure the complexity of the candidate models using approximate and sample entropy. The examples are given for waste gasification where the output data (percentage of each particular gas in the produced mixture) is given only for six values of the input data (temperature in the chamber in which the process takes place). In such cases instead, to produce expected simple spline-like curves, artificial intelligence tools can produce inappropriate oscillatory curves with sharp picks due to the known tendency of symbolic regression to produce overfitted and relatively more complex models if the nature of the physical model is simple.

1. Introduction

We developed a set of curves for the gasification of municipal solid waste [1] using symbolic regression [2]. The curves were tested statistically, and among those with satisfactory results in terms of Mean Square Error (MSE) and Pearson Correlation Coefficient (PCC) some are not acceptable because they show unexpected oscillatory behavior. To eliminate them, we applied dynamic system criteria by measuring complexity using approximate and sample entropy where the inappropriate curves can be eliminated to persist Occam’s Razor [3]. We prefer not to present more statistical metrics aside MSE and PCC because they cannot measure behavior between the testing points (on the other hand, dynamic system criteria can).

Gasification of municipal solid waste and biomass gives the different compositions of the synthetic gas (syngas) depending on the gasification temperature in the process [4]. Based on real measurements of a plasma gasifier, a symbolic model for an important component of the produced syngas is constructed where the percentage in the mixture is given depending on the gasification temperature. To construct these symbolic syngas composition models, the artificial intelligence provided outcomes using symbolic regression software tools AI Feynman [5] and PySR [6]. The candidate symbolic models were chosen among those with better statistical metrics; those with a lower Mean Square Error (MSE) and with the Pearson Correlation Coefficient (PCC) close to one. However, it was discovered that some of the obtained symbolic models, which fit very well the measured gasification datasets using statistical metrics, are of oscillatory nature, which was not expected and does not reflect the true physical properties of the modelled gasification process. However, using dynamic system criteria in addition to statistical metrics, such suspicious symbolic models that do fit well the measured gasification datasets were automatically detected and abandoned. This communication will show the universal method based on dynamic system criteria which can detect suitable models among all those which has good properties following statistical metrics. The dynamic system criteria measure the complexity of the candidate models using approximate and sample entropy. Our results indicate that candidate symbolic regression models with oscillations and other non-physical phenomena have higher complexity and can be automatically detected and excluded by approximate and sample entropy to persist Occam’s Razor “science always prefers the simpler model or representation of two which give similar accuracy” [6]. Consequently, we propose that the dynamic system criteria based on approximate or sample entropy should be used for the automated evaluation of symbolic regression models, as it is not enough to evaluate the models by statistical metrics.

2. Gasification Models

Gasification models were developed for the production of hydrogen (H₂) and carbon dioxide (CO₂) from municipal solid waste for only six different temperatures, while the measurements were repeated four times.

These functions were introduced in symbolic regression software to provide numerically stable logarithm-based functions, which are defined for all real numbers. As logarithms are defined only for positive non-zero numbers, logarithms pose numerical problems in the symbolic regression procedure, when the argument is negative.

Mean Square Error (MSE) and with the Pearson Correlation Coefficient (PCC) were calculated using functions of Python 3.9 by:

MSE = np.square(np.subtract(data,y_pred)).mean()

coef = np.corrcoef(data,y_pred)[0][1]

where “data” means the measured data set and “y_pred” means the predicted values using the selected symbolic regression model.

MSE and corr.coef were calculated for all measured data.

The presented models in Matlab notifications are given in Appendix A to this Communication.

2.1. Hydrogen H₂

The train set for hydrogen H₂ is given in

Table 1. Train data for H₂.

t (°C)	750	800	900	1000	1050	1100
H2 (%)	9.75	10.98	16.05	12.88	12.33	11.83

while three test sets are given in

Table 2. Test data for H₂.

t (°C)	750	800	900	1000	1050	1100
H2 (%)	9.69	11.54	15.91	12.77	12.48	11.56
H2 (%) 1	9.88	10.29	16.23	13.29	12.44	12.12
H2 (%) 2	9.68	11.12	16.01	12.58	12.08	11.82

¹ Second test measurement, ² Third test measurement.

.

The expected shape of the modelled curves for hydrogen H₂ is given in Figure 1 where the trendline is based on data from Table 1 and was produced in MS Excel as a polynomial curve of order 4.

Using symbolic regression tools, three different models were produced as follows.

2.1.1. Model 1 of Hydrogen H₂

The first developed model is given in Equation (1)

$$H_2(\%)=\frac{8.9065269419}{\cos \left(\cos \left(e^{\sin \left(t+1\right)}-2\right)\right)}.$$

(1)

This model performs good statistical metrics; MSE = 0.2897 and PCC = 0.9728 while anyway, it shows oscillatory tendencies as can be seen in Figure 2.

2.1.2. Model 2 of Hydrogen H₂

The second developed model is given in Equation (2)

$$\begin{array}{c}{H}_2(\%)=\mathrm{logm}2(7.744805·{(-9.837645+\mathrm{logm}2\left(t\right))}^{-1.5165994})+0.0054770974·t\\ \mathrm{logm}2\left(t\right)={\log }_2\left(\left|t\right|+{10}^{-8}\right)\end{array}\}.$$

(2)

This model performs good, as indicated by statistical metrics, MSE = 0.08143851 and PCC = 0.989470395. Model 2 has improved shape compared with Model 1, despite the undesired tendency towards a sharp peak. It is given in Figure 3.

2.1.3. Model 3 of Hydrogen H₂

The second developed model is given in Equation (3)

$$\begin{array}{c}{H}_2(\%)=6.5368786+\mathrm{logm}2({\left|-9.864973+\mathrm{logm}2\left(t\right)\right|}^{-2.1570945})\\ \mathrm{logm}2\left(t\right)={\log }_2\left(\left|t\right|+{10}^{-8}\right)\end{array}\}.$$

(3)

This model performs good statistical metrics, MSE = 0.362862372 and PCC = 0.952248777. It shows the same tendency as Model 2. Model 3 is given in Figure 4.

2.2. Hydrogen CO₂

The train set for hydrogen CO₂ is given in

Table 3. Train data for CO₂.

t (°C)	750	800	900	1000	1050	1100
CO2 (%)	8.13	9.8	11.93	11.33	11.53	12.3

while three test sets are given in

Table 4. Test data for CO₂.

t (°C)	750	800	900	1000	1050	1100
CO2 (%)	8.05	9.52	11.63	11.5	11.56	12.27
CO2 (%) 1	8.09	9.83	12.31	11.23	11.5	12.21
CO2 (%) 2	8.24	10.05	11.84	11.27	11.54	12.43

¹ Second test measurement, ² Third test measurement

.

The expected shape of the modelled curves for hydrogen CO₂ is given in Figure 5 where the trendline is based on data from Table 3 and was produced in MS Excel as a polynomial curve of order 4.

Using the symbolic regression tools, three different models were produced as follows.

2.2.1. Model 1 of Carbon Dioxide CO₂

The first developed model is given in Equation (4)

$$\begin{array}{c}{\mathrm{CO}}_2(\%)=\mathrm{logm}10{\left(x·e^{\sin \left(x+0.6718609\right)}\right)}^2\\ \mathrm{logm}10\left(t\right)=\ln \left(\left|t\right|+{10}^{-8}\right)\end{array}\}$$

(4)

This model performs good statistical metrics, MSE = 0.3481 and PCC = 0.9171, while it shows oscillatory tendencies, as can be seen in Figure 6.

2.2.2. Model 2 of Carbon Dioxide CO₂

The first developed model is given in Equation (5)

$$\begin{array}{c}{\mathrm{CO}}_2(\%)=\mathrm{logm}\left(-0.060511474·t^2\right)+2.0503674\\ \mathrm{logm}\left(t\right)=\ln \left(\left|t\right|+{10}^{-8}\right)\end{array}\}.$$

(5)

This model performs good statistical metrics, MSE = 0.1985 and PCC = 0.9519, with a good shape of the developed curve, as can be seen in Figure 7.

3. Dynamic System Criteria for Selection of Appropriate Models

As observed, it is possible to construct many different models of the investigated phenomena having comparable precision. Now, the task is to select one of them that can be signed as the best choice under the assumption of Occam’s Razor [3]. For this purpose, the qualification tools from the area of dynamical systems, like approximate $E_{app}$ and sample $E_{samp}$ entropy [7,8,9,10,11], can be applied.

Hence, the selection process, based on observation of the measure of complexity, works as follows. Firstly, construct models (e.g., as in the previous section). Secondly, measure the complexity of each model and order them with respect to this measure (in our case approximate and sample entropy). Finally, pick the model with the smallest complexity value (at this stage the assumption of Occam’s Razor [3] is applied). Here, note that since the minimum need not be unique the output of this selection process will always exist, but should not be unique in general as we will show in the next section.

3.1. Entropy Notions

Approximate entropy $E_{app}$ and sample entropy $E_{samp}$ are tools of complexity measurement that were investigated by many authors and applied in numerous research fields (e.g., [12,13,14]) to measure and compare studied cases’ complexity.

3.1.1. Approximate Entropy E_app

Recall these notions are defined for the input vector $X=x_1,x_2, \dots ,x_N$ of length $N$. Approximate entropy $E_{app}$ is defined in Equation (6)

$$E_{app}\left(X,m,r\right)={\mathsf{\Phi }}^m\left(r\right)-{\mathsf{\Phi }}^{m+1}\left(r\right),$$

(6)

where ${\mathsf{\Phi }}^m\left(r\right)=\left(N-m+1\right){{\displaystyle \sum }}_{i=1}^{M-m+1}\log \left(C_i^m\left(r\right)\right)$ and $C_i^m\left(r\right)$ is the number of $u_m\left(j\right)$ such that $d\left(u_m\left(i\right),u_m\left(j\right)\right)\le r$, divided by $N-m+1$. Here, $u_m\left(i\right)=\left[x\left(i\right),x\left(i+1\right),\dots ,x\left(i+m-1\right)\right]$ is an element of $m$-dimensional real space, $m, r$ are test parameters and $d\left(p,q\right)=\underset{a}{\max }|p\left(a\right)-q\left(a\right)|$ is the maximum metric. For these parameters holds: $m$ is the length of the window, $r$ is the diameter of the region with a similar subsequence.

3.1.2. Sample Entropy E_samp

On the other hand, sample entropy $E_{samp}$ is given in Equation (7)

$$E_{samp}\left(X,m,r\right)=-\ln \frac{A}{B},$$

(7)

where $A$ is the number of template vector pairs such that $d^c\left(u_{m+1}\left(i\right),u_{m+1}\left(i\right)\right)<r$, and $B$ is the number of template vector pairs such $d^c\left(u_m\left(i\right),u_m\left(i\right)\right)<r$. Here, $d^c$ is the Chebyshev distance and parameters $m, r$ have the same meaning as in the case of $E_{app}$.

3.2. Benchmark Models Application

To depict previously mentioned complexity measurement tools, classical models from the theory of dynamical systems can be applied. The well-known logistic function can be used as example of normalized models (also can be thought as predictive one). Recall $L_1\left(x\right)=4x\left(1-x\right)$ and $L_1:\left[0,1\right] \to \left[0,1\right]$, this model is well understood from dynamical point of view [15]. The next model can be constructed as a second iteration of $L_1\left(x\right)$ that is $L_2=L_1 · L_1=L_1^2$, and analogously $L_3=L_1 · L_1· L_1=L_1^3$, $L_4=L_1^4$, $L_5=L_1^5$. The evolution of these models is shown in Figure 8a and their corresponding entropies are in Figure 8b, clearly showing increase of entropy while the complexity of the model increases.

3.3. Simulation Outputs

The numerical simulations of our models were performed in Matlab while the continuous models were discretized from 750 °C to 1100 °C by the step of 0.001 °C. Entropies tests were set classically, that is, $r$ was picked as 20% of the standard deviation of the investigated vector and $m$ was set to 1 as the minimum window length. Firstly, note that the test performed on all models coincides so, for simplicity, we can use the abbreviation of entropy for both tests. These outputs, rounded on four decimal places, are summarized in

Table 5. Dynamic system criteria for hydrogen H₂.

	Model 1	Model 2	Model 3
$\mathrm{Approximate} \mathrm{entropy} E_{app}$	0.0442	0.0015	0.0015
$\mathrm{Sample} \mathrm{entropy} E_{samp}$	0.0197	0.0003	0.0003

for hydrogen H₂ and in

Table 6. Dynamic system criteria for carbon dioxide CO₂.

	Model 1	Model 2
$\mathrm{Approximate} \mathrm{entropy} E_{app}$	0.0252	0
$\mathrm{Sample} \mathrm{entropy} E_{samp}$	0.0235	0

for carbon dioxide CO₂.

Model 1 of H₂ (which is periodic with a period of 6.283183 °C) has higher complexity than Model 2 of H₂ and Model 3 of H₂, so Model 1 of H₂ can be denied. It is also observable from Table 5 that entropies of Model 2 of H₂ and Model 3 of H₂ are comparable and much less than Model 1 of H₂.

It is observable from Table 6 that entropy of the Model 1 of CO₂ is much higher than that of Model 2 of CO₂, proving that Model 2 has lower complexity than Model 1 and is then a better choice.

4. Conclusions

We developed symbolic regression models for the gasification of municipal solid waste [16,17,18,19,20,21]. However, these models were developed using limited points of data and so, between these points, it shows unpredicted behavior (sharp picks or oscillatory motions) where all such models were acceptable using statistic metrics (Mean Square Error and the Pearson Correlation Coefficient) as criteria. In the end, the proposed application of approximate $E_{app}$ and sample $E_{samp}$ entropy automatically detected those models with higher complexity contradicting Occam’s Razor assumption. Hence, the models with higher complexity can be excluded from further investigation. Moreover, it is possible to use these dynamic tools automatically in general for decision mechanisms. The example is about gasification of waste, but the shown method for rejection of inappropriate models is of general value and can be used in various scientific fields. It is based on dynamic system criteria and it is based on the measurement of entropy. In future, we would like to also test Symbolic Functional Evolutionary Search (SyFES), that automatically constructs accurate functionals in the symbolic form, which is more explainable to humans, cheaper to evaluate, and easier to integrate into existing software codes [22].

However, the proposed selection method is based on approximate and sample entropy, there are also other tools in the mathematical theory of dynamical systems that can be applied [23,24]. For example, metrics from recurrence quantification analysis (RQA) can be applied (or relevant alternatives mentioned in [25]). We propose these promising tools for further research.

In the end, since the proposed method is addressed in general to any set of models it can be also applied to prediction models. The application of the method to the prediction models is left for future research.