# Interpretable Feature Construction and Incremental Update Fine-Tuning Strategy for Prediction of Rate of Penetration

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

- Introducing advanced feature engineering techniques to construct interpretable features, which align with the physical drilling laws.
- Developing an incremental update fine-tuning strategy to adapt the model to changing drilling conditions.
- Evaluating the performance of the proposed model and comparing it with existing approaches in terms of accuracy and adaptability.

## 2. Methodology

#### 2.1. Dataset Preparation

#### 2.1.1. Data Pre-Processing

_{i}and Y

_{i}are the individual data points for variables X and Y, respectively; $\overline{X}$ and $\overline{Y}$ are the mean values of variables X and Y, respectively; $dCo{v}_{n}^{2}\left(X,Y\right)$ is the distance covariance of X and Y, which is calculated as the average of the pairwise product of the distance matrices of X and Y. $dCo{v}_{n}^{2}\left(X\right)$ is the standard deviation of the distance of X, and $dCo{v}_{n}^{2}\left(Y\right)$ is the standard deviation of the distance of Y.

#### 2.1.2. Sliding Window

#### 2.2. Interpretable Feature Construction with Genetic Programming

_{e}). Second, we generate new features by crossing engineering parameters and bit parameters (D, WOB, RPM, T, SPP, Q, BD, D

_{e}), which partially characterize the effect of bit wear on ROP. Figure 3 shows the original combination of features, which comprise the new features. In this section, we utilize genetic programming (GP), taking inspiration from the method implemented in Python gplearn library, to construct interpretable features. The process of constructing these features, as illustrated in Figure 4, can be described in detail as follows.

#### 2.3. Model Buliding and Incremental Update Fine-Tuning Framework

_{1}and LSTM

_{2}using a historical dataset. When incremental data D

_{1}are obtained, the weights of the LSTM

_{1}layer are frozen, and the feature extractor LSTM

_{2}is updated. Then, the extracted high-dimensional feature information is fed into a fully connected layer to obtain the predicted ROP, thereby incorporating both historical data and the newly acquired real-time data flow, obtaining Model 1. Similarly, when incremental data D

_{2}are obtained, LSTM

_{2}undergoes fine-tuning to update the current Model 2. This incremental update learning approach enables the intelligent model to continuously fine-tune using the real-time data stream and predict the ROP of drilling formation.

#### 2.4. Hyperparameter Tuning

#### 2.5. Evaluation Metrics

_{i}denotes the real ROP; and y

_{pre}denotes the predicted ROP, which are predicted by the developed models.

## 3. Results and Discussion

#### 3.1. Impact of Interpretable Feature Construction

_{1}represents the flow of drilling fluid between the drill bit and the wellbore, where higher values indicate reduced fluid flow, and lower values mean that the drilling fluid flows more freely, indicating better fluid circulation and cuttings removal. The constructed feature p

_{2}reflects the ratio of μ to D

_{e}, which serves as an indicator of wellbore conditions and formation properties. Additionally, p

_{3}explores the changing trend and temporal correlation between the drilling fluid and formation properties. Furthermore, p

_{4}and p

_{5}characterize the friction between the drill bit and rock formation, ultimately affecting the ROP.

#### 3.2. Evaluation of Incremental Update Steps

#### 3.3. Model Comparison Analysis

## 4. Conclusions

- The construction of interpretable features significantly enhances the accuracy of ROP prediction, as evidenced by a reduction in MAPE to 10.6%. The incorporation of these interpretable features effectively mitigates the risk of prediction shift and strengthens the model’s ability to generalize to new datasets.
- The utilization of the incremental update method for fine-tuning the model results in a further decrease in MAPE to 9.8%. The incremental update method offers distinct advantages over static historical data training.
- The proposed integration of interpretable feature construction and the incremental update fine-tuning strategy yields substantial enhancements in both the accuracy and stability of the model. Notably, a MAPE of 7.5% is achieved during testing on new datasets.

## Author Contributions

## Funding

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Khalilidermani, M.; Knez, D. A Survey on the Shortcomings of the Current Rate of Penetration Predictive Models in Petroleum Engineering. Energies
**2023**, 16, 4289. [Google Scholar] [CrossRef] - Maurer, W.C. The “Perfect—Cleaning” Theory of Rotary Drilling. J. Pet. Technol.
**1962**, 14, 1270–1274. [Google Scholar] [CrossRef] - Galle, E.; Woods, H. Best Constant Weight and Rotary Speed for Rotary Rock Bits. In Drilling and Production Practice; American Petroleum Institute: Washington, DC, USA, 1963. [Google Scholar]
- Bourgoyne, A.T.; Young, F.S. A Multiple Regression Approach to Optimal Drilling and Abnormal Pressure Detection. Soc. Pet. Eng. J.
**1974**, 14, 371–384. [Google Scholar] [CrossRef] - Walker, B.H.; Black, A.D.; Klauber, W.P.; Little, T.; Khodaverdian, M. Roller-Bit Penetration Rate Response as a Function of Rock Properties and Well Depth. In Proceedings of the SPE Annual Technical Conference and Exhibition, New Orleans, LA, USA, 5–8 October 1986. [Google Scholar] [CrossRef]
- Hareland, G.; Rampersad, P.R. Drag-Bit Model Including Wear. In Proceedings of the SPE Latin America/Caribbean Petroleum Engineering Conference, Buenos Aires, Argentina, 27–29 April 1994. [Google Scholar]
- Detournay, E.; Richard, T.; Shepherd, M. Drilling Response of Drag Bits: Theory and Experiment. Int. J. Rock Mech. Min. Sci.
**2008**, 45, 1347–1360. [Google Scholar] [CrossRef] - Wiktorski, E.; Kuznetcov, A.; Sui, D. ROP Optimization and Modeling in Directional Drilling Process. In Proceedings of the SPE Bergen One Day Seminar, Bergen, Norway, 5 April 2017. [Google Scholar]
- Norouzi, S.; Nazari, M.E.; Farahani, M.V. A Novel Hybrid Particle Swarm Optimization-Simulated Annealing Approach for CO
_{2}-Oil Minimum Miscibility Pressure (MMP) Prediction. In Proceedings of the 81st EAGE Conference and Exhibition 2019, London, UK, 3–6 June 2019. [Google Scholar] - Farahani, M.R.V.; Shams, R.; Jamshidi, S. A Robust Modeling Approach for Predicting the Rheological Behavior of Thixotropic Fluids. In Proceedings of the 80th EAGE Conference and Exhibition 2018, Copenhagen, Denmark, 11–14 June 2018; pp. 1–5. [Google Scholar]
- Zhu, Z.; Song, X.; Zhang, R.; Li, G.; Han, L.; Hu, X.; Li, D.; Yang, D.; Qin, F. A Hybrid Neural Network Model for Predicting Bottomhole Pressure in Managed Pressure Drilling. Appl. Sci.
**2022**, 12, 6728. [Google Scholar] [CrossRef] - Zhu, Z.; Liu, Z.; Song, X.; Zhu, S.; Zhou, M.; Li, G.; Duan, S.; Ma, B.; Ye, S.; Zhang, R. A Physics-Constrained Data-Driven Workflow for Predicting Bottom Hole Pressure Using a Hybrid Model of Artificial Neural Network and Particle Swarm Optimization. Geoenergy Sci. Eng.
**2023**, 224, 211625. [Google Scholar] [CrossRef] - Zhang, C.; Song, X.; Su, Y.; Li, G. Real-Time Prediction of Rate of Penetration by Combining Attention-Based Gated Recurrent Unit Network and Fully Connected Neural Networks. J. Pet. Sci. Eng.
**2022**, 213, 110396. [Google Scholar] [CrossRef] - Liu, W.; Fu, J.; Tang, C.; Huang, X.; Sun, T. Real-Time Prediction of Multivariate ROP (Rate of Penetration) Based on Machine Learning Regression Algorithms: Algorithm Comparison, Model Evaluation and Parameter Analysis. Energy Explor. Exploit.
**2023**, 01445987231173091. [Google Scholar] [CrossRef] - Barbosa, L.F.F.M.; Nascimento, A.; Mathias, M.H.; de Carvalho, J.A. Machine Learning Methods Applied to Drilling Rate of Penetration Prediction and Optimization—A Review. J. Pet. Sci. Eng.
**2019**, 183, 106332. [Google Scholar] [CrossRef] - Ben Aoun, M.A.; Madarász, T. Applying Machine Learning to Predict the Rate of Penetration for Geothermal Drilling Located in the Utah FORGE Site. Energies
**2022**, 15, 4288. [Google Scholar] [CrossRef] - Al-AbdulJabbar, A.; Mahmoud, A.A.; Elkatatny, S. Artificial Neural Network Model for Real-Time Prediction of the Rate of Penetration While Horizontally Drilling Natural Gas-Bearing Sandstone Formations. Arab. J. Geosci.
**2021**, 14, 117. [Google Scholar] [CrossRef] - Soares, C.; Gray, K. Real-Time Predictive Capabilities of Analytical and Machine Learning Rate of Penetration (ROP) Models. J. Pet. Sci. Eng.
**2019**, 172, 934–959. [Google Scholar] [CrossRef] - Elkatatny, S. Real-Time Prediction of Rate of Penetration in S-Shape Well Profile Using Artificial Intelligence Models. Sensors
**2020**, 20, 3506. [Google Scholar] [CrossRef] [PubMed] - Zhou, F.; Fan, H.; Liu, Y.; Ye, Y.; Diao, H.; Wang, Z.; Rached, R.; Tu, Y.; Davio, E. Application of Xgboost Algorithm in Rate of Penetration Prediction with Accuracy. In Proceedings of the International Petroleum Technology Conference, IPTC, Riyadh, Saudi Arabia, 21–23 February 2022; p. D012S111R002. [Google Scholar]
- Ao, L. Prediction of POR Based on Artificial Neural Network with Long and Short Memory(LSTM). In Proceedings of the 55th U.S. Rock Mechanics/Geomechanics Symposium, Virtual, 18 June 2021. [Google Scholar]
- Safarov, A.; Iskandarov, V.; Solomonov, D. Application of Machine Learning Techniques for Rate of Penetration Prediction. In Proceedings of the SPE Annual Caspian Technical Conference; SPE, Nur-Sultan, Kazakhstan, 15 November 2022; p. D021S013R002. [Google Scholar]
- Liu, H.; Jin, Y.; Song, X.; Pei, Z. Rate of Penetration Prediction Method for Ultra-Deep Wells Based on LSTM-FNN. Appl. Sci.
**2022**, 12, 7731. [Google Scholar] [CrossRef] - Ren, C.; Huang, W.; Gao, D. Predicting Rate of Penetration of Horizontal Drilling by Combining Physical Model with Machine Learning Method in the China Jimusar Oil Field. SPE J.
**2022**, 1–24. [Google Scholar] [CrossRef] - Bizhani, M.; Kuru, E. Towards Drilling Rate of Penetration Prediction: Bayesian Neural Networks for Uncertainty Quantification. J. Pet. Sci. Eng.
**2022**, 219, 111068. [Google Scholar] [CrossRef] - Pacis, F.J.; Alyaev, S.; Ambrus, A.; Wiktorski, T. Transfer Learning Approach to Prediction of Rate of Penetration in Drilling. In Computational Science—ICCS 2022, Proceedings of the 22nd International Conference on Computational Science, London, UK, 21–23 June 2022; Springer International Publishing: Cham, Switzerland, 2022; pp. 358–371. [Google Scholar]
- Li, Y.; Yang, C. Domain Knowledge Based Explainable Feature Construction Method and Its Application in Ironmaking Process. Eng. Appl. Artif. Intell.
**2021**, 100, 104197. [Google Scholar] [CrossRef] - Benesty, J.; Chen, J.; Huang, Y.; Cohen, I. Pearson Correlation Coefficient. In Noise Reduction in Speech Processing; Springer: Berlin/Heidelberg, Germany, 2009; pp. 1–4. ISBN 978-3-642-00296-0. [Google Scholar]
- Székely, G.J.; Rizzo, M.L.; Bakirov, N.K. Measuring and Testing Dependence by Correlation of Distances. Ann. Stat.
**2007**, 35, 2769–2794. [Google Scholar] [CrossRef] - Malinović, N.S.; Predić, B.B.; Roganović, M. Multilayer Long Short-Term Memory (LSTM) Neural Networks in Time Series Analysis. In Proceedings of the 2020 55th International Scientific Conference on Information, Communication and Energy Systems and Technologies (ICEST), Niš, Serbia, 10–12 September 2020; pp. 11–14. [Google Scholar]
- Salman, A.G.; Heryadi, Y.; Abdurahman, E.; Suparta, W. Single Layer & Multi-Layer Long Short-Term Memory (LSTM) Model with Intermediate Variables for Weather Forecasting. Procedia Comput. Sci.
**2018**, 135, 89–98. [Google Scholar] [CrossRef] - Probst, P.; Boulesteix, A.-L.; Bischl, B. Tunability: Importance of Hyperparameters of Machine Learning Algorithms. J. Mach. Learn. Res.
**2019**, 20, 1934–1965. [Google Scholar] - Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Müller, A.; Nothman, J.; Louppe, G.; et al. Scikit-Learn: Machine Learning in Python. J. Mach. Learn. Res.
**2011**, 12, 2825–2830. [Google Scholar] [CrossRef] - Bergstra, J.; Bardenet, R.; Bengio, Y.; Kégl, B. Algorithms for Hyper-Parameter Optimization. In Proceedings of the NIPS’11: Proceedings of the 24th International Conference on Neural Information Processing Systems, Granada, Spain, 12–15 December 2011; p. 24. [Google Scholar]

**Figure 1.**An overall workflow diagram. First, the experimental dataset is obtained through data cleaning, feature selection, normalization and sliding window methods, and it is divided into a training, validation and test set to enhance the data characterization capability by constructing interpretable domain features. Next, the optimal model is established by combining hyperparameter tuning methods. Finally, the model is trained in the form of incremental update fine-tuning, and evaluation analysis is given on the test set.

**Figure 2.**The process of sliding window in constructing sample sets, where w is the length of fixed window (i.e., how much data are used to predict ROP), h is the length of predicted ROP (predicted steps). In this paper, we define h as 3 m (i.e., we predict the ROP of the next 3 m).

**Figure 3.**The construction of new features. Combining formation properties (GR), drilling fluid properties (ρ, μ) and bit parameters (BD, De), some new features can be obtained through genetic programming. Similarly, additional new features are available through the bit parameters (BD, De) and engineering parameters (D, WOB, RPM, T, SPP, Q).

**Figure 5.**The internal structure of a LSTM cell, where x

_{t}is the current input; h

_{t}

_{−1}is the hidden state of the last step; h

_{t}is the hidden state of the next step; C

_{t}

_{−1}is the previous cell state; C

_{t}is the current cell state; f

_{t}is the output of the forget gate; i

_{t}controls the flow of information into the cell state; ${\stackrel{~}{C}}_{t}$ is a new estimation of the cell state; O

_{t}is the output of the output gate; σ is the sigmoid activation function; and tanh is the tanh activation function.

**Figure 8.**The results of real ROP and predicted ROP: (

**a**) Model A; (

**b**) Model B; (

**c**) Model C; (

**d**) Model D; (

**e**) Benchmark model.

**Table 1.**Two types of correlation coefficients between features and ROP. Drilling fluid viscosity here refers to the apparent viscosity of the drilling fluid, also called funnel viscosity, which is defined as the time required for a given volume of drilling fluid to flow through a small hole of a specified size.

Variables | Pearson Correlation Coefficient | Distance Correlation Coefficient |
---|---|---|

Depth (m) | 0.129 | 0.108 |

Weight on bit (kN) | −0.224 | 0.223 |

Rotary speed (rev/min) | 0.354 | 0.380 |

Torque (kN∙m) | 0.221 | 0.292 |

Standpipe pressure (MPa) | 0.349 | 0.323 |

Inlet flow rate (L/s) | −0.194 | 0.236 |

Bit drill distance (m) | 0.265 | 0.284 |

Bit size (mm) | −0.236 | 0.266 |

Drilling fluid density (g/cm^{3}) | 0.419 | 0.413 |

Drilling fluid viscosity (s) | 0.247 | 0.308 |

Gamma ray (API) | −0.049 | 0.225 |

Variables | Category |
---|---|

Depth, D | Operational Variables |

Weight on bit, WOB | Operational Variables |

Rotary speed, RPM | Operational Variables |

Torque, T | Operational Variables |

Standpipe pressure, SPP | Operational Variables |

Inlet flow rate, Q | Operational Variables |

Bit drill distance, BD | Operational Variables |

Bit size, D_{e} | Bit Properties |

Drilling fluid density, ρ | Drilling Fluid Properties |

Drilling fluid viscosity, μ | Drilling Fluid Properties |

Gamma ray, GR | Formation Properties |

Parameters | Minimum | Maximum | Mean Value | Standard Deviation |
---|---|---|---|---|

D (m) | 55 | 7510 | 3949.42 | 1778.15 |

WOB (kN) | 2.1 | 299.9 | 93.16 | 35.57 |

RPM (rev/min) | 7 | 190 | 75.24 | 18.37 |

TOR (kN∙m) | 1.5 | 29.90 | 12.35 | 5.16 |

SPP (MPa) | 4.9 | 33.6 | 19.80 | 5.15 |

Q (L/s) | 530 | 4169 | 2664.29 | 719.98 |

BD (m) | 0 | 1326 | 228.93 | 201.11 |

BS (mm) | 163.5 | 444.5 | 342.87 | 71.53 |

DEN (g/cm^{3}) | 1.08 | 2.3 | 1.63 | 0.38 |

VIS (s) | 37 | 230 | 60.13 | 14.58 |

GR | 11.15 | 237.43 | 64.04 | 22.36 |

ROP (m/h) | 0.26 | 19.35 | 3.75 | 3.16 |

Operator | Property | Function |
---|---|---|

Mutation | Original | −x, x + c, |x|, 1/x |

Crossover | Original | ${x}_{1}\ast {x}_{2}$$,\text{}{x}_{1}+{x}_{2}$ |

Exponent | Non-linear, dynamic | ${e}^{x}$ |

Sqrt | Non-linear, dynamic | $\sqrt{x}$ |

Log | Non-linear, dynamic | $\mathrm{log}\left(x\right)$ |

Delay | Temporal | ${x}_{t-{step}^{\text{}1}}$ |

^{1}step is an offset in a single operation, which is defined based on domain knowledge.

Hyperparameter | Value Range | |
---|---|---|

GP hyperparameters | Generations, Gen | 10, 15, 20 |

Population size, population | 800, 900, 1000, 1100 | |

Model hyperparameters | Window size | 10, 15, 20, 25, 30 |

Epochs | 50, 60, 70, 80, 90, 100 | |

Hidden units in LSTM | 16, 32, 48, 64 | |

Hidden units in fully connected layer | 8, 16, 32, 48 | |

Learning rate | 1 × 10^{−2}, 1 × 10^{−3}, 1 × 10^{−4} |

Number of New Features | MAPE | RMSE | MAE | |
---|---|---|---|---|

case 1 | Without new features | 12.5% | 0.57 | 0.39 |

case 2 | 3 new features + 3 new features | 10.6% | 0.53 | 0.33 |

case 3 | 5 new features + 5 new features | 11.4% | 0.56 | 0.35 |

Feature | Type | Expression |
---|---|---|

p_{1} | type 1 | $\frac{\rho}{\mu \xb7{D}_{e}}$ |

p_{2} | type 1 | $\sqrt{\frac{\mu}{\sqrt{{D}_{e}}}}$ |

p_{3} | type 1 | ${|GR-\mu |}_{t-2}$ |

p_{4} | type 2 | $\frac{W}{{D}_{e}}$ |

p_{5} | type 2 | $\mathrm{log}\left(Q\right)+\sqrt{SPP}$ |

p_{6} | type 2 | $\frac{\mathrm{log}\left(\rho \right)+D}{{D}_{e}}$ |

Incremental Update Steps/m | MAPE | RMSE | MAE |
---|---|---|---|

0 | 12.5% | 0.57 | 0.39 |

100 | 12.64% | 0.58 | 0.41 |

200 | 10.40% | 0.53 | 0.33 |

300 | 9.80% | 0.52 | 0.28 |

400 | 10.20% | 0.53 | 0.33 |

Model A | Model B | Model C | Model D | Benchmark | |
---|---|---|---|---|---|

One-layer LSTM | 🗸 | × | × | × | × |

Dual-layer LSTM | × | 🗸 | 🗸 | 🗸 | 🗸 |

Interpretable features | × | × | 🗸 | × | 🗸 |

Incremental update strategy | × | × | × | 🗸 | 🗸 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Ding, J.; Zhang, R.; Wen, X.; Li, X.; Song, X.; Ma, B.; Li, D.; Han, L.
Interpretable Feature Construction and Incremental Update Fine-Tuning Strategy for Prediction of Rate of Penetration. *Energies* **2023**, *16*, 5670.
https://doi.org/10.3390/en16155670

**AMA Style**

Ding J, Zhang R, Wen X, Li X, Song X, Ma B, Li D, Han L.
Interpretable Feature Construction and Incremental Update Fine-Tuning Strategy for Prediction of Rate of Penetration. *Energies*. 2023; 16(15):5670.
https://doi.org/10.3390/en16155670

**Chicago/Turabian Style**

Ding, Jianxin, Rui Zhang, Xin Wen, Xuesong Li, Xianzhi Song, Baodong Ma, Dayu Li, and Liang Han.
2023. "Interpretable Feature Construction and Incremental Update Fine-Tuning Strategy for Prediction of Rate of Penetration" *Energies* 16, no. 15: 5670.
https://doi.org/10.3390/en16155670