In Situ Structural Health Monitoring of Full-Scale Wind Turbine Blades in Operation Based on Stereo Digital Image Correlation

Abstract

Structural health monitoring (SHM) and the operational condition assessment of blades are greatly important for the operation of wind turbines that are at a high risk of disease in service for more than 5 years. Since certain types of blade faults only occur during wind turbine operation, it is more significant to perform in situ SHM of rotating full-scale blades than existing SHM of small-scale blades or static testing of full-scale blades. Considering that these blades are usually not prefabricated with relevant sensors, this study performed SHM and condition assessment of full-scale blades in operation with stereo digital image correlation. A self-calibration method adapted to the outdoors with a large field of view was introduced based on the speckled patterns. To accurately obtain the in- and off-plane deformation, a new reference frame is constructed at the center of the rotation of the blades. The 3D displacements of the points of interest (POIs) on the blade of a 2 MW wind turbine were characterized. Furthermore, the frequency spectrum of the measured 3D displacements of the blades was compared with the blades with the faults. The results showed that the introduced technique is a convenient and nondestructive technique that enables SHM of full-scale wind turbine blades in operation.

1. Introduction

During the past three decades, multiple sources of evidence show that the overuse of fossil fuels is a major factor in unabated global warming [1,2,3]. Wind energy has attracted extensive attention as one of the most promising clean renewable energy alternatives to fossil energy. According to the Global Wind Energy Council (GWEC), the total wind power capacity in the global market reached 837 GW in 2022 [4].

Meanwhile, wind turbines are designed with larger blade diameters to generate electricity more efficiently [5]. Moreover, the efficiency of power generation efficiency is strongly related with the operating conditions of the blades. Once a blade is faulted, its structural and aerodynamic characteristics change in a way that reduces the power generation efficiency [6,7,8]. Furthermore, as a critical component of wind turbines, the manufacturing cost of blades accounts for about 22% of the total cost of a wind turbine. Without effective techniques for early detection and risk warning of blade micro-defects, the accumulation of micro-defects can lead to significant blade faults [9]. If the faulty blade continues to operate, it can induce the failure of other components of the wind turbine or even cause a catastrophic accident in the unit. Not only is there a significant cost to maintain the wind turbine, but the power generated during maintenance is also lost. Therefore, structural health monitoring (SHM) and condition assessment of full-scale blades are significant for sustainability.

Currently, the SHM techniques for blades can be roughly categorized as contact and non-contact [10,11]. The current mainstream contact-based blade fault measurement methods include strain gauge measurement, vibration measurement, acoustic measurement, electromagnetic measurement, etc. [12,13,14,15]. Liu et al. proposed a strain prediction method for wind turbine blades based on genetic algorithm back propagation neural networks to investigate the strain prediction methods for full-scale blade static testing [16]. It is clear that the laboratory method of testing full-scale blades fixed on a test bench is not suitable for wind turbines in operation. In fact, certain types of faults can only be detected when the wind turbine is in operation, i.e., the germination stage of these types of faults can easily be missed by shutdown examinations [17,18]. In situ detection of the blade faults of wind turbines in operation is more effective than traditional off-line testing [19]. Khazaee et al. revealed the effect of faulty blades on tower vibration by installing accelerometers on the tower of a wind turbine to collect tower vibration signals. A convolutional neural network classifier was developed to categorize the tower vibration collected under blade health and fault conditions [20]. Sun et al. proposed a self-supervised health representation learning method for SHM of blades, where they accurately monitored blade icing by mounting accelerometers on the internal blade surface of a wind turbine [21]. Both of the above studies implemented blade fault detection by installing sensors. It is commendable that they have built convolutional neural network (CNN) models for fault recognition and categorization. However, the method is difficult to be applied to wind turbines that are at a high risk of disease in service for more than 5 years due to the absence of pre-assigned relevant sensors. Overall, contact measurement methods can hardly be used to measure the motion of large wind turbine blades due to the complexity of the layout and expensive cost.

Recent advances in photogrammetry provide new opportunities for non-contact SHM of full-scale blades [5,22]. Stereo digital image correlation (stereo DIC) typically uses two cameras to capture structural images from different perspectives and then uses pattern recognition techniques to compare these patterns frame by frame to measure 3D displacement, deformation, and motion [23,24]. As the main load-bearing component, blade faults can manifest in its 3D trajectory, and many faults only occur during blade rotation. Therefore, it is significant to prevent the deterioration of blade micro-defects by periodically performing in situ SHM of the blade to obtain the 3D motion and deformation of the blade. LeBlanc et al. performed static tests on a 9 m composite turbine blade using stereo DIC and showed that the technique can be readily expanded to accommodate large-scale blades [25]. Baqersad et al. identified the mode shapes of a wind turbine blade using the stereo DIC technique and compared its performance with the finite element and impact test approaches [26]. Both studies were static tests, as in [18]. Wu et al. tested the performance of the stereo DIC in blade fault examination with prefabricated faults in the blades of wind turbine models [27]. Bharadwaj et al. estimated the full-field dynamic strain of wind turbine blades using the DIC method and compared it with strain gauge measurements [28]. Both studies were conducted on small-scale wind turbines and did not really address the technical problems of full-scale wind turbine SHM. Lehnhoff et al. estimated the aeroelasticity of wind turbine blades based on the deformation measured by the DIC method and compared the results measured by the Spinner LiDAR method [29]. This study pasted speckle patterns only at the blade tips, while the rest of the blades were not considered. In summary, most of the existing DIC-based blade fault detection studies are for small-scale wind turbines or shutdown detection of full-scale wind turbines. Moreover, the apparent difference between laboratory and operating conditions also hinders the application of the traditional stereo DIC techniques for in situ SHM of full-scale wind turbine blades.

Due to the uncontrollable testing environment, the application of stereo DIC is still challenging for in situ SHM and condition assessment of full-scale wind turbine blade faults in operation. Therefore, this paper performs research on several major challenges, i.e., camera parameter calibration, reference frame construction, and test system construction, for in situ SHM of full-scale blades with stereo DIC. An in situ SHM method based on stereo DIC for operating full-scale blades is proposed. The effectiveness of the introduced method was verified by measuring the 3D motion of the blades of an operating 2 MW wind turbine. The operational condition of the blades was evaluated by transforming the 3D displacements of the blades into the frequency domain.

2. Methods

Figure 1 shows the flowchart of SHM and the condition assessment of full-scale wind turbine blades based on stereo DIC. It can be divided into 6 key processes, i.e., image capture, stereo camera calibration, point of interest localization, reference frame construction, 3D motion of points of interest (POIs) calculates, and condition assessment of the blade.

Figure 2 schematically shows the stereo DIC measurement system for in situ measuring 3D displacement of rotating wind turbine blades. The system consists of two measurement stations, which are connected by fiber optics. The measurement station consists of a camera, an attitude control unit, a battery, and a computer. A signal is sent to measurement station 2 via a program built into measurement station 1 and the fiber optics to ensure that both cameras capture images simultaneously. As we mentioned previously, since the system works in a non-cooperative environment, it has new hardware (pan-tilt and an inclinometer) to assist in camera attitude adjustment compared to the conventional stereo DIC system. In addition, the imaging system uses high-speed and high-resolution cameras. The configuration ensures that the captured blade images are sufficiently clear.

2.1. Stereo Digital Image Correlation

Stereo DIC technology typically uses two synchronized cameras to capture images and then measures the full-field 3D deformation of a plane or surface based on stereovision principles. Its working principle is to track the same spatial points before and after deformation and calculate their difference. The configuration of a typical stereo DIC and its perspective projection model are shown in Figure 3a. In this paper, subscript 1 and subscript 2 stand for the left and right camera. Set the camera 1 frame and the corresponding image frame as $O_1-X_1{Y}_1{Z}_1$ and $o_1-u_1{v}_1$, respectively. Similarly, $O_2-X_2{Y}_2{Z}_2$ and $o_2-u_2{v}_2$ represent the camera 2 frame and its image frame. The world frame is set to align with the frame of camera 1. The relative pose of the two cameras can be represented as rotation matrix R and translation vector t, respectively. Assume that any point of interest (POI) P has coordinate $\left(x_w,y_w,z_w\right)$ under the world frame, and the coordinates of it projected on the two camera imaging targets are $p_1\left(u_1,v_1\right)$ and $p_2\left(u_2,v_2\right)$. The projection geometry of the system is given by [30]:

$$s_1\left[\begin{array}{c}{u}_1\\ v_1\\ 1\end{array}\right]={\mathit{A}}_1\left[\begin{array}{cc}I& 0\end{array}\right]\left[\begin{array}{c}{x}_w\\ y_w\\ \begin{array}{c}{z}_w\\ 1\end{array}\end{array}\right], s_2\left[\begin{array}{c}{u}_2\\ v_2\\ 1\end{array}\right]={\mathit{A}}_2\left[\begin{array}{cc}\mathit{R}& \mathit{t}\end{array}\right]\left[\begin{array}{c}{x}_w\\ y_w\\ \begin{array}{c}{z}_w\\ 1\end{array}\end{array}\right]$$

(1)

where $s_1$ and $s_2$ are two scale factors, ${\mathit{A}}_1$ and ${\mathit{A}}_2$ are the intrinsic parameters of the two cameras, I is the identity matrix, and 0 is the zero matrix.

The gray gradient of the subset determines the performance of the correlation matching algorithm. In the case of a mismatch, a non-negligible error is introduced in the 3D reconstruction of the POIs. Therefore, for efficient matching performance, the sample surface is usually sprayed with artificial speckles [31]. As shown in Figure 3b, both cameras capture a set of images $im{g}_{1\_0}$ and $im{g}_{2\_0}$ simultaneously as reference images at the beginning of loading. During the loading process, the cameras capture the deformation images $im{g}_{1\_1}~im{g}_{1\_n}$ and $im{g}_{2\_1}~im{g}_{2\_n}$ of the specimen at the same frame rate. First, the image coordinate $p_{1\_0}$ of the POI P in the image $im{g}_{1\_0}$ is given, and its coordinates $p_{2\_0}$ in image $im{g}_{2\_0}$ are estimated using the correlation matching algorithm. Then, the coordinates $p_{1\_1}~p_{1\_n}$ and $p_{2\_1}~p_{2\_n}$ of the POIs in the deformed image are determined sequentially using the correlation matching algorithm.

The schematic diagram of the correlation matching is illustrated in Figure 4. Given any point P located at $\left(x_r, y_r\right)$ on the reference image, its correspondences $\left(x_d, y_d\right)$ on the deformed image can be located by first specifying a square reference subset centered at $\left(x_r, y_r\right)$ and then searching the deformed subset on the deformed images by optimizing the robust zero-mean normalized sum of squared difference (ZNSSD) criteria defined as [32]:

$$C_{\mathrm{ZNSSD}}\left(p\right)={{\displaystyle \sum }}_{i=-M}^M{{\displaystyle \sum }}_{j=-M}^M{\left[\frac{f\left(x_r^i, y_r^j\right)-\overline{f}}{\sqrt{{{\displaystyle \sum }}_{i=-M}^M{{\displaystyle \sum }}_{j=-M}^M{\left[f\left(x_r^i, y_r^j\right)-\overline{f}\right]}^2}}-\frac{g\left(x_d^i, y_d^j\right)-\overline{g}}{\sqrt{{{\displaystyle \sum }}_{i=-M}^M{{\displaystyle \sum }}_{j=-M}^M{\left[g\left(x_d^i, y_d^j\right)-\overline{g}\right]}^2}}\right]}^2$$

(2)

where $f\left(x_r^i, y_r^j\right) \mathrm{and} g\left(x_d^i, y_d^j\right)$ are the grayscales of point $\left(x_r^i, y_r^j\right)$ in the reference subset and point $\left(x_d^i, y_d^j\right)$ in the deformed subset, while $\overline{f}$ and $\overline{g}$ represent the mean grayscale intensity of the reference subset and deformed subset. The magnitude of the correlation coefficient $C_{\mathrm{ZNSSD}}$ varies from 0 to 1, with 1 signifying a perfect match between the two subsets. $P={\left(u,u_x,u_y,u_{xx},u_{yy},u_{xy},v,v_x,v_y,v_{xx},v_{yy},v_{xy}\right)}^{\mathrm{T}}$ in Equation (2) is the parameter vector involved in the 2nd-order shape function [33], which describes the deformation from reference subset to deformed subset:

$$\left\{\begin{array}{c}{x}_d^i=x_r+\Delta x+u+u_x\Delta x+1/2u_{xx}\Delta x^2+u_y\Delta y+1/2u_{yy}\Delta y^2+u_{xy}\Delta x\Delta y\\ y_d^j=y_r+\Delta y+v+v_x\Delta x+1/2v_{xx}\Delta x^2+v_y\Delta y+1/2v_{yy}\Delta y^2+v_{xy}\Delta x\Delta y\end{array}\right.$$

(3)

where $\left(u, v\right)$ denotes rigid-body translation from $\left(x_r, y_r\right)$ to $\left(x_d, y_d\right)$, $\Delta x=x_r^i-x_r$ and $\Delta y=y_r^i-y_r$ are the local coordinates in the subset, and $\left(u_x, u_y, v_x, v_y\right)$ and $\left(u_{xx}, u_{yy}, u_{xy}, v_{xx}, v_{yy}, v_{xy}\right)$ are the 1st-order and 2nd-order displacement gradient components [34].

2.2. Stereo Camera Extrinsic Parameter Calibration

Since the intrinsic parameters ${\mathit{A}}_1$ and ${\mathit{A}}_2$ depend on the camera’s sensor and lens, they can be pre-calibrated in the laboratory based on Zhang’s method [35]. Extrinsic parameters R and t calibration based on the scene control points can be divided into epipolar-based and homography-based methods. Both methods require sufficient control points uniformly distributed in the field of view. The former requires that the control points have depth variation, while the latter requires that the control points be coplanar. In this study, speckle patterns attached to blades can be used as control points for the calibration of extrinsic parameters. It is clear that the homography-based method is more suitable for the application scenario of this study. Theoretically, the control points are not strictly coplanar due to wind load and blade self-weight. In practice, the control points can be considered as approximately coplanar due to the long distance of the cameras from the wind turbine and the slight depth variation of the control points.

The homography of the projection points of the spatial points in the two views can be described in Figure 5. Assuming that P is any point on the spatial plane π, the normalized coordinates of the image points of its projection in the two perspective views are $p_1\left(x_1, y_1, 1\right)$ and $p_2\left(x_2, y_2, 1\right)$, respectively. The relationship between the two normalized coordinates can be described as:

$$\mathrm{c}\left[\begin{array}{c}{x}_2\\ y_2\\ 1\end{array}\right]=\mathit{H}\left[\begin{array}{c}{x}_1\\ y_1\\ 1\end{array}\right]=\left[\begin{array}{ccc}{h}_1& h_2& h_3\\ h_4& h_5& h_6\\ h_7& h_8& h_9\end{array}\right]\left[\begin{array}{c}{x}_1\\ y_1\\ 1\end{array}\right]$$

(4)

where c is any non-zero constant and H is the homography matrix. Equation (4) can also be written as:

$$\left\{\begin{array}{c}-h_1{x}_1-h_2{y}_1-h_3+\left(h_7{x}_1+h_8{y}_1+h_9\right)x_2=0\\ -h_4{x}_1-h_5{y}_1-h_6+\left(h_7{x}_1+h_8{y}_1+h_9\right)y_2=0\end{array}\right.$$

(5)

Equation (5) can be written in matrix form as:

$${\mathit{A}}_i\mathit{h}=0$$

(6)

where ${\mathit{A}}_i=\left[\begin{array}{ccccccccc}-x_1& -y_1& -1& 0& 0& 0& x_2{x}_1& x_2{y}_1& x_2\\ 0& 0& 0& -x_1& -y_1& -1& y_2{x}_1& y_2{y}_1& y_2\end{array}\right]$ and

$$\mathit{h}={\left[\begin{array}{ccccccccc}{h}_1& h_2& h_3& h_4& h_5& h_6& h_7& h_8& h_9\end{array}\right]}^T$$

In theory, H with 8 degrees of freedom can be calculated by 4 sets of corresponding points [36]. In practice, the corresponding points do not strictly satisfy the relation $p_2=\mathit{H}{p}_1$ due to image noise. Therefore, hundreds of correspondences with sub-pixel accuracy were used to calculate H in this study. Perform a singular value decomposition of the homography matrix H:

$$\mathit{H}=\mathit{U}\mathit{\Lambda }{\mathit{V}}^{\mathit{T}}$$

(7)

where $\mathit{U}$ and V are orthogonal matrices and Λ is the diagonal matrix containing the singular values of the matrix H. The diagonal matrix Λ can be decomposed as:

$$\mathit{\Lambda }={\mathit{R}}_{\mathit{\Lambda }}+{\mathit{t}}_{\mathit{\Lambda }}{\mathit{n}}_{\mathit{\Lambda }}^{\mathit{T}}$$

(8)

According to Equation (8), we can obtain up to 8 different solutions for the triplets: $\left\{{\mathit{R}}_{\mathit{\Lambda }}, {\mathit{t}}_{\mathit{\Lambda }}, {\mathit{n}}_{\mathit{\Lambda }}\right\}$. The rotation and translation matrix can be represented as follows:

$$\mathit{R}=\mathit{U}{\mathit{R}}_{\mathit{\Lambda }}{\mathit{V}}^{\mathit{T}},\mathit{t}=\mathit{U}{\mathit{t}}_{\mathit{\Lambda }}, \mathit{n}=\mathit{V}{\mathit{n}}_{\mathit{\Lambda }}$$

(9)

It is clear that not all of these 8 solutions are physically possible. Several constraints have been introduced to determine the unique solution [37].

For the measurement system in Figure 2, since the two cameras are mounted on separate mounts, possible ground vibrations or wind loads during measurements can invalidate the calibrated extrinsic parameters. Therefore, several control points were extracted in each deformed image pair, and the extrinsic parameters were updated with the fast extrinsic parameter re-correction algorithm in [38]. Once the refinement of R and t is determined, the 3D coordinates of the spatial point can be estimated via the triangulation principle.

2.3. Construction of the Reference Frame

The 3D deformation of the blades measured by stereo DIC depends on the chosen reference frame [39]. For 3D photogrammetry, the reference frame (world frame) is aligned with the left camera frame by default. This configuration results in the in- and off-plane deformation of the specimen not being measured accurately. Therefore, it is necessary to establish a new reference frame on the specimen surface before loading. Since there is no coplanar orthogonal line on the blade surface, fitting a reasonable reference plane is the key to establishing the reference frame.

As shown in Figure 6, image points ($\ge 3, \mathrm{white}\mathrm{points}$) are selected on the specimen surface, and their 3D coordinates in the left camera frame are calculated based on the calibrated stereo camera parameters. According to the 3D coordinates $\left(X_{1~n},Y_{1~n},Z_{1~n},\right)$, a reference plane can be fitted to construct a reference frame $O_r-X_r{Y}_r{Z}_r$. The displacement in the $Z_r$ direction is the out-plane displacement and the $X_r$ and $Y_r$ directions are the in-plane displacement components. Equation (10) represents the function of the reference plane where the reference frame is located [40].

$$aX+bY+cZ+d=0$$

(10)

The above equation can be written in the following form:

$$\left[\begin{array}{cccc}{X}_1& Y_1& Z_1& 1\\ X_2& Y_2& Z_2& 1\\ & & & ⋮\\ X_n& Y_n& Z_n& 1\end{array}\right]\left[\begin{array}{c}a\\ b\\ c\\ d\end{array}\right]=0$$

(11)

Perform singular value decomposition of the coefficient matrix:

$$\left[\mathit{S}, \mathit{V}, \mathit{D}\right]=svd\left[\begin{array}{cccc}{X}_1& Y_1& Z_1& 1\\ X_2& Y_2& Z_2& 1\\ & & & ⋮\\ X_n& Y_n& Z_n& 1\end{array}\right]$$

(12)

The coefficient in Equation (10) can be determined as $a=\mathit{D}\left(1,4\right),b=\mathit{D}\left(2,4\right),c=\mathit{D}\left(3,4\right)$ and $d=\mathit{D}\left(4,4\right)$. The intersection of the $Z_r$ axis with the fitted plane is used as the origin $O_r$ of the reference frame. The position of $O_r$ is ${\left[0, 0,-d/c\right]}^T$ in the $O_w-X_w{Y}_w{Z}_w$. Usually, this $X_r$ axis is determined based on the object’s design frame. Assuming that the unit vector of the $X_r$ axis is denoted by $u_x$, then the unit vector $u_z$ of the $Z_r$ axis is given by Equation (13). Subsequently, the unit vector $u_y$ of the $Y_{ref}$ axis can be derived using Equation (14).

$$u_z=\frac{{\left[a, b, c\right]}^T}{\sqrt{a^2+b^2+c^2}} \left(c<0\right)$$

(13)

$$u_y=u_z\times u_x$$

(14)

Once the origin $O_r$ and the unit vectors $u_x$, $u_y$, and $u_z$ are obtained, the pose relationship $\left({\mathit{R}}_{w-r}, {\mathit{t}}_{w-r}\right)$ between $O_r-X_r{Y}_r{Z}_r$ and $O_w-X_w{Y}_w{Z}_w$ is established. Thus, the coordinate $\left(X_n^{\prime },Y_n^{\prime },Z_n^{\prime }\right)$ of each interesting point in the newly constructed reference frame can be calculated using the following equation:

$${\left[X_n^{\prime },Y_n^{\prime },Z_n^{\prime }\right]}^T={\mathit{R}}_{w-r}{\left[X_n,Y_n,Z_n\right]}^T+{\mathit{t}}_{w-r}$$

(15)

3. Experiments and Results

3.1. Experimental Setup

In this experiment, the blades of an operating 2 MW wind turbine with a length of 47 m were tested. Since the coating of the blade’s surface is pure white, it causes the images of the blades captured by the stereo cameras to lack the gray gradient necessary for correlation matching. Therefore, we pasted random speckle patterns with a size of 40 × 40 cm on the pressure side of the blades before the test, as shown in Figure 7a.

According to the schematic in Figure 2, a measurement station of the test system constructed in this study is shown in Figure 7b. The imaging system consists of a CMOS camera (Mikrotron 25CXP, Unterschleißheim, Germany) with a spatial resolution of 5120 × 5120 pixels and a lens (Nikkor Lens, Tokyo, Japan) with a focal length of 75 mm. The pixel size is 3.45 µm pixel⁻¹. The baseline length of the stereo DIC was 63.9 m. The position of the camera can be adjusted through the attitude control unit, avoiding the inconvenience of manually adjusting the camera posture. Due to the high resolution and frame rate of the images, a high-speed image card (Active Silicon FireBird, Langley, UK) with a transfer rate of 6.25 Gbps was built into the computer (Intel Core (TM) i7 CPU @2.4 GHz, Santa Clara, CA, USA). All the algorithms are developed in C++ as the programming language. In addition, we used the open-source library OpenCV. The measuring system is arranged in front of the wind turbine according to the direction of the wind, as shown in Figure 8. The distance between the wind turbine and the camera was approximately 250 m.

The pan-tilt and controller shown in Figure 7b were utilized to adjust the positions of the two cameras to ensure that the entire blades were in the field of view of both cameras. When one of the blades is approximately perpendicular to the ground, the stereo camera starts capturing images of the wind turbine in operation. The stereo camera captured images of 560 pairs of blades over 5 rotation cycles.

3.2. Parameters of the Stereo Camera

Determining the parameters of the stereo camera is one of the most important steps in the tests of the SHM of wind turbine blades. The process of calculation is shown in Figure 9. Before the in-site measurement, the intrinsic parameters of the two cameras were pre-calibrated using Zhang’s method [35]. To estimate the extrinsic parameters of the stereo camera, six pairs of images were selected as calibration images from the capture images mentioned above. The control points are extracted and matched using the SIFT algorithm from the speckle patterns of the blade’s surface in the calibrated images. About 4000 pairs of control points are retained and used to calibrate the extrinsic parameters of the stereo camera according to the method introduced in Section 2.2, as shown in Figure 10. The intrinsic and extrinsic parameters are listed in

Table 1. The parameters of the stereo DIC and re-projection error.

Intrinsic parameters		$(u_0,v_0$)/Pixels	$\left(F_x,F_y\right)$/Pixels	$\left(K_1,K_2 \right)$
	Camera 1	(2194.99, 2374.92)	(16,716.56, 16,554.44)	(0.06, 4.58)
	Camera 2	(2175.87, 2474.15)	(16,195.47, 16,036.88)	(−0.01, −2.41)
Extrinsic parameters	Rotation vector (°)		Translation vector (mm)	Error (pixels)
	(−0.27, 13.41, −2.91)		(63,047.65, 2919.94, 9983.86)	0.19

.

The results showed that the re-projection error was 0.19 pixels. To further examine the performance of the proposed calibration method, the dimensions of the speckle patterns that were affixed to the blades were reconstructed using the stereo DIC parameters in Table 1. The statistical results showed that the mean absolute deviation of the reconstructed dimensions of the 30 randomly selected speckle patterns was 4.48 mm.

3.3. Reference Frame

To realistically illustrate the 3D trajectory of the blades, a reference frame was constructed at the center of rotation of the wind turbine in this study. Figure 11 shows the cabin portion of the first wind turbine image captured by the left camera. Several image points were arbitrarily selected on the contour of the front circular cover of the wind turbine cabin. The image coordinates of these image points in the corresponding right view can be determined by the DIC method. Since the extrinsic parameters of the stereo camera have been determined, the 3D coordinates of the image points in the left camera frame can be estimated. The 3D coordinates can be used for fitting a reference plane. The origin of the reference frame is located at the center of the reference plane. And then, the reference frame $O_r-X_r{Y}_r{Z}_r$ can be constructed according to the method described in Section 2.3. The pose relationship between the newly constructed reference frame and the left camera frame can be determined, as shown in

Table 2. Relative pose between the left camera frame and reference frame.

Rotation Angle (°)	Translation Vector (mm)
(12.075, 6.902, −3.105)	(−28,942.12, 56,292.19, −252,876.43)

.

3.4. Three-Dimensional Displacements of POIs

The first pair of images acquired by the stereo camera is used as the reference image, and the remaining are deformed images. To evaluate the health condition of the blade based on its 3D trajectory, eight POIs (P₁, P₂, …, P₈) were selected at roughly equal intervals in the root-to-tip of the blade in the left reference image, as shown in Figure 12. The image coordinates of these POIs in the right reference image can be determined by the DIC method. Similarly, the image coordinates in the deformed images can also be determined.

Figure 13 describes the amplitudes of pixel displacement and the corresponding rotation angles of the POIs in the right view over time for five cycles. The pixel displacements in the x- and y-directions show stable periodic variations with uniform amplitudes, and all eight POIs have the same rotation angle at each moment. They show the effectiveness of the DIC technique used in this study for the tracking of operating wind turbine blades.

The 3D coordinates of the POIs can be reconstructed according to the determined image coordinates of the POIs in the deformed image and the stereo DIC parameters, where the 3D coordinates are represented in the camera 1 frame. Then, these 3D coordinates were transformed to the newly constructed reference frame by the transformation matrix in Table 2. Figure 14a–h show the 3D displacement trajectories of the POIs in the $O_r-X_r{Y}_r{Z}_r$. It should be noted that according to the reference frame shown in Figure 11, the x-direction displacements represent horizontal motion, the y-direction displacements represent vertical motion, and the z-direction displacements represent off-surface motion. It can be observed that as the POIs are far away from the rotational center of the blade, the magnitude of the motion in each direction increases gradually. This is consistent with the motion law of the blade. Taking POI 5 as an example, its peak displacement in the y-direction is about 40 m, which is basically consistent with the distance between the point and the center of rotation of the blade. This sufficiently demonstrated the effectiveness of the reference frame construction method.

In addition, the motion amplitude of the POIs over five rotation cycles is approximate. This indicates that the wind speed and direction were relatively stable during the test period, and the wind turbine blades were in orderly operating condition. It should be noted that the off-surface displacement is caused by inertial and aerodynamic loads during blade rotation. In this case, the z-direction motion trajectories of the POIs close to the blade root approximate a sinusoidal curve, while the motion trajectories of the POIs close to the blade tip are more disorganized. This is due to the fact that the deflection of the blade tip is much larger than the blade root in the operating condition, and the forces are more complicated.

Transforming displacement signals to the frequency domain using fast Fourier transform (FFT) is one of the most common methods in blade fault diagnosis [41]. Therefore, the raw data were transferred to time frequency in this study to be considered as the fault indicators. As an example, the 3D motion of POI 5 was analyzed with Fourier transform, as shown in Figure 15a–c. In contrast, small-scale wind turbine blades with prefabricated faults were tested in the laboratory in another of our studies, as shown in Figure 15d [42]. Figure 15e presents the results after transforming the 3D displacements of the cracked blade measured in the operating condition to time frequency using FFT. It can be found that there is only one fundamental frequency signal in the frequency domain corresponding to the 3D displacements measured in this study, while there are multiple fundamental frequency signals for the 3D displacements of the blade with faults. This further verifies that the wind turbine blades shown in Figure 12 were in healthy operating conditions.

4. Discussion

The factors affecting the measurement accuracy mainly include image resolution, FOV size, FOV angle, system parameters, etc., in which the first three factors are closely related to the configuration of the stereo DIC system. To accommodate the size of the blades, a high-speed CMOS camera with high resolution was used to ensure the clarity of the captured images. For an ideal FOV angle, the two cameras were arranged at a distance of 63.9 m apart, which facilitated the 3D reconstruction of the POIs. In addition, for convenience in field environments, each camera was fitted with an inclinometer and a pan-tilt to assist in adjusting the camera’s attitude.

The accuracy and efficiency of calibration of the extrinsic parameters of the stereo camera are crucial for in situ SHM of a full-scale wind turbine in operation. The former is closely related to the reliability of the measured blade’s motion, and the latter determines the convenience of the testing. To improve the robustness of the self-calibration method for large FOV applications, this paper enhances the number and quality of scene control points by pasting spackle patterns on the blades, and the homography constraint is employed as a calibration principle to accommodate the spatial characteristics of the blades. Experimental results showed that the introduced method can efficiently and accurately estimate the extrinsic parameters of the stereo DIC for large FOV applications. The method of pasting the spackle patterns in the shutdown condition of the wind turbine has the disadvantage of high cost. As a suggestion, some markers can be created during the manufacturing process for large wind turbines.

For most of the studies, the reference frame is aligned with the left camera frame. This leads to the fact that the existing studies only obtain the trend of the blade motion, and the measured values are not reliable for the real deformation of the blade. In this study, the in- and off-plane displacements of the blades were accurately measured by constructing a new reference frame at the rotation center of the blades, which is important for the assessment of the operational condition of the blades. In fact, it is more appropriate to construct the reference frame at the blade’s root. Unfortunately, there is no available reference plane for the blade’s root in this test. In addition, since it is difficult to find a full-scale wind turbine with a disordered operating condition, this study used the motion frequency spectrum of a prefabricated fault small-scale wind turbine blade as a reference to verify the validity of the proposed techniques.

By analyzing the 3D displacement and frequency spectrum shown in Figure 14, it can be clearly identified that the measured blade is in good operating condition. According to Figure 15, it is known that the introduced technique can diagnose the presence of damage in the blade. The results of the analysis showed that the technique can serve as a reliable tool for SHM and the operational condition assessment of blades of full-scale wind turbines. Since the technique can detect potential blade damage in time, more critical blade or turbine accidents can be avoided. This is significant for the efficient utilization of wind energy and sustainability. It is also potentially helpful for the SHM of other outdoor engineering structures.

5. Conclusions

This study performed in situ SHM of full-scale wind turbine blades in operation and provided solutions to several challenges with existing technologies. A test system was constructed to accommodate the full-scale wind turbines. Given the difficulty in determining the pose relationship between cameras when applying the stereo DIC with a large FOV outdoors, control points for determining the camera poses are provided by pasting speckle patterns on the pressure side of the blades. A self-calibration method based on scene control points is introduced, and its effectiveness is verified by reconstructing the size of the pattern with a known geometric. To accurately obtain the in- and off-plane deformation of the blades, a new reference frame is constructed at the center of rotation of the blades.

An experiment was conducted to evaluate the operational condition of the blades of a 2 MW wind turbine. Eight POIs were selected at roughly equal intervals from the root to the tip of the blade. The experimental results showed that the displacements of the POIs in each direction over five rotational cycles approximated sinusoidal curves with the same amplitude. To further evaluate the operational condition of the blades, the frequency spectrum of the measured 3D displacements of the blades was compared with the blades with the faults. The results show that the proposed method is a convenient and nondestructive technique that enables online assessment of the operational condition of full-scale wind turbine blades.

The frequency spectrum of 3D motion can only distinguish whether the blade operation is orderly or not; it is not able to further identify the cause of disordered operation. In fact, different types of blade damage correspond to different 3D motion trajectories, which were not investigated more thoroughly in this study. In addition, the measurement errors caused by light refraction, illumination variations, and communication delays need to be considered. In the next step, we plan to incorporate the blade’s 3D motion data into a deep learning model to accurately identify and locate blade faults.