Critical Experiments for Structural Members of Micro Image Strain Sensing Sensor Based on Smartphone and Microscope

Abstract

Strain is one of the key concerns of structural health monitoring (SHM), and effective measurement of structural strain is very important for structural health condition assessment. The purpose of this paper is to investigate the application of the micro image strain sensing (MISS) sensor based on smartphone and microscope for measuring strain parameters of structural members. Firstly, metal tensile tests were performed to verify the effectiveness of the MISS sensor, and the strain results measured by the fiber Bragg grating (FBG) sensor and the MISS sensor were compared. The results showed that the mean error between the MISS and FBG sensor measured results is 7.1 με in the metal tensile test. Then, a concrete beam bending test was carried out; the mean relative error of strain values between the FBG sensor and the MISS sensor is 2.5%. Overall, the strain measurements of the MISS sensor matched well with those from the FBG sensor in each experiment. This study validates the reliability and accuracy of the MISS sensor for strain detection and the MISS sensors can play a more powerful role in SHM due to their robustness, simplicity, and low cost.

1. Introduction

The structural strain plays a significant role in the structural health monitoring (SHM) system, as it could intuitively reflect the local characteristics of the structure and easily perform the structural safety assessment [1,2,3,4]. Currently, the strain sensors for SHM commonly include electrical resistance strain gages (ESGs) [5,6,7], fiber optic strain sensors (FOSs) [8,9,10,11], vibrating wire strain gages (VWSGs) [12,13], and digital imaging correlation (DIC) [14,15,16,17]. Among these sensors, ESGs are usually required to form a measurement circuit with wires, which would cause complicated situations when there are many measurement points and is subject to electromagnetic signal interference. FOSs are costly to install in large structures and the fibers are prone to breakage. Simultaneously, the potential damage to structural integrity may occur when FOSs are embedded in the concrete or other materials. The great advantage of VWSGs is that they are stable and simple, but their slow response speed is unsuitable for dynamic monitoring, and the steel-string is prone to creeping and relaxation. DIC technology is a non-contact, non-destructive, full-field measurement method, which has received significant attention in recent years. However, the accuracy of DIC technology is influenced by image noise, image processing algorithms, and measurement equipment. Additionally, DIC technology can only measure the area within the camera’s field of view. It is worth noting that a larger field of view reduces the resolution of the camera, which affects the resolution and accuracy of the measurements. In other words, DIC technology cannot achieve a larger field of view and higher resolution at the same time [18]. The high price of DIC equipment also limited its widespread use in civil engineering structures. Furthermore, the above-mentioned traditional sensors often require special equipment and professional technicians to operate, which restricts the range of application in practical engineering. Therefore, it is necessary to develop an economical and practical strain sensor that can still provide reliable results.

Various novel strain sensors for SHM have been extensively studied [19]. For example, Tondolo et al. [20] designed a smart steel system to measure the strain of reinforced concrete members by implanting a commercial barometric pressure MEMS sensor into an appropriate sealed cavity in the reinforcement. Gregori et al. [21] invented a wireless strain gauge that combines an RFID tag with a common thin-film resistive strain gauge, overcoming the installation and maintenance problems of wireless sensor networks. Morgese et al. [22] developed a two-stage monitoring method combining distributed optical fiber sensors and digital image correlation techniques.

Notably, the development of smartphones offers new opportunities for low-cost sensors. One of the major advantages of using smartphones for SHM is that they can create a cyber-physical system using embedded operating systems and wireless communication hardware. Moreover, smartphones can be used as a crowdsourcing SHM platform that is fueled by citizens [23,24]. For example, some researchers employ smartphones to measure acceleration [25,26,27], displacement [28,29,30,31], cracking [32,33,34], construction safety [35,36,37], and other parameters of buildings. With smartphones equipped with improved cameras, some researchers (e.g., Yu et al. [38]) have been focused on using smartphones combined with image processing algorithms to implement visual monitoring of structural strain. However, the measurement area of the specimens studied by Yu et al. is only 20 mm × 25 mm, and the measurement error obtained is 56 με. Therefore, the method of measuring structural strain based on smartphones is not sufficiently mature and still needs further research.

To measure structural strain more conveniently and in a low-cost manner, the principle of micro image strain sensing (MISS) sensor was proposed by our team [39]. In the previous study, the deformation process of the structure was simulated by the movement of the stepper motor, and then it was found that the MISS sensor showed good accuracy and stability. However, there are still some problems to be solved if the MISS sensor is applied to the real structures. The first issue is the connection between the smartphone and the sensor. The smartphone camera needs to maintain a constant field of view during the image capture process, which requires a reliable connection between the smartphone and the sensor. The suitability of different smartphone models also needs to be taken into account. Another task is to carry out tests on the actual deformed elements. In previous studies, the deformation of the structure was simulated by the movement of a stepper motor, but the two processes are not identical. Further tests are required to verify whether the actual structural component strain results can be obtained accurately and effectively. The main purpose of this study is to solve the above two problems.

This paper researches the application of the MISS sensor based on smartphone and microscope in actual deformed structural members. We begin by introducing a fixed device that connects a smartphone with a MISS sensor and its operation. Then, the metal tensile tests under different loading conditions were carried out to verify the sensor performance. In addition, the MISS sensor was used for a bending experiment on reinforced concrete (RC) beam. The results show that the MISS sensor can effectively measure the development of the surface strain of deformed structures. In the end, the next stages of the research are discussed.

2. Measurement Method and Principle

2.1. Method for Fabricating the Sensor

A novel strain sensor for civil structural health monitoring has been proposed recently in the laboratory of the authors [39]. As shown in Figure 1a,b, the main body of the sensor consists of two carbon fiber tubes of different diameters, and the two tubes are referred to as inner and outer tubes respectively. During the operation of the sensor, the outer tube is fixed to the support 1, the inner tube is fixed to the support 2, and the inner and outer tubes move freely along the axial direction of the tubes in a piston connection. The outer tube of the sensor is mounted with a microscope that is paired with a smartphone camera to take photos. Underneath the microscope is a semi-cylindrical support platform, which is fixed to the outer tube of the sensor.

Figure 1c illustrates the field of view obtained by the smartphone camera. The image is split into two parts: the upper part is the region of interest (ROI) A, which is located on the support platform connected to the outer tube; the lower part is the ROI B, which is located on the carbon fiber rod connected to the inner tube. Some irregular shapes are preset on the support platform of the MISS sensor as feature points. The ROI A of the image is actually a collection of feature points containing the movement information of support 1, and similarly, the ROI B is actually a collection of feature points containing the movement information of support 2. Moreover, the black circle in the ROI A is also used for the pixel equivalent. When the structure is deformed, two supports are anchored to the surface of the structure, and the supports drive the inner and outer tubes to move relative to each other. At the same time, the support platform moves rigidly with the outer tube and the carbon fiber rod moves rigidly with the inner tube. As a result, the relative positions of the ROI A on the support platform and the ROI B on the carbon fiber rod change. Additionally, the mean strain on the surface of the measured structure is obtained by dividing the relative displacement of the ROI A and ROI B by the distance between the two supports.

In this paper, the speeded-up robust feature (SURF) algorithm [40] is used to identify the movement of the feature points within the ROI, and the M-estimator Sample Consensus (MSAC) algorithm [41] is used to eliminate the outliers of the feature points. The SURF algorithm is mainly divided into three steps: feature point detection, feature point description, and feature point matching, as shown in Figure 2. The feature point detector uses a fast Hessian matrix to detect feature points, such as corner points and T-intersection points. To describe the directional characteristics of the feature points, the integral image is used to generate the feature description operator, which performs Haar wavelet response operation on the image. Finally, the similarity measure of two feature point operators is described by Euclidean distance. However, in the feature point matching stage, a small deviation angle in the main direction can also cause errors in feature matching, so the MSAC algorithm is used to eliminate matching outliers. Figure 2 presents the calculation process of feature points.

2.2. Fixing Device

The MISS sensor requires a stable and constant view during the measurement process, so the camera of the smartphone needs to be relatively static with the microscope which provides the magnification function. In other words, it means that the smartphone needs to be consistent with the movement of the outer tube and the support 1. Since it is considered that fixing the phone to the outer tube might cause the outer tube to be deflected due to the smartphone’s self-weight, we finally decide to fix the smartphone to the support 1 of the sensor.

As we use various smartphones as measurement devices, a small phone holder was designed in this experiment. In the mobile phone market, if the model of phone A is different from phone B, then the locations of the cameras are usually different as well. Consequently, minor adjustments to the fixation of the smartphone to the sensor are required when the measuring device is a different smartphone. The holder can adjust the length and angle of the x-axis, y-axis, and z-axis by rotating the connecting part, so that the phone camera has the best view. So, this phone fixing device can easily adjust the placement of the phone; the specific design is shown in Figure 3.

2.3. Sensing Principle

When the structural surface is deformed, the position of the support 1 and support 2 of the sensor is changed, which drives the position of ROI A and ROI B to change as well. Define the coordinates of the initial reference image before deformation as a (x_a, y_a) for ROI A and b (x_b, y_b) for ROI B. Similarly, the coordinates of the feature points of the deformed image are a′ (x_a′, y_a′) and b′ (x_b′, y_b′) respectively. Specifically, x_a, y_a are the abscissa pixel values and ordinate pixel values of the feature point a in ROI A before deformation, and x_b, y_b represent ROI B. The coordinate origin is the upper left corner of the image, and the coordinates are in pixels. As mentioned above, the smartphone is fixed to the support 1 (see Figure 3), so that the camera remains relatively stationary with the ROI A fixed to the outer tube. This also means that the coordinates of the feature points located in the ROI A remain unchanged during the deformation process, but the values of a (x_a, y_a) and a′ (x_a′, y_a′) may be unequal due to focus drift. The structural strain is not accurate if only the feature points in ROI B are selected for calculation; it becomes necessary to modify the result.

The distance between markers before and after the deformation is expressed as Euclidean distance. Figure 4 presents the coordinate map of relative displacement.

The modified formula is:

$$\Delta l=\sqrt{{\left[\left(x_{b^{\prime }}-x_b\right)-\left(x_{a^{\prime }}-x_a\right)\right]}^2+{\left[\left(y_{b^{\prime }}-y_b\right)-\left(y_{a^{\prime }}-y_a\right)\right]}^2}$$

(1)

where Δl is the coordinate change value of the same feature point before and after deformation. As Δl is calculated using the pixel coordinate system, the unit of Δl is pixels. x_a, y_a are the abscissa values and ordinate values of the feature point in ROI A before deformation, x_b, y_b represent the ROI B. Similarly, x_a′, y_a′ are the abscissa values and ordinate values of the feature point in ROI A after deformation, and x_b′, y_b′ in ROI B are represented in the same format.

The single complete image acquired by the MISS sensor is exhibited in Figure 5a. Firstly, the image is grayed and opened to eliminate the influence of the interference point in the marked circle area. Then the grayscale image is binarized to obtain an image containing only the marked circle, as shown in Figure 5b.

The number of pixels contained in the circle area is calculated as N, and thus the number of pixels in the radius of the calibrated circle can be found as n:

$$n=\sqrt{\frac{N}{\pi }}$$

(2)

The actual size of the calibration circle radius is 0.5 mm, so the pixel equivalent k (mm/pixel) in the image can be obtained by the following equation:

$$k=2n$$

(3)

L (mm) is the distance between support 1 and support 2, then the mean strain:

$$\epsilon =\frac{\Delta l\times k}{L}$$

(4)

The effect of temperature on the MISS sensors has been studied in [39] and an empirical formula for the effect of temperature on the MISS sensor has been obtained:

$$D_T=-0.015T^2+2.051T$$

(5)

where D_T is the micro-strain value of the MISS sensor affected by temperature in με, and T represents the temperature of the environment where the MISS sensor is located in °C. The influence of temperature on the sensor is eliminated by the empirical formula.

3. Experimental Details

3.1. Metallic Materials Tensile Test

To study the effectiveness of the MISS sensor in measuring the strain on the surface of the structure, the metal specimens were selected as a simple illustration. The smartphone used in this test is Honor 7x, and the key parameters of the rear camera of the phone are shown in

Table 1. Rear camera parameters in the Honor 7x.

Parameters	Performance Indicators
Sensor type	CMOS
Flashlight	The LED light
Effective pixels	16 MP
Picture max resolution	4608 × 3456 pixel
Video shooting	1080 P (30 frames/sec)

. Furthermore, the Honor 7x is equipped with an MISS sensor with a gauge distance of 100 mm, so the strain resolution is 9.6 με/pixel.

The metal tensile test was conducted on a universal testing machine with a range of 200 kN. The test specimens (dog-bone shaped), with dimensions of 350 mm length, 30 mm width, and 4 mm thickness, were divided into steel and aluminum specimens according to the material. The response to strain at any point of the metal specimen is the same under loading conditions of uniaxial tension. Figure 6 shows a schematic diagram of the metal tensile test.

Figure 7 presents detailed diagrams of mounting sensors on the metal specimen. The photo of the MISS sensor and FBG sensor mounted on the metal specimen is revealed in Figure 7a. Both the MISS sensor and the FBG sensor were fixed on the surface of the metal specimen. As illustrated in Figure 7b, the fixed points of the MISS sensor were the same as those of the FBG, in order to ensure that the strain range of strain occurring in the specimen measured by both sensors is consistent. The gauge length of the MISS sensor is defined as the distance between the centers of the two supports. The MISS sensor with a gauge length of 100 mm was used for the metal tensile test. Figure 7c shows a technical diagram of the metal specimen.

The test was divided into staged loading and continuous loading according to the loading method. The applied load of each level of gradual loading was 2 kN, and the loading speed of continuous loading includes 0.1 mm/min, 0.2 mm/min, 0.5 mm/min. The specific test conditions are shown in

Table 2. Staged loading test conditions.

Test Case	Specimen Material	Loading Process	Load (kN)
1	Steel	Staged loading	10
2	Aluminum	Staged loading	10

and

Table 3. Continuous loading test conditions.

Test Case	Specimen Material	Loading Process	Loading Rate (mm/min)
1	Steel	Continuous loading	0.1
2	Steel	Continuous loading	0.2
3	Steel	Continuous loading	0.5
4	Aluminum	Continuous loading	0.1
5	Aluminum	Continuous loading	0.2
6	Aluminum	Continuous loading	0.5

.

3.2. Reinforced Concrete Beam Test

As a model for a four-point bending concrete beam structure, a 1.6 m beam with two-point supported rollers was designed as a test specimen. To investigate the performance of the MISS sensor in the structural evaluation of reinforced concrete beams, the MISS sensor with a 100 mm spacing length was arranged in the bent section of concrete beams, while the FBG sensor was used for comparative measurement. Figure 8 shows the schematic diagram of the RC beam test. The loading rate of each stage was set at 2 kN, the load was kept for around 30 s, and the increasing load was applied continuously to a load level of 70 kN. The development of strain was observed after each level of loading was completed. The sensors were installed in the purely curved section of the reinforced concrete beam, and the relative errors of the two methods of measuring strain were compared.

Figure 9a–d exhibit detailed photos of the MISS sensor and FBG sensor mounted to the RC beam. Figure 9a illustrates a photo of the RC beam bending experiment. The detailed photo of the MISS sensor and FBG sensor mounted on the RC beam surface is shown in Figure 9b. The gauge length of the MISS sensor is the distance between the centers of the two supports. The gauge length of the MISS sensor selected in the flexural test of the RC beam is 100 mm, which is the same as that of the metal tensile test. Figure 9c presents a photo of cracks in the measurement region. The RC beam was cast with C20 commercial concrete. The dimension size of the RC beam was 120 mm (width) × 200 mm (height) × 1600 mm (length). The thickness of the concrete cover was 20 mm. As shown in Figure 9d, two HRB 400 rebars with a diameter of 14 mm were used as the tensile longitudinal reinforcement. The longitudinal construction reinforcements were two HRB 400 rebars with a diameter of 8 mm. The stirrups were HPB 300 rebars with a diameter of 8 mm and a spacing of 150 mm.

4. Measurement Results and Discussion

4.1. Metallic Materials Tensile Test

FBG sensors are considered to have high accuracy in measuring structural strain, so a comparison experiment was conducted using the FBG sensor to verify the accuracy of the MISS sensor. Figure 10 shows the comparison of two results of structural strain under the condition of staged loading. For each test, the load was increased from 0 to 10 kN with an increment of 2 kN.

It is found that the strain data measured by the smartphone-based MISS sensor is in good agreement with the data measured by the FBG sensor. Both strains show a linear increasing and decreasing trend during loading and unloading.

In order to quantitatively evaluate the accuracy of strain measurement by the MISS sensor, we compared the difference between the MISS results and the FBG results. Steel specimens were loaded to 600 με. The mean relative errors were 5.4 με, 5.0 με, and 4.2 με for steel specimens with loading rates of 0.1, 0.2, and 0.5 mm/min, respectively. Aluminum specimens were loaded to 1200 με. For aluminum specimens loaded at 0.1, 0.2, and 0.5 mm/min, the mean relative errors were 7.6 με, 11.1 με, and 9.3 με. Using the loading time as the horizontal axis and the strain measured by the sensor as the vertical axis, the results of the strain development of the specimen under different working conditions are shown in Figure 11.

There may be two sources of error. One is that there may be a small adhesive slip between the outer tube and the inner tube wall of the sensor, and the other is that environmental noise has a certain impact on the test results. These errors are acceptable considering that the actual structure will undergo greater deformation.

To further evaluate the accuracy of the MISS sensor in measuring strain, the correlation coefficients of these two types of data were calculated and the results were shown in

Table 4. Continuous loading test conditions result of the two types of data.

Test Case	Specimen Material	Loading Rate (mm/min)	Mean Error (με)	Correlation Coefficient
1	Steel	0.1	5.4	0.9998
2	Steel	0.2	5.0	0.9998
3	Steel	0.5	4.2	0.9997
4	Aluminum	0.1	7.6	0.9998
5	Aluminum	0.2	11.1	0.9998
6	Aluminum	0.5	9.3	0.9996

. The correlation coefficient r was calculated using the formula:

$$r\left(x,y\right)=\frac{Cov\left(x,y\right)}{\sqrt{Var\left[X\right]Var\left[Y\right]}}$$

(6)

where Cov(x,y) is the covariance of X and Y, Var[X] is the variance of X and Var[Y] is the variance of Y.

As can be seen from Table 4, the lowest correlation coefficient is 0.9996, which occurs when the aluminum specimen was loaded at 0.5 mm/min. The mean error for the six loading conditions was 7.1 με. The MISS sensor shows a good ability to measure structural strain in the uniaxial tensile testing of metals.

4.2. Reinforced Concrete Beam Structure

The mean strain obtained by the MISS sensor within the gauge length of the concrete beam is shown in Figure 12. Where a gives the response curve of the load, b gives the absolute error measured and relative error measured by the MISS sensor and the FBG sensor.

In this experiment, the maximum value of the absolute error occurs at loading up to 54 kN, with a maximum value of 68 με. The maximum value of the relative error appears at the very beginning of the loading phase, which is due to the small, applied load and the small strain generated by the concrete beam, when the relative error is more sensitive and variable. Furthermore, with the increase of the load, the relative error is gradually reduced and basically stabilized within 2.5%. This verifies the accuracy of the MISS sensor in the strain measurement of concrete beams.

5. Conclusions

This paper researches the application of MISS sensors in real structural members and solves the problem of connecting the MISS sensors to smartphones in the process of measuring structural strain. A series of metal tensile tests and concrete beam bending test studies were conducted to verify the sensor’s performance in measuring strain. The accuracy of the MISS sensors was determined by comparing the strain measurements of FBG with those of MISS. The results are as follows:

1.

Metal tensile tests were carried out on various materials, loading speeds, and loading methods. It was found that the material of specimen and loading speeds had little effect on the measurement results. Under the two loading methods of graded loading and continuous loading, the MISS sensor can obtain the strain value of the specimens effectively. In the metal tensile test, the mean error is 7.1 με and the correlation coefficient is as high as 0.9997 between the FBG sensor and the MISS sensor measure results.

2.

The strain measurement of the pure bending section of the RC beam further verified the effectiveness of the MISS sensor. The mean relative error of strain values is 2.5% in the reinforced concrete beam bending test.

The MISS sensor is fixed on the structure, the mobile phone is connected to the MISS sensor when measuring the strain, and the smartphone can be removed during the non-measuring process. The strain images acquired by smartphones can be collated, transmitted, and archived by various means such as WIFI and Bluetooth, which is a low-cost, portable, and fast means of strain measurement and can be commonly used in practical engineering. Note that the MISS sensor was coupled with a smartphone in this research. However, in the case of citizen-participated structural strain measurement, it is necessary to study the synergistic work between the different actors. Besides, the next task is to make full use of the advantages of the MISS sensors to establish a public-participatory strain monitoring platform.