An Intelligent Breast Ultrasound System for Diagnosis and 3D Visualization

Abstract

Background: Ultrasonography is the main examination method for breast diseases. Ultrasound imaging is currently relied upon by doctors to form statements of characteristics and locations of lesions, which severely limits the completeness and effectiveness of ultrasound image information. Moreover, analyzing ultrasonography requires experienced ultrasound doctors, which are not common in hospitals. Thus, this work proposes a 3D-based breast ultrasound system, which can automatically diagnose ultrasound images of the breasts and generate a representative 3D breast lesion model through typical ultrasonography. Methods: In this system, we use a weighted ensemble method to combine three different neural networks and explore different combinations of the neural networks. On this basis, a breast locator was designed to measure and transform the spatial position of lesions. The breast ultrasound software generates a 3D visualization report through the selection and geometric transformation of the nodular model. Results: The ensemble neural network improved in all metrics compared with the classical neural network (DenseNet, AlexNet, GoogLeNet, etc.). It proved that the ensemble neural network proposed in this work can be used for intelligent diagnosis of breast ultrasound images. For 3D visualization, magnetic resonance imaging (MRI) scans were performed to achieve their 3D reconstructions. By comparing two types of visualized results (MRI and our 3D model), we determined that models generated by the 3D-based breast ultrasound system have similar nodule characteristics and spatial relationships with the MRI. Conclusions: In summary, this system implements automatic diagnosis of ultrasound images and presents lesions through 3D models, which can obtain complete and accurate ultrasound image information. Thus, it has clinical potential.

1. Introduction

As female sex organs, breasts have both internal function and external beauty, which are pivotal to women [1,2,3,4,5]. Usually, breasts are continuously undergoing developmental changes, and the structure of their glands is complex. Thus, treatment of breast nodules is always confusing for doctors [6,7,8]. At present, doctors mainly rely on the observation of breast images to find the developmental law of lesions. This can not only correctly recognize diseases and treat them reasonably, but also improve the understanding of breast nodule growth.

Ultrasonography has the characteristics of a real-time observation, low cost and high diagnostic level, and it has become the most important method of breast examination [9,10,11,12,13]. Hospitals generate many ultrasound images of the breast every day, which requires experienced ultrasound doctors to analyze the ultrasound images. However, most breast ultrasound images are tumor-free, so ultrasound doctors spend a lot of time every day analyzing tumor-free ultrasound images, which is a waste of ultrasound doctor resources. At the same time, the ultrasound image results of tumor patients mainly depend on the doctor’s statement of the image characteristics, which depends largely on the doctor’s professional level and expression ability. In addition, the subjective will of the patient is often the main factor in determining the treatment plan. Therefore, how to diagnose ultrasound images efficiently and quickly, and how to more accurately express the lesion structure, are urgent problems to be solved. It is of great scientific significance for patients to understand the results of breast ultrasound and explore the potential for breast disease [14] more intuitively.

Researchers have proposed several medical image nodule diagnosis algorithms based on convolutional neural networks, such as DenseNet [15], GoogLeNet [16], and ResNet [17] used in this paper. Due to the low resolution of the breast ultrasound image and the diversity of shape and texture features of lesions, a single network model is not generally effective for benign and malignant diagnosis of lesions, and has the disadvantage of poor generalization.

In addition, 3D models of breasts obtained by reconstruction allow patients to have more intuitive understanding and communication with doctors. The current methods for obtaining 3D breast structure include computed tomography (CT) and MRI imaging. Due to space limitations of the scanning machine, patients can only adopt a lying-down position when acquiring CT and MRI, which is different from the reclining posture on the operating table, resulting in greater deformation of the breast. Therefore, reconstructed 3D models cannot correctly realize surgical navigation. Although there are already some methods that enable doctors to collect 3D ultrasound data of patients, they require cumbersome positioning and take a long time [18].

In this work, we proposed a breast ultrasound system, which can simultaneously diagnose and visualize breast ultrasound nodules. The remainder of this paper is organized as follows: In Section 2, we introduce the method of neural network ensemble and 3D breast model generation. In Section 3, the experimental results and discussion of the ensemble network and 3D breast nodule model are presented. Section 4 provides a conclusion about the investigated methods, challenges, and future directions for the employment of the system in clinical applications.

2. Methods

2.1. 3D-Based Breast Ultrasound System

The breast ultrasound system consists of two parts, as shown in Figure 1. The first part is the intelligent diagnosis of breast ultrasound image by ensemble neural network. The ensemble method of neural network is shown in Figure 1a. The second part is the 3D visualization of breast locator, which is designed based on the structural characteristics of female breasts to realize the positioning of female breast nodules. It includes ultrasound scanning equipment, workstation (Figure 1b), and locator (Figure 1c and Figure 2). GE LOGIQ E9 Ultrasound Machine is adopted as ultrasound scanning equipment provided by Ultrasound Supply. The workstation environment is Microsoft Workstation, Python 3.6. Operating software is 3D-based Breast Ultrasound Software, which is mainly developed based on The Visualization Toolkit (VTK) official application programming interface (version 7.1.1). Locator is designed based on the structural characteristics of female breasts to realize the positioning of female breast nodules.

The flowchart of breast ultrasound system proposed in this work is shown in Figure 3, which is divided into intelligent diagnosis and 3D visualization. In the intelligent diagnosis part, the breast ultrasound images are input into a pre-trained network, and the prediction results of the ensemble neural network are obtained by weighted ensemble, to realize the automatic benign and malignant diagnosis of breast ultrasound images and save medical resources. In the 3D visualization part, the ultrasound doctor firstly analyzes the ultrasound image and obtains the spatial position and geometric characteristics of breast nodules. After the analysis, the corresponding parameters are input into the 3D visualization software, and the software automatically generates the 3D breast nodule model.

2.2. Ensemble Neural Network

Neural networks usually require a large amount of data to train the network. However, the amount of ultrasound image data in the breast is generally low. To achieve good performance in a small dataset, we select three networks from widely recognized network: DenseNet [15], GoogLeNet [16], ResNet [17], VGG16 [19], ResNeXt [20], AlexNet [21], MobileNet [22], and NasNet [23]. Then, we combine them via the weighted ensemble method. In our work, we use a public breast ultrasound image dataset [24] (647 in total) to train these three neural networks and test the performance of a single neural network and ensemble neural network.

The dataset is divided into 80% training set and 20% testing set. After training the three neural networks, the trained model is obtained. To take advantage of different models, we use a weighted ensemble strategy (Figure 4) to combine their results. It is assumed that the benign probability values of the neural network outputs are P₁, P₂ and P₃, respectively. By multiplying these three results by their respective weights (W₁, W₂ and W₃) and summing them up, the sum result is then mapped to the range 0–1 by Sigmoid function to make sure the ensemble result is a probability distribution. The benign and malignant probability value of the ensemble neural network is shown in the following formula:

$$P_{benign}=\mathrm{Sigmoid}\left(W_1{\times P}_1{+W}_2{\times P}_2{+W}_3{\times P}_3\right)$$

(1)

$$P_{malignant}{=1-P}_{benign}$$

(2)

In our work, W₁ = W₂ = W₃ = 0.33. The parameters above are evaluated by experienced breast ultrasound doctors according to the performance of a single neural network.

2.3. Locator

Locator is mainly worn on the patient’s breast. Its schematic is displayed in Figure 2. It is designed as a spherical structure that matches the shape of a human breast. The center of the inner side of the locator is equipped with a round hole-shaped groove that matches the position of the nipple, and the outer side is marked with a plurality of first- and second-measurement lines for measuring the breast of the wearer. The first-measurement lines with 15 degrees unit distance are used to obtain the angle of the breast, and the second-measurement lines with 0.1 mm unit distance measure the length of the breast. Each first-measurement line, respectively, extends from the center to the edge of the spherical structure. They are equal in length, and each is marked with an angle. Each second-measurement line is an annular line with the center of the spherical structure as the circle center. Second-measurement lines are parallel to each other, and each one is marked with a distance. Thus, locator can realize the structured measurement of patient’s breast, including measuring the height of the breast axis and the diameter of the breast base.

Conversion method of spatial information is as follows: Firstly, a straight line to connect origin and nodule center is made. Arc length from intersection of the straight line and skin to center of the nipple is defined as $l$, measured by locator’s second-measurement lines. Angle of nodule center on horizontal plane is $\theta$, measured by the first-measurement lines of locator. It ranges from 0 to 360 degrees, increasing in counterclockwise direction. In addition, vertical distance between nodule center and skin is defined as $d$, which can be measured by an ultrasound scanner. Then, sphere center of locator is taken as origin, and surface of locator is sphere. Thus, a coordinate system is established and its z-axis is from the center of the sphere to the nipple. Horizontal direction from right to left side of the chest is uniformly defined as x-axis, which satisfies the left-hand coordinate system. $\theta =0$ is satisfied on x-axis. Therefore, the ball radius is radius of curvature of nipple horizontal section.

Assuming that breast height is $H$ and width is $2W$, ${\left(R-H\right)}^2+W^2=R^2$ is satisfied and $R=\frac{H^2+W^2}{2H}$ is obtained (Figure 5a). $H$ and $W$ can be measured by doctors.

To obtain z-axis coordinates of the nodule position, $\alpha$ and $d_1$ are defined as shown in Figure 5b, where red area represents the nodule, which satisfies $\alpha =\frac{l}{R}$. Because of $d_1=R-d$, the nodule z coordinate can be expressed as:

$$\begin{array}{cc}z\hfill & =d_1\times cos\alpha \hfill \\ & =\left(R-d\right)\times cos\left(\frac{l}{R}\right)\hfill \end{array}$$

(3)

Calculation of the x-axis and y-axis coordinates is as follows: projection length on the XOY plane of the line from the nipple center to the origin is defined as $d_2$ (Figure 5c), which satisfies:

$$\begin{array}{cc}{d}_2\hfill & =d_1\times cos\left(90°-\alpha \right)\hfill \\ & =\left(R-d\right)\times sin\alpha \hfill \\ & =\left(R-d\right)\times sin\left(\frac{l}{R}\right)\hfill \end{array}$$

(4)

Therefore, x and y coordinates of the nodule are:

$$\{\begin{array}{c} x=d_{2}cos\theta =\left(R-d\right)sin\left(\frac{l}{R}\right)cos\theta \\ y=d_{2}sin\theta =\left(R-d\right)sin\left(\frac{l}{R}\right)sin\theta \end{array}$$

(5)

Since $z >0$ and the range of $\theta$ are $0–360$ degrees, positive and negative changes of x and y coordinates are consistent with Equation (3). Therefore, $\left(x,y,z\right)$ of the breast nodule in the 3D coordinate system can be converted through actual measurement parameters.

2.4. Model Preparation and Report Generation

Preparation of the models is mainly based on signs of breast ultrasonography. Referring to ultrasound images, four basic nodule models of smooth, burr, horned, and lobed cases are established. Dealing with nodules of different sizes, horizontal and vertical length, and height of nodules measured by ultrasound scanner correspond to the size in x-, y-, and z-axis. Thus, through size transformation of models, they are contracted or expanded. For a smoothing effect, noise and subdivision are performed on the basic geometric model. Some textures realize calcification (microcalcification, arc-shaped calcification, etc.) and blood flow (poor blood supply, rich blood supply) through basic geometric combinations. In addition, the setting of outer cover establishes a fuzzy high-echo halo feature. Some models need to be provided with materials. Cinema 4D model materials are applied to control color and transparency. According to input information, size transformation and morphological combination of models are carried out. Finally, three typical body types are established.

Generating module is composed of three parts, including 3D space positioning information of lesions, internal information of 3D lesions, and external information of 3D lesions. The 3D space positioning information marks the center position of the nodules, internal information shows textures and characteristics of nodules, and external information expresses the relative position of nodules in the breast. Thus, a 3D ultrasound diagnosis report is generated, which has the patients’ name and number, 3D models of the nodules, and diagnosis information. The 3D model is adjusted to appropriate angle, viewed through manual interaction of visualization module, and obtained through real-time screenshots. Finally, it is output according to the current report format commonly applied in hospitals and printed by the writing function.

3. Experiment Results and Discussion

3.1. Ensemble Network

To evaluate the performance of the network, we use the accuracy, recall, F1 score and the AUC (area under curve) of ROC (receiver operating characteristic curve) metrics [25,26]. Results of different neural networks and ensemble networks are shown in

Table 1. Results of different neural and ensemble networks on a testing set.

Neural Network	Accuracy	Recall	F1 Score	AUC
DenseNet	0.7752	0.5952	0.6329	0.8328
GoogLeNet	0.8294	0.7381	0.7381	0.8760
AlexNet	0.7984	0.7143	0.6977	0.8410
ResNet	0.7597	0.5238	0.5867	0.8240
MobileNet	0.7287	0.4524	0.5205	0.7466
NasNet	0.8062	0.6190	0.6753	0.8481
ResNeXt	0.7829	0.5952	0.6410	0.8106
VGG16	0.8372	0.6905	0.7342	0.8547
(NasNet, AlexNet, DenseNet)	0.8295	0.6667	0.7179	0.8719
(NasNet, GoogLeNet, AlexNet)	0.8450	0.7381	0.7561	0.8790
(VGG16, GoogLeNet, NasNet)	0.8527	0.6905	0.7532	0.8831
(VGG16, NasNet, DenseNet)	0.8992	0.7619	0.8312	0.8711

The bold numbers indicate that the network performs best on this metric.

. We first train all individual neural networks and calculate the metrics, then select the model with higher F1 score and combine the models through weighted ensemble strategy. Finally, we determine four groups of neural network combinations. They are (NasNet, AlexNet, DenseNet), (NasNet, GoogLeNet, AlexNet), (VGG16, GoogLeNet, NasNet), (VGG16, NasNet, DenseNet). By using the weighted ensemble strategy, the ensemble neural network is obviously improved compared with that of a single network. The performance and metrics of these networks have listed in the Table 1, Figure 6 and Figure 7.

Among combination groups, the (VGG16, NasNet, DenseNet) group achieved the best result, with acc of 0.8992 and F1 score of 0.8312. Compared with the best single neural network, VGG16, accuracy increased by 6.2%, and F1 score increased by 9.7%. All of the above indicate that the ensemble network obtained by the weighted ensemble method has better performance and is more suitable for benign and malignant diagnosis of breast ultrasound images.

3.2. Breast Lesion 3D Visualization

Two senior sonographers (both with over 25 years of experience) conducted experiments on three subjects. First, lesion space information of three patients was obtained through the locator. Then, 3D-based Breast Ultrasound Software realized the generation of models and reports. Information representation of the three patients is shown in

Table 2. Characterization information of three ultrasound volume data.

Data	Characterization Information Description
Subject 1	Medium breast shape; upper right half, 5 × 15 × 23 mm, smooth, lengthwise, clear and complete, and position (51 mm, 80°, 10 mm). Upper left half, 46 × 35 × 63 mm, burr, horizontal length, rich blood supply, position (0 mm, 0°, 30 mm).
Subject 2	Medium breast shape, upper right side, 42 × 31 × 44 mm, burr, horizontal and long, clear and complete border, lack of blood supply, position (0 mm, 0°, 27 mm). The upper left half, 50 × 58 × 60 mm, burr, rich blood supply, clear and complete border, position (0 mm, 0°, 35 mm).
Subject 3	Medium breast shape, upper right side, 41 × 30 × 39 mm, burr, lengthwise, clear and complete, rich blood supply, location (26 mm, 340°, 20 mm).

. In addition, three subjects underwent MRI scans at the same time, and the acquired MRI volume data were analyzed by the operating imaging physician in Mimics Research 19.0 to complete segmentation of the skin layer, lesion, and pectoralis major in sequence.

To unify the structure, all models used green nodules. Pectoralis major muscle tissues were shown as dark red with a certain degree of transparency, and white with high transparency were used as skin tissue. In the ultrasound reports of the first group, the most typical patient feature was that the size of the left thoracic nodule was 46 × 35 × 63 mm (Figure 8a), which was about 58.8 times larger than that of the right thoracic nodule (Figure 8b). Therefore, there were obvious differences in individual size between the two lesions, which are clearly shown in the green nodules in Figure 8c,d. Using the 3D-based breast ultrasound system, a basic burr nodule model was established on the left side of the chest, and the size was transformed according to the horizontal length and size. Basic geometry was added to show the state of rich blood supply (Figure 8e). On the right side of the chest, a smooth basic nodule model was established, and the size was transformed according to length and size (Figure 8f). Finally, all positionings were completed by the spatial information. Comparing the reconstruction results from MRI and models generated by the 3D-based breast ultrasound system, shape, size, and spatial location basically matched, meeting the clinical requirements of the doctor–patient information guidance.

Figure 9a,b reveal MRI imaging parts of the second patient, which are different from the first patient. Through a 3D model reconstructed by the MRI, it can be observed that nodules on the left and right sides of the chest are large and have obvious burr characteristics (Figure 9c,d). In the 3D-based breast ultrasound system, the left and right sides of the chest were both imported with the burr nodule model, and transformation was computed from the actual size. A basic geometric model to the left thoracic nodule was added to simulate the state of a rich blood supply (Figure 9e). In the end, the generated model basically matched the size and the spatial location of the reconstructed model’s structural features (Figure 9e,f).

Figure 10a introduces parts of the MRI imaging of the third group of cases. Based on the reconstruction structure, only the right side of the chest contained medium-sized nodules, showing burrs and a rich blood supply (Figure 10b). The 3D-based breast ultrasound system imported the burr nodule model and added basic geometric structure as blood structure. Finally, it was stretched according to scale. The generated model was basically the same as the reconstructed result (Figure 10c). It is worth mentioning that muscle fiber length characteristics of the third group of cases are obvious. However, the model generated in the 3D-based breast ultrasound system cannot reflect this feature. This is because the 3D-based breast ultrasound system only contains a small, medium, and large chest that can be contracted in proportion, so it cannot reflect the patient’s weight, chest width, and chest length. This will bring an error to the relative position of nodules and thoraxes.

3.3. Discussion

In breast nodule diagnosis, compared with the neural network models used in this paper (DenseNet, GooGleNet and AlexNet), the model generated by the weighted ensemble method improved in accuracy, recall and F1 score metrics to varying degrees. Among the experiments, the combination of VGG16, NasNet and DenseNet, with accuracy of 0.8992, recall of 0.7619 and F1 score of 0.8312, obtained the best score and verified that the ensemble neural network generated by the weighted ensemble method can effectively improve the performance of the neural network and better diagnose breast ultrasound images.

In breast nodule visualization, the size and location of nodules generated by the 3D breast ultrasound system basically matched the reconstructed structure. This is because the ultrasound probe and locator can accurately measure the size and position of the nodule. The error mainly comes from three aspects: (1) the measurement error of the ultrasonic breast nodule size may be due to human error; (2) when locating breast nodules on ultrasound, the selection of reference points (usually the area near the center of the nodules) may lead to localization errors; (3) structural errors, including calcification, blood supply and hyperechoic halo, may occur when the simulated edge shapes of basic nodules are different. It is worth mentioning that the 3D visualization of the breast ultrasound system aims to provide instructive 3D ultrasound reports to help doctors construct surgical navigation and facilitate communication with patients. Therefore, the 3D visualization of the breast ultrasound system does not need to establish a high-precision model system. However, how to more accurately simulate the patient’s body shape, simulate more accurate characterization methods, and enrich the basic model of nodules are still problems worth exploring. This not only deals with common nodule representations, but also generates a guiding model structure for complex situations. This also provides a direction for the next generation of 3D-based breast ultrasound systems.

4. Conclusions

This paper proposes an intelligent diagnosis and 3D visualization system based on ultrasonic breast images. The main work includes: first, through training widely recognized classical networks, the ultrasonic image experts evaluate the network performance and give the weight for the following ensemble. The ensemble neural network is obtained by the weight. Second, based on the texture characteristics of ultrasonic images, the basic model and geometric combination rules of breast nodules were used to construct a three-dimensional model of typical breast nodules. Third, a locater-based method for locating breast nodules was proposed to achieve spatial calibration of nodular information in ultrasound images. Fourthly, the performance of the network was evaluated on the test set with 129 samples on three separate networks and on the ensemble neural network. Lastly, to verify the 3D visualization performance of the system, three groups of clinical patients were tested. After comparison, the model generated based on the 3D breast ultrasound system was basically consistent with the reconstructed 3D nodular structure. Therefore, the 3D-based breast ultrasound system proposed in this paper can realize automatic diagnosis of breast ultrasound images, meet certain clinical needs, and solve the serious problems existing in doctor–patient communication and preoperative breast navigation. It has clinical benefits.