A Review of Large-Scale Simulations of Microstructural Evolution during Alloy Solidification

Abstract

During the past two decades, researchers have shown interest in large-scale simulations to analyze alloy solidification. Advances in in situ X-ray observations of the microstructural evolution of dendrites have shown defects that can be very costly for manufacturers. These simulations provide the basis for understanding applied meso-/macro-scale phenomena with microscale details using various numerical schemes to simulate the morphology and solve for transport phenomena. Methods for simulating methodologies include cellular automaton, phase field, direct interface tracking, level set, dendritic needle networks, and Monte Carlo while finite element, finite difference, finite volume, and lattice Boltzmann methods are commonly used to solve for transport phenomena. In this paper, these methodologies are explored in detail with respect to simulating the dendritic microstructure evolution and other solidification-related features. The current research, from innovations in algorithms for scaling to parallel processing details, is presented with a focus on understanding complex real-world phenomena. Topics include large-scale simulations of features with and without convection, columnar to equiaxed transition, dendrite interactions, competitive growth, microsegregation, permeability, and applications such as additive manufacturing. This review provides the framework and methodologies for achieving scalability while highlighting the areas of focus that need more attention.

1. Introduction

The research in the simulation of dendritic growth during alloy solidification used to be limited to small-scale, single dendrites that did not fully capture all of the physics behind solidification. The computation power and cost required for superior analysis have been the main hindrance for performing more complex, large-scale simulations. This article outlines the advances in both mathematical models and computational hardware/software that enable large-scale simulations of solidification microstructure.

While there is a significant amount of work on modeling solidification microstructure and dendrite growth including a few review papers from distinguished researchers [1,2], there has not been a review work on the large-scale modeling of solidification microstructure covering the various numerical techniques, computational methods, and examples of what has been accomplished thus far.

The goal of this review is to provide a comprehensive compilation of the most current research that has been conducted on large-scale simulations of solidification microstructure using advanced computing methods. Large-scale simulations of dendritic growth can greatly benefit manufacturing and materials engineers and scientists, offering a detailed understanding of microstructural evolution during solidification processes. Reducing the trial and error of the manufacturing process, the simulations save time, providing a greater understanding of the microstructure and defect formation. Many advances in in situ X-ray observations of solidification have identified areas that can benefit from the large-scale realistic modeling of dendrites [3,4,5,6,7,8,9,10]. Lee and Hunt [11] first examined slabs of Al–Cu alloys for hydrogen pores in situ during directional solidification (DS). They discovered that as the growth speed of the dendrites increased, along with hydrogen content, the shape of the pores evolved into a “worm-hole-like” shape. Defects that form can be caused by numerous conditions such as gas bubbles, segregation, and tears as well as other factors that can be difficult to predict due to the rheological behavior of the mushy zone [12,13]. This cycle of casting and taking x-rays of the samples are both costly and time consuming. Therefore, a lot of research has been conducted to understand the physics behind these defects [14,15,16,17,18,19,20,21] and reproducing them using computer simulations. The initial simulations produced two-dimensional (2D) axisymmetric, branchless dendrite features that lacked the details needed to understand the issues that develop during the manufacturing processes. Therefore, there are three-dimensional (3D) aspects that are not captured by 2D simulations, which will be discussed in detail in later sections. However, 3D simulations require extensive computing power and algorithms to scale. Recent advances in the large-scale simulation of the solidification microstructure have produced some astonishing new discoveries [22]. This has been made possible thanks to developments in parallelization and supercomputing. State-of-the-art hardware and algorithm techniques have enabled researchers to produce computations that used to be seen as impossible. The ability to see high resolution in the formation of microstructures has never been more accessible. The size, spacing, and morphology of dendritic arms show a significant effect on the properties of the solidified materials [1].

Solidification modeling is a complex multiphysics problem including fluid flow, heat, and solute diffusion aspects. There is also a need to solve equations for phase change or track the interface to simulate the morphology. The morphology of the dendrites can be captured using cellular automaton (CA), phase field (PF), direct interface tracking (DIT), level set (LS), or dendritic needle networks (DNN) methods [21]. In terms of reproducing the physics, PF is one of the best methods to simulate dendrite growth accurately when compared to the front-tracking, boundary integral methods, and CA. However, depending on the aspect under consideration, each method has its own strengths and shortcomings. The majority of large-scale simulations of dendrite growth have used PF and CA methods. The transport phenomena, which consists of either heat transfer, diffusion, and fluid flow, can be solved using finite element (FE), finite difference (FD), finite volume (FV), or lattice Boltzmann (LB) methods. Scalability is accomplished using computational approaches that can take advantage of increasing the number of processing units. The selected numerical methods need to be scalable, making it easier to implement different parallelization techniques by using threads, message passing interface (MPI) [23,24,25,26,27,28], and compute unified device architecture (CUDA) [29,30,31,32], among other methods. Reducing the computational time is still a challenge for large-scale simulation. Therefore, many strategies have been developed over the years to address the computational time. Some researchers have focused on utilizing parallel computing algorithms to speed up the simulation while others have tried to optimize the meshing method and reduce the overall computational time by using the adaptive mesh refinement (AMR) strategy [33] or by applying different grid size and time step methods [34,35].

From a computational power point of view, technology is increasing toward Moore’s law and the exascale era is already upon us [36,37,38]. Supercomputing has provided a small deviation from expectations, now known as the post-Moore era [39,40]. Now, the strategy to increase the ability to model dendritic solidification is finding limitations in mathematical models and algorithms. This limitation is partly associated with interpolation of the quantities around the interface of the dendrite where sharp gradients are present. By implementing an adaptive grid refinement model, the code’s capability will be extended to a more accurate prediction of the interfacial region. In the AMR algorithm, finer grids are used for sharp gradient regions such as the solid–liquid interface, while the mesh coarsens in the rest of the computational domain. As a result, most computation time is spent solving the region occupied by the mushy zone. This method has been used by many researchers separately or in combination with other parallelization methods to reduce the computational time [41,42,43]. Recently, parallel-GPU AMR has shown great promise in combining the effectiveness of the AMR scheme with the power of GPU computing [44,45,46]. This allows for high-speed simulations of purely diffusive dendrite growth.

These outstanding large-scale simulations have inspired many researchers to perform analysis on realistic grain growth. As an example, Miyoshi et al. [47] conducted a PF simulation study of ideal grain growth in an ultra-large-scale domain. Using a supercomputer, TSUBAME 2.5 at the Tokyo Institute of Technology, which consists of 1408 nodes, the team was able to investigate the number of sample grains that are necessary for repeatable results for grain size distributions. The largest domain consisted of 2560³ grid points, which consisted of 3,125,00 initial grains. At this time and scale, it was the largest simulation performed by a factor of 10. This was an effective performance for observing and quantifying the steady-state growth behaviors. Another impressive simulation of large-scale was performed by Sakane et al. [48], who used a quantitative PF model to simulate a dilute binary alloy’s (Al-3 wt.% Cu) grain growth under the influence of forced convection. A directional solidification condition was produced in a system with dimensions of 3.072 × 3.078 × 3.072 mm³. The computation of 1024³ mesh points, during 60,000 steps, was completed using 128 GPUs within four hours. Their PF model could provide consistent results independent of the interface thickness. An LB model was coupled to simulate the melt flow.

The focus of this paper is to highlight important research on the large-scale simulation of dendritic growth and other microscale solidification features. We will discuss numerical methods as well as computational approaches used to produce large-scale simulations of the solidification microstructure. The actual domain size of these simulations depends on the physics considered and the features that are being simulated. The structure of this review will focus on the increasing complexity of features. Many case studies will be discussed in the following order: dendritic interactions and competitive growth; columnar to equiaxed transition (CET); solute transport and segregation; natural and forced convection; permeability. The details for each subject are outlined with respect to both the mathematical/computational models as well as the physical nature of the simulations.

2. Methods for Simulating the Morphology

The process of capturing the interface morphology in simulations can be accomplished with the CA, PF, DIT, or LS methods. Each method has its unique advantages and deficiencies. CA, which is relatively simple, stands alone from the others because of its local structure and will be discussed first. PF is probably the most powerful and will be analyzed with respect to some of the most notable case studies. Third, DIT will be discussed to feature the differences using a couple of research examples. Finally, the LS method will be reviewed, which is similar to DIT but requires knowledge about the temperature gradients at the interface. Dendrite needle network (DNN) and Monte Carlo (MC) methods will also be discussed. By simulating the morphology of dendrites, the complexity in geometry can be incorporated for the large-scale domain for manufacturing purposes.

2.1. Cellular Automation

CA is ideal for large-scale modeling simulations as its structure is considered local with respect to the objective domain. The ability to approach the morphology naturally, in a localized manner, allows for easy scalability on many processors. In the 1940s, the CA method was developed by John von Neumann using simple microscopic laws to model complex, macroscopic phenomena physically [49]. With a successful application in metallurgy, the CA method can accurately simulate processes such as grain growth, cracking, diffusion, or mechanical deformation. By superimposing a grid of cells on the area of interest, the evolution of these physical processes can be reproduced. State indices and variables are attributed to cells that evolve based on transition rules [50], outperforming the standard GPU implementations, making this effective for scalability as the interactions of cells can be defined in three dimensions. Advantages over PF include being able to develop a spatial resolution by the order of magnitude of the smallest microstructure feature size [51]. CA is highly parallelizable, making it two orders of magnitude faster than alternative PF methods [52].

One of the largest strictly CA methodologies was recently performed by Zhang et al. (shown below in Figure 1) [53]. The study focused on the simulation of an Fe–C alloy during isothermal and directional solidification while using the CA model to simulate the CET. Using the CA model in combination with GPUs and MPI, the maximal speed-up ratio was measured to be 153.19. They were able to study the effects of increasing the cooling rate among other conditions, which promotes the occurrence of equiaxed dendrites ahead of the solidification front. The CA code was able to process 768³ grids within 27.42 min for 8000 time steps. They were able to solve the problem of data race in the CA model by introducing an additional field variable with a modification to the capture rule. CA is regularly coupled with methods to solve for solute transport given its functional, elemental nature that allows it to be scaled [54,55,56]. The CA algorithm accounts for heterogeneous nucleation, the preferential growth directions, and growth kinetics of the dendrites [57]. Rafii-Tabar and Chirazi provide a comprehensive analysis of modeling solidification techniques in their review on multiscale modeling using CA and coupling techniques [58]. CA is often coupled with other methods such as FE and FD, which are described in later sections of this review in context to AM applications.

2.2. Phase Field

The PF method is a powerful and versatile tool to model microstructural dynamics [59,60]. Many phenomena can be represented using a simple set of differential equations [61]. Using continuous fields to describe the interfaces, the discontinuities of properties and boundary conditions are represented by a variation of one or more auxiliary field. These phase fields across a diffused interface are solved by integrating partial differential equations for the whole domain. Free-boundary problems with arbitrary complex interfaces are notoriously challenging [62].

Kobayashi [63] first brought the PF modeling of dendritic growth to the public’s attention in 1993, showing a simple simulation for one component melt growth and showed its ability to solve the free-boundary problem. PF solves the time evolution equation of the PF variable, Ø. This was introduced to express the phase state (solid or liquid) of the material, in which the sharp interface is replaced by a diffuse interface. With a finite thickness, the PF variable has a smooth, but steep change. The versatility of the PF method allows for the simulation of the interface migration without tracking of the interface [64]. By simply solving an equation, PF can solve for the solid concentration, curvature of the interface energy, and capture the interface based on computational tracking. Kobayashi actually produced movies of grain growth simulation in 3D prior to 1990 [65], which inspired researchers to reproduce this method for larger scale domains.

PF is very computationally taxing, where the meshes for the interface require high density. AMR has been implemented on a large scale under certain conditions [66] such as 3D modeling [67,68,69] (but mostly 2D conditions due to limitations) as computational efficiency decreases as the volume fraction of the interface increases [70]. Therefore, parallel computational schemes are necessary for very large-scale simulations using a quantitative PF model. Shibuta et al. [71] utilized a supercomputer and PF to simulate the competitive growth of dendrite assemblages. They discussed convergence behavior that enabled the use of a large interface thickness. At the time (2011), it was the largest reported simulation of dendrite growth for a domain of 3.072 mm³ for a total time of 100 s, using 768 GPUs on the supercomputer TSUBAME 2.0 (TSUBAME 2.5′s predecessor). It is computationally oppositional to obtain a realistic looking microstructure and an extremely thin interface [72]. PF parameters are derived at a thin interface limit [73], which is competitively being pushed further.

Two years later, Takaki et al. [74] used the TSUBAME 2.0 for a very-large-scale 3D PF simulation of directional solidification of 3.072 × 3.078 × 3.072 mm³, which was equivalent to 4096 × 4104 × 4096 meshes. This research produced interesting results (Figure 2), where unfavorably oriented dendrites and highly complicated interactions could be observed. As a result, controlling microstructures in terms of crystallographic structures has been considered for more complex structures. This is important as it expanded on both the domain size and complexity that was previously discussed by Shibuta et al. [71].

In order to simulate large-scale microstructures using PF models, the PF interface is required to be scaled up much larger than the physical interface to remove several artifacts that step from the thickness of the interface [75]. Using interpolation functions for average diffusivities and grand potentials of the bulk phases, the interface stretching is eliminated. The interface also faces (unrealistic) movement from the relaxation that can be countered by asymptotic analysis. Finally, artificial solute trapping can be mitigated by anti-trapping formulations [76]. These conditions need to be met to scale to higher resolutions while maintaining accuracy using PF methods. PF’s advantage over the other methods lies within the field variables, which eliminate the need to identify and track the interface. Tourret et al. highlighted some of the most recent applications, perspectives, and challenges in PF modeling in the most recent review [60].

2.3. Direct-Interface-Tracking

DIT, a front tracking method, is a successful methodology for solving the energy equation as well as momentum and mass conservation equations. This requires treating the interface as incompressible. Since it requires the calculation of the temperature gradients at the interface in combination with the normal velocity and curvature of the interface, DIT is less powerful in comparison to the other techniques already mentioned [22].

Zhu et al. [77] used a quantitative virtual front tracking model, which is characterized by its mesh independency, to simulate 2D dendrite growth in the low Pėclet number regime. This means that the results converge to a finite value when the mesh size is refined. It has been observed that the mesh dependency is influenced by the methods of curvature and solid fraction calculations [78]. The simulated columnar dendrite evolution of an Al-4 wt.% Cu alloy was calculated in a domain of 600 × 1200 meshes on a single CPU in about 11 h of computational time. Figure 3 shows the evolution of the dendrites after 12 s when comparing the grain boundaries and solute map. The equilibrium composition was compared in good agreement to the LGK model.

This process is considered with direct numerical simulations of flows with phase change. Tryggvason et al. [79] expanded on a multiphase flow model where different material properties in each phase were added at the phase boundaries. Juric et al. [80] used this method to observe complex dendritic structures such as liquid trapping, tip-splitting, side branching, and coarsening. This efficiently handles discontinuities in material properties between liquid and solid phases. DIT produces a method for simulating morphology that can be further defined based on tracking criteria. Front tracking explicitly provides the location of the interface at all times.

2.4. Level Set

Tracking the interface can be accomplished either explicitly or implicitly. Explicit tracking such as front tracking requires special care for topological changes [81]. Overcoming this is difficult in 3D; however, implicit representations such as LS or PF can handle these topological changes in a straightforward manner. LS represents the front as a level set of continuous functions. LS simulations are simple, which is an advantage, especially when computing the curvature. LS is similar to PF [82] as it also interchanges the interface with a field variable; however, it requires the knowledge of the direction in which the solid front is advancing (along with its velocity and calculation of the normal vector from the interface) [83].

Tan et al. [84] utilized a LS simulation that combined the features of front-tracking and fixed-domain. A domain decomposition of eight domains was performed using the Cornell CTC supercomputer [85] for 5 h, while the 3D simulation of a single dendrite required 12 h with 16 nodes (each of the nodes consisted of two 2048 MHz CPUs). They were able to prove that the method provides accurate tracking of the interfaces, computation of the heat/mass/momentum transport avoiding boundary conditions, adaptive meshing, and the capability of multiple solid phases. The research involved a reasonable mesoscale value for a mesh width of 12.2 µm for the diffused interface for a full mesh of 1024 × 2048. In another study, Tan et al. [86] modeled multiple dendrite interaction with undercooling in the front using a LS method in a 3D domain. A solidification speed of 3000 μm/s and thermal gradient of 1400 K/cm was required for nucleation and is shown in the simulation below (Figure 4).

The LS method is a helpful alternative to PF to avoid the asymptotic analysis needed for the PF method [62,87]. It is an extension of front tracking methods, where energy conservation issues derive from discretization errors. This is based on the direct application of temperature boundary conditions and the computation of heat fluxes, which was first addressed in Stefan problems [62], where the interface velocity is calculated from interpolation of the heat flux nodes near the interface. A LS method is useful for a direct calculation of the growth and shape of the solidification of dendrites without the need to apply the boundary condition explicitly at the freezing interface. Using a fixed FE grid, LS avoids the need for moving or adaptive gridding while providing an explicit and accurate tracking of the interface front [88]. Gibou et al. [89] provide a technical analysis of the LS methods and their applications in the computational review, which provides insight to adaptive Cartesian grids and parallel architectures.

2.5. Dendritic Needle Network

DNN is another novel method that is featured with respect to simulating the complex structure of dendrites. A mesoscale simulation resolution is needed for this, as each branch of the dendritic grain is considered a thin needle crystal, modeled as a network. PF and CA simulations are used to predict the dynamics of the individual branches for solidification or complex dendritic networks at a scale much larger than the diffusion length. Tourret and Karma [90] created a 2D multiscale model for DNN for both isothermal and directional solidification that was validated through the comparison with analytic solutions. This DNN approach allows for investigations into the dynamics and stability of spatially extended dendritic arrays. The optimization of this model was proven to be four times faster than normal PF simulations. A 3D DNN simulation was scaled from 2D by defining the flux intensity factor (FIF), which is the strength of the concentration gradient at the tip of the needle. This is comparable to calculating the stress factor of a crack’s tip, which is calculated using a contour integral and modeled as a parabolic tip. The main discrepancy for a 3D analysis is caused by Laplace’s equation having no solution for a line terminating at a single point, which was accounted for by considering the solute flux intensity for a needle of finite thickness. The results were compared with the microscopy of a sample from NASA’s microgravity solidification experiments, which showed a similar characteristic scale of the array spacing. Tourret and Karma [91] elaborated on this 3D DNN model using the same parameters with isothermal and directional solidification. An Al-7 wt.% Si alloy that was modeled in the previous microgravity experiment was analyzed with more scrutiny. They utilized a new 2D formulation for thick branches with paraboloidal tips. The comparison of the 3D simulations to samples from the CET in the Solidification Processing Project (CETSOL) [92] is shown in Figure 5. The simulation results of DNN, featured in white, followed a similar primary dendritic spacing to the markers of the samples featured in purple. This research provided a basis for quantitatively predicting the full 3D microstructure spacing of individual branches, which determine the mechanical strength of the structure.

Tourret et al. [93] expanded on the DNN research by focusing on the isothermal growth of an equiaxed grain in a supersaturated liquid in 3D. Using PF, needle-based, and envelope-based approaches, benchmarks for an undercooled isothermal equiaxed growth were used to compare steady-state growth predictions. The theoretical Ivantsov solution provided the Péclet number that formed the basis of comparison, scaled with respect to the tip radius and velocity. The comparison of the model types is shown in Figure 6. While the lack of truncation radius bounding was apparent in the shapes, the coarseness of the dendrites caused the velocity to decrease due to the effect of the boundaries.

Nonetheless, this illustrates the effectiveness of DNN and grain envelope models (GEM) at reproducing PF results with reasonable accuracy. This comparison of models features an operational compromise between resolution and computational efficiency. While accuracy is important for large-scale simulations, multiscale models enable predictions of primary dendritic spacings, similar to industrial casting processes [94].

In context with PF, DNN solves a broad range of phenomena, where strong assumptions are made for the mesoscale models. Normally, these models do not account for the detailed evolution of the solid/liquid interface. DNN’s representation of the dendritic structure as an array of needles allows for both steady-state and transient growth regimes to be described while accounting for the diffusive interactions between them. This bridges the PF approach with coarse grained stochastic models. Multiscale DNN allows for convective effects to be more effectively studied. Most recently, Isensee and Tourret [95] compared the oscillatory growth behavior to X-ray in situ imaging, identifying the fundamental mechanisms in which the gravity-induced buoyancy alters the dynamics of the crystal growth. DNN is a helpful approach for scaling simulations of morphology; however, in order to predict realistic microstructural evolution, incorporating fluid flow is crucial.

To compare the simulation methods that were just described, it is essential to produce simulations of a similar nature. However, as not many simulations have been created for this purpose, an analysis of the most outstanding simulations discussed in the above sections will serve as the basis for comparison (

Table 1. Comparison of the simulation techniques.

	CA [53]	PF [74]	LS [84]	DIT [77]
Hardware	4 NVIDIA RTX 2080 Ti GPUs	TSUBAME 2.0	CTC supercomputer	single CPU
Compute time	27.42 min	288 h	5 h	11 h
Domain size (mm)	5 × 2 × 0.5	3.072 × 3.078 × 3.072	12.5 × 25.0	1.2 × 0.6
Mesh size	768 × 768 × 768	4096 × 4101 × 4096	1024 × 2048	600 × 1200

).

Monte Carlo (MC) simulations have also been widely used in studying solidification and grain growth processes as they allow scientists to model and predict how microstructures/grains evolve over time [96,97]. The Monte Carlo method involves making random changes to the system and then deciding whether to accept or reject the change based on a probability that depends on the free energy of the system [98]. This deterministic method is used to simulate grain boundaries on a large-scale, which is featured in the section for additive manufacturing (AM).

3. Mathematical Methods for Solving Transport Phenomena

The complex physics associated with the solidification process requires solving a number of differential equations for heat transfer, diffusion, fluid flow, and phase transformations. The numerical methods for solving these equations often include finite element (FE), finite difference (FD), finite volume (FV), and lattice Boltzmann (LB). The first three are compared with respect to each other due to the many similarities shared between them, while LB is featured separately. Solving the transport phenomena can be accomplished in either the mesoscale (FE, FD, FV) or microscale by using molecular dynamics (MD). Solving a large-scale problem is impossible using this MD method; however, LB bridges the gap between the two by not considering each particle’s behavior in isolation, as per MD. LB uses a distribution function to represent the behavior properties of a collection of particles.

The conventional method for the mesoscale is usually divided into two approaches, continuous or discrete. Using the continuous approach, an infinitesimal control volume and the conservation of energy, mass, and momentum are used to obtain partial differential equations, which is difficult to solve with complex geometry, boundary conditions, and nonlinearity. This problem is solved by discretizing the domain into finite elements, grids, or volumes in a macroscopic scale. Physical properties such as temperature, pressure, and velocity are represented at nodal points or averaged or assumed linearly/bilinearly across a finite volume/nodal point [99].

3.1. Finite Element/Difference/Volume

Many scholars have used FE, FD, and FV methods to solve the governing partial differential equations. Time and space are structured in a grid, where a continuous field variable is approximated at discrete grid points [100]. When extending into higher volumes, the FV methodology enables conformation to an irregularly shaped grid compared to the FD method’s cartesian grid. The FE method divides a domain into discrete units, which is distinguished by the connectivity between the nodes and is suited for irregular structures. The trade-off between resolution and computational speed is dependent on the methodology chosen.

The FE method is also used to solve the governing equations for solute transport. Feng et al. [101] used FE to implement a fluid flow model for a mesoscale simulation. Utilizing the Galerkin FE method [102], an elemental matrix was developed and solved using an open access program C++ library (IML++) to solve both systematic and nonsystematic linear systems [103]. These resources have been helpful in developing models for many uses such as quantitatively predicting the fluid flow behavior induced by solidification shrinkage. The resulting mesoscale simulations were measured for variations in permeability from the shrinkage (Figure 7). With an average grain size of 300 μm, the domain contained 8000 realistic grains. The methodology for scaling will be discussed in more detail in context with the dendrite interactions and competitive growth.

FV, on the other hand, is functionally designed to work in a 3D environment. Finite volume works as an integral scheme across an area, similar to FE, in which the chance for error is minimized. Integral schemes are computationally slower than differential schemes (FD), but this is dependent on the boundary conditions. FV is a conservative formulation that allows for a mesh to apply boundary conditions for a flow [94]. FV is effective for coupling with CA, which was used to study artificial anisotropy [104]. Much effort has been made to simplify this process in mesoscale [105], for example, by creating a framework for FV [106] for casting processes (OpenCast) or integrating with sharp solid–liquid interfaces with the Eulerian–Lagrangian framework [107]. Navier–Stokes is unable to simply combine thermodynamics with the source term, therefore, FV is a common strategy for integrating these extra physics. LB, however, can naturally inject the complex physics of combining thermodynamics with the source term into the model for applications such as phase changes [101].

Despite their challenges, finite methods have provided useful integrations for modeling fluid flow with respect to meshing. Integration was applied as a moving mesh algorithm for quantitative PF equations by Karma et al. [108]. The sharp interface equations define the dendrite evolution in terms of the diffusion equation, which is expressed in a moving frame. Li et al. [109] expanded on the moving mesh framework for 3D multiphase flows. This adaptive grid method is useful for the disparity between the relatively small thickness and global length scale, which requires a locally refined grid inside the interface. This research has inspired others to utilize these methods for greater potential. Wang et al. [110] used these adaptive FE methods to measure the scalability of these domains. Comparing the tip velocities for undercooling cases, complex dendrites were studied for the potential to simulate more realistic physical problems (with a CPU that required about 60 h of time for 47,905 nodes). This numerical technique presents a framework for multicomponent (MC) alloys. The previously stated research has its limitations with respect to computation and time. FE has been used historically to model the transient thermal stresses and residual stresses from laser heat treatments [111]. With this context, it requires a higher degree of computation for realistic models. 3D finite element models provide a model for scalability [112,113,114] while other algorithms can enable multiscale complexity. Prakash et al. [115] provide a review on melting and solidification strategies in general purpose computational fluid dynamics software, which typically employs these types of methods.

3.2. Lattice Boltzmann

The LB method is a relatively newer approach for solving the solute transport and fluid flow. It is different from conventional modeling techniques as the system is modeled as a collection of particles moving on the discrete computational space’s lattices. It is powerful for simulating both single and multiphase flows in complex geometries, which gives it an advantage over the conventional Navier–Stokes solvers [116,117,118,119]. It is very efficient in describing the fluid flow computationally [120] and can be coupled with different interface capturing methods. Interesting observations about the movement of dendrites can be made using large-scale LB models such as translation and rotation [121,122,123,124].

LB has been an important tool for our research on the large-scale simulation of dendritic solidification. Jelinek et al. [125] were able to visualize the flow of an Al-3 wt.% Cu alloy melt in 2D (Figure 8). The arrows represent the velocity vectors of the melt, while the contours represent temperature, where cooling occurs at the back and front as heat is applied at the sides. By incorporating the effects of melt convection, solute diffusion, and heat transfer, LB was able to be scaled using MPI and matched to CA using an identical mesh. LB’s simple formulation is split into two steps: collision, which is completely local, and streaming, where MPI is used to transfer the distribution functions.

LB is very suitable for parallel processing, as this method does not rely on the assembly of a large global matrix, which makes CA a natural approach for coupling. However, LB has also been coupled with PF [121,126], where the equations of motion are solved for tracking the translational and rotational motion of the solid phase. Medvedev et al. [122] proposed a mesoscopic scheme to simulate dendritic solidification with both the motion and rotation of grains, which laid the framework for larger-scale simulations. This scheme was translated into 3D for multiple solid particles by Subhedar et al. [127], which optimized the diffuse interface-flow simulations. Software such as OPENPHASE [128] utilize PF and LB methods to combine dendrite growth simulations with fluid dynamics.

LB has advantages that are clear; computations are local and easy to handle in terms of complexity. It is efficient for parallelization while handling accuracy, numerical stability, and constitutive versatility. Therefore, the transport phenomena can be computed in a variety of use cases outside of fluid flow such as reaction systems, phase changes, material processing as well as heat transfer. Typically, simple simulations are performed using LB; however, with greater computation power, LB can be utilized for more than one use-case in a single simulation such as by studying phase change in combination with heat transfer. Ren et al. [129] verified this with experimental temperature profiles. They studied the influence of the dispositive position and relative position of the adjacent component and analyzed it with the metal droplet deposition method (MDDM) under heat conduction. LB was used to predict the heat transfer and phase change in the multi-layer deposition. Sakane et al. [130] used a domain decomposition method to simulate the free growth of an equiaxed dendrite in a domain of 2 × 2 × 6 mm³. This allowed for multiple-GPU parallelization, where the boundaries moved to divide the dendrite evenly.

When the heat transfer equation is solved in large-scale simulation for most metals, since thermal diffusivity is 100–1000 times larger than the solute diffusivity, the temporal resolution required for the solute diffusion equation is too fine for the energy equation [35]. If all the numerical models for simulating microstructure growth are explicit, then separate spatial scale and temporal scales for each physics can be employed to alleviate the problem and reduce the computational time. In other words, unique grid sizes and time steps can be used for different physics. The grid size and time steps are selected through the Courant–Friedrichs–Lewy condition [131] for each physics. However, the length scales should be fine enough to capture the secondary dendritic arm spacing (SDAS) and inter-dendritic flow for fluid flow and solidification growth features. This will result in a much coarser grid for fluid flow that reduces the overall computational cost [132]. This method has been successfully applied by many researchers to reduce the computational time of large-scale simulation [34,35,132,133]. Recently, our group [133] studied the accuracy and computational efficiency of a multiple grid and time step scheme for a natural convection benchmark problem. We showed that through the appropriate selection of the grid and time step, computational savings up to tenfold could be obtained compared to when the same time step and grid size were used. The model was also accurate and only lost 9% accuracy for the worst case [133]. Chen et al. [134] presented a critical review on LB methods and applications that focused on solving multiphase problems.

4. Computational Approaches

In recent years, general-purpose computing on CPUs and GPUs with MPI and CUDA has been employed to speed up large-scale simulations of the solidification microstructure [135,136,137,138,139,140,141]. When MPI is used for parallelization, the computational domain is decomposed into finite subdomains in all directions. The information exchange between subdomains is carried through halo regions or ghost nodes. Based on the location of the subdomain, ghost nodes are either physical boundary conditions or contain neighbor subdomain boundary information. The ghost nodes are updated at each temporal iteration using MPI and the intercommunication between the subdomains [132]. The host program (CPU) divides the computational domain into thousands of thread blocks. Each thread block consists of a multiple of 32 threads. The kernel function is executed by the total number of threads in parallel in the device. It was shown that the computation time for a GPU with massive computation capacity and bandwidth is two orders of magnitude faster when compared with a serial CPU core [142].

Under certain conditions, CPUs are effective for simulations in comparison to GPUs. Sun et al. [143] characterized the accelerated performance quantitatively based on the total node points run on a single Intel Xeon E5-2699 v4 core and four NVIDIA GPUs. Figure 9 shows that at a lower number of nodes, the GPU efficiency was low. At 64³ nodes, the speedup ratio was only 8.83, while the 640³ case had a speedup of 155.29. The data communication between the devices was time consuming, while the calculation source of the GPU was largely unoccupied. The decline in speedups (1024³ nodes = 152.30) is believed to be caused by the time used for output. Furthermore, the GPU efficiency increased as the number of nodes increased. With greater development, the acceleration in GPUs can be doubled by using shared memory [142], which is located at and shared by the same block.

Using multi-GPU cellular automata models, Wang et al. [144] simulated solidification structures in continuous casting blooms, which is known to be a computationally time-consuming process. The model utilized a heterogeneous GPU-CA parallel algorithm to optimize the calculation parallelism and a multi-stream communication scheme to overlap computations and data transfers. They validated the model by the analytical LGK model and applied it to simulate the solidification structure. In comparison to single-core CPUs, the proposed model achieved speedups of 700× while maintaining similar calculation precision and producing results with minimal relative error.

The evolution of technology has given rise to a diversity in applications. Computer processing for large-scale simulations has been improved due to the number of processors. Supercomputers were the source of this amount of computation required, while CPUs were soon replaced by GPUs, which allowed for even faster calculations. Eventually, the bottlenecks for the speed were determined by the programming. Innovations in GPU technology such as NVLink, which allows for direct communication between GPUs, were assisted with toolkits such as NVIDIA’s CUDA to program the GPUs directly. TSUBAME 3.0 (the most current of the series of supercomputers based at the Tokyo Institute of Technology) employs this technology, allowing four 20 GB/s data link per GPU for a total of 47.2 PFlop performance in half-precision [145]. MPI has shown effective use throughout trials, and the combination of techniques and optimizations has been proven to show significant success in GPU-rich supercomputers.

Jelinek et al. [125] produced a large-scale simulation for 2D dendritic growth, which was accomplished through MPI parallelization. Parallel programming utilizing this type of communication between the distributed-memory systems is the standard for large-scale simulations. The programming is approached using a single program, multiple data streams (SPMD) [146]. SPMD uses each processor, executing the same program on different data for MPI processes. Using a notion of rank to distinguish processes, the point-to-point communication is the fundamental primitive for sending and receiving. The scalability enabled this type of research to utilize Oak Ridge National Laboratory’s Cray XT5 system (Kraken supercomputer) [147,148]. The dendrites were grown to a reasonable size in an “incubation region”, then the results were stored to restart using the scalability of 41,472 of a total of 112,000 cores of the Kraken supercomputer. Figure 10 shows the final snapshot of the dendrite incubation domain, where the magnified portion shows the flow of the alloy melt between solidifying dendrites.

The parallelization required for this amount of detail for such a large domain is only possible with a technique of spatial domain decomposition. This popular method of splitting the spatial domain into equally-sized domains is specific to the number of cores used for the computation. The benefit of the CA-LB model is that only the subdomain boundary values need to be exchanged between the subdomains [125]. Implementing this in binary Hierarchical Data Format 5 (HDF5) yielded a high efficiency that resulted in a 50% reduction in the memory and computational time required, enabling high scalability.

Continuing this research, Eshraghi et al. [137] utilized the CA-LB model to simulate 3D dendrite growth in a macroscale domain of approximately 36 billion grid points (1 mm³). The scale-up performance (strong scaling), where the number of processors was increased with the fixed domain size, was compared with the speed-up performance of a fixed processor load by scaling the domain size (weak scaling). Using this combination, the entire domain was filled with dendrites, as shown in Figure 11. The competition between the dendrites shows that the ones with orientations other than 90 degrees were blocked by the perpendicular dendrites.

Shimokawambe et al. [135] were able to perform the first petascale PF simulation of dendrite growth that we know to date. The process was implemented locally on a single GPU using CUDA; then using 4000 GPUs, MPI was implemented, so host CPUs were used as a bridge for the data exchange. They used overlapping techniques to utilize both GPUs and CPUs to optimize the scalability, which were defined as Hybrid-YZ and Hybrid-Y. First, Hybrid-YZ exploited the data independency for array elements by dividing each subdomain into five regions, which were computed separately. The CPU cores were used to compute four y and z boundary regions, while the GPU was used to compute inside the region to produce halo regions. Hybrid-Y instead assigned only boundary regions in the y orientation of the boundary to CPUs [146]

Hybrid techniques for computational simulations have been the focus of research since modern high-performance computing (HPC) systems have been introduced to simulations. Yamanaka et al. [142] proved that GPU-accelerated PF simulation was faster than its CPU counterpart by two orders of magnitude. Heterogeneous computing has resulted in the development of many frameworks [149,150]. These many frameworks have been developed to enhance the computation and portability of these HPC techniques. “Multiphysics Object-Oriented Simulation Environment” (MOOSE) [151], FEniCS [152], “Portable, Extensible Toolkit for Scientific Computation” (PETSc) [153], Mesoscale Microstructural Simulation Project (MMSP) [154], and Structured Adaptive Mesh Refinement Application Infrastructure (SAMRAI) [155] are a few resources that have been built upon the models previously discussed. High performance interconnection networks [156] have also optimized research regarding many of the features described below.

5. Features and Applications

There are many unique use-cases for large-scale simulations of the solidification microstructure. The original focus of initial works was to provide a large macroscale domain for the direct numerical simulation of dendrites. The simulations grew in both dimensions and scale. Stochastic-based methods of modeling [54,55] have been used to produce larger simulation domains, but researchers had to develop special dynamic allocation techniques to minimize the computation costs for the large number of cells [57]. The trade-off between 2D and 3D is important for distinct use cases. A dendrite growing in a 3D domain versus a 2D domain in the presence of convection will experience less effect of flow on its growth and morphology, which is because of the obvious reason that the moving melt can flow around the dendrite arms in 3D. We studied the comparison of natural convection and forced convection on a single dendrite evolution in 2D and 3D using LB [157]. 3D dendrites grow faster than 2D, while secondary arms are more likely to form. Jegatheesan et al. [158] studied the effects of distributing nanoparticles in a buoyancy driven convection solidification by using an enthalpy-based method. When considering methods for convection, a reduction in volume from solidification (shrinkage driven flow) was not considered as it was studied only in 2D. However, adding a third dimension can improve the nanoparticle transport due to the enhanced diffusion, which is the result of the extra simulated room for the convection. This comparison serves as a helpful example of how observations in 2D can provide effective results. The simulation also highlighted the accumulation of nanoparticles at the grain boundaries while the presence of a large number of dendrites or even nanoparticles, with or without convection, could reduce the growth of a dendrite.

In a 3D space, competitive growth can be studied in detail, which cannot be expressed in 2D. When a dendrite enters the space of another, dendritic branches can block growth based on orientation. Sakane et al. [138] studied the dendrite interactions in the directional solidification of an Al–Cu binary alloy bicrystal using 512 GPUs with 1024³ meshes. A PF-LB model was utilized with MPI to demonstrate that 3D phase field simulations could be performed within a reasonable computation time (12 h) while showing tertiary dendrite growth.

Another helpful comparison of 2D and 3D simulations of dendrite growth was accomplished by Sun et al. [143] by comparing the effectiveness of the MC PF model. In this case, a 2D simulation of Ti6Al4V was performed using Al and V solute simultaneously, which was a first of its kind. The results showed that the driving force and growth were magnified artificially in the pseudo-binary in comparison to the MC (ternary) PF model, violating the sharp interface (Figure 12). As a result of the benchmark comparisons, the MC PF model was scaled into a 3D model using a 512³ nodal box with a capillary length of 0.840 µm. Using four Tesla K80 GPUs for a total of nearly 20 k cores, the resulting simulation showed that the inhibited growth of the shortest dendrite by diluted Al (V) was merged by its surrounding dendrites. This phenomenon has been studied experimentally in AM for the incremental arm spacing and coarsening of primary dendrites.

5.1. Dendrite Interactions and Competitive Growth

Dendritic growth is dependent on many factors that can generate many different phenomena. These effects need to be understood in detail to be able to predict the changes in the microstructure. Traditionally, the Walton and Chalmers competitive growth model [159] is utilized for the selection of growing multiple dendrites. Nickel-based superalloys have shown some astonishing growth phenomena [160,161,162] that cannot be simulated using the conventional model. First, this section will highlight the traditional model, then will expand on the phenomenon and other effects associated with competitive growth.

In 2017, Yang et al. [163] used PF simulation to create a 3D dendrite growth in a nickel-based superalloy. The results were generated using a single NVIDIA GTX1080 GPU for a total of 774.292 GFLOPS (giga floating-point operations per second). From the dendrite arm spacing to inclination angles, the dendrite morphology evolution during this superalloy’s casting can be understood to optimize the mechanical properties. The simulation was accomplished using a simplified approach by assuming the alloy as a pseudo-binary alloy, which was first popularized by Raghavan in 2012 [164]. PF has the capacity to simulate both isothermal and non-isothermal dendrite growth, which makes it versatile for applications such as asynchronous concurrent GPU computations. This allowed Yang et al. to show the interactions between the dendrites in an equiaxed multi-dendrite domain, which caused the dendrite arms to grow with a deviation from their initial crystal orientations [163]. As they fill the entire domain, the dendrites coarsen and coalesce, causing the melting of the secondary arms.

The comparison was accomplished using directional grain growth, allowing the height to be competitive. This illustrates that the growth is dependent on both the inclination angle and the position of the adjacent dendrites. This simulation showed how the favorably oriented (FO) dendrites outperformed the unfavorably oriented (UO) dendrites. This simulation was performed using an asynchronous concurrent algorithm to show that 774.29 GFLOPS were obtained in 512³ computational grids on a single NVIDIA GTX1080 GPU. The growth of the dendrites obtained a height of 6.42 mm. However, in contrast to this study, the following study illustrates conditions where the opposite is true (UO outgrows FO).

Takaki et al. [165] published a study focused on simulating the competitive growth with converging grain boundaries in a large domain of 3.072 × 6.144 mm². The directional solidification of 3D dendrites was simulated to compare the effect of unfavorably oriented (UO) grain inclination angles. Using the Walton–Chalmers model [159], Takaki modeled a contrasting phenomenon where at the grain boundaries, the UO dendrites overgrew the FO dendrites. This unusual overgrowth was more common when the thickness of the sample and the UO grain inclination angle were small. The secondary arms had higher growth at boundaries. Tertiary arm growth was enlarged at the convergence with the domain boundary.

Takaki used this competitive growth model for 3D analysis, where a collision of FO and UO grains occurred in the middle of the domain to form a straight GB. The FO and UO grains shared different properties, where the arrangements of the dendrites gradually became ordered [124]. The FO grains form a hexagonal arrangement, while the UO grain migrates in a lateral direction. This occurred as the UO dendrites penetrated deeper into the FO grains with respect to the reduction in the angle of orientation. This interaction was termed as “space-to-face interaction” [124]. Figure 13 shows the convergence of the differently oriented grains with respect to different degrees of orientation. This perspective from above the columnar dendrites shows that the UO dendrites with a lesser degree of orientation had a greater convergence, which is expected from previous competitive growth patterns.

In large-scale grain growth, the Walton–Chalmers model is largely accepted as the general competitive-growth model for grain selection. In this model, FO grains block UO grains. In contrast, when analyzing primary arm spacing, Fourier transformation (FT), Voronoi decomposition, and minimum spacing tree have been employed to evaluate the arm arrays [57]. This has been the basis for new models to be developed for greater accuracy. Voronoi tessellation (VT) has been applied to approximate the morphology of equiaxed dendritic grains. The dendrites are formed from a mesoscale domain, where Voronoi grains are formed in the triangulation technique exemplified below (Figure 14) forming a polyhedral structure. These assumption models allow for complex model domains to be generated. For example, Feng et al. [101] simulated a 3D semi-solid microstructure using 1000 grains, where VT was used to approximate the final grain morphology to compare the volumetric inflow caused by shrinkage. Permeability will be discussed in more detail in a later section.

In order to evaluate the arrays, ordering, and spacing of primary arms, especially in 3D directional growth, different methods have been employed historically such as minimum spacing tree [166], Fourier transformation [167], Voronoi decomposition [168], and many others [169,170,171,172]. Using a modified Voronoi decomposition technique, where the small sides are removed from the Voronoi cells, the primary arm array was evaluated by Takaki et al. [173]. The hexagonal pattern showed typical-hepa defects that decreased with respect to time for tilted columnar dendrites in a computational domain of 1.152 × 1.152 × 0.768 mm³. These enabled predictions of the primary arm spacing, which is crucial for the integrity of the material. This was ultimately possible through a convergence analysis of the tip undercooling of the dendrite/cell.

The ability to predict dendrite spacing is important for permeability research, as the fluid flow can only be researched under certain conditions. Porosity, which is an important type of defect that can form from casting, can be caused mostly by either shrinkage, where the volume changes upon solidification with a restricted feeding of the liquid, or the condensation of dissolved gases in the melt upon freezing. This has a strong negative effect on ductility and fatigue life, where internal pores create initiation sites for cracks and local stress concentrators [100]. As convergence occurs, permeability is another condition that requires a lot of research for the consideration of realistic predictions. Li et al. [174] produced a review article on Ti–6Al–4V solidification simulations for AM, where competitive growth is featured in context to different modeling techniques.

5.2. Columnar to Equiaxed Transition

In the context of design, columnar or equiaxed grains may be desired depending on the expected properties. The transition between columnar and equiaxed has been investigated for many years, where heterogeneous nuclei are commonly used to promote equiaxed grains. An example where equiaxed grains are preferred is for processes of the direct-chill casting of aluminum alloys [175]. However, its high angle grain boundaries can reduce creep rupture life. Therefore, it is important to control the grain structures using conditions such as high thermal gradients and low growth rates. We recently performed a 3D PF simulation of CET for an Inconel 718 alloy in a domain of 0.2 × 0.1 × 0.4 mm³. A CET solidification map was created to compare the growth rates and temperature gradients for the evolution of dendrites in equiaxed, columnar, and mixed regimes, as depicted in Figure 15. A model was developed to predict the primary dendrite arm spacing (PDAS) of columnar growth in a wide range of temperature gradients, solidification rates, and initial grain sizes. This novel approach is effective for optimizing process parameters for melting and solidification on a preexisting substrate such as in AM or welding applications.

Studies to determine the mechanisms behind CET have been a popular focus for research. Hunt [176] proposed an analytical model to predict CET based on the potential for equiaxed grains to nucleate ahead of the columnar front’s undercooled region. The solid fraction was calculated using the truncated Scheil equation empirically to relate the cooling rate to the tip undercooling. Next, it was modified by Gäumann, Trivedi, and Kurz (GTK model) [177] to include the non-equilibrium effects of rapid solidification. Hunt’s model shares the same relationship for CET based on key parameters of pulling velocity, thermal gradient, and composition.

The process of predicting the CET has been demonstrated by comparing 2D (10 × 30 mm²) and 3D (10 × 30 × 2 mm³) models using a stochastic model for alloy 718 [178]. CET begins earlier in the 3D simulation because the grain growth is not confined in the third (z) direction, and there are more nucleation sites.

Dong et al. [179] showed a situation where equiaxed grains were placed ahead of the growth of the columnar front during solidification of Ni-based superalloys. This real-life situation results in casting defects such as stray grains or “tree rings” [180]. This was the precursor of the CET simulations, where it showed the gradual transition, realistic dendrites, and complex solute concentration profiles. The size of the domain was 2.5 × 4 mm² for a total of 500 × 800 cells in the domain. In another work, Dong et al. [176] focused on the solute interactions within the CET modeling. It was a unique finding that the solute interactions were strong in the secondary and tertiary arms, while a weak interaction between the solute and arms was observed in the columnar tips [1].

DNN provides a promise for describing the transient growth dynamics of higher order branches, which normally relies on strong assumptions. Geslin et al. [181] effectively simulated the CET in 2D, using DNN in a large domain with a 22.5 × 30 mm² size. Using a sharp interface model for directional solidification, they were able to observe complex phenomena such as “circular growth” (branchless growth), abrupt and progressive transitions with different grain structures. Therefore, when casting large samples in a crucible, the columnar front that progresses toward the inside of the sample is accurately simulated, where the slower solidification rates favor the nucleation of equiaxed grains [100]. Balasubramani et al. [182] reviewed the origin of equiaxed grains in solidification within the context of manufacturing treatments, which provides a greater understanding to the application of these simulations.

5.3. Solute Transport and Segregation

Solute transport is a phenomenon that is very flow dependent and can have effects on the development of the dendrites under well-defined thermal conditions. Effects on dendrite spacing and symmetry have been notably studied [183,184]. Wang et al. [55] were able to utilize a CA-FD model to simulate a controlled solute diffusion in the solidification of a binary alloy. Using a 2D model, they simulated a tertiary dendritic arm growth from a secondary dendrite arm, which was then blocked by another secondary dendrite. This was helpful in predicting the primary dendrite arm spacing. The average primary arm spacing was found to be dependent on the current growing conditions and the way those conditions were reached.

These features for prediction models serve as a helpful analysis to develop a real-life understanding of the grain interactions. From interactions of both equiaxed and directional grain growth, the competition models show a clear understanding of the evolutional interactions. Even highlighting the transition between the two phases, comparing the mesoscale interactions on a microscale level requires a manageable distribution of resources. The prediction models for spacing utilize innovative techniques that provide a new understanding of competitive grain growth, even within the same primary arm. The features for this analysis have provided the foundation for more progress to be made with more complexity in dendritic evolution. Solute plumes are a resulting situation that occur in a directional solidification of Ga–In alloys. This solute-rich liquid that flies up is believed to be a crucial factor that causes, during solidification, a freckle defect [185]. Takaki et al. [186] studied the effects of natural convection in both 2D and 3D simulations. Performing a series of simulations where gravity was changed, they were able to show that as the gravity decreased in the negative region, primary arm spacing increased. Furthermore, the downward flow enhanced the growth of secondary arms, as unstable dendrite growth is caused by a large upward flow. Figure 16 shows the differences in the dendrite tips in 3D, while 2D shows the plume and freckle-like solidification defects forming. Using PF-LB simulations on TSUMBAME 2.5, they were able to simulate a domain of 0.384 × 0.384 × 1.536 mm³ with 95 h of computation time.

The flow influences both the solute transport variations and the upstream–downstream dendritic growth. The effects of convection can limit the downstream growth, while the upstream dendrites are promoted by the convection-induced anisotropy. The solute plumes are the result of gravity assisting in the direction of the dendrite growth. The presence of convection can change both the time- and length-scales, which can result in differences in the morphologies from the purely solute transport. Our team studied the buoyancy-induced flow during the directional solidification of 3D thin domains by comparing the Al–Cu and Pb–Sn alloy systems [187]. Sn is lighter than Pb, where more solute is rejected into the melt and its concentration around the interface increases. Upward buoyant force results from the decrease in the density of the liquid mixture around the interface. This large solute boundary layer that forms with a peak at the center is shown in Figure 17 and is also referred to as a chimney. This phenomenon has the potential to be stable during the solidification process, which carries the solute up, forming a recirculating flow. This can lead to decreased growth or remelting in this region during later stages of solidification and can form a freckle defect once completely solidified.

Macrosegregation is the solute composition inhomogeneities at a macroscopic scale of a casting [188]. Heat treatment can remove these imperfections; however, spatial variations of mechanical properties can occur with respect to the nature, amount, and size of these defects. Gross compositional defects such as the formation of freckles or segregated chimneys is caused by macrosegregation. This is associated with the following causes: solidification shrinkage [189,190,191,192], natural and forced convection [193,194,195], grain movement [196], mushy deformation [195], and cavitation bubbles [197]. Fragmentation has been researched as an important phenomenon using large-scale simulations as it is responsible for grain refinement [198,199,200,201]. It is undesirable for certain manufacturing processes such as single crystal turbine blade casting [202] or AM of metastable β-Ti alloys [203,204]. Process parameters can define the formation of freckles that can be caused by remelting, where there is a sudden rise in temperature, deceleration of a growth front, or a change in flow conditions. Simulation is an effective method to study this capillary force driven process [59].

Kao et al. [132] developed a large-scale model to simulate freckle formation for the casting of Ga-25 wt.% In. Freckles form due to the remelting and fragmentation of dendrite arms by thermosolutal fluid flow, especially for alloys where the partitioned solute is lighter than the bulk fluid. Examples of such phenomena include Ni-based superalloy and Ga-25 wt.% In. The authors utilized the LB method for fluid flow and CA to simulate the solidification growth. The coupling between these two main equations is achieved by natural convection force and energy and solute transport equations at each node, so they utilized the MPI parallel algorithm to accelerate the large-scale simulation [205]. The formation of the freckles in the simulation is shown in Figure 18. The domain for the numerical model was a total of 32 × 32 × 0.16 mm³, which is equivalent to 164 million cells.

Zang et al. [185] studied a comparison of forced and natural convection simulations in both equiaxed and columnar dendrite evolution. Understanding the effects on an Al–Cu alloy using PF-LB in an AMR algorithm, they were able to illustrate some unique phenomena. Dendritic fragmentation, angulation of dendrite arms, and splitting are dendritic growth behaviors that were the subjects of interest for the forced convection simulations, where the direction and intensity of the convection had a significant influence. Fragmentation occurred regardless of orientation for columnar dendrites, but was instead dependent on convection type.

Freckle formation, also known as channel segregation, is a unique byproduct of vacuum arc remelting (VAR). The solute is ejected upward and solidifies as freckles. Research has been conducted in understanding the segregation defects on the VAR process, simulating the ingot evolution under different arc distributions [206]. However, a lot of detail can be obtained by employing large-scale simulations of the microstructure, where solute transport is a driving factor. Hecht et al. [207] provided a review on multiphase solidification, where the effects of solute transport are mentioned in context to large-scale simulations.

5.4. Natural and Forced Convection

Forced convection has been a subject of interest for large-scale simulations as it has a lot of influence on solute distribution. Research has provided context for dendritic growth and compared the effects with and without convection. Another helpful comparison is between natural convection and forced convection. Convection was originally solved using NS solutions [208]. Simulations of dendrite growth with melt flow was first possible in the 1990s using PF [209,210,211], while being limited to 2D [42,212]. Early simulations of 3D were limited to a single dendrite [213], however, it is important to scale these algorithms for complex microstructures. Yuan et al. [214] studied the effects of dimensionality on dendritic growth simulations for convection by using a modified projection method of NS. When the 2D flow has a blockage from the primary dendrite arms, the 3D flow has the ability to wrap around the primaries. Therefore, it demonstrated that 3D simulations are necessary to correctly predict unconstrained solidification microstructures. Forced convection was studied by Jakhar et al. [215] in combination with thermal isotropy, where pressure fields were solved using the SIMPLER algorithm. The model was extended to multi-dendritic simulations with random distributions and orientations in order to study microstructural evolution. Takaki et al. [186] performed a large-scale PF and LB simulation (0.384 × 0.384 × 1.536 mm³) to study the effects of natural convection during directional solidification. Comparing 2D and 3D, the effects of gravity were smaller for 3D, while the average primary arm spacing increased as gravity decreased, similar for both. Downward flow enhanced the growth of secondary arms, while upward flow, larger than a critical value, could produce plumes and freckle-like solidification defects. Using the TSUBAME 2.5 supercomputer, the computation took about 95 h for 1.5 × 10⁶ computation steps in 3D.

Computational cost has been a limiting factor for research on convection, where LB is the most efficient for multi-GPU computation as the growth is able to be simulated with the solid motion, liquid flow, collision, and coalescence of multiple solids, and subsequent grain growth. However, much effort has been made to enhance this ability. Sakane et al. [216] created a 2D simulation for a large number of dendrites (350) utilizing PF and LB methods. Figure 19 shows the evolution of the solute concentration and flow velocities, where a sedimentation path formed from the dendrites. Assuming inelastic collisions, the coalescence of the grains was observed. The nuclei generated at the top of the domain settled downward while growing equixially. The ability to scale performance was widely studied in this research, utilizing active parameter tracking (APT) [217,218]. Tracking the execution time with and without APT was compared to grains with and without motion, proving the efficiency of the model, where APT excels with convection. The parallel efficiency of the model showed that a simulation could be performed in a 64x larger domain with only twice the amount of time compared to a single GPU. The simulation was performed with 2048 × 2048 grid points and five grains per one GPU. APT algorithms were employed to simulate coalescence-free grain growth within a reasonable computation time [219].

Zhang et al. [220] combined the PF and LB with a parallel AMR algorithm for several studies of convection, both natural and forced, in 2D and 3D. They quantified the effect of both convection and undercooling by comparing the length ratio of dendrite arms. The investigation showed that as the undercooling decreased and the effect of convection increased, the length ratio had a peak value. This was due to the decrease in the crystal size compared to a higher undercooling. Second, the effect of gravity with a lateral force of convection was studied, where columnar dendrites grew anti-parallel to gravity. The accumulation of the solute from gravity stunted the growth; however, with the convection, the primary trunks of the dendrites showed a constant deflection angle until reaching a critical value. Sun et al. [221] used a 2D LB model to show that asymmetrical dendrites grew faster in an upstream direction, while slower downstream. Studies like this can help to optimize the solidification conditions for manufacturing by analyzing the effects of convection on dendrite growth.

5.5. Permeability

Permeability predictions have produced an effective method to analyze the distinct solidification conditions of the liquid flow through a mushy zone. Anisotropic porous media use Darcy’s law, which is derived from the Navier–Stokes equation using an averaging procedure [222]. Interfacial stresses occur when the solid fraction is high enough for the solid to form a continuous structure. In a mesoscale domain, the solute distribution is limited by the permeability of the solidification structure, with consequences for grain refinement [223]. 3D interdendritic flow simulations have been performed using microtomography mappings to measure the permeability in Al–Cu [224]. However, the compromise between resolution and sample size is limited to the camera. Therefore, simulations have provided a solution with parallel programming, simulating both the morphology and the fluid flow. The complexity of the transport phenomena has resulted in interesting studies in large-scale such as cross-permeability [225], where experiments measuring permeability have a limitation of experimental volume fraction [226].

Permeability research has been characterized both numerically and experimentally. Permeability simulations have been validated in hypoeutectic aluminum alloys by Khajeh et al. [227], where the simulated microstructures for a dendritic network were modeled using the CA technique. 3D printing has been a helpful tool to perform experimental measurements from a simulated model. Most recently, Berger et al. [228] used a fused filament fabrication (FFF) technique to scale a PF model from 150 µm³ to 5 cm³ of an Al–Si–Mg alloy sample with a fraction solid varying from 0.61 up to 0.91.

Validation of these models requires more detail to ensure that the correct physics is being modeled [180]. Mitsuyama et al. [229] performed an analysis on the permeability of a large domain of 1.152 × 1.152 × 0.768 mm³ using PF and LB, and approximated using the Kozeny–Caman (KC) equation [230], which is used most frequently to express permeability. Alternatively, the Poirier–Heinrich (PH) [231] equation can be used, but with uncertain accuracy due to its derivation from 2D simulations and experimental data [232]. Therefore, a lot of research has gone into validating these models. Such discrepancy has invoked studies based on different types of flow and growth patterns. In parallel flow, a solid fraction can change in dimensions with respect to the liquid that flows through the entire columnar dendrites, as shown in Figure 20. The simulations validated the use of a KC coefficient of ${\mathrm{K}}_{\mathrm{c}}=3$ and the permeability tensor for this use case. Thus, the permeabilities in arbitrary directions for columnar solidification structures can be calculated without simulation. The main area of interest regards the specific interface areas and the temporal changes, described by the solid fraction.

While approximation for equiaxed dendrites is effective, the KC equation is not as certain for directional solidification [233,234]. Takaki et al. [235] utilized a new permeability prediction method [209,225,232,236,237] using a parallel process of GPUs. They performed an analysis on permeability for columnar solidification structures with a periodic regular hexagonal array simulated using PF and LB. They were able to develop smooth variations in the qualities of solidification morphologies, where the permeability was shown to be independent of the array ordering of a consistent primary arm structure. This dimensionless permeability for a specific interface area can be attributed to the parallel GPU computing that performed this large-scale simulation. This reiterates the importance of large-scale simulation to study what is normally not possible with experiments.

5.6. Additive Manufacturing Applications

The study of AM is an important application of large-scale simulations where the solidification microstructure and related phenomena can significantly alter the material properties [238,239]. The porosity, propagation of cracks, or precipitation of second phases can have unique effects on the mechanical properties of the material. Many factors can contribute to this, and this section elaborates on the different types of AM methods and studies that produce advances in our understanding of these processes. The molten pool is an interesting area of focus, where different dendrite morphologies can be obtained by controlling the thermal gradient and cooling rates. As an example, a FE-CA model was used by Yin et al. [240] to simulate dendritic growth in the molten pool of the laser engineered net shaping (LENS) process to study the laser moving speed, layer thickness, and substrate size. Comparing the simulations to experimental results is an effective measure for success. Yu et al. [241] researched a multigrid CA model to simulate these properties from an electron beam selective melting (EBSM) of a Ni-based superalloy, Inconel 718. Using the experimental data, the growth of tertiary dendrite arms was validated. This thermal-fluid model was compared to the experimental results of single-track scans, as shown in Figure 21. By studying how primary dendrite arms grow in the melt pool at the mesoscale, this research provided a promising approach for studying shrinkage porosity and the propagation of hot cracking.

The complexity of AM requires many components to efficiently model the processes. An interesting aspect of AM processes is the potential for location-specific microstructure control. Shi et al. [242] studied the effect of laser beam shaping on the morphology, size, and crystallographic texture for the laser powder bed fusion (L-PBF) of stainless steel. First, they used a process modeling code, ALE3D, for solving the continuity, momentum, and energy equations, which was developed at Lawrence Livermore National Laboratory using a hybrid finite element and finite volume formulation [243]. The output of the ALE3D was the temperature field for all nodes through time. The transient temperature field was then imported to a simplified CA model to capture the grain structure. The ALE3D solver needed a coarser grid compared to the CA model. Therefore, the output temperature field from the ALE3D solver was projected on a finer CA mesh. They utilized the DREAM 3D software [244] and experimental measurements for the initial grain structure required in the CA model.

The properties of AM builds are highly dependent on the process parameters. Lian et al. [245] proposed a 3D CA-FV method to study the process parameters including laser scan speed and laser power to predict the grain structure for the single track directed energy deposition (DED) AM process of the Inconel 718 alloy. They also presented the grain growth for a multiple-layer deposition process with different raster patterns. The comparison of the 3D simulation results with electron backscattered diffraction (EBSD) and pole figure experimental results are shown in Figure 22. In their proposed method, the cellular automaton method, enriched with a grain nucleation scheme, was used to predict columnar, equiaxed, and mixed grains, while the FV method was used to solve heat transfer and thermocapillary flow.

L-PBF is the most popular process for manufacturing functional parts for different applications [138,157,158,159]. The simulation of L-PBF requires considering localized phenomena that are highly transient, making the simulation very complex. Marangoni convection in the melt pool, rapid solidification, topological depression of the melt, and thermal cycling are examples of such complex phenomena [242]. Elahi et al. [246] recently presented a computational framework for this type of simulation by using a combination of CALPHAD calculations for alloy properties, macroscale FE thermal simulations, and microscopic PF models for the melt pool solidification. They were able to calculate a billion grid points on a single cluster node of eight GPUs, providing insight into the grain texture selection with the details of dendrites for a realistic multiscale SLM simulation.

The large-scale simulations have enabled innovative melting strategies for AM such as localized melt-scanning to control the grain size and spacing of the primary dendrite arms. Raghavan et al. [247] produced predictions for grain sizing for a corresponding qualitative texture plot. This process allows for a consistent solidification microstructure across the build. By comparing the experimental results with the simulations, context for the types of microstructures was validated with the types of melt strategies.

Kinetic Monte Carlo [248] has been used to simulate complex, non-traditional geometries, which are observed in AM processes. 3D grain structures have been simulated under diverse experimental conditions by Rogers et al. [249]. The open-source Stochastic Parallel Particle Kinetic Simulator (SPPARKS) distribution [250] was built to scale using MPI and a spatial-decomposition of the simulation domain. The 3D simulation below (Figure 23) was both quantitatively and qualitatively compared to the experimental results of AM processes such as LENS and L-PBF to study the temperature dependent grain boundary mobility and molten zones, respectively, which were both in good agreement.

The research by Rogers et al. provides a comprehensive comparison of AM microstructure simulation methods such as CAFE [57,251], CA-LB [252], Monte Carlo [253,254], and empirical microstructure models [255]. In comparison, the CA models have a higher computation cost requirement. In addition, they are limited to a few passes of a heat source, which currently does not simulate solid-state grain evolution after solidification. On the other hand, Monte Carlo does not allow for the direct coupling of thermal and microstructural models or the incorporation of the material texture/anisotropy. However, Monte Carlo is open source in comparison, and utilizes idealized molten zones without the need to parameterize for specific material systems.

In their comparative study of phase field (PF) and cellular automaton (CA) computational models, Elahi et al. [256] sought to forecast the microstructure evolution during solidification in AM. The researchers discovered that PF simulations were superior in terms of accurately capturing microscopic features, whereas the CA simulations necessitated grid refinement to achieve a partial congruence with PF outcomes. Despite certain discrepancies, the average grain distributions drawn from various simulations demonstrated a satisfactory concurrence between the PF and CA models. However, the researchers identified differences between PF and CA in aspects such as transient growth regimes and the morphology of the solid–liquid interface. PF simulations depicted an initial period of near-planar solidification, which then destabilized morphologically into cells or dendrites, an aspect overlooked in CA simulations. Additionally, the PF simulations accounted for the dynamics of side-branching post destabilization, leading to the removal of less favorably oriented grains, a phenomenon absent in CA simulations, which, in contrast, yielded smoother grain boundaries compared to those obtained from the PF simulations.

The potential for simulating AM is ever growing, as are the processing parameters. Because of AM’s complex geometries, many novel techniques have been applied to determine the most optimal conditions. New simulation approaches are necessary to represent the performance and validate experiments.

6. Future Developments

The progression of large-scale simulations of the solidification microstructure has shown that the factors for realistic results are constantly improving. In prediction analysis, the complexity has grown with respect to technological advances. An interesting concept of utilizing machine learning for simulating grain growth has a lot more potential for future development. In 2017, Hu et al. [257] used a CA method, along with machine learning, to simulate the grain and pore growth in aluminum alloys. The back propagation neural network (BPNN) was used to create a correlation between the solidification parameters and pore growth; however, the domain was restricted to a size of 200 × 200 µm. Machine learning has the unique potential to assist in the modeling of dendritic features and has been implemented in the prediction of secondary arm spacing in aluminum alloys [258]. While deep learning has been used for the prediction of porosity defects in aluminum alloys [259], applications with simulations have been limited. Most recently, Hu et al. [260] used recurrent neural networks to accelerate PF predictions. By comparing different dimensionality-reduction methods such as linear (principal component analysis (PCA) [261]) and nonlinear embedding (isometric feature mapping (Isomap)) [262] and uniform manifold approximation and projection (UMAP) [263] techniques, the latent space can preserve the PF input parameters. The autocorrelation-based PCA proved to be the most efficient, while a computation speedup of 3x was able to be implemented using recurrent neural network (RNN) models with a fewer number of cells and a gating mechanism such as the gated recurrent unit (GRU) [264] or long short-term memory (LSTM) [265]. Figure 24 shows the implementation of the RNN model. Future development shows promise with the integration of mesoscale simulation systems.

Another potential area for higher development is the complexity of remelting in larger domains. The evolution of dendrite remelting consists of four stages, while the last stage consists of fragmentation [266], which enables the further discretion of large-scale realistic simulations. An interesting phenomenon, where secondary arms remelt at the roots, detaching from the primary trunk, and moving toward unsolidified melt can be better understood in larger domains [267]. The combination of improved physics models and computational algorithms with thermodynamic databases would enable the calculation of multicomponent phase equilibria, allowing for more reliable simulations for real-world industrial applications.

7. Conclusions

The applications of simulating the evolution of the microstructure during solidification are as diverse as the observable features. The large-scale domain requires extensive computing power; therefore, a combination of hardware and numerical techniques are necessary to enable scaling. Employing the most efficient numerical methods together with communication techniques over a large number of processing units, the research into solidification simulations has resulted in innovative methodologies, enabling the investigation of various features during alloy solidification.

To enable efficient simulation of the solidification microstructure, various numerical methods have been developed to improve the simulation results over larger domains. Cellular automaton (CA), with its simplicity and computational efficiency, has been featured to simulate dendrite growth in large 2D and 3D domains. Phase field (PF), being the most popular, has been utilized to reproduce the physics more accurately. Although more computationally demanding compared to CA, PF has been used in the largest dendrite growth simulation to this date. Direct interface tracking (DIT) and level set (LS) methods are less popular for large-scale simulations. However, they provide a direct method to handle the sharp interface front, avoiding PF’s asymptotic analysis, where techniques of adaptive mesh refinement (AMR) and parallelization have become a drawback in some situations. Scaling with a dendrite needle network (DNN) method allows for analytical and coarse-grained models to be used in combination, which can serve as a useful multiscale modeling technique.

Solving the transport phenomena is necessary for realistic simulations with a trade-off in complexity. Solving the differential equation can be integral to the performance of the simulation. While finite element, finite difference, and finite volume have been conventionally used to solve the heat transfer, diffusion, and fluid flow equations, the lattice Boltzmann method offers an alternative local method to reduce the computation cost, especially for the case of fluid flow. However, there are many trade-offs between the methods of simulating the transport phenomenon.

Many features were discussed in detail of increasing complexity. Competitive growth and dendrite interactions are better understood in large-scale simulations, which can result in a more realistic interpretation for columnar to equiaxed transition (CET). Simulating the solute transport can provide a deeper understanding of segregation, which can be greatly influenced by convection, both natural and forced. The mushy zone permeability is another area that can greatly benefit from large-scale parallel simulations. Additive manufacturing (AM) processes often involve complex physics, which can be influenced by processing parameters specific to the simulation, demanding higher computational power.

Several manufacturing and materials processing techniques can take advantage of the prediction capabilities offered by large-scale simulations of the solidification microstructure. The modeling approaches still have a lot of future improvements to innovate upon, with promising developments in machine learning and computing power.