# Improvement in Runtime Speed for Frequency Domain Soil–Structure Interaction Analysis Using a Coarray Parallel Processing Technique

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Coarray Parallel Processing Technique

- Step 1. Assign frequency points to be analyzed in each image.
- Step 2. Run all images simultaneously to construct element matrices of infinite elements and effective seismic loads (OpenMP+CAF part).
- Step 3. Construct system matrices by running a single image and save them to the hard drive frequency by frequency.
- Step 4. Run all images simultaneously for the pre-assigned frequency points to solve the matrix equation and save the solution to the hard drive (OpenMP+CAF part). At this time, OpenMP is used to run on each image.
- Step 5. Construct frequency domain responses by running a single image, which is done by collecting and interpolating solutions at the frequency points for further post-processing.

## 3. Numerical Evaluation

## 4. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Conflicts of Interest

## References

- ASCE/SEI 4-16; Seismic Analysis of Safety-Related Nuclear Structures. American Society of Civil Engineers: Reston, VA, USA, 2017.
- Wolf, J.P. Dynamic Soil Structure Interaction Analysis; Prentice-Hall Inc.: Hoboken, NJ, USA, 1985. [Google Scholar]
- Wong, H.; Luco, J. Dynamic interaction between rigid foundations in a layered half-space. Soil Dyn Earthq Eng
**1986**, 5, 149–158. [Google Scholar] [CrossRef] - Lysmer, J.; Tabatabaie, M.; Tajirian, F.; Vahdani, S.; Ostadan, F. SASSI: A System for Analysis of Soil-Structure Interaction; Dept Civil Eng, University of California: Berkeley, CA, USA, 1981. [Google Scholar]
- Bechtel National Inc. User’s Manual for SASSI2010, Version 1.0; Bechtel National Inc.: Reston, VA, USA, 2011. [Google Scholar]
- Ghiocel, D.M. User’s Manual for ACS SASSI—An Advanced Computational Software for 3D Dynamic Analysis Including Soil-Structure In-teraction, version 3.0; Ghiocel Predictive Technologies Inc.: Pittsford, NY, USA, 2016. [Google Scholar]
- MTR & Associates. System for Analysis of Soil-Structure Interaction—User’s Manual; MTR & Associates: Lafayette, CA, USA, 2016. [Google Scholar]
- Berenger, J.P. A perfectly matched layer for the absorption of electromagnetic waves. J. Comput. Phys.
**1994**, 114, 185–200. [Google Scholar] [CrossRef] - Basu, U.; Chopra, A.K. Perfectly matched layers for transient elastodynamics of unbounded domains. Int. J. Numer. Methods Eng.
**2004**, 59, 1039–1074. [Google Scholar] [CrossRef] - Guddati, M.N.; Tassoulas, J.L. Continued-fraction absorbing boundary conditions for the wave equation. J. Comput. Acoust.
**2000**, 8, 139–156. [Google Scholar] [CrossRef] - Bolisetti, C.; Whittaker, A.S.; Coleman, J.L. Linear and nonlinear soil-structure interaction analysis of buildings and safety-related nuclear structures. Soil Dyn. Earthq. Eng.
**2018**, 107, 218–233. [Google Scholar] [CrossRef] - OpenMP. OpenMP Application Programming Interface, Version 5.1; OpenMP: Antwerp, Belgium, 2020. [Google Scholar]
- Message Passing Interface Forum. MPI: A Message-Passing Interface Standard, Version 3.1; Message Passing Interface Forum: Dallas, TX, USA, 2015. [Google Scholar]
- Zhao, B.; Liu, Y.; Goh, S.; Lee, F. Parallel finite element analysis of seismic soil structure interaction using a PC cluster. Comput. Geotech.
**2016**, 80, 167–177. [Google Scholar] [CrossRef] - Shterenlikht, A. Parallel Programming with Fortran 2008 and 2018 Coarrays; Mech Eng Dept, The University of Bristol: Bristol, UK, 2018. [Google Scholar]
- Numrich, R.W.; Reid, J. Co-array Fortran for parallel programming. ACM SIGPLAN Fortran Forum
**1998**, 17, 1–31. [Google Scholar] [CrossRef] - Reid, J.K.; Numrich, R.W. Co-Arrays in the Next FORTRAN Standard. J. Sci. Program.
**2007**, 15, 9–26. [Google Scholar] [CrossRef] - DO CONCURRENT isn’t necessarily concurrent--LLVM Flang. Available online: https://flang.llvm.org/docs/DoConcurrent.html (accessed on 2 February 2023).
- Sharma, A.; Moulitsas, I. MPI to Coarray Fortran: Experiences with a CFD Solver for Unstructured Meshes. Sci. Program.
**2017**, 2017, 1–12. [Google Scholar] [CrossRef] - Tracy, F.T.; Oppe, T.C.; Corcoran, M.K. A comparison of MPI and co-array Fortran for large finite element variably saturated flow simulations. Scalable Comput. Pract. Exp.
**2018**, 19, 423–432. [Google Scholar] [CrossRef] - Metcalf, M.; Reid, J.; Cohen, M. Modern Fortran Explained: Incorporating Fortran 2018; Oxford University Press: Oxford, UK, 2018. [Google Scholar]
- Yang, S.-C.; Yun, C.-B. Axisymmetric infinite elements for soil-structure interaction analysis. Eng. Struct.
**1992**, 14, 361–370. [Google Scholar] [CrossRef] - Yun, C.-B.; Kim, J.-M.; Hyun, C.-H. Axisymmetric elastodynamic infinite elements for multi-layered half-space. Int. J. Numer. Methods Eng.
**1995**, 38, 3723–3743. [Google Scholar] [CrossRef] - Yun, C.-B.; Chang, S.-H.; Seo, C.-G.; Kim, J.-M. Dynamic infinite elements for soil-structure interaction analysis in a layered soil medium. Int. J. Struct. Stab. Dyn.
**2007**, 7, 693–713. [Google Scholar] [CrossRef] - Ryu, J.-S.; Seo, C.-G.; Kim, J.-M.; Yun, C.-B. Seismic response analysis of soil–structure interactive system using a coupled three-dimensional FE–IE method. Nucl. Eng. Des.
**2010**, 240, 1949–1966. [Google Scholar] [CrossRef] - Seo, C.-G.; Kim, J.-M. KIESSI program for 3-D soil-structure interaction analysis. Comput. Struct. Eng.
**2012**, 25, 77–83. [Google Scholar] - Amdahl, G.M. Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities. In Proceedings of the Spring Joint Computer Conference, Atlantic City, NJ, USA, 18–20 April 1967; pp. 483–485. [Google Scholar]

**Figure 1.**Parallel processing example using OpenMP and CAF: (

**a**) Shared memory of OpenMP; (

**b**) Distributed memory of CAF.

**Figure 2.**Modeling of soil-structure interaction system using KIESSI-3D program utilizing finite and dynamic infinite elements: (

**a**) Modeling concept; (

**b**) Modeling examples.

**Figure 4.**Conceptual diagram of the proposed KIESSI-3D with CAF incorporated (e.g. with a three coarray, 12-core computer system): Orange boxes represent accelerated parallelization by CAF combined with OpenMP, gray boxes represent parallelization using OpenMP only. NF denotes the number of frequency points to be analyzed.

**Figure 5.**Numerical examples for evaluating the performance of the proposed CAF parallelization technique for frequency domain soil–structure interaction analysis: (

**a**) Shallow foundation (number of nodes = 22,690); (

**b**) Deep foundation (number of nodes = 45,137).

**Figure 6.**Speedup ratios for OpenMP and OpenMP+CAF for the shallow foundation example using various multi-core computer systems (abbreviation: ca{${n}_{1}$}nc{${n}_{2}$ }; ${n}_{1}$ = number of coarrays; ${n}_{2}$ = number of cores/image): (

**a**) Single-CPU computer systems; (

**b**) Dual-CPU computer systems.

**Figure 7.**Speedup ratios of OpenMP and OpenMP+CAF for the deep foundation example using various multi-core computer systems (abbreviation: ca{${n}_{1}$}nc{${n}_{2}$ }; ${n}_{1}$ = number of coarrays; ${n}_{2}$ = number of cores/image): (

**a**) Single-CPU computer systems; (

**b**) Dual-CPU computer systems.

**Figure 8.**Maximum speedup ratios for OpenMP+CAF and OpenMP using dual-CPU computer systems: (

**a**) Shallow foundation example; (

**b**) Deep foundation example.

**Figure 10.**Parallelization ratio P achieved by the OpenMP+CAF version using dual-CPU computer systems: (

**a**) Shallow foundation example; (

**b**) Deep foundation example.

**Table 1.**Comparison of features of the parallel processing methods [15].

Parallel Method/Language | Coarray Fortran (CAF) | DO CONCURRENT | OpenMP | MPI |
---|---|---|---|---|

Fortran standard | Yes | Yes | No | No |

Shared memory | Yes | Possibly | Yes | Yes |

Distributed memory | Yes | Possibly | No | Yes |

Ease of use | Easy | Easy | Easy | Hard |

Flexibility | High | Poor | Limited | High |

Incremental improvement | Yes | Yes | Yes | No |

Performance | High | Poor | Limited | High |

PC Identification | Number of CPUs | Cores/CPU | CPU Model (Clock Speed) | RAM (GB) | Runtime for 45 Frequencies with Ca1nc1 Option (Minutes) | |
---|---|---|---|---|---|---|

Shallow Foundation | Deep Foundation | |||||

PC-1 | 1 | 8 | Intel Xeon W-2245 (3.9 GHz) | 256 | 11.7 | 31.9 |

PC-2 | 1 | 16 | Intel i9-9960X (3.1 GHz) | 128 | 12.7 | 34.6 |

PC-3 | 2 | 6 | Intel Xeon X5690 (3.46 GHz) | 64 | 47.6 | 141.4 |

PC-4 | 2 | 8 | Intel Xeon E5-2687w (3.1 GHz) | 192 | 19.2 | 58.5 |

PC-5 | 2 | 12 | Intel Xeon E5-2687w v4 (3.0 GHz) | 192 | 20.1 | 54.0 |

PC-6 | 2 | 16 | Intel Xeon Gold 6142 (2.6 GHz) | 384 | 13.6 | 37.0 |

PC-7 | 2 | 24 | Intel Xeon Gold 6248R (3.0 GHz) | 768 | 12.4 | 34.4 |

PC-8 | 2 | 32 | AMD EPYC 7601 (2.2 GHz) | 384 | 62.2 | 121.3 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Kim, J.-M.; Lim, J.-S.; Lee, H.-J.
Improvement in Runtime Speed for Frequency Domain Soil–Structure Interaction Analysis Using a Coarray Parallel Processing Technique. *Appl. Sci.* **2023**, *13*, 2356.
https://doi.org/10.3390/app13042356

**AMA Style**

Kim J-M, Lim J-S, Lee H-J.
Improvement in Runtime Speed for Frequency Domain Soil–Structure Interaction Analysis Using a Coarray Parallel Processing Technique. *Applied Sciences*. 2023; 13(4):2356.
https://doi.org/10.3390/app13042356

**Chicago/Turabian Style**

Kim, Jae-Min, Jae-Sung Lim, and Hyeok-Ju Lee.
2023. "Improvement in Runtime Speed for Frequency Domain Soil–Structure Interaction Analysis Using a Coarray Parallel Processing Technique" *Applied Sciences* 13, no. 4: 2356.
https://doi.org/10.3390/app13042356