A New Parallel Code Based on a Simple Piecewise Parabolic Method for Numerical Modeling of Colliding Flows in Relativistic Hydrodynamics

Abstract

A new parallel code based on models of special relativistic hydrodynamics is presented for describing interacting flows. A new highly accurate numerical method is considered and verified. A parallel implementation of the method by means of Coarray Fortran technology and its efficiency are described in detail. The code scalability is 92% on a cluster with Intel Xeon 6248R NKS-1P with 192 Coarray Fortran images. Different interacting relativistic flows are considered as astrophysical applications.

1. Introduction

Many astrophysical phenomena proceed with relativistic speeds. Examples are relativistic jets in active galactic nuclei [1,2] or Herbig–Haro objects [3], collapse of massive stars [4], black holes [5,6,7], neutron stars [8,9,10], and gravitational waves [11,12]. Of particular interest are interacting relativistic flows of matter [13,14,15]. Such processes are observed well with modern instruments [16,17,18,19], and observations are used to construct models [20] and perform laboratory experiments [21,22]. Important tools for research are provided by mathematical modeling with supercomputers [23].

Godunov’s method, which was developed almost 70 years ago [24], is used in most software codes to simulate hydrodynamic (in particular, relativistic) flows. It has been studied in detail for convergence [25] and the behavior of entropy [26,27]. It has been modified to take into account the transfer of matter [28] and extended to solve equations of relativistic hydrodynamics [29], in particular for magnetic field flows [30,31]. Almost immediately after the invention of Godunov’s method, works began to appear on simplified solutions of the Riemann problem (for instance, Rusanov’s scheme [32]) and decreased dissipation methods (for instance, Kolgan’s scheme [33]). In 2000, a scheme called MUSCL was proposed [34]. It combines the advantages of the Rusanov and Kolgan schemes. The next step in decreasing the dissipation of Godunov-type numerical methods was the use of piecewise parabolic [35,36] and piecewise cubic [37] presentations of solutions. A piecewise cubic presentation of the solution was extended by the authors to HLL-type methods [38] and an operator separation method [39]. The separation of operators and the use of a combination of Godunov- and Rusanov-type schemes made it possible to formulate efficient numerical methods [40,41] and solve the problem of sound point in Godunov’s method with linearized discontinuity breakdown [42]. There is a tendency towards increasing the order of the polynomial for presenting the physical variables to be reconstructed in Godunov’s scheme [43]). Nevertheless, there are many original works on using Kolgan’s approach to decrease the dissipation in solving the Riemann problem [44,45]. The present paper is essentially an extension of paper [42], using a new numerical method for solving the equations of special relativistic hydrodynamics and adding the degenerate Rusanov method with velocities limited to the speed of sound [46]. This approach will allow simulating relativistic flows in highly rarefied zones, which are often present in relativistic jets [47].

One of the major advantages of the MPI 3.0 standard is an efficient implementation of one-way communications, which allows using one of the most promising parallel programming models, called Partitioned Global Address Space (PGAS). Computational experiments with continuum mechanics models have shown [48,49,50] that the code obtained by the authors using Coarray Fortran is as efficient as the MPI code. At the same time, this code is much less complicated, which allows the development of new libraries [51] and language extensions [52]. Note that the multidimensional decomposition of computations in the code has practically no effect on its efficiency [53,54].

The limitations of existing solutions in terms of reproducing individual relativistic gas flows, which are primarily associated with the numerical technique used, as well as the lack of codes that use all the advantages of one-way network communications implemented in MPI 3.0 and especially in Coarray Fortran, show us the need to develop a new numerical code for modeling relativistic gas flows. The implementation priority is Coarray Fortran for us, since its use significantly simplifies the code in terms of parallel implementation, which will simplify its implementation. Section 2 provides a description of a new numerical method for solving the equations of special relativistic hydrodynamics. Section 3 is devoted to a parallel implementation of the method. In Section 4, the numerical method is verified on standard tests. Section 5 presents two astrophysical applications. Section 6 provides conclusions to the paper.

2. Numerical Method

Let us first introduce the physical variables: $\rho$—density, $\overrightarrow{v}$—velocity vector, and p—pressure. For definiteness, we take the speed of light $c\equiv 1$. In this case the Lorentz factor $\mathrm{\Gamma }$ is defined by the formula

$$\mathrm{\Gamma }=\frac{1}{\sqrt{1-{\left(v/c\right)}^2}}=\frac{1}{\sqrt{1-v^2}}.$$

(1)

The condition imposed on the speed of light ensures that the velocity modulus is limited by unity. This fact will be used in the construction of the numerical scheme. In the code, we will use an ideal gas model whose equation of state is written in the form of an equation for special enthalpy h:

$$h=1+\frac{\gamma }{\gamma -1}\frac{p}{\rho },$$

(2)

where $\gamma$ is the adiabatic index. The speed of sound $c_s$ is defined by the formula

$$c_s^2=\frac{\gamma p}{\rho h}.$$

(3)

To write the equations of special relativistic hydrodynamics, we introduce the following conservative variables: relativistic density $D=\mathrm{\Gamma }\rho$, relativistic momentum $M_j={\mathrm{\Gamma }}^2\rho h{v}_j$, where $v_j$ are components of the velocity vector $\overrightarrow{v}$ at $j=x,y,z$, and total relativistic energy $E={\mathrm{\Gamma }}^2\rho h-p$. We write the system of equations in vector conservative form from classic book [55]:  

$$\frac{\partial }{\partial t}\left(\begin{array}{c}\mathrm{\Gamma }\rho \\ {\mathrm{\Gamma }}^2\rho h{v}_j\\ {\mathrm{\Gamma }}^2\rho h-p\end{array}\right)+\sum _{k=1}^3\frac{\partial }{\partial x_k}\left(\begin{array}{c}\rho \mathrm{\Gamma }{v}_k\\ \rho h{\mathrm{\Gamma }}^2{v}_j{v}_k+p{\delta }_{jk}\\ ({\mathrm{\Gamma }}^2\rho h-p)v_k+p{v}_k\end{array}\right)=0,$$

(4)

where ${\delta }_{jk}$ is the Kronecker symbol. The conservative variables are easily calculated from the physical variables, but for the inverse transformation the following nonlinear equation has to be solved:

$$f\left(p\right)={\mathrm{\Gamma }}^2\rho h-p-E=0,$$

(5)

For this, Newton’s iterative method is used:

$$p_{m+1}=p_m-\frac{f\left(p_m\right)}{f^{\prime }\left(p_m\right)},$$

(6)

where the derivative of the function f is

$$f^{\prime }\left(p\right)=\frac{\gamma }{\gamma -1}{\mathrm{\Gamma }}^2-\frac{M^2{\mathrm{\Gamma }}^3}{{\left(E+p\right)}^3}\left(D+2\frac{\gamma }{\gamma -1}p\mathrm{\Gamma }\right)-1.$$

(7)

When a given accuracy is reached, the iterative process is terminated. The value from the previous time-step is used as the initial pressure approximation.

The system of Equation (4) can be rewritten in vector form as follows:

$$\frac{\partial U}{\partial t}+\sum _{k=1}^3\frac{\partial F_k}{\partial x_k}=0,$$

(8)

Then for any cell, Godunov’s scheme can be written as

$$\begin{array}{c}\frac{U_{i+\frac{1}{2},k+\frac{1}{2},l+\frac{1}{2}}^{n+1}-U_{i+\frac{1}{2},k+\frac{1}{2},l+\frac{1}{2}}^n}{\tau }+\frac{F_{x,i+1,k+\frac{1}{2},l+\frac{1}{2}}^{*}-F_{x,i,k+\frac{1}{2},l+\frac{1}{2}}^{*}}{{\Delta }_x}+\\ +\frac{F_{y,i+\frac{1}{2},k+1,l+\frac{1}{2}}^{*}-F_{y,i+\frac{1}{2},k,l+\frac{1}{2}}^{*}}{{\Delta }_y}+\frac{F_{z,i+\frac{1}{2},k+\frac{1}{2},l+1}^{*}-F_{z,i+\frac{1}{2},k+\frac{1}{2},l}^{*}}{{\Delta }_z}=0,\end{array}$$

(9)

where ${\Delta }_{x,y,z}$ are the grid spacings along the corresponding coordinates, $\tau$ is a time-step, and $F^{*}$ are the fluxes of the corresponding quantities through the cell boundaries obtained by solving the Riemann problem. Let us describe a method for constructing a numerical scheme for solving the Riemann problem.

To construct the numerical scheme, we consider the one-dimensional equations:

$$\frac{\partial }{\partial t}\left(\begin{array}{c}\mathrm{\Gamma }\rho \\ {\mathrm{\Gamma }}^2\rho h{v}_x\\ {\mathrm{\Gamma }}^2\rho h{v}_y\\ {\mathrm{\Gamma }}^2\rho h{v}_z\\ {\mathrm{\Gamma }}^2\rho h-p\end{array}\right)+\frac{\partial }{\partial x}\left(\begin{array}{c}\rho \mathrm{\Gamma }{v}_x\\ \rho h{\mathrm{\Gamma }}^2{v}_x{v}_x+p\\ \rho h{\mathrm{\Gamma }}^2{v}_y{v}_x\\ \rho h{\mathrm{\Gamma }}^2{v}_z{v}_x\\ ({\mathrm{\Gamma }}^2\rho h-p)v_x+p{v}_x\end{array}\right)=0,$$

(10)

With the operator separation scheme, the system of Equation (10) is divided into the Eulerian stage with the pressure forces at work:

$$\frac{\partial }{\partial t}\left(\begin{array}{c}\mathrm{\Gamma }\rho \\ {\mathrm{\Gamma }}^2\rho h{v}_x\\ {\mathrm{\Gamma }}^2\rho h{v}_y\\ {\mathrm{\Gamma }}^2\rho h{v}_z\\ {\mathrm{\Gamma }}^2\rho h-p\end{array}\right)+\frac{\partial }{\partial x}\left(\begin{array}{c}0\\ p\\ 0\\ 0\\ p{v}_x\end{array}\right)=0$$

(11)

and the Lagrangian stage at which the advective transport takes place:  

$$\frac{\partial }{\partial t}\left(\begin{array}{c}\mathrm{\Gamma }\rho \\ {\mathrm{\Gamma }}^2\rho h{v}_x\\ {\mathrm{\Gamma }}^2\rho h{v}_y\\ {\mathrm{\Gamma }}^2\rho h{v}_z\\ {\mathrm{\Gamma }}^2\rho h-p\end{array}\right)+\frac{\partial }{\partial x}\left(\begin{array}{c}\rho \mathrm{\Gamma }{v}_x\\ \rho h{\mathrm{\Gamma }}^2{v}_x{v}_x\\ \rho h{\mathrm{\Gamma }}^2{v}_y{v}_x\\ \rho h{\mathrm{\Gamma }}^2{v}_z{v}_x\\ ({\mathrm{\Gamma }}^2\rho h-p)v_x\end{array}\right)=0.$$

(12)

For the systems O (Original)—(10), E (Eulerian)—(11), and A (Lagrangian)—(12), we will use a Rusanov-type scheme, which can be written for an arbitrary cell in a one-dimensional statement in the following form:

$$\frac{U_{i+\frac{1}{2}}^{n+1}-U_{i+\frac{1}{2}}^n}{\tau }+\frac{F_{i+1}^{O,E,A}-F_i^{O,E,A}}{\Delta }=0,$$

(13)

where $\Delta$ is the grid spacing, $\tau$ is a time-step, and $F_i^{O,E,A}$ are the solutions of Riemann problems for equations O, E, and A on the interface between cell $i-\frac{1}{2}$ and cell $i+\frac{1}{2}$. We have

$$F^{E,A}=\frac{F_L^{E,A}+F_R^{E,A}}{2}+\frac{\lambda }{2}\left(U_L-U_R\right),$$

(14)

where L denotes the left cell and R the right one. Let us solve the system at each stage of the method.

To solve system (11), we find the eigenvalues of the Jacobian. Omitting long but obvious manipulation, we obtain

$${\lambda }_R=max\left(v_{n,R}\frac{\left(1-c_{s,R}^2\right)+c_{s,R}{\mathrm{\Gamma }}_R^{-1}{\omega }_R}{1-c_{s,R}^2{v}_R^2}-v_{n,R},0\right),$$

(15)

$${\lambda }_L=min\left(v_{n,L}\frac{\left(1-c_{s,L}^2\right)-c_{s,L}{\mathrm{\Gamma }}_L^{-1}{\omega }_L}{1-c_{s,L}^2{v}_L^2}-v_{n,L},0\right),$$

(16)

where $\omega =\sqrt{1-v_n^2-c_s\left(v^2-v_n^2\right)}$, and $v_n$ is the normal velocity component. To implement Rusanov’s scheme, at the Eulerian stage we use the maximum characteristic angle:

$${\lambda }^E=max\left(|{\lambda }_R|,|{\lambda }_L|\right).$$

(17)

The scheme for calculating the flux is

$$F^E=\frac{F_L^E+F_R^E}{2}+\frac{{\lambda }^E}{2}\left(U_L-U_R\right),$$

(18)

Note that the characteristics are limited by the angle associated with the speed of light, that is, the condition $\lambda \le 1$ is satisfied. In this way we can considerably simplify the numerical scheme.

When solving the system of Equation (12), the eigenvalue of the Jacobian matrix is the numerical advection speed calculated from the equation

$${\lambda }^A=\frac{\sqrt{{\rho }_L}{v}_{n,L}+\sqrt{{\rho }_R}{v}_{n,R}}{\sqrt{{\rho }_L}+\sqrt{{\rho }_R}},$$

(19)

where, taking into account Rusanov’s scheme (14) and the form of Equation (12), we obtain the following simple scheme:

$$F^A={\lambda }^A\left\{\begin{array}{c}{U}_L,{\lambda }^A\ge 0\\ U_R,{\lambda }^A<0\end{array}\right..$$

(20)

Thus, we have obtained fluxes for the Eulerian (E) and Lagrangian (A) stages. The flux for solving the full system of equations of special relativistic hydrodynamics is the sum of these fluxes:

$$F^{*}=F^O={\lambda }^A\left\{\begin{array}{c}{U}_L,{\lambda }^A\ge 0\\ U_R,{\lambda }^A<0\end{array}\right.+\frac{F_L^E+F_R^E}{2}+\frac{{\lambda }^E}{2}\left(U_L-U_R\right).$$

(21)

Taking into account that the velocity is naturally limited by the speed of light, the flux in trivial form may be written as follows:

$$F^{*}=\frac{F_L+F_R}{2}+\frac{U_L-U_R}{2},$$

(22)

To reduce the dissipation of the numerical method, we will use a piecewise parabolic reconstruction of the physical variables. For definiteness, a piecewise parabolic function of a physical variable $q\left(x\right)$ will be constructed on a regular grid with spacing $\Delta$ on the interval $[x_i,x_{i+1}]$. For simplicity, we will use subscript i. In general form, the parabola may be written as follows:

$$q\left(x\right)=q_i^L+\xi \left(△q_i+q_i^{\left(6\right)}(1-\xi )\right),$$

where $q_i$ is the value in the cell center, $\xi =(x-x_i){\Delta }^{-1}$, $△q_i=q_i^L-q_i^R$, and $q_i^{\left(6\right)}=6(q_i-1/2(q_i^L+q_i^R))$; the conservation condition is

$$q_i={\Delta }^{-1}{\int }_{x_i}^{x_{i+1}}q\left(x\right)dx.$$

Let us describe the construction of the parabola step by step.

At the first step, we construct $\delta q_i=1/2(q_{i+1}-q_{i-1})$. To avoid the extrema of the functions, we use the formula

$${\delta }_m{q}_i=\left\{\begin{array}{c}min\left(\right|\delta q_i|,2|q_{i+1}-q_i|,2|q_i-q_{i-1}\left|\right)sign\left(\delta q_i\right),\\ (q_{i+1}-q_i)(q_i-q_{i-1})>0\\ 0,(q_{i+1}-q_i)(q_i-q_{i-1})\le 0\end{array}\right..$$

Then, we recalculate the values at the boundary using the fourth-order interpolant

$$q_i^R=q_{i+1}^L=q_{i+1/2}=1/2(q_i+q_{i+1})-1/6({\delta }_m{q}_{i+1}-{\delta }_m{q}_i).$$

At the second step, we begin to construct the local parabola itself using the formula

$$△q_i=q_i^L-q_i^R,q_i^{\left(6\right)}=6(q_i-1/2(q_i^L+q_i^R)).$$

If the local parabola is nonmonotone (this may take place at discontinuities), we rearrange the values at the boundaries $q_i^L,q_i^R$ by the formulas

$$q_i^L=q_i,q_i^R=q_i,(q_i^L-q_i)(q_i-q_i^R)\le 0,$$

$$q_i^L=3q_i-2q_i^R,△q_i{q}_i^{\left(6\right)}>{(△q_i)}^2,$$

$$q_i^R=3q_i-2q_i^L,△q_i{q}_i^{\left(6\right)}<-{(△q_i)}^2.$$

Thus, the boundary values satisfy the monotonicity conditions.

At the third step, the parameters of the parabola are reconstructed taking into account the new values at the cell boundaries:

$$△q_i=q_i^L-q_i^R,$$

$$q_i^{\left(6\right)}=6(q_i-1/2(q_i^L+q_i^R)).$$

Recall that the local parabolas are used as important parts of the Riemann problem, since there may be discontinuities at the interfaces.

At the fourth step, the parabola is finally reconstructed, taking into account the new values at the cell boundaries:

$$△q_i=q_i^L-q_i^R,$$

$$q_i^{\left(6\right)}=6(q_i-1/2(q_i^L+q_i^R)).$$

As a result, we obtain a local parabola in each cell $[x_{i-1/2},x_{i+1/2}]$.

The left and right values of the physical variables are recalculated using the formulas

$$q_L(-{\lambda }_L\tau )=q_i^R-\frac{{\lambda }_L\tau }{2\Delta }\left(△q_i-q_i^6\left(1-\frac{2{\lambda }_L\tau }{3\Delta }\right)\right),$$

otherwise

$$q_R\left({\lambda }_R\tau \right)=q_i^L+\frac{{\lambda }_R\tau }{2\Delta }\left(△q_i+q_i^6\left(1-\frac{2{\lambda }_R\tau }{3\Delta }\right)\right),$$

where ${\lambda }_{L,R}$ is the corresponding eigenvalue at each step.

The Courant–Friedrichs–Lewy (CFL) condition is a necessary condition for the stability of an explicit Godunov-type scheme. From this condition, we determine the time-step. Completing the description of the numerical method, we again use the limitation of velocities by the light speed to limit the time-step. Then, the CFL condition can be formulated as follows:

$$\frac{\tau }{\Delta }\le CFL,$$

(23)

where $CFL$ is the Courant–Friedrichs–Lewy number.

3. Parallel Code

When using the ${256}^3$ computational grid, one time-step is calculated in approximately a minute; a complete calculation takes approximately a day.

Of course, it is necessary to develop a parallel implementation of the code based on the geometric decomposition of the computational domain with overlapping subdomains. Decomposition occurs on the images of the Coarray Fortran program. The description and study of the parallel implementation will be described below. In this section, we consider the structure of the parallel code. For simplicity, we will describe in detail a one-dimensional version of the code. However, all computational experiments and efficiency studies will be made using a three-dimensional version (see https://gitflic.ru/project/igorkulikov/rhd3dhllk, accessed on 29 May 2022).

The parallel implementation is based on a one-dimensional geometric decomposition of the computational domain with overlapping over two cells. This is due to the use of a five-point stencil for calculating the parabolas. The interprocess communications are performed by using Coarray Fortran based on one-way communications implemented in MPI-3.0 standard. Note that the interprocess communication tools are standards of the language, not of an external library.

This implies that the software of the code being developed is portable. A schematic diagram of the geometric decomposition of calculations and the one-way communications using, as an example, a one-dimensional implementation, are shown in Figure 1, where the blue color shows the cells for the boundary conditions, the green color denotes the calculation cells, and the gray color is the overlapping of subdomain cells. Note that in one-way interaction the exchange process is always initiated by the left process (image for Coarray Fortran programs). This is due to the fact that reading and writing data of a right image is always in the first cells. Therefore, the information about the local computational domain size contained in each image need not be stored in each program image. Nevertheless, this organization of data exchanges ensures data integrity when calculating the next time-step. Implementation details of the Coarray Fortran code are given in Appendix A.

To study the parallel implementation, we use two nodes of an NKS-1P cluster, each with two 24-core Intel Xeon 6248R processors with Hyper-Threading support. Let us perform a preliminary study of the program speedup on a ${256}^3$ grid with various numbers of nodes, sockets, cores, and threads, which, upon multiplication, forms the number of images.

A study of the calculation time shows that to increase the program images optimally, the following scheme should be used: increase the number of nodes, threads, processors, and cores, as shown in

Table 1. Calculation time for one time-step when using a ${256}^3$ grid.

Nodes	Sockets	Cores	Threads	Images	Time (s)
1	1	1	1	1	27.542
1	1	2	1	2	14.724
1	1	1	2	2	14.443
1	2	1	1	2	14.122
2	1	1	1	2	14.072
1	1	2	2	4	7.345
1	1	4	1	4	7.485
1	2	1	2	4	7.395
2	1	2	1	4	7.398
2	1	1	2	4	7.318
2	2	1	1	4	7.378
1	1	8	1	8	3.952
1	1	4	2	8	4.045
1	2	2	2	8	4.051
1	2	4	1	8	3.985
2	2	1	2	8	3.945
2	1	2	2	8	3.946
2	2	2	1	8	3.971

.

To study the speedup of the software implementation, we use a ${256}^3$ calculation grid with different numbers of Coarray Fortran program images, and to study the scalability, a ${128}^3$ grid per one program image.

A 17-fold speedup was achieved with 32 images and 92% efficiency when using 192 Coarray Fortran images, results are shown in Figure 2.

4. Verification

For verification, we formulate three one-dimensional tests to consider the dynamics of a relativistic gas on the interval $[0,1]$ up to $t=0.4$. At point $x_0=0.5$ from the initial time the gas has a discontinuity. The adiabatic index $\gamma =5/3$. We use non-reflecting boundary conditions. In the simulation, 200 cells for all one-dimensional tests are used. In addition to comparing a numerical solution with an analytical one, we will analyze the order of convergence of the method in $L_1$ norm:

$$L_1=\sum _{i}h|u_i-u\left(x_i\right)|,$$

where $u\left(x_i\right)$ is the exact solution at point $x_i$, $u_i$ is the numerical result, and h is the uniform grid spacing.

First, consider a one-dimensional shock wave test with the following parameters: pressure $p^L=40/3$, density ${\rho }^L=10$, and velocity $v^L=0$. The parameters of the gas to the right of the discontinuity are: pressure $p^R={10}^{-8}$, density ${\rho }^R=1$, and velocity $v^R=0$. The simulation results are shown in Figure 3 and Figure 4.

Figure 5 shows the behavior of the numerical solution error and the order of convergence of the method at grid refinement.

The method correctly reproduces all components of the solution. The shock wave dissipates only over two cells, which is due to the use of piecewise parabolic methods. The shock wave peak in the density function is reproduced correctly. No additional dissipation is observed on the rarefaction wave. Note that there are no entropy traces in the pressure and velocity functions. Although the error curves decrease rather smoothly, for small grids there are jumps in the order of convergence curves. With significant grid refinement, the density and velocity functions converge to 0.8, whereas the pressure function converges to 0.9.

As a second test, consider a stronger shock wave with the following parameters: pressure $p^L=1000$, density ${\rho }^L=1$, and velocity $v^L=0$. The parameters of the gas to the right of the discontinuity are: pressure $p^R={10}^{-2}$, density ${\rho }^R=1$, and velocity $v^R=0$. The simulation results are shown in Figure 6 and Figure 7.

Figure 8 shows the behavior of the numerical solution error and the order of convergence of the method at grid refinement.

The main difficulty of this test is that behind the shock wave front there is a very thin shell, which covers only two cells in the experiment. Even at this resolution, the numerical method was able to reproduce the shell location, and only 60% of the density peak was reproduced. Note that in the shell zone the solution dissipates over two cells in the direction of the shock wave and over two cells behind the shell front. At the same time, the mass of the dissipation zone corresponds well to the mass of the shell. The other parts of the solution are reproduced correctly. The difficulties in reproducing the shell impose some restrictions on the order of convergence of the density function. Note that on small grids the shell is reproduced with a small, increasing order of convergence to 0.6. At the same time, the convergence of the velocity and pressure functions is quite stable; they converge to the first-order for the pressure function since the smooth part of the rarefaction wave prevails in the solution, and to 0.8 for the velocity function due to a large shock wave amplitude.

Let us add complexity to the second test by adding a nonzero tangential initial velocity. The parameters of the gas to the left of the discontinuity are: pressure $p^L=1000$, density ${\rho }^L=1$, normal velocity component $v_x^L=0$, and tangential velocity component $v_y^L=0$. The parameters of the gas to the right of the discontinuity are: pressure $p^R={10}^{-2}$, density ${\rho }^R=1$, normal velocity component $v_x^R=0$, and tangential velocity component $v_y^R=0.99$. The simulation results are shown in Figure 9 and Figure 10.

Figure 11 shows the behavior of the numerical solution error and the order of convergence of the method at grid refinement.

In contrast to ideal hydrodynamics, in the equations of special relativistic hydrodynamics, the tangential velocity greatly affects the normal velocity component due to the nonlinearity of the Lorentz factor. The test is much more complicated in that the tangential velocity is given in a cold gas. Therefore, the method must correctly reproduce the interaction of the velocity resulting from the shock impact of the hot gas. Note that the method reproduces well the shock wave zone with dissipation over two cells at the front and over four cells behind the shock wave shell zone, as in the previous test. Not only the location but also the density peak amplitude is reproduced correctly. The absence of shell-type solutions causes more-regular behavior of the error and the order of convergence of the method. Thus, for the pressure function, the order of convergence is one (the reason being the same as in the previous test), and for the density and velocity functions it is 0.8.

Let us take up the problem of discontinuity breakdown in a two-dimensional statement. For this, consider gas dynamics in the ${[-1/2;1/2]}^2$ domain up to time $t=0.4$. Let us divide the domain into four parts in geographical directions: NW—$[-1/2;0]\times [0;1/2]$, NE—$[0;1/2]\times [0;1/2]$, SW—$[-1/2;0]\times [-1/2;0]$, SE—$[-1/2;0]\times [0;1/2]$. Let us define the parameters of the gas in each part as follows:

$${\left(\rho ,v_x,v_y,p\right)}^{NE}=\left(0.1,0,0,0.01\right),$$

$${\left(\rho ,v_x,v_y,p\right)}^{NW}=\left(0.1,0.99,0,1\right),$$

$${\left(\rho ,v_x,v_y,p\right)}^{SE}=\left(0.1,0,0.99,1\right),$$

$${\left(\rho ,v_x,v_y,p\right)}^{SW}=\left(0.5,0,0.99,1\right),$$

The adiabatic index $\gamma =5/3$. Nonreflecting boundary conditions are set on all boundaries. In the simulation, a grid of ${200}^2$ cells is used. The results of the simulation are presented in Figure 12.

The interaction between domains NW/SW and SE/SW form contact discontinuities with a jump in the tangential velocities. These contact discontinuities result in a numerical artifact that expands over time in domain SW. The formation of two interacting shock waves causes a density jump in domain NE, which has the same amplitude and limited dissipation. In this test, only the interaction of different waves is investigated, since the reproduction of individual waves has already been considered in the previous test problems.

5. Numerical Simulation

Let us consider the problem of shock wave formation for a gas at rest in a three-dimensional statement. For this, consider gas dynamics in the ${[-1;1]}^3$ domain up to time $t=0.4$. Let the domain have a spherical radius $r_0=0.5$. In the internal part of the domain, the gas parameters are: pressure $p_0=40/3$ and density ${\rho }_0=10$. In the external part of the domain, the gas parameters are: pressure $p={10}^{-8}$ and density $\rho =1$. The adiabatic index $\gamma =5/3$. Nonreflecting boundary conditions are used. The results of the simulation are presented in Figure 13. A grid of ${400}^3$ cells is used in the simulation.

According to the results of this test, the solution is invariant with respect to rotation in the three-dimensional case. The density peak is reproduced correctly, and the shock wave is reproduced well (its analytical solution is shown by a dotted line). This problem is considered as a central explosion of a supernova without analyzing the cause of the explosion (thermonuclear reactions or a core collapse) in the model of special relativistic hydrodynamics.

As a second problem, consider interacting relativistic jets. We will simulate two interacting galactic jets with density ${\rho }_J={10}^{-3}$ cm${}^{-3}$ and radius $R_J=200$ pc moving with Lorentz factor $\mathrm{\Gamma }=10$ and having relativistic Mach number $\mathcal{M}=8$. The atmosphere of the galaxy has temperature $T_A={10}^7$ K and density ${\rho }_A={10}^{-2}$ cm${}^{-3}$. The adiabatic index $\gamma =5/3$. Figure 14 presents the results of simulating the evolution of the galactic jets.

In both jets, a spherical shock wave propagates forward at the speed of light. Behind the shock front there is a shell with a system of interacting shock waves. Inside the shell there is a sparse cocoon area. During the interaction of the jets, the hot point zone becomes extended and covers about 1 kpc.

6. Discussion and Conclusions

Distinctive features of the code are as follows:

• To increase the order of accuracy on smooth solutions and decrease dissipation at discontinuities, a piecewise parabolic reconstruction is used for the physical variables since they are responsible for the thermodynamic state of the continuous medium [43].
• We use a piecewise-parabolic reconstruction of physical variables although the calculations in the scheme are based on the use of vectors of conservative variables and their flows through cell boundaries. This is due to the fact that they determine the thermodynamic state of a continuous medium. This was emphasized in  [43] when studying the accuracy of the discontinuous Galerkin method on the Einfeldt problem.
• When using Rusanov-type methods as part of Godunov’s method, no full spectral problem is solved; this is the case, for instance, when using an equation of state of the Taub–Mathews type [56].
• We plan to extend the model for a multiphase medium, where the equation of state is more complicated [57]. In the present paper, we neglect the right-hand side of the energy equation, although this term can be easily included in our numerical scheme [58].
• On the basis of the above computational experiments, we will try to avoid using the MPI technology where possible. The Coarray Fortran technology is a good alternative to develop parallel programs for distributed memory architectures.
• In a number of problems, a significant part of the computational domain is occupied by flows with a high value of the Lorentz factor and low pressure values. In these cases, the energy equation is used in the form:

  $$\frac{\partial }{\partial t}\left({\mathrm{\Gamma }}^2\rho h-p-\mathrm{\Gamma }\rho \right)+\sum _{k=1}^3\frac{\partial }{\partial x_k}\left(\rho h{\mathrm{\Gamma }}^2{v}_k-\rho \mathrm{\Gamma }{v}_k\right)=0.$$

  (24)

The above statements are disputable. However, we believe they are appropriate when discussing the code. To conclude: in the present paper, a new code for describing flows in a model of special relativistic hydrodynamics and implemented with Coarray Fortran parallel programming technology was presented. During the development of the code, a new original numerical method was constructed with high-order accuracy on smooth solutions and low dissipation at discontinuities. The numerical method has been thoroughly verified. A code scalability of 92% is obtained on a cluster with Intel Xeon 6248R NKS-1P with 192 Coarray Fortran images. As an astrophysical application, we considered two interacting relativistic jets.