An Iteration Algorithm for American Options Pricing Based on Reinforcement Learning

Abstract

In this paper, we present an iteration algorithm for the pricing of American options based on reinforcement learning. At each iteration, the method approximates the expected discounted payoff of stopping times and produces those closer to optimal. In the convergence analysis, a finite sample bound of the algorithm is derived. The algorithm is evaluated on a multi-dimensional Black-Scholes model and a symmetric stochastic volatility model, the numerical results implied that our algorithm is accurate and efficient for pricing high-dimensional American options.

1. Introduction

The pricing of American options is an important issue in quantitative finance and stochastic processes [1,2]. Many popular derivative products in various financial sectors are of the American type, and can be exercised at any time before maturity. Therefore, considerable effort has been spent to obtain accurate and efficient methods for pricing American options (see, e.g., Hull [3]). When the dimension of the option is small, methods based on partial differential equations [4] and binomial trees [5] can be applied. However, the calculation costs of these methods increase exponentially as the dimension gets larger, thus making them inefficient for pricing options on many underlying assets, such as the widely used high-dimensional symmetric stochastic volatility models [6].

To treat American options on multi-dimensional underlying assets, many pricing methods based on Monte Carlo simulation have been proposed. The most popular are the regression-based methods proposed by Longstaff and Schwartz [7] and Tsitsiklis and Roy [8]. Through a backward iteration scheme, these methods can approximate the continuation value and a feasible exercise policy, such as linear regression [7], neural network [9], Gaussian process regression [10] and kernel ridge regression [11], all of which produce lower price bounds for American options. The dual approaches for American options were developed by Rogers [12] and Haugh and Kogan [13], these methods produce upper price bounds for options. However, in the computation of these methods, the continuation value at each date is approximated by different functions, and only the data at this date are used.

A different strand of the literature focuses on finding the optimal exercise policy by Monte Carlo sample [14,15]. These approaches consider a parametric class of exercise regions and maximize an estimate of the value function within the parametric class. Through optimization, all the sample data are used to approximate the optimal exercise policy. Recently, Bayer et al. [16] and Becker et al. [17] consider randomized stopping times in approximating the optimal exercise regions. However, in these approaches, the resulting loss function may still be non-concave and exhibit isolated local optima; thus, it is difficult to find the global optimum for the loss function [18,19].

Reinforcement learning, especially the policy iteration method, has achieved empirical success in high-dimensional control problems [20,21,22]. The basic idea of policy iteration is to compute the evaluation function of the policy in each iteration, after which an improved policy is computed from the function for the next iteration [23]. The pricing of American options could be seen as a control problem, but there are only two possible actions, and these do not influence the underlying process. Reinforcement learning methods have already been applied to optimal stopping problems. Tsitsiklis and Roy [8] introduced the fitted Q-iteration for American option pricing based on the least-squares method. Yu and Bertsekas [24] proposed an algorithm based on projected value iteration, the convergence of this method for finite-state models was also obtained. Li et al. [25] considered a least squares policy iteration in the pricing of American options, and they give a finite-time bound for the algorithm. Becker et al. [26] developed an algorithm related to policy optimization for high-dimensional optimal stopping problems. Chen et al. [27] applied Zap-Q-learning for the optimal stopping problem, and established consistency of the algorithm for linear function approximation. Herrera et al. [28] considered a fitted Q-iteration based on randomized neural networks for the optimal stopping problems. However, most of these methods are a direct transformation of the method in reinforcement learning. Furthermore, there is a lack of analysis on the accuracy and efficiency of reinforcement learning in pricing high-dimensional American options.

In this paper, we propose an iteration algorithm for American options based on reinforcement learning. In each iteration, the expected discounted payoff of a family of stopping times is approximated by regression; thus, the data of all dates are used to improve the approximation of all dates. An improved family of stopping times was obtained based on the constructed function. After this procedure, an approximate optimal exercise policy was obtained. To provide theoretical guarantees, we developed a finite sample-error bound for the algorithm. In the numerical experiments, we considered the data generated by the multi-dimensional Black–Scholes model and a symmetric stochastic volatility model. The results showed that (a) our algorithm was accurate and efficient in pricing high-dimensional American options; (b) by using a function of time and underlying process, the continuation values can be approximated using a fraction of the parameters; (c) the methods based on reinforcement learning outperform the state-of-the-art methods in the pricing of American options.

The paper is organized as follows. In Section 2, we introduce the problem of pricing American options and illustrate the relationship of continuation values and stopping times. The efficient algorithm is described in Section 3. In Section 4, convergence rates of the algorithm are discussed. Numerical experiments of high-dimensional American options on multi-dimensional Black–Scholes model and a symmetric stochastic volatility model are given in Section 5. Finally, we conclude in Section 6. All proofs are found in the Appendix A.

2. Pricing of American Options and Stopping Times

In this section, we introduce to the pricing of American options. Let $\{X_t,0\le t\le T\}$ be a ${\mathbb{R}}^d$-valued Markov process, this process is defined on a filtered measurable probability space with a risk-neutral measure $\mathrm{P}$. We assumed that the process records all relevant financial variables. In practice, the price of the American option was approximated by the price of a Bermudan option [11], which could be exercised at discrete time points $0<t_1<\cdots <t_N=T$. For $0\le n\le N$, we represented $t_n$ by n to simplify the notation in the following. We assumed that the risk-free discount factor between time points was constant, which was denoted by $\gamma \in (0,1)$. The price $V_n\left(x\right)$ of the option at $n=1,\dots ,N$ was given by the optimal stopping problems

$$V_n\left(x\right)=\underset{\tau \in {\mathcal{T}}_n}{sup}\mathrm{E}\left[{\gamma }^{\tau -n}g\left(X_{\tau }\right)|X_n=x\right],$$

(1)

where $g\left(x\right)$ is the non-negative payoff function, ${\mathcal{T}}_n$ denotes the set of stopping times such that $n\le \tau$. The price at time 0 is given by $V_0\left(x\right)=\mathrm{E}\left[\gamma V_1\left(X_1\right)\right|X_0=x]$. We assume that g satisfies $\parallel g\left(X_n\right){\parallel }_{\infty }\le B$ for $n=1,\dots ,N$, where ${\parallel ·\parallel }_p$ denotes the $L^p$-norm and $B>0$. As we will see later, this assumption can be relaxed.

The optimal stopping problem (1) is solved by a family of optimal stopping times ${\tau }_n^{*}$, $n=1,\dots ,N$, that satisfies the consistency property ${\tau }_n^{*}>n⟹{\tau }_n^{*}={\tau }_{n+1}^{*}$[16]. By a dynamic programming principle, ${\tau }_n^{*}$ can be determined by the continuation values [29]. The value in state x at time n is $C^{*}(N,x)\equiv 0$ for $n=N$ and

$$C^{*}(n,x)=\mathrm{E}\left[{\gamma }^{{\tau }_{n+1}^{*}-n}g\left(X_{{\tau }_{n+1}^{*}}\right)\right|X_n=x],$$

(2)

for $n=0,\dots ,N-1$. Then, ${\tau }_n^{*}$ can be written as

$${\tau }_n^{*}=inf\left\{i\ge n:g\left(X_i\right)\ge C^{*}\left(i,X_i\right)\right\}.$$

(3)

In other words, the option should be exercised when the current payoff is larger than the continuation value. A meaningful family of suboptimal stopping times should be obtained by replacing the continuation values by a good approximation.

Motivated by the fitted policy iteration method in reinforcement learning [23], we considered iteratively approximating the family of optimal stopping times. In this paper, we dealt with consistent families of stopping times ${\tau }_n,n=1,\dots ,N$. These times satisfied $n\le {\tau }_n\le N$ with ${\tau }_N=N$ and ${\tau }_n>n⟹{\tau }_n={\tau }_{n+1}$. We define the function $C^{\tau }:\{0,1,\dots ,N-1\}\times {\mathbb{R}}^d\to \mathbb{R}$ by

$$C^{\tau }(n,x)=\mathrm{E}\left[{\gamma }^{{\tau }_{n+1}-n}g\left(X_{{\tau }_{n+1}}\right)\right|X_n=x].$$

(4)

This function represents the expected discounted payoff achieved when $X_n=x$, and the option is not exercised at n after which the stopping time ${\tau }_{n+1}$ is followed. Conversely, given $C:\{0,1,\dots ,N-1\}\times {\mathbb{R}}^d\to \mathbb{R}$, we defined a new family of stopping times by

$$\begin{array}{cc}\hfill & {\tau }_N^{{}^{\prime }}=N,\hfill \\ & {\tau }_n^{{}^{\prime }}=\left\{\begin{array}{cc}n,\hfill & \mathrm{if}g\left(X_n\right)\ge C(n,X_n),\hfill \\ {\tau }_{n+1}^{{}^{\prime }},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(5)

It was immediately seen that the obtained family of stopping times ${\tau }_n^{{}^{\prime }}$, $1\le n\le N$ was consistent. The following result shows that, the exercise policy ${\tau }_1^{{}^{\prime }}$ constructed from $C^{\tau }$ yielded a higher than expected discounted payoff than the original policy ${\tau }_1$.

Theorem 1.

For any family of consistent stopping times ${\tau }_n,n=1,\dots ,N$, the stopping time ${\tau }_1^{{}^{\prime }}$ constructed from $C^{\tau }$ by (5) satisfies

$$\mathrm{E}{\gamma }^{{\tau }_1^{{}^{\prime }}}g\left(X_{{\tau }_1^{{}^{\prime }}}\right)\ge \mathrm{E}{\gamma }^{{\tau }_1}g\left(X_{{\tau }_1}\right).$$

(6)

By Theorem 1, if we can approximate $C^{\tau }(n,X_n)$ for a family of stopping times ${\tau }_n$, $1\le n\le N-1$, we can construct an improved family of stopping times closer to the optimal family.

3. Iteration Algorithm

In this section, we propose a two-step iteration algorithm for American options. In the evaluation step, $C^{\tau }(n,x)$ of a family of stopping times is estimated. In the improvement step, the estimated function is used to construct an improved family of stopping times by (5).

We first defined the approximation architecture used for estimating $C^{\tau }(n,x)$ in the algorithm. Contrary to the regression-based algorithms, we used a single function by taking the time as an argument in the computation throughout the paper, denoted by $\mathcal{F}$: = $\{f:{\mathbb{R}}^{d+1}\to \mathbb{R}\}$, the choosing set of real-valued functions. We also introduced the truncation operator for the approximation architecture. Let ${\psi }_B$ denote the truncation operator with level B defined by

$${\psi }_{B}f=\left\{\begin{array}{ll}f,& \mathrm{if}\left|f\right|\le B,\\ sign\left(f\right)·B,& \mathrm{otherwise}.\end{array}\right.$$

(7)

For a set of functions $\mathcal{F}$, we set ${\psi }_B\mathcal{F}=\{{\psi }_{B}f:f\in \mathcal{F}\}$.

To obtain a good approximation of $C^{\tau }$, it was a straightforward matter to consider

$$\frac{1}{N}\sum _{n=0}^{N-1}\mathrm{E}{\left(C^{\tau }(n,X_n)-\mathrm{E}\left[{\gamma }^{{\tau }_{n+1}-n}g\left(X_{{\tau }_{n+1}}\right)\right|X_n]\right)}^2.$$

(8)

To obtain a practical procedure, we considered the sample-based approximation to (8) in the algorithm.

To approximate the optimal stopping times numerically, our method was initialized with arbitrary $C_0\in \mathcal{F}$ and corresponding stopping times ${\tau }_n^1$ constructed from (5), $n=1,\dots ,N$. For $j=1,\dots ,J-1$, we generated a set of Monte Carlo paths $(x_0^i,\dots ,x_N^i)$, $i=1,\cdots ,M$ of process $X_n$, this sample set was independent of all previously generated paths. If at time n, stopping time ${\tau }_n^j$ was applied, the discounted payoff along the i-th simulated path was denoted by ${\gamma }^{{\tau }_n^{j,i}-n}g\left(x_{{\tau }_n^{j,i}}^i\right).$ To obtain the approximation of $C^{{\tau }^j}$, we considered minimizing the empirical counterpart of (8). Let ${\widehat{f}}_j\in \mathcal{F}$ satisfy

$${\widehat{f}}_j=arg\underset{f\in \mathcal{F}}{min}\frac{1}{NM}\sum _{i=1}^M\sum _{n=0}^{N-1}{\left(f\left(n,x_n^i\right)-{\gamma }^{{\tau }_{n+1}^{j,i}-n}g\left(x_{{\tau }_{n+1}^{j,i}}^i\right)\right)}^2,$$

(9)

we used the truncation ${\widehat{C}}^{{\tau }^j}={\psi }_B{\widehat{f}}_j$ as the approximation. In the next iteration, an improved family of stopping times ${\tau }_n^{j+1}$, $n=1,\dots ,N$ was obtained by (5). Starting from any family of consistent stopping times and computing inductively, we finally constructed the exercise policy ${\tau }_1^J$. Note that the optimization problem (9) is easily solved for some linear function space such as polynomial basis functions. For other approximation architecture such as neural networks, gradient-based methods can be applied to find the infimum, since (9) is differentiable with respect to f.

To estimate $V_0$, we generated another independent Monte Carlo sample path $(x_0^i,\dots ,x_N^i)$, $i=1,\cdots ,M^{{}^{\prime }}$ and approximate $V_0$ by the average

$${\widehat{V}}_0=\sum _{i=1}^{M^{{}^{\prime }}}{\gamma }^{{\tau }_1^{J,i}}g\left(x_{{\tau }_1^{J,i}}^i\right).$$

(10)

Our method is summarized into Algorithm 1. In next section, we discuss the convergence of the algorithm and derive a finite sample bound.

Algorithm 1 Iteration algorithm for pricing American options.
Require: the number of sample path M, $M^{{}^{\prime }}$, the number of iterations J and function space $\mathcal{F}$
Ensure:  the approximating optimal stopping time ${\tau }_1^J$, the price estimate ${\widehat{V}}_0$
1:	Generate sample paths of the underlying process;
2:	Generate a random function $C_0\in \mathcal{F}$;
3:	for$j=1,\dots ,J-1$do
4:	Obtain ${\tau }_n^j$, $n=1,\dots ,N$ using ${\widehat{C}}^{{\tau }^{j-1}}$ from (5);
5:	Construct ${\widehat{f}}_j$ by the regression optimization problem $${\widehat{f}}_j=arg\underset{f\in \mathcal{F}}{min}\frac{1}{NM}\sum _{i=1}^M\sum _{n=0}^{N-1}{\left(f\left(n,x_n^i\right)-{\gamma }^{{\tau }_{n+1}^{j,i}-n}g\left(x_{{\tau }_{n+1}^{j,i}}^i\right)\right)}^2;$$
6:	Obtain the approximation by ${\widehat{C}}^{{\tau }^j}={\psi }_B{\widehat{f}}_j$;
7:	end for
8:	Obtain ${\tau }_1^J$ using ${\widehat{C}}^{{\tau }^{J-1}}$ from (5);
9:	Generate another independent sample path of the underlying process;
10:	Calculate the option price by (10);
11:	return${\tau }_1^J$ and ${\widehat{V}}_0$;

4. Convergence Analysis

In this section, we consider the convergence of the algorithm introduced in Section 3. Before describing the main result, we present some necessary definitions. To measure the complexity of a functional class, we introduced the definition of covering numbers. For a class of functions $\mathcal{F}$ and points ${\mathbf{z}}_1^M:=\left(z_1,\dots ,z_M\right)$, the covering number ${\mathcal{N}}_1\left(ϵ,\mathcal{F},{\mathbf{z}}_1^M\right)$ is the minimal number $Q\in \mathbb{N}$ such that there exist functions $f_1,\dots ,f_Q$ with the property that for every $f\in \mathcal{F}$ there is a $q\in \{1,\dots ,Q\}$ such that

$$\frac{1}{M}\sum _{i=1}^M\left|f\left(z_i\right)-f_q\left(z_i\right)\right|<ϵ.$$

(11)

For $f:{\mathbb{R}}^{d+1}\to \mathbb{R}$, we introduce $\parallel ·\parallel$ by

$${\parallel f\parallel }^2=\frac{1}{N}\sum _{n=0}^{N-1}{∥f(n,X_n)∥}_2^2.$$

(12)

Denoted by $\tau \left(f\right)$, the family of stopping times was obtained from (5) with respect to f. Let ${\mathrm{E}}_M$ stands for the expectation conditioned by the samples used to approximate the function $C^{\tau }$. We now state our main result about the convergence of our algorithm.

Theorem 2.

Assume that $B<\infty$. Fix the set of admissible functions $\mathcal{F}$ and positive integer M. For $j=1,\dots ,J$, define ${\tau }^j$ by (5) and define ${\widehat{C}}^{{\tau }^j}$ by (9). Then

$$\begin{array}{cc}\hfill {\mathrm{E}}_M& ∥C^{*}(n,X_n)-C^{{\tau }^J}(n,X_n)∥\hfill \\ \hfill & \le \frac{c_{1}logM{sup}_{0\le n\le N-1}\left({sup}_{{\mathbf{x}}_1^M\in {(\left\{n\right\}\times {\mathbb{R}}^d)}^M}{log}^{1/2}\left({\mathcal{N}}_1\left(\frac{1}{MB},{\psi }_B\mathcal{F},{\mathbf{x}}_1^M\right)\right)\right)}{M^{1/2}}\hfill \\ & +c_2\underset{f^{{}^{\prime }}\in \mathcal{F}}{sup}\underset{f\in \mathcal{F}}{inf}∥f-C^{\tau \left(f^{{}^{\prime }}\right)}∥+c_3{\gamma }^{J/2}B,\hfill \end{array}$$

(13)

where $c_1,c_2,c_3>0$.

There are three terms in the bound (13). The first is the estimation error caused by the sampling step in the approximation. The second is the approximation error of $\mathcal{F}$ with respect to the $C^{{\tau }^j}(n,x)$ appearing in the iteration. The third comes from the error remaining after running the iteration algorithm for J iterations. This term decays at a geometric rate.

Remark 1.

$log{\mathcal{N}}_1\left(ϵ,{\psi }_B\mathcal{F},{\mathbf{x}}_1^M\right)$ is bounded by $logM·{\nu }_{{\psi }_B\mathcal{F}+}$ under some mild conditions, ${\nu }_{{\psi }_B\mathcal{F}+}$ is the VC-dimension of ${\psi }_B\mathcal{F}+$ (see the definition in Kohler and Langer [30]). Theorems 1 immediately apply for linear, finite-dimensional approximation architecture, since the corresponding VC-dimension is bounded [31]. For neural networks with L hidden layers, λ neurons per layer and ReLU activation function, a bound of $c_4\lambda Llog\lambda$ with $c_4>0$ on the corresponding VC-dimension is also known [30]. Hence, the Theorem applies for deep neural networks as well.

The next corollary provides a bound on the difference between $V_0$ and $\mathrm{E}{\widehat{V}}_0$.

Corollary 1.

Assume that $B<\infty$ and $X_0=x_0$ a.s. for some $x_0\in \mathbb{R}$. Fix the set of admissible functions $\mathcal{F}$ and positive integer M. Define ${\tau }_1^J$ by (5) with respect to ${\widehat{C}}^{{\tau }^{J-1}}$ and define ${\overline{V}}_0:=\mathrm{E}{\gamma }^{{\tau }_1^J}g\left(X_{{\tau }_1^J}\right)$. Then

$$\begin{array}{cc}\hfill {\mathrm{E}}_M& \left|V_0-{\overline{V}}_0\right|\hfill \\ \hfill & \le \frac{c_{5}logM{sup}_{0\le n\le N-1}\left({sup}_{{\mathbf{x}}_1^M\in {(\left\{n\right\}\times {\mathbb{R}}^d)}^M}{log}^{1/2}\left({\mathcal{N}}_1\left(\frac{1}{MB},{\psi }_B\mathcal{F},{\mathbf{x}}_1^M\right)\right)\right)}{M^{1/2}}\hfill \\ & +c_6\underset{f^{{}^{\prime }}\in \mathcal{F}}{sup}\underset{f\in \mathcal{F}}{inf}∥f-C^{\tau \left(f^{{}^{\prime }}\right)}∥+c_7{\gamma }^{J/2}B,\hfill \end{array}$$

(14)

where $c_5,c_6,c_7>0$ and $0<\gamma <1$.

5. Numerical Examples

In this section, the performances of the algorithm were tested on various American options for bounded and unbounded payoffs.

The computations were carried out on a laptop with an Intel i5-10300H 2.50 GHz CPU and a NVIDIA GeForce GTX 1650 GPU.

To evaluate our method, we considered two function spaces: the linear spaces of polynomials and neural networks. Polynomial basis functions have been used in Longstaff and Schwartz [7] and are a popular basis function for regression-based methods. To include interaction terms in the basis, we considered the classical polynomial basis functions up to the third order. Neural network approximates nonlinear functions by successive compositions of an affine transformation and non-linear activation function. This model showed good performance for pricing American options, especially in high dimensions [32].

We compared our method with two state-of-the-art methods: the least squares Monte Carlo (LSM) proposed in Longstaff and Schwartz [7] and deep optimal stopping (DOS) proposed in Becker et al. [26]. To have a fair comparison for accuracy and efficiency, we used the same number of sample paths and time steps for both methods. Furthermore, we used the same network architecture in DOS and in the method with the neural network except the activation function. There were 3 hidden layers and $40+d$ neurons per hidden layer in the networks. The activation function in our method was a leaky ReLU function.

5.1. Multi-Dimensional Black–Scholes Model

In this subsection, we consider high-dimensional American options in the Black–Scholes model. Assume the risk-neutral dynamics of the assets prices $S_t=(S_t^1,\dots ,S_t^d)$ are given by

$$S_t=S_{0}exp\left(\left[r-\delta -\frac{1}{2}{\sigma }^2\right]t+\sigma W_t\right),$$

(15)

where $S_0$ is the initial value; r is the risk-free interest rate; $\delta$ is the dividend rate; and $W_t$ is d-dimensional Brownian motions with covariance matrix $\rho$. The parameters are set as $T=1$ and $N=10$, and we used $r=5\%$, $\delta =0$ for each asset. We assumed that $\rho$ was a diagonal matrix and all the assets had the same volatility $\sigma =0.2$ and initial value $S_0=100$.

We considered three types of high-dimensional options: max call options with payoff ${\left({max}_{1\le i\le d}{S}_t^i-K\right)}^{+}$, arithmetic put options with payoff ${(K-\frac{1}{d}{\sum }_{i=1}^d{S}_t^i)}^{+}$ and geometric put options with payoff ${(K-{\left({\prod }_{i=1}^d{S}_t^i\right)}^{1/d})}^{+}$. We considered $d=\{5,10,20,30,40,60,80,100\}$, and set $K=100$. In the computation we set M = 20,000; $M^{{}^{\prime }}$ = 100,000; $J=5$; and we used the payoff as a regressor. For American options with unbounded payoff, we omitted the truncation step.

Table 1. Pricing results for d-dimensional max call options.

d	Ref.	LSM-2	Time	LSM-3	Time	LFPI	Time	DOS	Time	NNFPI	Time
5	29.63	29.635	2.3	29.722	2.0	29.732	2.6	27.483	12.8	29.683	19.1
10	38.96	39.062	5.9	39.131	9.2	39.219	7.1	35.772	11.9	39.097	18.9
20	47.84	47.279	14.4	47.912	91.6	48.000	23.2	45.839	12.8	47.968	22.8
30	52.91	50.082	28.7	52.786	667.2	52.965	53.8	51.301	13.1	52.991	23.6
40	56.37	56.175	90.6			56.347	138.7	56.132	14.4	56.496	24.3
60	61.24							60.416	16.7	61.362	32.0
80	64.72							63.457	19.3	64.704	35.7
100	67.28							66.333	20.4	67.323	40.4

,

Table 2. Pricing results for d-dimensional arithmetic put options.

d	Ref.	LSM-2	Time	LSM-3	Time	LFPI	Time	DOS	Time	NNFPI	Time
5	2.05	2.045	1.4	2.046	1.2	2.047	2.6	2.046	12.6	2.048	19.0
10	1.39	1.376	3.3	1.378	4.7	1.378	7.0	1.378	11.7	1.378	18.7
20	1.06	1.042	6.7	1.045	35.7	1.046	25.5	1.047	12.5	1.047	22.7
30	0.64	0.626	11.7	0.628	215.0	0.630	55.1	0.629	13.2	0.630	23.5
40	0.66	0.645	26.6			0.646	141.0	0.646	14.1	0.646	24.3
60	0.89							0.864	16.8	0.865	31.9
80	0.74							0.712	19.2	0.713	35.6
100	0.32							0.311	20.5	0.312	40.2

and

Table 3. Pricing results for d-dimensional geometric put options.

d	Ref.	LSM-2	Time	LSM-3	Time	LFPI	Time	DOS	Time	NNFPI	Time
5	2.05	2.033	1.3	2.036	1.5	2.038	2.6	2.036	12.6	2.038	19.1
10	1.39	1.368	3.8	1.376	5.1	1.379	6.9	1.373	11.9	1.379	18.8
20	1.06	1.027	7.7	1.043	40.7	1.049	25.2	1.051	12.6	1.052	22.8
30	0.64	0.604	13.7	0.616	271.5	0.625	54.9	0.625	13.2	0.625	23.7
40	0.66	0.645	35.1			0.654	135.8	0.646	14.3	0.655	24.4
60	0.89							0.869	16.6	0.887	32.0
80	0.74							0.719	19.3	0.722	35.6
100	0.32							0.308	20.8	0.311	40.3

report the pricing results. Column Ref. provides the benchmark value computed by Premia (https://www.rocq.inria.fr/mathfi/Premia, accessed date is 29 September 2021), a freely available software for derivative pricing and hedging. Column Time is the computational times in seconds. For large d the computation time of LSM and our method with a polynomial basis function exceeded a reasonable amount of time, so the results were omitted. The columns in tables labeled as LSM-2, LSM-3, LFPI and NNFPI correspond to, respectively, LSM with a second-order polynomial basis function, LSM with a third-order polynomial basis function, our method with a second-order polynomial basis function, and our method with a neural network. Because ${\widehat{V}}_0$ is a lower-biased price estimate, higher price estimates implied better experimental performance.

We observed that both LFPI and NNFPI provided accurate results. LFPI was relatively faster in low-dimensional cases, but the computation time of NNFPI increased little with d. For the linear approximation architecture, the results showed that our method outperformed the LSM with respect to the required polynomial degree in pricing. This phenomenon is crucial when the number of underlying assets is large. For the computation time, although LFPI is slower than the LSM algorithm with same polynomial degree, LSM needed larger polynomial degrees for accurate results so that our method returned accurate results faster than the LSM in high-dimensional examples. In the case of nonlinear architecture, NNFPI generally outperformed DOS, which has similar network architecture.

5.2. Stochastic Volatility Model

This subsection is devoted to the American options on the Heston model [33], a well-known symmetric stochastic volatility model in options pricing. The evolution of underlying asset $S_t$ and instantaneous variance ${\nu }_t$ is described by the following stochastic differential equation

$$\begin{array}{cc}\hfill \mathrm{d}{S}_t& =r{S}_t\mathrm{d}t+\sqrt{{\nu }_t}{S}_t\mathrm{d}{W}_t^1,\hfill \\ \hfill \mathrm{d}{\nu }_t& =\kappa \left(\theta -{\nu }_t\right)\mathrm{d}t+\xi \sqrt{{\nu }_t}\mathrm{d}{W}_t^2,\hfill \end{array}$$

(16)

where $r\ge 0,\kappa >0,\theta >0,\xi >0$, $W_t^1$ and $W_t^2$ are d-dimensional Brownian motions. For a reliable price reference for high-dimensional American options, we used the same parameter settings as in the cases studied in Herrera et al. [28]. The max call options and geometric put options were considered in the experiments. Specifically, we choose the parameters $T=1$; $N=10$; $\kappa =2$; $\theta =0.01$; $\xi =0.2$; the initial stock price $S_0=100$; the initial variance ${\nu }_0=0.01$; $r=0\%$ for max call options; and $r=2\%$ for geometric put options. We assumed that the dynamics of different assets were independent, the correlation between the Brownian motion driving the price process and variance process of single asset was $\rho =-0.3$. In the computation we used M = 20,000; $M^{{}^{\prime }}$ = 100,000; and $J=5$ and same network architecture as in last subsection.

To obtain a Markovian model, we included price and variance as inputs. In practical, stochastic volatility models, they need to be calibrated from observed data; then, our algorithm can be applied to sample data generated from the models. In the experiments, we tested our method under max call options and geometric put options with $K=100$. The results are reported in

Table 4. Pricing results for d-dimensional max-call options in the Heston model.

d	Ref.	LSM-2	Time	LSM-3	Time	LFPI	Time	DOS	Time	NNFPI	Time
5	8.33	8.252	5.7	8.258	6.3	8.262	6.1	8.192	12.8	8.207	19.8
10	11.83	11.460	27.4	11.662	76.6	11.623	23.1	11.286	13.5	11.796	21.0
20		14.992	89.0			15.058	132.0	14.885	15.6	15.330	26.5
30								17.091	18.4	17.465	27.5
40								18.485	21.5	18.974	31.7
50	20.09							19.563	23.7	20.100	35.0
80								21.993	34.3	22.487	43.5
100	23.69							22.927	40.3	23.613	50.1

and

Table 5. Pricing results for d-dimensional geometric put options in the Heston model.

d	Ref.	LSM-2	Time	LSM-3	Time	LFPI	Time	DOS	Time	NNFPI	Time
5	2.43	2.366	4.9	2.368	5.0	2.420	6.0	2.309	12.8	2.443	19.7
10	2.01	1.773	21.1	1.959	52.6	1.986	22.5	1.931	13.6	2.015	21.0
20	1.71	1.631	67.5			1.658	134.0	1.643	15.4	1.712	26.5
30								1.292	18.1	1.593	27.5
40								1.304	20.8	1.522	31.6
50	1.48							1.291	22.5	1.474	34.9
80								1.194	33.3	1.426	43.3
100	1.40							1.141	39.9	1.402	49.8

. The results were similar to those under the multi-dimensional Black–Scholes model. The pricing results obtained from LFPI and NNFPI were close to the reference values. The LFPI computation time was smaller than that of NNFPI for low-dimensional situations, but the NNFPI was more efficient for $d\ge 10$. It could be seen that NNFPI is generally the most accurate method for high-dimensional American options, and the computation time of NNFPI is close to that of DOS, especially for large d. For the linear approximation architecture, the results showed that LFPI outperformed LSM. Note that our method had much fewer trainable parameters.

5.3. Convergence with Respect to the Number of Iteration

In this subsection, we study numerically the convergence of our method with respect to the hyperparameter J. We considered max-call options under the Black–Scholes model with same parameters setting as in Section 5.1. Figure 1 presents the results for LFPI with $d=5$ and NNFPI with $d=20$. The errors were computed with respect to the final value. It could be seen that our method converged fast with respect to the number of iterations. The results confirmed that limiting the number of iterations below to 5 was reasonable.

6. Conclusions

We introduced a novel method for American options based on reinforcement learning that was accurate and efficient in high-dimensional situations. We provided a convergence analysis for the algorithms in the number of training samples and iterations. We also considered the applicability of the algorithm and carried out comprehensive numerical experiments in multivariate Black-Scholes and Heston models. The results showed that (a) the NNFPI achieved good performance in high-dimensional situations, and the LFPI outperformed the LSM in the pricing of American options; (b) the algorithm had high efficiency and accuracy under different model assumptions; (c) our algorithm had a fast convergence rate with respect to the number of iterations; (d) the continuation values can be approximated with a fraction of the parameters by using a function of time and the underlying process. To summarize, the results reconfirmed that reinforcement learning methods surpass backward induction methods for pricing of high-dimensional American options.

There are several directions for future research. First, upper price bounds and confidence interval could be constructed based on the approximating the optimal stopping time. Furthermore, it would be desirable to remove the condition that the payoff function is bounded in $L^{\infty }$. One idea is to use the truncation technique in Zanger [34]. The proofs are technically more challenging and are left for future research.