Stock Portfolio Management by Using Fuzzy Ensemble Deep Reinforcement Learning Algorithm

Abstract

The research objective of this article is to train a computer (agent) with market information data so it can learn trading strategies and beat the market index in stock trading without having to make any prediction on market moves. The approach assumes no trading knowledge, so the agent will only learn from conducting trading with historical data. In this work, we address this task by considering Reinforcement Learning (RL) algorithms for stock portfolio management. We first generate a three-dimension fuzzy vector to describe the current trend for each stock. Then the fuzzy terms, along with other stock market features, such as prices, volumes, and technical indicators, were used as the input for five algorithms, including Advantage Actor-Critic, Trust Region Policy Optimization, Proximal Policy Optimization, Actor-Critic Using Kronecker Factored Trust Region, and Deep Deterministic Policy Gradient. An average ensemble method was applied to obtain trading actions. We set SP100 component stocks as the portfolio pool and used 11 years of daily data to train the model and simulate the trading. Our method demonstrated better performance than the two benchmark methods and each individual algorithm without fuzzy extension. In practice, real market traders could use the trained model to make inferences and conduct trading, then retrain the model once in a while since training such models is time0consuming but making inferences is nearly simultaneous.

1. Introduction

The goal of stock trading is to optimize some relevant measures of performance, such as profit and utility (Moody and Wu 1997). Traditionally, trading is performed by humans, who are classified as fundamentalists and technicalists (Murphy 1999), depending on the strategies one use. Stock trades based on algorithms have received attention and have started to take over the finance industry in the last two decades with the development of online trading platforms (Creamer and Freund 2010; Ozbayoglu et al. 2020). Since the successes that Deep Learning (DL) and Reinforcement Learning (RL) algorithms achieved in areas such as Go game (Silver et al. 2016) and video games (Vinyals et al. 2017), researchers have wanted to train computers to be skilled traders using these algorithms.

One of the ideas is supervised learning, for example, trading based on the prediction of market trends. Studies by (Oelschläger and Adam 2021) and (Leung et al. 2000) used statistical and DL algorithms to label the market as bullish and bearish; while (Sezer et al. 2017) provided a finer labeling by further giving each direction three levels. Another type of effort was given to the prediction of stock prices. Long short-term memory (LSTM) networks were used for the prediction of the Standard and Poor 500 (SP500) index in (Fischer and Krauss 2018). It, together with a number of other DL models, was implemented for the prediction of ten stock prices in (Balaji et al. 2018). In (Singh et al. 2023), the author implemented the Logistic model, Gompertz model, and Harvey model to predict the increasing trend of the adoption of Unified Payment Interface. However, maximizing profit by predicting market behavior or prices was a suboptimal approach since the prediction step itself suffered from forecast errors (Moody and Saffell 1998). Therefore, a number of past works have been performed to train the program to trade directly by optimizing some objective functions. The authors in (Moody et al. 1998) presented an adaptive algorithm called Recurrent Reinforcement Learning (RRL) to train the trading system for a single asset; (Gold 2003) investigated the foreign currency exchange market by using RRL; In (Zhang et al. 2020), the authors designed trading strategies by adopting Deep RL algorithms for future contracts and tested them on 50 future contracts individually. Although these approaches were able to obtain remarkable returns, their limitation was that the trading was only on one asset; that is, it was an allocation of money between a risk-free investment and only one risky investment.

Financial theories state that to achieve better investment return, one would need to diversify the investment or create a portfolio that usually has much smaller volatility or variance of returns compared to a single asset (Rubinstein 2002). In (Iliev et al. 2023), the authors provided a creative way of constructing the pool for building a portfolio. They focused on stocks that were best related to the concepts of Circular Economy and Environmental, Social, and Governmental principles and narrowed down to a top 100 stocks from a list of 6000 globally traded ones. The authors were able to compare three portfolios built from these 100 stocks to some benchmark measurements, such as a BlackRock ETF, and found that all portfolios outperformed them with larger returns and equal or smaller volatilities. In (Li and Hoi 2014), the author grouped related work into four categories depending on the theories or models that the strategies followed, including Follow-the-Winner, Follow-the-Loser, Pattern-Matching, and Meta-Learning. In (Ozbayoglu et al. 2020), the author also surveyed several studies regarding portfolio management. For instance, in (Fu et al. 2018), multiple stocks were analyzed and selected by using various DL algorithms; (Lee and Yoo 2020) proposed threshold-based portfolios that were built from a LSTM-based prediction model; (Liang et al. 2018) proposed an Adversarial Training method that was adopted from Policy Gradient algorithm and tested it on the Chinese stock market.

Jiang and Liang (2017) applied Convolutional Neural Networks (CNN) to the Deterministic Policy Gradient (DPG) algorithm to develop a portfolio of 12 cryptocurrencies and received remarkable return rates. In (Nikolaev and Petrova 2021), the author classified the stocks’ weekly change into seven categories and implemented two models involving CNN to make predictions. Their results showed the effectiveness of CNN in predicting graphical representations such as stock price trends. Yang et al. (2020) used an ensemble strategy of three RL algorithms for a portfolio of 30 Dow Jones stocks. Our work was to build a model-free algorithm that can manage a stock portfolio and achieve high returns and small volatility without predicting stock prices. Sharpe ratio, a measurement introduced by Sharpe (1998), was computed by the expected return relative to its standard deviation. It is an effective measurement to evaluate portfolios with similar risk characteristics over the same period. For example, the authors in (Em et al. 2022) created some metrics for some and used the Sharpe ratio and its variants to evaluate performance differences among bond portfolios in the global market.

In sum, there is room to improve the existing literature. The originality of this proposed method includes the following: first, no prediction is needed, and attention is paid only to trading since predicting stocks’ prices has errors and is different from making profits. Secondly, the method is able to include a much larger number of stocks to select from. We use SP100 index stocks, but the method could be scaled up easily to include more. Third, we want to include market trend information, such as bullish and bearish, quantitatively and continuously to help on decision-making. Last but not least, the method assumes no trading or financial knowledge. The trading agent (machine) will be learning by trial and error.

The motivation of this work is to use several RL algorithms to train a computer to manage a portfolio of a relatively large number of stocks and to receive higher measures of trading, such as profit or Sharpe ratio, by using only past data and assuming no other trading knowledge. To incorporate the market direction, e.g., bullish/bearish, but in a more quantitative way, we incorporated fuzzy extension. We also use average ensemble to lower the volatility and, therefore, gain higher Sharpe ratios. The remaining parts of this work are organized as follows. Section 2 reviewed RL algorithms and proposed our method; Section 3 presented the data sets and data preprocessing; Section 4 illustrated the results; and Section 5 summarized the work in conclusions and some possible future work.

2. Methodology

2.1. Model Terminology and Assumptions

We explain some notations and their meanings as well as assumptions needed for them in our problem.

• Reward, denoted by r, is a scalar feedback to reflect the performance of actions. The goal of most RL algorithms is to optimize the cumulative reward. For the stock trading problem, it depends on the current market and portfolio information and the trading actions taken, and, therefore, it is denoted as $r\left({\mathbf{s}}_{\mathbf{t}-\mathbf{1}},{\mathbf{a}}_{\mathbf{t}-\mathbf{1}},{\mathbf{s}}_{\mathbf{t}}\right)$, meaning the change in the total portfolio value when facing state ${\mathbf{s}}_{\mathbf{t}-\mathbf{1}}$ and taking action ${\mathbf{a}}_{\mathbf{t}-\mathbf{1}}$, and, therefore, the arriving state ${\mathbf{s}}_{\mathbf{t}}$, where ${\mathbf{s}}_{\mathbf{t}}$ and ${\mathbf{a}}_{\mathbf{t}}$ are defined later in this section.
• We consider a pool of M stocks over T time periods and assume a trader may buy or sell stocks but may not short sell. For each stock i, and time t, where $i=1,2\dots M$ and $t=1,2\dots T$, let $p_t^i$ be the price and $n_t^i\ge 0$ be the numbers of share holding in the trader’s portfolio. Respectively, combining $p_t^i$ and $n_t^i$ yields ${\mathbf{p}}_{\mathbf{t}}={[p_t^1,p_t^2,\dots ,p_t^M]}^T$ and ${\mathbf{n}}_{\mathbf{t}}={[n_t^1,n_t^2,\dots ,n_t^M]}^T$. Then, the portfolio’s return at time t is

  $$R_t={({\mathbf{p}}_{\mathbf{t}}-{\mathbf{p}}_{\mathbf{t}-\mathbf{1}})}^T{\mathbf{n}}_{\mathbf{t}}-cost·{{\mathbf{p}}_{\mathbf{t}-\mathbf{1}}}^T|{\mathbf{n}}_{\mathbf{t}}-{\mathbf{n}}_{\mathbf{t}-\mathbf{1}}|$$

  (1)

  where $cost$ is the transaction cost rate, usually ranging from $0\%$ to $1\%$. Therefore, our objective is to maximize the total return, the sum of period-wise returns over all periods.

  $$J=\sum _{t=1}^T{R}_t$$

  (2)
• We define s as the environment state space vector, which will be the model input. In our problem, the environment state space includes two types of components: the quantities that are dependent on trading actions, e.g., account cash balance and stock shareholdings, which is denoted by vector ${\mathbf{s}}_{\mathbf{1}}$; and the quantities that are independent of trading actions, e.g., stock prices and technical indicators, such as MACD, denoted by vector ${\mathbf{s}}_{\mathbf{2}}$. Thus, $\mathbf{s}=[{\mathbf{s}}_{\mathbf{1}},{\mathbf{s}}_{\mathbf{2}}]$.
• Action a is a vector with a dimension equaling the number of stocks, M, whose positive components indicate buying and negative components indicate selling of the corresponding stocks. Each component of action a is a number between $-1$ and 1, indicating the number of shares we buy or sell. For example, a number $-0.1$ would mean that we sell $0.1H$ shares of a stock, where H is a hyperparameter defining the maximum amount of shares for any single transaction. There are usually two ways to claim the maximum: the first way is to choose the same number of shares for all stocks; for example, $H=1000$ would mean a single transaction can, at most, buy or sell 1000 shares of any stock; the second way is to choose the same value for all stocks, for example, a single transaction can at most have a total value of 100,000 USD. In this case, the number of maximum shares for each stock is different and is computed by 100,000 divided by the current stock price. In this work, we used the second way. $\pi \left(\mathbf{a}\right|\mathbf{s})$ is the stochastic policy that gives the probability of action a when facing state s. We denote ${\mathbf{a}}_{\mathbf{t}}$ as the action taken in period t. In our problem, for example, say for stock A, the action could be (1) buying $k_1$ shares of the stock and resulting in $n_t=n_{t-1}+k_1$ shares, or (2) holding, which results in $n_t=n_{t-1}$, or (3) selling $k_2$ stocks and yielding $n_t=n_{t-1}-k_2$. The action will also update the cash balance and other stocks and, therefore, reach the state ${\mathbf{s}}_{\mathbf{t}}$. This action results in a reward, denoted by $r\left({\mathbf{s}}_{\mathbf{t}-\mathbf{1}},{\mathbf{a}}_{\mathbf{t}-\mathbf{1}},{\mathbf{s}}_{\mathbf{t}}\right)$, which is the change in the portfolio’s total value.
• We also make the following assumptions and constraints. There will be no short selling. All selling transactions in the action vector are conducted before buying, and all buying transactions’ total value does not exceed the current cash amount plus the total value of all sell transactions. That is, the traders could use the cash received from selling stocks in a period to buy other stocks in the same period, but they could not borrow other cash. In addition, every single transaction’s total value does not exceed $K\%$ of the total portfolio value, where K is a hyperparameter.
• Value function $V^{\pi }\left(\mathbf{s}\right)$ and Q function $Q^{\pi }(\mathbf{s},\mathbf{a})$ are defined to be the value function of state s and the value function of state-action pair $(\mathbf{s},\mathbf{a})$ by following policy $\pi$. In stock trading, $V^{\pi }\left(\mathbf{s}\right)$ means the expected future gain when facing state s, and $Q^{\pi }(\mathbf{s},\mathbf{a})$ means the expected return when facing state s and taking action a if the trader follows policy $\pi$.
• Advantage function, denoted by $A^{\pi }(\mathbf{s},\mathbf{a})=Q^{\pi }(\mathbf{s},\mathbf{a})-V^{\pi }\left(\mathbf{s}\right)$, can be thought of as how much better state s would be if we take action a opposed to average. In many algorithms, subtracting the value function is called a baseline. In our approach, value function, Q function, and advantage function are predefined and embedded in the programming packages and need no extra attention.

2.2. Reinforcement Learning Algorithms

A review of the classic Policy Gradient algorithm will be given by using the notations in the previous section, then followed by five RL algorithms, some of which are highly related to the classic Policy Gradient. These five algorithms are all suitable for continuous action and are used in our method.

In classic Policy Gradient, the objective of many RL learning algorithms is to maximize the expected value of the total reward $J=E\left[r\right(\tau \left)\right]$ where $\tau$ represents a trajectory of states and actions, and $r\left(\tau \right)$ is the total reward by taking this trajectory. Policy Gradient Theorem (Sutton and Barto 2018) provided a simplified form of the gradient of J that allowed the update of parameter $\theta$ by using gradient ascending, $\theta \leftarrow \theta +\eta {\nabla }_{\theta }J$. The theorem states that the gradient term is proportional to $E[{\sum }_t{Q}^{\pi }(s_t,a_t){\nabla }_{\theta }ln{\pi }_{\theta }\left(a_t\right|s_t)]$. One may subtract a base-line value function $V^{\pi }\left(s_t\right)$ from function $Q^{\pi }(s,a)$ to make negative values possible; then the Policy Gradient could be expressed as follows, which is then used in the gradient ascending method.

$${\nabla }_{\theta }J=E[(Q^{\pi }(s,a)-V^{\pi }\left(s_t\right)){\nabla }_{\theta }ln{\pi }_{\theta }\left(a\right|s)]$$

(3)

(1)

Advantage Actor-Critic (A2C)

A2C (Mnih et al. 2016) is an on-policy actor-critic algorithm. The method uses the advantage function to replace the return function with the advantage function $A(s_t,a_t)=Q(s_t,a_t)-V\left(s_t\right)$ in the Policy Gradient, which then becomes

$${\nabla }_{\theta }J=E[\sum _{t=1}^T(Q(s_t,a_t)-V\left(s_t\right)\nabla {\pi }_{\theta }\left(a_t\right|s_t)].$$

(4)

Next, $Q(s_t,a_t)$ can be approximated by the reward that is observed at time t, $r_t$ plus the value at the next time, $V\left(s_{t+1}\right)$. Thus, the gradient for parameter $\theta$ updating is

$${\nabla }_{\theta }J=E[\sum _{t=1}^T(r_t+V\left(s_{t+1}\right)-V\left(s_t\right)\nabla {\pi }_{\theta }\left(a_t\right|s_t)].$$

(5)

The algorithm uses one network to estimate the policy and another network to estimate the value function, which may share some parameters.

(2)

Trust Region Policy Optimization (TRPO)

TRPO (Schulman et al. 2015) is an off-policy algorithm that uses an old policy, denoted as ${\pi }_{{\theta }^{\prime }}$, to collect data for updating a new policy ${\pi }_{\theta }$. The expected return object function (Sutton and Barto 2018) could be written in terms of both policies as

$$J\left(\theta \right)=E_t(\frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }^{\prime }}\left(a_t\right|s_t)}\widehat{A}(s_t,a_t)),$$

(6)

where $\widehat{A}(s_t,a_t)$ is the estimated advantage function by using importance sampling (Kloek and Van Dijk 1978; Van Dijk and Kloek 1983), which allows approximating the expectation of the distribution of interest from samples generated from a different distribution. However, the expectation might be unstable if these two approximations are too far away; therefore, the two policies need to be somehow closed in the above objective function. TRPO uses a trust region defined by KL divergence (Kakade and Langford 2002) to restrict the choice of the new policy. In particular, the algorithm maximizes $J\left(\theta \right)$ among $\theta$ whose distance defined KL divergence to ${\theta }^{\prime }$ is no larger than $\delta$, or $D_{KL}(\theta ,{\theta }^{\prime })<\delta$, where $\delta$ is a small positive hyperparameter.

(3)

Proximal Policy Optimization (PPO)

The idea of PPO (Schulman et al. 2017) started from the same objective function as in TRPO $J\left(\theta \right)=E_t(\frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }^{\prime }}\left(a_t\right|s_t)}\widehat{A}(s_t,a_t))$ and, therefore, would avoid have old and new policies too far away. Instead of using a trust region, PPO penalizes the policies distance by adding a clip function and using objected function

$$J^{clip}\left(\theta \right)=E_t[min\{\frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }^{\prime }}\left(a_t\right|s_t)}\widehat{A}(s_t,a_t),clip(\frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }^{\prime }}\left(a_t\right|s_t)},1-ϵ,1+ϵ)\widehat{A}(s_t,a_t)\}]$$

(7)

where $ϵ$ is a small positive number around 0.1–0.2, function $clip(a,b,c)$ returns a if $b<a<c$, returns b if $a<b$, and returns c if $a>c$. Therefore, it clips the ratio between new and old policies to be within $1-ϵ$ and $1+ϵ$. The method was proven to have better sample complexity and be simpler to implement than some other online Policy Gradient methods and provides a favorable balance between sample complexity, simplicity, and wall time (Achiami et al. 2016; Schulman et al. 2017).

(4)

Actor-Critic Using Kronecker Factored Trust Region (ACKTR)

ACTKR (Wu et al. 2017) is a natural Policy Gradient method with trust region optimization and uses a more accurate second-order optimization. Similar to A2C, ACTKR uses two networks for policy and another network for the advantage function. On some classical testing tasks, the algorithm obtained two to three-fold sample efficiency and only had 10–25% more computation cost due to the Kronecker-factored approximated curvature (K-FAC) (Martens and Grosse 2015).

The algorithm uses $\theta \to \theta +\eta F^{-1}{\nabla }_{\theta }J$ for updating, where $\eta$ is the learning rate, F is the Fisher information matrix that will be approximated by K-FAC (Martens and Grosse 2015), and ${\nabla }_{\theta }J$ is the Policy Gradient. The learning rate $\eta$ in ACKTR involves trust region radius hyperparameter $\delta$ and is given by $\eta =\sqrt{\frac{2\eta }{{\left({\nabla }_{\theta }J\right)}^T{F}^{-1}{\nabla }_{\theta }J}}$.

(5)

Deep Deterministic Policy Gradient (DDPG)

DDPG (Lillicrap et al. 2015) uses deep neural networks for fitting a policy $\pi \left(s_t\right)$ called the actor with actor parameters ${\theta }^{\pi }$ and critic Q function, $Q(S_t,a_t)$ with critic parameters ${\theta }^Q$. It adapts Deep Q-Learning (DQN) to continuous action cases. The method is an off-policy algorithm using a replay buffer that stores (state, action, reward, next state) or $(s_t,a_t,r_t,s_{t+1})$ to train networks and has a deterministic actor policy function $\pi \left(s_t\right)$.

To update parameter ${\theta }^Q$, DDPG uses the recursive relationship from the Bellman equation

$$Q(s_t,a_t)=E[r(s_t,a_t)+\gamma Q(s_{t+1},a_{t+1})]$$

(8)

where $\gamma$ is a discount factor whose value is usually between $0.9$ and $0.99$ and minimizes the mean square error between $Q(s_t,a_t)$ and $r(s_t,a_t)+Q(s_{t+1},\pi \left(s_{t+1}\right))$, or

$$L\left({\theta }^Q\right)=E\left[{(r(s_t,a_t)+Q(s_{t+1},\pi \left(s_{t+1}\right))-Q(s_t,a_t))}^2\right]$$

(9)

Then, the method updates the parameters of the actor, using the gradient of expected return with respect to ${\theta }^{\pi }$

$${\nabla }_{{\theta }^{\pi }}=E_t\left[{\nabla }_{a}Q(s_t,a_t|{\theta }^Q){\nabla }_{{\theta }^{\pi }}\pi \left(s_t\right|{\theta }^{\pi })\right]$$

(10)

2.3. Fuzzy Extension

It has been known that trading strategies work differently in a bullish market and in a bearish market (Chen et al. 2020; Dai et al. 2012). Effort has been made for bullish/bearish market classification(Chong et al. 2017; Oelschläger and Adam 2021; Wu et al. 2020). However, many classification methods could only tell until a turning signal was shown, and they only qualitatively classified the market. The idea of a fuzzy set state is where the state of the membership function, whose range is between 0 and 1, measures the grade that an element belongs to a set (Pal and Bezdek 1994). Our approach, instead of classifying each trading day as bullish, bearish, or oscillation, wanted to quantify, for example, a trading day to be 0.7 oscillation degree, 0.2 bullish degree, and 0.1 bearish degree. To achieve this, we adopted the approach in (Deng et al. 2016; Lin et al. 2006), as follows:

For each stock, first, find the rate of return for each of the time periods by $r_t=\frac{p_t-p_{t-1}}{p_{t-1}}$. To obtain its degree for $k=3$ fuzzy sets representing bullish, bearish, and oscillation markets for a time period t:

• choose moving window length $T_0$, and using k-mean clustering on the rates of return $r_t,r_{t-1},r_{t-2},\dots ,r_{t-T_0+1}$ to get $k=3$ groups with means and standard deviations, $m_j$ and $s_j$, $j=1,2,3$, respectively.
• For period $t_0$, compute fuzzy degree $f^j$, $j=1,2,3$ via the Gaussian membership function (Krasnyuk et al. 2022; Lin et al. 2006) as

  $$f^j=e^{-{(r_t-m_j)}^2/s_j^2}$$

  (11)
• These fuzzy degrees, $f_t^1,f_t^2,f_t^3$, are added as fuzzy extensions to obtain the final state space, which was stated in Section 3.
• repeat this process for all stocks and for all periods.

2.4. Ensemble

We use the RL algorithms, including A2C, PPO, DDPG, ACKTR, and TRPO, which are all suitable for continuous action to train the agent individually. These algorithms each have their strengths and weaknesses, and even for stock trading, they may behave well in one period but bad in another (Yang et al. 2020), which we observe in our initial trials as well. We train the model individually and take the average over the five output actions as our final action decision. Concretely, for any time period t, the model implements each of the five algorithms mentioned above on a training data set and obtains five action vector outputs, say ${\mathbf{a}}_{\mathbf{1}}$, ${\mathbf{a}}_{\mathbf{2}}$, ${\mathbf{a}}_{\mathbf{3}}$, ${\mathbf{a}}_{\mathbf{4}}$, and ${\mathbf{a}}_{\mathbf{5}}$, with each of them representing the decision for buying/holding/selling each of the stocks and the shares of the transaction as defined in the Model Terminology section. Then, the ensemble technic averages the five actions and yields action $\overline{\mathbf{a}}=\frac{{\mathbf{a}}_{\mathbf{1}}+{\mathbf{a}}_{\mathbf{2}}+{\mathbf{a}}_{\mathbf{3}}+{\mathbf{a}}_{\mathbf{4}}+{\mathbf{a}}_{\mathbf{5}}}{5}$ as the decision for trading in period t. It is worth noting that since each of the five ${\mathbf{a}}_{\mathbf{i}}$ satisfies the trading constraints claimed in the Model Terminology section, their average action $\overline{\mathbf{a}}$ will also satisfy them.

A complete diagram of our work-frame is shown in Figure 1. First, environment information is observed, and fuzzy extension is computed and added to feed the model as input for trading period t. Second, five RL algorithms are trained and provide five actions, which are then averaged to receive the final action decision. Next, this action is conducted to yield a reward and update part of the environment’s information, such as shareholdings and cash. Finally, the updated environment information, together with the reward, is used for period $t+1$, and the whole process repeats until all time periods are used.

3. Datasets and Preprocessing

We considered all component stocks in the S&P 100 index during the period of 2011–2021, except for those that were listed on the market later than 2011. The data were available at the daily level from yahoo finance, which yielded $M=94$ stocks that made up our portfolio pool with $T=2537$ trading days. For each trading day, t, we used $M=96$ dimension vectors to denote the shareholdings ${\mathbf{n}}_{\mathbf{t}}$, the daily open price ${\mathbf{o}}_{\mathbf{t}}$, high price ${\mathbf{h}}_{\mathbf{t}}$, low price ${\mathbf{l}}_{\mathbf{t}}$, adjusted closed prices ${\mathbf{p}}_{\mathbf{t}}$, and volumes ${\mathbf{v}}_{\mathbf{t}}$. Following the steps in Section 2.3 for the fuzzy extensions, we chose a moving window length of $T_0=60$, and obtained three 96-d vectors, ${\mathbf{f}}_{\mathbf{t}}^{\mathbf{1}},{\mathbf{f}}_{\mathbf{t}}^{\mathbf{2}},{\mathbf{f}}_{\mathbf{t}}^{\mathbf{3}}$. A snapshot of the computation for the fuzzy extensions on one sample trading day is given in Figure 2 and

Table 1. The fourth column shows the fuzzy extension of AAPL on 8 April 2019, which was based on a daily return of 0.0115, or 1.15%.

	Cluster Mean	Cluster SD	Fuzzy Extension Value
increasing	0.0411	0.0133	$e^{-{(0.0115-0.0411)}^2/{0.0133}^2}=0.0083$
oscillation	0.0029	0.01	$e^{-{(0.0115-0.0029)}^2/{0.01}^2}=0.4369$
decreasing	−0.0323	0.0108	$e^{-{(0.0115+0.0323)}^2/{0.0108}^2}=4.9\times {10}^{-8}$

.

There were a larger number of technical indicators to measure stock movement in various ways, among which we chose the ones that were considered related to stock trends and buy/sell signals (Di 2014; Vargas et al. 2018), including Moving Average Convergence Divergence (MACD) $\mathbf{M}\mathbf{A}\mathbf{C}{\mathbf{D}}_{\mathbf{t}}$, Williams Overbought/Oversold index (WR) $\mathbf{W}{\mathbf{R}}_{\mathbf{t}}$, Relative Strength Index (RSI) $\mathbf{R}\mathbf{S}{\mathbf{I}}_{\mathbf{t}}$, Commodity Channel Index (CCI) $\mathbf{C}\mathbf{C}{\mathbf{I}}_{\mathbf{t}}$, and Average Directional Index (ADX) $\mathbf{A}\mathbf{D}{\mathbf{X}}_{\mathbf{t}}$. Combining these indicators into one vector yielded $\mathbf{T}{\mathbf{I}}_{\mathbf{t}}$. The daily account cash balance is denoted by $c_t$. Following the notations in Section 2.1, we have ${\mathbf{s}}_{\mathbf{1}}=[c_t,{\mathbf{n}}_{\mathbf{t}}]$, and ${\mathbf{s}}_{\mathbf{2}}=[{\mathbf{o}}_{\mathbf{t}},{\mathbf{h}}_{\mathbf{t}},{\mathbf{l}}_{\mathbf{t}},{\mathbf{p}}_{\mathbf{t}},{\mathbf{v}}_{\mathbf{t}},\mathbf{T}{\mathbf{I}}_{\mathbf{t}}]$.

Finally, we obtained the state space for our RL algorithms as $\mathbf{s}=[f_t^1,f_t^2,f_t^3,{\mathbf{s}}_{\mathbf{1}},{\mathbf{s}}_{\mathbf{2}}]=[f_t^1,f_t^2,f_t^3,c_t,n_t,o_t,h_t,l_t,p_t,v_t,T{I}_t]$, which has dimensions of $14\times 94+1=1317$.

4. Results

For the first round of training, the dataset was split into a training interval from the beginning of 2011 to the end of 2019, a testing interval ranging from 18 December 2020 to 20 January 2021 (20 trading days), and a validation interval containing any trading day in between, which had about 250 days. In each of the following rounds, training, testing, and validation intervals were all moved forward by 20 trading days. The process repeated until the last day of the data was used. Notice the reason that we chose 18 December 2020 as the first testing trading day was to ensure that suing periods of 20 trading days, the last trading day could exactly be 31 December 2021. A diagram showing the training/validation/testing periods update is shown in Figure 3.

We used a hypothetical initial cash amount of 10 million US dollars and conducted the trial 100 times. The histograms for final return rates and Sharpe ratios were given in Figure 4 and Figure 5. The red vertical line in Figure 4 indicated a SP100 rate of return during the same period. We observed that most of the trials were able to gain higher rates of return than the index. In Figure 5, the red vertical line marked the SP100 index Sharpe ratio, which was about the same as the average Sharpe ratio for our 100 trials.

We took the average portfolio values over the 100 trials at the end of each trading period. A time series plot of our results is shown in Figure 6, as well as two benchmark portfolios with SP100 representing buying and holding the S&P100 index and rebalance (Davis and Nairn 2012) being the strategy that money was evenly spread over all stocks and the weights were rebalanced to maintain the even value allocation at the end of each trading interval. One could see that the average final portfolio value was over 14.9 million or a return rate of $49\%$. A $95\%$ simultaneous confidence band for the portfolio value using Bonferroni’s adjustment (Dunn 1961) was given by the shaded region. The final value for the S&P100 index was 12.9 million or a return rate of $29\%$; the rebalancing strategy yielded a return rate of $29.7\%$. We could see that our method’s confidence band was able to stay above the S&P100 index and rebalancing for the whole period. Thus, our method was able to achieve a higher return while keeping about the same Sharpe ratio as the market index.

For comparison purposes, we respectively used each of the five classic RL methods without a fuzzy extension to conduct the experiment 100 times and obtained their mean portfolio values. The result is given in Figure 7. One can see that only the ACKTR model was able to have a final portfolio value a bit higher than our method, and it was still lower than the upper limit of the $95\%$ confidence band. However, all methods, including ACKTR, showed more volatility, especially during the last three periods when the market had large oscillations. More quantitative results are summarized in

Table 2. The results of total returns and Sharpe ratios.

	Ours	SP100	Rebalance	A2C	ACKTR	DDPG	TRPO	PPO
Return Rate	49.4%	29.4%	29.9%	38.2%	44.8%	27.9%	35.9%	42.3%
Sharpe Ratio	0.627	0.622	0.679	0.452	0.565	0.347	0.478	0.563

. One could see that our method had a Sharpe ratio of 0.627, which was close to the S&P100 index Sharpe ratio of 0.622 and rebalancing portfolio Sharpe ratio of 0.679, but had a much higher return rate. None of the five RL methods had a Sharpe ratio of more than 0.57, with the lowest Sharpe ratio of 0.347 belonging to DDPG, which indicated that to reach similar expected return levels, one must suffer from larger volatilities.

5. Discussion and Conclusions

In this work, we proposed and trained an agent that could conduct automated stock trading from a pool of stocks to form a portfolio and beat the market. The work had two theoretical contributions.

First, the method was able to handle a vast number of stocks and gain a satisfactory return with no more risks. In our case, the model built and managed a portfolio from S&P 100 component stocks. The model could easily include an even larger number of stocks. In the existing literature, the difficulties of dealing with many of stocks came from two aspects. The first one was the restriction of the model. For example, time series type analysis or Markov Decision Process were naturally limited to small dimension state spaces. The other one is the computing capacity of the computer. In our approach, the most time-consuming part was training, but this training and retraining could be performed during closed market periods daily or weekly. The trading operations were nearly simultaneous after the model was trained. The model earned an average return of about $49\%$ during a one-year period. This was significantly higher than the return of the index and rebalanced equal-weight portfolio, which was about $29\%$. Their associated Sharpe ratios were similar and around 0.6.

Second, the method incorporated ensemble technic and fuzzy extension in addition to existing RL algorithms. The averaging ensemble technic reduced the volatility compared to any single algorithm, and the fuzzy extension itself also provided a numerical way to describe the market trend as opposed to the traditional categorical classifications as bullish/bearish/oscillation. As the comparison results suggested, our method achieved higher returns and Sharpe ratios than each single RL algorithm without using the fuzzy extension.

Our work added evidence that machines have the potential to beat humans and provided investors with a practical way to make profit in the market. Most investors, especially individual investors and small funds used fundamental analysis and technical analysis techniques for trading. It was not easy for them to beat the indices consistently, such as S&P 100, S&P 500, Dow Jones Industrial Average, and NASDAQ Composite. The reasons were that for fundamental analysis, most information they collected was public information and, therefore, was already reflected in current prices (price-in); while for technical analysis, the patterns or technical indicators might be used or interpreted in more than one way, which might contradict others. Our method assumed no fundamental analysis, technical analysis, or other trading strategies, creating a possibility for them to achieve higher returns without running into more risks. The approach could also be used by large investment companies that have more and quicker information and data on a larger scale. It was desirable that the model could be trained to achieve even higher performance.

This work took a step into algorithm-based trading; however, it has limitations, which were from the nature of the market assumptions we used. We assumed that in the market: (1) Each of our transaction amounts was insignificant compared to the total volumes, thus had no impact on the market prices, and (2) we were able to sell or buy stocks right at the stocks’ closing prices and the transactions were executed immediately.

The limitation of the first assumption was the scaling problem. When the portfolio value became large, this was no longer true. While it was somehow realistic as long as the total portfolio values were small, which in this work was 10 million US dollars, or the stocks volumes and market caps were reasonably large, which all S&P 100 stocks should satisfy. The limitation of the second assumption was that trading at the closing prices was not always possible, and only using daily closing prices would ignore the intraday trends and extended-hours trends and, therefore, miss many profitable opportunities.

There could be three possible ways for future study. The first way is based on the second assumption mentioned above. For example, if one wants to use computer algorithms for real automated trading, it would be necessary to use market information on finer time intervals, such as hourly data or minute data. Secondly, we could include training periods that include some recent long bullish and long bearish markets and tune the parameters so the model could be more robust. Historically speaking, U.S. stock markets had much longer bullish periods than bearish periods, which was also true in our data time frame. We are observing a long ongoing bearish market starting year 2022. Thus the authors think this period would be valuable for any algorithm-based trading strategy and would like to include it in the future. The third one is to improve the algorithms themselves. This includes adopting existing ones, such as implementing CNN to pre-transform the inputs, integrating additional features, such as ESG and macroeconomics indicators, into the model environment, and exploring more complicated structures or models.