Decision Support from Financial Disclosures with Deep Reinforcement Learning Considering Different Countries and Exchange Rates ^†

Abstract

The era of low-interest rates is coming. In addition to their basic salary, people hope to increase their income by doing part-time work, understanding how to use assets already on hand, and investing in assets to earn extra rewards. Goldman Sachs reports that over the past 140 years, the 10-year stock market return has averaged 9.2%. The investment firm also noted that the S&P 500 outperformed its 10-year historical average with an annual average return of 13.6% between 2010 and 2020. Nowadays, with increased computing power and advancements in artificial intelligence algorithms, the effective use of computing power for investment has become an important topic. In the investment process, venture capitalists form portfolios, a practice that improves investment efficiency and reduces risks in a relatively safe situation. Current investments are not limited to one country, allowing for investments in other countries. Given this situation, we must pay attention to Uncovered Equity Parity (UEP) conditions. Thus, we aim to find optimal dynamic trading strategies with Deep Reinforcement Learning, considering the aforementioned properties.

1. Introduction

The era of low-interest rates is coming. In addition to regular jobs, people hope to do part-time work to increase their income, understand how to use assets already on hand, and invest in them. Central Banks around the world are adopting negative interest rate policies. People who previously saved their money now tend to invest more [1]. Goldman Sachs reports that the 10-year stock market return averages 9.2%, with the S&P 500 outperforming its 10-year historical average from 2010 to 2020, achieving an annual average return of 13.6%.

However, while the S&P 500’s Index achieved a high return, the average market return for the typical investor was only 5.19% during the same period. The main reason for this is that investors are sometimes illogical and irrational when investing [2]. We usually invest in various stocks, real estate, or gold. Different investments have different attributes and features. In the past, investors invested based on their experience, but were lacking a complete formula or exact rules. Nowadays, with increased computing power and developed artificial intelligence algorithms, the effective use of computing power for investment is an important topic.

People constantly adjust their various investment portfolio holdings, with past studies focusing on univariate time series analysis due to excessive noise in multivariate data causing inaccuracies [3]. However, daily stock prices are highly positively correlated with each other. As a result of recent advances in neural network research, particularly in Long Short-Term Memory (LSTM) networks, multivariate models have been shown to outperform univariate networks [4].

In the context of global investing, it is essential to consider Uncovered Equity Parity (UEP), which suggests that a country’s stock market often moves in the opposite direction to the exchange rate [5]. This means that the exchange rate can significantly impact investment performance, therefore making it crucial to incorporate this factor into portfolio design. Previous studies have shown that adding USD and gold to a portfolio can improve its hedging effect, with the optimal weighted stock-dollar-gold combination emerging as the best portfolio strategy, regardless of the reference return or risk [6].

To tackle the challenge of investing in different countries, we propose a deep reinforcement learning-based approach. Specifically, we use an A2C-based model to adjust the shares owned of each stock at each time step, with the goal of maximizing the portfolio return per period by selecting sequential trading directions for individual assets based on time-varying market characteristics.

2. Methodology

Stock returns become a time series forecasting problem, due to the daily changes in stock prices. Many researchers have undertaken extensive studies on this topic [3,7]. Recurrent Neural Networks (RNNs) are a type of neural network that contain feedback loops in their recurrent layers. Through feedback loops, RNNs can store information in “memory cells”. Several studies have demonstrated that various forms of Recurrent Neural Networks (RNNs) perform better than conventional financial time series models across different markets and financial investment applications [8,9]. However, they perform poorly if learning has long-term time dependencies.

Reinforcement learning (RL) is a type of machine learning that allows an agent to learn by interacting with an environment and receiving feedback in the form of rewards or penalties. In contrast to supervised learning, where feedback provides explicit instruction on how to perform a task, RL learns through trial and error by receiving signals indicating whether its actions are positive or negative. Essentially, RL enables an agent to make optimal decisions based on past experiences and feedback from the environment, incorporating knowledge about when it acts and how its environment encourages rewards.

Recent research tries to find a learning method to optimize trading strategies when interacting with the dynamic real-world financial environment. Pendharkar and Cusatis [10] show two different RL methods: on-policy (SARSA) and off-policy (Q-learning). They also compared the performance of the two methods. Almahdi and Yang [11] considered real-world constraints and high transaction cost conditions to build a hybrid method that combines recurrent RL and particle swarm optimization.

These studies used different RL-based methods depending on the different problem settings they face. Advantage Actor Critic (A2C) is a typical actor-critic algorithm to improve policy gradient updates. In the basic Actor—Critic algorithm, the critic network estimates the value function, but there is a problem with the large variance of the policy gradient. The critic network in A2C not only estimates the value function of action, but also evaluates how the action could be performed better. To address the high variance of the policy gradient, A2C employs a baseline mechanism to make the model more robust.

In A2C, many agents work independently to pass average gradients to the global network.

The A2C’s objective function $\nabla {\mathcal{J}}_{\theta }\left(\theta \right)$ can be written as follows:

$$\nabla {\mathcal{J}}_{\theta }\left(\theta \right)=\mathbb{E}\left[{\sum }_{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }\left(a_t|s_t\right)A\left(s_t,a_t\right)\right]$$

(1)

${\pi }_{\theta }\left(a_t|s_t\right)$ is the policy network at time step—t. $A\left(a_t|s_t\right)$ is the advantage function and can be written as:

$$A\left(s_t,a_t\right)=Q\left(s_t,a_t\right)-V\left(s_t\right)$$

(2)

or

$$A\left(s_t,a_t\right)=r\left(s_t,a_t,s_{t+1}\right)+\gamma V\left(s_t\right)-V\left(s_t\right)$$

(3)

Portfolio optimization involves adjusting the allocation of shares in a portfolio to achieve the objective of maximizing returns while minimizing risk. This can be achieved by considering various factors such as expected returns, volatility, and correlation between assets. In our study, we use the approach of changing the volume of shares in our portfolio to find the optimal allocation for maximizizing returns while minimizing risk. We aim to strike a balance between generating high returns and managing the risk of losses, achieved through efficient diversification and asset allocation.

The Sharpe ratio is a commonly used metric for evaluating the risk-adjusted performance of an investment strategy. It measures the excess return of an asset or portfolio over the risk-free rate per unit of volatility or risk. A higher Sharpe ratio indicates better risk-adjusted performance. In our study, we use the Sharpe ratio to measure the performance of our proposed model in optimizing portfolio return with respect to risk. By evaluating the Sharpe ratio of our model, we can gain insights into its ability to generate higher returns while controlling risk, and compare its performance with other models or benchmark indices.

In this study, assumptions are made to ensure that individual investor trades do not have any impact on the market price of portfolio assets since trades are small. This means that if we buy or sell any number of shares, the stock price does not change. In the portfolio trading process, we have the freedom to buy and sell any number of shares at any time. To maximize the portfolio return per period, we formulate this process as a generalized Markov decision process (MDP), where the agent selects the sequential trading direction of individual assets in the portfolio based on time-varying market characteristics.

The state space of the agent in this system is where the agent observes the financial environment. The state at period, t, can be expressed as follows:

$$S_t=\left(X_t,{cp}_t,{sh}_t{,{ch}_t,b}_t\right)$$

(4)

($X_t$: encoding features at time step—t

${cp}_t$: each stock’s close price at time step—t

${sh}_t$: shares owned of each stock at time step—t

${ch}_t:$ exchange rate at time step—t

$b_t$: investment available balance at time step—t)

For each stock, our action space is defined as {−i …, −1, 0, 1, …, j}, where i represents the maximum number of stocks to buy and j is the maximum number of stocks to sell at each action step. For continuous actions, we normalized the action space to [−1, 1]. In each state, we first perform an action selection, followed by a sell operation, a buy, and a hold operation.

The encoding features consist of 5 features and OHCLV (Open, High, Close, Low, Volume) data for each stock. The five features are Simple Moving Average (SMA), Exponential Moving Average (EMA), Moving Average Convergence Divergence (MACD), Relative Strength Index (RSI), and On-Balance Volume (OBV). These five features are selected from widely used financial technical indicators to predict stock price trend.

Table 1. Summary of five features.

Type	Financial Technical Indicators
Moving average	Simple Moving Average (SMA)
Moving average	Exponential Moving Average (EMA)
Trend	Moving Average Convergence Divergence (MACD)
Momentum	Relative Strength Index (RSI)
Volume	On-Balance Volume (OBV)

shows the technical indicators used in the proposed model.

From the state processing network, the agent observes the financial environment and leverages its experience through the Actor—Critic network. After updating the average gradients to the global network, the agent performs the final action, as depicted in Figure 1.

Our goal is to obtain the maximum return at the end of the investment process. For this goal, we define the reward function as recomputing the whole portfolio value in a new state—$s$ using action—$a$.

$$R\left(s_t,a_t,s_{t+1}\right)=\left(b_{t+1}+{cp}_{t+1}{s}_{t+1}{ch}_{t+1}\right)-\left(b_t+\mathrm{c}{p}_t{s}_t{ch}_t\right)$$

(5)

3. Experiment and Evaluation

To evaluate the effectiveness of the proposed A2C-based model for financial portfolio management, we conducted experiments on a real-world stock market dataset. The dataset includes historical stock prices of various assets, as well as economic indicators and news sentiment data that may affect the stock market performance.

We used Python, NumPy, pandas, and the scikit-learn machine learning library to implement and train the A2C model on this dataset.

To accelerate the computation process, we used a PC with an Intel four-core CPU 2.7 G, DDR4 16 G RAM, and the Ubuntu Desktop 20.04.5 LTS operating system. The experiment environment is further described in

Table 2. Experimental environment.

Numerical and Machine Learning Package
Python 3.9.2	scikit-learn
	imbalanced-learn
	NumPy
	SciPy
	pandas

.

To construct the portfolio, we selected Dow Jones Industrial Average (DJIA), SPDR S&P 500 ETF Trust (SPY), and Yuanta/P-shares Taiwan Top 50 ETF (0050.tw). In the portfolio, the first two stocks are in the US stock market and the last one is in the Taiwan stock market. We used daily data from 1 January 2010 to 1 January 2020 for training and data from 1 January 2020 to 7 June 2022 for validation (back-testing). Figure 2 shows the relative price of these three stocks from 1 January 2020 to 8 July 2022.

The close price of SPY is the highest at the end of this back-testing period, which is the reason why we choose SPY as the baseline. We would have obtained almost 30% if we bought SPY on 1 January 2020 and kept this stock until 7 June 2022.

The portfolio comprises SPY, DJIA, and 0050.tw, and the A2C model, which has been trained, is used by the agent to adjust the shares of each stock owned at each time step. The time step in this research is daily. Each day, the agent observes the financial environment and uses the trained A2C model to take action.

Table 3. Performance of portfolio for stock trading.

	SPY	DJIA	0050.tw	Portfolio
Annual Return	0.11728	0.05538	0.12267	0.21484
Cumulative returns	0.30851	0.13962	0.30875	0.60423
Sharp Ratio	0.56783	0.33789	0.65887	0.81368

presents the experimental results of a real financial dataset, covering the period from 1 January 2020 to 7 June 2022.

The portfolio outperforms other stocks that use a buy-and-hold strategy.

Figure 3 shows the cumulative returns of the portfolio and the baseline. The baseline is the buy-and-hold strategy, only investing in SPY stock.

4. Conclusions and Future Work

Investments in today’s global economy are not limited to a single country. As a result, when constructing a portfolio, it is essential to consider the Unintended Effects Paradigm (UEP) that can affect the portfolio’s performance. In this study, we propose an A2C-based model for financial portfolio management that takes into account different countries and exchange rates. Although exchange fees and stock transaction costs were not factored into the model, our findings indicate that investing in a portfolio yields better returns than investing in a single stock. Future research should focus on incorporating exchange fees and stock transaction costs into the model, and we will explore another deep reinforcement learning model that can be integrated into our proposed framework.