Imperfect-Information Game AI Agent Based on Reinforcement Learning Using Tree Search and a Deep Neural Network

Abstract

In the field of computer intelligence, it has always been a challenge to construct an agent model that can be adapted to various complex tasks. In recent years, based on the planning algorithm of Monte Carlo tree search (MCTS), a new idea has been proposed to solve the AI problems of two-player zero-sum games such as chess and Go. However, most of the games in the real environment rely on imperfect information, so it is impossible to directly use the normal tree search planning algorithm to construct a decision-making model. Mahjong, which is a popular multiplayer game with a long history in China, attracts great attention from AI researchers because it contains a large game state space and a lot of hidden information. In this paper, we utilize an agent learning approach that leverages deep learning, reinforcement learning, and dropout learning techniques to implement a Mahjong AI game agent. First, we improve the state transition of the tree search based on the learned MDP model, the player position variable and transition information are introduced into the tree search algorithm to construct a multiplayer search tree. Then, the model training based on a deep reinforcement learning method ensures the stable and sustainable training process of the learned MDP model. Finally, we utilize the strategy data generated by the tree search and use the dropout learning method to train the normal decision-making agent. The experimental results demonstrate the efficiency and stability performance of the agent trained by our proposed method compared with existing agents in terms of test data accuracy, tournament ranking performance, and online match performance. The agent plays against human players and acts like real humans.

1. Introduction

Since the advent of computer games, game AI [1] has been one of the most important branches of research in AI [2]. In recent decades, game AI has made significant progress in games with perfect information, such as AlphaGo [3] and AlphaZero [4], through deep convolutional networks [5] and tree search techniques. In contrast, related research on games with imperfect information [6] has not had such outstanding success. Currently, some imperfect-information games are solved with CFR [7] or reinforcement learning methods [8,9]. However, due to the complexity of the environment rules and the high simulation cost, both models are inefficient and unstable.

In the game field of imperfect information, the research and development are completely different from that of perfect information, and it can be said that the development is very slow. The main reason is that, in an imperfect-information environment, participants or agents usually cannot fully access all the information about the current situation. It is difficult to utilize search algorithms to improve agent strategies. Furthermore, the amount of state information that needs to be processed is huge. In most cases, participants or agents can only observe private information and partly public features, and the key information that has a significant impact on the agent’s strategies is usually hidden. As a result, the traditional search-based learning methods cannot be directly applied to imperfect-information game scenarios.

The intelligent agent-based modeling approach has become popular in recent studies, because it can provide a direct and specific way to overcome complex real-world problems. With the help of machine learning (ML) and reinforcement learning (RL) techniques, an opportunity to build and improve an agent-based model has arrived. Shahbazi et al. [10] proposed an intelligent agent-based recommendation system based on NLP techniques and semantic analysis approaches and achieved state-of-the-art prediction accuracy. Alejandro et al. [11] proposed an extension to the design concepts and details (ODD) protocol to support agent-based modeling and offered a standardized description of ML approaches. Furthermore, some other researchers attempted to integrate RL with a neural network. Arash Heidari et al. [12] combined the lightweight version of the RL technique and a convolutional neural network to successfully provide a good performance of an offloading strategy. Other researchers have provided a review of the literature: Zahra Amiri et al. [13] reviewed and reported a well-organized classification of distributed system, and divided several recent studies into seven divisions to analyze their advantages and drawbacks. Currently, most agent-based modeling approaches are based on a certain strategy or pure supervised learning methods, which is ineffective and provides poor performance.

In this paper, we design an agent model of Mahjong based on reinforcement learning, deep learning, and dropout learning to overcome some of the mentioned limitations. The detailed rules of Mahjong can be found in Appendix A. However, implementing and training an agent model directly from the simulation of imperfect-information games with complex rules, such as Mahjong, presents a huge challenge. Without a carefully designed agent model and an effective learning approach, the agent may not be able to effectively learn the game strategies for good performance. To tackle the problem of the huge hidden information and large state-action space, in this paper, we design an extended multiplayer game search tree in a way that allows for more flexible state transitions and dynamic decision making. After that, by using dropout learning based on a dynamic Bernoulli random matrix, we transform it into a normal agent with imperfect information to extend the applicability of the agent. The contributions of this article are as follows:

• An extended multiplayer game search tree combined with a reinforcement learning (RL) technique and deep learning is proposed. We incorporate an internalized MDP network model to internalize the simulation state and reduce the simulation cost. Then, we also introduce the additional player position and state-action information into the search tree and network policy estimation to realize the dynamic transition of the internal state.
• A normal agent with imperfect information is trained through dropout learning. With the utilization of the improved strategies achieved by the RL agent and the dropout learning technique, our approaches can effectively handle the decision-making problems of Mahjong.
• To increase the learning efficiency of the proposed agent, residual blocks [5], an experience replay buffer [14], and an L2 regularization term [15,16] are used. In addition, extra noise and a multi-thread simulation are applied to the search tree to improve the performance of the search algorithm.
• The convergence and performance of our method are demonstrated by successfully implementing a Mahjong agent with good performance and comparing it to several existing models through accuracy on test data and tournament confrontation. Finally, we conduct tests on online game platforms against human players to further validate the practicality of the proposed agent.

The structure of this paper is as follows: Section 2 reviews the related work, including both model-free and model-based agent learning approaches. In Section 3, we describe the game scenario of Mahjong and detail the framework of the RL methodology. In Section 4, we report the simulation and experiment results, and compare them with previous work to demonstrate the performance of our agent. Finally, in Section 5, we conclude this article and discuss improvement works in the future.

2. Related Work

At present, reinforcement learning methods generally have two principal categories: model-based, and model-free [17]. It is very popular among researchers to utilize model-free methods, such as policy gradient [18] and advantage function [19], to directly implement decision-making agents. The model-free methods are usually simple and direct, and can quickly train an agent with good performance for most uncomplicated games.

However, when interacting with complex rule environments, the model-free RL method encounters problems such as a low convergence speed and large gradient fluctuations during the training, e.g., TD learning [20] and Q learning [21]. In addition, additional labeling of training data is usually required for the agent to speed up model training. In contrast, the model-based RL method first constructs a specific environment model, generally based on the Markov decision process (MDP) [22]. Then, it is combined with the tree search planning and deep learning techniques to handle more complex decision-making problems, such as the TreeQN [23] model. However, the model-based methods usually need to solve two key issues additionally. First, the tree search algorithm usually not only needs to implement a simulation accurately but also needs to maintain and restore all the simulation states, which is highly reliant on the knowledge of the environment, e.g., an accurate environmental simulator; secondly, it cannot be directly applied to imperfect-information environments, because a large amount of hidden information could weaken the performance of the search algorithm. As a result, an agent could only obtain a suboptimal strategy.

Aiming at the first problem, Julian Schrittwieser et al. [24] proposed a learned model that uses the locally learned MDP model to replace the accurate environmental simulator. The model introduces the concept of internal state and learns and fits the state transition of tree search simulations by using the deep-learning-based MDP model that is iteratively updated. The MDP model is viewed as the hidden layers of deep neural networks, and the internal state is used to approximate the optimal policy function and value function. However, it is only suitable for single-player or two-player games with simple rules and it cannot be directly applied to the mostly imperfect-information games.

Imperfect-information agents are generally difficult to implement because observers or participants usually only observe part of the information. To solve this problem, some researchers enumerated all possible states and used determinization technology to achieve improved strategies [25]. Since all possible results also need to be averaged, this method may also obtain a suboptimal strategy [26]. As a result, it does not find the global optimal solution but a local solution. Different from the above approaches, some researchers have proposed a learning agent based on the dropout technique [27], which trains the agent from scratch and slowly drops part of the global features during RL training, and realizes the conversion from a perfect information agent to a normal agent with imperfect information. However, its training method has a slow convergence and low effectiveness. Essentially, it is based on the model-free approach. Not only does this strategy lack optimality but also the learning process is prone to fluctuations, making it hard to optimize the agent model.

Mahjong is one of the most complex imperfect-information games. Players or agents need to take action based on their hand’s tiles and other players’ public tiles. It is very difficult to directly apply tree search algorithms to Mahjong because the game state and state transitions are complicated. Tsuruoka et al. [28] considered the opponents’ possible moves or actions and proposed a Mahjong agent combined with the opponent model. Van Rijn et al. [29] constructed a Mahjong search tree based on statistical methods and machine learning approaches to find the action with the maximum possible return. Another researcher proposed a strategy model based on deep learning [30] and deep convolutional networks. Researchers from Microsoft Asia proposed Suphx [31], based on distributed reinforcement learning combined with other novel improvement methods, which defeated professional players on the Riichi Mahjong platform.

In recent years, Gao et al. [32] used DenseNet [33] to extract all the features of Mahjong and obtained a better strategy with the XGBoost [34] algorithm, which achieved great accuracy on the test dataset. Zheng et al. [35] described recent Mahjong AI works in detail, summarizing the current research, analyzing the differences of each approach, and proposing the future work of Mahjong AI.

In conclusion, the studies reviewed above emphasize the potential of RL approaches in designing agents for imperfect-information games that could solve decision-making problems and outperform human players.

3. Reinforcement Learning Algorithm

3.1. Tree Search Learning Algorithm for Mahjong

3.1.1. Internalized MDP-Based MCTS Learning Algorithm

Due to the complexity of the game rules of Mahjong, it is difficult to implement the search simulations with accurate Mahjong simulators. For example, not only should the game state of the Mahjong be maintained, but also the backup and restore operations of the simulators should be performed. This is to ensure the accuracy of the game state and state transition. To reduce the dependency on accurate simulators, we use the internalized MDP model-based RL method in combination with the multiplayer tree search algorithm to improve and obtain the agent strategy. In this work, the internalized MDP model was used instead of the accurate Mahjong simulator, combined with the MCTS algorithm to decrease the execution cost of simulators and further strengthen the Mahjong game strategy. Note that the tree search process can access all the feature information of Mahjong to speed up the agent learning process and ensure the accuracy and efficiency of the tree search.

The internalized MDP model extracts the characteristic information of Mahjong based on a deep convolutional network [36] and then relies on the information processing and integration capabilities of a recurrent neural network [37] to further improve the generalization and estimation capabilities of the functions to obtain more accurate value estimation and prior policy estimation of the Mahjong situation. It is also combined with the MCTS algorithm to strengthen the agent strategy. The internalized MDP model consists of three sub-networks: the representation network is used to calculate the network state of the root node of the search tree; the dynamic network is used to calculate the internal state of the next node during the simulation; the prediction network is used to calculate the prior policy estimation and the multi-value estimation corresponding to the player state of the current leaf node. We utilize the residual blocks as the main body of the representation and dynamic network to extract the features of the Mahjong situation. The information on the Mahjong situation is represented as the internal state of the network during the search process. Now, we describe the internalized MDP-based MCTS algorithm for Mahjong. In addition to the original search algorithm, we also introduce the player position variable into the search algorithm, which can handle dynamic player priority and makes dynamic decision-making possible. Note that the search algorithm also conforms to the principle of the Minimax algorithm. To simplify, the player position variables are represented in relative order and start from 0. In addition, the player’s basic action is combined with additional dynamic information to let the search tree adapt to the dynamic Mahjong game rules. Formally, at each step t, the model performs a strategy search process to determine the current player’s best action. A search process consists of $\mathrm{N}$ multiple simulations and each simulation involves several steps to calculate and predict $\mathrm{k}$ $\left(\mathrm{k}=0,1,\mathrm{l}\right)$ hypothetical actions in the future. After the $\mathrm{N}$ simulations are implemented completely, the agent will obtain the strategy probability vector corresponding to the number of visit counts about all edges of the root node of the search tree. The tree search process and training process are shown in Figure 1.

To better describe the steps of each simulation, we first declare the variables used in the MCTS algorithm [38]. Each node stores its game state, represented as the internal network state S, and the information associated with all edges, such as the number of visits N, the average state-action value Q, the player position variable D, and the prior policy estimation P. Then, we perform the following five steps sequentially, starting from the root node for each simulation:

• Initialization of the root node. If the root node of the Mahjong game tree has not been initialized before, we use the representation network to calculate its internal state:

$$S^0=I_{\theta }\left(O_t\right)$$

(1)

where ${\mathrm{I}}_{\mathsf{\theta }}\left({\mathrm{O}}_{\mathrm{t}}\right)$ denotes the representation network parameterized by $\mathsf{\theta }$ with the features vector ${\mathrm{O}}_{\mathrm{t}}$ of Mahjong as the input. Then, skip to step 4. Otherwise, go to step 2.

2.

Selection of edges. Selecting the edge along a path with the upper confidence bound (UCB) algorithm [39] until an uninitialized (empty) leaf node is reached. In detail, the statistics of the current node corresponding to the current player are used to determine the next search edge. The UCB algorithm used is shown below:

$$a^k=argma{x}_a\left[Q\left(s,a\right)+C_1\frac{\sqrt{{\Sigma }_{b}N\left(s,b\right)}}{1+N\left(s,a\right)}P\left(s,a\right)\right]$$

(2)

In Equation (2), the key point is that the exploitation and exploration of the search depend on the value $\mathrm{Q}$ and the policy estimation $\mathrm{P}$. It can be seen that the parameter $\frac{\sqrt{{\Sigma }_{\mathrm{b}}\mathrm{N}\left(\mathrm{s},\mathrm{b}\right)}}{1+\mathrm{N}\left(\mathrm{s},\mathrm{a}\right)}$ is related to the edge visits $\mathrm{N}\left(\mathrm{s},\mathrm{a}\right)$ and the parent node visits ${\Sigma }_{\mathrm{b}}\mathrm{N}\left(\mathrm{s},\mathrm{b}\right)$.

The smaller the edge visits, the higher the priority of the edge, and the parent node visits provide an additional exploration for each child edge. The variable ${\mathrm{C}}_1$ controls the overall exploration of the search. Due to the large action and state space of Mahjong, ${\mathrm{C}}_1$ = 1.5 in our experiment. The purpose is to encourage more exploration.

3.

State transition. After reaching an uninitialized leaf node, we utilize the dynamic network to compute the internal state ${\mathrm{s}}^{\mathrm{k}}$ of this node with its parent internal state (the latest state) ${\mathrm{s}}^{\mathrm{k}-1}$ and the transition action ${\mathrm{a}}^{\mathrm{k}}$ combined with additional Mahjong transition information as the network input:

$$s^k=T_{\mu }\left(s^{k-1},a^k\right)$$

(3)

where ${ \mathrm{T}}_{\mathsf{\mu }}$ denotes the dynamic network parameterized by $\mathsf{\mu }$. After that, the resulting internal state ${\mathrm{s}}^{\mathrm{k}}$ will be used to compute related estimation variables of the leaf node in step 4.

4.

Initialization of the leaf node. After obtaining the internal state ${\mathrm{s}}^{\mathrm{k}}$($\mathrm{k}=0,1,\dots$) of the new leaf node, we need to create and initialize all edges of this node. First, we use the prediction network to calculate the prior policy ${\mathrm{p}}^{\mathrm{k}}$ in combination with rule-related transition information ${\mathrm{q}}^{\mathrm{k}}$ of Mahjong and the value estimation vector ${\mathrm{v}}^{\mathrm{k}}$ of all players:

$$p^k\times q^k,v^k=E_{\sigma }\left(s^k\right)$$

(4)

where ${\mathrm{E}}_{\mathsf{\sigma }}$ denotes the prediction network parameterized by $\mathsf{\sigma }$. Then, we initialize all edges of this node, e.g., $\mathrm{N}\left(\mathrm{s},\mathrm{a}\right)=0$,$\mathrm{Q}\left(\mathrm{s},\mathrm{a}\right)=0$, then,

$$P\left(s,a\right)=p^k\times q^k$$

(5)

After that, we also need to calculate the current player position variable $\mathrm{D}$ of Mahjong using player position variables and the rule-related transition information:

$$D\left(s^{k-1},a^k\right)=\left(D\left(s^{k-2},a^{k-1}\right)+T\left(a^k\right)\right)MOD M$$

(6)

In Equation (6), $\mathrm{T}\left({\mathrm{a}}^{\mathrm{k}}\right)$ denotes the mapping function from the transition information to the change of the player position variable, which depends on the special Mahjong game rules. $\mathrm{M}$ denotes the number of players, $\mathrm{MOD}$ denotes the modulo operation. Note that we always set the player position variable of the root node to 0, e.g., $\mathrm{D}\left({\mathrm{S}}^0\right)=0$, and the player position variable is assumed to be in the relative order, such as $0,1,2,3,0,1,\dots$, and so on, which is used to calculate the value estimation for each player.

5.

Backpropagation and updating. Finally, we update the statistics of all nodes along the path. For each $\mathrm{k}=1,\mathrm{\dots },\mathrm{leaf}$, where $\mathrm{leaf}$ denotes the depth index of the new leaf node, we compute the change of the player position variable corresponding to the stored variable $\mathrm{D}$:

$$shif{t}^k=\left(D\left(s^{k-1},a^k\right)-D\left(s^{leaf-1},a^{leaf}\right)+M\right)MOD M$$

(7)

Then, we obtain the state-action value of the current edge according to the value vector ${\mathrm{v}}^{\mathrm{leaf}}$ and the ${\mathrm{shift}}^{\mathrm{k}}$ value:

$$Va{l}^k=V_{ shif{t}^k}^{leaf},shif{t}^k\in \left(0,1,\dots ,M-1\right)$$

(8)

Then, we update the value $\mathrm{Q}$ and the edge visits $\mathrm{N}$ of the current edge:

$$Q\left(s^{k-1},a^k\right)=\frac{N\left(s^{k-1},a^k\right)\ast Q\left(s^{k-1},a^k\right)+Va{l}^{k-1}}{1+N\left(s^{k-1},a^k\right)}$$

(9a)

$$N\left(s^{k-1},a^k\right)=N\left(s^{k-1},a^k\right)+1$$

(9b)

Note that the output range of the value estimation is [−1, 1], for example, the target values −1, 0, and 1 mean loss, no change, and gain of the round score of the game, respectively, which also means the final result of a single round.

3.1.2. Game Information Representation

Since Mahjong has high-dimensional features, we use binary planes to denote each tile and use the column of the plane to denote each type of all tiles. Considering Mahjong contains numbers from 1 to 9 and three suits of characters, dots, and bamboo, we use 27 different tiles and 4 different counts to represent a 27 × 4 two-dimensional matrix realized by a 27 × 1 × 4 three-dimensional plane. This encoding of the tiles’ representation allows our agent to better understand the Mahjong features. Figure 2 shows an example.

Figure 2 shows a hand of 366 Man (character), 777 Pin (dot), and 123456 Sou (bamboo) tiles.

3.1.3. MDP Model Input

Because Mahjong has complex game rules, we design feature planes with all possible special rules of Mahjong to improve the learning process of the agent. At each step, the player acts according to their hand tiles, other players’ public discard tiles, and other features. In addition, we also consider the last tiles and the last action of the latest player. As a result, the following features in

Table 1. Input features of representation network.

Features	Number of Channels	Description
Hand tiles of each player	4 × 4	Private tiles of each player
Total discard tiles of each player	4 × 4	Discard tiles of each player
Recent 4 discard tiles of each player	4 × 4	Recent discard of each player
Melds tiles of each player	4 × 4	Melds tiles of each player
Wall tiles	10	Top ten tiles of wall tiles
Current score of each player	11 × 4	Score is divided into 11 integer intervals for each player

are used as the input of the representation network. Each feature is encoded by several 27 × 1 × 1 planes for integer features or 27 × 1 × 4 planes for tiles.

Then, we need to design the input features of the dynamic network for Mahjong. In addition to the last internal state, the transition action combined with other information is also provided as an input of the dynamic network. As shown in

Table 2. Input features of the dynamic network.

Features	Number of Channels	Description
Tile type	1	Tile type of action
Action type	10	10 types of actions 1
The change of the player position variable	4	The change of next player’s position variable after taking action 2
Additional player state	4	The next player state after taking action 3

¹ Actions encoded with one-hot form, as shown in Table 3. ² e.g., player position +1, +2, +3, +4. ³ e.g., Win, Pong/Kong, Chow, Pass of the player state.

, we utilize the tile and action type to specify the basic action information. Since Mahjong is a multiplayer game, the change of the player position variable and the player state are also provided as input. The goal of providing extra information is to improve the accuracy of the value estimation ${\mathrm{v}}^{\mathrm{k}}$ and the prior policy estimation ${\mathrm{p}}^{\mathrm{k}}\times {\mathrm{q}}^{\mathrm{k}}$ of the prediction network. All features are encoded by several 27 × 1 planes and then stacked with the last internal network state.

3.1.4. MDP Model Output

There are two types of output estimations: the prior policy estimation ${\mathrm{p}}^{\mathrm{k}}\times {\mathrm{q}}^{\mathrm{k}}$ and the value estimation vector ${\mathrm{v}}^{\mathrm{k}}$. First, we use a 27 × 1 × 55 convolutional layer to represent all possible actions combined with the transition information. The proposed output design shows faster model convergence and consistent performance compared to a full connection output layer. The detailed output representation is shown in

Table 3. Prior policy representation.

Action Type	Number of Channels	Action Type	Number of Planes
Discard	4 × 4	Chow_Right	1
AddKong	4 × 4	Chow_Middle	1
Pass	4 × 4	Pong	1
ClosedKong	1	OpenKong	1
Chow_Left	1	Hu (Win)	1

. Specifically, 48 channels are used to denote the three types of actions that can dynamically trigger one of four possible changes in the player position and one of four possible player states, such as discard, add Kong, and Pass. So, there are 3 × 4 × 4 = 48 different types of dynamic actions. The remaining seven types of deterministic actions have the normal state transition.

The value estimation vector ${\mathrm{v}}^{\mathrm{k}}$ involves the evaluation value of four Mahjong players, each value within [−1, 1]. It also uses the relative order, starting from the player position of the current node. The detailed improvement of the target value for Mahjong is left for future work.

3.1.5. Network Implementation

It can be seen from Figure 3 that the input of the representation network consists of the features mentioned in Table 1 as a three-dimensional 27 × 1 × 118 input layer. The input of the dynamic network consists of the last internal state and transition features mentioned in Table 2 as a stacked input layer, a three-dimensional plane of 27 × 1 × (128 + 19). The output layers of the prediction network are the two types of estimations mentioned in Section 3.1.4. We do not utilize the internal state directly but just store the network output of the internal state as a representation of a specific Mahjong game situation. The representation and dynamic networks have a main body of 12 repeated residual blocks, with each residual block consisting of two batch normalized, 128 channels, a convolutional layer with a 3 × 1 kernel, and a prediction network that only serves as computing and outputting two types of estimation. Furthermore, the prediction network outputs P player estimate values, where P is the number of players. In Mahjong scene, P = 4. The stride of all convolutional layers is 1 × 1. We train all three networks as a joint network, shown in Figure 4, with K = 6 unfolded time steps for each training iteration.

The total training loss of the network is as follows:

$$L_t\left(\theta \right)={\displaystyle \sum }_{k=0}^K {\displaystyle \sum }_{i=0}^N l_i^v\left(Z_{t+k},v_t^k\right)+l^p\left({\pi }_{t+k},p_t^k\right)+c{\left|\left|\theta \right|\right|}^2$$

(10)

In Equation (10), K = 6 is the unfolded time steps; N = 4 is the number of players; ${\mathrm{l}}^{\mathrm{v}}$ and ${\mathrm{l}}^{\mathrm{p}}$ are the loss functions of the value estimation and prior policy, which are the mean square error (MSE) loss and cross-entropy loss, and $\mathrm{c}{\left|\left|\mathsf{\theta }\right|\right|}^2$ is the L2 regularization term of the network parameters, which could mitigate the network overfitting problem.

3.2. Dropout Learning

After the internalized MDP-based MCTS model is trained completely, then we use the MCTS model to accelerate the training process of the normal agent with imperfect information. Because Mahjong contains a lot of hidden information, if a normal agent cannot access this hidden information, the performance of the normal agent is more likely to be worse than the MCTS model. Anyway, it is not effective to directly train the normal agent from scratch and we cannot ensure that the normal model can learn the best strategy with a lack of information. To overcome the above problems, we propose dropout learning combined with some improvement approaches.

3.2.1. Learning Algorithm with Bernoulli Random Matrix

Instead of directly training the normal agent from scratch, we use the game strategy data learned and generated by the internalized MDP-based MCTS model as the training dataset of the normal agent. Note that the input of the game strategy data involves all global feature information, but we only want the normal agent to eventually access the normal feature information, such as the player’s private tiles, all players’ discard tiles, and other public information, during the evaluation. The other players’ hand tiles are not available to the normal agent during the normal situation. To achieve this, we combine dropout learning with the special discard features matrix. In detail, we train the normal agent starting from all features with perfect information and then gradually add features with imperfect information. For this purpose, we introduce the Bernoulli random matrix [40] λ and multiply the input layer of the normal agent by the matrix $\mathsf{\lambda }$:

$${\mathsf{\lambda }}_{ij}=\{\begin{array}{c}1, Inpu{t}_{ij} \in \Phi \\ ~ B\left(1,p\right) otherwise\end{array}$$

(11)

where Φ is a set of normal input features containing all features with imperfect information described in

Table 4. Imperfect-information features.

No	Feature
1	Player’s own hand tiles
2	Discard tiles of each player
3	Melds tiles of each player
4	Current score of each player

. ${\mathrm{Input}}_{\mathrm{ij}}$ denotes the input feature in the $\mathrm{i}$-th row and $\mathrm{j}$-th column of the matrix of the input layer. $\mathrm{B}\left(1,\mathrm{p}\right)$ denotes the Bernoulli function parameterized by $\mathrm{p}$, and ${\mathsf{\lambda }}_{\mathrm{ij}}$ takes a random value depending on this function for each training iteration. Then, we gradually reduce the value $\mathrm{p}$ from 1 to 0 with 0.1 for every 60 thousand steps during the training. Finally, $\mathrm{p}=0$ is fixed during the evaluation. The total training loss is shown in Equation (12):

$$L_{ }\left(\theta \right)= E_{s,{\pi }^{\prime }\left(s\right)~B} \left[l^p\left({{\pi }^{\prime }}_{ }\left(s\right),{\pi }_{\theta }\left(a|\lambda \ast O_{ }\left(s\right)\right)\right)\right]+c{\left|\left|\theta \right|\right|}^2$$

(12)

In Equation (12), $\mathrm{B}$ denotes the experience replay buffer [41], ${\mathrm{O}}_{ }\left(\mathrm{s}\right)$ involves all input features described in Table 1 in the state $\mathrm{s}$, ${\mathsf{\pi }}_{\mathsf{\theta }}$ denotes the output policy of the normal agent parameterized by the network $\mathsf{\theta }$, ${{\mathsf{\pi }}^{\prime }}_{ }\left(\mathrm{s}\right)$ denotes the target policy obtained by the MCTS model in state $\mathrm{s}$, ${\mathrm{l}}^{\mathrm{p}}$ denotes the cross-entropy loss function, and $\mathrm{c}{\left|\left|\mathsf{\theta }\right|\right|}^2$ denotes the L2 regularization term.

3.2.2. Network Implementation of the Normal Agent

The initial input of the normal agent is the representation layer, which consists of the features listed in Table 1; the priority policy of the network output has been simplified from the original 27 × 55 game actions to 27 × 10 game actions without additional transition information. The main body of the normal agent consists of 64 residual blocks and is used with 256 hidden channels. The network structure is shown in Figure 5. The agent model consists of an initial feature layer, a main body of residual blocks, and several convolutional layers with batch normalization [42] to output the probability vector of all possible player actions.

3.3. Training Process

The complete training process of the Mahjong agent is divided into two parts: (1) the tree search learning process; (2) the dropout learning process.

First, we train the “perfect agent” with perfect information based on the MCTS-based tree search reinforcement learning. The detailed agent’s strategy flow is shown in Appendix B. The brief learning process is shown in Figure A2 and Appendix C, which consists of three parts: the simulation process, the model training process, and the model evaluation process. The detailed tree search Algorithm 1 is as follows:

Algorithm 1 The internalized MDP-based MCTS tree search

Input: $O_t$: input features vector in step t; $\theta ,\mu ,\sigma$: network parameters Output: $\pi$: policy probability vector

1.  Initialize three functional networks $I_{\theta }\leftarrow {\theta }_{ }$, $T_{\mu }\leftarrow \mu$, $E_{\sigma }\leftarrow \sigma$; set the maximum number of simulations per move $\mathrm{K}=250$;

2.  for $(\mathrm{int} \mathrm{i}=0;\mathrm{i}<\mathrm{K};\mathrm{i}++)$

3.  $\mathrm{Node}=\mathrm{Root}$: set the root node as current node; $k=0$: reset the index of search depth;

4.  $\mathrm{while} \mathrm{Node} != \mathrm{leaf} \mathrm{Node}$ //current node isn’t leaf node

5.  k++

6.  ${\mathrm{a}}^{\mathrm{k}}=\mathrm{UCB}\left(\mathrm{Node}\right)$ //determine the next edge according to Formula (2)

7.  $\mathrm{Node}={\mathrm{Node}}_{{\mathrm{a}}^{\mathrm{k}}}$ //go to the child node and set it as current node

8.  end while

9.  if $(\mathrm{Node}==\mathrm{Root})$ //current node is root node

10. ${\mathrm{s}}^0={\mathrm{I}}_{\mathsf{\theta }}\left({\mathrm{O}}_{\mathrm{t}}\right)$, $\mathrm{D}\left({\mathrm{s}}^0\right)=0$ //utilize the representation network to compute the internal state of root node and initialize the player position variable of root node

11. else

12. ${\mathrm{s}}^{\mathrm{k}}={ \mathrm{T}}_{\mathsf{\mu }}\left({\mathrm{s}}^{\mathrm{k}-1},{\mathrm{a}}^{\mathrm{k}}\right)$ //utilize the dynamic network to compute the internal state of the child node after state transition.

13. ${\mathrm{p}}^{\mathrm{k}}{\mathrm{x} \mathrm{q}}^{\mathrm{k}},{\mathrm{v}}^{\mathrm{k}}={\mathrm{E}}_{\mathsf{\sigma }}\left({\mathrm{s}}^{\mathrm{k}}\right)$, store $\mathrm{P}\left(\mathrm{s},\mathrm{a}\right)$, $\mathrm{D}\left({\mathrm{s}}^{\mathrm{k}-1},{\mathrm{a}}^{\mathrm{k}}\right),\mathrm{etc}.$ //utilize the prediction network to compute related estimations and initialize related statistics information according to Formula (5) and (6).

14. Backpropagation and update related statistics of each node along the path according to Formula (7)–(9)

15. end for

16. Get the tree search policy π corresponding to the visit count of all edges of the root node.

After the MCTS model is fully trained, we continue to train the normal agent based on the dropout learning approach described in Section 3.2. The detailed training parameters are shown in

Table 5. Experiment parameters.

Method	Parameters	Value
Tree Search Learning	Dirichlet_noise_alpha	0.9
	Simulation_count	250
	Total_training_steps	1.2 million
	L2_penalty_alpha	1.0 × 10−4
	Batch_size	256
	Initial training speed	0.01
	Optimizer	SGD + momentum
	Simulation CPU threads per agent	25
	Simulation GPU threads per agent	5
	Data generation agent used	10
Dropout Learning	Batch_size	512
	Channels	256
	Initial_learning_rate	0.001
	L2 penalty item	1.0 × 10−4
	Momentum_alpha_beta	0.9, 0.99
	Total_training_steps	0.66 million
	Dropout_decay_rate	0.1
	Dropout_decay_steps	60 thousand

.

In our actual experiments, the Mahjong simulator code runs in Python to facilitate cross-platform operation, while the agent model and the tree search code are implemented by C++ programs to improve decision-making efficiency and implement concurrence on multiple threads. The interaction between the two is realized through Python’s embedding technology.

4. Experimental Results and Analysis

In this section, we validate the learning process of the internalized MDP-based MCTS model, the dropout learning with and without the Bernoulli random matrix. In addition, we test the normal agent on the online game platform of Mahjong with human players and conduct tournaments with other AI models to demonstrate the performance of the proposed agent.

4.1. Experiment Settings

The experiment environment of this paper is IntelCore (TM) i7 12700 CPU, 32 GB memory, Nvidia Tesla V100, and Windows 10 64-bit operating system. The experiment code is written in C/C++ and Python, and implemented based on Cuda, Cublas, and other libraries.

The Mahjong test data is obtained from the Mahjong platform of JJ World Game Company. In addition to removing the faulty game records and the defective data, we also use the data of the game records where the score of the players is more than 5000 points. The score represents the player’s performance of playing Mahjong. The more points, the better the player’s level. After removing the data of players with poor performance, there are about 400,000 state-action pairs of Mahjong played by human players. According to the total number of game data pairs, we divide them into a verification set and a test set in a ratio of 1:2. To reduce the influence of other factors, we conduct 10 independent experiments and use the average value of them as the final result; each independent experiment is performed with a different random number generator. For the MCTS-based model, the corresponding action with the highest number of edge visits is selected as the output action; for the normal agent model, the action with the highest probability is selected (if there are multiple available actions with the same probability, then one of them is randomly selected). The detailed experiment configurations are shown in Appendix D.

4.2. Model Performance Validation

First, we test the MCTS-based models with different simulations during the RL training. All models use the same internalized MDP network as described in Figure 3, and the same training parameters mentioned in Table 5, except for different simulation counts: 10, 50, 100, and 250 per move during training. Furthermore, all models utilize the same 500 simulations per move during the evaluation of the Mahjong test data. The training loss of the models with different simulations is shown in Figure 6a. The accuracy rate on the Mahjong test date set of models with different simulations is shown in Figure 6b.

Figure 6a shows that the training loss starts to decrease for all models and the accuracy rate of all models slowly increases from 0.2 million steps. The accuracy rate on the test data is improved during the RL training, however, the growth rate continues to decline. At the beginning of training, the loss of the 250-simulation per move model is slightly lower than that of the other models. As training progresses, the training loss of the 250-simulation model is obviously lower and smoother than that of all the other models, this is because the 250-simulation model not only learns the prior policy estimation well but also predicts more accurately the value estimation. The experiment results show that the 250-simulation model finally has the lowest training loss and the 100-simulation model is second. Further, it can be seen from Figure 6b that models with 50 or fewer simulations cannot learn game strategy very well, and their accuracy rates are lower than 30% after half of the training steps. The model with 10 or 50 simulations per move could not reach a 50% accuracy rate after the complete training. In contrast, when given more than 50 simulations per move during training, the accuracy rate of the model has a great improvement. The more simulations, the higher the magnitude of accuracy increase. The highest accuracy rate reached is 72%, with 250 simulations per move. The results show the effectiveness and stability of the internalized MDP-based MCTS model on Mahjong test data.

To better describe the effects of feature access on the performance of the MCTS-based model, we compare the MCTS-based models with access limitations on different features. The original version could access all feature information, and we also designed two variants based on the original model: one of which removes all unconventional input features, such as other players’ private hand tiles, and the other replaces all unconventional input features with random features instead of empty features. Both models utilize 250 simulations per move during the training. The detailed experiment setup for both models are mentioned in Section 3.3. The results comparing the accuracy rate between the three different models over different evaluation simulations per move are shown in Figure 7a.

We can see from Figure 7a that the accuracy rate of the model with all of the unconventional input features removed could not reach 35%, and cannot be improved by increasing the evaluation simulations. In contrast, the other model, replacing unconventional features with random features, has a better accuracy rate, although it is still lower than that of the all-features-access model. This is because the former could utilize additional random features instead of empty features to improve the tree search process. Regardless of whether the random features information is correct or not, the game strategy can be improved to a certain extent through the value-averaging approach of tree search. Compared with the other two versions, the accuracy rate of the all-features-access model still increases as the number of simulations increases. Finally, it reaches 72% and is better than the other two variants. This comparison confirms our inference that the feature access has a great influence on the performance of the MCTS-based model. Further, to verify whether the internalized MDP network improves on the performance of the MCTS-based model, we train and compare two different versions: one is based on the internalized MDP network and the other is based on the accurate Mahjong simulator (it can be equivalent to utilize the representation and prediction network to calculate the prior policy and value estimation of nodes, and it does not implement the internal state transition of Mahjong) over different evaluation simulations. The accuracy rate of the two models is shown in Figure 7b. When less than 500 simulations of evaluation are used, the accuracy rate of the internalized MDP-based model is higher than that of the accurate simulator-based model, except when using a single simulation, which is the same as directly using the prior policy as model output. However, when the number of simulations is more than 500, the magnitude of accuracy increase in the internalized MDP-based model is suddenly decreased but the accuracy rate of the simulator-based model can still steadily increase. By analyzing the MDP model, the possible reason is that, as the number of simulations increases, the depth of the nodes will also grow and the internalized MDP-based model is more likely to compute a noisy value estimation, due to the limitation of the recurrent neural network structure and the precision of the matrix multiplication operations, etc. In short, the experiment results indicate that the internalized MDP model could still make effective improvements on the model performance under a certain number of simulations.

The test accuracy of the models is reported in

Table 6. Accuracy results.

Model	Highest Accuracy	Mean Accuracy
Simulator-based MCTS Model	70.5%	69.6%
MDP-based MCTS Model	73.0%	72.0%
Previous Works	88.2%	68.8%

. The MCTS-based model utilizes 134 thousand validation data and 266 thousand test data. Considering the time and computing cost, the simulator-based and MDP-based MCTS models both use 250 simulations per move during evaluation. In addition to the above models, we also report the accuracy rate obtained by other models in previous works [43] for comparison. It can be seen from Table 6, that we obtain a 69.6% average accuracy rate for the simulator-based MCTS model and a 72.0% average accuracy rate for the MDP-based MCTS model. However, we do not achieve the highest accuracy among all models and more research could focus on improving the target game reward to obtain higher data accuracy. Note that due to different test data and different Mahjong game rules, we just list the data and indirectly compare our models with previous works.

4.3. Offline Experiment

In this section, we conduct offline tournaments between the learning-based normal agent and the open-source Mahjong engine. We use the public open-source Mahjong engine as a baseline player for offline evaluation. This Mahjong engine, called Tenhou-Bot (https://github.com/MahjongRepository/tenhou-python-bot (accessed on 12 July 2022)), which is programmed with a hand-designed value function, has conducted a large number of anonymous online confrontation tests on the online Mahjong website Tenhou.net, with a total of about 1000 online games and the final rating is between three dan and four dan (the ranking system adopted by Tenhou.net, https://tenhou.net/man/ (accessed on 15 July 2022), which starts from rookie, 9 kyuu down to 1 kyuu, and then 1 dan up to 10 dan), with a first place rating of 23.65%. To be compatible with the Mahjong rules in our paper, we remove the extra seven tiles (four winds and red, green, and white tiles) and make some additional modifications to the winning condition on the above engine. To ensure the fairness and effectiveness of the evaluation, the agent ranking adopts the standard ranking formula proposed by Tenhou:

$$Rank=\frac{5\ast n_1+2\ast n{–}_2}{n_4}-2$$

(13)

In Formula (13), $n_1$, $n_2$, and $n_4$ represent the total times of the first, second, and fourth rankings, respectively. The ranking rating is computed from the results of the tournament between iterations of the dropout-learning-based agent during training and the baseline player (engine). Since Mahjong has a lot of random feature information, such as the initial player hand tiles, it is generally assumed that not less than a hundred games are needed to obtain a stable rank. So, we set up 10 random independent experiments, each has 100 random games, each game has four rounds, and we use the average ranking rating as the final result. We randomly select three of the four players as the baseline player in each game.

First, we compare the effects on the performance of the agent with and without the discard feature matrix during training, with the same training data obtained by the internalized MDP-based MCTS model. All parameter settings and training processes of both models are the same, except one is combined with the Bernoulli random matrix, the other does not directly use it and only takes conventional features as input by setting $\mathrm{p}=0$ throughout training. The number of training steps for both models is set to about 0.6 million steps. The training results are shown in Figure 8. The ranking results are shown on the y-axis, which is calculated corresponding to Formula (13), millions of training steps on the x-axis. The dark line denotes the median score across 10 separate experiments and the shaded region denotes the 25th to 75th percentile. We can see from Figure 8 that at the beginning, the rating perform of the agent with the Bernoulli random matrix is lower than that of the agent without it because the former needs more time and steps to extract the related features from randomly dropping features during the training iterations. As training continues, the rating of the agent with the Bernoulli random matrix begins to be higher than that of the agent without it. This is because it is hard for the agent without feature dropping to establish the network connection between the conventional features and the target action learned by the “perfect model” without the help of additional feature information, such as the hand tiles of other players. From another perspective, it can be explained that the model has not been pre-trained for enough time to better initialize the network parameters, making the network easier to overfit and thus converge earlier, and it finally falls into the local strategy. As a result, it cannot be trained very well and only reaches an average rating of about 4 dan. In contrast, the agent with the feature dropping achieves an average rating of about 5 dan and a highest rating of about 6 dan. However, its rating cannot be further improved even given more training steps. There are two possible reasons: one is that the training data itself is not perfect, and the learning algorithm of the MCTS model needs to be further improved first; the other is that the hyperparameters in the training affects the convergence performance of the agent, and future investigation could focus on adjusting the learning rate and the decrease rate of the parameter $\mathrm{p}$ to achieve a better rating performance of the normal agent.

Finally, we report the ratings of our agent and baseline player in

Table 7. Ranking comparison with different Mahjong engines or human player.

AI Model/Engine	Number of Games	Stable Rank
Proposed agent	3328	5 dan
Baseline player Tenhou-Bot	1095	3 dan
Bakuuchi	30,516	6 dan
Professional human player	-	5 dan

, and we also list the Mahjong model proposed by other researchers [28] as a reference. We can see that our model is 2 dan higher than the baseline player Tenhou-Bot and very close to the Bakuuchi AI engine, with fewer games, and also the professional human player, which indicates a certain effectiveness of our proposed agent.

4.4. Online Experiment

In this section, we conducted the Mahjong test on an online Mahjong platform. The purpose is to make a good-performance Mahjong agent that can replace the human player to take action and win the game. One player is the proposed agent during the online evaluation, and the other three are human players.

It can be seen from Figure 9 that the agent draws six character tiles and makes a decision to discard one of the hand tiles. Due to the winning tiles combinations of Mahjong, and the most similar type of player hand tiles is normal, the agent is ready to discard the unnecessary single six characters. The program of our agent is shown on the left-hand side of Figure 9. As the input features are passed to the program, the agent outputs the fifth action (starting from 0), which exactly means the discard of six characters. We can see that our agent takes the discard action effectively and follows the rules of the Mahjong game.

Figure 10 shows a certain situation of Mahjong. In the above situation, the available action Hu will end the game immediately and obtain the score points from other players, and the Chow action will make a meld and continue the game. In most situations, the player should choose Hu if possible. So, our agent decides to take Hu action and outputs the 258th action (the Hu action type with 7 dots) according to Figure 2 and Table 3, which corresponds to the Hu of 7 dots. This shows that our agent knows the winning conditions of Mahjong and takes the Hu action if possible.

As shown in Figure 11, after the left player discards 8 dots, it is time for the agent to choose whether to take Chow or Pong or Pass. Because it has three different types of tiles and Pong 8 dots, which do not influence the other two types of tiles, our agent decides to make a Pong meld and speeds up the progress to a winning hand. In addition, we can see that the program of the agent receives the information of the game situation and is ready to take the 205th action after seconds of computing, which is the Chow of the 8 dots according to Table 3. It shows that the agent will make a meld in the proper situation and act like a human player.

In the situation of Figure 12, the last player discards 7 dots, and our agent has two 7 dots. If we choose Pong, it will not disassemble the hand tiles. However, the agent outputs the 69th action and decides to choose Pass instead of Pong (204th action), which will miss an opportunity to make a meld quickly. In view of lacking information, the possible reason is that there are too few discarded tiles and our agent hesitates to take Pong, which shows that the agent may not have a good understanding of when to take Pong.

To learn about the more detailed winning strategies, the amounts of different winning scores obtained by our agent are shown in

Table 8. The distribution results of the winning scores.

Score Multiplier	One	Two	Four	Eight	More
Total	378	38	8	3	1
Occurrence Rate	88.3%	8.9%	1.9%	0.7%	0.2%

, where one, two, four, and eight mean the corresponding scores multiplier. The occurrence of the one multiplier is very frequent, because the agent does not consider the score factor as one of the learning objectives, so the strategy is more inclined to the strategy that can win quickly, and usually the strategy with very few points. The more rounds of the game, the greater the impact. Therefore, in the future work, the factor of score should also be taken as one of the goals of the model learning.

In addition, we also participated in the Mahjong group of the 2022 Competitive World Cup national computer game tournament [44] and eventually won third place after combining the proposed agent with other improvement methods. A total of 16 teams were divided into four groups for a total of three rounds. The first-place and second-place teams in each group advanced to the next round of the final round, which was scored based on total points. Our Mahjong agent successfully finished second in the group and moved on to the final round, where it eventually finished third, indicating some performance and efficiency.

5. Discussions and Conclusions

In this paper, we present a new AI learning method and successfully train a Mahjong AI agent to demonstrate the convergence and effectiveness of our method. Different from the existing learning methods, this work first takes advantage of the internalized MDP model and multiplayer MCTS algorithm to obtain the improved game strategy from model-based reinforcement learning. With the almost “perfect” game strategy and the Bernoulli random matrix, the proposed dropout-learning-based normal agent enables faster convergence, better performance, and increased effectiveness in the imperfect-information Mahjong game. The experiments demonstrate the good performance of the proposed MCTS-based model by comparing between different simulation models during RL training. To further verify the effect of feature access and the internalized MDP model on the performance of the model-based learning approach, the models are analyzed with different influencing factors over evaluation simulations. Moreover, the proposed agent is evaluated against the other baseline engine. The results show that the agent trained by our methods achieves a good ranking performance and can play like a human in the Mahjong game.

However, the proposed methods and agent also have some limitations. First, it is very computationally intensive due to the huge search tree and the large state-action space of Mahjong. Second, it is currently only validated for Mahjong and it is not certain whether the proposed methods can be applied in any other game scenarios. Future work will focus on expanding the proposed methods to more complex and general game scenarios and scaling it for training agents under multiple scenarios with optimization of the reward signal. This will be more challenging, as the dimensions of the state and action space will increase exponentially.

Furthermore, the proposed agent utilizes the round result of the game as a learning signal, ignoring the detailed winning score information. The utilized learning signal is too simple, and it is difficult to distinguish the different performances of the different winning scores. Utilizing the additional reward predictor, such as the global reward prediction model, has the potential to reduce the variance of the learning signal for each round and optimize the learning signal by combining the winning score information and the final game result, and further improve our agent.

When faced with more complex game scenarios, how to better represent the current state, transition actions, and the impact of the state transition process in the network is particularly important. In addition, how to better integrate it with convolutional networks or combining the characteristics of other learning networks is also the key to further improving the performance of the model. The feature representation of other games will obviously affect the final model performance. In the dropout learning, in the face of different games, the more hidden information there is, the more difficult it is to train the agent model and the more training time is required.

The proposed method mainly combines several general agent training approaches and the learned network still requires some necessary prior knowledge related to the specific environment. For other multiplayer games, if more prior knowledge and transition information related to the performance of the agent are provided during the state transition, a corresponding better performance can be obtained in theory, and the agent learning method has a certain generalization. Overall, the proposed method gradually constructs the agent model by decomposing the decision-making problems related to specific scenarios, which has certain adaptability and can also be applied in related fields such as robot control and human–computer interaction, providing a new AI agent-based modeling idea.