Design and Implementation of EinStein Würfelt Nicht Program Monte_Alpha

Abstract

The game of EinStein würfelt nicht involves an element of uncertainty due to die rolling, which poses a big challenge in the development of computer game programs. However, the intriguing nature of probabilistic elements has made this game popular in computer game competitions. This study aimed to develop a high-strength EinStein würfelt nicht program that utilizes an efficient bitboard representation for the game board as well as pre-established probability distribution tables and extensively uses bitwise operations to improve the efficiency of game tree expansion. Additionally, this study attempted to replace random simulation with an evaluation function to enhance the accuracy of the Upper Confidence bounds applied to Trees algorithm. Through this design, we improved the strength of our program, and we hope that this program will be able to achieve additional excellent results in future computer game tournaments.

1. Introduction

EinStein würfelt nicht [1], designed by Ingo Althöfer in 2004, is a two-player board game in which each player has six pieces. The players take turns rolling a die to determine which piece to move. It is a game with complete information but has a probabilistic nature, and designing a strong EinStein würfelt nicht AI is not an easy task due to the probabilistic uncertainty of the game. However, the game’s combination of challenge and entertainment has made EinStein würfelt nicht a popular game in computer game tournaments such as the International Computer Game Association (ICGA) [2], the Taiwanese Association for Artificial Intelligence (TAAI) [3], the Taiwan Computer Game Association (TCGA) [4], and the Chinese Association for Artificial Intelligence (CAAI) [5].

EinStein würfelt nicht is played on a 5 × 5 board, with both red and blue players each having six pieces numbered from one to six. At the start of the game, the pieces can be placed at random or in a particular order within their respective triangular positions, as shown in Figure 1a. The movement rules are as follows: red pieces can only move one square to the right, bottom-right, or down, while blue pieces can only move one square to the left, top-left, or up. Then, the two players take turns rolling the die to determine which piece they can move. If there is already a piece in the destination square, it must be removed from the board, regardless of whether it belongs to the opponent or the player, as shown in Figure 1b. If the die lands on a number corresponding to a piece that is no longer on the board, the player can choose to move a piece whose number is next-higher or next-lower to the rolled number, as shown in Figure 1c. The goals for both red and blue players are situated in opposite corners, as indicated by the flags in Figure 1d. The winner is the player who reaches the goal first with one piece or who captures all of their opponent’s pieces. Therefore, the outcome of an EinStein würfelt nicht game can only be a win or loss, and there are no draws.

In most games, having more pieces typically confers an advantage, but this is not the case in EinStein würfelt nicht. When there are more pieces on the board, the probability of moving any particular piece decreases. For example, if a player has all 6 pieces on the board, the probability of moving any given piece each turn is $\frac{1}{6}$. This makes it difficult to move a specific piece to the goal quickly. However, if a player has only one piece left on the board, the probability of moving that piece each turn is one, thereby allowing this piece to reach the goal quickly. Therefore, strategically reducing one’s own pieces on the board can be a smart tactic. However, if a player has too few pieces on the board, there is also a relative increase in risk because the player may not have a backup force to eliminate enemy forces that are on the verge of victory.

The rules of EinStein würfelt nicht might seem straightforward, but the gameplay is filled with strategy, with striking a balance between offense and defense presenting a significant challenge. Therefore, this study aimed to develop a high-strength EinStein würfelt nicht program and validate its game-playing abilities through participation in computer games tournaments. This paper extends our previous work presented in [7] by correcting errors in the probability distribution table and by replacing random simulation with an evaluation function, which significantly improved the strength of our EinStein würfelt nicht program. We also hope that these improvements will enable our program to achieve additional excellent results in future competitions.

The remainder of this paper is organized as follows: Section 2 describes the background of the Upper Confidence bounds applied to Trees (UCT) algorithm [8,9]. Section 3 provides a brief overview of related works on the topics under investigation. Section 4 introduces the design and implementation of our EinStein würfelt nicht program, Monte_Alpha. Section 5 presents the experimental results, and Section 6 describes the tournament results. Section 7 introduces further enhancements, and the conclusions are reported in Section 8.

2. Background

The Monte Carlo method [10] is a statistical technique that enables numerical solutions to intricate problems via random sampling. In 2006, Coulom introduced the Monte-Carlo Tree Search (MCTS) [11], which applies the Monte Carlo method to game tree search. Later that same year, Kocsis and Szepesvári formulated the Upper Confidence bounds applied to Trees (UCT) algorithm [8] to address MCTS’s shortcomings.

The UCT algorithm uses the Monte Carlo methods to sample the search space and utilizes the multi-armed bandit strategy [12] to determine the subsequent exploration path. UCT allocates more computational resources toward the most promising areas of the game tree, as determined by the outcomes of the Monte Carlo simulations. If given unlimited computational resources, the UCT is theoretically capable of converging toward the optimal solution. The process of the UCT algorithm, depicted in Figure 2, consists of the following four stages [9]:

• Selection: The process begins at the root node and descends through the tree until reaching a leaf node that has unvisited child nodes. In this study, we employed the UCB’ Formula (1) [9] to maintain a balance between the aspects of exploitation and exploration within the search tree.

$$\mathrm{U}\mathrm{C}\mathrm{B}\mathrm{\prime }\left({\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{l}\mathrm{d}}_{\mathrm{i}}\right)=\left\{\begin{array}{c}\frac{{\mathrm{w}}_{\mathrm{i}}}{{\mathrm{n}}_{\mathrm{i}}}+\sqrt{\frac{2\ln {\mathrm{n}}_{\mathrm{p}}}{{\mathrm{n}}_{\mathrm{i}}}}, \mathrm{c}\mathrm{u}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{t} \mathrm{n}\mathrm{o}\mathrm{d}\mathrm{e} \mathrm{i}\mathrm{s} \mathrm{t}\mathrm{h}\mathrm{e} 1\mathrm{s}\mathrm{t} \mathrm{p}\mathrm{l}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\\ \left(1-\frac{{\mathrm{w}}_{\mathrm{i}}}{{\mathrm{n}}_{\mathrm{i}}}\right)+\sqrt{\frac{2\ln {\mathrm{n}}_{\mathrm{p}}}{{\mathrm{n}}_{\mathrm{i}}}}, \mathrm{c}\mathrm{u}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{t} \mathrm{n}\mathrm{o}\mathrm{d}\mathrm{e} \mathrm{i}\mathrm{s} \mathrm{t}\mathrm{h}\mathrm{e} 2\mathrm{n}\mathrm{d} \mathrm{p}\mathrm{l}\mathrm{a}\mathrm{y}\mathrm{e}\mathrm{r}\end{array}\right.$$

(1)

where ${\mathrm{n}}_{\mathrm{i}}$ stands for the total number of simulations that the i-th child node (${\mathrm{c}\mathrm{h}\mathrm{i}\mathrm{l}\mathrm{d}}_{\mathrm{i}}$) has undergone, ${\mathrm{w}}_{\mathrm{i}}$ stands for the total number of wins within ${\mathrm{n}}_{\mathrm{i}}$, and ${\mathrm{n}}_{\mathrm{p}}$ stands for the total number of simulations that the current (parent) node has undergone. Therefore, the term $\frac{{\mathrm{w}}_{\mathrm{i}}}{{\mathrm{n}}_{\mathrm{i}}}$ represents the win rate of the i-th child from the perspective of the first player, and the term $\left(1-\frac{{\mathrm{w}}_{\mathrm{i}}}{{\mathrm{n}}_{\mathrm{i}}}\right)$ represents the win rate of the i-th child from the perspective of the second player. The first component (win rate) of the UCB’ formula promotes the exploitation of nodes with higher rewards. The other component ($\sqrt{\frac{2\ln {\mathrm{n}}_{\mathrm{p}}}{{\mathrm{n}}_{\mathrm{i}}}}$) encourages the exploration of nodes that have been visited less frequently. In each layer of the game tree, the child node with the highest UCB’ value is selected as the most promising move.

2.

Expansion: A child node (or multiple) is expanded to the existing leaf node unless the selected node represents the end of the game. In this work, we restricted the expansion to a single child node for every simulated game.

3.

Simulation: Execute a random playout (or multiple) from the newly expanded node until the game concludes. In this study, we limited ourselves to a single random playout for each game simulation.

4.

Backpropagation: Accumulate the game’s results from each playout, initiating from the newly expanded node and proceeding upwards to the root node.

3. Related Works

After the release of EinStein würfelt nicht in August 2004, Althöfer guided Schäfer in the development of an EinStein würfelt nicht program over the following year [13]. This was the first publicly available EinStein würfelt nicht program that utilized the Minimax algorithm [14], the Monte Carlo method [10], and heuristics as its computational techniques. Since then, many stronger EinStein würfelt nicht programs have used the Minimax algorithm. In 2011, Lorentz [15] used Monte Carlo Tree Search to develop an EinStein würfelt nicht program that could reach the same level as the Minimax algorithm. Subsequently, most programs have been implemented using the Monte Carlo Tree Search (MCTS) algorithm [15,16,17,18,19]. Li et al. [17] optimized the Upper Confidence bounds applied to Trees (UCT) algorithm as a node selection strategy [20], significantly enhancing the strength of EinStein würfelt nicht programs. Wang et al. [19] proposed an optimized UCT algorithm with an evaluation function that provided a more precise evaluation of the game situation.

However, many studies have designed EinStein würfelt nicht programs based on the Monte Carlo method and have implemented fine-tuned evaluations [18,19,21,22,23,24]. Zhang et al. [24] designed a set of evaluation functions targeting the distances of pieces and the probability of pieces being rolled, achieving a good balance between offense and defense in the program. Additionally, some EinStein würfelt nicht programs continue to use conventional search algorithms from the Minimax series. For example, Li et al. [25] proposed an Offensive and Defensive Expect Minimax Algorithm to find optimal moves and achieved a high win rate using the traditional Expect Minimax Algorithm. Bonnet and Viennot [26] used a *-Minimax algorithm [27,28] that included both α-cuts and β-cuts to prune branches while executing expectiminimax.

In addition, regarding the theoretical research on EinStein würfelt nicht, Turner [29] established an endgame database in 2012 to analyze the game records of the ICGA 2011 EinStein würfelt nicht competition. Lu et al. [22] discussed the optimal placement of pieces on the initial board to increase the chances of winning. Furthermore, Bonnet and Viennot [30] not only analyzed the optimal solutions when both players have only one piece remaining but also calculated the optimal winning probability on a 4 × 4 board size using the Nash equilibrium [26].

In recent years, the prevalent approach for implementing programs in EinStein würfelt nicht has been the utilization of the Monte Carlo Tree Search algorithm. Advanced EinStein würfelt nicht programs have adopted algorithms from the AlphaZero series for training purposes [31,32]. Hsueh et al. [33] employed the AlphaZero algorithm to learn the theoretical values and optimal strategies for stochastic games. This is due to the fact that the theoretical values for stochastic games are anticipated win rates rather than simple win, loss, or draw outcomes. Furthermore, some programs have incorporated techniques that have combined the MCTS algorithm with n-tuple networks [34,35].

4. Design and Implementation of Monte_Alpha

This section introduces the design and implementation of our EinStein würfelt nicht program, Monte_Alpha. We will elaborate on three aspects: the representation of the game board, the search algorithm, and the design of the die roll. The following sections will explain each of these three parts in detail.

4.1. Bitboard Representation

Monte_Alpha uses a one-dimensional array data structure to record the state of the board. In order to make the program execution more efficient, many bitwise operations are used in the development of the program. Therefore, the board representation is also presented in hexadecimal format. The Board (which comprises 256 spaces (16$\times$16 elements)) is used to record the state of the board and is as shown in Figure 3. Herein, the value of the board position, Board [position], ranges from −1 to 12, where −1 denotes that this position is outside of the range of the game board, 0 indicates that there is no piece in this position, 1 to 6 represent the blue pieces numbered 1 to 6, respectively, and 7 to 12 represent the red pieces numbered 1 to 6, respectively.

Additionally, the used fields are the 25 squares in the orange area (0x11~0x55) in Figure 3. Although this design scheme wastes a large amount of storage space and is not as efficient as the pure bitboard scheme [35,36], it has the following advantages:

• Fast move generation: Taking Figure 3 as an example, the red piece with the number 3 (0x44) can move to the right (0x45), down (0x54), or diagonally down to the right (0x55). The move can be generated quickly by adding an increment (INC). The operation is as follows:

$$\mathrm{R}\mathrm{e}\mathrm{d}\left(3\right)=0\mathrm{x}44+\mathrm{I}\mathrm{N}\mathrm{C}$$

(2)

where INC = {0x01, 0x10, 0x11}.

• Fast validation of move legality: The performance of the bitboard is much faster than using a two-dimensional array data structure [37]. It can be quickly determined whether the movement of a piece is within the valid range of the board. Continuing the example in Figure 3, suppose the right move (0x45) for the red piece with number 3 is expanded. The validation operation is as follows:

$$\mathrm{B}\mathrm{o}\mathrm{a}\mathrm{r}\mathrm{d}\left[0\mathrm{x}45\right]!=-1$$

(3)

Since there is no piece in position 0x45, $\mathrm{B}\mathrm{o}\mathrm{a}\mathrm{r}\mathrm{d}\left[0\mathrm{x}45\right]$= 0 and the value of Formula (3) is true, which means the right move (0x45) for the red piece with the number 3 is legal. In another example, suppose the diagonally down move (0x36) for the red piece with the number 6 is expanded. The validation operation is as follows:

$$\mathrm{B}\mathrm{o}\mathrm{a}\mathrm{r}\mathrm{d}\left[0\mathrm{x}36\right]!=-1$$

(4)

Since the position 0x36 is outside of the range of the game board, as shown in the gray area in Figure 3, Board [0x36] = −1 and the value of Formula (4) is false, which means the diagonally down move (0x36) for the red piece with number 6 is illegal.

• Quick access to board coordinates: Continuing the example in Figure 3, suppose the right move (0x45) is selected. The board coordinates can be obtained quickly through the following bitwise operations:

$$\mathrm{R}\mathrm{o}\mathrm{w}=0\mathrm{x}45\gg 4$$

(5)

$$\mathrm{C}\mathrm{o}\mathrm{l}=0\mathrm{x}45 \& 0\mathrm{x}0\mathrm{F}$$

(6)

where its corresponding coordinates are the fourth row (high-order hexadecimal digit) and the fifth column (low-order hexadecimal digit).

• Convenient for developers to debug: Using a two-dimensional array or tuple has the advantage of high readability while using a bitboard has the advantage of better performance. Although the overall execution performance is a little lower than when using a pure bitboard, the representation of our bitboard is as simple and clear as using a two-dimensional array.

4.2. Search Algorithm

Due to the probabilistic nature of the die-rolling phase in Einstein würfelt nicht, using the Monte Carlo method represents a good option. Therefore, the search algorithm of Monte_Alpha adopts the Upper Confidence bounds applied to Trees (UCT) algorithm introduced in Section 2, which uses the UCB’ formula in Monte Carlo Tree Search (MCTS). Because of the die-rolling steps involved in Einstein würfelt nicht, the expansion of the game tree becomes more complex. As a result, the original UCT cannot be directly applied to Einstein würfelt nicht. Figure 4 illustrates the search algorithm architecture of Monte_Alpha. The root node is special because its board state includes the number of points rolled on the die, determining which pieces (two pieces at most) can be moved, and each piece has at most three possible moves. In the example in Figure 4, the given board comes with a die roll of 4, but red piece number 4 is no longer on the board. Therefore, the red player can choose either piece number 3 or 6, where red piece number 3 has three possible moves, and red piece number 6 can only move downward to capture blue piece number 6.

Furthermore, the nodes expanded below the root node are different from the root node. The expansion process requires simulating the rolling of the die to determine which pieces can be moved. Therefore, all possible die rolls from one to six must be considered. In the example in Figure 4, the only blue piece that remains on the board is piece number 6, so no matter what number the die rolls, the blue player can only move piece number 6. In addition, the expansion process of the game tree follows the four steps of UCT. Finally, the move with the highest number of simulations is selected as the best decision for the given game state.

4.3. Design of Die Rolling

When designing the die-rolling action, the most intuitive method would be to first randomly generate a number between one and six and then check whether the piece with this number is still on the board. If the piece with this number is no longer on the board, the program would then look to both the left and right to find the piece with the closest number that is still on the board. For instance, as illustrated in Figure 3, suppose the red player rolls a three, and the red piece labeled number 3 is still on the board. This is the best case and requires searching once. Alternatively, suppose the blue player rolls a one, and the blue piece labeled number 1 is no longer on the board. The program would then need to look to the right to find the adjacent piece that is still on the board, which is piece number 6. This is the worst case and requires searching six times. In general, if the rolled piece is no longer on the board, the program needs to search for the adjacent piece, which requires searching approximately three times on average, but the program also needs to check the pieces on the left and the right, so the overall number of searches is approximately six.

During implementation, Monte_Alpha uses bitwise operations, adding two variables to record the state of the blue and red pieces. For instance, with reference to Figure 3, the binary value of the state of the blue pieces is as follows:

$$\mathrm{b}\mathrm{l}\mathrm{u}\mathrm{e}\_\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{v}\mathrm{e}=0\mathrm{b}100000$$

(7)

where the least significant bit represents piece number 1, and the most significant bit represents piece number 6. Similarly, the binary value of the red player’s pieces state is as follows:

$$\mathrm{r}\mathrm{e}\mathrm{d}\_\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{v}\mathrm{e}=0\mathrm{b}100101$$

(8)

To check whether the blue player still has pieces on the board, it is only necessary to determine whether blue_alive is not equal to zero. For example, in the right branch of Figure 4, red piece number 6 will capture blue piece number 6. Afterward, blue_alive will become zero, indicating that the red player has captured all enemies and won the game. Similarly, the red player can also quickly determine whether there are still pieces in play using this command. Since there are only 2⁶ combinations of a single player’s piece status, a probability distribution table can be pre-established for all piece combinations to speed up the program’s processing time.

In order to accelerate the efficiency of rolling the die, the probability of each die number appearing is converted into the number of times it appears and is stored in a probability table to speed up obtaining information about which pieces can be moved. Taking the red pieces in Figure 3 as an example, the binary representation of the red piece status is 100101, which means that the pieces numbered 1, 3, and 6 are still on the board. The probabilities of those three pieces that can be moved are $\frac{1.5}{6}$ (the probability of rolling a 1 is $\frac{1}{6}$, and there is a half chance of moving piece 1 when rolling a 2, with a probability of $\frac{0.5}{6}$), $\frac{2.5}{6}$ (the probability of rolling a 3 is $\frac{1}{6}$, and there is a half chance of moving piece 3 when rolling a 2, 4 or 5, with a probability of $\frac{0.5}{6}+\frac{0.5}{6}+\frac{0.5}{6}$), and $\frac{2}{6}$ (the probability of rolling a 6 is $\frac{1}{6}$, and there is a half chance of moving piece 6 when rolling 4 or 5, with a probability of $\frac{0.5}{6}+\frac{0.5}{6}$), respectively.

Position 100,101 in the probability table is a 10,000-element array with a value of one for elements 0 to 2499, a value of three for elements 2500 to 6666, and a value of six for elements 6667 to 9999. This indicates that when rolling the die 10,000 times, there are 2500 chances to move piece 1, 4167 chances to move piece 3, and 3333 chances to move piece 6. The contents of the array are as follows:

$$\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y}\left[100101\right]=\{1,\dots ,1,3,\dots ,3,6,\dots ,6\}$$

(9)

In this way, it only takes checking which pieces can be moved once by looking up the table. The operation is as follows:

$$\mathrm{m}\mathrm{o}\mathrm{v}\mathrm{e}=\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y}\left[100101\right]\left[\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{d}\right()\mathrm{\%}10000$$

(10)

During the process of game tree expansion, each simulation needs to go through a series of moves to reach the end state of the game. Assuming the depth of the simulation is $d$, each simulation with Monte_Alpha only takes $d$ times, whereas the conventional method requires $6\times d$ times. Therefore, as long as the number of simulations increases, the acceleration effect of Monte_Alpha is considerable, and this is where the advantage of Monte_Alpha lies.

5. Experimental Results

This study conducted experiments on a personal computer equipped with an AMD R5-3600 CPU (6 cores; 12 threads) and a Micron Ballistix DDR4-3200 16G RAM (8G × 2). The counterparts for comparison are three other Einstein würfelt nicht programs based on different search algorithms:

• Monte_Alpha: This is the EinStein würfelt nicht program developed in this paper. It can simulate approximately 200,000 times per second.
• Greedy: This strategy uses the shortest path algorithm to move a piece toward the goal.
• UCT_cut: This strategy uses the UCT algorithm with pruning and can simulate approximately 50,000 times per second.
• ABS_eval: This strategy uses alpha–beta pruning combined with a transposition table and a game evaluation function based on the level of threat [38].

At the beginning of the experiment, we conducted 400 games (200 as the first player and 200 as the second player) to compare Greedy and Monte_Alpha with different numbers of simulations.

Table 1. Win rates of Monte_Alpha against Greedy with different simulation numbers. Data obtained from [7].

Simulations	Monte_Alpha
	Wins	Losses	Win Rate
10,000	163	237	40.75%
50,000	292	108	73.00%
100,000	285	115	71.25%
500,000	264	136	66.00%
1,000,000	262	138	65.50%
1,500,000	289	111	72.25%
2,000,000	300	100	75.00%
2,500,000	271	129	67.75%
3,000,000	271	129	67.75%
3,500,000	277	123	69.25%
4,000,000	283	117	70.75%
4,500,000	273	127	68.25%
5,000,000	291	109	72.75%

shows the win rates of different versions of Monte_Alpha against Greedy. From the experimental results, it can be seen that the version of Monte_Alpha with 10,000 simulations cannot outperform the Greedy strategy, with a win rate of only 40.75%. However, all versions of Monte_Alpha with more than 50,000 simulations significantly outperform Greedy. However, due to the probabilistic nature of the game, the win rate exhibits slight fluctuations. The experimental results show that increasing the number of simulations will not continue to increase the win rate. It can be inferred that the statistical results obtained from a number of simulations exceeding 50,000 converged to the local optimal solution, so the win rate of Monte_Alpha against Greedy’s shortest path is approximately 70% (±5%).

Next, we conducted extensive matches between Monte_Alpha and the three other strategies, with each program allotted the same execution time (approximately 1.2 s per move). The results of these matches are shown in

Table 2. Win rates of Monte_Alpha against other search algorithms. Data obtained from [7].

Opponent	Monte_Alpha
	Wins	Losses	Win Rate
Greedy	10,597	4403	70.65%
UCTcut	7699	7301	51.33%
ABSeval	7925	7075	52.83%

. The experimental results demonstrate that Monte_Alpha does indeed have a win rate of approximately 70% against Greedy. Additionally, the strengths of Monte_Alpha, UCT_cut, and ABS_eval are comparable, which confirms that simply increasing the number of simulations does not significantly improve the strength of the program.

6. Computer Game Tournaments

Our EinStein würfelt nicht program, Monte_Alpha, participated in the ICGA 2022 and TAAI 2022 computer game tournaments. The results of each competition will be described in the following section.

6.1. ICGA 2022

The ICGA 2022 competition was held from July 23 to 29 and was conducted online. Although there were only three teams participating in the EinStein würfelt nicht competition this year, the opponents were all very strong. The techniques used by Monte_Alpha have been introduced in Section 4 of this paper. Meanwhile, Reinstein utilized the bitboard, MCTS, and n-tuple networks techniques. Additionally, YJ_EinStein employed the bitboard and alpha–beta pruning techniques.

During the regular competition, each pair of competitors played 12 games, with each playing 6 games as the first player and 6 games as the second player. At the end of the regular competition, both leading teams, Monte_Alpha and Reinstein, scored 15 points, leading them into the overtime playoffs. The overtime match consisted of an additional six games, with each team having an equal number of turns to play first and second. However, the two teams remained neck and neck during the overtime match, resulting in another tie. As a result, they proceeded to a second overtime match. During the second overtime match, Monte_Alpha maintained a lead throughout, ultimately securing a 5 to 1 victory and winning the gold medal for the EinStein würfelt nicht competition at ICGA 2022.

Table 3. Results of the EinStein würfelt nicht competition at ICGA 2022. Data obtained from [7].

Program	Monte_Alpha	Reinstein	YJ_EinStein
Monte_Alpha	-	6	3
Reinstein	6	-	3
YJ_EinStein	9	9	-
Total	15	15	6
Playoff 1	3	3	-
Playoff 2	5	1	-
Final	8	4	-
Rank	1	2	3

presents the results of the competition. The related competition rules and competition platform can be found on the official website of the ICGA.

6.2. TAAI 2022

The TAAI 2022 competition was held in Tainan on December 3rd. This year, there were four teams participating, including Monte_Alpha and Reinstein, as well as two newcomers, Deku and Ssunoo. Since Section 4 of this paper was published on December 1st, some participants knew about our design and implemented our method in their program. Deku used UCT coupled with pruning techniques (UCT_cut, as mentioned in Section 5) and the method mentioned in Section 4.3; Ssunoo used alpha–beta search paired with a fine-tuned evaluation function (ABS_eval, as mentioned in Section 5) and the method mentioned in Section 4.3.

The competition made use of the automated competition platform developed by National Taipei University. This platform significantly reduced the time required for human operation. Furthermore, to minimize the influence of luck on the rankings, each pair of competitors played 100 games during the regular competition, with each player going first in 50 games. At the end of the regular competition, Deku led all competitors, while Monte_Alpha and Ssunoo tied for second place, and Reinstein trailed behind all of the other competitors. As Monte_Alpha and Ssunoo had the same score in the regular competition, they went into overtime. The overtime competition consisted of 100 games, and Monte_Alpha and Ssunoo tied again with 50 games each, leading to a second overtime match. Due to time constraints, the second overtime competition consisted of only 10 games, and ultimately, Ssunoo defeated Monte_Alpha 6 to 4. Unfortunately, Monte_Alpha only achieved third place in this competition.

Table 4. Results of the EinStein würfelt nicht competition at TAAI 2022.

Program	Deku	Ssunoo	Monte_Alpha	Reinstein
Deku	-	45	47	35
Ssunoo	55	-	41	60
Monte_Alpha	53	59	-	44
Reinstein	65	40	56	-
Total	173	144	144	139
Playoff 1	-	50	50	-
Playoff 2	-	6	4	-
Final	-	56	54	-
Rank	1	2	3	4

shows the results of the EinStein würfelt nicht competition at TAAI 2022, with the competition rules following those of ICGA 2022.

7. Further Enhancement

Given that Deku and Ssunoo leveraged the method outlined in Section 4.3 to enhance the efficiency of their programs, ultimately surpassing Monte_Alpha, this study has made several improvements to Monte_Alpha, which are detailed in the sections below.

7.1. Enhancement of Die Rolling

In the design of the method described in Section 4.3, the probability table is an array that includes 10,000 elements, introducing a slight error. Let us take the red piece in Figure 3 as an example. For position 100,101 in the probability table, there should be 2500 ($\mathrm{10,000}\times \frac{1.5}{6}$) values of 1, 4166.67 ($\mathrm{10,000}\times \frac{2.5}{6}$) values of 3, and 3333.33 ($\mathrm{10,000}\times \frac{2}{6}$) values of 6. Herein, we observe that if the denominator of the probability of a piece appearing is expanded to 12, the numerator will also be an integer. This implies that we only need 12 elements in the probability table and that the number of times each piece appears will be an integer. Therefore, position 100,101 in the probability table in Figure 3 changes to having 3 ($12\times \frac{1.5}{6}$) values of 1, 5 ($12\times \frac{2.5}{6}$) values of 3, and 4 ($12\times \frac{2}{6}$) values of 6. Consequently, the calculation of the probability of a piece appearing becomes more accurate and avoids errors.

In addition, the revised probability is calculated using an integer data type, which also enhances the performance of reading from the probability table.

Table 5. Performance of different table sizes.

Size	108 Accesses	Access Number/ms	Speedup
10,000 elements	3342 ms	29,922	1
12 elements	2750 ms	36,364	1.2

lists the time taken to access the probability table using the old and new designs, with the new design achieving a speedup ratio of 1.2 times.

7.2. Enhancement of Simulation

The results of a single Upper Confidence bounds applied to Trees (UCT) simulation are not reliable. A larger number of simulations is required to gradually converge toward the optimal solution based on the statistical results. To make the simulation results more reliable, this study replaced the simulation phase of UCT with an evaluation function [9,39]. As a result, the simulation process no longer needs to unfold until the end of the game, and, moreover, the results of excessively long-term simulations are also less reliable.

Herein, our evaluation function was designed based on a goal-oriented concept and was specifically designed using the reciprocal of the shortest distance from the pieces to the goal. Taking Figure 3 as an example, if the red player rolls a three, corresponding to a specific piece, and the shortest distance from that piece to the goal is one, it will receive an update of one point. Similarly, if the red player rolls a six and the shortest distance from that piece to the goal is three, it will receive an update of $\frac{1}{3}$ of a point. This design aligns with the idea that the closer a piece is to the goal, the higher its winning probability.

As the improvements mentioned in Section 7.1 only corrected probability errors and improved the speed of table lookup without significantly impacting the strength of the program, this study did not conduct independent testing specifically for the method in Section 7.1. However, a new version was developed incorporating improvements from both 7.1 and 7.2 called Monte_Alpha v2. We conducted 5000 matches between Monte_Alpha v2, the previous version of Monte_Alpha, and Deku. The results in

Table 6. Win rates of Monte_Alpha v2 against Monte_Alpha and Deku.

Opponent	Monte_Alpha v2
	Wins	Losses	Win Rate
Monte_Alpha	3300	1700	66%
Deku	2684	2316	53.68%

show that Monte_Alpha v2 won 3300 matches, achieving a win rate of 66% against the previous version of Monte_Alpha. Additionally, in the 5000 matches against Deku, Monte_Alpha v2 won 2684 matches, achieving a win rate of 53.68%. These experimental results demonstrate the significant effect of the improvements discussed in Section 7.

8. Discussion

The EinStein würfelt nicht program developed in this study called Monte_Alpha utilizes an efficient bitboard representation for the game board. It is complemented with an efficient die-rolling design scheme and extensively employs bitwise operations, enabling the program to quickly expand nodes during Upper Confidence bounds applied to Trees (UCT) search. The experimental results in Section 5 demonstrate that although our program’s execution speed outperforms other competitors, continuously increasing the number of simulations beyond a certain threshold does not effectively enhance the program’s playing strength. In addition, this study attempted to replace random simulation with an evaluation function, aiming to enhance the quality of UCT and make the search process both faster and more accurate. The experimental results of the new version Monte_Alpha v2 in Table 6 demonstrate that using an evaluation function instead of random simulation significantly improves the program’s playing strength. We hope that this program could achieve commendable results in future computer game competitions.

Currently, our EinStein würfelt nicht program has achieved a high level of efficiency, so improving its execution speed is no longer our primary focus for future enhancements. Instead, we plan to explore developing more advanced evaluation functions. For example, we will consider the opponent’s pieces as they appear on the path of movement or strategically reduce the player’s own number of pieces to increase the chances of movement for other pieces. Additionally, integrating the utilization of opening [13] and endgame databases [29,30] and incorporating calculations based on the Nash equilibrium [26,40] are also worth considering. These ideas may enhance the accuracy of the simulation results and further elevate the program’s performance.