Improving Strategic Decisions in Sequential Games by Exploiting Positional Similarity

Round 1

Reviewer 1 Report

Referee Report on: “Improving Strategic Decisions in Sequential Games by Exploiting Positional Similarity”

 

Brief Summary: this paper proposes similarity metrics for comparing different game trees to better detect trap states in two-player extensive form games. The aim is to improve the performance of heuristic-like game-playing agents. The similarity metrics are applied to some chess examples to observe how well they perform in detecting trap states.

 

Comments:

1. It seems like the main contribution of the paper is to show that using positional similarity can better detect trap states in games. This contribution is aimed at improving on MCTS artificial intelligence programs, which the authors posit struggle at detecting forcing lines. However, I find the evidence that MCTS-based engines to be deficient along this margin lacking. They provide one example from a Chess Championship match where DeepMind’s AlphaZero (presumably MCTS-based) engine. However, I have trouble assessing how important this deficiency is. It strikes me that a natural question to explore would be to run an off-the-shelf MCTS-based engine on numerous settings along with an MCTS engine augmented with similarity measures and show that the similarity measures do indeed provide an improvement. As it is now, the performance is measured by looking at a handful of chess examples and seeing how effective they are at detecting trap states. However, there is no comparison to how an off-the-shelf MCTS-based engine would perform in these settings. Moreover, it’s clear that a handful of cases are not nearly enough to judge the generality of the performance of these similarity measures.

 

2. I find the use of the Rubinstein quotes on bounded rationality and the general introduction discussing human strategic interactions puzzling in the context of this paper. This paper is about artificial intelligence programs, not about human decision making. Clearly, humans have limits on cognitive ability, and yes, clearly humans use heuristics, but the paper makes no attempt to show that, indeed, humans use similarity-like measures in their heuristic decision making, so I see no connection between what the paper is doing and human decision making. I would suggest re-writing the introduction to avoid presenting things in this context. The paper, at its core, is about improving game-playing agents and is not about human behavior.

 

3. The paper is clearly written by more computer science focused authors. Seeing as Games casts quite a large net in terms of fields, I would suggest defining many terms used in the paper. (e.g. \alpha - \beta methods, forcing sequence, trap states, forcing lines, etc.)

 

4. I would suggest fleshing out the intuition a bit more in the paragraph starting on line 59. From my understanding of the paper, what the similarity measures do are to pick up pieces of different game trees that look very similar/identical. The reason these may be similar/identical is that, for example, the opponent may put you in check and force certain moves going forward. These forcing sequences are likely (? Not sure how generally true this is, so, if possible, some quantification would be helpful) to lead to loss of the game or a significant disadvantage. Insofar as these forcing sequences are detrimental, it behooves players (agents) to avoid them, and the similarity measures help to identify these situations. As I write this, however, I also wonder about similarities that could arise for other reasons. Suppose the agent in question is the aggressor and the optimal strategy is to engage the opponent into a forcing sequence. Shouldn’t, then, a high similarity measure be a good thing? Another way of framing this is that both the aggressive and defensive positions would likely have very similar game trees, how do we leverage similarity metrics to distinguish whether this is good or bad? Of course, if the nodes analyzed are terminal, you can compare the utilities of them, but since the MCTS engines don’t analyze the full game trees, it’s easy to see cases where this is ambiguous. This, of course, relates to my comment above about more thorough testing of this additional measure in a MCTS framework to illustrate that it is indeed an added benefit. At the very least, more discussion of this is needed in the paper.

 

Other minor comments:

• Line 2, remove comma before “, by”
• I think you should entirely re-write the introduction, but in any event, on line 10, you should say “… position to [fully] calculate the….
• Reference to Li and Zhang in the “Tree Edit Similarity” section is inconsistent with the other reference style.
• Line 254 “In general, there was no significant difference…” I see no statistical testing anywhere in the paper…

 

Author Response

Dear Reviewer 1,

We thank you for your detailed comments, which helped us improve our manuscript. We have corrected all the typos and inaccuracies, as suggested.

Responding to your criticism emphasising the mismatch between human and artificial behaviour in decision making, we would like to stress how our model intends to capture any decision-making agent operating in large extensive games. This includes humans and computers, who both need to resort to heuristic assessments in large extensive games in order to take decisions. Artificial Intelligence can then provide us with stylised and testable models of boundedly rational decision-making which we can use to support human decision-making as well. We clarified this point with more strength and restructured the introduction to make it more readable.

 

Reviewer 2 Report

See attached PDF.

Comments for author File: Comments.pdf

Author Response

Dear Reviewer 2,

Thanks so much for your encouraging words and suggestions, which helped us improve the manuscript nonetheless. We endorse your approach to suggestions in review, point to an issue that we often see ourselves. We have corrected all the inaccuracies as suggested, adding further explanation to the definitions which were pointed at.

Round 2

Reviewer 1 Report

The revised manuscript has changed very little. Beyond fixing typos they seem to only respond to one of my comments, and poorly at that.

The fundamental issue remains that they do not test these similarity metrics in MCTS settings. So, there is no way to assess whether these similarity metrics provide a meaningful improvement in the performance of game-playing agents. This fundamental issue is clear to see in the Applications section: "We suggest adding a similarity measure to two MCTS adaptations: the killer heuristic, where decisive moves are evaluated first, and killer RAVE, which only applies RAVE to decisive moves [58]. MCTS may more quickly detect a trap ahead when combined with these similarity-based adaptations." (emphasis added). In other words, the whole paper is about how similarity metrics can improve game-playing agents, but they do not test this conjecture. So how do we know if, indeed, there will be an improvement, and if there are improvements, how meaningful will they be? Absent this evidence, readers have no ability to assess the value of similarity metrics.