Next Article in Journal
Achievement in Fundamental Movement Skills, Spatial Abilities, and Mathematics among Lower Key Stage 2 Children
Next Article in Special Issue
Why Should We Study the Foreign Language Effect: Debiasing through Affecting Metacognition?
Previous Article in Journal
Does Thinking in Opposites in Order to Think Differently Improve Creativity?
Previous Article in Special Issue
Overconfidence in the Cognitive Reflection Test: Comparing Confidence Resolution for Reasoning vs. General Knowledge
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
J. Intell.
Volume 11
Issue 5
10.3390/jintelligence11050086
jintelligence-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Tracing Cognitive Processes in Insight Problem Solving: Using GAMs and Change Point Analysis to Uncover Restructuring

by
Mario Graf
1,*,
Amory H. Danek
2,
Nemanja Vaci
3 and
Merim Bilalić
4
1
Institute of Psychology, University of Klagenfurt, 9020 Klagenfurt, Austria
2
Department of Psychology, Heidelberg University, 69117 Heidelberg, Germany
3
Department of Psychology, Sheffield University, Sheffield S10 2BP, UK
4
Department of Psychology, University of Northumbria at Newcastle, Newcastle upon Tyne NE1 8ST, UK
*
Author to whom correspondence should be addressed.
J. Intell. 2023, 11(5), 86; https://doi.org/10.3390/jintelligence11050086
Submission received: 16 March 2023 / Revised: 24 April 2023 / Accepted: 26 April 2023 / Published: 3 May 2023
(This article belongs to the Special Issue Metareasoning: Theoretical and Methodological Developments)
Browse Figures
Versions Notes

Abstract

:
Insight problems are likely to trigger an initial, incorrect mental representation, which needs to be restructured in order to find the solution. Despite the widespread theoretical assumption that this restructuring process happens suddenly, leading to the typical “Aha!” experience, the evidence is inconclusive. Among the reasons for this lack of clarity is that many measures of insight rely solely on the solvers’ subjective experience of the solution process. In our previous paper, we used matchstick arithmetic problems to demonstrate that it is possible to objectively trace problem-solving processes by combining eye movements with new analytical and statistical approaches. Specifically, we divided the problem-solving process into ten (relative) temporal phases to better capture possible small changes in problem representation. Here, we go a step further to demonstrate that classical statistical procedures, such as ANOVA, cannot capture sudden representational change processes, which are typical for insight problems. Only nonlinear statistical models, such as generalized additive (mixed) models (GAMs) and change points analysis, correctly identified the abrupt representational change. Additionally, we demonstrate that explicit hints reorient participants’ focus in a qualitatively different manner, changing the dynamics of restructuring in insight problem solving. While insight problems may indeed require a sudden restructuring of the initial mental representation, more sophisticated analytical and statistical approaches are necessary to uncover their true nature.

1. Introduction

In cognitive science, the temporal dynamics of problem-solving processes have always been an important topic of investigation. Most problems are assumed to be solved gradually, by piecing together information in order to arrive at a solution (Newell and Simon 1972). To investigate these problems, several tools have been developed, which allow for the observation of each step of the problem-solving process (e.g., Tower of Hanoi, Hobbits and Orcs problem). In the case of “insight problems”, the solution often comes seemingly out of nowhere (Duncker 1945), despite the problem appearing unsolvable just a moment earlier. To be solved, insight problems are thought to require a fundamental, sudden change in the way the problem is perceived, a process referred to as restructuring or representational change (Ohlsson 1992; Wertheimer 1925). The restructuring from the initial, incorrect mental representation to the correct one is the key component in modern theories such as representational change theory (RCT) (Knoblich et al. 1999; Ohlsson 1984, 1992, 2011).
Although the sudden nature of the underlying restructuring process is a main theoretical assumption about insight, the evidence for this claim is inconclusive. Ohlsson (1992) even hypothesized that “the sudden appearance of the complete solution in consciousness is an illusion caused by our lack of introspective access to our cognitive processes (...)” (p. 17). To truly understand the temporal nature of insight, the cognitive component of insight (restructuring) must be examined with appropriate tools. Observing changes in solvers’ mental problem representation is a methodological and statistical challenge, which is addressed in the present work. Among the reasons for this lack of clarity is that many measures of insight rely solely on the solvers’ subjective experience of the solution process. Using matchstick arithmetic problems, we demonstrate that it is possible to objectively trace problem-solving processes.
We first review the research on representational change, focusing on the experimental designs. After that, we describe a novel analytical approach that improves upon previous attempts. Finally, and arguably most importantly, we show that this analytical approach needs to be combined with appropriate statistical tools in order to work properly. We demonstrate the feasibility of this approach by re-analyzing eye-tracking data from an already published study (Bilalić et al. 2019). The paper is accompanied by an online supplement, with technical details, such as data and code for the analysis, which is freely available at https://osf.io/pwuhs/?view_only=7c52bda4e6fa481e826e5d7570b6ef3 (accessed on 25 April 2023).

1.1. Temporal Dynamics of the Restructuring Process

In 1994, Durso and colleagues conducted an early study on the temporal dynamics of insight problem solving. They asked participants to rate the relatedness of word pairs in a word puzzle and found that, on average, solution-relevant pairs were rated as increasingly similar as participants approached a solution. The authors concluded that “[l]ike dynamite, the insightful solution explodes on the solver’s cognitive landscape with breathtaking suddenness, but if one looks closely, a long fuse warns of the impending reorganization” (Durso et al. 1994, p. 98). Novick and Sherman (2003; Experiment 2) provided similar evidence. They asked participants to indicate within a short time window (250 ms after stimulus offset) whether presented anagrams were solvable. They found that, although participants could not find the solution within the allotted time, they were increasingly better at differentiating between solvable and unsolvable anagrams as the presentation time of the anagrams increased. The authors concluded that solvers gradually accumulate information relevant for solving the anagrams.
Several studies have focused on the concept of restructuring in insight problem solving, but have typically not measured the dynamics of the solving process (e.g., Ash et al. 2012; Ash and Wiley 2006, 2008; Fleck and Weisberg 2013; MacGregor and Cunningham 2009). However, a number of studies have attempted to measure the temporal dynamics of restructuring, using different methods to acquire trace data. Some used repeated ratings of problem elements, either regarding their similarity (Durso et al. 1994) or with regard to their relevance for the solution (Cushen and Wiley 2012; Danek et al. 2020). Others recorded eye movements (Ellis et al. 2011; Knoblich et al. 2001; Bilalić et al. 2019; Tseng et al. 2014) or employed solvability judgments (Novick and Sherman 2003). In some of these studies, both incremental and sudden solution patterns were found (Cushen and Wiley 2012; Danek et al. 2020; Novick and Sherman 2003), whereas other studies found only incremental patterns (Durso et al. 1994).

1.2. Eye Movements and Matchstick Arithmetic Problems

Here, we will take a closer look at using eye movement recordings to measure the temporal dynamics of restructuring in insight problems (for a comprehensive overview on eye movements, please see Holmqvist et al. 2011). In general, eye movements provide an objective measure of cognitive processes, as they are closely linked to attention (e.g., Just and Carpenter 1976; Rayner 1995; Reingold et al. 2001). Specifically, eye fixations reveal when people pay attention to certain features of a problem and for how long. More importantly, eye tracking is particularly useful when participants might not remember or even concurrently report that they are paying attention to these elements (Bilalić and McLeod 2014; Bilalić et al. 2008, 2010; Kuhn and Land 2006; Kuhn et al. 2009). This is particularly relevant in the case of insight problems, where it is possible that people are not aware of the dynamics of their solution process.
We use the matchstick arithmetic problems introduced by Knoblich et al. (1999). Matchstick arithmetic problems are suitable for investigation with eye tracking, as was powerfully demonstrated by the seminal study of Knoblich et al. (2001). A matchstick arithmetic problem consists of a false arithmetic statement written using Roman numerals, arithmetic operators, and equal signs, all formed using matchsticks (Knoblich et al. 1999, 2001; see also Figure 1 below). The task is to transform the false arithmetic statement into a true statement by moving only a single stick. Four types of matchstick arithmetic problems have been defined with varying levels of difficulty, depending on the constraints that need to be relaxed and the tightness of the chunks that need to be decomposed. These problem types were theoretically derived from the representational change theory (Ohlsson 1992) and have been empirically confirmed (Knoblich et al. 1999; Öllinger et al. 2006, 2008). The use of matchstick arithmetic problems enables us to build on a well-researched task domain. It is known which problem type should elicit the restructuring process (Knoblich et al. 1999; Öllinger et al. 2006, 2008), and it is possible to contrast it with a type which requires no restructuring. Additionally, based on Knoblich’s study (2001), predictions about eye movement patterns can be made. Furthermore, the matchstick arithmetic domain is well suited for eye tracking because each problem consists of individual matchsticks that do not overlap, allowing for precise differentiation of fixations. In other words, we can determine at any point in time which aspect of the problem is attended to.
Knoblich et al. (2001) investigated constraint relaxation type problems, which are considered to require restructuring; see Figure 1 for an example. They found that for this problem (constraint relaxation type), both solvers and non-solvers examined the values in the beginning and spent most of their time doing so. This can be seen as an indication that participants were using an initial incorrect problem representation, triggered by previous knowledge, where only values can be changed. Only in the final third of the problem- solving period did later solvers change their mental representation, as demonstrated by their eye movements. Solvers started to pay attention more to the operators and less to the values. In contrast, non-solvers remained stuck in their initial representation, as they continued to attend to values rather than to operators. Similar results for the same problem were found by another eye-tracking study (Tseng et al. 2014).
The Knoblich et al. (2001) study provides strong evidence for the claim that in problems that require constraint relaxation, a restructuring of the problem representation took place. However, it did not answer the question of whether this change was a sudden or a gradual one. In the final third of the allotted time, solvers paid attention to the important but previously ignored features, which could be interpreted as a result of sudden restructuring. It is nevertheless not that clear, since the final period may have lasted minutes, given that they took around five minutes to solve the problem. Thus, the restructuring might have been a continuous process over time. On the other hand, an eye-tracking study on anagrams by Ellis et al. (2011; see also Ellis and Reingold 2014) found that participants started disregarding the irrelevant problem elements several seconds before they came up with the solution. The viewing times on that problem elements were decreasing gradually. Most intriguingly, both participant groups, those who experienced pop-out insight-like solutions and those who did not, displayed the same gradual accumulation of solution knowledge.

1.3. Metacognitive Processes and Insight Problems

There is evidence that the problem-solving process benefits from hints (e.g., Bowden 1997; Bilalić et al. 2019; Ammalainen and Moroshkina 2021; Becker et al. 2021; Korovkin and Savinova 2021; Spiridonov et al. 2021). This is the case even when hints were unreportable; that is, hints even work when presented briefly below the threshold of consciousness. Ammalainen and Moroshkina (2021) found evidence that hints can influence the problem-solving ability, which can be both, positive and negative. In a positive way, hints which are helpful to find the solution increase solution rates. On the other hand, misleading hints can negatively affect solution rates by distracting problem solvers and leading to a decrease in their success rate. In our paper (Bilalić et al. 2019), we also provided hints when participants were unable to find the correct solution after a certain time.
These hints serve two purposes: a practical and a theoretical one. On a practical level, they provide an additional check on the main assumption behind the restructuring process. On a theoretical level, they serve as explicit clues that tap into metacognitive processes (Takeuchi et al. 2019; Metcalfe and Shimamura 1994). Hints make participants aware of important aspects in the problem, drawing their attention towards elements that may have been neglected. They also change participants’ knowledge about the problem, potentially affecting the way they solve insight problems (Bowden 1997; Bilalić et al. 2019; Korovkin and Savinova 2021).

2. Methods

The present work is a re-analysis of our paper (Bilalić et al. 2019). In our paper, we also combined solving of insight and non-insight problems with eye tracking. We presented first a non-insight matchstick problem and then the matchstick insight problem depicted here (see Figure 1) to 61 participants (5 male; Mage = 22.8; SDage = 6.5). The study was designed to take into account the methodological issue discussed in the previous section. It built upon previous attempts that utilized more time periods and sometimes presented the last 5 or 10 s separately (see also Bilalić et al. 2008, 2010, 2014). In the 2019 study, we provided a more fine-grained temporal analysis of the solution process by using ten time periods of equal length for our eye movement analysis1 (for more information, please refer to Bilalić et al. 2019). We demonstrated that the restructuring is a gradual process on the insight problem as the solvers started paying attention to the important aspects of the problem long before they found the solution. Here, we provide another set of data where the jump is sudden; that is, the solvers started paying attention to the important aspects immediately before they found the solution (as reported by Knoblich et al. 2001). This is done to illustrate (1) how classical ways of analyzing data, such as ANOVA, are inappropriate for discovering the sudden changes, and (2) how other non-linear approaches are required.
We expected that all participants would initially focus on the values. Solvers would shift their attention towards the critical element (the “+” operator), while non-solvers would remain fixated on the values. The first question of interest is whether the representational shift in eventual solvers will be sudden or rather incremental. The second question of interest is whether the explicit cue, that is, the hint, will produce a sudden rearrangement of attention towards the critical elements (here “+”, but also “=” because “=” is also an operator). In our design, we included hints for participants who had not solved the problem within five minutes. The hint provided at this point was ‘You can change the operators, too.’ We were interested in whether the hints change the dynamics of problem solving, specifically whether the solution process remains sudden even after receiving an explicit cue.

3. Results

The problem proved difficult as only 34% found the solution. After the hint was provided, an additional 11% of participants were able to find the solution. We present the eye data analysis below, with a particular focus on the critical element of the problem, the plus sign (+). Additionally, when analyzing the impact of hints, we also focused on the equal sign (=) as the hints should also affect the attention drawn to this operator through metacognitive control. For analysis of other problem elements, please refer to the supplementary materials.

3.1. Is Insight Sudden or Incremental? (Solvers vs. Non-Solvers: First 5 Min Analysis)

In Figure 2, raw data and means for each bin of the critical element for the first five minutes are presented.2 The solving pattern follows the typical sudden pattern, where there is not much difference between eventual solvers and non-solvers with regard to the time spent on the critical element (+) until the end of the first five minutes. Solvers suddenly increase their dwell time just before announcing the solution, while non-solvers continue to observe the critical element sporadically until the end of the solving period.
The crucial question is how to analyze the temporal changes presented in Figure 2. The traditional method, which we had chosen in our previous paper (Bilalić et al. 2019), is to use an analysis of variance (ANOVA) where the bins and groups are factors that predict the amount of time spent on the critical element. However, ANOVA not only requires a completely balanced dataset, but it also ignores the clustered nature of data (van Rij et al. 2020). Furthermore, it is based on linear regression, which is not suitable for capturing sudden attentional shifts, which are nonlinear in nature. In order to capture the sudden shift as depicted in Figure 2 (the 100% bin for the solvers), ANOVA would need to adjust the linear trend throughout the whole problem-solving period. In other words, a sudden trend may appear as an incremental one as ANOVA adjusts by increasing previous periods (see Figure 3, left panel).
ANOVA can be expressed as linear regression, where an additional quadratic polynomial term is included next to the linear one, in an attempt to capture the shift. However, even in this case, the predicted shift by the ANOVA model would begin earlier, namely at the 80% bin, than it does in the raw data (see Figure 3, right panel). The general limitation of linear regression, with or without polynomial terms, is that it heavily relies on previous trends. If the change is sudden, the previous time periods will also be adjusted accordingly.
One way around this problem is generalized additive (mixed) modeling (GAM). These models are specifically designed to handle nonlinear relationships, as they are data-driven and use non-linear mixed-effects regression (van Rij et al. 2020). A key benefit of GAMs is that they do not require the user to specify the shape of the nonlinear regression line, as the model determines this based on the data. However, while GAMs have a high level of flexibility in modeling nonlinear changes in time series data, they only allow for the exploration of changes in the function and do not provide parametric estimates such as standard error of estimate or its impact on predictive accuracy of the model. More specifically, GAMs do not provide parametric estimates, which means that they do not give us a set of parameters that describe the shape of the nonlinear function. However, the present work intends to demonstrate the advantages and downsides of the available analysis tools in question, which is why GAMs are included here.
Arguably the most reliable way of checking the assumption of suddenness is the use of change point analysis, which looks for significant deviance from previous trends (Raftery and Akman 1986). Unlike the standard regression analysis (ANOVA) and nonlinear GAMs, change point regression estimates the moment of the function inflection. In other words, it includes the possibility to estimate additional parameters, such as intercept and slope of regression, time point when the function changes, and how the intercept and/or slope of regression changes (see the figures of the MCP analysis for illustrations). This makes the technique particularly valuable in detecting increasing patterns as one would expect several points of change in the attentional pattern on the way towards the solution. In this instance, we use the one implemented in the Multiple Change Points package (MCP; Lindeløv 2020).
Below, we address the three main questions using both GAM and MCP analysis. In the supplemental material, we provide the model-estimated values for each case, which include the results and, in the case of the MCPs, how well the model fits the data and which model was used. We begin with the GAM analysis of solvers and non-solvers for the first five minutes to determine whether the insight is sudden or incremental. Figure 4 provides the estimated trend lines for both solvers and non-solvers, as well as the time periods (shaded in orange) where the difference between the two is statistically significant. The model estimates closely follow the raw data (see Figure 2), and the difference between solvers and non-solvers is indeed significant at the beginning of the solving phase, as well as at the 90% bin and the 100% bin.
Figure 5 shows the results of the MCP analysis for the same data as the GAM above. Similarly to the GAM, the MCP analysis identified a change point around the 90% bin for the solvers, which captures an attentional shift they made. While some non-solvers also shifted their attention towards the “+” sign at the end, it was not as clear as in the case of the solvers.

3.2. Do Explicit Cues Rearrange Attentional Distribution? (An Immediate Change after the Hint)

Figure 6 illustrates the impact of providing an explicit hint to the non-solvers from the first five minutes (presented here as a single group; solvers from the first five minutes are not included in this graph). The attentional shift from values towards operators, “+” and “=”, is substantial immediately after the hint. The operator “=” is attended to twice as much immediately after the hint than before. The change for “+” is slightly less dramatic at first (only 4%), but by the 20% bin, the dwell time has doubled in comparison to before the hint was provided. Note that only non-solvers are shown here, since solvers did not receive any hints.
The GAM analysis effectively captures the attentional shift, as depicted in Figure 7. However, it predicts that the change occurs prior to the hint being provided, starting already at the 90% bin, which is not a correct reflection of the actual data. While GAM is considerably more flexible than regressions with polynomial terms, the same problem of interdependence of neighboring phases remains. The shift caused by the explicit cue is so drastic that the GAM needs to adjust the increase to begin earlier in order to account for it.
In contrast, the switch points of the MCP analysis correctly capture where the change in attention allocation happens (see Figure 8).

3.3. Does Metacognition Influence Insight Problem Solving? (Solvers vs. Non-Solvers after the Hint)

The final question we aimed to address was whether the explicit cue, and the additional knowledge about the problem associated with it, would alter the way the problem was solved. Figure 9 indicates that both solvers and non-solvers maintain the level of attention on the critical aspects throughout the problem-solving period, which is a direct consequence of the explicit cue. However, this was not sufficient for finding the solution. The eventual solvers initially shifted their attention to “=” around the 30% bin, but starting from the 50% bin, they increasingly focused on “+”. This means that at this point in time, the solvers may have realized that the “+” symbol was the critical element they needed to solve the problem. Consequently, they gathered more information about the symbol by attending to it more closely.
This incremental pattern of solving is well captured by GAMs, as Figure 10 illustrates. While the non-solvers attended to the critical “+” operator consistently over the entire problem-solving period, but on a rather low level of 25% of their time, solvers gradually increased their attention towards it. Significant differences were found in the middle and the end of the problem-solving period. This was also the case for the other operator (=). Non-solvers attended to “=” in a consistent manner throughout the problem-solving period, while the solvers attended to “=” more in the middle of the problem-solving period and less at the very end of it, probably because they were then already focusing more on the “+” sign which needs to be changed for a solution.
Figure 11 illustrates that the attentional shifts after receiving a hint are effectively captured by the MCP analysis. Again, non-solvers attended to the “+” operator on a consistently low level throughout the entire problem-solving process, while solvers attended to the “+” operator more and more. The same trend is observed for the “=” operator. Non-solvers attended to it less, while solvers shifted their attention to it in the middle of the problem-solving process. Towards the end of the problem-solving process, the data suggest that solvers became aware that the “=” operator was not as important for solving the problem and began to focus more on the “+” operator.

4. Discussion

We have demonstrated that recording eye movements is a valuable method for gaining insight into complex cognitive processes, including mental restructuring in insight problems. It is also an adequate tool for investigating attentional shifts after receiving hints. However, it is important to use eye movement recording with appropriate analytical approaches. Our results show that it is necessary to conduct a more fine-grained analysis of the eye movement data to capture the temporal dynamics of the problem-solving process. This is particularly relevant for insight problems such as the one used here, which are believed to feature a sudden change in eye movement patterns reflecting a change in mental representation.
We were able to identify the point at which solvers and non-solvers start to differ in their attentional patterns by dividing the problem-solving period into ten equal bins. The temporal resolution of the problem-solving period is one aspect, but it is also important to choose an appropriate statistical procedure. We have demonstrated that nonlinear statistical models, such as GAM and MCP, can effectively capture the sudden change that is a hallmark of insight problem solving. The GAM analysis can effectively capture the attentional shift; however, it predicts that the change occurs prior to the correct reflection of the actual data. While GAM is considerably more flexible than regressions with polynomial terms, the same problem of interdependence of neighboring phases remains. The shift caused by the explicit cue is so drastic that the GAM needs to adjust the increase to begin earlier to account for it. In contrast, the change points of the MCP analysis correctly capture where the change in attention allocation happens. A change point is a time point where the statistical properties of a time series change abruptly. However, in contrast to GAMs, one needs a priori knowledge about the number of change points and the form of the segments in between (Lindeløv 2020). Therefore, one might decide from case to case which statistical procedure is appropriate.
Our example illustrates the importance of considering theoretical assumptions when choosing analytical and statistical procedures. The restructuring of mental representations is a key concept in theories of insight (Knoblich et al. 1999; Ohlsson 1984, 1992, 2011). It is a nonlinear process in essence, which can be operationalized as a sudden burst of attention to the relevant aspects of a problem (Bilalić et al. 2019). The shift inevitably deviates significantly from participants’ previous problem solving. Seen as a part of the overall problem-solving continuum, the sudden shift is difficult to capture with linear statistical procedures. Only truly nonlinear statistical procedures can appropriately capture the sudden nature of representational change.
Providing explicit hints typically alters the dynamics of problem solving. It is obvious that the given hints were effective, as participants’ patterns of attention show a drastic change, which is very well captured by both GAM and MCP. However, it is important to note that the eventual solvers, after receiving the hint, exhibit a gradual, incremental shift, with increasing attention to the main elements during the problem-solving period. In contrast, non-solvers display an immediate burst of refocusing following the hint, but subsequently, their attention to the important aspects diminishes.
Both the analytical procedure for capturing the temporal resolution and the nonlinear statistical procedures can be easily extended beyond eye movements to other tracing methods. For example, “importance-to-solution” ratings of individual problem elements that are made repeatedly during the solving process (Durso et al. 1994; Cushen and Wiley 2012; Danek et al. 2020; Danek and Wiley 2020) often reveal patterns of sudden change which could be effectively captured by GAMs and MCPs. Similarly, “Feelings-of-Warmth” that are used to assess metacognitive knowledge about solution progress (Kizilirmak et al. 2018; Hedne et al. 2016; Pétervári and Danek 2020) are another suitable candidate for nonlinear modeling with GAMs. Other tracing methods, such as mouse-tracing data (Loesche et al. 2018; van Rij et al. 2020), think-aloud protocols (Gilhooly et al. 2010; Schooler et al. 1993; Blech et al. 2020), or even self-reports (Fedor et al. 2015), are also better modeled with GAMs than with commonly applied linear methods, even if they are more appropriate than the classical ANOVA.

5. Conclusions

Our results indicate that for insight problems, the restructuring process leaves a discernible trace of suddenness. Eye movements suggest that just prior to solving the problems, participants shift their focus from elements that constituted the initial problem representation to those crucial for the solution. Our results also demonstrate that receiving hints leads to attentional shifts towards critical aspects, which in turn facilitates the generation of a correct solution. However, in order to accurately capture the sudden shift in attention, a combination of the appropriate methodological approach and statistical procedure is necessary. These nonlinear processes are best captured by nonlinear statistical procedures, such as GAMs and MCPs.

Supplementary Materials

The following supporting information can be downloaded at: https://osf.io/pwuhs/?view_only=7c52bda4e6fa481e826e5d7570b6ef34.

Author Contributions

Conceptualization, M.B. and M.G.; methodology, M.B. and M.G.; software, M.G.; validation, M.B., N.V., A.H.D., and M.G.; formal analysis, M.G. and N.V.; investigation, M.G.; data curation, M.G.; writing—original draft preparation, M.G.; writing—review and editing, M.B., A.H.D., N.V., and M.G.; visualization, M.G., N.V., and M.B.; project administration, M.G. and M.B.; funding acquisition, M.G. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by Talent Austria der OeAD-GmbH, finanziert aus Mitteln des österreichischen Bundesministeriums für Wissenschaft, Forschung und Wirtschaft (BMWFW), grant number ICM-2017-07423 given to the first author.

Institutional Review Board Statement

Since this is a re-analysis of our paper (Bilalić et al. 2019), please refer to the original paper for the Institutional Review Board Statement.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Technical details, such as data and code for the analysis, is available at https://osf.io/pwuhs/?view_only=7c52bda4e6fa481e826e5d7570b6ef34 (accessed on 25 April 2023).

Acknowledgments

The help and cooperation from participants is greatly appreciated, as is Matthew Bladen’s contribution in preparing the text.

Conflicts of Interest

The authors declare no conflict of interest.

Notes

1
The length of time taken to solve (or not solve) a problem is different from person to person, meaning that one cannot compare the eye tracking data directly between people. For example, some may need only 45 s to solve the problem, whereas others need four minutes to find a solution. In consequence, the data must be transformed in order to be able to compare the data between people properly. While the problem-solving period can be extended by adding more time phases, it is important to note that the duration should not be prolonged beyond a certain point. Utilizing too many time frames may leave too little data (e.g., a 10-second trial should not be divided into 100 bins, as each bin will have the duration of only 100 ms). This can lead to distorted eye movement patterns, masking the underlying effects present before the data were binned. On the other hand, choosing too few bins may not capture the full temporal dynamics of the problem-solving process. In either case, ANOVA is not suitable for analyzing a large number of problem-solving periods, unlike GAM and multiple change point analysis, which can easily accommodate a large number of time frames. MCP analysis is another adequate tool for this type of analysis as it can capture the shift of attention. However, in contrast to GAMs, one needs a priori knowledge about the number of change points and the form of the segments in between (Lindeløv 2020).
2
Please note that the data presented here are simulated to represent a sudden shift, which is difficult to capture by classical analyses. The original data in Bilalić et al. (2019) indicate a gradual shift.

References

  1. Ammalainen, Artur, and Nadezhda Moroshkina. 2021. The effect of true and false unreportable hints on anagram problem solving, restructuring, and the Aha!-experience. Journal of Cognitive Psychology 33: 644–58. [Google Scholar] [CrossRef]
  2. Ash, Ivan K., and Jennifer Wiley. 2006. The nature of restructuring in insight: An individual-differences approach. Psychonomic Bulletin & Review 13: 66–73. [Google Scholar]
  3. Ash, Ivan K., and Jennifer Wiley. 2008. Hindsight bias in insight and mathematical problem solving: Evidence of different reconstruction mechanisms for metacognitive versus situational judgments. Memory & Cognition 36: 822–37. [Google Scholar] [CrossRef]
  4. Ash, Ivan K., Benjamin Jee, and Jennifer Wiley. 2012. Investigating insight as sudden learning. The Journal of Problem Solving 4: 150–76. [Google Scholar] [CrossRef]
  5. Becker, Maxi, Simone Kühn, and Tobias Sommer. 2021. Verbal insight revisited—Dissociable neurocognitive processes underlying solutions accompanied by an AHA! experience with and without prior restructuring. Journal of Cognitive Psychology 33: 659–84. [Google Scholar] [CrossRef]
  6. Bilalić, Merim, and Peter McLeod. 2014. Why good thoughts block better ones. Scientific American 310: 74–79. [Google Scholar] [CrossRef]
  7. Bilalić, Merim, Peter McLeod, and Fernand Gobet. 2008. Why good thoughts block better ones: The mechanism of the pernicious Einstellung (set) effect. Cognition 108: 652–61. [Google Scholar] [CrossRef]
  8. Bilalić, Merim, Peter McLeod, and Fernand Gobet. 2010. The Mechanism of the Einstellung (Set) Effect: A Pervasive Source of Cognitive Bias. Current Directions in Psychological Science 19: 111–15. [Google Scholar] [CrossRef]
  9. Bilalić, Merim, Mario Graf, Nemanja Vaci, and Amory H. Danek. 2019. The temporal dynamics of insight problem solving–restructuring might not always be sudden. Thinking & Reasoning 27: 1–37. [Google Scholar]
  10. Blech, Christine, Robert Gaschler, and Merim Bilalić. 2020. Why do people fail to see simple solutions? Using think-aloud protocols to uncover the mechanism behind the Einstellung (mental set) effect. Thinking & Reasoning 26: 552–80. [Google Scholar]
  11. Bowden, Edward M. 1997. The effect of reportable and unreportable hints on anagram solution and the Aha! Experience. Consciousness and Cognition 6: 545–73. [Google Scholar] [CrossRef] [PubMed]
  12. Cushen, Patrick J., and Jennifer Wiley. 2012. Cues to solution, restructuring patterns, and reports of insight in creative problem solving. Consciousness and Cognition 21: 1166–75. [Google Scholar] [CrossRef] [PubMed]
  13. Danek, Amory H., and Joshua Wiley. 2020. What causes the insight memory advantage? Cognition 205: 104411. [Google Scholar] [CrossRef]
  14. Danek, Amory H., Joshua Williams, and Jennifer Wiley. 2020. Closing the gap: Connecting sudden representational change to the subjective Aha! experience in insightful problem solving. Psychological Research 84: 111–19. [Google Scholar] [CrossRef] [PubMed]
  15. Duncker, Karl. 1945. On problem-solving. Psychological Monographs 58: 270. [Google Scholar] [CrossRef]
  16. Durso, Francis T., Cornelia B. Rea, and Tom Dayton. 1994. Graph-theoretic confirmation of restructuring during insight. Psychological Science 5: 94–97. [Google Scholar] [CrossRef]
  17. Ellis, Jessica J., and Eyal M. Reingold. 2014. The Einstellung effect in anagram problem solving: Evidence from eye movements. Frontiers in Psychology 5: 679. [Google Scholar] [CrossRef]
  18. Ellis, Jessica J., Mackenzie G. Glaholt, and Eyal M. Reingold. 2011. Eye movements reveal solution knowledge prior to insight. Consciousness and Cognition 20: 768–76. [Google Scholar] [CrossRef] [PubMed]
  19. Fedor, Anna, Eörs Szathmáry, and Michael Öllinger. 2015. Problem solving stages in the five square problem. Frontiers in Psychology 6: 1050. [Google Scholar] [CrossRef]
  20. Fleck, Jessica I., and Robert W. Weisberg. 2013. Insight versus analysis: Evidence for diverse methods in problem solving. Journal of Cognitive Psychology 25: 436–63. [Google Scholar] [CrossRef]
  21. Gilhooly, Ken J., Evie Fioratou, and N. Henretty. 2010. Verbalization and problem solving: Insight and spatial factors. British Journal of Psychology 101: 81–93. [Google Scholar] [CrossRef] [PubMed]
  22. Hedne, Mikael R., Elisabeth Norman, and Janet Metcalfe. 2016. Intuitive feelings of warmth and confidence in insight and noninsight problem solving of magic tricks. Frontiers in Psychology 7: 1314. [Google Scholar] [CrossRef]
  23. Holmqvist, Kenneth, Marcus Nyström, Richard Andersson, Richard Dewhurst, Jarodzka Halszka, and Joost Van de Weijer. 2011. Eye Tracking: A Comprehensive Guide to Methods and Measures. Oxford: OUP Oxford. [Google Scholar]
  24. Just, Marcel A., and Patricia A. Carpenter. 1976. Eye fixations and cognitive processes. Cognitive Psychology 8: 441–80. [Google Scholar] [CrossRef]
  25. Kizilirmak, Jasmin M., Violetta Serger, Judith Kehl, Michael Öllinger, Kristian Folta-Schoofs, and Alan Richardson-Klavehn. 2018. Feelings-of-warmth increase more abruptly for verbal riddles solved with in contrast to without Aha! Experience. Frontiers in Psychology 9: 1404. [Google Scholar] [CrossRef] [PubMed]
  26. Knoblich, Günther, Stellan Ohlsson, Hilde Haider, and Detlef Rhenius. 1999. Constraint relaxation and chunk decomposition in insight problem solving. Journal of Experimental Psychology: Learning, Memory, and Cognition 25: 1534–55. [Google Scholar] [CrossRef]
  27. Knoblich, Günther, Stellan Ohlsson, and Gary E. Raney. 2001. An eye movement study of insight problem solving. Memory & Cognition 29: 1000–9. [Google Scholar] [CrossRef]
  28. Korovkin, Sergey, and Anna Savinova. 2021. The Effectiveness of Metacognitive Hints in Insight Problem Solving. In Advances in Cognitive Research, Artificial Intelligence and Neuroinformatics. Edited by Boris M. Velichkovsky, Pavel M. Balaban and Vadim L. Ushakov. Intercognsci 2020. Advances in Intelligent Systems and Computing. Cham: Springer, vol. 1358. [Google Scholar] [CrossRef]
  29. Kuhn, Gustav, and Michael F. Land. 2006. There’s more to magic than meets the eye. Current Biology 16: R950–R51. [Google Scholar] [CrossRef]
  30. Kuhn, Gustav, Benjamin W. Tatler, and Geoff. G. Cole. 2009. You look where I look! Effect of gaze cues on overt and covert attention in misdirection. Visual Cognition 17: 925–44. [Google Scholar] [CrossRef]
  31. Lindeløv, Jonas K. 2020. mcp: An R Package for Regression with Multiple Change Points. OSF Preprints. [Google Scholar] [CrossRef]
  32. Loesche, Frank, Jeremy Goslin, and Guido Bugmann. 2018. Paving the way to eureka—Introducing “dira” as an experimental paradigm to observe the process of creative problem solving. Frontiers in Psychology 9: 1773. [Google Scholar] [CrossRef]
  33. MacGregor, James N., and J. Barton Cunningham. 2009. The effects of number and level of restructuring in insight problem solving. The Journal of Problem Solving 2: 7. [Google Scholar] [CrossRef]
  34. Metcalfe, Janet, and Arthur P. Shimamura, eds. 1994. Metacognition: Knowing about Knowing. Cambridge: MIT Press. [Google Scholar]
  35. Newell, Allen, and Herbert A. Simon. 1972. Human Problem Solving. Englewood Cliffs: Prentice Hall. [Google Scholar]
  36. Novick, Laura R., and Steven J. Sherman. 2003. On the nature of insight solutions: Evidence from skill differences in anagram solution. The Quarterly Journal of Experimental Psychology Section A 56: 351–82. [Google Scholar] [CrossRef] [PubMed]
  37. Ohlsson, Stellan. 1984. Restructuring revisited: II. An information processing theory of restructuring and insight. Scandinavian Journal of Psychology 25: 117–29. [Google Scholar] [CrossRef]
  38. Ohlsson, Stellan. 1992. Information-processing explanations of insight and related phenomena. In Advances in the Psychology of Thinking. Edited by M. Keane and K. J. Gilhooly. London: Harvester-Wheatsheaf, pp. 1–44. [Google Scholar]
  39. Ohlsson, Stellan. 2011. Deep Learning: How the Mind Overrides Experience. New York: Cambridge University Press. [Google Scholar]
  40. Öllinger, Michael, Gary Jones, and Günther Knoblich. 2006. Heuristics and representational change in two-move matchstick arithmetic tasks. Advances in Cognitive Psychology 2: 239–53. [Google Scholar] [CrossRef]
  41. Öllinger, Michael, Gary Jones, and Günther Knoblich. 2008. Investigating the effect of mental set on insight problem solving. Experimental Psychology 55: 270–82. [Google Scholar] [CrossRef]
  42. Pétervári, Judit, and Amory H. Danek. 2020. Problem solving of magic tricks: Guiding to and through an impasse with solution cues. Thinking & Reasoning 26: 502–33. [Google Scholar] [CrossRef]
  43. Raftery, Adrian E., and V. E. Akman. 1986. Bayesian analysis of a Poisson process with a change-point. Biometrika 73: 85–89. [Google Scholar] [CrossRef]
  44. Rayner, Keith. 1995. Eye movements and cognitive processes in reading, visual search, and scene perception. In Studies in Visual Information Processing. Amsterdam: Elsevier, vol. 6, pp. 3–22. [Google Scholar]
  45. Reingold, Eyal M., Neil Charness, Richard S. Schultetus, and Dave M. Stampe. 2001. Perceptual automaticity in expert chess players: Parallel encoding of chess relations. Psychonomic Bulletin & Review 8: 504–10. [Google Scholar]
  46. Schooler, Jonathan W., Stellan Ohlsson, and Kevin Brooks. 1993. Thoughts beyond words: When language overshadows insight. Journal of Experimental Psychology: General 122: 166. [Google Scholar] [CrossRef]
  47. Spiridonov, Vladimir, Nikita Loginov, and Vladlen Ardislamov. 2021. Dissociation between the subjective experience of insight and performance in the CRA paradigm. Journal of Cognitive Psychology 33: 685–99. [Google Scholar] [CrossRef]
  48. Takeuchi, Naoyuki, Takayuki Mori, Yoshimi Suzukamo, and Shin I. Izumi. 2019. Activity of prefrontal cortex in teachers and students during teaching of an insight problem. Mind, Brain, and Education 13: 167–75. [Google Scholar] [CrossRef]
  49. Tseng, Chien-Chih, Ching-Hui Chen, Hsueh-Chih Chen, Yao-Ting Sung, and Kuo-En Chang. 2014. Verification of Dual Factors theory with eye movements during a matchstick arithmetic insight problem. Thinking Skills and Creativity 13: 129–40. [Google Scholar] [CrossRef]
  50. van Rij, Jacolien, Nemanja Vaci, Lee Wurm, and Laurie Feldman. 2020. Alternative quantitative methods in psycholingistics: Implications for theory and design. In Word Knowledge and Word Usage: A Cross-Disciplinary Guide to the Mental Lexicon. Edited by Vito Pirrelli, Ingo Plag and Wolfgang Dressler. Berlin and Boston: De Gruyter Mouton, pp. 83–126. [Google Scholar] [CrossRef]
  51. Wertheimer, Max. 1925. Über Schlussprozesse im produktiven Denken. In Drei Abhandlungen zur Gestalttheorie. Edited by Max Wertheimer. Erlangen: Verlag der Philosophischen Akademie, pp. 164–84. [Google Scholar]
Jintelligence 11 00086 g001 550
Figure 1. Matchstick arithmetic problem. Participants are required to transform the false arithmetic statement to a true statement by moving a single matchstick. This problem requires restructuring, because the initial assumption that only the matchsticks from values can be manipulated needs to be changed. In this case, the operator “+” can be decomposed and its vertical matchstick moved to make another “=” sign (VI = VI = VI). The “+” sign is the critical element that needs to be changed for solution.
Figure 1. Matchstick arithmetic problem. Participants are required to transform the false arithmetic statement to a true statement by moving a single matchstick. This problem requires restructuring, because the initial assumption that only the matchsticks from values can be manipulated needs to be changed. In this case, the operator “+” can be decomposed and its vertical matchstick moved to make another “=” sign (VI = VI = VI). The “+” sign is the critical element that needs to be changed for solution.
Jintelligence 11 00086 g001
Jintelligence 11 00086 g002 550
Figure 2. Raw data and means for each bin of the critical element (+). The raw data represents every data point of each participant over the entire problem-solving period. The problem-solving period was divided in 10 proportional bins, each representing 10% of the total problem-solving time. The error bars represent the 68% confidence interval. This figure illustrates a nonlinear increase in the amount of time that solvers spend on the critical element. In the case of solvers, the 100% bin means the participant provided a solution.
Figure 2. Raw data and means for each bin of the critical element (+). The raw data represents every data point of each participant over the entire problem-solving period. The problem-solving period was divided in 10 proportional bins, each representing 10% of the total problem-solving time. The error bars represent the 68% confidence interval. This figure illustrates a nonlinear increase in the amount of time that solvers spend on the critical element. In the case of solvers, the 100% bin means the participant provided a solution.
Jintelligence 11 00086 g002
Jintelligence 11 00086 g003 550
Figure 3. Estimated model means based on (a) ANOVA with linear term; (b) ANOVA with both linear and quadratic terms. Y-Axis: Time on the problem element (%). Please refer to supplementary material for the detailed analysis.
Figure 3. Estimated model means based on (a) ANOVA with linear term; (b) ANOVA with both linear and quadratic terms. Y-Axis: Time on the problem element (%). Please refer to supplementary material for the detailed analysis.
Jintelligence 11 00086 g003
Jintelligence 11 00086 g004 550
Figure 4. GAM: the difference between two estimated trend lines for solvers and non-solvers of the critical element (+). This figure illustrates that the GAM also found a nonlinear increase in the amount of time that non-solvers spend on the critical element. The orange area determines where the differences between solvers and non-solvers were significant.
Figure 4. GAM: the difference between two estimated trend lines for solvers and non-solvers of the critical element (+). This figure illustrates that the GAM also found a nonlinear increase in the amount of time that non-solvers spend on the critical element. The orange area determines where the differences between solvers and non-solvers were significant.
Jintelligence 11 00086 g004
Jintelligence 11 00086 g005 550
Figure 5. MCP analysis of the critical element (+) for non-solvers and solvers. This figure illustrates every data point of each participant over the problem-solving period. Lines at the bottom of the figure illustrate the posterior density (estimated likelihood) of the change point for each MCMC chain. There is a nonlinear increase in the amount of time that solvers spend on the critical element.
Figure 5. MCP analysis of the critical element (+) for non-solvers and solvers. This figure illustrates every data point of each participant over the problem-solving period. Lines at the bottom of the figure illustrate the posterior density (estimated likelihood) of the change point for each MCMC chain. There is a nonlinear increase in the amount of time that solvers spend on the critical element.
Jintelligence 11 00086 g005
Jintelligence 11 00086 g006a 550Jintelligence 11 00086 g006b 550
Figure 6. Raw data and means for each bin of the operators (“+” upper panel; “=” lower panel) in the period before and after the hint. The raw data represents every data point of each participant (non-solvers only) over the problem-solving periods. Each of both problem-solving periods (before and after the hint was provided) were divided in 10 proportional bins, each representing 10% of the total problem-solving time. It is necessary to view the problem-solving periods as distinct periods; therefore, each period is labeled from beginning to end (10% to 100%) to differentiate them. The error bars represent the 68% confidence interval. This figure illustrates the attentional shifts from values (mostly attended to before the hint) towards operators (attended to after the hint was given).
Figure 6. Raw data and means for each bin of the operators (“+” upper panel; “=” lower panel) in the period before and after the hint. The raw data represents every data point of each participant (non-solvers only) over the problem-solving periods. Each of both problem-solving periods (before and after the hint was provided) were divided in 10 proportional bins, each representing 10% of the total problem-solving time. It is necessary to view the problem-solving periods as distinct periods; therefore, each period is labeled from beginning to end (10% to 100%) to differentiate them. The error bars represent the 68% confidence interval. This figure illustrates the attentional shifts from values (mostly attended to before the hint) towards operators (attended to after the hint was given).
Jintelligence 11 00086 g006aJintelligence 11 00086 g006b
Jintelligence 11 00086 g007a 550Jintelligence 11 00086 g007b 550
Figure 7. GAM: estimated trend line for non-solvers of the critical element (+; upper panel) and the other operator (=; lower panel). This figure illustrates that the GAM also found a nonlinear increase in the amount of time that non-solvers spend on the critical element after receiving a hint. The orange area indicates where there is a significant shift in attention.
Figure 7. GAM: estimated trend line for non-solvers of the critical element (+; upper panel) and the other operator (=; lower panel). This figure illustrates that the GAM also found a nonlinear increase in the amount of time that non-solvers spend on the critical element after receiving a hint. The orange area indicates where there is a significant shift in attention.
Jintelligence 11 00086 g007aJintelligence 11 00086 g007b
Jintelligence 11 00086 g008a 550Jintelligence 11 00086 g008b 550
Figure 8. MCP analysis of the critical element (+; upper panel) and the other operator (=; lower panel). This figure illustrates that the switch points of the MCP analysis correctly captures where the shift in attention happens.
Figure 8. MCP analysis of the critical element (+; upper panel) and the other operator (=; lower panel). This figure illustrates that the switch points of the MCP analysis correctly captures where the shift in attention happens.
Jintelligence 11 00086 g008aJintelligence 11 00086 g008b
Jintelligence 11 00086 g009 550
Figure 9. Raw data and means for each bin of the critical element (+; upper panel) and the other operator (=; lower panel) after the hint was provided. The raw data represent every data point of each participant over the remaining problem-solving period after the hint was given. The error bars represent the 68% confidence interval. This figure illustrates a nonlinear increase or decrease in the time solvers spend on the critical element.
Figure 9. Raw data and means for each bin of the critical element (+; upper panel) and the other operator (=; lower panel) after the hint was provided. The raw data represent every data point of each participant over the remaining problem-solving period after the hint was given. The error bars represent the 68% confidence interval. This figure illustrates a nonlinear increase or decrease in the time solvers spend on the critical element.
Jintelligence 11 00086 g009
Jintelligence 11 00086 g010 550
Figure 10. GAM: the difference between two estimated trend lines for solvers and non-solvers of the critical element (+; upper panel) and the other operator (=; lower panel). This figure illustrates that the GAM also found a nonlinear increase in the time solvers spend on that particular element. The orange area in the figure indicates regions where there are statistically significant differences between the attention patterns of solvers and non-solvers.
Figure 10. GAM: the difference between two estimated trend lines for solvers and non-solvers of the critical element (+; upper panel) and the other operator (=; lower panel). This figure illustrates that the GAM also found a nonlinear increase in the time solvers spend on that particular element. The orange area in the figure indicates regions where there are statistically significant differences between the attention patterns of solvers and non-solvers.
Jintelligence 11 00086 g010
Jintelligence 11 00086 g011 550
Figure 11. MCP analysis of the critical element after the hint (+; upper panel) and the other operator (=; lower panel) for non-solvers and solvers. This figure demonstrates that the switch points of the MCP analysis correctly captures the incremental pattern of solving for the critical element (+). It also demonstrates that after the hint, the non-solvers attended to the noncritical element (=) more in the beginning but not the critical element (+). As the GAMs showed already, the non-solvers attended to the critical element (+) in the same way throughout the whole problem-solving period.
Figure 11. MCP analysis of the critical element after the hint (+; upper panel) and the other operator (=; lower panel) for non-solvers and solvers. This figure demonstrates that the switch points of the MCP analysis correctly captures the incremental pattern of solving for the critical element (+). It also demonstrates that after the hint, the non-solvers attended to the noncritical element (=) more in the beginning but not the critical element (+). As the GAMs showed already, the non-solvers attended to the critical element (+) in the same way throughout the whole problem-solving period.
Jintelligence 11 00086 g011
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Graf, M.; Danek, A.H.; Vaci, N.; Bilalić, M. Tracing Cognitive Processes in Insight Problem Solving: Using GAMs and Change Point Analysis to Uncover Restructuring. J. Intell. 2023, 11, 86. https://doi.org/10.3390/jintelligence11050086

AMA Style

Graf M, Danek AH, Vaci N, Bilalić M. Tracing Cognitive Processes in Insight Problem Solving: Using GAMs and Change Point Analysis to Uncover Restructuring. Journal of Intelligence. 2023; 11(5):86. https://doi.org/10.3390/jintelligence11050086

Chicago/Turabian Style

Graf, Mario, Amory H. Danek, Nemanja Vaci, and Merim Bilalić. 2023. "Tracing Cognitive Processes in Insight Problem Solving: Using GAMs and Change Point Analysis to Uncover Restructuring" Journal of Intelligence 11, no. 5: 86. https://doi.org/10.3390/jintelligence11050086

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop