Judgment of learning (JOL) is a predictive judgment about the performance of a learned item in subsequent tests (Nelson and Dunlosky 1991
). As an important form of metacognitive monitoring, the accuracy of JOL plays an essential role in determining whether individuals can learn efficiently (Koriat et al. 2006
; Metcalfe and Finn 2008
; Metcalfe 2009
). Furthermore, researchers have found that metacognitive monitoring performance is more accurate when JOLs are made at an interval after learning (e.g., 30 s of studying other items), compared with immediate JOL, known as the delayed JOL effect (Dunlosky and Nelson 1997
; Koriat and Ma’ayan 2005
; Son and Metcalfe 2005
; Rhodes and Tauber 2011
; Kelley et al. 2021
). Additionally, studies also have found that delayed JOL not only improves the accuracy of metamemory monitoring, but also improves the recall performance of the judged items (Sikström and Jönsson 2005
; Serra and Dunlosky 2006
; Jönsson et al. 2012
; Sundqvist et al. 2012
; Myers et al. 2020
; Li et al. 2022
). Jönsson et al.
) suggested that this facilitation is related to the covert retrieval attempts of the target information resulting from the delayed JOL, which itself facilitates learning and is often referred to as the retrieval practice effect (McDaniel et al. 2007
; Karpicke and Roediger 2008
; Carpenter et al. 2008
; Hausman and Rhodes 2018
). In other words, when individuals make metacognitive judgments, they engage in covert retrieval attempts to predict their own test performance (Kubik et al. 2022
). Thus, compared to restudy, one of the most common learning strategies without any item retrieval, delayed JOL can more effectively promote learners’ memory for learned items.
The current research focuses on the forward effects of delayed JOL, which are receiving increasing attention as an indirect but future-oriented benefit of delayed JOL. Recent studies have found that delayed JOL not only promotes the retention of previously learned information, but also has a positive effect on the subsequent learning of new material (Lee and Ha 2019
; Kubik et al. 2022
). To examine the forward effect of JOL, methods were developed based on the paradigm of the forward test effect (FTE). In this approach, the material to be memorized is divided into several separate learning blocks (e.g., Szpunar et al. 2008
; Sohlberg et al. 2021
; Kubik et al. 2022
), and participants are divided into at least two groups: a delayed JOL group and a control group. The JOL group undergoes the metacognitive assessment after each individual study block, while the control group often restudies the content a second time. Both groups then take a test on the final study block. The results of the interim test in the final learning block for both groups demonstrates the effect of forward effect of JOL. The difference between the two groups in terms of the results achieved in the final block is likely due to the JOL group making learning judgements in the previous block, resulting in the forward effect.
There are several possible explanations for the forward effect of delayed JOL. On the one hand, the reasons may be similar to the forward test effect (for an overview, see Yang et al. 2018
). Context change theory emphasizes the role of contextual change induced by an interim task. This theory postulates that contextual changes between learning items enhance individual learning by reducing proactive interference and increasing learning engagement (Szpunar et al. 2008
; Bäuml and Kliegl 2013
; Sohlberg et al. 2021
). A recent study by Kubik et al.
) demonstrated that delaying JOL using an interim task inserted during learning multiple lists attenuated the number of intrusions from prior lists. On the other hand, various forms of metacognitive judgments have been shown to impact learners’ future learning behaviors. For example, the JOL group requires learners to consider their own learning status after a learning process. This monitoring and evaluation of their learning behavior encourages learners to adjust their learning efforts by changing their study behavior to optimize subsequent learning, such as allocating study time, selecting restudy materials, and adjusting learning strategies (Sahakyan et al. 2004
; Kornell and Metcalfe 2006
; Metcalfe and Finn 2008
; Mitchum et al. 2016
; Wang et al. 2020
). In studies on the forward effect, interim tasks inserted during the learning of multiple lists are thought to have metacognitive benefits that can prompt learners to adopt more effective coding strategies during subsequent learning (Cho et al. 2017
; Lee and Ahn 2018
). Lee and Ha
) found that even when learners were not given the freedom to choose what they study and for how long, the mere act of making metacognitive judgments directly facilitated the learners’ subsequent learning of new material.
However, not all acts of making JOLs will produce a facilitative forward effect. Kubik et al.
) used single German words as experimental material and examined the forward effects of delayed JOLs with word stems (e.g., ele_____?) versus JOLs with complete words (e.g., elephant), in comparison to restudy and retrieval practice. Participants consecutively studied five lists of 20 words with the goal of recalling as many of them as possible on a final cumulative recall test. After the presentation of each of the first four lists, the participants either restudied the list, made JOLs with complete words, made JOLs with word stems, or were tested on word stems. The results showed that practicing retrieval and making JOLs with word stems, but not making JOLs with complete words, facilitated interim recall performance of the last list compared to restudy. This suggests that creating JOLs with incomplete information can promote new learning, perhaps because it elicits covert retrieval attempts, but making JOLs with complete information cannot. Similarly, in another related study, Lee and Ha
) examined whether the act of making judgments of learning (JOLs) on prior material contributed to the learning of new material in the category learning of painters’ styles in three experiments. The results showed that the act of making category-level JOLs and global metacognitive judgments facilitated the learning of new material. However, it is worth noting that simply asking learners to make item-level JOLs did not facilitate learning compared to restudy. One explanation for the lack of forward effects in these item-level JOLs is that they required learners to evaluate their learning about specific painting–artist pairs and thus did not meet the learning goal of abstracting the artist’s general painting style from the studied examples. However, as Lee and Ha pointed out, another explanation is that the participants were unable to adequately evaluate their learning due to the simultaneous presentation of cues (paintings) and target items (artist names). Simultaneous presentation may place learners in a restudy-like state, thereby reducing the probability that learners will engage in covert retrieval practice.
Given that only a few studies have explored the future-oriented benefit of delayed JOL, the purpose of the present study was to replicate and extend this work on the forward effect of item-based JOL to word pair memory and category learning tasks, and to further explore the boundary conditions of this effect. We pursued three main research questions. First, we examined whether the same forward effect of delayed JOL could be observed when we used typically associated word pairs rather than single words as experimental material. To our knowledge, no previous study has used associated word pairs (e.g., dog–cat) to explore the forward effect of delayed JOL. If this effect is robust, we hypothesize that cue-only delayed JOL (the first word in a word pair, e.g., dog) might similarly facilitate learners’ performance on subsequent material, since this process also involves covert retrieval attempts for the missing target word. In the present experiments, we examined whether delayed JOLs affect subsequent learning by comparing them to restudy, which serves as a common and neutral control condition that does not involve any overt or covert retrieval practice, ensuring that the two conditions differed only in the behavior of performing delayed JOLs.
As a second goal, we explored whether the item-level delayed JOL could similarly improve the learning efficiency of new category material. In a real-life educational context, what is important to learn often goes beyond memorizing specific items. Students also need to be able to generalize what they have learned from specific examples to other examples in a particular category, that is, category learning (Smith et al. 2010
; Lee and Ahn 2018
). Different from item learning, inductive learning is a process where learners are required to abstract rules from a set of exemplars (Yang et al. 2019
). For example, if students have mastered Van Gogh’s painting style in art class, they will later classify a painting that is unfamiliar but fits that style as Van Gogh’s work. Likewise, if a teacher has shown some examples of swallows in a biology class, then when students see a small blue–black bird with pointed wings and a forked tail in real life, they will be able to classify it as a swallow. Although Lee and Ha
) were the first to explore the role that delayed JOL plays in category learning, it remains unclear whether item-based delayed JOL can facilitate subsequent learning of new categories, and we do not know whether the ineffectiveness of the forward effect is due to a mismatch between item-based JOLs and category learning goals, or if it is due to the lack of a retrieval practice process when making JOLs. Therefore, we made some adjustments based on Lee and Ha
) to achieve this goal. First, we used cue-only JOLs rather than cue-target JOL, thus enabling learners to make retrieval attempts. Second, we used pictures of artificial conceptual animals as learning materials, which effectively excluded the influence of learners’ prior experience compared to materials for learning drawing styles. Third, we further manipulated the difficulty level of the category learning material to determine whether there were boundary conditions for the forward effects of delayed JOL.
Material difficulty, as an important factor influencing learner performance after retrieval, is the third focus of the current study. Previous studies have found that retrieving materials of different difficulty levels leads to different memory performances. For example, Pyc and Rawson
) suggested that memory performance after successfully retrieving items with a higher difficulty is better than for items with lower difficulties. According to the strength and retrieval effort theory, the enhancement of knowledge is proportional to the effort of retrieval, and the key to the retrieval practice effect lies in the retrieval effort. Compared with simple retrieval activities, learners will invest more effort in retrieving difficult materials, which will help to produce better learning results (Bjork 1975
; Jacoby et al. 1978
; Pyc and Rawson 2009
; Kliegl et al. 2019
). Then, in the delayed JOL condition involving covert retrieval, does the difficulty of the material also have a different impact on the learning effect? More importantly, how does material difficulty affect the forward effect? Compared with the direct influence of retrieval practice on “learned” materials, the forward effect focuses on its indirect promotion and the transfer of subsequent “unlearned” materials. In this process, the learner’s effort and other motivational factors may play a more significant role (Yang et al. 2018
). Difficult items may motivate learners to make a greater effort to retrieve and code new content due to the difficulty or failure of retrieval (Cho et al. 2017
). To explore this, we introduced an independent variable of learning material difficulty by manipulating the correlation of word pairs and feature dimensions during category learning, expecting that the forward effect of delayed JOL might be more advantageous in more difficult materials.
In summary, three experiments were designed to explore the questions posed above. Experiment 1A focused on determining whether the forward effect of delayed JOL occurs during encoding of memories of associated word pairs by comparing the delayed JOL and restudy group, while Experiment 1B further refined the level of difficulty of the memorized material based on Experiment 1A to explore the role that material difficulty played in this process. Experiment 2 then examined whether it was possible to replicate Experiment 1 when the task was extended from the memory domain to the category learning task.
3. Experiment 1B
In Experiment 1A, the forward effect of delayed JOL was confirmed using word pairs of medium difficulty, which had certain correlations but were still difficult to remember. However, we wanted to further explore whether the failure to retrieve the previous list caused the learner to put more effort into learning the subsequent difficult material. Thus, based on Experiment 1A, the purpose of Experiment 1B was to investigate the effect of material difficulty on the forward effect of delayed JOL. We hypothesized that the target list performance of the delayed JOL group would be better than that of the control group for the low correlation word pairs, while there would be no difference for the high correlation word pairs.
3.1. Materials and Methods
We conducted a priori sample size calculations using G*Power3.1.9 to determine a reliable forward effect with an α = 0.05, a power of 1−β = 0.80, and an effect size of d = 0.74. These calculations indicated that a sample size of n = 30 per group (total N = 60) would be sufficient. Sixty participants (44 females; Mage = 20.44, SDage = 1.32) were recruited from the ZJNU participant pool and were randomly divided into the delayed JOL and restudy conditions (n = 30 in each group). The participants gave informed consent and received a corresponding reward after the experiment.
A total of 42 cue–target word pairs were selected (for details, see Supplementary Materials, Table S2
). Among them, 36 pairs were used for formal experiments and were randomly divided into 3 lists, each consisting of 6 high-related word pairs and 6 low-related word pairs. The remaining 6 pairs were used in the practice phase, and the results were not included in the final analysis. All the materials were assessed in advance, and there was a significant difference in difficulty between high-related word pairs (4.36 ≤ M ≤ 5.80) and low-related word pairs (1.14 ≤ M ≤ 1.78).
3.1.3. Design and Procedure
This experiment employed a 2 (Learning strategy: delayed JOL vs. restudy) × 2 (Material difficulty: high-related vs. low-related) mixed design, with learning strategy used as a between-subjects variable and material difficulty as a within-subjects variable. Experiment 1B also adopted the multi-list learning paradigm. Except for the difficulty of word pairs (6 high-related word pairs and 6 low-related word pairs were randomly presented), the process was the same as 1A.
For the recall of the target list, a 2 (Learning strategy: Delayed JOL vs. Restudy) × 2 (Material difficulty: high-related vs. low-related) two-factor ANOVA revealed no significant difference between the delayed JOL group (M
= 0.77, SD
= 0.25) and restudy group (M
= 0.71, SD
= 0.25), F
(1, 58) = 3.564, p
= 0.058. However, there was a significant main effect of material difficulty, indicating that the target list recall of high-related word pairs (M
= 0.84, SD
= 0.13) was significantly higher than that of low-related word pairs (M
= 0.63, SD
= 0.17), F
(1, 58) = 185.245, p
= 0.762. Moreover, the analysis revealed a significant interaction between learning strategy and material difficulty, F
(1, 58) = 8.479, p
= 0.128. Further simple effect analysis showed that there was no significant difference between the delayed JOL group (M
= 0.85, SD
= 0.13) and the restudy group (M
= 0.83, SD
= 0.14) for high-related word pairs, F
(1, 58) = 0.396, p
= 0.007. However, for low-related word pairs, the delayed JOL group (M
= 0.69, SD
= 0.16) performed significantly better than the restudy group (M
= 0.58, SD
= 0.17), F
(1, 58) = 6.978, p
= 0.107 (see Figure 3
A similar ANOVA on cumulative test recall revealed a significant main effect on material difficulty, F (1, 58) = 202.353, p < .001, = 0.777, indicating better memory for high-related word pairs (M = 0.83, SD = 0.13) and worse memory for low-related word pairs (M = 0.61, SD = 0.15). Moreover, significant interactions were found between learning strategy and material difficulty, F (1, 58) = 5.835, p = .019, = 0.091. Further simple effect analysis showed that there was no significant difference between the delayed JOL group (M = 0.84, SD = 0.11) and the restudy group (M = 0.82, SD = 0.14) for high-related word pairs, F (1, 58) = 0.355, p = .554, = 0.006. However, for low-related word pairs, the delayed JOL group (M = 0.65, SD = 0.15) performed significantly better than the restudy group (M = 0.56, SD = 0.15), F (1, 58) = 6.153, p = .016, = 0.096. No main effect was observed on learning strategy, F (1, 58) = 3.166, p = .08, = 0.052.
Experiment 1B explored the effect of material difficulty on the forward effect of delayed JOL by manipulating the correlation of word pairs at the memory task level, which had not been previously explored. Firstly, for the target list recall performance, whether under the condition of delayed JOL or restudy, the recall performance of high-related word pairs was significantly better than that of low-related word pairs, indicating that the experimental materials we selected were reliable. Secondly, the interaction between the learning strategy and material difficulty was significant, with the delayed JOL group performing significantly better than the restudy group for low-related word pairs, but no significant difference for high-related word pairs. This suggests that the forward effect of delayed JOL was only present in materials with some difficulty compared to materials that were very easy to learn. This result is in line with our hypothesis and consistent with previous related studies (Pastötter et al. 2013
; Yang et al. 2022
). However, in Experiment 1B, although the delayed JOL group showed better memory performance than the restudy group, the difference in retention between the two groups did not reach statistical significance. This may have been due to the null effect of the study strategy on easy items, which attenuates the overall main effect. In other words, the lack of a forward effect for easy materials resulted in a lower overall forward effect, which masked the main effect of the learning strategy.
For cumulative test performance, there was also a significant interaction between the learning strategy and material difficulty, which demonstrated that there was no significant difference between the delayed JOL group and the restudy group on the high-related word pairs. In contrast, the delayed JOL group performed significantly better than the restudy group on the low-related word pairs. This demonstrates that, under the multiple-list learning paradigm, the facilitation effect of delayed JOL were more evident in difficult material, whether for new items or learned items.
5. General Discussion
Using three experiments, the present study tested whether delayed JOL as a learning strategy could improve performance in subsequent learning of new material, both in a word pair memory task and in category learning. This not only provided empirical support for the forward effect of delayed JOL, but also further refined the conditions under which this facilitation effect applies.
Making metacognitive judgments can positively affect future memory performance (Soderstrom and Bjork 2015
; Witherby and Tauber 2017
). Delayed JOL is an important form of metacognitive judgment, and to examine learners’ subjective monitoring of their own learning, research participants are often asked to assess their likelihood of success on future tests. Whether learning single words or word pairs, if the process of delayed JOL involves implicit retrieval of the target item (e.g., presentation of only the stem or only the target word), then it can lead to a forward facilitation effect. This is consistent with the results found by Kubik et al.
) and perhaps also helps to explain the ineffectiveness of the results found in Experiment 1 by Lee and Ha
). Lee and Ha presented cues (paintings) and target items (artists’ names) simultaneously, which may have caused the learners to place themselves in a state similar to restudy without sufficient metacognitive assessment. In other words, the participants were not encouraged to retrieve the target words according to the cue items when making JOLs; therefore, the retrieval practice step was missing. Moreover, the meta-analysis results of Rhodes and Tauber
) showed that delayed JOL improved memory when presented the cue words only instead of a word pair, which was the only condition that allowed learners to make diagnostic attempts at long-term memory retrieval before the output of JOLs (Dunlosky and Nelson 1992
; Weaver and Kelemen 2003
; Rhodes and Tauber 2011
). There also seems to be a similar effect in terms of source memory monitoring, where participants may try to retrieve an item’s source when a Judgment of Source (JOS) is delayed and only items are available, whereas they do not engage in source retrieval when the complete source–item pair is presented (Schaper et al. 2022
In addition, Metcalfe and Finn
) found that learners experience at least two stages in the process of delayed JOL: the stage of familiarity judgment and the stage of target retrieval. Only after the target retrieval stage will the memory trace be deepened, thus improving the learning performance. In our study, only cue words were presented in the delayed JOL stage so that the participants could predict and judge the possibility of correctly recalling the target word in the future test. This process enabled learners to adopt more effective retrieval or coding strategies in subsequent learning according to the assessment of their learning performance. Therefore, compared with the restudy strategy which did not involve retrieval practice, the delayed JOL of only presenting cue words could significantly promote the learners’ subsequent learning of new material.
In Experiment 1A, the forward effect of delayed JOL was found to occur in materials of medium difficulty. Experiment 1B and Experiment 2 further explored the effect of material difficulty on the forward effect of delayed JOL by manipulating word–pair relevance in the memory task and the number of dimensions in the category task, respectively. The results showed that the forward effect of delayed JOL was present only for material of medium or high difficulty, and not for very easy to learn tasks. The failure-encode-effort theory suggests that failure to retrieve in the prior interim task leads participants to realize that their memories are more deficient than they thought they would be, which motivates learners to commit more effort to encode new information in order to narrow the perceived gap (Cho et al. 2017
). Previous studies have shown that retrieval failures or errors made in previous tests can encourage encoding in subsequent study phases (Potts and Shanks 2014
; Soderstrom and Bjork 2014
; Yang et al. 2017
In addition to enhancing encoding effort, the retrieval–effort theory proposed by Cho et al.
) may also help us to understand that the forward effect found in the current study is more obvious in materials with higher levels of difficulty. This theory hypothesizes that retrieval failure in a prior interim task causes learners to be dissatisfied with their learning performance. This dissatisfaction motivates them to exert more effort in subsequent learning to retrieve new information, and that greater retrieval effort yields better recall performance (Cho et al. 2017
; Yang et al. 2018
). Evidence for the retrieval–effort theory comes from Yang et al.’s Experiment 3. The experiment consisted of two groups, a test group and a restudy group, using the painter’s painting style and works as the experimental material. In the interim test of the last list, Yang et al. assessed their participants’ retrieval effort by measuring how much time they spent classifying the paintings. The result showed that the interim test group spent more time classifying the paintings than the restudy group, which was consistent with their hypothesis (Yang et al. 2019
). Thus, we might conclude that the learner’s effort and motivation play a moderating role in the forward effect. When presented with materials of varying difficulty in random order, the learners realized that learning the difficult material was not been as successful as they expected, and this experience of failure led them to exert more effort into encoding, using more effective retrieval strategies, or both, to prepare for subsequent learning.
For cumulative test scores, we found in Experiment 1A that the JOL group outperformed the restudy group. This result did not surprise us, as many previous studies have shown that making item-by-item JOLs can retroactively alter memory itself, a phenomenon known as the memory reactivity effect (Tauber et al. 2015
; Witherby and Tauber 2017
; Double et al. 2018
; Janes et al. 2018
; Tekin and Roediger 2021
; Li et al. 2022
). A positive reactivity theory proposes that making item-by-item JOLs may motivate participants to exert greater encoding effort, use more effective encoding strategies, and engage in more elaborate processing, leading to improved retention and a positive reactivity effect (Mitchum et al. 2016
; Li et al. 2022
). The results of Experiment 1B suggest that the advantage of delayed JOL is only demonstrated when the learning material is of some difficulty, perhaps because the easy items are not challenging for most learners and require little retrieval effort for them to learn, so retrieval practice seems to play a minimal role in the process. For the difficult items, the desirable difficulty theory suggests that activities that are more difficult and require more effort to retrieve are better for performance than those that are relatively easy to retrieve (Bjork and Bjotk 1992
). Bjork R. A and Bjork E. L. proposed the concepts of retrieval strength and storage strength and believed they could be improved by increasing retrieval difficulty to promote long-term retention of memory. If the ability to retrieve and store an item is weak, it is relatively difficult to retrieve the item. In other words, the higher the retrieval ability and storage intensity of the item after successful retrieval, the better the learning performance will be. Thus, the delayed JOL strategy that involves covert retrieval is significantly more effective than the restudy strategy. However, slightly different from previous studies, it is worth noting that since List 3 was tested once in the current study, the performance of the two groups on the cumulative test may have been affected by the testing effect of List 3 and not only the effect of the delayed JOL. Nevertheless, because both groups of participants were affected by the interim test in List 3, there were still significant differences on the final cumulative test, which we believe may also reflect, to some extent, the different effects of the two learning strategies on the learned material.
For educational practice, the discovery of the forward effect of delayed JOL provides a feasible learning strategy for practical learning. In the traditional view, tests are often used to measure students’ knowledge by using a standardized measure to evaluate the students’ achievements (Roediger and Butler 2011
). However, in actual educational practice, frequent testing is not generally accepted. Some educators believe that tests are time-consuming and laborious, which not only occupy teachers’ time to prepare, but also cause some students to suffer from test anxiety, depression tendencies, decreased learning motivation, and poor performance due to frequent tests (Hancock 2001
). However, together with the results of previous studies, this study shows that delayed JOL has great potential to improve learning performance not only on old and new material, but also on memory and categorical tasks. Therefore, teachers can introduce metacognitive strategies to students throughout the daily teaching process so that students can use cue-only metacognitive judgment to prepare for subsequent learning, especially when the learning material is difficult. For example, a teacher may provide students with metacognitive questions about learned concepts before learning new concepts.
The first limitation of this study is that its participants were all college students from the same school; therefore, the population structure of participants was relatively lacking in diversity, and the representation was insufficient to generalize the results. In particular, the students at this age have a good memory, have received a good education, and may use particular strategies proficiently in learning while learners at other ages may adopt different memory methods. Thus, in future research, researchers can choose primary and secondary school students as research groups to observe whether there is a difference in the results.
Second, our study used word pairs and artificial concept materials to explore the learners’ performance. Although these have enriched the experimental materials to some extent, it is not enough to prove that the forward effect of delayed JOL is a general phenomenon in learning. Future research could examine whether the forward effect of delayed JOL is robust in other learning materials or tasks.
Third, this study demonstrates that delayed JOL as a learning strategy can improve subsequent performance in new material and examines the necessary role of retrieval practice in this process. Based on the covert retrieval hypothesis, although delayed JOL involves covert retrieval and testing involves overt retrieval, the two learning strategies in paired-associate learning paradigms often produce similar benefits relative to restudy on delayed recall (Smith et al. 2013
; Putnam and Roediger 2013
). For example, Putnam and Roediger’s study (Experiment 3) compared restudy, covert retrieval, and overt retrieval and found that covert and overt retrieval yielded similar levels of recall, with both groups of participants recalling better than the restudy group. However, the truncated retrieval hypothesis challenges this view by suggesting that, relative to overt retrieval practice, people exert less effort into retrieval during delayed JOLs and thus truncate their retrieval attempts prematurely (Son and Metcalfe 2005
; Tauber et al. 2015
; Tekin and Roediger 2021
). Tekin and Roediger
) examined the effect of delayed JOLs on subsequent recall by comparing them to restudy, overt retrieval, and several other delayed JOL conditions. Their results showed that providing delayed JOLs reliably improved recall relative to restudy, but providing only delayed JOLs did not produce similar benefits relative to overt retrieval. Therefore, a strict covert retrieval hypothesis may not explain the promotion effects of delayed JOLs. Although this study provides some support for the covert retrieval hypothesis, it does not show that the effects of covert and overt retrieval are identical because they were not directly compared to the test group. Future studies could test the truncated retrieval hypothesis by directly comparing the delayed JOL with the test. In previous studies of the forward effects of tests, researchers have proposed the context-change theory, strategy-change theory, activation facilitation theory, test-expectancy theory, and others. Considering that the delayed JOL and test have a certain common basis, whether these theories are also applicable to the delayed JOL needs to be further tested (Sundqvist et al. 2012
; Akdoğan et al. 2016
Fourth, the results of previous related studies support the conclusion that cue-only JOL encourages learners to make retrieval attempts, both in terms of making delayed JOLs for the studied material (e.g., Rhodes and Tauber 2011
) and in terms of the forward effect of delayed JOLs (e.g., Kubik et al. 2022
). While the present study supports this conclusion by showing that delayed cue-only JOLs had a positive effect on word-pair and category learning, it is important to note that the effect of delayed JOLs with full information cannot be fully ruled out since they were not directly compared in this experiment.
Finally, this study found that material difficulty was one of the factors affecting the forward effect of delayed JOLs, but we did not directly measure and collect data about the students’ motivation. Although previous hypotheses, such as the failure-encoding-effort theory and the retrieval-effort theory, have received empirical support, providing data such as response times may explain our results more intuitively. Future research could investigate the influence of motivational factors on the forward effect by measuring the self-paced learning time of participants on new material.