“A Bias Recognized Is a Bias Sterilized”: The Effects of a Bias in Forecast Evaluation

Abstract

Are traditional tests of forecast evaluation well behaved when the competing (nested) model is biased? No, they are not. In this paper, we show analytically and via simulations that, under the null hypothesis of no encompassing, a bias in the nested model may severely distort the size properties of traditional out-of-sample tests in economic forecasting. Not surprisingly, these size distortions depend on the magnitude of the bias and the persistency of the additional predictors. We consider two different cases: (i) There is both in-sample and out-of-sample bias in the nested model. (ii) The bias is present exclusively out-of-sample. To address the former case, we propose a modified encompassing test (MENC-NEW) robust to a bias in the null model. Akin to the ENC-NEW statistic, the asymptotic distribution of our test is a functional of stochastic integrals of quadratic Brownian motions. While this distribution is not pivotal, we can easily estimate the nuisance parameters. To address the second case, we derive the new asymptotic distribution of the ENC-NEW, showing that critical values may differ remarkably. Our Monte Carlo simulations reveal that the MENC-NEW (and the ENC-NEW with adjusted critical values) is reasonably well-sized even when the ENC-NEW (with standard critical values) exhibits rejections rates three times higher than the nominal size.

1. Introduction

“Fortunately for serious minds, a bias recognized is a bias sterilized” Benjamin Haydon.

Diebold and Mariano (1995) [1] and West’s (1996) [2] seminal papers are typically pinpointed as the Big Bang of the forecast evaluation literature in economics and finance. Even though both papers propose asymptotically normal tests for forecast evaluation, the proper environment of each paper is different. The former considers the case of comparing forecasts (i.e., the forecasts are assumed to be given), while the latter focuses on the case of comparing models (i.e., the forecasts are constructed through estimated parametric models). Put simply, the key contribution of West’s asymptotic theory is that it accounts for parameter uncertainty.

While the asymptotic theory of [2] is quite general and allows a variety of estimation techniques and loss functions, it is not universal. One of the key assumptions in West’s theory is a full rank condition over the long-run variance of the loss function when parameters are set at their true values. One of the most iconic cases in which this condition is not fulfilled is the comparison of two competing nested models: Under the null hypothesis of no encompassing, both models are identical, and standard tests of forecast evaluation become degenerate. As pointed out by Clark and West (2006) [3] and West (2006) [4], for the case of MSPE, this degeneracy is not only important in a theoretical sense, but also in an empirical one: “[…] use of standard critical values usually results in very poorly sized tests, with far too few rejections. As well, the usual statistic has very poor power.” [4] p. 119 (A note of caution here. We are not arguing that this degeneracy necessarily imply that tests become undersize. Some of the simulations in [5] suggest both types of size distortions: sometimes tests are undersized, sometimes they are oversized.).

Not surprisingly, an important strand of this literature focuses on the case of nested models. Some of the most influential empirical papers in economic forecasting (such as Meese and Rogoff (1983, 1987) [6,7] and Goyal and Welsh (2003, 2008) [8,9], among many others) consider the random walk as one of the most relevant benchmarks. Anecdotally, and just to emphasize the relevance and popularity of nested models comparisons, to date, the number of citations of [2] is slightly lower than Clark and West (2007) [10] (1552 and 1615, respectively), even though the latter is exclusively useful for nested models (and published a decade later).

An interesting approach dealing with nested models is the one of Clark and McCracken (2001, 2005) [5,11] and McCracken (2007) [12] (henceforth, CM). Let P be the number of forecasts, R the initial number of observations used to estimate our parameters, T = P + R, and $f_{t+1}({\hat{\beta }}_t)$ the loss function for one-step-ahead forecasts, constructed with a vector of estimated parameters ${\hat{\beta }}_t$. The intuition of CM asymptotics is as follows: when models are nested, traditional tests of encompassing/accuracy become degenerate under the null hypothesis in the sense that $P^{-0.5}{\sum }_{t=R}^T{f}_{t+1}({\hat{\beta }}_t)\stackrel{plim}{\to }0$. However, CM notice that ${\sum }_{t=R}^T{f}_{t+1}({\hat{\beta }}_t)$ need not be degenerate. Based on this intuition, CM derive the asymptotic distribution of some traditional tests, such as the encompassing test of Harvey, Leybourne, and Newbold (1998) [13] (ENC-t), and a simple modification of the ENC-t (label as ENC-NEW). CM show that the asymptotic distribution of these tests is non-standard, but functions of integrals of quadratic Brownian motions. Under specific conditions, however, these statistics are free of nuisance parameters (See Clark and McCracken (2013) [14] for a detailed discussion on these conditions.). For those cases, CM tabulate their critical values, which depend on how we update our parameters (either rolling, recursive, or fixed), the number of excess parameters in the nesting model ($k_2$), and the asymptotic limit of the ratio P/R.

Even though the asymptotic theory of CM is useful in many contexts, it has some important caveats. In particular, Assumption 3 in [5,12] requires that the generalized forecast errors form a martingale difference sequence. This condition may be, of course, violated in the case of misspecification in the null model. As commented by [12]: “The assumption has the side effect of imposing a type of correct model specification […] and thus rules out applications where the predictive models are misspecified” [12], p. 725. In our opinion, this is a somewhat overlooked aspect of CM results (Even though they take a completely different approach, the most related papers addressing misspecification in predictive models are those of Chao, Corradi and Swanson (2001) [15], Armah and Swanswon (2008) [16], and Corradi and Swanson (2004, 2006, 2007) [17,18,19]. These authors propose bootstrap procedures that are robust to different types of misspecifications. In contrast, we derive the new asymptotics distributions in the presence of bias and propose a modification to the ENCNEW.). We show that the use of CM critical values ignoring the effects of these misspecifications may lead to severe size distortions; in specific, we address the effects of a bias in the predictive models over the asymptotic distributions (The implication of a bias in the nested model is that, under the null hypothesis, the forecasting errors of both models are biased. In essence, we show that a bias in the forecast errors distorts the size properties of traditional tests.).

Why not simply use a correctly specified model, then? First, our view about econometric forecasting models coincides with Box (1976) [20] in the sense that “All models are wrong, but some of them are useful.” In this context, as forecasting models are just parsimonious representations of complex phenomena, it is reasonable to expect some misspecification in the predictive models. Moreover, the econometric framework of CM considers exclusively linear models estimated by OLS; thus, any nonlinearity in the true model may introduce some type of misspecification. Second, if we knew in advance the “correct” model, then “there may be no compelling reason for performing an out-of-sample version of the particular test (…) in the first place.” [17], p. 188. As noticed by Inoue and Killian (2002) [21], in-sample analyses tend to outperform their out-of-sample counterparts in terms of power (when models are correctly specified); as a consequence, if we knew for sure that our models are “correct”, then the framework of CM may not be relevant to begin with: “After all, under the assumption of correct specification under the null, why not simply carry out in-sample inference, for the sake of efficiency?” [16], p. 196.

If there is indeed a bias in the null model, then by construction, the critical values provided by CM are no longer valid as they rule out this possibility. In this context, we distinguish between two relevant cases: (i) there is a bias both in-sample and out-of-sample, and (ii) we find the bias exclusively out-of-sample. A leading case in which (i) is relevant is the following: If the expected value of the target variable is not zero, and the benchmark is the zero-forecast from a driftless random walk (DRW), then by construction, both models are biased under the null hypothesis, and inference may be severely incorrect when using CM critical values. As the DRW is one of the most important benchmarks in the economic/financial forecasting literature, we think this is an important extension for CM work, and we focus on this leading case (Some empirical papers using the DRW as a benchmark and the ENC-NEW with CM critical values are Pincheira and Hardy (2019, 2021) [22,23], Pincheira et al. (2021) [24,25], to name a few. We emphasize, however, that these papers consider multiple benchmarks as the nested model (not exclusively the DRW). In this sense, we are not questioning they main conclusions by any means.). Regarding (ii), an important case is a shift in the expected value of the target variable: because out-of-sample analyses require multiple in-sample estimations of the parameters, a structural break in the expected value of the target variable (outside the estimation window) may introduce an out-of-sample bias.

Based on CM, we relax Assumption 3 in [5] and derive the new distribution for the ENC-NEW imposing a bias in the null model (either out-of-sample or both in-sample and out-of-sample). While this is a subtle change in the set of assumptions, it has important implications over the asymptotic theory. In particular, the quadratic Brownian motions in CM arise because of the martingale difference terms: put simply, the orthogonality condition is assumed to hold both in-sample and out-of-sample. In contrast, in our case, the quadratic Brownian motions arise because of both the predictors and the martingale difference terms. Moreover, if the bias appears both in-sample and out-of-sample, the persistency of the predictors shift the expected value of the integrals of quadratic Brownian motions, thus a re-centering is required: our MENC-NEW is simply a re-centered version of the ENC-NEW. Of course, in the absence of a bias, our MENC-NEW reduces to the ENC-NEW. As expected, we show that this new asymptotic distribution depends on the magnitude of the bias and the persistency of the predictors. Even though the asymptotic distribution of the MENC-NEW (and the ENC-NEW with adjusted critical values) is not pivotal, the nuisance parameters can be easily estimated (one of these nuisance parameters being, of course, the magnitude of the bias).

To assess the adequacy of our approach, we provide five different sets of Monte Carlo simulations. The firsts two sets are designed to evaluate the size properties of our MENC-NEW with both an in-sample and out-of-sample bias. The third and fourth sets are designed to evaluate the size properties of the ENC-NEW with our adjusted critical values when there is only an out-of-sample bias (e.g., a structural break). Finally, the last set of simulations evaluates the power properties of our test. We compare the performance of our approach against the ENC-NEW (with CM critical values), the ENC-t ([5]), and the Wild Clark and West (WCW-t) proposed by Pincheira, Hardy, and Muñoz (2021) [26]. Our results are crystal clear in terms of size: (i) A bias in the null model may severely distort the size properties of the ENC-NEW and the ENC-t. For instance, under the null hypothesis, some of our simulations show that both tests may exhibit an empirical size of over 30% when the nominal size is just 10%. (ii) In sharp contrast, our approach is reasonably well-sized in all our simulations, and it seems to outperform the other tests in most of our simulations. As expected, our approach dominates the rest of the tests when the data-generating process considers a greater bias. (iii) Next to our approach, the WCW-t exhibits the best size properties. However, as Pincheira, Hardy, and Muñoz (2021) [26] noticed, this advantage comes with a cost: the WCW-t displays significantly less power than the competitors. (iv) Even though the ENC-NEW is significantly more oversized than our test, there are no significant differences between both approaches in terms of power.

The rest of the paper is organized as follows. Section 2 provides a brief literature review on forecast evaluation in economics and finance. Section 3 considers the case of the DRW being the benchmark model (thus, a presence of a bias both in-sample and out-of-sample). In this case, we propose the MENC-NEW as the proper approach. Section 4 considers that the in-sample bias may be different than the out-of-sample bias. The interesting case here is an in-sample bias of zero but an out-of-sample bias different from zero (e.g., a shift in the drift of the target variable). In this case, there is no need for a re-centering term, although we derive the new asymptotic distribution of the ENC-NEW (thus, it is necessary to simulate new critical values). Section 5 considers four different Monte Carlo simulations to evaluate the size properties of our approach compared to traditional tests. At the end of Section 4 we evaluate the power of our test. Finally, Section 6 concludes.

2. Literature Review

Forecasting is a crucial area of study in financial econometrics: “It is obvious that forecasts are of great importance and widely used in economics and finance. Quite simply, good forecasts lead to good decisions.” Diebold and Lopez (1996) [27], page 241. Of course, a forecast has to be good in order to be useful, but how do we assess the quality of the forecast? This is the raison d’être of the literature on forecast evaluation. See West (2006) [4] and Clark and McCracken (2013) [28] for great reviews on forecast evaluation.

Forecast evaluation has a long tradition in economics and finance. For instance, West (2006) [4] recognizes Wilson (1934) [29] as one of the earlier examples. Other famous works are Fair and Shiller (1989, 1990) [30,31] and Meese and Rogoff (1983, 1988) [6,7]. The main intuition in these empirical papers is the following: if a model is a reasonably good representation of the target variable, then this model should accurately forecast this target variable. In this sense, forecast evaluation is at the core of empirical time series works.

Diebold and Mariano (1995) [1] (DM) and West (1996) [2] (DM) are two famous seminal papers addressing formal procedures to evaluate forecasts. In short, DM propose a direct application of the Central Limit Theorem for stationary series. In some sense, the DM approach is appealing since it relies on weak assumptions on the loss functions and the forecasting errors. Moreover, the Central Limit Theorem ensures asymptotic normality, and consequently, conducting inference in this framework is straightforward. Nevertheless, an important caveat of DM approach is that they consider forecasts as primitives: they do not consider the effects of parameter uncertainty. Intuitively, the DM approach is valid for evaluating forecasts but not for inference about models.

The main contribution of [2] is considering the case of forecasting models. The author provides a formal procedure for testing predictive ability at the population level. In essence, West (1996) [2] addresses how estimation error on the model’s parameters may affect the proper inference about predictive ability. West notices that, under specific conditions, estimation error vanishes asymptotically; if that is the case, then the approaches of DM and West are equivalent. Notably, the asymptotic theory of West rules out the comparison of nested models: West requires a full rank condition over the variance of the loss function. Intuitively, in nested models, the null hypothesis of equal predictive ability means that both models are identical; thus the asymptotic distribution of the test becomes degenerate.

The comparisons of nested models are extremely relevant in economics and finance. A common practice is to compare the predictive performance of a model with a simple benchmark. One of the most commonly considered benchmarks in the empirical literature is the random walk, which is typically nested on more complex predictive models. Even though the random walk is very simple, it is difficult to outperform empirically. As famously noticed by Meese and Rogoff (1983) [6]: “We find that a random walk model would have predicted major-country exchange rates during the recent floating-rate period as well as any of our candidate models” [6], page 3. See Goyal and Welch (2008) [9], Rossi (2013) [32], and Meese and Rogoff (1988) [7] for more examples of how difficult it is to outperform a simple random walk with more complex nesting models.

Due to its relevance, an important strand of the literature focuses on formal procedures to compare the predictive performance of nested models. [3,10] (CW) propose an adjusted mean squared prediction error test and show via simulations that this test works reasonably well with standard normal critical values. Recently, Pincheira, Hardy, and Muñoz (2021) [26] (PHM) consider a simple modification of the CW test. This modification prevents the core statistic from becoming degenerate under the null hypothesis. Using the asymptotic theory of [2], PHM show their test is asymptotically standard normal. The main caveat of their approach is that their modification erodes some of the power properties of the CW test.

A different approach to deal with nested models is Clark and McCracken (2001, 2005) [5,11] and McCracken (2007) [12]. Using a completely different asymptotic theory, the authors consider some of the most popular tests on forecast evaluation and derive its asymptotic distributions for the special case of nested models. The authors show that the asymptotic distribution of these statistics are no longer standard, but they are functionals of quadratic Brownian motions. These distributions depend on the estimation scheme considered to update the parameters of the model (either rolling or recursive), on the number of excess parameters in the nesting model, and the asymptotic limit of the ratio P/R. Under some specific conditions, CM show that these distributions are pivotal, and they provide tables with the relevant critical values. Additionally, CM propose a different regularization of the encompassing test proposed by [13]: they label this new encompassing test as ENC-NEW. Notably, the simulations of CM suggest that the ENC-NEW has an edge in terms of power compared to other traditional tests.

One problem with the approach of CM is that it requires the strong assumption that the nested model is correctly specified. In essence, this assumption rules out the possibility of a bias in the benchmark model: CM critical values are no longer valid if that is the case. To the best of our knowledge, this is the first paper that addresses how this type of misspecification affects the asymptotic distribution of these tests. First, using the same asymptotic theory as CM, we analyze how a bias in the null model modifies the asymptotic distribution of the ENC-NEW. This asymptotic distribution is not pivotal and relies on the magnitude of the bias and the persistency of the additional predictor in the nesting model. Second, we show via simulations that ignoring this bias, and using CM critical values, may lead to severe size distortions. Not surprisingly, these size distortions depend on the magnitude of the bias and the persistency of the additional predictor.

3. The DRW as the Null Model and the MENC-NEW Test

The theory of CM imposes that the benchmark model is correctly specified. This section illustrates the effects of failing this condition because of a bias both in-sample and out-of-sample. The leading case is the DRW as the benchmark model: If the expected value of the target variable is not zero, then both models are biased under the null hypothesis of no encompassing. Considering the relevance of the DRW in the forecasting literature, we focus exclusively on this leading case. With some technical subtleties, the generalization to a linear parametric benchmark is straightforward and considered in Section 4. In fact, the results of Section 3 are special cases of our decompositions in Section 4.

Following CM notation, let $k_2$ be the number of parameters in the nesting model, and W(s) be a ($k_2$ × 1) vector Brownian Motion with covariance kernel equal to the Identity matrix. Let $Y_{t+1}$ be our scalar target variable, such that $Y_{t+1}=\Delta +u_{t+1}$, where $u_{t+1}$ is simply white noise with variance ${\sigma }_u^2$, and $\Delta$ captures the expected value of $Y_{t+1}$. Consider two competing models for one-step-ahead forecasts. The first one is simply the zero-forecast from a DRW, with forecast errors denoted by $u_{1,t+1}$

$$Y_{t+1}=u_{1,t+1} (model 1:null model)$$

The second one is a linear parametric model

$$Y_{t+1}=X_t^{\prime }{\beta }^{*}+u_{2,t+1} (model 2:alternative model)$$

Notice that ${\beta }^{*}$ denotes the ($k_2$ × 1)vector of true (population) parameters, and $X_t$ is assumed to be a zero-mean covariance stationary vector with $k_2$ predictors. Let $\widehat{{\beta }_t}$ be the OLS estimator of ${\beta }^{*}$ constructed with either a rolling, recursive or fixed scheme.

As noticed by [2], under the null hypothesis of no encompassing, ${\beta }^{*}=0$ and both models are identical $\forall t$. Therefore, traditional tests of encompassing/accuracy become degenerate under West’s asymptotic theory. To address this issue, CM proposes a new statistic (ENC-NEW) and derives its distribution using a different set of asymptotics:

$$ENC-NEW=P\frac{P^{-1}{\sum }_t{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1})}{P^{-1}{\sum }_t{\hat{u}}_{2,t+1}^2}$$

The limit distribution of the ENC-NEW depends on how parameters are updated: either ${\int }_{R/T }^1{s}^{-1}{W}^{\prime }(s)dW(s)$ (recursive), or $T/R{\int }_{R/T }^1(W(s)-W(s-\frac{R}{T}))\prime dW(s)$ (rolling) (see [5], page 93). One of the key assumptions in CM is that model 1 is correctly specified under the null hypothesis. Notice, however, that the DRW is free of parameter uncertainty (${\hat{u}}_{1,t+1}=u_{1,t+1}$), and

$$Y_{t+1}={\hat{u}}_{1,t+1}=u_{1,t+1}=\Delta +u_{t+1}$$

In contrast, for the case of model 2

$$Y_{t+1}=X_t^{\prime }\widehat{{\beta }_t}+{\hat{u}}_{2,t+1}$$

where ${\hat{u}}_{2,t+1}=X_t^{\prime }({\beta }^{*}-\widehat{{\beta }_t})+u_{2,t+1}$.

Under the null of no encompassing ${\beta }^{*}=0$, and $u_{1,t+1}=u_{2,t+1}=\Delta +u_{t+1}$. This is the key difference with CM asymptotic theory: Under the null hypothesis, both models are misspecified (biased); the reason is that none of them capture the mean of our target variable. By construction, we will observe this type of misspecification whenever $\Delta =E{Y}_{t+1}\ne 0$ and the nested model is the zero-forecast from a driftless random walk. We focus on this leading case as the DRW is one of the most frequently used benchmarks in financial forecasting.

Let us now illustrate the effects of this bias over the asymptotic distribution of the ENC-NEW. For clarity of exposition, we focus our discussion and most of our proofs on the recursive scheme (since they are all similar). However, we do provide the main results for both the rolling and recursive schemes in Theorem 2.

From West (1996), under the null hypothesis, we have $P^{-1}\sum _t{\hat{u}}_{2,t+1}^2\to {\sigma }_2^2=E{u}_{t+1}^2+{\Delta }^2={\sigma }_u^2+{\Delta }^2$ as $P,R\to \infty$. In what follows, we assume that $X_t{u}_{t+1}$ (rather than $X_t{u}_{2,t+1}$) is a martingale difference sequence. Now consider the following decomposition of the core statistic

$$\begin{gathered} \sum _{t=R}^{T-1}{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1})=\sum _{t=R}^{T-1}{u}_{1,t+1}(u_{1,t+1}-{\hat{u}}_{2,t+1})=\sum _{t=R}^{T-1}{u}_{1,t+1}{X}_t^{\prime }(\widehat{{\beta }_t}-{\beta }^{*}) \\ =\sum _{t=R}^{T-1}(\Delta +u_{t+1})X_t^{\prime }(\widehat{{\beta }_t}-{\beta }^{*}) \end{gathered}$$

$$\begin{gathered} =\sum _{t=R}^{T-1}\left(\Delta +u_{t+1}\right)X_t^{\prime }{\left({\sum }_{j=1}^{t-1}{X}_j{X}_j^{\prime }\right)}^{-1}\left({\sum }_{j=1}^{t-1}{X}_j{\hat{u}}_{2,j+1}\right) \\ =\sum _{t=R}^{T-1}\left(\Delta +u_{t+1}\right)X_t^{\prime }{\left({\sum }_{j=1}^{t-1}{X}_j{X}_j^{\prime }\right)}^{-1}\left({\sum }_{j=1}^{t-1}{X}_j\left(\Delta +u_{j+1}\right)\right) \end{gathered}$$

$$=\sum _{t=R}^{T-1}{u}_{t+1}{X}_t^{\prime }{\left({\sum }_{j=1}^{t-1}{X}_j{X}_j^{\prime }\right)}^{-1}\left({\sum }_{j=1}^{t-1}{X}_j{u}_{j+1}\right)$$

(1)

$$+\Delta \sum _{t=R}^{T-1}{X}_t^{\prime }{\left({\sum }_{j=1}^{t-1}{X}_j{X}_j^{\prime }\right)}^{-1}\left({\sum }_{j=1}^{t-1}{X}_j{u}_{j+1}\right)$$

(2)

$$+\Delta \sum _{t=R}^{T-1}{u}_{t+1}{X}_t^{\prime }{\left({\sum }_{j=1}^{t-1}{X}_j{X}_j^{\prime }\right)}^{-1}\left({\sum }_{j=1}^{t-1}{X}_j\right)$$

(3)

$$+{\Delta }^2\sum _{t=R}^{T-1}{X}_t^{\prime }{\left({\sum }_{j=1}^{t-1}{X}_j{X}_j^{\prime }\right)}^{-1}\left({\sum }_{j=1}^{t-1}{X}_j\right)$$

(4)

This decomposition explicitly shows the effects of parameter uncertainty on the core statistic. This decomposition shows that the asymptotic distribution of the ENC-t and the ENC-NEW depend on how parameters are updated (either rolling or recursive). Notice that (1) is precisely the same term as Lemma A6 in [33] for the numerator of the ENC-NEW in a recursive scheme. We may interpret (1) as the core statistic when there is no bias; if that is the case, then $\Delta =0,$and consequently $(2)=0,(3)=0,$ $(4)=0,$ and the distribution of the ENC-NEW is given by Theorem 3.3 of [5]. Nevertheless, (2–4) in the ENC-NEW arise because $\Delta$ may not be zero. In some sense, we can view Clark and McCracken (2001) [5] results as the special case in which $\Delta =0$. To establish the new asymptotic distribution of the ENC-NEW, we use the following results.

Lemma 1.

(a) 

$$\begin{gathered} {\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{u}_{t+1}{X}_t^{\prime }{(\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_j{X}_j^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j{u}_{j+1})= \\ {\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{u}_{t+1}{X}_t^{\prime }{(E{X}_t{X}_t^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j{u}_{j+1})+o_p(1) \end{gathered}$$

(b) 

$$\begin{gathered} \Delta {\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{X}_t^{\prime }{(\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_j{X}_j^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j{u}_{j+1})= \\ \Delta {\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{X}_t^{\prime }{(E{X}_t{X}_t^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j{u}_{j+1})+o_p(1) \end{gathered}$$

(c) 

$$\begin{gathered} \Delta {\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{u}_{t+1}{X}_t^{\prime }{(\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_j{X}_j^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j)= \\ \Delta {\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{u}_{t+1}{X}_t^{\prime }{(E{X}_t{X}_t^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j)+o_p(1) \end{gathered}$$

(d) 

$$\begin{gathered} {\Delta }^2{\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{X}_t^{\prime }{(\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_j{X}_j^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j)= \\ {\Delta }^2{\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{X}_t^{\prime }{(E{X}_t{X}_t^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j)+o_p(1) \end{gathered}$$

(e) 

$$\begin{gathered} {\sum }_{t=R}^{T-1}{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1})={\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{u}_{t+1}{X}_t^{\prime }{(E{X}_t{X}_t^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j{u}_{j+1})+ \\ \Delta {\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{X}_t^{\prime }{(E{X}_t{X}_t^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j{u}_{j+1})+ \\ \Delta {\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{u}_{t+1}{X}_t^{\prime }{(E{X}_t{X}_t^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j)+ \\ {\Delta }^2{\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{X}_t^{\prime }{(E{X}_t{X}_t^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j)+o_p(1) \end{gathered}$$

Proof of Lemma 1.

See Appendix A. □

Theorem 1 next establishes the asymptotic distribution of $\sum _t{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1})$

Theorem 1.

(a) 

${\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{u}_{t+1}{X}_t^{\prime }{(E{X}_t{X}_t^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j{u}_{j+1})\to {\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s){\sigma }_u^{2}d{W}_1(s)$

(b) 

$$\begin{gathered} \Delta {\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{X}_t^{\prime }{(E{X}_t{X}_t^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j{u}_{j+1})\to \\ \Delta {\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)\sqrt{{(E{X}_t{X}_t^{\prime })}^{-1}}{\sigma }_u\sqrt{V_{LP}(X_t)}d{W}_2(s) \end{gathered}$$

(c) 

$$\begin{gathered} \Delta {\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{u}_{t+1}{X}_t^{\prime }{(E{X}_t{X}_t^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j)\to \\ \Delta {\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)\sqrt{V_{LP}(X_t)}{\sigma }_u\sqrt{{(E{X}_t{X}_t^{\prime })}^{-1}}d{W}_1(s) \end{gathered}$$

(d) 

$$\begin{gathered} {\Delta }^2{\sum }_{t=R}^{T-1}(\frac{T}{t})T^{-0.5}{X}_t^{\prime }{(E{X}_t{X}_t^{\prime })}^{-1}(T^{-0.5}{\sum }_{j=1}^{t-1}{X}_j)\to \\ {\Delta }^2{\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)V_{LP}(X_t){(E{X}_t{X}_t^{\prime })}^{-1}d{W}_2(s)+{\mathsf{\Lambda }}_{\mathsf{\lambda }} \end{gathered}$$

(e) 

$$\begin{gathered} {\sum }_{t=R}^{T-1}{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1})\to {\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s){\sigma }_u^{2}d{W}_1(s)+ \\ \Delta {\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)\sqrt{{(E{X}_t{X}_t^{\prime })}^{-1}}{\sigma }_u\sqrt{V_{LP}(X_t)}d{W}_2(s)+ \\ \Delta {\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)\sqrt{V_{LP}(X_t)}{\sigma }_u\sqrt{{(E{X}_t{X}_t^{\prime })}^{-1}}d{W}_1(s)+ \\ {\Delta }^2{\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)V_{LP}(X_t){(E{X}_t{X}_t^{\prime })}^{-1}d{W}_2(s)+{\mathsf{\Lambda }}_{\mathsf{\lambda }} \end{gathered}$$

Proof of Theorem 1.

See Appendix B. □

Where $V_{LP}(X_t)$ is the long-run variance of $X_t$ and $\lambda =R/T$. As expected, if $\Delta =0$ (no bias), the asymptotic distribution of $\sum _t{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1})$ is simply Theorem 1a ${\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s){\sigma }_u^{2}d{W}_1(s)$ (the same as CM). In contrast, if $\Delta \ne 0$, the limit distribution of $\sum _t{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1})$ is given by Theorem 1e. Compared to CM, this distribution depends on three additional nuisance parameters: the magnitude of the bias $\Delta$, the persistency of the additional predictors $V_{LP}(X_t)$, and ${\mathsf{\Lambda }}_{\mathsf{\lambda }}$ (which is also related to the persistency of the predictors). Consequently, our Monte Carlo simulations in Section 5 reveal that the size distortions of the ENC-NEW with CM critical values are more severe with greater bias and persistency. Finally, Theorem 1e establishes that the asymptotic distribution of $\sum _t{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1})$ depend not only on $W_1$, but also on $W_2$; while the former appears because of $X_t{u}_{t+1}$ (as in CM), the latter arises because of $X_t$. Both standard Brownian motions are assumed to be independent.

Notice that the term ${\mathsf{\Lambda }}_{\mathsf{\lambda }}$ in Theorem 1d,e arises because of the correlation between $X_t$ and $X_j$ (i.e., the correlation between the predictor in the estimation window and the evaluation window). Put simply, ${\mathsf{\Lambda }}_{\mathsf{\lambda }}$ depends on the covariances $E(X_t{X}_j)$. As $W_1$ and $W_2$ are assumed to be independent, we must re-center the term ${\Delta }^2{\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)V_{LP}(X_t){(E{X}_t{X}_t^{\prime })}^{-1}d{W}_2(s)$ whenever $E(X_t{X}_j)\ne 0$. See White (2014) [34], p.198 for a similar discussion. Notably, our proposed MENC-NEW is simply a re-centered version of the ENC-NEW, corrected by the autocovariances that shift the distribution. The re-centering term is simply the expected value of ${\Delta }^2{\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)V_{LP}(X_t){(E{X}_t{X}_t^{\prime })}^{-1}d{W}_2(s)$. For clarity of exposition, consider the following example: suppose that $k_2=1$, then, the re-centering term is simply given by

$${\mathsf{\Lambda }}_{\mathsf{\lambda }}={\Delta }^2{(E{X}_t^2)}^{-1}\sum _{t=R}^{T-1}(\frac{1}{t}){\sum }_{j=1}^{t-1}E(X_t{X}_j)$$

While the sum of the covariances $E(X_t{X}_j)$ are unknown, they can be estimated. Our proposed MENC-NEW is simply

$$MENC-NEW=P\frac{P^{-1}{\sum }_t{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1})-{\hat{\mathsf{\Lambda }}}_{\mathsf{\lambda }}}{P^{-1}{\sum }_t{\hat{u}}_{2,t+1}^2}$$

As ${\hat{\mathsf{\Lambda }}}_{\mathsf{\lambda }}$ depends on the sum of the covariances, its estimation depends on $\mathsf{\lambda }$ (i.e., how we split the data). Notice that this re-centering term will be particularly important for both the recursive and the rolling scheme. Theorem 2 next establishes the asymptotic distribution of our MENC-NEW for each updating scheme

Theorem 2.

The asymptotic distribution of the MENC-NEW =$P\frac{P^{-1}{\sum }_t{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1})-{\hat{\Lambda }}_{\lambda }}{P^{-1}{\sum }_t{\hat{u}}_{2,t+1}^2}$is given by

$$\begin{array}{l}MENC-NE{W}_{P,R\to \infty }(recursive)\\ \to \frac{1}{{\sigma }_u^2+{\Delta }^2}\{{\sigma }_u^2{\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)d{W}_1(s)\\ +\Delta {\sigma }_u{\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s){(E{X}_t{X}_t^{\prime })}^{-0.5}\sqrt{V_{LP}(X_t)}d{W}_2(s)\\ +\Delta {\sigma }_u{\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s){(E{X}_t{X}_t^{\prime })}^{-0.5}\sqrt{V_{LP}(X_t)}d{W}_2(s)\\ +{\Delta }^2{\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)V_{LP}(X_t){(E{X}_t{X}_t^{\prime })}^{-1}d{W}_2(s)\}\end{array}$$

$$\begin{array}{l}MENC-NE{W}_{P,R\to \infty }(rolling)\\ \to \frac{{\lambda }^{-1}}{{\sigma }_u^2+{\Delta }^2}\{{\sigma }_u^2{\int }_{\lambda }^1{[W_1(s)-W_1(s-\lambda )]}^{\prime }d{W}_1(s)\\ +\Delta {\sigma }_u{\int }_{\lambda }^1{[W_1(s)-W_1(s-\lambda )]}^{\prime }{(E{X}_t{X}_t^{\prime })}^{-0.5}\sqrt{V_{LP}(X_t)}d{W}_2(s)\\ +\Delta {\sigma }_u{\int }_{\lambda }^1{[W_2(s)-W_2(s-\lambda )]}^{\prime }\sqrt{V_{LP}(X_t)}{(E{X}_t{X}_t^{\prime })}^{-0.5}d{W}_1(s)\\ +{\Delta }^2{\int }_{\lambda }^1{[W_2(s)-W_2(s-\lambda )]}^{\prime }{V}_{LP}(X_t){(E{X}_t{X}_t^{\prime })}^{-1}d{W}_2(s)\}\end{array}$$

where $R/T_{T,R\to \infty }\to \mathsf{\lambda }$.

Proof of Theorem 2.

This is a direct application of Theorem 1 and the definition of the MENC-NEW. □

Not surprisingly, if we set $\Delta =0$ (i.e., no bias), the asymptotic distribution of the MENC-NEW is either ${\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)d{W}_1(s)$ (recursive) or ${\lambda }^{-1}{\int }_{\lambda }^1{[W_1(s)-W_1(s-\lambda )]}^{\prime }d{W}_1(s)$, which is precisely Theorem 3c in [5]. In other words, Theorem 2 encompasses the case of unbiasedness.

The main caveat of our approach is that these distributions are not pivotal: to obtain our critical values, we must simulate the quadratic Brownian motions using estimates of these nuisance parameters. Notably, relaxing the assumption of $\Delta =0$ has the implication of significantly complicating the asymptotic distribution of our test. Notice that the distribution of the MENC-NEW depends on (i) The number of excess parameters $k_2$. (ii) How we update our parameters (either recursive or rolling). (iii) The persistency of the additional predictors (through $V_{LP}(X_t)$). (iv) The magnitude and sign of the bias (through $\Delta$), (v) The variances ${(E{X}_t{X}_t^{\prime })}^{-1}$ and ${\sigma }_u^2$. (vi) How we split our dataset (through $\mathsf{\lambda }$)). In contrast, the distribution of the ENC-NEW simply depends on (i), (ii), and (vi); for this reason, CM are able to provide a set of critical values. We emphasize, however, that ignoring these additional parameters may distort the empirical size of these tests, as we show in Section 5.

Let us illustrate numerically how the critical values of the ENC-NEW might change with a bias. Suppose we are interested in a recursive scheme, and we use fifty percent of the sample to estimate our parameters and fifty percent to evaluate our forecasts ($\frac{R}{T}=0.5$, or $\frac{P}{R}=1$). Suppose the alternative model consider just one predictor $X_t$ (i.e., $k_2=1$). Under the assumption of no bias, CM establishes the asymptotic distribution of the ENC-NEW in a recursive scheme as ${\int }_{R/T }^1{s}^{-1}W(s)dW(s)$, and tabulate the corresponding critical values. In this case, the critical values for 10%, 5%, and 1% significance levels are 0.984, 1.584, and 3.209, respectively.

Nevertheless, Theorem 1 establishes that the proper asymptotic distribution of the ENC-NEW with a bias under the null model should be

$$\begin{array}{l}\frac{1}{\Delta +{\sigma }_u^2}({\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s){\sigma }_u^{2}d{W}_1(s)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\Delta {\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)\sqrt{{(E{X}_t{X}_t^{\prime })}^{-1}}{\sigma }_u\sqrt{V_{LP}(X_t)}d{W}_2(s)\end{array}$$

$$\begin{array}{l}+\Delta {\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)\sqrt{V_{LP}(X_t)}{\sigma }_u\sqrt{{(E{X}_t{X}_t^{\prime })}^{-1}}d{W}_1(s)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\Delta }^2{\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)V_{LP}(X_t){(E{X}_t{X}_t^{\prime })}^{-1}d{W}_2(s)+{\mathsf{\Lambda }}_{\mathsf{\lambda }})\end{array}$$

Suppose the predictor in the alternative model follows an AR(1) process: $X_t=0.5X_{t-1}+u_{X,t}$, with $V(u_{X,t})=0.1$. If that is the case, $V_{LP}(X_t)=\frac{0.1}{{(1-0.5)}^2}=0.4$ and $E{X}_t^2=0.1333$. Set $\Delta =0.5$ and ${\sigma }_u^2=0.1$. The critical values for 10%, 5%, and 1% significance levels in this new scenario are 2.380, 3.760, and 7.676, respectively. In this case, the bias in the null model shifts the critical values significantly to the right: in this illustration, the critical values are twice those tabulated by CM. Consequently, using CM critical values may lead to severely oversized tests. This is supported by our Monte Carlo simulations.

4. The General Case of a Parametric Nested Benchmark and the Asymptotic Distribution of the ENC-NEW

This section generalizes our previous result considering a parametric nested benchmark (possibly different from the DRW). Here, we allow the possibility of having different in-sample and out-of-sample biases. As we estimate our parameters by OLS, it is reasonable to think that the bias is present exclusively out-of-sample. For this reason, we derive the asymptotic distribution of the ENC-NEW considering an in-sample and an out-of-sample bias (${\Delta }_{\mathrm{IS}}$ and ${\Delta }_{\mathrm{oos}}$ respectively and possibly ${\Delta }_{\mathrm{IS}}\ne {\Delta }_{\mathrm{oos}}$), and then we focus on the particular (but more relevant) case of ${\Delta }_{\mathrm{IS}}=0$.

Let $X_1$ and $X_2$ be the ($k_1$ × 1) and ($k$ × 1) vectors of predictors of the nested and nesting models, respectively. In contrast to the previous section, $k=k_1+k_2$; in other words, $k_2$ indicates the additional predictors considered in the nesting model. Let J be a $k_1$ × $k_2$ indication matrix, constructed as a ($k_1$ × $k_1)$ identity matrix, and zeroes elsewhere. In contrast to the previous section, additional terms arise in our decomposition because model 1 is also affected by parameter uncertainty. We focus our discussion and most of our proofs on the recursive scheme. We do provide, however, the main results for both the rolling and recursive methods. First, notice that

$${\hat{u}}_{1,t+1}=u_{1,t+1}-X_{1t}^{\prime }({\hat{\beta }}_{1t}-{\beta }_1^{*})=(u_{t+1}+{\Delta }_{\mathrm{oos}})-X_{1t}^{\prime }({\hat{\beta }}_{1t}-{\beta }_1^{*})$$

$${\hat{u}}_{2,t+1}=u_{2,t+1}-X_{2t}^{\prime }({\hat{\beta }}_{2t}-{\beta }_2^{*})=(u_{t+1}+{\Delta }_{\mathrm{oos}})-X_{2t}^{\prime }({\hat{\beta }}_{2t}-{\beta }_2^{*})$$

Then, the core statistic from the ENC-NEW is simply given by

$$\sum _{t=R}^{T-1}{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1})$$

$$=\sum _{t=R}^{T-1}[(u_{t+1}+{\Delta }_{\mathrm{oos}})-X_{1t}^{\prime }({\hat{\beta }}_{1t}-{\beta }_1^{*})][X_{2t}^{\prime }({\hat{\beta }}_{2t}-{\beta }_2^{*})-X_{1t}^{\prime }({\hat{\beta }}_{1t}-{\beta }_1^{*})]$$

$$=\sum _{t=R}^{T-1}(u_{t+1}+{\Delta }_{\mathrm{oos}})[X_{2t}^{\prime }({\hat{\beta }}_{2t}-{\beta }_2^{*})-X_{1t}^{\prime }({\hat{\beta }}_{1t}-{\beta }_1^{*})]$$

(5)

$$-\sum _{t=R}^{T-1}{X}_{1t}^{\prime }({\hat{\beta }}_{1t}-{\beta }_1^{*})[X_{2t}^{\prime }({\hat{\beta }}_{2t}-{\beta }_2^{*})-X_{1t}^{\prime }({\hat{\beta }}_{1t}-{\beta }_1^{*})]$$

(6)

To establish the distribution of (5) and (6), we need additional results, provided by Lemmas 2 through 5. While these Lemmas are similar to those of CM, they are not immediate from their results.

Lemma 2.

$\underset{t}{\sup }{T}^{0.5}|\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{lj}|=O_p(1)$, with $l=1,2$.

Proof Lemma 2.

See Appendix C. □

Lemma 3.

(a) 

$\begin{array}{c}{\sum }_{t=R}^{T-1}{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{u}_{j+1}\}}^{\prime }{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{X}_{1j}^{\prime }\}}^{-1}{X}_{1t}{X}_{2t}^{\prime }{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{2j}{X}_{2j}^{\prime }\}}^{-1}\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{2j}{u}_{j+1}\}\\ ={\displaystyle \sum _{t=R}^{T-1}}{\left\{\frac{1}{t}{\displaystyle \sum _{j=1}^{t-1}}{X}_{1j}{u}_{j+1}\right\}}^{\prime }{\{E{X}_1{X}_1^{\prime }\}}^{-1}E{X}_{1t}{X}_{2t}^{\prime }{\{E{X}_2{X}_2^{\prime }\}}^{-1}\left\{\frac{1}{t}{\displaystyle \sum _{j=1}^{t-1}}{X}_{2j}{u}_{j+1}\right\}+o_p(1)\end{array}$

(b) 

$\begin{array}{c}{\sum }_{t=R}^{T-1}{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{u}_{j+1}\}}^{\prime }{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{X}_{1j}^{\prime }\}}^{-1}{X}_{1t}{X}_{1t}^{\prime }{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{X}_{1j}^{\prime }\}}^{-1}\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{u}_{j+1}\}\\ ={\displaystyle \sum _{t=R}^{T-1}}{\left\{\frac{1}{t}{\displaystyle \sum _{j=1}^{t-1}}{X}_{1j}{u}_{j+1}\right\}}^{\prime }{\{E{X}_1{X}_1^{\prime }\}}^{-1}\left\{\frac{1}{t}{\displaystyle \sum _{j=1}^{t-1}}{X}_{1j}{u}_{j+1}\right\}+o_p(1)\end{array}$

(c) 

$\begin{array}{c}{\sum }_{t=R}^{T-1}{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{u}_{j+1}\}}^{\prime }{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{X}_{1j}^{\prime }\}}^{-1}{X}_{1t}{X}_{1t}^{\prime }{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{X}_{1j}^{\prime }\}}^{-1}\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}\}\\ ={\displaystyle \sum _{t=R}^{T-1}}{\left\{\frac{1}{t}{\displaystyle \sum _{j=1}^{t-1}}{X}_{1j}{u}_{j+1}\right\}}^{\prime }{\{E{X}_1{X}_1^{\prime }\}}^{-1}\left\{\frac{1}{t}{\displaystyle \sum _{j=1}^{t-1}}{X}_{1j}\right\}+o_p(1)\end{array}$

(d) 

$\begin{array}{c}{\sum }_{t=R}^{T-1}{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{u}_{j+1}\}}^{\prime }{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{X}_{1j}^{\prime }\}}^{-1}{X}_{1t}{X}_{2t}^{\prime }{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{2j}{X}_{2j}^{\prime }\}}^{-1}\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{2j}\}\\ ={\displaystyle \sum _{t=R}^{T-1}}{\left\{\frac{1}{t}{\displaystyle \sum _{j=1}^{t-1}}{X}_{1j}{u}_{j+1}\right\}}^{\prime }{\{E{X}_1{X}_1^{\prime }\}}^{-1}E{X}_{1t}{X}_{2t}^{\prime }{\{E{X}_2{X}_2^{\prime }\}}^{-1}\left\{\frac{1}{t}{\displaystyle \sum _{j=1}^{t-1}}{X}_{2j}\right\}+o_p(1)\end{array}$

(e) 

$\begin{array}{c}{\sum }_{t=R}^{T-1}{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}\}}^{\prime }{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{X}_{1j}^{\prime }\}}^{-1}{X}_{1t}{X}_{2t}^{\prime }{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{2j}{X}_{2j}^{\prime }\}}^{-1}\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{2j}{u}_{j+1}\}\\ ={\displaystyle \sum _{t=R}^{T-1}}{\left\{\frac{1}{t}{\displaystyle \sum _{j=1}^{t-1}}{X}_{1j}\right\}}^{\prime }{\{E{X}_1{X}_1^{\prime }\}}^{-1}E{X}_{1t}{X}_{2t}^{\prime }{\{E{X}_2{X}_2^{\prime }\}}^{-1}\left\{\frac{1}{t}{\displaystyle \sum _{j=1}^{t-1}}{X}_{2j}{u}_{j+1}\right\}+o_p(1)\end{array}$

(f) 

$\begin{array}{c}{\sum }_{t=R}^{T-1}{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}\}}^{\prime }{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{X}_{1j}^{\prime }\}}^{-1}{X}_{1t}{X}_{1t}^{\prime }{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{X}_{1j}^{\prime }\}}^{-1}\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}\}\\ ={\displaystyle \sum _{t=R}^{T-1}}{\left\{\frac{1}{t}{\displaystyle \sum _{j=1}^{t-1}}{X}_{1j}\right\}}^{\prime }{\{E{X}_1{X}_1^{\prime }\}}^{-1}\left\{\frac{1}{t}{\displaystyle \sum _{j=1}^{t-1}}{X}_{1j}\right\}+o_p(1)\end{array}$

(g) 

$\begin{array}{c}{\sum }_{t=R}^{T-1}{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}\}}^{\prime }{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{1j}{X}_{1j}^{\prime }\}}^{-1}{X}_{1t}{X}_{2t}^{\prime }{\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{2j}{X}_{2j}^{\prime }\}}^{-1}\{\frac{1}{t}{\sum }_{j=1}^{t-1}{X}_{2j}\}\\ ={\displaystyle \sum _{t=R}^{T-1}}{\left\{\frac{1}{t}{\displaystyle \sum _{j=1}^{t-1}}{X}_{1j}\right\}}^{\prime }{\{E{X}_1{X}_1^{\prime }\}}^{-1}E{X}_{1t}{X}_{2t}^{\prime }{\{E{X}_2{X}_2^{\prime }\}}^{-1}\left\{\frac{1}{t}{\displaystyle \sum _{j=1}^{t-1}}{X}_{2j}\right\}+o_p(1)\end{array}$

Proof of Lemma 3.

The proofs for Lemma 3a,b follow from Lemma A2 and A3b in CM. While the proofs for Lemmas 3c through 3g are not immediate from CM, they follow the same arguments and strategies. Since the proofs are very similar, Appendix D shows our proofs exclusively for Lemma 3c. □

Lemma 4.

$$\sum _{t=R}^{T-1}{X}_{1t}^{\prime }({\hat{\beta }}_{1t}-{\beta }_1^{*})[X_{2t}^{\prime }({\hat{\beta }}_{2t}-{\beta }_2^{*})-X_{1t}^{\prime }({\hat{\beta }}_{1t}-{\beta }_1^{*})] is o_p(1)$$

Proof Lemma 4.

See Appendix E and Appendix F. □

Lemma 5.

$$\begin{array}{l} {\displaystyle \sum _{t=R}^{T-1}}{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1})\\ = {\displaystyle \sum _{t=R}^{ T-1}}{u}_{t+1}{X}_{2t}^{\prime }[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]{\displaystyle \sum _{j=1}^{t-1}}{X}_{2j}{u}_{t+1}\\ +{\Delta }_{\mathrm{IS}}{\displaystyle \sum _{t=R}^{T-1}}{u}_{t+1}{X}_{2t}^{\prime }[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]{\displaystyle \sum _{j=1}^{t-1}}{X}_{2j}\\ +{\Delta }_{\mathrm{oos}}{\displaystyle \sum _{t=R}^{T-1}}{X}_{2t}^{\prime }[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]{\displaystyle \sum _{j=1}^{t-1}}{X}_{2j}{u}_{t+1}\\ +{\Delta }_{\mathrm{IS}}{\Delta }_{\mathrm{oos}}{\displaystyle \sum _{t=R}^{T-1}}{X}_{2t}^{\prime }[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]{\displaystyle \sum _{j=1}^{t-1}}{X}_{2j}+o_p(1)\end{array}$$

Proof of Lemma 5.

See Appendix G. □

Theorem 3.

Let $S_{uX}={\sigma }_u\sqrt{{(E{X}_{2t}{X}_{2t}^{\prime })}^{-1}}$,

(a) 

$$\begin{gathered} {\sum }_{t=R}^{T-1}{u}_{t+1}{X}_{2t}^{\prime }[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]{\sum }_{j=1}^{t-1}{X}_{2j}{u}_{t+1}\to \\ {\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)S_{uX} [{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] S_{uX}d{W}_1(s) \end{gathered}$$

(b) 

$$\begin{gathered} {\Delta }_{\mathrm{IS}}{\sum }_{t=R}^{T-1}{u}_{t+1}{X}_{2t}^{\prime }[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]{\sum }_{j=1}^{t-1}{X}_{2j}\to \\ {\Delta }_{\mathrm{IS}}{\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)\sqrt{V_{LP}(X_{2t})} [{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] S_{uX}d{W}_1(s) \end{gathered}$$

(c) 

$$\begin{gathered} {\Delta }_{\mathrm{oos}}{\sum }_{t=R}^{T-1}{X}_{2t}^{\prime }[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]{\sum }_{j=1}^{t-1}{X}_{2j}{u}_{t+1}\to \\ {\Delta }_{\mathrm{oos}}{\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)S_{uX}[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]\sqrt{V_{LP}(X_{2t})}d{W}_2(s) \end{gathered}$$

(d) 

$$\begin{gathered} {\Delta }_{\mathrm{oos}}{\Delta }_{\mathrm{IS}}{\sum }_{t=R}^{T-1}{X}_{2t}^{\prime }[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]{\sum }_{j=1}^{t-1}{X}_{2j}\to \\ {\Delta }_{\mathrm{oos}}{\Delta }_{\mathrm{IS}}{\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)\sqrt{V_{LP}(X_{2t})}[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] \sqrt{V_{LP}(X_{2t})}d{W}_2(s)+{\mathsf{\Lambda }}_{\mathsf{\lambda }} \end{gathered}$$

(e) 

$$\begin{gathered} {\sum }_{t=R}^{T-1}{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1})\to {\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)S_{uX} [{(E{X}_2{X}_2^{\prime })}^{-1}- \\ J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] S_{uX}d{W}_1(s)+{\Delta }_{\mathrm{IS}}{\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)\sqrt{V_{LP}(X_{2t})} [{(E{X}_2{X}_2^{\prime })}^{-1}- \\ J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] S_{uX}d{W}_1(s)+{\Delta }_{\mathrm{oos}}{\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)S_{uX} [{(E{X}_2{X}_2^{\prime })}^{-1}- \\ J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] \sqrt{V_{LP}(X_{2t})}d{W}_2(s)+{\Delta }_{\mathrm{oos}}{\Delta }_{\mathrm{IS}}{\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)\sqrt{V_{LP}(X_{2t})} [{(E{X}_2{X}_2^{\prime })}^{-1}- \\ J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] \sqrt{V_{LP}(X_{2t})}d{W}_2(s)+{\mathsf{\Lambda }}_{\mathsf{\lambda }}+o_p(1) \end{gathered}$$

(f) 

If there is no in-sample bias (i.e.,${\Delta }_{IS}=0$), then

$${\displaystyle \sum _{t=R}^{T-1}}{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1}){\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)S_{uX}^{\prime } [{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] S_{uX}d{W}_1(s)$$

$$+{\Delta }_{oos}{\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)S_{uX}^{\prime }[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]\sqrt{V_{LP}(X_{2t})}d{W}_2(s)$$

Proof of Theorem 3.

Theorem 3e is simply the combination of Theorems 3a through 3d. Theorem 3f is equivalent to Theorem 3e imposing ${\Delta }_{\mathrm{IS}}=0$. The proof for Theorems 3a–3d is direct from Lemmas 3 through 5. □

Three things from Theorem 3 are worth mentioning. First, the re-centering term of the MENC-NEW is relevant in the distribution because of the term ${\Delta }_{\mathrm{oos}}{\Delta }_{\mathrm{IS}}{\sum }_{t=R}^{T-1}{X}_{2t}^{\prime }[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]{\sum }_{j=1}^{t-1}{X}_{2j}$. The reason is that the correlation between $X_{2j}$ and $X_{2t}$ may not be zero; in other words, we may find a correlation between the predictor in the estimation and evaluation window. Second, if there is no in-sample bias (i.e., ${\Delta }_{\mathrm{IS}}=0$) then the re-centering term is not required (thus, we could simply use the ENC-NEW with different critical values). In essence, the term ${\Delta }_{\mathrm{oos}}{\sum }_{t=R}^{T-1}{X}_{2t}^{\prime }[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]{\sum }_{j=1}^{t-1}{u}_{t+1}{X}_{2j}^{\prime }$ does not require a re-centering term since $X_{2j}{u}_{t+1}$ (the in-sample term) and $X_{2t}$ (out-of-sample predictor) are assumed to be uncorrelated. Finally, the first term in the distribution of Thereom 3f is akin to CM if we are willing to assume that errors are not autocorrelated and homoscedastic. Notice, however, that even if that is the case, our distribution is different because of the out-of-sample bias. In contrast to CM, these additional terms depend on the long-run variance of $X_{2t}$ ($V_{LP}(X_{2t})$), on an additional Brownian motion $W_2$ and, of course, the out-of-sample bias ${\Delta }_{oos}$. Theorem 4a next establishes the asymptotic distribution of the MENC-NEW in the general case in which ${\Delta }_{IS},{\Delta }_{oos}\ne 0$. In Theorem 4b, we consider the special (although more interesting) case in which ${\Delta }_{IS}=0, {\Delta }_{oos}\ne 0$; in this case, we establish the new asymptotic distribution of the ENC-NEW, as the re-centering term is not required.

Theorem 4.

(a) 

The asymptotic distribution of our MENC-NEW$=P\frac{P^{-1}{\sum }_t{\hat{u}}_{1,t+1}({\hat{u}}_{1,t+1}-{\hat{u}}_{2,t+1})-{\hat{\Lambda }}_{\lambda }}{P^{-1}{\sum }_t{\hat{u}}_{2,t+1}^2}$is simply given by

$$MENC-NE{W}_{P,T\to \infty }(Recursive)$$

$$\begin{array}{l}\frac{1}{{\sigma }_u^2+{\Delta }_{\mathrm{oos}}^2}[{\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)S_{uX} [{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] S_{uX}d{W}_1(s)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\Delta }_{IS}{\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)\sqrt{V_{LP}(X_{2t})} [{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] S_{uX}d{W}_1(s)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\Delta }_{\mathrm{oos}}{\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)S_{uX}[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]\sqrt{V_{LP}(X_{2t})}d{W}_2(s)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\Delta }_{\mathrm{oos}}{\Delta }_{\mathrm{IS}}{\int }_{\lambda }^1{s}^{-1}{W}_2^{\prime }(s)\sqrt{V_{LP}(X_{2t})} [{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] \sqrt{V_{LP}(X_{2t})}d{W}_2(s)]\end{array}$$

$$MENC-NE{W}_{P,T\to \infty }(Rolling)\to$$

$$\begin{array}{l}\frac{{\lambda }^{-1}}{{\sigma }_u^2+{\Delta }_{\mathrm{oos}}^2}[{\int }_{\lambda }^1{(W_1(s)-W_1(s-\lambda ))}^{\prime }{S}_{uX} [{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] S_{uX}d{W}_1(s)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\Delta }_{IS}{\int }_{\lambda }^1{(W_2(s)-W_2(s-\lambda ))}^{\prime }\sqrt{V_{LP}(X_{2t})} [{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] S_{uX}d{W}_1(s)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\Delta }_{\mathrm{oos}}{\int }_{\lambda }^1{(W_1(s)-W_1(s-\lambda ))}^{\prime }{S}_{uX}[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]\sqrt{V_{LP}(X_{2t})}d{W}_2(s)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\Delta }_{\mathrm{oos}}{\Delta }_{\mathrm{IS}}{\int }_{\lambda }^1{(W_2(s)-W_2(s-\lambda ))}^{\prime }\sqrt{V_{LP}(X_{2t})} [{(E{X}_2{X}_2^{\prime })}^{-1}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] \sqrt{V_{LP}(X_{2t})}d{W}_2(s)]\end{array}$$

(b) 

If there is no in-sample bias (i.e.,${\Delta }_{IS}=0$), then the asymptotic distribution of the ENC-NEW is simply

$$ENC-NE{W}_{P,T\to \infty }(recursive)$$

$$\begin{array}{l}\frac{1}{{\sigma }_u^2+{\Delta }_{\mathrm{oos}}^2}[{\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)S_{uX} [{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] S_{uX}d{W}_1(s)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+{\Delta }_{\mathrm{oos}}{\int }_{\lambda }^1{s}^{-1}{W}_1^{\prime }(s)S_{uX}[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]\sqrt{V_{LP}(X_{2t})}d{W}_2(s)]\end{array}$$

$$ENC-NE{W}_{P,T\to \infty }(rolling)\to$$

$$\frac{{\lambda }^{-1}}{{\sigma }_u^2+{\Delta }_{\mathrm{oos}}^2}\left[{\int }_{\lambda }^1{(W_1(s)-W_1(s-\lambda ))}^{\prime }{S}_{ux} [{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] S_{uX}d{W}_1(s)\right]$$

$$+\frac{{\lambda }^{-1}{\Delta }_{\mathrm{oos}}}{{\sigma }_u^2+{\Delta }_{\mathrm{oos}}^2}\left[{\int }_{\lambda }^1{(W_1(s)-W_1(s-\lambda ))}^{\prime }{S}_{ux} [{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }] \sqrt{V_{LP}(X_{2t})}d{W}_2(s)\right]$$

Proof of Theorem 4.

The proofs of Theorem 4a,b are immediate from Theorem 3e,f and the definition of the MENC-NEW and the ENC-NEW. □

As expected from Theorem 3, even in the case of ${\Delta }_{IS}=0$, we will see that the out of sample bias shift the asymptotic distribution of the ENC-NEW because of the term ${\Delta }_{\mathrm{oos}}{\sum }_{t=R}^{T-1}{X}_{2t}^{\prime }[{(E{X}_2{X}_2^{\prime })}^{-1}-J{(E{X}_1{X}_1^{\prime })}^{-1}{J}^{\prime }]{\sum }_{j=1}^{t-1}{X}_{2j}{u}_{t+1}$. Akin to CM, the distribution of the ENC-NEW depends on how parameters are estimated (either rolling or recursive), on the number of excess parameters $k_2$, and how we split our database (through $\mathsf{\lambda }=\underset{\mathrm{P},\mathrm{T}\to \infty }{\lim }R/T$). Nevertheless, the new asymptotic distribution of the ENC-NEW depend on more nuisance parameters than CM: it depends on the magnitude of the out-of-sample bias ${\Delta }_{\mathrm{oos}}$, on the persistency of the predictors in the nesting model ($V_{LP}(X_{2t}))$, and it is necessary to simulate a second Brownian motion because of $W_2$. In Section 5 next, we show that ignoring these additional terms may severely distort the empirical size of the ENC-NEW.

5. Monte Carlo Simulations

In this section, we consider four different sets of simulations to study the size properties of our test compared to some traditional out-of-sample tests: the ENC-NEW and the ENC-t with CM critical values, and the “Wild Clark and West” (WCW) of [26] (henceforth PHM).

The data generating process (DGP) in Section 5.1 is designed to study the effects of a bias both in-sample and out of sample (e.g., the benchmark model is the DRW). The experiment in Section 5.2 is akin to Section 5.1, but this time we set our parameters to match the empirical specifications of [22,23,24] using the Chilean peso and lead prices. In these two experiments, we consider the MENC-NEW according to Theorem 2.

Section 5.3 and Section 5.4 consider two different DGPs that introduce an out-of-sample bias. In these cases, the benchmark model the random walk (with drift). As a consequence, there is no in-sample bias, and no re-centering term is required (according to Theorem 3). We use the ENC-NEW with corrected critical values in these two cases, following Theorem 4 (ENC-NEW*).

Finally, Section 5.5 studies the power properties of the MENC-NEW. To this end, we impose the alternative hypothesis over the DGP of Section 5.2.

5.1. Size Properties of the MENC-NEW: The Benchmark Model Is the DRW and $E{Y}_t\ne 0$

To illustrate the effects of a bias in the nested model, we consider a simple DGP for both the target variable $Y_t$ and the predictor $X_t$:

$$Y_{t+1}=\alpha +\beta X_t+{\epsilon }_{t+1}$$

(7)

$$X_{t+1}=\rho X_t+u_{t+1}$$

(8)

where ${\epsilon }_{t+1}$ and $u_{t+1}$ are drawn simply by independent standard normal distributions. The initial value $Y_1$ for each simulation is set at the unconditional expected value of $Y_{t+1}$ ($E{Y}_t=\alpha$). We consider two competing one-step-ahead forecasts. The first model is simply a zero-forecast (e.g., the DRW), and the second model uses exclusively $X_t$ as a predictor with no drift (i.e., $Y_t^f(1)=\gamma X_t$).

To evaluate the empirical size of each test, we set $\beta =0$. In this simple DGP, we introduce the bias through the parameter $\alpha$, and we modify the persistency of the predictor $X_{t+1}$ through $\rho$. If $\alpha =0$, then the proper critical values for this setup are those of CM. We argue, however, that severe distortions may appear because of $\alpha \ne 0$, especially for highly persistent predictors. To illustrate the severity of these size distortions, we consider different sets of values for these two parameters $\alpha$ and $\rho$. We set $\alpha$ to either 0.3, 0.5, 0.7 or 1, and $\rho$ to either 0.6 or 0.8. Additional simulations with different values of these parameters are available upon request. While this is an extremely simple DGP, it clearly illustrates the size distortions over traditional tests of out-of-sample evaluation.

We consider sample sizes of T = 100, 150, and 200. As the asymptotic distribution of most of these tests depends on how we split this sample, we consider P/R = 2 and 4, using a recursive scheme of estimation (Additional results considering P/R = 0.4, 1, and 3 are available upon request with similar conclusions.). We consider a total of 5000 Monte Carlo simulations for each exercise.

To obtain the critical values suggested by Theorem 2, we closely follow the simulations in CM. In each of the 5000 simulations, we generate 10,000 Brownian motions through simple random walks, each using an independent sequence of 10,000 i.i.d increments, drawn from a normal distribution, with a mean of 0 and a variance of 1/10,000. Then, we obtain the integrals by summing the weighted quadratic of the random walks for the entire evaluation window (from R + 1/T through T/T). In every Monte Carlo simulation, we estimate the nuisance parameters of the MENC-NEW with the synthetic sample data (such as the bias and the long-run variance of $X_t$), according to Theorem 2. Finally, the critical values are simply the 90th and 95th percentiles of our 10,000 simulated stochastic integrals for 10% and 5% significance levels, respectively.

To evaluate the empirical size of the ENC-t and ENC-NEW, we consider the critical values available in CM for the recursive scheme with the corresponding P/R and $k_2=1$. In a recent paper, PHM propose a new asymptotically normal test label as the Wild Clark and West (WCW). The authors show that parameter uncertainty in their framework is asymptotically irrelevant. The strategy of PHM is to introduce an independent random variable ${\theta }_t$ in the ENC-t core statistic that prevents degeneracy under the null hypothesis of no encompassing. PHM propose a gaussian distribution for ${\theta }_t$ (${\theta }_t~N(1,{\varphi }^2)$) such that

$$WCW-t=\sqrt{P-1}\frac{P^{-1}{\sum }_{t=R+1}^T{\hat{e}}_{1,t+1}({\hat{e}}_{1,t+1}-{\theta }_t{\hat{e}}_{2,t+1})}{\sqrt{\widehat{S_{ff}}}}$$

where ${\hat{e}}_{1,t+1}$ and ${\hat{e}}_{2,t+1}$ are the forecasting errorrs of the nested and nesting models, respectively, and $\widehat{S_{ff}}$ is a consistent estimate of the long-run variance of ${\hat{e}}_{1,t+1}({\hat{e}}_{1,t+1}-{\theta }_t{\hat{e}}_{2,t+1})$. In this case, we consider the HAC estimator proposed by Newey and West (1987,1994) [35,36]. While PHM do not provide an optimal value for the tuning parameter $\varphi$, they give some recommendations. In particular, we set $\varphi =\sqrt{V({\hat{e}}_{2,t})}\ast 0.1$, as suggested by the authors (see PHM, page 6).

Table 1. Empirical Size with P/R = 2.

	Nominal Size 10%				Nominal Size 5%
	ENC-NEW	ENC-t	WCW-t	MENC-NEW	ENC-NEW	ENC-t	WCW-t	MENC-NEW
			α = 0.3    ρ = 0.8
T = 100	0.192	0.195	0.134	0.136	0.119	0.122	0.069	0.074
T = 150	0.188	0.188	0.133	0.128	0.119	0.121	0.069	0.072
T = 200	0.187	0.184	0.114	0.123	0.119	0.116	0.061	0.065
Ave	0.189	0.189	0.127	0.129	0.119	0.120	0.066	0.070
			α = 0.5    ρ = 0.6
T = 100	0.183	0.187	0.116	0.107	0.114	0.117	0.060	0.057
T = 150	0.182	0.185	0.113	0.103	0.115	0.122	0.057	0.056
T = 200	0.182	0.181	0.105	0.100	0.116	0.112	0.053	0.050
Ave	0.182	0.184	0.111	0.103	0.115	0.117	0.057	0.054
			α = 0.7    ρ = 0.6
T = 100	0.233	0.227	0.112	0.109	0.157	0.156	0.064	0.064
T = 150	0.230	0.227	0.104	0.105	0.154	0.162	0.055	0.062
T = 200	0.230	0.220	0.101	0.097	0.154	0.148	0.047	0.051
Ave	0.231	0.225	0.106	0.104	0.155	0.155	0.055	0.059
			α = 1.0    ρ = 0.6
T = 100	0.291	0.280	0.113	0.114	0.208	0.198	0.062	0.067
T = 150	0.279	0.270	0.106	0.105	0.204	0.198	0.055	0.067
T = 200	0.276	0.261	0.098	0.096	0.201	0.186	0.050	0.053
Ave	0.282	0.270	0.106	0.105	0.204	0.194	0.056	0.062

Notes: Each entry report the percentage of rejections under the null hypothesis $\beta =0$. The table’s left (right) side reports our results with a nominal size of 10% (5%). ρ is related to the persistency of the predictor and α with the magnitude of the bias. For each exercise we consider 5000 Monte Carlo simulations. T stands for the sample size. We use a recursive scheme to update our parameters. P and R stand for the number of observations in the evaluation and estimation windows, respectively. We determine the critical values (CV) for the MENC-NEW according to Theorem 2. CV for the WCW are standard normal, while CV for the ENC-t and the ENC-NEW are those tabulated in [5] for $k_2$ = 1, P/R = 2 and a recursive method.

and

Table 2. Empirical Size for P/R = 4.

	Nominal Size 10%				Nominal Size 5%
	ENC-NEW	ENC-t	WCW-t	MENC-NEW	ENC-NEW	ENC-t	WCW-t	MENC-NEW
			α = 0.3    ρ = 0.8
T = 100	0.205	0.199	0.124	0.137	0.122	0.118	0.061	0.079
T = 150	0.201	0.195	0.120	0.131	0.121	0.122	0.064	0.078
T = 200	0.192	0.187	0.104	0.119	0.116	0.114	0.054	0.069
Ave	0.199	0.194	0.116	0.129	0.120	0.118	0.060	0.075
			α = 0.5    ρ = 0.6
T = 100	0.198	0.192	0.114	0.111	0.120	0.114	0.056	0.061
T = 150	0.193	0.193	0.106	0.105	0.121	0.121	0.054	0.059
T = 200	0.191	0.187	0.096	0.098	0.117	0.109	0.044	0.049
Ave	0.194	0.191	0.105	0.105	0.119	0.115	0.051	0.056
			α = 0.7    ρ = 0.6
T = 100	0.278	0.270	0.113	0.134	0.187	0.176	0.057	0.081
T = 150	0.276	0.261	0.107	0.126	0.193	0.183	0.056	0.077
T = 200	0.274	0.256	0.095	0.110	0.180	0.165	0.045	0.062
Ave	0.276	0.262	0.105	0.123	0.187	0.175	0.053	0.073
			α = 1.0    ρ = 0.6
T = 100	0.321	0.301	0.103	0.120	0.218	0.209	0.052	0.067
T = 150	0.302	0.288	0.095	0.114	0.220	0.206	0.052	0.064
T = 200	0.301	0.284	0.089	0.095	0.209	0.189	0.043	0.052
Ave	0.308	0.291	0.096	0.110	0.216	0.201	0.049	0.061

Notes: Each entry report the percentage of rejections under the null hypothesis $\beta =0$. The table’s left (right) side reports our results with a nominal size of 10% (5%). ρ is related to the persistency of the predictor and α with the magnitude of the bias. For each exercise we consider 5000 Monte Carlo simulations. T stands for the sample size. We use a recursive scheme to update our parameters. P and R stand for the number of observations in the evaluation and estimation windows, respectively. We determine the critical values (CV) for the MENC-NEW according to Theorem 2. CV for the WCW are standard normal, while CV for the ENC-t and the ENC-NEW are those tabulated in [5] for $k_2$= 1, P/R = 4 and a recursive method.

display our results for nominal sizes 10% and 5%, using P/R = 2, and 4 respectively. Each panel considers different parametrizations of our DGP. Three things about Table 1 and Table 2 are worth mentioning. First, consistent with Theorem 2, the ENC-t and the ENC-NEW become severely oversized with an increasing bias and higher persistency on the predictor. Notably, the last panel of Table 2 reports an empirical size about three times higher than the nominal size for both statistics. The average size of the ENC-NEW across all exercises goes from 18.2% (11.5%) to 30.8% (21.6%) with a nominal size of 10% (5%). Second, the MENC-NEW is reasonably well-sized in most of our exercises. Notice that the size properties of our test improve significantly with larger samples; this is reasonable since the asymptotic distribution of our test relies on nuisance parameters estimated in each simulation. The average size of our test goes from 10.3% (5.4%) to 12.9% (7.5%) across all exercises. Moreover, if we focus on the simulations with the larger sample T = 200, our results range from 9.5% (4.9%) to 12.3% (6.9%). Finally, while the WCW-t is generally well-sized in these exercises, we make a note of caution. While we see improvements in size compared to the ENC-t and ENC-NEW, PHM show that these improvements come with a cost: the WCW tends to exhibit less power than other tests. Our results in Section 5.5 generally support this idea: the power of the WCW is sometimes less than a half than our test.

5.2. Size Properties of the MENC-NEW with a DGP

In this section, we use the same DGP as Section 5.1, but we set the parameters in our simulation to match the econometric setup of [22,23,24] (PH). Based on the present-value model for exchange rate determination (Campbell and Shiller (1987) [37] and Engel and West (2005) [38]), PH show that the Chilean exchange rate has the ability to predict base-metal prices. Table 2 in PH suggest the following econometric specification:

$$\Delta ln(C{P}_t)=\beta [\Delta ln(E{R}_{t-1})+\Delta ln(E{R}_{t-2})]+e_t$$

where $E{R}_t$ stands for the Chilean peso and $C{P}_t$ for a generic commodity price (in this case, we use lead prices). PH use this econometric specification to determine whether this model can outperform the zero-forecast model (e.g., they evaluate the null hypothesis $H_0:\beta =0$). We argue that some distortions may appear if $E[\Delta ln(C{P}_t)]\ne 0$, as both models are biased under the null hypothesis.

Let $Y_t=\Delta ln(C{P}_t)$ and ${\mathrm{X}}_{\mathrm{t}}=[\Delta ln(E{R}_{t-1})+\Delta ln(E{R}_{t-2})]$. Using the annualized monthly returns of the Chilean peso and lead prices from September 1999 through August 2020, and using the same DGP of Section 5.1 (under the null hypothesis $\beta =0$), we estimate $\rho$, ${\sigma }_{\epsilon }^2=V({\epsilon }_{t+1})$, ${\sigma }_u^2=V(u_{t+1})$, and $\delta =Corr({\epsilon }_{t+1},u_{t+1})$:

$${\mathrm{Y}}_{\mathrm{t}}=\alpha +\beta X_t+{\epsilon }_{t+1}$$

(9)

$$X_t=0.53X_{t-1}+u_{t+1}$$

(10)

where $(\begin{array}{c}{\epsilon }_t\\ u_t\end{array})\to N(\begin{array}{cc}{\sigma }_{\epsilon }^2& \delta {\sigma }_u{\sigma }_{\epsilon }\\ \delta {\sigma }_u{\sigma }_{\epsilon }& {\sigma }_u^2\end{array})$ with $\delta =-0.36$, ${\sigma }_{\epsilon }=1.05$, and ${\sigma }_u=0.50$. To introduce the bias, we consider two different values for $\alpha$ (0.3 and 0.6). Akin to Section 5.1, we compare out-of-sample the zero-forecast against a model that uses exclusively $X_t$ ($Y_t^f(1)=\gamma X_t$). We estimate recursively by OLS the parameter $\gamma$.

Table 3. Empirical Size for P/R = 1.

	Nominal Size 10%				Nominal Size 5%
	ENC-NEW	ENC-t	WCW-t	MENC-NEW	ENC-NEW	ENC-t	WCW-t	MENC-NEW
				Bias = 0.3
T = 100	0.127	0.136	0.142	0.105	0.075	0.082	0.075	0.059
T = 150	0.120	0.134	0.132	0.100	0.074	0.081	0.073	0.054
T = 200	0.121	0.125	0.121	0.097	0.064	0.073	0.059	0.043
Ave	0.123	0.132	0.132	0.101	0.071	0.079	0.069	0.052
				Bias = 0.6
T = 100	0.177	0.186	0.133	0.105	0.113	0.120	0.065	0.061
T = 150	0.175	0.183	0.123	0.103	0.112	0.119	0.058	0.052
T = 200	0.169	0.172	0.113	0.091	0.106	0.108	0.056	0.044
Ave	0.174	0.180	0.123	0.100	0.110	0.116	0.060	0.052

Notes: Each entry report the percentage of rejections under the null hypothesis. The table’s left (right) side reports our results with a nominal size of 10% (5%). We introduce the bias in this GDP through the parameter α. For each exercise, we consider 5000 Monte Carlo simulations. T stands for the sample size. We use a recursive scheme to update our parameters. P and R stand for the number of observations in the evaluation and estimation windows, respectively. We determine the critical values (CV) for the MENC-NEW according to Theorem 2. CV for the WCW are standard normal, while CV for the ENC-t and the ENC-NEW are those tabulated in [5] for $k_2$= 1, P/R = 1 and a recursive method.

and

Table 4. Empirical Size for P/R = 2.

	Nominal Size 10%				Nominal Size 5%
	ENC-NEW	ENC-t	WCW-t	MENC-NEW	ENC-NEW	ENC-t	WCW-t	MENC-NEW
				Bias = 0.3
T = 100	0.120	0.127	0.128	0.098	0.069	0.073	0.070	0.056
T = 150	0.127	0.134	0.129	0.101	0.069	0.079	0.060	0.051
T = 200	0.113	0.121	0.121	0.090	0.062	0.069	0.061	0.043
Ave	0.120	0.127	0.126	0.096	0.067	0.074	0.064	0.050
				Bias = 0.6
T = 100	0.174	0.175	0.120	0.097	0.105	0.114	0.058	0.057
T = 150	0.180	0.183	0.112	0.100	0.115	0.124	0.058	0.054
T = 200	0.176	0.177	0.101	0.085	0.101	0.104	0.049	0.045
Ave	0.177	0.178	0.111	0.094	0.107	0.114	0.055	0.052

Notes: Each entry report the percentage of rejections under the null hypothesis $\beta =0$. The table’s left (right) side reports our results with a nominal size of 10% (5%). We introduce the bias in this GDP through the parameter α. For each exercise, we consider 5000 Monte Carlo simulations. T stands for the sample size. We use a recursive scheme to update our parameters. P and R stand for the number of observations in the evaluation and estimation windows, respectively. We determine the critical values (CV) for the MENC-NEW according to Theorem 2. CV for the WCW are standard normal, while CV for the ENC-t and the ENC-NEW are those tabulated in [5] for $k_2$= 1, P/R = 2 and a recursive method.

exhibit our results for P/R = 1 and 2, respectively. Additional exercises are available upon request. Notice that the only difference in each panel is the magnitude of the bias.

The main conclusions of Table 3 and Table 4 are similar to those in Table 1 and Table 2. First, consistent with Theorem 2, the ENC-t and ENC-NEW exhibit substantial size distortions when the bias increases. The average size of both tests is ~12–13% (7%) when $\alpha =0.3$. Nevertheless, when the bias increases ($\alpha =0.6)$, the average size of both tests is ~17–18% (11%). Notably, the MENC-NEW seems to be reasonably well-sized in all our exercises. Specifically, the empirical size of our test ranges between 8.5 and 10.5% (4.3–6.1%), with an average empirical size across all our exercises of 9.8% (5.1%). Akin to the previous section, the size of our test seems to improve with the sample size; this is reasonable since we need to estimate some nuisance parameters to determine our critical values. Finally, the WCW exhibit mixed results compared to the ENC-t and ENC-NEW. In particular, the WCW is less oversized for $\alpha =0.6$, but it tends to underperform for $\alpha =0.3$.

5.3. Size Properties of the ENC-NEW with Adjusted Critical Values: The Effects of an Out-of-Sample Bias

In this DGP we introduce an out-of-sample bias by shifting the expected value of the target variable $Y_t$. The key point is that this shift happens in the evaluation window $t>R$. As the parameters are recursively estimated in each period, the OLS estimator will not fully accommodate this shift in small samples. As a consequence, we will observe an out-of-sample bias in both models under the null hypothesis. In contrast to Section 5.1, there is no need to re-center the ENC-NEW, as suggested by Theorem 3. Nevertheless, the asymptotic distribution of the ENC-NEW is now given by Theorem 4b, and it is necessary to obtain new critical values.

To illustrate the effects of this bias, we introduce this shift immediately after the estimation window at $t=R+1$ through a change in the drift. Let ${\alpha }_1$ and ${\alpha }_2$ be the drifts before and after the shift, respectively:

$$Y_t={\alpha }_1+\beta X_{t-1}+{\epsilon }_t, t\le R$$

$$Y_t={\alpha }_2+\beta X_{t-1}+{\epsilon }_t, t>R+1$$

where ${\epsilon }_t$ is an independent standard normal random variable and $X_t$ is a stationary AR(1) process

$$X_t={\alpha }_x+\rho X_{t-1}+v_t,\forall t$$

We consider two competing models for the target variable $Y_t$

$$Y_t=c_1+v_{1,t} (M1)$$

$$Y_t=c_2+\gamma X_{t-1}+v_{2,t} (M2)$$

where $c_1$, $c_2$, and $\gamma$ are regression parameters, and $v_{1,t}$,$v_{2,t}$ stand for the forecast errors. Notice that M1 forecasts $Y_t$ using its historical sample mean, and it is identical to M2 whenever $\gamma =0$. As the parameters in M1 and M2 are estimated in-sample in each recursive window by OLS, we should expect no in-sample bias whatsoever. Nevertheless, the OLS estimator will not fully capture the out-of-sample change on the drift in small samples. According to Theorem 3, if there is no in-sample bias, then the re-centering term of the MENC-NEW is unnecessary. In this case, we may simply consider the ENC-NEW with corrected critical values (as suggested by the asymptotic distribution in Theorem 4b). We consider different parametrizations for this DGP. In particular, we allow different levels of persistency on the predictor $X_t$ (ρ = 0.4 and 0.8), and different changes on the drift parameter (${\alpha }_1$ = 0 ${\alpha }_2$ = 0.3, ${\alpha }_1$ = 0 ${\alpha }_2$ = 0.3, and ${\alpha }_1$ = 0 ${\alpha }_2$ = 0.3). The idea of using different changes on the drift is to manage the magnitude of the out-of-sample bias.

Similar to Section 5.1 and Section 5.2, we consider 5000 Monte Carlo simulations using a recursive scheme with P/R = 1 and T = 100, 150, and 200. To evaluate the empirical size of each test, we set $\beta =0$. To simulate the corrected critical values of the ENC-NEW, we consider the asymptotic distribution suggested by Theorem 4b. Akin to the previous section, in each of the 5000 simulations, we generate 10,000 Brownian motions, each using an independent sequence of 10,000 i.i.d increments, drawn from a normal distribution, with mean 0 and variance 1/10,000. Then, we obtain the integrals by summing the weighted quadratic of the random walks for the entire evaluation window (from R + 1/T through T/T). We estimate the respective nuisance parameters in each simulation using the simulated data, according to Theorem 4b. The critical values are the 90th and 95th percentiles of our 10,000 simulated stochastic integrals for 10% and 5% significance levels, respectively.

Each panel in

Table 5. Empirical Size with an out-of-sample bias and P/R = 1.

	Nominal Size 10%				Nominal Size 5%
	ENC-NEW	ENC-t	WCW-t	ENC-NEW*	ENC-NEW	ENC-t	WCW-t	ENC-NEW*
			$\rho =0.8\hspace{1em}{\alpha }_1$$=0\hspace{1em}{\alpha }_2$$=0.3\hspace{1em}{\alpha }_X$ = 0.2
T = 100	0.142	0.145	0.141	0.116	0.086	0.088	0.077	0.066
T = 150	0.146	0.156	0.136	0.115	0.086	0.093	0.074	0.060
T = 200	0.141	0.145	0.121	0.113	0.088	0.088	0.067	0.058
Ave	0.143	0.149	0.133	0.115	0.087	0.090	0.073	0.061
			$\rho =0.8\hspace{1em}{\alpha }_1$$=0.1\hspace{1em}{\alpha }_2$$=0.3\hspace{1em}{\alpha }_X$ = 0.2
T = 100	0.130	0.126	0.141	0.104	0.073	0.072	0.078	0.054
T = 150	0.128	0.135	0.135	0.099	0.073	0.076	0.072	0.048
T = 200	0.125	0.129	0.127	0.098	0.073	0.072	0.072	0.049
Ave	0.128	0.130	0.134	0.100	0.073	0.073	0.074	0.050
			$\rho =0.4\hspace{1em}{\alpha }_1$$=0.2\hspace{1em}{\alpha }_2$$=-0.3\hspace{1em}{\alpha }_X$ = 0.4
T = 100	0.120	0.136	0.137	0.094	0.069	0.077	0.074	0.048
T = 150	0.112	0.126	0.132	0.080	0.060	0.071	0.071	0.040
T = 200	0.121	0.131	0.128	0.090	0.068	0.071	0.069	0.045
Ave	0.118	0.131	0.132	0.088	0.066	0.073	0.071	0.044

Notes: Each entry report the percentage of rejections under the null hypothesis $\beta =0$. The table’s left (right) side reports our results with a nominal size of 10% (5%). We introduce the out-of-sample bias in this GDP through the parameters ${\alpha }_1$(the initial drift of $Y_t)$ and ${\alpha }_2$ (the drift of $Y_t$ after the change). The parameter $\rho$ is related to the persistency of the predictor $X_t$. ${\alpha }_X$ denotes the drift of the predictor. For each exercise, we consider 5000 Monte Carlo simulations. T stands for the sample size. We use a recursive scheme to update our parameters. P and R stand for the number of observations in the evaluation and estimation windows, respectively. We determine the corrected critical values (CV) for the ENC-NEW according to Theorem 2 (ENC-NEW*). CV for the WCW are standard normal, while CV for the ENC-t and the ENC-NEW are those tabulated in [5] for $k_2$= 1, P/R = 1 and a recursive method.

exhibits our results using different parameterizations of our DGP. Three features in Table 5 are worth mentioning. First, the ENC-NEW with corrected critical values (ENC-NEW*) is generally well-sized in all exercises. In particular, the average size of the ENC-NEW* goes from 8.8% (4.4%) to 11.5% (6.1%), where the most oversized exercise exhibits a rejection of 11.6% (6.6%), and the average size across all exercises is 10.0% (5.16%). Second, the ENC-NEW and the ENC-t are again significantly more oversized. For instance, the average size of the ENC-t goes from 13.0% (7.3%) to 14.9% (9.0%), with an average size across all exercises of 13.7% (7.87%). Finally, consistent with Theorem 4, the worsts results for the ENC-NEW and ENC-t are those in simulations with a stronger persistency in the predictor and a bigger positive shift in the drift (first panel). In these exercises, the ENC-t ranges from 14.5% (8.8%) through 15.6% (9.3%), with an average size of 14.9% (9.0%). In sharp contrast, the ENC-NEW with corrected critical values has an empirical size ranging from 11.3% (5.0%) to 11.6% (6.6%), with an average size of 11.5% (6.1%).

5.4. Size Properties of the ENC-NEW with Adjusted Critical Values: Persistency in the Drift

In this DGP, we introduce an out-of-sample bias by drifting the expected value of the target variable $Y_t$. The key point is that the drift in $Y_t$ follows a persistent, although stationary, AR(1) process. As the parameters are recursively estimated in each period, the OLS estimator will not be able to accommodate this shift in small samples, and as a consequence, we will observe an out-of-sample bias in both models under the null hypothesis. In contrast to Section 5.1, there is no in-sample bias, and there is no need to re-center the ENC-NEW, as suggested by Theorem 3. Nevertheless, the asymptotic distribution of the ENC-NEW is now given by Theorem 4b.

Consider the following DGP for the target variable $Y_t$ and the predictor $X_t$

$$Y_t={\alpha }_{y,t}+\beta X_{t-1}+u_t$$

$$X_t={\alpha }_x+\rho X_{t-1}+{\epsilon }_t$$

where $u_t$ and ${\epsilon }_t$ are independent standard normal random variables. We consider two competing models

$$Y_t=c_1+v_{1,t} (M1)$$

$$Y_t=c_2+\gamma X_{t-1}+v_{2,t} (M2)$$

where $c_1$, $c_2$, and $\gamma$ are simply regression parameters, and $v_{1,t}$,$v_{2,t}$ stand for the forecast errors. Notice that M1 forecasts $Y_t$ using its historical sample mean, and it is identical to M2 whenever $\gamma =0$. The interesting thing about this DGP is that we allow the drift of $Y_t$ to follow a persistent (although stationary) AR(1) process

$${\alpha }_{y,t}={\alpha }_{\alpha }+{\rho }_{\alpha }{\alpha }_{y,t-1}+e_t$$

where $e_t$ is an independent standard normal random variable. As the parameters in M1 and M2 are estimated in-sample in each recursive window by OLS, we should expect no in-sample bias whatsoever. Nevertheless, the OLS estimator will not capture the out-of-sample change on the drift. As the drift is persistently changing in each period, it may introduce an out-of-sample bias. According to Theorem 4b, if there is no in-sample bias, then the re-centering term of the MENC-NEW is unnecessary. In this case, we may simply consider the ENC-NEW with corrected critical values (as suggested by the asymptotic distribution in Theorem 4b).

Table 6. Empirical size for P/R = 1.

	Nominal Size 10%				Nominal Size 5%
	ENC-NEW	ENC-t	WCW-t	ENC-NEW*	ENC-NEW	ENC-t	WCW-t	ENC-NEW*
			$\rho =0.4\hspace{1em}{\rho }_{\alpha }$$=0.9\hspace{1em}{\alpha }_{\alpha }$$=0.01\hspace{1em}{\alpha }_x$ = 0.3
T = 100	0.219	0.235	0.120	0.125	0.145	0.157	0.060	0.065
T = 150	0.207	0.220	0.112	0.106	0.137	0.141	0.056	0.049
T = 200	0.215	0.218	0.107	0.104	0.149	0.151	0.058	0.044
Ave	0.214	0.224	0.113	0.112	0.144	0.150	0.058	0.053
			$\rho =0.5\hspace{1em}{\rho }_{\alpha }$$=0.95\hspace{1em}{\alpha }_{\alpha }$$=0.01\hspace{1em}{\alpha }_x$ = 0.3
T = 100	0.264	0.276	0.128	0.131	0.189	0.204	0.063	0.072
T = 150	0.260	0.271	0.108	0.110	0.186	0.193	0.054	0.053
T = 200	0.270	0.274	0.115	0.105	0.197	0.201	0.058	0.052
Ave	0.265	0.274	0.117	0.115	0.191	0.199	0.058	0.059
			$\rho =0.6\hspace{1em}{\rho }_{\alpha }$$=0.9\hspace{1em}{\alpha }_{\alpha }$$=0.05\hspace{1em}{\alpha }_x$ = 0.4
T = 100	0.303	0.302	0.131	0.115	0.222	0.229	0.068	0.064
T = 150	0.289	0.292	0.113	0.090	0.215	0.214	0.060	0.042
T = 200	0.300	0.294	0.108	0.085	0.225	0.221	0.057	0.040
Ave	0.297	0.296	0.117	0.097	0.221	0.221	0.062	0.049

Notes: Each entry report the percentage of rejections under the null hypothesis $\beta =0$. The table’s left (right) side reports our results with a nominal size of 10% (5%). ${\rho }_{\alpha }$ and ${\alpha }_{\alpha }$ denote the persistency and the drift of ${\alpha }_{y,t}$, respectively. The parameter $\rho$ is related to the persistency of the predictor $X_t$. ${\alpha }_X$ denotes the drift of the predictor. For each exercise, we consider 5000 Monte Carlo simulations. T stands for the sample size. We use a recursive scheme to update our parameters. P and R stand for the number of observations in the evaluation and estimation windows, respectively. We determine the corrected critical values (CV) for the ENC-NEW according to Theorem 2 (ENC-NEW*). CV for the WCW are standard normal, while CV for the ENC-t and the ENC-NEW are those tabulated in [5] for $k_{2 }$= 1, P/R = 1 and a recursive method.

and

Table 7. Empirical Size for P/R = 2.

	Nominal Size 10%				Nominal Size 5%
	ENC-NEW	ENC-t	WCW-t	ENC-NEW*	ENC-NEW	ENC-t	WCW-t	ENC-NEW*
			$\rho =0.4\hspace{1em}{\rho }_{\alpha }$$=0.9\hspace{1em}{\alpha }_{\alpha }$$=0.01\hspace{1em}{\alpha }_x$ = 0.3
T = 100	0.216	0.236	0.111	0.121	0.145	0.156	0.053	0.063
T = 150	0.210	0.219	0.096	0.104	0.140	0.145	0.047	0.050
T = 200	0.220	0.218	0.096	0.099	0.148	0.155	0.048	0.042
Ave	0.215	0.224	0.101	0.108	0.144	0.152	0.049	0.052
			$\rho =0.5\hspace{1em}{\rho }_{\alpha }$$=0.95\hspace{1em}{\alpha }_{\alpha }$$=0.01\hspace{1em}{\alpha }_x$ = 0.3
T = 100	0.273	0.291	0.107	0.126	0.192	0.208	0.056	0.064
T = 150	0.272	0.285	0.092	0.103	0.186	0.203	0.046	0.050
T = 200	0.278	0.286	0.096	0.099	0.197	0.207	0.047	0.047
Ave	0.274	0.287	0.098	0.109	0.192	0.206	0.050	0.054
			$\rho =0.6\hspace{1em}{\rho }_{\alpha }$$=0.9\hspace{1em}{\alpha }_{\alpha }$$=0.05\hspace{1em}{\alpha }_x$ = 0.4
T = 100	0.316	0.319	0.116	0.115	0.226	0.239	0.062	0.059
T = 150	0.309	0.314	0.101	0.086	0.225	0.227	0.052	0.040
T = 200	0.310	0.303	0.091	0.073	0.234	0.232	0.049	0.030
Ave	0.312	0.312	0.103	0.091	0.228	0.233	0.054	0.043

Notes: Each entry report the percentage of rejections under the null hypothesis $\beta =0$. The table’s left (right) side reports our results with a nominal size of 10% (5%). ${\rho }_{\alpha }$ and ${\alpha }_{\alpha }$ denote the persistency and the drift of ${\alpha }_{y,t}$, respectively. The parameter $\rho$ is related to the persistency of the predictor $X_t$. ${\alpha }_X$ denotes the drift of the predictor. For each exercise, we consider 5000 Monte Carlo simulations. T stands for the sample size. We use a recursive scheme to update our parameters. P and R stand for the number of observations in the evaluation and estimation windows, respectively. We determine the corrected critical values (CV) for the ENC-NEW according to Theorem 2 (ENC-NEW*). CV for the WCW are standard normal, while CV for the ENC-t and the ENC-NEW are those tabulated in [5] for $k_2$= 1, P/R = 2 and a recursive method.

report our results for P/R = 1 and 2, respectively (Additional results are available upon request.). Consistent with our previous simulations, the ENC-NEW with corrected critical values is reasonably well-sized, especially with large sample sizes. The size of the ENC-NEW* ranges between 7.3% (3.0%) to 13.1% (7.2%), with an average size across all exercises of 10.5% (5.2%). In sharp contrast, the ENC-NEW and the ENC-t are severely oversized in all our simulations: some exercises exhibit an empirical size four times bigger than its nominal size. For instance, the last panel of Table 7 shows an average empirical size of 31.2% (22.8%) and 31.2% (23.3%) for the ENC-NEW and ENC-t, respectively. Moreover, the average size of the ENC-NEW across all exercises is 26.3% (18.6%), more than two (three) times its nominal size. Finally, the WCW-t seems to be correctly sized for most of our exercises, with an empirical size ranging from 9.1% (4.6%) to 13.1% (6.8%).

5.5. Power Properties of the MENC-NEW

In this section, we provide some simulations evaluating the power properties of the MENC-NEW compared to the ENC-NEW, ENC-t, and WCW-t. We use the same DGP of Section 5.2, but this time we impose the alternative hypothesis setting $\beta \ne 0$.

Table 8. Raw power with nominal sizes 10% and 5%, and P/R = 1.

	Nominal Size 10%				Nominal Size 5%
	ENC-NEW	ENC-t	WCW-t	MENC-NEW	ENC-NEW	ENC-t	WCW-t	MENC-NEW
				Bias = 0.15
T = 100	0.925	0.883	0.419	0.923	0.878	0.801	0.364	0.871
T = 150	0.985	0.963	0.467	0.984	0.969	0.919	0.440	0.967
T = 200	0.997	0.991	0.501	0.997	0.994	0.976	0.489	0.993
Ave	0.969	0.946	0.462	0.968	0.947	0.899	0.431	0.944
				Bias = 0.3
T = 100	0.905	0.856	0.395	0.885	0.848	0.765	0.338	0.823
T = 150	0.970	0.945	0.453	0.963	0.947	0.890	0.417	0.933
T = 200	0.993	0.984	0.493	0.991	0.988	0.955	0.471	0.982
Ave	0.956	0.928	0.447	0.947	0.928	0.870	0.409	0.913
				Bias = 0.6
T = 100	0.807	0.751	0.330	0.738	0.739	0.659	0.267	0.630
T = 150	0.909	0.873	0.395	0.861	0.868	0.802	0.343	0.792
T = 200	0.962	0.931	0.450	0.928	0.936	0.885	0.407	0.892
Ave	0.892	0.851	0.392	0.842	0.847	0.782	0.339	0.771

Notes: Each entry report the percentage of rejections under the alternative hypothesis $\beta =0.6$. The table’s left (right) side reports our results with a nominal size of 10% (5%). We introduce the bias in this GDP through the parameter α. For each exercise, we consider 5000 Monte Carlo simulations. T stands for the sample size. We use a recursive scheme to update our parameters. P and R stand for the number of observations in the evaluation and estimation windows, respectively. We determine the critical values (CV) for the MENC-NEW according to Theorem 2. CV for the WCW are standard normal, while CV for the ENC-t and the ENC-NEW are those tabulated in [5] for $k_{2 }$= 1, P/R = 1 and a recursive method.

and

Table 9. Raw power with nominal sizes 10% and 5%, and P/R = 2.

	Nominal Size 10%				Nominal Size 5%
	ENC-NEW	ENC-t	WCW-t	MENC-NEW	ENC-NEW	ENC-t	WCW-t	MENC-NEW
				Bias = 0.15
T = 100	0.938	0.929	0.450	0.934	0.896	0.870	0.409	0.893
T = 150	0.985	0.980	0.485	0.985	0.973	0.961	0.465	0.971
T = 200	0.997	0.997	0.507	0.996	0.994	0.989	0.501	0.994
Ave	0.973	0.969	0.481	0.972	0.954	0.940	0.458	0.953
				Bias = 0.3
T = 100	0.914	0.906	0.432	0.900	0.865	0.835	0.385	0.837
T = 150	0.976	0.968	0.473	0.969	0.954	0.937	0.440	0.941
T = 200	0.995	0.992	0.502	0.994	0.989	0.978	0.488	0.982
Ave	0.962	0.955	0.469	0.954	0.936	0.917	0.438	0.920
				Bias = 0.6
T = 100	0.830	0.814	0.364	0.748	0.757	0.728	0.303	0.655
T = 150	0.918	0.905	0.416	0.864	0.875	0.847	0.373	0.795
T = 200	0.969	0.958	0.468	0.938	0.946	0.925	0.438	0.902
Ave	0.906	0.892	0.416	0.850	0.859	0.833	0.372	0.784

Notes: Each entry report the percentage of rejections under the alternative hypothesis $\beta =0.6$. The table’s left (right) side reports our results with a nominal size of 10% (5%). We introduce the bias in this GDP through the parameter α. For each exercise, we consider 5000 Monte Carlo simulations. T stands for the sample size. We use a recursive scheme to update our parameters. P and R stand for the number of observations in the evaluation and estimation windows, respectively. We determine the critical values (CV) for the MENC-NEW according to Theorem 2. CV for the WCW are standard normal, while CV for the ENC-t and the ENC-NEW are those tabulated in [5] for $k_2$= 1, P/R = 2 and a recursive method.

next exhibit our power results introducing a bias of $\alpha =0.15, 0.30,$ and 0.60, considering P/R = 1 and 2, respectively. In both tables, we impose the alternative by setting $\beta =0.6$. Additional simulations using different values for $\beta$ deliver a similar message, and they are available upon request.

First, consistent with [26] (PHM), the size improvements of the WCW-t come with a cost: a deterioration in power. As commented by PHM: “In terms of power, results have been rather mixed, although CW has frequently exhibited some more power. All in all, our simulations reveal that asymptotic normality and size corrections come with a cost: the introduction of a random variable erodes some of the power of WCW.” [26], page 3. In particular, the power of WCW-t ranges from 33.0% (26.7%) to 50.7% (50.1%), with an average power of 44.5% (40.7%). Second, the ENC-NEW, ENC-t, and MENC-NEW exhibit significantly more power than WCW-t, generally more than double. For instance, the average power of the MENC-NEW is 92.2% (88%). Finally, while the power of the ENC-NEW and the MENC-NEW are very similar, we do observe a small edge on the ENC-NEW. This is, of course, reasonable as the ENC-NEW and ENC-t are extremely oversized in this environment, according to the results of Section 5.2. Not surprisingly, the differences in power become more important with a higher bias (e.g., last panel): the advantage of the ENC-NEW in terms of power coincides with the ENC-NEW being particularly more oversized. We see the results of Table 8 and Table 9 encouraging since the differences in power are, most of the time, neglectable, even though the size of the ENC-NEW is sometimes more than double than the nominal size.

6. Concluding Remarks

In this paper, we show analytically and via simulations that a bias in the null model may severely distort the asymptotic distribution of some traditional tests of forecast evaluation in nested models comparisons. These distortions are more severe with an increasing bias and a higher persistency on the additional predictors. We address two relevant cases: (i) The presence of a bias in-sample and out-of-sample (e.g., the driftless random walk as the benchmark model). (ii) The presence of a bias exclusively out-of-sample (e.g., a shift in the expected value of the target variable). To deal with the former case, we consider a simple modification of the ENC-NEW (MENC-NEW) robust to a bias in the null model. In essence, the MENC-NEW introduces a re-centering term in the ENC-NEW that appears because of the additional predictors’ bias and persistency. In both cases, the relevant asymptotic distribution is not pivotal; akin to CM, they are functionals of stochastic integrals, but they also depend on the magnitude of the bias and the persistency of the predictors.

Based on CM, we derive the new distribution for the ENC-NEW imposing a bias in the null model (either out-of-sample or both in-sample and out-of-sample). While this is a subtle change in the set of assumptions, it has important implications over the asymptotic theory. In particular, the quadratic Brownian motions in CM arise because of the martingale difference terms: put simply, the orthogonality condition is assumed to hold both in-sample and out-of-sample. In contrast, in our case, the quadratic Brownian motions arise because of both the predictors and the martingale difference terms. Moreover, if the bias appears both in-sample and out-of-sample, the persistency of the predictors shift the expected value of the integrals of quadratic Brownian motions, thus a re-centering is required: our MENC-NEW is simply a re-centered version of the ENC-NEW. Of course, in the absence of a bias, our MENC-NEW reduces to the ENC-NEW. As expected, we show that this new asymptotic distribution depends on the magnitude of the bias and the persistency of the predictors. Even though the asymptotic distribution of the MENC-NEW (and the ENC-NEW with adjusted critical values) is not pivotal, the nuisance parameters can be easily estimated (one of these nuisance parameters being, of course, the magnitude of the bias).

Our Monte Carlo simulations reveal that our MENC-NEW (and the ENC-NEW with adjusted critical values) is reasonably well-sized even when the ENC-NEW and ENC-t (with CM critical values) exhibits rejections three to four times higher than the nominal size. The severity of the size distortion is, of course, related to the magnitude of the bias, as demonstrated by our decompositions.

Our results are important since the “correct specification” assumption in CM is often overlooked in empirical works. We show that ignoring this type of misspecification (bias) may severely change the asymptotic distribution of traditional tests. We suggest four interesting avenues for future research: First, it is important to study the effects of other cases of misspecification. For instance, a usual finding in the empirical literature is that forecasts are often auto-inefficient, in the sense that $Corr(X_t,u_{t+1})\ne 0$. This type of misspecification may also affect the asymptotic distribution of out-of-sample tests and, consequently, distort the size and power properties. Second, in our view, the main caveat of our approach is that the asymptotic distribution of our test is not pivotal. In this sense, the empirical researcher must simulate its critical values to conduct inference correctly. An interesting contribution may be the development of asymptotically normal tests, or at least an approach free of nuisance parameters, in order to simplify the forecast evaluation. Third, the extension of our approach to a direct multi-step-ahead framework, along the lines of [11]. Fourth, akin to CM, all our derivations are based on non-linear least squares (where the least square estimator is, of course, a special case). It may be interesting to study how the asymptotic distribution of these tests is affected by different estimation methods (e.g., ridge regressions).