Analytical Approximation of the Jackknife Linking Error in Item Response Models Utilizing a Taylor Expansion of the Log-Likelihood Function

Abstract

Linking errors in item response models quantify the dependence on the chosen items in means, standard deviations, or other distribution parameters. The jackknife approach is frequently employed in the computation of the linking error. However, this jackknife linking error could be computationally tedious if many items were involved. In this article, we provide an analytical approximation of the jackknife linking error. The newly proposed approach turns out to be computationally much less demanding. Moreover, the new linking error approach performed satisfactorily for datasets with at least 20 items.

1. Introduction

Item response theory (IRT) models [1,2,3] are an important class of multivariate statistics methodologies for analyzing dichotomous random variables used to model testing data from educational or psychological applications. This class aims to summarize a high-dimensional contingency table by a few latent factor variables of interest. Of particular relevance is the application of item response models in educational large-scale assessment [4], such as the studies programme for international student assessment (PISA; [5]) or progress in international reading literacy study (PIRLS; [6]).

In this article, only unidimensional IRT models are considered. Let $\mathit{X}=(X_1,\dots ,X_I)$ be the vector of I dichotomous random variables $X_i\in \{0,1\}$ (also referred to as items). A unidimensional item response model [1,7] is a statistical model for the probability distribution $\mathrm{P}(\mathit{X}=\mathit{x})$ for $\mathit{x}=(x_1,\dots ,x_I)\in {\{0,1\}}^I$, where

$${\displaystyle \mathrm{P}(\mathit{X}=\mathit{x};\mathit{\delta },\mathit{\gamma })={\int }_{-\infty }^{\infty }\prod _{i=1}^I\left[{P_i(\theta ;{\mathit{\gamma }}_i)}^{x_i}{\left(1-P_i(\theta ;{\mathit{\gamma }}_i)\right)}^{1-x_i}\right]\varphi (\theta ;\mu ,\sigma )\mathrm{d}\theta ,}$$

(1)

where $\varphi$ is the density of the normal distribution with mean $\mu$ and standard deviation $\sigma$. The vector $\mathbf{\delta }=(\mu ,\sigma )$ contains the distribution parameters. The vector $\mathbf{\gamma }=({\mathbf{\gamma }}_1,\dots ,{\mathbf{\gamma }}_I)$ contains all estimated item parameters of item response functions $P_i(\theta ;{\mathbf{\gamma }}_i)=\mathrm{P}\left(X_i=1\right|\theta )$.

The one-parameter logistic (1PL) model (also referred to as the Rasch model; [8]) uses the item response function $P_i\left(\theta \right)=\mathsf{\Psi }(\theta -b_i)$, where $\mathsf{\Psi }$ denotes the logistic distribution function, and $b_i$ is the item difficulty of item i. In this case, the vector of item parameters ${\mathbf{\gamma }}_i$ only consists of one entry; that is, ${\mathbf{\gamma }}_i=\left(b_i\right)$. The two-parameter logistic (2PL) model [9] includes the item discrimination $a_i$ in addition (i.e., ${\mathbf{\gamma }}_i=(a_i,b_i)$), and the item response function is given by $P_i\left(\theta \right)=\mathsf{\Psi }\left(a_i(\theta -b_i)\right)$.

Please note that distribution parameters $\mathbf{\delta }$ and item parameters $\mathbf{\gamma }$ cannot be simultaneously identified. If the parameters $(\mu ,\sigma ,\left\{(a_i,b_i)\right|i\in \{1,\dots ,I\}\})$ parametrize the 2PL model, an equivalent parametrization would be $(\mu =0,\sigma =1,\left\{(a_i\sigma ,{\sigma }^{-1}(b_i-\mu ))\right|i\in \{1,\dots ,I\}\})$. In applications like PISA in which a country mean $\mu$ and country a standard deviation $\sigma$, item parameters ${\mathbf{\gamma }}_i$ are often fixed at values ${\mathbf{\gamma }}_i^{*}$ that are used for all countries. In this case, $\mu$ and $\sigma$ can be identified. If sample data ${\mathit{x}}_1,\dots ,{\mathit{x}}_N$ for N persons are available, unknown model parameters in (1) can be estimated by (marginal) maximum likelihood (ML) using an expectation-maximization algorithm [10,11].

In practice, data-generating item parameters ${\mathbf{\gamma }}_i$ differ from assumed fixed item parameters ${\mathbf{\gamma }}_i^{*}$. This property is also referred to as differential item functioning (DIF; [12]). DIF effects ${\mathit{e}}_i$ are defined as deviations ${\mathit{e}}_i={\mathbf{\gamma }}_i-{\mathbf{\gamma }}_i^{*}$. The occurrence of DIF causes additional variability in the estimated (country) mean $\mu$ and standard deviation $\sigma$ [13,14]. Consequently, the estimated distribution parameters depend on the choice of selected items, even in infinite sample sizes of persons. This variability is quantified in the linking error [15,16,17,18,19,20]. There exist simple formulas for linking errors based on variance components for the 1PL model [16,18]. For more complex models, resampling techniques [21,22] such as jackknife [16,18] or (balanced) half sampling [19] of items can be employed. In the computation of the jackknife linking error, the model is repeatedly estimated by excluding a single item i at each item resulting in slightly differing estimates ${\widehat{\mu }}_{(-i)}$ and ${\widehat{\sigma }}_{(-i)}$ compared to the estimates $\widehat{\mu }$ and $\widehat{\sigma }$ in the full sample of items. The jackknife linking error for the estimated mean $\widehat{\mu }$ is defined as

$$\begin{array}{c}\hfill \mathrm{LE}\left(\widehat{\mu }\right)=\sqrt{\frac{I-1}{I}{\displaystyle \sum _{i=1}^I}{({\widehat{\mu }}_{(-i)}-\widehat{\mu })}^2}.\end{array}$$

(2)

The disadvantage of the linking error formula (2) is that $I+1$ model estimations of the IRT model based on the log-likelihood function l are required. In this article, a computational shortcut for determining increments ${\widehat{\mu }}_{(-i)}-\widehat{\mu }$ in (2) based on a Taylor expansion of the log-likelihood function is presented. Only second-order derivatives and one additional estimation of the IRT model are required in our proposed approach. Hence, the computational effort is significantly reduced.

The rest of the article is structured as follows. The newly proposed analytical approximation to the jackknife linking error is presented in Section 2. A simulation study compares the performance of our new approach with the jackknife linking error in Section 3. Finally, the article closes with a discussion in Section 4.

2. Analytical Approximation of the Jackknife Linking Error

This section provides details for our analytical approximation to the jackknife linking error. A Taylor expansion of the log-likelihood function l is employed to approximate increments in the jackknife linking error formula.

Let $\mathbf{\delta }=(\mu ,\sigma )$ be the vector that includes the mean $\mu$ and the standard deviation $\sigma$. Let $\mathbf{\gamma }=({\mathbf{\gamma }}_1,\dots ,{\mathbf{\gamma }}_I)$ be the vector that includes all item parameters ${\mathbf{\gamma }}_i$ ($i=1,\dots ,I$). Furthermore, let ${\mathbf{\delta }}_0$ and ${\mathbf{\gamma }}_0$ be the true distribution parameter and item parameters, respectively. In the computation of $\widehat{\mathbf{\gamma }}$, the item parameters in the scaling model to $\mathbf{\gamma }={\mathbf{\gamma }}^{*}$ are fixed. The difference $\mathit{e}={\mathbf{\gamma }}_0-{\mathbf{\gamma }}^{*}$ indicates misspecification. If the scaling model involves data of a country and ${\mathbf{\gamma }}^{*}$ are international item parameters, the vector $\mathit{e}$ includes DIF effects.

The approximation of the jackknife linking error relies on a Taylor expansion of the first derivative of the log-likelihood function l with respect to $\mathbf{\delta }$ (i.e., the score equations) around true data-generating parameters $({\mathbf{\delta }}_0,{\mathbf{\gamma }}_0)$. In the application of IRT models, the log-likelihood function is typically twice continuously differentiable to guarantee the applicability of the Taylor approximation. Define $l_{\mathbf{\delta }}=(\partial l)/(\partial \mathbf{\delta })$, $l_{\mathbf{\delta }\mathbf{\delta }}=\left({\partial }^{2}l\right)/(\partial {\mathbf{\delta }}^2)$, and $l_{\mathbf{\delta }{\mathbf{\gamma }}_i}=\left({\partial }^{2}l\right)/(\partial \mathbf{\delta }\partial {\mathbf{\gamma }}_i)$. With a sufficiently long test, estimated item parameters ${\widehat{\mathbf{\gamma }}}_i$ are independent across items [23]. Hence, $l_{\mathbf{\delta }}$ can be approximated around $({\mathbf{\delta }}_0,{\mathbf{\gamma }}_0)$ as

$$\begin{array}{c}\hfill l_{\mathbf{\delta }}(\mathbf{\delta },\mathbf{\gamma })\approx l_{\mathbf{\delta }}({\mathbf{\delta }}_0,{\mathbf{\gamma }}_0)+l_{\mathbf{\delta }\mathbf{\delta }}({\mathbf{\delta }}_0,{\mathbf{\gamma }}_0)(\mathbf{\delta }-{\mathbf{\delta }}_0)+{\displaystyle \sum _{i=1}^I}{l}_{\mathbf{\delta }\mathbf{\delta }}({\mathbf{\delta }}_0,{\mathbf{\gamma }}_{i0})({\mathbf{\gamma }}_i-{\mathbf{\gamma }}_{i0})\end{array}.$$

(3)

The distribution parameter estimates $\widehat{\mathbf{\delta }}=(\widehat{\mu },\widehat{\sigma })$ are obtained by setting (3) to zero and using fixed but misspecified item parameters ${\mathbf{\gamma }}_i={\mathbf{\gamma }}_i^{*}$. Hence, we obtain from (3)

$$\mathbf{0}=l_{\mathbf{\delta }}({\mathbf{\delta }}_0,{\mathbf{\gamma }}_0)+l_{\mathbf{\delta }\mathbf{\delta }}({\mathbf{\delta }}_0,{\mathbf{\gamma }}_0)(\widehat{\mathbf{\delta }}-{\mathbf{\delta }}_0)+\sum _{i=1}^I{l}_{\mathbf{\delta }{\mathbf{\gamma }}_i}({\mathbf{\delta }}_0,{\mathbf{\gamma }}_{i0})({\mathbf{\gamma }}_i^{*}-{\mathbf{\gamma }}_{i0}).$$

(4)

We now determine the distribution parameter estimate ${\widehat{\mathbf{\delta }}}_{(-i)}$ in which item i is omitted from the log-likelihood function. Empirical evidence shows that the distribution parameters can be equivalently estimated if the item parameters of item i were freely estimated. This means that one can set ${\mathbf{\gamma }}_i={\mathbf{\gamma }}_{i,0}$ for a sufficiently large number of items I. Then, (4) can be rewritten as

$$\mathbf{0}=l_{\mathbf{\delta }}({\mathbf{\delta }}_0,{\mathbf{\gamma }}_0)+l_{\mathbf{\delta }\mathbf{\delta }}({\mathbf{\delta }}_0,{\mathbf{\gamma }}_0)({\widehat{\mathbf{\delta }}}_{(-i)}-{\mathbf{\delta }}_0)+\sum _{\genfrac{}{}{0pt}{}{j=1}{j\ne i}}^I{l}_{\mathbf{\delta }{\mathbf{\gamma }}_j}({\mathbf{\delta }}_0,{\mathbf{\gamma }}_{j0})({\mathbf{\gamma }}_j^{*}-{\mathbf{\gamma }}_{j0}).$$

(5)

By subtracting (4) from (5), we obtain

$${\widehat{\mathbf{\delta }}}_{(-i)}-\widehat{\mathbf{\delta }}=-{\left[l_{\mathbf{\delta }\mathbf{\delta }}({\mathbf{\delta }}_0,{\mathbf{\gamma }}_0)\right]}^{-1}{l}_{\mathbf{\delta }{\mathbf{\gamma }}_i}({\mathbf{\delta }}_0,{\mathbf{\gamma }}_{i0})({\mathbf{\gamma }}_i^{*}-{\mathbf{\gamma }}_{i0}).$$

(6)

Now, Equation (6) is now specialized for the 2PL model. In this case, ${\mathbf{\gamma }}_i=(a_i,b_i)$ consists of two parameters. We assume that fixed item discriminations were correct and fixed item intercepts $b_i^{*}$ do not equal true data-generating item intercepts $b_{i0}$. We obtain from (6)

$$\begin{array}{c}\hfill {\widehat{\mathbf{\delta }}}_{(-i)}-\widehat{\mathbf{\delta }}\approx -{\left[l_{\mathbf{\delta }\mathbf{\delta }}({\mathbf{\delta }}_0,{\mathbf{\gamma }}_0)\right]}^{-1}{l}_{\mathbf{\delta }{b}_i}({\mathbf{\delta }}_0,{\mathbf{\gamma }}_{i0})(b_i^{*}-b_{i0})\end{array}.$$

(7)

In the following subsections, it is discussed how the finding can be used in the practical implementation (Section 2.1) of the jackknife linking error and how to efficiently compute the necessary derivatives of the log-likelihood function (Section 2.2). The estimation of linking errors is also prone to sampling errors. To circumvent a biased estimation of the linking error, we propose a bias-corrected version of the analytical approximation of the jackknife linking error in Section 2.3. Finally, a variant of the jackknife linking error computation in subsets of items is discussed in Section 2.4.

2.1. Use of the Approximation in Scaling

We now discuss how to apply the analytical approximation formula (6) for the deviations ${\widehat{\mathbf{\delta }}}_{(-i)}-\widehat{\mathbf{\delta }}$ in the jackknife linking error formula. First, we compute the distribution parameters $\widehat{\mathbf{\delta }}$ by fixing item parameters to ${\mathbf{\gamma }}^{*}$. Second, we estimate item parameters $\widehat{\mathbf{\gamma }}$ by fixing the distribution parameters to $\widehat{\mathbf{\delta }}$. The motivation is that differences of $\widehat{\mathbf{\delta }}-{\mathbf{\delta }}_0$ and $\widehat{\mathbf{\gamma }}-{\mathbf{\gamma }}_0$ are close to zero for a sufficiently large number of items I. Hence, we replace unknown parameters in (6) with their empirical counterparts, and we arrive at

$${\widehat{\mathbf{\delta }}}_{(-i)}-\widehat{\mathbf{\delta }}=-{\left[l_{\mathbf{\delta }\mathbf{\delta }}(\widehat{\mathbf{\delta }},\widehat{\mathbf{\gamma }})\right]}^{-1}{l}_{\mathbf{\delta }{\mathbf{\gamma }}_i}(\widehat{\mathbf{\delta }},{\widehat{\mathbf{\gamma }}}_i)({\mathbf{\gamma }}_i^{*}-{\widehat{\mathbf{\gamma }}}_i).$$

(8)

For the special case of the 2PL model with misspecified item intercepts $b_i^{*}$, we obtain from (7)

$${\widehat{\mathbf{\delta }}}_{(-i)}-\widehat{\mathbf{\delta }}=-{\mathit{H}}_i(b_i^{*}-{\widehat{b}}_{i0})$$

(9)

for estimated item parameters $\widehat{\mathit{b}}=({\widehat{b}}_1,\dots ,{\widehat{b}}_I)$ and ${\mathit{H}}_i={\left[l_{\mathbf{\delta }\mathbf{\delta }}(\widehat{\mathbf{\delta }},\widehat{\mathit{b}})\right]}^{-1}{l}_{\mathbf{\delta }{b}_i}(\widehat{\mathbf{\delta }},\widehat{\mathit{b}})$. The analytical approximation ${\mathrm{LE}}_{\mathrm{AN}}$ of the jackknife linking error is given as

$$\begin{array}{c}\hfill {\mathrm{LE}}_{\mathrm{AN}}\left(u\right)=\sqrt{\frac{I-1}{I}{\displaystyle \sum _{i=1}^I}{h}_{iu}^2{(b_i^{*}-{\widehat{b}}_{i0})}^2},u=\mu \mathrm{or}u=\sigma \end{array},$$

(10)

where ${\mathit{H}}_i={(h_{i\mu },h_{i\sigma })}^{\top }$. For the linking error of $\widehat{\mu }$, the first entry $h_{i\mu }$ is chosen. For the linking error of $\widehat{\sigma }$, the second entry $h_{i\sigma }$ is chosen.

2.2. Implementation Details for Computing Derivatives of the Log-Likelihood Function

We now discuss how to efficiently compute the necessary derivatives of the log-likelihood function required in the analytical approximation of the jackknife linking error. We evaluate the integral in (1) by a rectangular integration on a finite grid ${\theta }_1,\dots ,{\theta }_T$ of $\theta$ points. Hence, the continuous normal distribution is approximated by a discretized normal distribution [24]. We set $w_t=w_t(\mu ,\sigma )=C\varphi ({\theta }_t;\mu ,\sigma )$ using an appropriate scaling constant C that ensures ${\sum }_{t=1}^T{w}_t=1$.

Let $l_p=log{L}_p$ denote the contribution in the log-likelihood function of person p based on item response data ${\mathit{x}}_p=(x_{p1},\dots ,x_{pI})$. It holds that

$$L_p=\sum _{t=1}^T{w}_t\prod _{i=1}^I{f}_{pti}=\sum _{t=1}^T{w}_t{f}_{pt},\mathrm{where}$$

(11)

$$f_{pti}=P_i{({\theta }_t,{\mathbf{\gamma }}_i)}^{x_{pi}}{\left[1-P_i({\theta }_t,{\mathbf{\gamma }}_i)\right]}^{1-x_{pi}}$$

(12)

and $f_{pt}={\prod }_{i=1}^I{f}_{pti}$. We now compute the partial derivative of $l_p$ for a scalar parameter u

$$\frac{\partial l_p}{\partial u}=\frac{{\displaystyle \frac{\partial L_p}{\partial u}}}{L_p}.$$

(13)

The second-order derivative with respect to another parameter v is given as

$$\frac{{\partial }^2{l}_p}{\partial u\partial v}=\frac{{\displaystyle \frac{{\partial }^2{L}_p}{\partial u\partial v}{L}_p-\frac{\partial L_p}{\partial u}\frac{\partial L_p}{\partial v}}}{L_p^2}.$$

(14)

We can compute for $u=\mu$ or $u=\sigma$

$$\frac{\partial L_p}{\partial u}=\sum _{t=1}^T\frac{\partial w_t}{\partial u}{f}_{pt}.$$

(15)

The second-order derivative for $v=\mu$ or $v=\sigma$ can be obtained as

$$\frac{{\partial }^2{L}_p}{\partial u\partial v}=\sum _{t=1}^T\frac{{\partial }^2{w}_t}{\partial u\partial v}{f}_{pt}.$$

(16)

In the analytical approximation of the jackknife linking error, we need to compute $\left({\partial }^2{L}_p\right)/(\partial u\partial b_i)$ for an item parameter $b_i$. We obtain

$$\frac{{\partial }^2{L}_p}{\partial u\partial b_i}=\sum _{t=1}^T\frac{\partial w_t}{\partial u}{f}_{pt}\frac{1}{f_{pit}}\frac{\partial f_{pit}}{\partial b_i}=\sum _{t=1}^T\frac{\partial w_t}{\partial u}{f}_{pt}\frac{\partial log{f}_{pit}}{\partial b_i}.$$

(17)

Equations (15)–(17) indicate that the necessary computations for the first- and second-order derivatives are computationally inexpensive. Hence, the analytical approximation of the linking error is computationally cheap if DIF effects were available.

2.3. Bias Correction due to Sampling Error

The estimation of linking errors is also prone to sampling errors because estimated item parameters are involved in the computation. To avoid a biased estimation of the linking error, we now propose a bias-corrected version of the analytical approximation of the jackknife linking error.

Assume that the variance of $b_i^{*}-{\widehat{b}}_{i0}$ in (10) is $v_i$, and the estimates are approximately independent across items [23]. The bias-corrected analytical linking error can be obtained by subtracting variability that is due to sampling error quantified in $v_i$. We obtain

$$\begin{array}{c}\hfill {\mathrm{LE}}_{\mathrm{AN},\mathrm{bc}}\left(u\right)={\mathrm{sqrt}}_{+}\left(\frac{I-1}{I}{\displaystyle \sum _{i=1}^I}{h}_{iu}^2\left[{(b_i^{*}-{\widehat{b}}_{i0})}^2-v_i\right]\right),u=\mu \mathrm{or}u=\sigma \end{array},$$

(18)

where ${\mathrm{sqrt}}_{+}\left(x\right)=\sqrt{max(x,0)}$.

A similar bias-correction method was used in the 1PL method in trend estimation [18]. Alternatively, a bias-correction term can also be estimated using resampling techniques regarding persons (e.g., bootstrap or half sampling; [19,22]).

2.4. Jackknife Linking Error Based on Testlets

The jackknife linking error is frequently evaluated at groups of items (so-called testlets; refs. [25,26]) instead of single items. The reason for this is that subsets of items in a test are often presented jointly with a common (text) stimulus. Hence, DIF effects pertain to all items in a testlet and often have the same sign. Therefore, the testlet structure must be taken into account when computing linking errors [17,18,27].

Assume that there are H testlets. That is, the set of item integers $i=1,\dots ,I$ is partitioned into distinct sets ${\mathcal{I}}_1,\dots ,{\mathcal{I}}_H$. The linking error based on testlets for $\widehat{\mu }$ is defined as

$$\begin{array}{c}\hfill \mathrm{LE}\left(\widehat{\mu }\right)=\sqrt{\frac{H-1}{H}{\displaystyle \sum _{h=1}^H}{({\widehat{\mu }}_{(-{\mathcal{I}}_h)}-\widehat{\mu })}^2}\end{array},$$

(19)

where ${\widehat{\mu }}_{(-{\mathcal{I}}_h)}$ is the estimate in which all items from testlet h were removed. In the analytical approximation, we can approximate the relevant jackknife difference in (19) by

$$\begin{array}{c}\hfill {\widehat{\mu }}_{(-{\mathcal{I}}_h)}-\widehat{\mu }={\displaystyle \sum _{i\in {\mathcal{I}}_h}}{h}_{i\mu }(b_i^{*}-b_{i0})\end{array}.$$

(20)

A corresponding bias-corrected variant of the linking error (see (18)) can be similarly obtained.

3. Simulation Study

3.1. Method

In this simulation study, we investigate the performance of our analytical approximation of the jackknife linking error. We illustrate the performance based on the 2PL model. Equal discriminations $a_i\equiv 1$ were assumed.

In the simulation study, we varied two factors: the number of items and the DIF standard deviation ${\tau }_{\mathrm{DIF}}$. We chose $I=10$, 20, 30, and 40 items to cover a range of test lengths that are obtained in empirical practice. The goal is to assess the mean $\mu$ and the standard deviation $\sigma$ in a group (e.g., a country in a large-scale assessment study such as PISA). We defined $\mu =-0.2$ and $\sigma =0.9$ in the simulation. Assumed item difficulties $b_i^{*}$ were chosen equispaced in the interval $[-2,2]$ with increments $4/(I-1)$. For example, for $I=10$ items, assumed item difficulties were −2.00, −1.56, −1.11, −0.67, −0.22, 0.22, 0.67, 1.11, 1.56, and 2.00. In each replication, data-generating item difficulties $b_{i0}$ were simulated according

$$b_{i0}=b_i^{*}+e_i,e_i\sim \mathrm{N}(0,{\tau }_{\mathrm{DIF}})$$

(21)

Hence, the estimated distribution parameters $\widehat{\mu }$ and $\widehat{\sigma }$ will vary across replications even with infinite sample sizes of persons because the true data-generating item parameters vary. The standard deviation of DIF effects was either ${\tau }_{\mathrm{DIF}}=0.25$ or 0.50.

Item response data were simulated according to a quasi-Monte Carlo simulation method ([28], see also [29] for a similar approach). Our motivation was to assess the performance of the linking error by reducing the uncertainty due to sampling error. Simulated item responses $\mathit{X}$ should be as close as possible to the true distribution $\mathrm{P}(\mathit{X};\mu ,\sigma ,{\mathit{b}}_0)$ (see [29]). To facilitate this, we chose $\theta$ values from the same discrete grid ${\theta }_1=-3.5,\dots ,{\theta }_{21}=3.5$ of equidistant $\theta$ points that were also used in fitting the 2PL model. We fixed a pseudo-sample size of 10,000 persons. Then, we computed the number of persons at each ${\theta }_t$, which is given by $N{w}_t(\mu ,\sigma )$ (some rounding is necessary). For each $\theta ={\theta }_t$ and for each item i, we can compute $P_i({\theta }_t;a_i,b_{i0})$ according to the item response function. Hence, there are $N{w}_t(\mu ,\sigma )P_i({\theta }_t;a_i,b_{i0})$ persons with $\theta ={\theta }_t$ with $X_i=1$ and $N{w}_t(\mu ,\sigma )\left(1-P_i({\theta }_t;a_i,b_{i0})\right)$ with $X_i=0$. The zeroes and ones for item $i=1,\dots ,I$ are randomly allocated to the corresponding persons $\theta ={\theta }_t$. Although the empirical frequencies for multivariate item response patterns $\mathit{X}=\mathit{x}$ do not match the population probabilities, conditional marginal probabilities of the item response functions are (almost) correctly simulated. Hence, one can conclude that this quasi-Monte Carlo approach reduces the impact of sampling errors to a minimum and only reflects variability due to linking errors; that is, using incorrect item parameters $b_i^{*}$ that differ from the data-generating item parameters $b_i$.

In each of the $4\times 2=8$ cells of the simulation, 2000 replications were conducted. We fitted the 2PL model with item discriminations fixed to 1 and fixed item difficulties $b_i^{*}$ and obtained the estimated mean $\widehat{\mu }$ and the estimated standard deviation $\widehat{\sigma }$. Then, we determined item difficulties ${\widehat{b}}_i$ by fixing the mean and standard deviation to $\widehat{\mu }$ and $\widehat{\sigma }$, respectively. Using these quantities, we calculated the analytical approximation (AN) of the jackknife linking error given in (10). We compared the analytical linking error with the jackknife linking error (JK). To evaluate the quality of the estimated linking errors, we computed the mean, the standard deviation, the standard error ratio (${\mathrm{SE}}_{\mathrm{ratio}}$; defined as the quotient of the average linking error and the empirical standard deviation of the estimate $\widehat{\mu }$ or $\widehat{\sigma }$), and coverage rates at the 95% confidence level.

The R software [30] was used for simulation and analysis. The R package TAM [31] was used for estimating the 2PL model.

3.2. Results

Table 1. Simulation study: Results as a function of the standard deviation of DIF effects (${\tau }_{\mathrm{DIF}}$) and number of items (I).

				Mean			${\mathbf{SE}}_{\mathbf{ratio}}$			SD		COV95
${\mathbf{\tau }}_{\mathrm{DIF}}$	$\mathit{I}$	Bias	SD	EXP	JK	AN	JK	AN	AAD	JK	AN	JK	AN
		Linking error of estimated mean $\widehat{\mu }$
0.25	10	0.000	0.079	0.079	0.079	0.070	0.990	0.875	0.0091	0.0202	0.0177	91.6	88.6
0.25	20	0.003	0.056	0.056	0.056	0.053	1.000	0.945	0.0031	0.0096	0.0091	93.2	91.7
0.25	30	0.000	0.045	0.046	0.046	0.044	1.007	0.970	0.0017	0.0065	0.0062	94.3	93.3
0.25	40	0.003	0.039	0.040	0.040	0.039	1.013	0.986	0.0011	0.0047	0.0045	96.3	95.7
0.5	10	0.003	0.161	0.158	0.153	0.135	0.949	0.837	0.0181	0.0397	0.0354	91.6	87.9
0.5	20	0.002	0.111	0.112	0.111	0.104	1.000	0.942	0.0065	0.0185	0.0177	93.5	91.8
0.5	30	0.004	0.090	0.091	0.091	0.087	1.011	0.972	0.0036	0.0130	0.0127	94.5	93.5
0.5	40	0.008	0.080	0.079	0.078	0.076	0.978	0.948	0.0025	0.0097	0.0095	93.2	92.3
		Linking error of estimated standard deviation $\widehat{\sigma }$
0.25	10	−0.007	0.039	—	0.041	0.033	1.051	0.866	0.0073	0.0111	0.0091	95.2	92.3
0.25	20	−0.008	0.023	—	0.023	0.021	1.009	0.914	0.0022	0.0042	0.0038	94.7	93.0
0.25	30	−0.008	0.019	—	0.017	0.016	0.901	0.848	0.0011	0.0026	0.0024	92.4	91.7
0.25	40	−0.008	0.016	—	0.014	0.014	0.898	0.856	0.0007	0.0019	0.0017	90.9	90.8
0.5	10	−0.030	0.078	—	0.077	0.062	0.985	0.796	0.0155	0.0206	0.0166	94.9	90.8
0.5	20	−0.029	0.053	—	0.043	0.040	0.827	0.752	0.0047	0.0079	0.0068	89.3	87.5
0.5	30	−0.027	0.042	—	0.033	0.031	0.774	0.727	0.0026	0.0048	0.0044	85.9	84.7
0.5	40	−0.027	0.037	—	0.027	0.026	0.723	0.690	0.0019	0.0034	0.0033	85.0	83.0

Note. EXP = expected value of linking error for $\widehat{\mu }$; JK = jackknife linking error; AN = linking error estimated by analytical approximation LE_AN from Equation (10); SE_ratio = standard error ratio; AAD = average absolute difference between JK and AN linking error estimates; SD = standard deviation of estimated linking error; COV95 = coverage rate for confidence level 95%.

contains the results of the two linking error methods as a function of the standard deviation of DIF effects ${\tau }_{\mathrm{DIF}}$ and the number of items I.

It can be seen that the mean can be almost unbiasedly estimated in the presence of DIF effects. As expected, the standard deviation of the estimated mean $\widehat{\mu }$ decreases with a larger number of items. Because all item discriminations were equal to one, the linking error of $\widehat{\mu }$ can be analytically predicted as ${\tau }_{\mathrm{DIF}}/\sqrt{I}$ (referred to as EXP in Table 1; [16,18]). It can be seen that empirical standard deviations of $\widehat{\mu }$ were close to these expected values. Moreover, the mean of the jackknife linking error JK was very similar to the empirical standard deviation of $\widehat{\mu }$, while the linking error AN based on the analytical approximation was slightly too small. This was particularly the case for a low number of items $I=10$. However, with a larger number of items, the analytical approximation performed well. This behavior is also reflected in the standard error ratio ${\mathrm{SE}}_{\mathrm{ratio}}$, which attained desired values close to 1 for the jackknife linking error and was smaller than 1 for the linking error based on the analytical approximation. However, for at least 20 items, the analytical approximation might be considered to have satisfactory performance. Overall, it can also be seen that the estimated linking error AN was a bit smaller on average than the jackknife linking error. However, the average absolute deviation (AAD in Table 1) demonstrated that the analytical approximation was very close to the jackknife linking error in each replication, particularly for a large number of items such as $I=40$. It can also be seen that coverage rates were satisfactory for the jackknife linking error, while the analytical approximation showed issues in the condition of a few items (i.e., $I=10$). Finally, we observed that the empirical standard deviation of estimated linking errors was slightly smaller for the analytical approximation compared to the jackknife approach.

The estimated standard deviation $\widehat{\sigma }$ showed a small bias in the condition of a larger standard deviation of DIF effects (i.e., ${\tau }_{\mathrm{DIF}}=0.5$). Hence, one can expect that coverage rates will perform satisfactorily because the expected value deviated from the true value $\sigma =0.9$. Interestingly, the mean of both linking errors was smaller than the empirical standard deviation of $\widehat{\sigma }$. This finding was also reflected in the standard error ratios, which were substantially smaller than 1. Hence, one might conclude that both linking errors (jackknife JK and the analytical approximation AN) were unsatisfactory to reflect the variability in estimated standard deviations in our application. We suspect these results might be explained by the fact that the standard deviation was obtained using fixed but incorrect item parameters. Moreover, such a finding would likely not be observed in a linking approach such as log-mean-mean linking [32].

Across all conditions, the analytical approximation provided linking errors that were slightly too small. In the computation of the jackknife linking error in Equation (10), the multiplication factor $(I-1)/I$ is used. In other applications, the multiplication factor 1 is used [22]. We recomputed linking errors based on the analytical approximation with the multiplication factor of 1, and it turned out that the results grew closer to the jackknife linking error. However, the standard error ratio was still smaller than 1. Nevertheless, there could be a benefit to using the modified formula in empirical applications.

On the Bias in the Estimated Standard Deviation $\widehat{\sigma }$

At first sight, the negative bias in the estimated standard deviation $\widehat{\sigma }$ is surprising, although such a finding was also found in other studies that use fixed but incorrect item parameters in estimation [32]. We now present a heuristic derivation of the bias that appears close to the empirically obtained bias. Please note that the estimation of the item response model using a logistic item response function can be approximated by the probit link function [33]. In this case, weighted least squares estimation based on tetrachoric correlations [34] can be used to estimate the standard deviation $\sigma$. The approach relies on modeling underlying continuous variables $X_i^{*}$ for dichotomous items $X_i$. The variable $X_i$ takes the value if $X_i^{*}$ exceeds the item parameter $b_i$. The tetrachoric correlation ${\rho }_{ij}$ for items i and j is given by

$${\rho }_{ij}=\frac{\mathrm{Cov}(X_i^{*},X_j^{*})}{\sqrt{\mathrm{Var}\left(X_i^{*}\right)\mathrm{Var}\left(X_j^{*}\right)}}.$$

(22)

Now assume equal item discriminations $a_i$ and a correctly specified model. In this case, (22) simplifies to

$${\rho }_{ij}=\frac{{\sigma }^2}{{\sigma }^2+\mathcal{L}},$$

(23)

where $\mathcal{L}={\pi }^2/3\approx 3.29$ is the variance of the logistic distribution. If incorrect item parameters $b_i^{*}$ were used, the covariance $\mathrm{Cov}(X_i^{*},X_j^{*})$ in (22) is not affected on average. However, the variance $\mathrm{Var}\left(X_i^{*}\right)$ of the underlying latent variable $X_i^{*}$ increases, and the expected value can be determined by the DIF variance ${\tau }_{\mathrm{DIF}}^2$. The estimated tetrachoric correlation can then be computed as

$${\rho }_{ij}^{*}=\frac{{\sigma }^2}{{\sigma }^2+{\tau }_{\mathrm{DIF}}^2+\mathcal{L}}.$$

(24)

In the computation of $\widehat{\sigma }$, one, therefore, essentially solves

$${\rho }_{ij}^{*}=\frac{{\sigma }^2}{{\sigma }^2+{\tau }_{\mathrm{DIF}}^2+\mathcal{L}}=\frac{{\widehat{\sigma }}^2}{{\widehat{\sigma }}^2+\mathcal{L}}.$$

(25)

The estimated standard deviation $\widehat{\sigma }$ can be determined as (see [35] for a similar approach)

$$\widehat{\sigma }=\sqrt{\mathcal{L}\frac{{\rho }_{ij}^{*}}{1-{\rho }_{ij}^{*}}}=\sigma \sqrt{\frac{\mathcal{L}}{\mathcal{L}+{\tau }_{\mathrm{DIF}}^2}}.$$

(26)

Hence, the estimated standard deviation is negatively biased in the presence of DIF effects. The predicted bias in the estimated standard deviation based on (26) is $-0.008$ for ${\tau }_{\mathrm{DIF}}=0.25$ and $-0.032$ for ${\tau }_{\mathrm{DIF}}=0.50$, which is similar to the empirically obtained bias for $\widehat{\sigma }$ in Table 1.

4. Discussion

In this article, an analytical approximation of the jackknife linking error by means of a Taylor expansion of the log-likelihood function has been proposed. It turned out that the analytical approximation performed well for estimated means for at least 20 items. The approximation has the advantage because it only requires one additional estimation of an item response model and second-order derivatives of the log-likelihood function. In contrast, the jackknife linking error requires I additional estimations of the item response model for I items which is computationally much more demanding. One might argue that the analytical approximation provides at least a computationally cheap proxy of the linking error. However, the jackknife linking error would be preferred due to a more reliable statistical inference because our simulation findings indicated that it provided slightly better coverage rates.

In the simulation study, we did not consider sampling errors because a quasi-Monte Carlo simulation method was utilized that minimized the impact of sampling errors. Future studies could simultaneously assess sampling errors and linking errors based on the analytical approximation. In particular, the performance of our proposed bias-correction method could be evaluated.

In the analytical derivation and the simulation study, we restricted ourselves to the computation of a single group which can be interpreted as country means and standard deviations in a cross-sectional assessment or the computation of trend between two time points. In future research, our proposed simplified computation formula for the linking error might be applied for trend estimates in the country means in educational assessment studies such as PISA [18].

It should be noted that our proposed approach of the analytical approximation of the jackknife linking error bears similarity with the infinitesimal jackknife technique [36,37,38]. The difference is that the jackknife linking error removes columns (i.e., items) from the dataset, while infinitesimal jackknife addresses contributions of rows (i.e., persons) in a dataset.

In the linking literature [39,40], linking errors are also sometimes referred to as sampling errors (of persons) of obtained linking constants that can be means or standard deviations [41,42,43]. It is important to note that sampling errors due to the sampling of persons and linking errors due to item choice must be distinguished [19,20]. The computation of linking errors can be justified for random items and fixed items [19]. For random items, the used items are thought to be a (representative) draw from a larger population of items [44,45]. For fixed items, DIF effects can be stochastically modeled by some distribution [19]. The latter case might also be conceived as quantifying model error [46]. We would like to point out that we see great potential in using linking errors by incorporating the extent of model misspecification in the reported parameter uncertainty.