Cognitive Trait Model: Measurement Model for Mastery Level and Progression of Learning

Abstract

This paper seeks to establish a framework which operationalizes cognitive traits as a portion of the predefined mastery level, the highest level expected to successfully perform all of the relevant tasks of the target trait. This perspective allows us to use and interpret the cognitive trait levels in relative quantities (e.g., %s) of the mastery level instead of relative standings (i.e., rankings) on an unbounded continuum. To facilitate the proposed perspective, this paper presents an analytical framework that has support on the [0, 1] trait continuum with truncated logistic link functions. The framework provides a solution to cope with the chronic question of “relative standings or magnitudes of learning outcome?” in measuring cognitive traits. The proposed framework is articulated relative to the traditional models and is illustrated with both simulated and empirical datasets within the Bayesian framework, estimated with the Markov chain Monte Carlo method.

1. Introduction

The primary interest of the social and behavioral sciences is typically the traits of human beings, where the developments of tools to model these traits are of critical theoretical and practical interest. Messick defined a trait as “a relatively stable characteristic of a person … which is consistently manifested to some degree when relevant, despite considerable variation in the range of setting and circumstances” [1]. According to [2], the social and behavioral sciences encounter two major domains of human traits: cognitive and affective. In the cognitive domain, researchers would like to model a wide range of traits:

Various classes of cognitive and neuropsychological functioning, including intelligence, broad ability domains, and more focused domains (e.g., abstract reasoning and categorical thinking; academic achievement; attention; cognitive ability; executive function; language; learning and memory; motor and sensorimotor functions and lateral preferences; and perception and perceptual organization/integration) [3] (p. 155).

In the affective domain, on the other hand, the researchers are customarily interested in feelings or emotions, such as attitude or self-efficacy. Although the field of psychometrics covers both of the domains, in educational measurement or learning science specifically, the traits in the cognitive domain have been of primary interest.

Ref. [4], for example, defined the primary goal of educational and psychological measurement as “the determination of how much of such a latent ability (trait) a person possesses” (p. 3). They continued, “The underlying idea here is that, if one could physically ascertain the ability of a person, this ruler would be used to tell how much ability a given person has, the ability of several persons could be compared” (p. 3). As such, the current work will be framed largely within the cognitive domain.

A cognitive trait has been conceptualized as an ability or proficiency to perform relevant tasks to some degree (e.g., [4]). What does it mean to have the ability to do relevant tasks? What does it mean to have the ability to recall the brand of the phone you are using, to compute 2 + 2, or to ride a bicycle? One natural answer could be “to have an ability is to know how to do some relevant tasks to some degree”. To have the ability to name the brand of a phone is to know how to name the brand of a phone, to have the ability to compute 2 + 2 is to know how to solve 2 + 2, and to have the ability to ride a bicycle is to know how to ride a bicycle. In other words, a cognitive trait (i.e., the ability to do or knowledge how to do) is manifested in performing the relevant tasks identified, regardless of whether the target knowledge is related to knowledge-how or knowledge-that.

From the observations of the performance of related tasks, we infer the levels of the target cognitive trait, that is, “how much ability a learner has acquired (or possessed) to perform relevant tasks?” Therefore, to appropriately infer a cognitive trait level, the features of the trait and their interdependencies should be agreed among the relevant experts and carefully melted into these tasks. In other words, the domain (i.e., a complete list) of tasks must be pre-defined to measure (especially, to measure) a cognitive trait.

Given a domain of tasks for a cognitive trait, we can naturally assume there is a cognitive trait level that is expected to perform all of those tasks successfully. This, as the highest expected level, should be called the mastery level. Under the same logic, there is a level that is expected where one fails at all of those tasks, which is the lowest-level, and which should be called the ignorance level. Then, the levels between these two boundaries represent the sizes of the portions (parts) of the mastery. This induces an analytical framework that maps the trait levels on a bounded continuum from the lowest level to the highest (i.e., mastery) level. Consider an instrument of measurement (an examination or test) for a problem-solving trait: “For any number from 1 to 9, find the number that makes 10 when added to the given number” [5]. Let us further assume that there are only 10 tasks (i.e., the 10 tasks are the domain of the tasks) for the trait. Ignoring the measurement error and bias for the moment, if a person fails to solve all of the 10 tasks, we would infer that this person is at the lowest trait level. Likewise, if a person successfully solves all of the 10 tasks (again temporarily ignoring the issue of measurement error and bias), we would conclude that this person is at the mastery level.

To model the connections between the levels of cognitive (or affective) domain traits (constructs or latent variables) and the observations from relevant tasks (items or indicators), latent variable models (LVMs) are especially useful. LVMs include the factor models (FMs), structural equation models (SEMs), and item response theory models (IRTMs). In traditional LVMs, the trait levels have typically been assumed to be on the unbounded real line continuum (i.e., from negative infinity to positive infinity), with a midpoint of zero. This assumption, however, may be more due to the nature of the mathematical model being employed than a theoretical belief that the levels of a trait can keep decreasing or increasing indefinitely.

Although the traditional analytical framework has been proved to be useful in analyzing the relationship among constructs or to measure the relative standing levels of cognitive traits for the single point, group comparison, and/or summative assessment purposes, it has limitations in continuous (i.e., longitudinal) and/or individually adaptive assessments, such as formative assessment, the learning growth model, or real-time learning analytics in the digital environment. Suppose that someone’s trait level is estimated as −1.5, with an unbounded real line continuum. Although the (unbounded) continuum value is useful for placing and differentiating the individuals’ relative location (i.e., ranking), it does not directly provide information on the level of achievement or acquisition compared to a mastery level before making additional efforts (some additional transformation techniques will be discussed later in this paper).

In fact, researchers have proposed several options that better conceptualize and represent trait estimates differently. Perhaps the most popular, and simplistic, transforms the latent variable estimates to a distribution with a designated mean and standard deviation (e.g., μ = 100 and σ = 15, as for some intelligence test scores). Within the IRTM arena, ref. [6] suggested a method using the number of correct scores as the basis for the item characteristic curves (ICCs) rather than the trait score; however, this approach was not further developed, because the ICCs were dependent on the set of items included in a test (i.e., test-specific). Reference [7] suggested a domain score that is an expected number-correct score, derived by transforming an examinee’s IRTM trait estimate via a weighted sum of each ICC. The authors argued that the domain scores are preferable in the context of student qualification. Although this domain score approach enhances the interpretation of the assessment result, it is still an indirect interpretation of the trait estimates and requires transformations with IRT estimation results. More complex approaches, within IRTMs, Item Maps, or Wright Maps [8] have been used to display both the person parameter (i.e., trait level) and the item difficulty parameter estimates along a common scale. Based on this approach, one can interpret a trait estimate of −1.5 as an examinee being expected to have a 0.5 probability of correctly answering an item whose difficulty is at −1.5 on the normal continuum (low difficulty). This interpretation, however, primarily describes the relation between the trait and items (again, the expected probability of correct answer) rather than the trait estimate itself, and it is still an indirect characterization/conceptualization of what the person parameter means.

Several LVMs to conceive of the latent variables as bounded have also been suggested, such as the Beta-Binomial model [9]. In this model, the person parameter represents a true score in the [0, 1] metric with a Beta prior distribution, and the model is used as a method for classifying the accuracy of the IRTMs [10]. Diagnostic classification models (DCMs; ref. [11]), which provide each subject with a profile detailing the attributes (latent variables), can also model latent categories. These models, however, are based on discrete traits—such as dichotomous (0 = no-mastery status or 1 = mastery status) or polytomous (0/1/2/…) variables—to represent the stages of mastery and provide the expected probabilities of stage mastery. Another popular assessment modeling technique, Bayesian networks (BNs; ref. [12]), represent a set of latent variables and their conditional dependencies with a directed acyclic graph (DAG). However, in the same way as with CDMs, the BNs also model the latent variables as discrete, and provide probability estimates for each latent category.

For formative assessment, learning growth, or adaptive learning systems, which have rapidly gained in popularity in recent years, the researchers, practitioners, educators, and learners are in dire need of a new analytical framework that supports more direct and intuitive uses and interpretations of ‘the magnitude of learning outcomes and its changes.’ To help in the communication between the observations from the tasks and the trait continuum of interest in such uses and demands, therefore, it would be useful to have an analytical framework that facilitates the trait levels on the [0, 1] continuum as follows:

(1)

Assigning zero (0) to the lower boundary (the lowest or ignorance level): a trait level as an expected proportion of the succeeded tasks out of all of the tasks is 0 (or 0%);

(2)

Assigning unit (1) to the upper boundary (the highest or mastery level): a trait level for an expected proportion of the succeeded tasks out of all of the tasks is 1 (or 100%);

(3)

Assigning a number between the 0 and 1 boundaries: a trait level as an expected proportion of the succeeded tasks out of all of the tasks.

This paper proposes an analytical framework for measuring cognitive traits (so-called, Cognitive Trait Model) based on truncated logistic functions to serve this purpose. First, analytical specifications of the traditional and proposed models are presented. Second, the various psychometric aspects of the proposed models are illustrated, alongside those of the traditional models. Third, a simulated dataset and an empirical dataset are analyzed to illustrate and evaluate the proposed model, using a Markov chain Monte Carlo (MCMC) estimation method [13]. Lastly, this paper concludes with a discussion and suggested future works.

2. Developments

2.1. Traditional Models

Let y_ij be the observed data of the ith person on the jth item (or task), and without loss of generality, let the possible values for each y_ij be 1 (indicating successful completion of a task, a correct answer to an item, etc.) and 0 (indicating failure to complete a task, an incorrect answer to an item, etc.). IRTMs [14,15], widely used LVMs, can be expressed as a version of:

$$\mathrm{Pr}(y_{ij}=1|{\theta }_i,a_j,b_j,c_j)=c_j+(1-c_j)\Psi (a_j({\theta }_i-b_j)),$$

(1)

where ${\theta }_i$ represents the level (score) of a cognitive trait for person i. The symbols $a_j$, $b_j$, and $c_j$ represent the parameters for item j. Specifically, $a_j$ is often referred to as the slope or discrimination parameter for the item, because it relates to how well an item differentiates (or discriminates) among the people with lower and higher values along the $\theta$ continuum. The symbol $b_j$ represents the location parameter for the jth item, often referred to as the difficulty parameter for item j. Finally, $c_j$ is often referred to as the pseudo-guessing parameter, reflecting the probability that even the least capable person will occasionally give a correct response, that is, Pr$\left(y_{ij}=1|\theta \right)$ > 0, due to guessing (e.g., the probability of giving a correct response by probability is 0.25 on a selected-response item with four possible choices). The symbol $\mathsf{\Psi }$, referred to as link function (or item response function in the context of IRTMs), is a monotonically increasing function that maps $\theta$ to the probability of $y_{ij}=1$.

In traditional LVMs, the cumulative normal distribution function (i.e., normal ogive function) and the logistic function are popular choices for $\Psi$, with the latter being more common for IRTMs. In the case of choosing $\Psi$ in Equation (1) to be a logistic function, this yields the familiar three-parameter logistic (3PL) IRTM [15]:

$$\mathrm{Pr}(y_{ij}=1|{\theta }_i,a_j,b_j,c_j)=c_j+(1-c_j)\times \frac{\exp (a_j({\theta }_i-b_j))}{1+\exp (a_j({\theta }_i-b_j))}.$$

(2)

Setting $c_j=0$, effectively assuming that guessing on items cannot yield a correct answer, yields the two-parameter logistic (2PL) IRTM [14]:

$$\mathrm{Pr}(y_{ij}=1|{\theta }_i,a_j,b_j)=\frac{\exp (a_j({\theta }_i-b_j))}{1+\exp (a_j({\theta }_i-b_j))}.$$

(3)

Finally, assuming all of the items have the same ability to discriminate among individuals, setting $a_j=1$ yields the one-parameter logistic (1PL) IRTM or Rasch model [16]:

$$\mathrm{Pr}(y_{ij}=1|{\theta }_i,b_j)=\frac{\exp ({\theta }_i-b_j)}{1+\exp ({\theta }_i-b_j)}.$$

(4)

Whether logistic function or cumulative normal distribution function (i.e., normal ogive function), all of the models in Equations (2)–(4) are supported on an entire real number line for θ. In other words, the support (the set of θ points where the function is not zero-valued) of these functions is from −∞ to +∞. (for information on defining the latent distribution by assuming some non- or semi-parametric form, and simultaneously estimating the distribution with item parameters, see [17].

2.2. Proposed Models

In order to derive link functions supported on the [0, 1] continuum, that is, the levels of a cognitive trait assumed to be on a bounded 0 to 1 continuum ($0\le \theta \le 1$), one can analytically transform the traditional 2-parameter logistic (2PL) IRTM function as follows. First, let z be the function from Equation (3):

$$z=\frac{\exp (a_j({\theta }_i-b_j))}{1+\exp (a_j({\theta }_i-b_j))}$$

(5)

Subtracting 0.5 from z shifts the function to the left,

$$\frac{\exp (a_j\left({\theta }_i-b_j\right))}{1+\exp (a_j\left({\theta }_i-b_j\right))}-\frac{1}{2}.$$

(6)

Creating a function that passes through the point ($b_j$, 0) with the upper horizontal asymptote 0.5 and lower horizontal asymptote −0.5. Next, multiplying the shifted function by a factor $\delta$ stretches/shrinks the function in Equation (6) vertically:

$$\delta \left(\frac{\exp (a_j\left({\theta }_i-b_j\right))}{1+\exp (a_j\left({\theta }_i-b_j\right))}-\frac{1}{2}\right).$$

(7)

Lastly, adding a constant $\epsilon$ shifts the above function vertically:

$$\delta \left(\frac{\exp (a_j\left({\theta }_i-b_j\right))}{1+\exp (a_j\left({\theta }_i-b_j\right))}-\frac{1}{2}\right)+\epsilon .$$

(8)

The resulting function in Equation (8) now passes through the point (b_j, $\epsilon$), with new horizontal asymptotes +0.5δ + ε and −0.5δ + ε.

In order to make the above function in Equation (8) bounded on [0, 1], values for $\delta$ and $\epsilon$ must be found so the function equals 0 when $\theta =0$ and the function equals 1 when $\theta =1$. Substituting accordingly yields the system of two equations:

$$\{\begin{array}{l}\delta \left(\frac{\exp (-a_j{b}_j)}{1+\exp (-a_j{b}_j)}-\frac{1}{2}\right)+\epsilon =0\\ \delta \left(\frac{\exp (a_j(1-b_j))}{1+\exp (a_j(1-b_j))}-\frac{1}{2}\right)+\epsilon =1\end{array}$$

(9)

with unknowns $\delta$ and $\epsilon$. Solving this system results in the following:

$$\begin{array}{l}\delta =\frac{(1+\exp (a_j(1-b_j)))(1+\exp (a_j{b}_j))}{\exp (a_j)-1},\\ \epsilon =\frac{1}{2}\delta -\frac{(1+\exp (a_j(1-b_j)))}{\exp (a_j)-1}.\end{array}$$

(10)

After substituting (10) in for $\delta$ and $\epsilon$ into the function in Equation (8), the function can be simplified as:

$$\frac{1-\exp (a_j\times {\theta }_i)}{1+\exp (a_j\left({\theta }_i-b_j\right))}\times \frac{1+\exp (a_j\left(1-b_j\right))}{1-\exp (a_j)}.$$

(11)

The resulting function (11) is a monotonically increasing link function with respect to ${\theta }_i$ with [0, 1] support. Finally, paralleling the 3PL IRTM in Equation (2), a pseudo-guessing parameter γ_j may be incorporated back into the function, yielding the desired three-parameter cognitive trait model (3P CTM):

$$\mathrm{Pr}(y_{ij}=1|{\theta }_i,{\alpha }_j,{\beta }_j,{\gamma }_j)={\gamma }_j+(1-{\gamma }_j)\times \frac{1-\exp ({\alpha }_j\times {\theta }_i)}{1+\exp ({\alpha }_j({\theta }_i-{\beta }_j))}\times \frac{1+\exp ({\alpha }_j(1-{\beta }_j))}{1-\exp ({\alpha }_j)}.$$

(12)

In the 3P CTM, I is the person index (i = 1,…, n), j is the item index (j = 1,…, J), α_j is a CTM slope or discrimination parameter (similar to the a parameter in the traditional model), β_j is a CTM location (or difficulty) parameter (similar to the b parameter in the traditional 2PL and 3PL IRTMs), and γ_j is a CTM pseudo-guessing parameter for item j.

Further parallels to IRTMs may also be derived. Setting ${\gamma }_j=0$ yields a two-parameter cognitive trait model (2P CTM):

$$\mathrm{Pr}(y_{ij}=1|{\theta }_i,{\alpha }_j,{\beta }_j)=\frac{1-\exp ({\alpha }_j\times {\theta }_i)}{1+\exp ({\alpha }_j({\theta }_i-{\beta }_j))}\times \frac{1+\exp ({\alpha }_j(1-{\beta }_j))}{1-\exp ({\alpha }_j)}.$$

(13)

In addition, setting ${\alpha }_j$ to an arbitrary positive value, such as ${\alpha }_j$ = 10, yields a one-parameter cognitive trait model (1P CTM):

$$\mathrm{Pr}(y_{ij}=1|{\theta }_i,{\beta }_j)=\frac{1-\exp (10\times {\theta }_i)}{1+\exp (10\times \left({\theta }_i-{\beta }_j\right))}\times \frac{1+\exp (10\times (1-{\beta }_j))}{1-\exp (10)}.$$

(14)

For someone who wants the value fitted to Beta(2, 2) cdf instead of 10, use 9.3827 which is estimated with fminsearch in MATLAB (2020). All of the 1, 2, and 3P CTMs are not undefined when ${\alpha }_j$ = 0. $\mathrm{Pr}(y_{ij}=1|{\theta }_i,{\alpha }_j,{\beta }_j,{\gamma }_j)$ in 3P CTM Equation (12) converges to ${\theta }_i+{\gamma }_j$, and $\mathrm{Pr}\left( y_{ij}=1|{\theta }_i,{\alpha }_j, {\beta }_j\right)$ in 2P CTM Equation (13) converges to ${\theta }_i$ as ${\alpha }_j$ approaches to 0, according to L’Hôpital’s rule (e.g., Larson et al., 1999).

Figure 1 depicts a selection of 2P CTM Task Characteristic Curves (Item Characteristic Curves (ICCs) in the context of IRTMs) based on Equation (13). One of the well-known and attractive features of IRTMs is that the item location and person parameters can be presented on the same scale. For the proposed CTMs, the difficulty of an item also represents a location along the latent continuum, in this case from 0 to 1. As such, β_j = 0 represents the lowest location for item j, that is, the easiest item that everyone is expected to answer correctly, while β_j = 1 represents the highest location for item j, that is, the most difficult item, expected to be correctly answered only by a person at the mastery level.

The second technical facet is the CTM discrimination parameter α_j, which describes how well an item can differentiate between the people with traits below the item location and those with traits above the item location. Figure 2 depicts 2P CTM link function based on Equation (13), showing the different item discriminations. As with the traditional IRTMs, the discrimination parameter of the proposed model also reflects the steepness of the link function, such that the greater the value of α_j, the steeper the curve, and hence the better the item can discriminate at that location along the θ continuum. Conversely, the lower the value of α_j, the flatter the curve, and hence the less effectively the item can discriminate, because the probability of a correct response at low trait levels is similar to that at high trait levels.

The third component of the link function is the pseudo-guessing parameter γ_j, which conventionally characterizes the probability of the least capable person obtaining the correct answer due to guessing. Figure 3 depicts several 3P CTM link functions, based on Equation (12), constructed to show the different pseudo-guessing parameters. In the traditional 3PL models, this parameter represents the height of the lower asymptote of an ICC, and it shows the probability of y_ij = 1 for a person with a very low (lower asymptote) trait score. For the 3P CTM, however, γ_j represents the probability of getting a correct response when a person is at the lowest boundary, $\mathrm{Pr}(y_{ij}=1|{\theta }_i=0)$, without introducing a low-left asymptote concept.

Table 1. Trait Level Interpretations over Different Measurement Models.

	Classical Test Theory Model	Diagnostic Classification Model	Logistic IRT Model and Factor Model	Cognitive Trait Model
Type	Continuous	Discrete	Continuous	Continuous
Range	0 to total score	0 or 1	(−∞, +∞)	[0, 1]
Interpretation of 0	All answers were wrong	No-mastery status	Mean of trait *	Ignorance level: The level of trait at which none of the tasks are expected to be successfully performed.
Interpretation of 0.5	Half of answers were correct	50% chance of mastery status **	Half SD above mean of trait *	Half of the mastery: The level of trait at which 50% of the tasks are expected to be successfully performed
Interpretation of 1	All answers were correct	Mastery status	One SD above mean of trait *	Mastery level; The level of trait at which all of the tasks are expected to be successfully performed

* 50% chance of answering correctly for the item difficulty value equal to the score 0 for 1, 2PL IRTMs; ** Probability of mastery status.

summarizes the types and interpretations of the scores over the different models.

2.3. Likelihood Function and Markov Chain Monte Carlo Estimator

In the same way as those used in the traditional models, the likelihood function of the proposed models can be defined analytically without great difficulty. Furthermore, by virtue of previous work developing the estimator for the traditional model, it can be relatively straightforward to develop the Marginal Maximum Likelihood (MML; ref. [18]) estimator and other Bayesian estimates (expected a posteriori [EAP], maximum a posteriori [MAP]; refs. [17,19,20]), and MCMC for the proposed models.

During the last few decades, the MCMC estimation option has become an explosively popular technique for estimating a variety of statistical models, including IRTMs. Most importantly, this simulation-based estimation option enables one to simultaneously estimate both the items’ and the subjects’ parameters. Recently, in the context of IRTMs, this MCMC estimation has proved to be useful by providing great flexibility in dealing with a variety of modeling situations, including polytomous responses [21], nominal responses [22], missing data [21], hierarchical models [23], and so forth. For a more thorough treatments of the MCMC estimation in other psychometric models, see [24].

Beyond these modeling flexibilities, most importantly, an MCMC estimator can simultaneously estimate both the item and the person parameters, so that all of the parameter estimates appropriately reflect the estimation uncertainties of the other parameters. Furthermore, the MCMC estimation for the proposed models is as straightforward as the traditional model cases, as will be shown using the general purpose and freely available MCMC software, WinBUGS [25] to estimate both the person and the item parameters for the proposed models.

2.4. MCMC Specifications

The joint density specification for MCMC estimation of a 3P CTM can be expressed as:

$${\displaystyle \prod _{i=1}^n{\displaystyle \prod _{j=1}^J\mathrm{Pr}(y_{ij}|{\theta }_i,{\alpha }_j,{\beta }_j,{\gamma }_j)}}\mathrm{Pr}({\theta }_i|a_{\theta }^{*},b_{\theta }^{*})\mathrm{Pr}({\alpha }_j|{\mu }_{\alpha },{\sigma }_{\alpha }^2)\mathrm{Pr}({\beta }_j|a_{\beta }^{*},b_{\beta }^{*})\mathrm{Pr}({\gamma }_j|a_{\gamma }^{*},b_{\gamma }^{*}),$$

(15)

where

$$y_{ij}|{\theta }_i,{\alpha }_j,{\beta }_j,{\gamma }_j~Bern[\mathrm{Pr}(y_{ij}=1|{\theta }_i,{\alpha }_j,{\beta }_j,{\gamma }_j)],$$

(16)

$$\mathrm{Pr}(y_{ij}=1|{\theta }_i,{\alpha }_j,{\beta }_j,{\gamma }_j)={\gamma }_j+(1-{\gamma }_j)\times \frac{1-\exp ({\alpha }_j\times {\theta }_i)}{1+\exp ({\alpha }_j({\theta }_i-{\beta }_j))}\times \frac{1+\exp ({\alpha }_j(1-{\beta }_j))}{1-\exp ({\alpha }_j)},$$

(17)

$${\theta }_i~Beta(a_{\theta }^{*},b_{\theta }^{*}),$$

(18)

$${\alpha }_j~N^{+}({\mu }_{\alpha },{\sigma }_{\alpha }^2)\mathrm{or}LN({\mu }_a,{\sigma }_a^2),$$

(19)

$${\beta }_j~Beta(a_{\beta }^{*},b_{\beta }^{*}),\mathrm{and}$$

(20)

$${\gamma }_j~Beta(a_{\gamma }^{*},b_{\gamma }^{*}).$$

(21)

Several aspects of the above specification are worth addressing. First, in contrast with the traditional IRT model, the proposed models assume that the latent variable follows a Beta distribution, with two hyper-parameters $a_{\theta }^{*}$ and $b_{\theta }^{*}$. These two parameters can be determined empirically or purposefully by a researcher. Note that, while the standard Beta distribution with a [0, 1] support interval can be easily generalized to an arbitrary interval using a linear transformation, the standard Beta distribution is adapted in this paper for modeling the person and the location parameter specifically with a [0, 1] interval. Note also that the traditional models (e.g., 2PL) would allow the discrimination parameter to be negative for fitting items with incorrectly scored. However, the CTM α parameter settings in Equation (21) only allow positive slopes. Unlike the traditional IRTMs, the CTMs only allow a positive slope, and as such cannot detect the aberrant items that are negatively related to the construct.

Many practical applications of the traditional models specify arbitrary values for the hyper-parameters to resolve the location and scale indeterminacies; the most popular choice is a normal mean = 0 and variance = 1 for the MML or MCMC estimators. Likewise, the CTMs also require one to specify values for $a_{\theta }^{*}$ and $b_{\theta }^{*}$, such as $a_{\theta }^{*}=2$ and $b_{\theta }^{*}=2$. If these hyper-parameters are unknown, these can also be handled by hyper-prior distribution(s). For the discrimination parameters in (21), recognize that ${\alpha }_j$ is often used to specify a prior distribution that is restricted to the positive real line either with N⁺ (a normal distribution truncated to the positive real line distribution) or with LN (a log-normal distribution). Again, the hyper-parameters ${\mu }_{\alpha }$ and ${\sigma }_{\alpha }^2$ are determined by the researcher, and the distributional specification of the discrimination parameter for the proposed models is the same as that of the traditional IRTMs. While the traditional models assume an unbounded continuum for the difficulty or location parameter, for the proposed model ${\beta }_j$, it is continuous but is bounded (from 0 to 1). Therefore, as seen in Equation (17), it is natural to adapt a [0, 1] continuous distribution, such as a standard Beta distribution, where $a_{\beta }^{*}$ and $b_{\beta }^{*}$ are the hyper-parameters that can be specified by the researcher. Therefore, similar to the traditional models, the difficulty and the person parameters are also located on the same [0, 1] continuum for the proposed models. Finally, in the traditional models, the pseudo-guessing parameters are the lower asymptote parameters and reflect the probability of a correct answer on the item when the proficiency is very low. For the proposed models, the pseudo-guessing parameters ${\gamma }_j$ are also bounded by 0 and 1, and the Beta distribution can be the natural choice, as seen in Equation (21) with the two hyper-parameters, $a_{\gamma }^{*}$ and $a_{\gamma }^{*}$. The next section of the paper illustrates the simultaneous estimation of the person and item parameters for the CTMs, using the MCMC estimator within the WinBUGS software [25].

3. Illustrations

3.1. Illustration with a Simulated Dataset

In this part of the illustration, I analyzed a simulated item responses dataset to evaluate the parameter replicability of the proposed models. In other words, I investigated how well the proposed model parameters are replicated, using a simulated dataset generated from the known item and person parameters. The details of these analysis steps are as follows.

First, generate 1000 person parameters from Beta(2, 2) distribution. Second, using the generated person parameters and the 20 items parameter values in Table 1, compute the probability of getting the right answer for all of the items using Equation (13). Third, generate the 20 item response data for each person as:

$$\{\begin{array}{l}{y}_{ij}=1\mathrm{if}\mathrm{Pr}(y_{ij}=1|{\theta }_i,{\alpha }_j,{\beta }_j)>Uni(0,1)\\ y_{ij}=0\mathrm{if}\mathrm{Pr}(y_{ij}=1|{\theta }_i,{\alpha }_j,{\beta }_j)\le Uni(0,1)\end{array},$$

(22)

where Uni (0, 1) is a uniform distribution ranges from 0 to 1. Then, fit the 2P CTM in Equation (13) with the following prior distribution specifications (assuming prior independence among the parameters):

$${\theta }_i~Beta(2,2),$$

(23)

$$\begin{array}{c}{\alpha }_j~LN(7.5,0.1),\\ {\beta }_j~Beta(2,2).\end{array}$$

(24)

Identical to the empirical data illustration case, I used the same $Beta(2,2)$ for the prior distributions of both the $\theta$ and $\beta$ parameters, and these prior distributions for the latent variable and item parameters. A log-normal distribution $LN(7.5,0.1)$ is also used for each ${\alpha }_j$ prior distribution (Appendix A gives the WinBUGS code for this analysis). The model was run with three chains with different starting values which are determined by the software. The first 2000 iterations were discarded as burn-in. Each chain was run for 3000 more iterations after burn-in, yielding 9000 iterations for use in summarizing the marginal distributions for all of the parameters. Convergence under the MCMC estimation has a different meaning than under the ML methods: a Markov chain is considered converged when it becomes stationary, and this can be examined visually with trace plots and the diagnostic statistics developed by Brooks–Gelman–Rubin (BGR; ref. [26]).

In the outputs, no chain gets stuck in certain areas of the parameter space for all of the parameters, and all three chains of each parameter are mixed together. In other words, in this simulation data analysis, no trace plot of Markov chain was found unstationary after the burn-in period. Another way to visually inspect the convergence is the BGR diagnostic statistic. This ANOVA-type diagnostic compares within- and among-chain variance, and WinBUGS displays the BGR diagnostic statistic (red-line) when the “bgr diag” button is pressed in the “Sample monitor tool”. The values around 1 indicate convergence, with 1.1 considered an acceptable limit [26,27]. The BGR diagnostic statistic also showed satisfactory convergence as the ratio (red) curves for all of the parameters approach 1. The marginal densities for the parameters are mostly unimodal and asymmetric, although some are asymmetric. Due to the space limitations, these trace plots are not provided in this paper. The results in their entirety can be requested from the author of this paper.

Table 2. Simulation Data Marginal Density Summary Statistics.

Para.	True	Mean	Median	SD	Para.	True	Mean	Median	SD	Para.	True	Mean	Median	SD
α1	5.000	5.166	5.083	0.600	β1	0.100	0.107	0.107	0.060	θ1	0.320	0.287	0.286	0.072
α2	5.000	4.784	4.717	0.704	β2	0.200	0.163	0.172	0.071	θ2	0.281	0.270	0.268	0.069
α3	5.000	4.316	4.289	0.753	β3	0.300	0.211	0.227	0.079	θ3	0.133	0.157	0.152	0.060
α4	5.000	4.105	4.137	0.837	β4	0.400	0.302	0.322	0.077	θ4	0.814	0.729	0.732	0.067
α5	5.000	3.674	3.873	1.221	β5	0.500	0.544	0.533	0.083	θ5	0.261	0.233	0.230	0.070
α6	5.000	3.389	3.438	0.927	β6	0.600	0.667	0.636	0.097	θ6	0.429	0.506	0.508	0.075
α7	5.000	3.465	3.376	0.541	β7	0.700	0.856	0.857	0.081	θ7	0.109	0.123	0.118	0.056
α8	5.000	4.690	4.662	0.759	β8	0.800	0.796	0.781	0.073	θ8	0.439	0.411	0.409	0.075
α9	5.000	4.736	4.690	0.373	β9	0.900	0.944	0.953	0.042	θ9	0.566	0.662	0.663	0.071
α10	5.000	5.480	5.413	0.463	β10	1.000	0.937	0.944	0.044	θ10	0.419	0.321	0.319	0.073
α11	10.000	9.890	9.733	1.060	β11	0.100	0.063	0.062	0.037	θ11	0.344	0.352	0.349	0.076
α12	10.000	11.740	11.680	1.458	β12	0.200	0.187	0.190	0.025	θ12	0.641	0.755	0.758	0.068
α13	10.000	10.120	10.090	0.968	β13	0.300	0.304	0.305	0.016	θ13	0.764	0.761	0.765	0.064
α14	10.000	10.790	10.760	0.962	β14	0.400	0.391	0.391	0.013	θ14	0.863	0.862	0.867	0.055
α15	10.000	9.271	9.262	0.826	β15	0.500	0.480	0.480	0.013	θ15	0.935	0.893	0.899	0.051
α16	10.000	10.940	10.910	0.969	β16	0.600	0.600	0.600	0.012	θ16	0.748	0.762	0.764	0.065
α17	10.000	9.936	9.911	1.017	β17	0.700	0.721	0.720	0.019	θ17	0.637	0.615	0.616	0.075
α18	10.000	9.509	9.487	1.147	β18	0.800	0.831	0.825	0.034	θ18	0.492	0.542	0.543	0.076
α19	10.000	10.600	10.500	1.352	β19	0.900	0.900	0.895	0.040	θ19	0.303	0.316	0.314	0.073
α20	10.000	12.460	12.250	1.358	β20	1.000	0.957	0.961	0.029	θ20	0.642	0.570	0.571	0.076

gives the summary statistics of the marginal distributions for the item parameters and the first twenty person parameters. Table 2 also includes the true values for the item and person parameters. I examined the following summary statistics regarding the parameter replicability for the 2P CTM with the simulation data. First, I examined the correlation between the true parameter values and the point (i.e., the median of the marginal density) estimates. Second, for evaluating the bias of point estimates, I employed mean bias (MB) and mean relative bias (MRB) as:

$$MB={\displaystyle {\sum }_{i=1}^{NP}\left[{\widehat{\rho }}_i-{\rho }_i\right]}/NP,$$

(25)

$$MRB={\displaystyle {\sum }_{i=1}^{NP}\left[({\widehat{\rho }}_i-{\rho }_i)/{\rho }_i\right]}/NP,$$

(26)

where ${\rho }_i$ was the true value of ith parameter; ${\widehat{\rho }}_i$ was the ith parameter point estimate given the estimation method converged; and NP was the number of parameters. Third, for evaluating the point estimates, I examined the root mean squared error (RMSE) with the following definitions:

$$RMSE={\left[{\displaystyle {\sum }_{i=1}^{NP}{({\widehat{\rho }}_i-{\rho }_i)}^2/NP}\right]}^{1/2}.$$

(27)

As seen in

Table 3. Statistics for the Parameter Replicability Analysis.

Parameter	NP	MB	MRB	RMSE	Pearson’s r
α	20	−0.087	−0.040	0.962	0.968
β	20	−0.002	−0.032	0.050	0.987
θ	1000	−0.004	0.021	0.071	0.947

, the MB values are low (i.e., less than 0.1) for all three parameters. The MB values for the $\beta$ and $\theta$ parameters are especially low (i.e., less than 0.01). All three of the parameters’ MRB values are also low (i.e., less than 0.05; that is, the bias size relative to a parameter value is less than 5%). The $\beta$ and $\theta$ parameters’ RMSE values are also low (i.e., less than 0.05). The RMSE values (i.e., 0.962; more than 0.05) of the α parameter shows the parameter estimation is relatively unstable than the $\beta$ and $\theta$ parameters. It is recommended that readers evaluate the proposed model by noting the instability of the α parameter estimation. Lastly, the correlations between the true parameter values and the estimates are high (i.e., greater than 0.947) for all three parameters. In Figure 4, the scatter plots of true person trait values versus the 2P CTM estimates are shown, that also represent a linear relationship between the true values and the estimates.

Table 4. Correlation Table of the Simulation Data Person Parameter Estimates.

		${\mathit{\theta }}_{\mathit{T}\mathit{r}\mathit{u}\mathit{e}}$	${\widehat{\mathit{\theta }}}_{\mathit{C}\mathit{P}}$	${\widehat{\mathit{\theta }}}_{2\mathit{P}\mathit{L}}$	${\widehat{\mathit{\theta }}}_{2\mathit{P}\mathit{T}\mathit{L}}$
${\theta }_{True}$	Pearson’s r	1.000	0.944 **	0.944 **	0.947 **
	Spearman’s rho	1.000	0.945 **	0.947 **	0.947 **
${\widehat{\theta }}_{CP}$	Pearson’s r	0.944 **	1.000	0.993 **	0.995 **
	Spearman’s rho	0.945 **	1.000	0.996 **	0.995 **
${\widehat{\theta }}_{2PL}$	Pearson’s r	0.944 **	0.993 **	1.000	0.997 **
	Spearman’s rho	0.947 **	0.996 **	1.000	1.000 **
${\widehat{\theta }}_{2PTL}$	Pearson’s r	0.947 **	0.995 **	0.997 **	1.000
	Spearman’s rho	0.947 **	0.995 **	1.000 **	1.000

** Correlation is significant at the 0.01 level (2-tailed); ${\theta }_{True}$ is the true trait given 2P CTM is correct model; ${\widehat{\theta }}_{CP}$ is the correct answer ratio (i.e., the number of correctly answered items/the total number of items) estimate; ${\widehat{\theta }}_{2PL}$ is 2PL estimate; ${\widehat{\theta }}_{2PTL}$ is 2P CTM estimate.

also provides several interesting aspects of the person parameter estimates among the different models. First, ${\theta }_{True}$ is highly correlated with all of the other three estimates with similar degrees (i.e., around 0.94 coefficients). Second, the Spearman’s rho coefficient between ${\widehat{\theta }}_{2PL}$ and ${\widehat{\theta }}_{2PTL}$ is perfect as 1. In other words, the rank order between the two model estimates is exactly the same in the simulated data case (especially for these 20 items). Third, ${\widehat{\theta }}_{2PTL}$ is highly correlated (i.e., >0.99) with ${\widehat{\theta }}_{CP}$, but the correlation coefficients are less than those between ${\widehat{\theta }}_{2PL}$ and ${\widehat{\theta }}_{2PTL}$. That means, although ${\widehat{\theta }}_{CP}$ and ${\widehat{\theta }}_{2PL}$ are both a score from 0 to 1 and highly correlated, ${\widehat{\theta }}_{2PL}$ and ${\widehat{\theta }}_{2PTL}$ are more statistically similar.

The interpretation of the high correlations between the parameters of the two model frameworks is as follows. CTM’s link function was analytically derived by transforming IRTM’s link function, the logistic function. Therefore, it is quite natural that the parameters of the two models have similar properties and are highly correlated. However, because the individual parameter in CTM has a certain value in the 0 to 1 bounded continuum, the ‘conceptualization’ and ‘interpretation’ of the parameter are different. Thus, the usage would be different for the different purposes of the assessments, such as summative vs. formative assessments. These different usages will be described in more detail in the discussion section.

Note that both ${\widehat{\theta }}_{2PL}$ and ${\widehat{\theta }}_{2PTL}$ are based on LVMs, and ${\widehat{\theta }}_{CP}$ is an average of the observed variables. For example, in the simulation dataset, the 809th and 231st subjects’ ${\widehat{\theta }}_{CP}$ are both 0.8 (80% correct). However, the ${\widehat{\theta }}_{2PTL}$ are different as 0.8064 (interpreted as 80.640% of the mastery level, or the trait level at which 80.64% of the tasks are expected to correctly be answered) for the 809th subject with a response vector {1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0} and 0.7741 (interpreted as 77.41% of the mastery level, or the trait level at which 77.41% of the tasks are expected to correctly answered) for the 231st subject with a response vector {1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0}. This means that, although the averages of the observed variables (${\widehat{\theta }}_{CP}$) are the same, the LVM-based estimates (${\widehat{\theta }}_{2PTL}$) can be different, because ${\widehat{\theta }}_{2PTL}$ is taking into account the different characteristics (i.e., difficulties and/or discriminations) of the items (see Table 2 for the population item difficulty parameter values). For the 211st subject with a response vector {1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0}, ${\widehat{\theta }}_{2PL}$ = −1.284 and ${\widehat{\theta }}_{2PTL}$ = 0.2037. The ${\widehat{\theta }}_{2PL}$ value of −1.284 provides a relative standing information as “1.284 SD below than average”. However, the ${\widehat{\theta }}_{2PTL}$ value of 0.2037 can be interpreted as “20.37% of the mastery level, or the trait level at which 20.37% of the tasks are expected to be correctly answered”. In sum, this simulated dataset analysis demonstrates that it is possible not only to satisfactorily estimate all of the item and person parameters, but also to successfully replicate the proposed 2P CTM parameters.

3.2. Illustration with an Empirical Dataset

To illustrate the proposed models, I analyzed the item responses data originally provided by [28]. These were responses from 1000 examinees to five dichotomously scored items on the Section 6 of the Law School Admissions Test (LSAT). There are two reasons for choosing this example as follows. First of all, this example has been used in a wide variety of LVMs, especially IRTM studies. Second, all of the items were in the multiple choice format with five response options. Therefore, the 3P CTM in Equation (12) can be demonstrated with this dataset.

For the MCMC estimation, the model was fitted with the following prior distribution specifications (assuming prior independence among the parameters):

$${\theta }_i~Beta(2,2),$$

(28)

$${\alpha }_j~LN(5,10),$$

(29)

$${\beta }_j~Beta(2,2),\mathrm{and}$$

(30)

$${\gamma }_j~Beta(7,25).$$

(31)

The same $Beta(2,2)$ is used for both the $\theta$ and $\beta$ parameters’ prior distributions, to reflect a moderately strong and symmetric belief on the parameters around 0.5. Obviously, other a priori knowledge and empirical evidences of the $\theta$ and $\beta$ distributions can be incorporated by choosing different values of the shape parameters. $Beta(2,2)$ is symmetric with 0.5 mean, variance as $({\alpha }^{*}\times {\beta }^{*})/\left\{{({\alpha }^{*}+{\beta }^{*})}^2\times ({\alpha }^{*}+{\beta }^{*}+1)\right\}=$ $4/80=0.05$, and it is flatter than the normal distributions (i.e., less informative or relatively diffuse than the normal prior). Given the assumption that the chance of having a correct response increases monotonically with latent trait, using a log-normal normal distribution, ${\alpha }_j~LN(5,10)$, which is also relatively diffuse, ensures that each ${\alpha }_j>0$.

Because the assessment is a five-response option multiple choice format, as a moderately informative prior out of several distributions, $Beta(7,25)$ has been adapted as the prior distribution of $\gamma$. $Beta(7,25)$ mode is $({\alpha }^{*}-1)/({\alpha }^{*}+{\beta }^{*}-2)=$ $6/30=0.2$, mean is ${\alpha }^{*}/({\alpha }^{*}+{\beta }^{*})=$$7/32=0.21875$, and the variance is $({\alpha }^{*}\times {\beta }^{*})/\left\{{({\alpha }^{*}+{\beta }^{*})}^2\times ({\alpha }^{*}+{\beta }^{*}+1)\right\}$$=$ $175/33792\approx 0.00517$. Note that $Beta(7,25)$ is a Beta distribution for reflecting the five choices item pseudo-guessing parameter (i.e., 1/Number of Choices = 1/5 = 0.2). Other Beta distributions could be adapted to model the multiple choice items, e.g., Beta distributions with mode as $({\alpha }^{*}-1)/({\alpha }^{*}+{\beta }^{*}-2)=$ 1/Number of Choices. Note that the sensitivity analysis of the prior selection on the estimates would be an important issue in MCMC. In general, the prior sensitivity is a function of the data size, i.e., a prior selection will be more influential to the estimates as the data size gets smaller.

The WinBUGS code for this analysis is given in Appendix A. The model was run with five chains. The first 1000 iterations were discarded as burn-in, and each chain was run for 1000 more iterations after burn-in, yielding 5000 iterations for use in summarizing the marginal distributions for all of the parameters. The trace plot (i.e., chain history) is a plot of the MCMC iteration number against the value of the draw of the parameter at each iteration. In the chain history, no chain gets stuck in certain areas of the parameter space, and all five chains of each parameter are mixed together. In this dataset result, the ratio (red) curves for all of the parameters approach to 1. In the MCMC outputs, the marginal densities are mostly unimodal and asymmetric, with some of them are asymmetric. Due to space limitations, the chain history and BGR diagnostic plots are not provided in this manuscript. The results in their entirety can be requested from the author of this paper. Accordingly,

Table 5. LSAT Data Marginal Density Summary Statistics.

	3PL IRTM					3P CTM
Parameter	Mean	Median	SD	2.5% Perc.	97.5% Perc.	Parameter	Mean	Median	SD	2.5% Perc.	97.5% Perc.
A1	1.08	1.03	0.31	0.65	1.83	α1	8.86	8.60	8.60	1.70	6.52
a2	0.96	0.88	0.42	0.47	2.04	α2	1.25	0.95	0.95	1.17	0.02
a3	1.41	1.26	0.65	0.62	3.26	α3	2.61	2.18	2.18	2.44	0.06
a4	0.84	0.79	0.29	0.46	1.54	α4	1.98	1.96	1.96	1.42	0.03
a5	0.85	0.83	0.22	0.49	1.36	α5	5.21	5.19	5.19	1.03	3.33
b1	−2.50	−2.45	0.52	−3.64	−1.59	β1	0.08	0.08	0.08	0.04	0.01
b2	−0.58	−0.61	0.43	−1.38	0.30	β2	0.43	0.42	0.42	0.21	0.08
b3	0.34	0.33	0.24	−0.11	0.84	β3	0.67	0.70	0.70	0.18	0.21
b4	−1.10	−1.09	0.42	−1.94	−0.27	β4	0.35	0.32	0.32	0.20	0.05
b5	−2.13	−2.06	0.51	−3.29	−1.27	β5	0.13	0.11	0.11	0.08	0.02
c1	0.26	0.25	0.10	0.10	0.47	γ1	0.33	0.33	0.33	0.10	0.16
c2	0.26	0.25	0.10	0.09	0.49	γ2	0.38	0.39	0.39	0.05	0.26
c3	0.23	0.22	0.08	0.09	0.37	γ3	0.20	0.19	0.19	0.07	0.09
c4	0.25	0.25	0.09	0.09	0.46	γ4	0.45	0.47	0.47	0.07	0.29
c5	0.27	0.26	0.10	0.10	0.48	γ5	0.44	0.44	0.44	0.10	0.26
θ1	−1.75	−1.75	0.75	−3.26	−0.26	θ1	0.14	0.12	0.12	0.09	0.02
θ2	−1.75	−1.76	0.75	−3.22	−0.26	θ2	0.14	0.13	0.13	0.08	0.02
θ3	−1.74	−1.72	0.74	−3.21	−0.30	θ3	0.14	0.13	0.13	0.08	0.02
θ4	−1.36	−1.35	0.77	−2.89	0.12	θ4	0.19	0.17	0.17	0.11	0.03
θ5	−1.36	−1.35	0.77	−2.91	0.12	θ5	0.19	0.17	0.17	0.11	0.03

contains the summary statistics of the marginal distributions for the item parameters and the first five-person parameters. The overall analysis of the results not only shows that the 3P CTM (all of the item and person parameters) can be estimated satisfactorily with the LSAT data, but also shows that one can straightforwardly analyze the proposed models via the MCMC estimation option with conventional MCMC software, such as WinBUGS (Appendix A gives the code for this analysis).

For the guessing parameters in Table 5, the CTM guessing parameters tend to be bigger than those from the PL model. This result also clearly shows the differences between the corresponding parameters of the two models. In the case of the PL model, the guessing parameter value (c₁ = 0.26) can be interpreted as the probability of getting the right answer for the first item is 0.26 for someone whose latent trait level is negatively very low (theoretically, it requires the low-left asymptote concept). In the case of the CTM, the CTM guessing parameter estimate (γ₁ = 0.33) can be straightforwardly interpreted as the probability of getting the right answer for the first item is 0.33 for a person at the lowest (a trait level for 0% chance of successfully completing any task) level.

Table 6. Correlation Table of the LSAT Data Person Parameter Estimates.

		${\widehat{\mathit{\theta }}}_{2\mathit{P}\mathit{L}}$	${\widehat{\mathit{\theta }}}_{2\mathit{P}\mathit{T}\mathit{L}}$	${\widehat{\mathit{\theta }}}_{3\mathit{P}\mathit{L}}$	${\widehat{\mathit{\theta }}}_{3\mathit{P}\mathit{T}\mathit{L}}$
${\widehat{\theta }}_{2PL}$	Pearson’s r	1.000	0.953 **	0.985 **	0.940 **
	Spearman’s rho	1.000	0.869 **	0.948 **	0.922 **
${\widehat{\theta }}_{2PTL}$	Pearson’s r	0.953 **	1.000	0.987 **	0.986 **
	Spearman’s rho	0.869 **	1.000	0.896 **	0.918 **
${\widehat{\theta }}_{3PL}$	Pearson’s r	0.985 **	0.987 **	1.000	0.980 **
	Spearman’s rho	0.948 **	0.896 **	1.000	0.957 **
${\widehat{\theta }}_{3PTL}$	Pearson’s r	0.940 **	0.986 **	0.980 **	1.000
	Spearman’s rho	0.922 **	0.918 **	0.957 **	1.000

** Correlation is significant at the 0.01 level (2-tailed).

also provides the association measures of the person parameter estimates among the different models. Note that all of Spearman’s rho coefficients between the IRTM and CTM estimates are high (i.e., >0.85), but those are not perfect 1 as in Table 4. In other words, the ranking information between the two models (i.e., 2PL IRTM and 2P CTM; 3PL IRTM and 3P CTM) are not exactly same for this empirical data simulation dataset. This difference would be due to the difference in the number of items (i.e., 20 items vs. 5 items). In sum, this simulated dataset analysis demonstrates that it is possible to satisfactorily estimate all of the 3P CTM parameters.

4. Discussion

This paper makes several contributions to psychometrics. First, this paper proposes a perspective of operationalizing the quantity of a cognitive domain trait (such as knowledge, skill, ability, expertise, intelligence, achievement, competence, and proficiency) as a level between two boundaries. (i.e., the lowest ignorance level and the highest mastery level). This perspective enables us to conceptualize and interpret the trait levels as a relative magnitude to the mastery level, instead of relative standings on an unbounded continuum. The perspective makes us rethink the question of “relative standings or magnitudes?” on cognitive trait levels and provides an option to cope with it. The placement of a person parameter’s boundaries as 0 and 1 allows us to conceptualize a person’s ability level, based on the construction’s 0 (ability level where no tasks can be performed correctly) and 1 (ability level where all tasks are performed correctly). Thus, such conceptualization of the person parameter value facilitates the usages and interpretations in domain mastery and/or learning growth over time/stimulus of a person. This new framework will be able to better respond to the demands of digital assessments, which require th e interpretation and analysis of individual growth and change beyond inter-individual comparisons, such as formative assessment, learning progression/growth, or adaptive learning systems.

Second, for the above point of view, this paper presents an analytical framework for cognitive domain traits: cognitive trait models (CTM) which have supports on the [0, 1] latent continuum using truncated logistic functions. After illustrating how the new model parameters can be estimated and replicated via the MCMC estimation method within a Bayesian framework, the second primary contribution becomes possible. Specifically, it is possible to simultaneously estimate both the item and the person parameters by making minor modifications of the MCMC specifications of the traditional model (two illustrations were exemplified, with program syntaxes provided). The implementation illustrations provide modeling accessibility to applied researchers who are interested in adopting the proposed models.

Third, due to the popularity of the logistic function family, the above contributions of this paper have great potential to be extended beyond psychometrics. Since initially introduced by [29], the logistic function and variants of it have proved useful for modeling a wide variety of phenomena in diverse research domains, such as population ecology [30], technological innovation infusion [31], biological and genetics [32], artificial neural networks [33,34], and nonlinear growth processes [35,36]. Due to such a wide range of applications of the logistic function family, the proposed framework with [0, 1] support may have great potential to provide theoretically and/or practically sound analytical option in diverse fields.

Note that, as articulated earlier, the link function of CTM is analytically derated from the IRTMs’ logistic function. Therefore, the mathematical/analytical properties of the two model frameworks are similar. However, because the person parameters of CTM are bound to 0 and 1, the interpretations and usages are different from those of the IRTMs. The CTM is useful for criterion/domain-referenced and/or within-person comparisons, such as formative assessment, learning analytics, and learning progression model applications. In addition, the IRTMs, as is already well known, are useful for the use and interpretation of norm-referenced or summative assessment/testing, with the primary purpose of operationalizing the differences between the subjects. It is important to understand the difference between the two modeling frameworks, and accordingly to use them appropriately for the purposes of making relevant interpretations.

It should be noted that the proposed framework requires the clarity and stability of the mastery level, which is a cognitive level where all of the tasks are expected to be successfully performed. Therefore, the meanings and interpretations of the CTM model parameters are subject to the degree of identification and specifications of the domain of the tasks (as the traditional framework does). Note that the interpretations and inferences of the CTMs also require the assumption of “random samples from the domain of tasks”, as the other traditional frameworks do. In the proposed framework, because the trait quantity is a number from the 0 to 1 continuum, the uncertainties of the domain of tasks (especially in the top range) would impact the whole metric of a trait. Analytical investigations, adjustments, and treatments for such issues (for example, more difficult or easy tasks are added to the domain; equating or vertical scaling) remain as future research topics.

Although the applicability of the proposed models is imminent for the traits in the cognitive domain, it is not imminent for the traits in the affective domain (e.g., beliefs or attitudes; ref. [2]). In the affective domain, a negative estimate from the traditional models could be more useful to conceptualize or interpret. An attempt to integrate the proposed framework into the affective domain induces bounded affective continuums, such as from “completely not believe” to “completely believe”, or from “completely disagree” to “completely agree”. Such applications would require a substantial amount of theoretic considerations for each affective trait, and remains as future work for the experts.

In the last few decades, IRTMs [6] based on the logistic function with unbounded support have been popular measurement models in the educational and psychological domain. Although this study focuses on introducing a new conceptualization of the trait level itself, the proposed models can be considered as a type of IRTM, so called Truncated Logistic IRTMs (i.e., 1, 2, 3PTL IRTMs). From this point of view, the straightforward connection between the person and item difficulty parameters (i.e., when θ = β, the probability of getting an item right is always 0.5) is lost in CTMs. In other words, the probability of getting an item right when θ = β depends on α in the CTMs. Considering the CTMs as a type of IRTM would lead to a need for IRTM-specific features and analyses, such as test characteristic curves (TCCs), item information functions (IIFs), test information functions (TIFs), standard error of measurement (SEM), or model-parameter invariance investigation, etc. [37,38]. In particular, the TCC and TIF for the CTMs will be useful improvements, because they allow us to investigate the relationship between observations and trait levels at the level of the task domain, not at the item level. Furthermore, a didactic presentation with a walkthrough of the modification on the original and the transformed sigmoid with other dataset examples would be beneficial to someone who would like to understand more details of the analytic characteristics of the CTM. Such extensions and articulations are left to future research.

Beyond the conceptualization and interpretability of the model parameters, other modeling aspects (e.g., the robustness of the parameter invariance, model fits, sample or item size requirements) should be further evaluated for someone who is considering the new models.

It would be interesting to investigate how the CTM trait levels are numerically different from the DCM or the Beta-Binomial model [9] estimates. Note that other traditional LVMs (such as IRTMs, DCMs, or FMs) and CTMs are not within a nested structure. Therefore, the comparisons of these aspects would require an intense series of investigations (e.g., Monte Carlo simulation studies varying several conditions: ground truth models for generating data, sample sizes, and/or number of tasks, etc.) which also remain subject to future work. Lastly, the traditional psychometric modeling options have been vigorously extended, including multi-latent traits, polytomous observables, longitudinal, and multi-level/nested structure, etc. [39]. As this paper has sought to demonstrate, one may implement these extensions for the proposed analytical framework by virtue of the Bayesian framework, which is flexible in dealing with a variety of modeling situations. Such developments are eagerly awaited.