Cognitive Assessment Models With Few
Assumptions, and Connections With
Nonparametric Item Response Theory
Brian W. Junker, Carnegie Mellon University
Klaas Sijtsma, Tilburg University
Some usability and interpretability issues for
single-strategy cognitive assessment models are con-
sidered. These models posit a stochastic conjunctive
relationship between a set of cognitive attributes to be
assessed and performance on particular items/tasks
in the assessment. The models considered make few
assumptions about the relationship between latent
attributes and task performance beyond a simple
conjunctive structure. An example shows that these
models can be sensitive to cognitive attributes, even
in data designed to well fit the Rasch model. Several
stochastic ordering and monotonicity properties are
considered that enhance the interpretability of the
models. Simple data summaries are identified that
inform about the presence or absence of cognitive
attributes when the full computational power needed
to estimate the models is not available. Index terms:
cognitive diagnosis, conjunctive Bayesian inference
networks, multidimensional item response theory,
nonparametric item response theory, restricted
latent class models, stochastic ordering, transitive
reasoning.
There has been increasing pressure in educational assessment to make assessments sensitive to
specific examinee skills, knowledge, and other cognitive features needed to perform tasks. For
example, Baxter & Glaser (1998) and Nichols & Sugrue (1999) noted that examinees’ cognitive
characteristics can and should be the focus of assessment design. Resnick & Resnick (1992)
advocated standards- or criterion-referenced assessment closely tied to curriculum as a way to
inform instruction and enhance student learning. These issues are considered in fuller detail by
Pellegrino, Chudowsky, & Glaser (2001).
Cognitive assessment models generally deal with a more complex goal than linearly ordering
examinees, or partially ordering them, in a low-dimensional Euclidean space, which is what item
response theory (
IRT) has been designed and optimized to do. Instead, cognitive assessment models
produce a list of skills or other cognitive attributes that the examinee might or might not possess,
based on the evidence of tasks that he/she performs. Nevertheless, these models have much in
common with more familiar
IRT models.
Interpretability of
IRT-like models is enhanced by simple, monotone relationships between model
parts. For example, Hemker, Sijtsma, Molenaar, & Junker (1997) considered in detail stochastic
ordering of the manifest sum-score by the latent trait (
SOM), and stochastic ordering of the latent trait
by the manifest sum-score (
SOL), in addition to the usual monotonicity assumption (see below). All
three properties are considered here for two conjunctive cognitive assessment models. Additionally,
a new monotonicity condition is considered, which asserts that the more task-relevant skills an
examinee possesses, the easier the task should be.
258
Applied Psychological Measurement, Vol. 25 No. 3, September 2001, 258–272
©2001 Sage Publications
at EASTERN KENTUCKY UNIV on February 10, 2015apm.sagepub.comDownloaded from
剩余14页未读，继续阅读
资源评论
