Table 8 Summary of problematic items as indicated by the current study’s analysis and Kalinowski et al. (2016)’s IRT analysis
From: Measuring undergraduate evolution learning using the CANS: psychometric strengths and limitations
Item | Psychometric Pattern | Recommended Improvements | |||
|---|---|---|---|---|---|
IRT Factor Loading1 | Unidimensional Rasch Model Fit 2 | Resistance to Instruction2 | Construct Irrelevant Variation2 | ||
4 | Poor factor loading | Acceptable | Not resistant. Item difficulty decreased. | Absent | No specific changes recommended by Kalinowski et al. and our study did not find problems with this item. The item requires reading of an introductory paragraph at the beginning of the section for full context. It may be beneficial to provide this information directly in the question itself. |
5 | Acceptable factor loading | Unacceptable. Unproductive for measurement likely due to redundancy with item 19. | Not resistant. Item difficulty decreased. | Absent | No improvements recommended |
6 | Poor factor loading | Unacceptable. Degrading to measurement | Not resistant. Item difficulty decreased. | Present | Address possible construct irrelevant variation related to the nature of science or remove the item from the instrument. |
12 | Poor factor loading | Unacceptable. Degrading to measurement | Resistant. Item difficulty increased but topic not addressed3 | Present | Address possible construct irrelevant variation related to the nature of science or remove the item from the instrument. |
13 | Poor factor loading | Acceptable | Resistant. Item difficulty remained almost constant. | Absent | Change text in answer option C from “different genes” to “different alleles” or “different versions” of the same gene. |
18 | Poor factor loading | Acceptable | Not resistant. Item difficulty decreased. | Absent | No specific changes recommended by Kalinowski et al. and our study did not find problems with this item. |
19 | Acceptable factor loading | Unacceptable. Unproductive for measurement likely due to redundancy with item 5. | Not resistant. Item difficulty decreased. | Absent | No improvements recommended |
20 | Poor factor loading | Unacceptable. Unproductive for measurement. | Resistant. Item difficulty increased. | Absent | Define seedlings in the introductory paragraph and the question stem. |