How the heck do you screw up a test this bad? Trying to understand my distribution of scores.

Multiple choice items are fine, but multiple-multiple choice items are frowned upon by the overwhelming majority of assessment experts & psychometricians. Might be worth checking out Haladyna & Downing's work (e.g., 1989, 1993, 2002) as a starting point.

My take is that multiple-multiple choice items (M-MCI) are common in test banks and are often writen by item writers who haven't had a measurement theory class (e.g., psychometrics, scale development, etc.) because such items are (1) comparatively easy to write, (2) allow the test giver to incorporate a greater amount of material into a single question (thereby reducing the necessary number of items on the exam), and (3) "feel" like "smarter" questions that will better "separate the wheat from the chaff" (or something along that line).

As it turns out, these M-MCI's usually have pretty poor psychometric properties. The usual recommendation is to convert a single M-MCI into two or more regular MCI or T/F questions. That probably goes back at least as far as Cronbach (1944?), but differences in psychometric properties between M-MCI & multiple regular MCI or multiple T/F questions have been studied a fair bit. Splitting the M-MCI into multiple questions (1) increases the test's diagnostic power and (2) reduces constract irrelevant item variance, which often leads to (3) improved item information functions/better discriminability. Basically, we want to be sure that the test taker misses an item because he/she does not know/understand the tapped material (and we want to be clear about precisely what part(s) of the material are unmastered) and not because he/she does not successfully parse the item (which is more of an issue with negatively worded items, especially M-MCI with some positively and some negatively worded distractors). Applied MCI and items that demand critical thinking are great, but increasing item verbal complexity is not the same as writing a well-written, high level MCI.

/r/Professors Thread Parent