Ψlogical
Testing

Chapter 6
Item writing & Evaluation

House keeping 🧹🗑

• First exam 3/6 (Thurs)
  • In class, but administered via Canvas (Ch 1 \(\rightarrow\) 8)
• Project:
  • keep revising
  • Hinkin (2005)
  • item writing

🔭🌠ASTEROID YR4 PANIC METER!!!🌌💥

Item writing!!

Terms used:

• Stem – Stimulus (prompt)
• Option – Response (reaction)
• Distractor – Incorrect response option
• Key – scoring algorithm

Item “Types”

Dichotomous vs Polytomous not very common or helpful distinction – we’re not going to use it

• How many response categories are offered
• Has implications, but not good differentiator of different types of items

Common types

• 😇true–false🤥
• forced choice
• multiple choice
• Likert-type
• categories
• checklists

Note

Items may also require scoring rubrics – for example, essay, short-answer, and fill-in-the-blank item types

True–false

• sometimes yes–no
• 50% chance of “getting correct” regardless of construct standing
• familiar
• easy to score
  • aka recoding

Forced choice¹

• aka paired comparisons
• scoring may be ipsative
• typically multidimensional construct(s)
• elaborate scoring key
  • response option–aggregate scale associations

1. Some also use this term to refer to multiple choice-type items (with more than 2 alternative response options)

Multiple choice

• 20–25% chance¹ of “getting correct” regardless of construct standing
• familiar
• easy to score
• poor distractors can be response cues

1. technically “1” \(\div\) number of response options

Likert-type

• options progress along a response continuum
• most common item type in inventories
  • both unidimensional and multidimensional
• familiar
• elaborate scoring key
  • partially due to reverse scoring

“Categories”

• for example, standard 10-point
• not common in Psych assessment

• familiar
• non-psychometricians tend to like 😍

Checklists

• aka multiple response
  • adjective checklist historically popular
• choose as many descriptors as desire
• more common in demographic than substantive trait measurement
• (likely) complex scoring algorithm

“Other”

• your choice should not be tied to any of these formats
  • methods others have already tried
• choice of item type is a research decision
  • maximize the likelihood of accurately reflecting the intended construct

Item Writing Guidelines

1. Define construct
2. Sample¹ from content domain
   • Shorter preferred
   • Reading level??
   • Double-barrelled??
   • Phrasing direction??

1. rule of thumb is 3 or 4 times the number you’d like to eventually keep

Item analysis

Item characteristics

• difficulty (aka extremity)
  • average response
• discrimination (aka discriminability)
  • differentiation across trait levels
  • item functioning with someone high vs low on trait

Difficulty & Discrimination indices

Several common options exist:

1. p-value (% correct)
   • true tests only
     
2. corrected item-total correlation
   • aka “point biserial method”
3. visual plot (ICCs)

Item response theory

• Popular alternative to CTT
• Scale score estimation more complex
• Item characteristics permit targetted administrations
  • 🐱Computerized Adaptive Testing🦁

Item Characteristic Curves

• more common with true test items
• always a good idea to supplement numerical indices
• several different strategies
  • most focus on (at least) difficulty & discrimination

Activity!!

Name the capital of…

1. Maine Augusta
2. Mexico Mexico City
3. Missouri Jefferson City
4. Mongolia Ulaan Baatar
5. Uruguay Montevideo

These distractors are…

• really great!! I love them!! Gimme more!!
• bad
• about as useful as a hippopotomous
• better than some I’ve seen boy howdy

With inventory responses, item difficulty may also be referred to as item…

• extremity
• consistency
• discrimination
• equivocation

A visual representation of difficulty and discrimination can be found in a/an…

• ICC
• ODD
• IRT
• CTT

This question type is most accurately described as…

• Multiple Choice
• Forced Choice
• Checklist
• Categories

What’s the capital of Mongolia?

• Ulaan Baatar
• Umaami Baaktari
• Unaan Baaltam
• Utaashi Baaltari

References

Hinkin, T. R. (2005). Research in organizations: Foundations and methods of inquiry (R. A. Swanson & E. F. Holton III, Eds.; pp. 161–179). Berrett-Koehler Publishers.