3 TIMSS (2019)

A multiple-choice item asks a question or establishes the situation for a response. For the TIMSS 2019 assessments, this type of item provides response choices, or options, from which the correct answer is selected. A multiple-choice item is characterized by the following components:

  • The stimulus presents the contextual information relevant to the item
  • The stem is the initial part of the item in which the task is defined
  • The response options refer to the entire set of labeled response choices presented under the stem
  • The key is the correct response option
  • The distracters are the incorrect response options

At least half of the items developed for TIMSS 2019 will be multiple-choice items. The next sections present guidelines specific to multiple-choice items, and include information about writing the stem, structuring the response options, developing plausible distracters, and providing response rationales.

Please keep the guidelines for writing multiple-choice questions in mind.In particular, ask a direct question, make sure there is one and only one correct answer, and provide plausible distracters.

3.1 The stem

For the TIMSS 2019 assessments, since clarity is of vital importance, please phrase all stems as a direct question. An example of a stem phrased as a direct question is provided in Exhibit 2.

Exhibit 2: Example of a Stem that Asks a Direct Question – Grade 4 Mathematics

Figure 3.1: Exhibit 2: Example of a Stem that Asks a Direct Question – Grade 4 Mathematics

Provide sufficient information in the stem to make the question clear and unambiguous to students. In nearly all cases, the question must be able to stand alone, and be answerable without the response options.

  1. The stem should not include extraneous information. Extraneous information is liable to confuse students who might otherwise have determined the correct answer.
  2. Avoid using negative stems—those containing words such as “not,” “least”, “worst”, “except”, etc.If it is absolutely necessary to use a negative stem, put the word in bold type so that it stands out for the student. If the stem is negative, use only positive response options—do not use double negatives.
  3. If there is not one universally agreed upon answer to the question, it is best to include “of the following” or some similar qualifying phrase in the stem.
  4. Avoid questions for which a wrong method yields the correct answer (e.g., a question about a circle with a radius of \(2\), since \(2r = r^2\), students computing either the area or the circumference get \(4\pi\)).

3.2 Structure of the Response Options

  1. Write multiple-choice items with four response options, labeled A–D (as shown in the example item about planting trees in Exhibit 2).
  2. Make sure that one of the four response options is the correct answer (key). Make sure there is only one correct or best answer.
  3. Make sure that the four response options are independent. For example, response options should not represent subsets of other options. Also, do not include pairs of response options that constitute an inclusive set of circumstances (e.g., day or night, does or does not).
  4. Make sure that the grammatical structure of all response options “fit” the stem. Inconsistent grammar can provide clues to the key or eliminate incorrect response options. Avoid writing items where the options complete a sentence begun in the stem, because these can cause problems with translation.
  5. Make sure all (or sets) of the response options are parallel in length, level of complexity, and grammatical structure. Avoid the tendency to include more details or qualifications in the correct response, thus making it stand out. If the options are not parallel in length, please order the options short to long if at all possible.
  6. Do not use words or phrases in the stem that are repeated in one of the response options and, therefore, act as a clue to the correct response.
  7. Do not use “none of these” and “all of these” as response options.
  8. Arrange the response options in a logical order to save the student time in reading the options (e.g., years in chronological order, numbers from least to greatest).
  9. Avoid writing items where students can work backwards from the response options to find the correct answer (e.g., solving for x in an equation). Sometimes described as “plug and chug” items, such questions or problems will not be included in any of the TIMSS 2019 assessments.In such cases, a constructed response item may be more appropriate than a multiple-choice item.

3.3 Compound Multiple-Choice Items

Some mathematics and science concepts lend themselves to measurement using compound multiple-choice items. Exhibit 3 shows an example of a compound multiple-choice item.

Exhibit 3: Example of a Compound Multiple-Choice Item – Grade 4 Science

Figure 3.2: Exhibit 3: Example of a Compound Multiple-Choice Item – Grade 4 Science

The individual multiple-choice elements included in a compound multiple-choice item each address a particular aspect of a mathematics or science concept. Taken together, the group of multiple-choice elements provides better measurement than any single element. Generally, compound multiple-choice items are scored as fully correct (1 point) if all parts are answered correctly. For all compound multiple-choice items, please use the template to provide the rules for scoring and key for each part as shown in Exhibit 3.

3.4 Plausibility of Distracters

Use plausible incorrect response options (distracters) that are based on likely student errors or misconceptions. This reduces the likelihood of students arriving at the correct response by eliminating other choices and, equally important, may allow identification of widespread student misunderstandings or tendencies that could lead to curricular or instructional improvements. If there are no plausible errors or misconceptions, still make the options “reasonable.” For example, they should be from the same area of content. However, avoid the use of “trick” distracters.

3.5 Response Option Analysis

For all multiple-choice items, please use the template to include a brief analysis of each response option or rationale for inclusion of specific response options (one sentence at the most for each response option). Label the correct response option as the KEY and provide the supporting calculations for the answer or an explanation of why it is correct. For each incorrect response option, describe the common misconception the option represents or the process by which a student could have arrived at that incorrect answer. Exhibits 4 and 5 show multiple-choice items together with their response option analyses.

Exhibit 4: Example Multiple-Choice Item and Response Rationales – Grade 4 Science

Figure 3.3: Exhibit 4: Example Multiple-Choice Item and Response Rationales – Grade 4 Science

Exhibit 5: Example Multiple-Choice Item and Response Rationales – Grade 8 Mathematics

Figure 3.4: Exhibit 5: Example Multiple-Choice Item and Response Rationales – Grade 8 Mathematics

3.6 References

Mullis, I., Martin, M., Cotter K. & Centurino. (2019). TIMSS 2019 Item Writing Guidelines. International Association for the Evaluation of Educational Achievement.