To show or not to show: the effects of item stems and answer options on performance on a multiple-choice listening comprehension test

The purpose of this study is to examine whether the choice between three multiple-choice listening comprehension test formats results in any difference in listening comprehension test performance. The three formats entail (a) allowing test takers to preview both the question stem and answer options prior to listening; (b) allowing test takers to preview only the answer options; and (c) allowing preview of the question stems, but not answer options. Account is taken of additional, intervening factors relating to the content of the items that were tested; and we report on the extent to which they too may have contributed to difficulty. The results indicate that format (b) produces significantly fewer correct answers than the other two formats. No significant difference in test scores is found between the other two formats. The analysis suggests that the differences may be associated with reliance on a lexical matching strategy on the part of test takers. Implications are drawn for test developers and test takers.