Distractor analysis of multiple choice questions: A descriptive study of physiology examinations at the Faculty of Medicine, University of Khartoum

Abstract

Background: Distractor analysis is an important component of item analysis. It helps in the design and construction of items with functional effective distractors which are valid for future use and development
of proper question banks.Objectives: This study aimed at investigating distractors’ functionality of physiology multiple choice
question (MCQ) items at the Faculty of Medicine, Khartoum University using post- examination item analysis data.

Methods: Analyzing test statistics reports produced by Remark-Software, the frequency distribution of non-functioning (NFD) or non-effective distractors (NED) on ten summative physiology MCQ exams administered to undergraduate medical students in the period from September 2015 to September 2016 was assessed. Item analysis reports provided detailed options analyses in the form of response frequency and discrimination power of each option. Each exam paper consisted of 60-80 (five-option) items and the number of the examinees ranged from 332 to 359. A total of 654 items were reviewed, including 2580 distractors. NFD/NED were defined as distractors with either response frequency<5% of the examinees or non-negative discrimination coefficient (rpbis) criteria. Furthermore, the number of NFD/NED per item were computed to measure distractor efficiency (DE %) of an item as follows; 0 NED = 100 % DE, 1 NED = 75%
DE, 2 NED = 50% DE, 3 NED = 25% DE, 4 NED = 0% DE.

Results: Although less than one-third (31%) of total number of distractors were non-functioning because they were chosen by less than 5% of examinees, only 4 (0.15%) distractors had a choice frequency of
zero. In addition, 9.38% of total number of distractors was non-functioning because they had non-negative discrimination correlation coefficient. Defining functioning distractors using response frequency or discrimination ability criterion, the majority (61%) of our distractors were functioning effectively. The mean DE was 59.46±28.376% and the proportion of functional distractors ranged from 40.27% - 75% in the ten exams. Our study proportions of items with 100%, 75%, 50%, 25%, 0% DE were: 32.7%, 31.3%, 20.9%, 9.9% & 5.1% respectively based on response frequency criterion alone, compared to: 8.2%, 12.3%, 28.7%, 35.2% & 15.7% respectively when using any of the two criterion for NED definition.

Conclusion: The findings that the majority of our distractors were functioning and the high proportions of our items that have high distractor efficiency are explained by the fact that our expert staff members
carefully reviewed distractors to ensure unambiguity of the correct answers and generated plausible effective distractors.