How to Write a Multiple-Choice Question

Basic Terminology 13. The shortage of a product of resource is called -- STEM

a) mass consumption b) planned obsolescence }DISTRACTORS c) materialism ALTERNATIVE *d) scarcity KEY * keyed answer

I don't like the term "distractors" because it suggests you are attempting to trick the student, which is not really the idea - we're adults, they're kids, it's not that hard to trick'em.... -- but we're stuck with the term from long usage, so "distractors" it is How to WRITE a Multiple-Choice Question Guidelines for the Basic Format of a multiple-choice question and some examples of Poor Formats. Several Types of multiple-choice questions Whole Item Checklist. Does your question have everything it needs? Checklist for Stem. Will your stem give away the answer? Checklist for Alternatives. Are they all believable? Keying the Item

Basic Format for Multiple-Choice Questions

1. Each Alternative listed on separate lines, one under another. o o it is a good clear format it is the one used on provincial tests, so should get students used to it

since it is modeled on provincial tests, probably the format most commonly found in other classrooms -- consistency is helpful to some students

2. Use correct grammar.

3. Use capital letters, periods, colons -- but be consistent.

POOR FORMATS FOR MULTIPLE-CHOICE: 47. The shortage of a product or resource is called a. mass consumption b. planned obsolescence c. scarcity d. materialism

47. The shortage of a product or resource is called a. mass consumption c. scarcity b. materialism d. planned obsolescence

Hard to read without space between Stem and Alternatives.

the logic for placing (b) and (c) is equally compelling in both versions of #47 so some students will choose the right answer but record the wrong letter.

48. 21 X 10 is 1. 2.1 2.10 3.4721 4.210 48. 21 X 10 A) .21 B) 2.1

C) 21 D) 210

use letters in front of alternatives instead of numbers (especially in anything involving numbers)

47. Which of the following authors is female? A) Tennessee Williams B) Andre Norton C) Sean Stewart D) Guy G. Kay

Several Types of Multiple-Choice Questions

DISCRETE item: stands on its own without any additional directions or information. It may take the form of a question or an incomplete statement. The PASSAGE or CONTEXT DEPENDENT item provides information separate from the question stem. Thus, the stem has meaning only in relation to the passage or information provided. - sometimes called a DBQ in social studies "Document Based Questions"

CORRECT ANSWER, where there is only one correct answer; the rest are definitely wrong.

BEST ANSWER

- they have different types of stems - this where the student is asked to make a fine distinction between several alternatives which are very similar, so all have some element of truth

- seldom done correctly, this is one parents have trouble with too

Whole Item Checklist

1. The item must reflect an important curricular objective

-if I could only ask one question, is this the question I would ask? -if I could have the student remember one thing from this course ten years from now, would this be it? -avoid temptation to ask question just because you have one written

2. The item should focus on a single problem. -the solution can involve several steps, but only one question should be asked

3. The item is totally independent of all other items for its correct answer. 4. The item should be free of cultural, gender, or other biases If all your math problems are about baseball with nothing on sewing, it might be biasing math scores; -if poetry question asks about allusion to English fairy tales, it may be bit rough on transfer students from China and Nigeria -can get subtle -- Christian educator complained that literature choices on English 30 exam were all downbeat because we seemed to think depressing was sophisticated, whereas he liked to accentuate the positive, so his kids werediscriminated against...

5. The item is not unduly demanding of the student's total time for the test. - you can'have one question require five readings, 20 minutes with a calculator and count it the same as a simple recall question - if student is going to read some passage or analyze data, try to amortize it out over several questions -- more of which we will consider later

6. The item should be clear and contain simple language.

The sentence structure and vocabulary should not present any unnecessary

challenge to comprehension --Unnecessary wording and details should be eliminated from both the stem and alternatives -key words from a reading passage or data set should not be repeated in the stem or in an alternative 7. Avoid asking opinion type questions. -we don't mark people for their opinions in this country some confusion here because: - can use multiple choice formats for opinion surveys, but can't be on same test, or even same day as regular test

8. Make sure whole item is on same page - I thought this was self-evident but lately have had a rash of students who allowed their word processor to put last alternative on top of next page;

or

another problem occurs when testers have passage on one page and question on another page-- so half kids miss passage altogether, and other half rustling pages back and forth going nuts...

Remember to proofread whole item: some folks forget to check distracters, just read stem, right answer, and go on -so kid knows if he finds a typo, probably not in correct answer -less of an issue with spell checkers on computers

Checklist for Stem

1. The stem should be self-contained. The students should be able to read the stem (and passage or chart if there is one) and answer the question without reference to the alternatives.

o o

places an unfair reading burden on some students because there is no context incomplete stems tempt you to write distracters which are unrelated to the specific learning objective -if question is about market economy all distracters should relate to that objective

-if question is about who America is named after, distractors should all be people it could have been named after, not random thoughts that takes kid in 12 different directions ---> no diagnostic information available from such a question: -if child gets it wrong, is because he got the person wrong, or because got tangled in one of the other curricular areas you've mixed in?

-stem does NOT have to be phrased as a complete question, but should be complete in the sense that student can finish the phrase based on what's there -SOME research suggests that complete questions are best for young children who have trouble keeping first half of an incomplete statement in their head and have to go back and re-read the stem each time they read an alternative 2. 3. The stem should be clearly worded and free of ambiguity. -easier said than done, but keep chipping away at it -- can I simplify, clarify? 4. The stem should be free of irrelevant or unnecessary detail.

-same thing, just keep whittling down until have bare essentials -temptation is to do some teaching at the same time, stick in some new knowledge, but resist temptation -danger of confusing the student -increases reading burden -too nervous to really learn

all that much anyway 1. The stem asks a question that has a definite answer -again, should be able to answer the question before seeing alternatives -and their answer should be among the alternatives when they get there... When BEST-answer items are used the qualifier in the stem has been emphasized by the use of capital letters. -anytime you have a key word that would change the answer if the kid misses it, boldface it, capitalize it, underline it or all three If you have a "best answer" question, don't say, which is the "BEST" answer: you must specify in what way it is the "best"

NOT which is the best procedure for playing a flute -Which of the following ways of playing the flute will give the clearest sound? The stem must be grammatically correct within itself and in its relationship to each of the alternatives i.e., has to make a question, four answers; or stem makes complete statement with answers - the stem should not provide grammatical clues to any alternative

The use of absolutes (e.g., always, never) should be avoided.

-absolute statements are usually incorrect, some student will know an exception to the keyed answer and so panic

Negatively worded stems should be avoided if possible.

If you MUST use a negative, the negative wording should be stressed using uppercase

"Of the following" --> only use this where student needs alternatives to answer It is a "sometimes"kind of thing -"which of the following" suggests that there may be other equally valid answers, but that given these four choices, which is the best?

In a correct answer question (where there is only one possible answer) "which of the following" is both redundant and confusing

........is this appropriate use or no?

Checklist for Alternatives

1. The alternatives are all appropriate to the question asked or implied by the stem. 2. -sometimes have trouble thinking of a fourth alternative, toss in something essentially irrelevant as a red herring -- confuses kids, since they suddenly wonder what that has to do with anything, they waste time checking to see if they have misinterpreted the question.

3. Alternatives must be grammatically consistent with the stem, and parallel in form

4. 5. Avoid overlapping distracters

6. Alternatives are stated as briefly and simply as possible -keep reading load down -problem with provincial tests and committees -- constantly adding more qualifiers so will be completely unarguably correct -- good instinct but hard to read -in classroom context you have a little more leeway to cut

7. Words common to all the alternatives should be placed in the stem

8. All distracters should be plausible -common misconceptions

or -errors in calculation

choosing errors and common misconceptions helps you to identify which error this student is making, and therefore how to help them --which you find out by making it a short answer question the year before and see what the students give you as wrong answers

-throwing in some impressive sounding jargon like "important" -all alternatives should have same complexity and length as correct answer 9. The use of trickery has been avoided.

there is no point in finding out if you can trick them-- you can

10. DO NOT use "all of the above"

(a) -all of the above automatically produces over lapping alternatives

-as we saw in

from #3: overlapping distractors

(b) -if chose (a), skips ahead without reading rest of alternatives, can think he'got the right answer (and he has) but gets it wrong all of the above is unfair, because it means answer (A) is correct, so student anticipates (a), looks down, sees (a), chooses it, then moves on. Gets it wrong, even though KNEW (A) was the correct answer -gets answer right but question wrong -- is that FAIR????

(c) -"of the Above"also messes up your statistics -if can eliminate one alternative, automatically kicks out "all of the above" as an alternative

-Even worse is "both a and b" --> becomes exercise in logic

11.

DO NOT use "none of the above"

Never-ever in best answer questions because who is to say how close is close enough?

o o

"None of the above" can always be defended as best alternative... don't use even for correct answer questions because

-wasted alternative --> other wrong answers tell you which mistake student is making, "None of the above" doesn't WHY they think none of these apply -maybe if you go to five alternatives...but what's the point?

Exception is in math

12.

Key words from the stem are not repeated in the alternatives

13.

Avoid stating the correct answer in greater length

14. 15.

Avoid absolute terms like "always" "never" Avoid stating the correct answer in textbook language or stereotyped phraseology

-tempted to take correct answer from curriculum guide or text, but have to make up incorrect answers yourself, kids see difference in style

16. 17.

All distracters must remain consistent regardless of higher levels of learning

-->it is considered BAD form for an answer to be correct for a Grade 10 that would be considered wrong for a third year university; or vice versa -->can't say, "oh well, close enough for grade 6" --> E.g.. water freezes at 0&#176C, except some kids will know that is at sea level maybe that's all you expect from a grade 4, but you don't want to teach grade 4's something a grade 8 teacher is going to have to unteach

18.

Do not present false information in alternatives

19.

The branch has the policy of not saying WRONG things in alternatives, but that's kind of a high standard to reach for

o o o

idea is that students will read wrong information, and even if they realize that it is not the answer in this question, may think it is true often remember "that somewhere once"and if it happened to be a false alternative on a test, could be very counter productive not as damaging as true false because more distracters to remind that its only one possibility, false statements balanced by reading correct ones at same time in same context

20.

Always use the same number of alternatives

-confuses students -sometimes kids can get clues by number of alternatives

-statistical analysis is problematic if they vary from 3 to 6

-but not really that crucial -I sometimes have varying number if using some common items with other teachers

USED TO SAY: - the number of alternatives to use is 4 -3 is generally too few --> 33% by guessing -studies have shown that 5 are usually too many to think of --> four is tough enough -no hard and fast rule, but take my word for it, four makes best sense

NOW SAY:

Last 18 months a lot of new research is coming out that says, really, 3 is best. Math is way over my head but seems to be... argue 4th alternative is too hard, most teachers write rubbish....

21.

If you're ever doing science or math, must have correct number of significant digits (unless that's the concept you're testing of course)

Keying The Item

1. Make certain that intended answer is only correct or clearly the best -pretty obvious, but some kids can think their way around anything

2. Vary the distribution of the keyed item in random manner -should have same number of a's, b's, c's and d's as correct answer

-most kids know that if in doubt you choose "c" -because we all tend to want to "hide" keyed answer in middle -because after you've gone to all that work of thinking up wrong answers want to make sure the little monsters read all of them

-this is very seductive -- subconscious even when at the test branch so we used: -random tables to force yourself -or dice -or use pyramiding

3. Distracters should be arranged in ascending or descending order.

4.

o o o o o 5. Make o o

numbers in order dates in chronological order lines from a passage in order they appear in passage alphabetize if nothing else, others in length (pyramiding) --not vital, last choice sure that every item is independent of every other item that answer to one question doesn't tip them off on another one or that wrong answers confuses them into choosing wrong answer on another question --> though that's a bit harder to spot o answer to one cannot depend on getting answer to previous one right -e.g.., can't break 4 step math problem into four questions because error on first step costs four marks, even if he got last three steps right o ALL of which the department found away around---> we developed families of items --> bit more advanced.... but you can simulate 4 step math step one, what is the answer step two: "another student doing this experiment got 148 grams. Had your answer been 148, the next step would be..." this was an Alberta invention, now used everywhere

6. Each item should be worth the same number of marks: no surprises means no fair telling student #37 was worth 12 marks confusing on what basis would you decide? Discriminating against students who don't do well on that one concept --> if you think concept is twice as important as another, you ask twice as many questions about it o since each item is worth the same, you have to try to make them all about equal work --> in terms of time to do them, not difficulty can't spend two hours reading booklet to answer one question....