Multiple answer (select all that apply) questions share many of the advantages and disadvantages of multiple choice questions, their close cousin. However, instead of asking the learner to select exactly one choice, multiple answer questions ask you to choose all the options that fit the description in the question stem. For example, you could be given a list of numbers and be asked to click on those that are prime, or you could be given a list of animals and be asked to select all of the insects, and so on.

Multiple answer questions are one of the standard question types in the assessment toolbox, and they are useful, but they don’t get used often in high-stakes assessments. Here are the trade-offs when using this question type:

Better discrimination

A multiple answer question tends to be a better discriminator than a multiple choice question with the same number of choices because the multiple answer question has many more potential responses. A five-part multiple choice question has five potential answers. A five-part multiple answer question involves five distinct yes/no decisions. As a result, it has 32 potential responses, which makes luck much less relevant.

Like a series of true/false questions

Multiple answer options could all be true, or all false, or any mix of true and false. This means that multiple answer questions function like a series of true/false questions, except that typically you have to get all of the decisions correct to earn the point.

Absolutely true or absolutely false

Multiple answer questions share a key weakness with true-false questions: each option must be either absolutely true or absolutely false, and that tends to make these questions either very simple or too subjective. If just one of the options is ambiguous then the entire question is ambiguous.

For example:

In the United States, which of the following are legally-valid reasons to deny employment? Select all that apply.

1. Age
2. Gender
3. Religion
4. Education

At first, it might seem like only the last choice, education, would be a valid reason to deny employment. In most contexts, age, gender, and religion are prohibited grounds for hiring, firing, and promotion. But “most” is not the same thing as “all.” There are exceptions. You may have to be of legal drinking age to be a bartender. Gender can be relevant when hiring a washroom attendant. Religion can be relevant when hiring a priest or a rabbi. So it’s actually hard to say whether each option is absolutely true or absolutely false, and that makes the whole question ambiguous. Even worse. the sort of people who really understand the subject matter might be most likely to get it wrong. In general, any multiple answer question that invites interpretation doesn’t work. That’s a real problem, because it’s often difficult to write options that have absolutely no room for doubt, and statements that leave no room for doubt are often too obvious to be useful assessments.

When are multiple answer questions useful?

These disadvantages aside, multiple answer questions can be useful when things are clear cut, with no ambiguity. In math, for example, you can ask which of the following are factors of a giving polynomial, which fractions are greater than 1/3, or which conclusions are inferable from a data set. No problem there. In science, multiple answer questions could ask learners to identify the symptoms of a disease, the animals from a list that are carnivores, and so on. There are plenty of other potential uses for this question type, but it remains relatively uncommon in the assessment landscape.