UGH—If you think having to pass multiple choice exam to
graduation from massage school and become licensed and then finally certified
is a pain you should try writing them. I
write textbooks for massage education and each book has a huge test bank. I also
write a review guide for the tests used for licensing. I am in process of writing 10 practice tests
150 questions each and my brain is exhausted.
I have five of them completed and five to go. I write really good multiple choice question.
I see every question as a little problem to be solved and a little lesson to be
learned. I wish I could tell you that I approve of multiple choice questions as a method for
evaluation but I don’t (even though I write really good multiple choice
questions). Doesn’t matter if I approve
or not- the licensing exams are based on multiple choice exams. Multiple-choice questions can be used to
measure knowledge outcomes and various types of learning outcomes. They are most widely used for measuring
knowledge, comprehension, and application outcomes.
I think the biggest
reason multiple choice questions are used because scores are more reliable than
subjectively scored items (e.g., essays),
scoring is easy, objective, and reliable and they can cover a lot of
material very efficiently (about one item per minute of testing time).
Constructing good
questions is time consuming and it is difficult to find plausible wrong answers.
My biggest concern is that real world problem solving differs – a different process
is involved in a real critical thinking and clinical reasoning processes. If a
multiple choice exam is used that the questions should be well written.
Multiple-Choice Item Writing Guidelines
From: http://ctl.utexas.edu/assets/Evaluation--Assessment/Writing-Good-Multiple-Choice-Exams-04-28-10.pdf
Multiple-choice questions typically have 3 parts: a stem,
the correct answer – called the key, and
several wrong answers, called distractors.
Procedural Rules:
• Use either the best answer or the correct answer format.
• Best answer format refers to a list of options that can
all be correct in the sense that each
has an advantage, but one of them is the best.
• Correct answer format refers to one and only one right
answer.
• Use good grammar, punctuation, and spelling consistently.
• Minimize the time required to read each item.
• Avoid trick items.
• Use the active voice.
• The ideal question will be answered correctly by 60-65% of
the tested population.
• Have your questions peer-reviewed.
• Avoid giving unintended cues – such as making the correct
answer longer in length than the
distractors.
Content-related Rules:
• Base each item on an educational or instructional
objective of the course, not trivial
information.
• Test for important or significant information.
• Focus on a single problem or idea for each test item.
• Keep the vocabulary consistent with the examinees’ level
of understanding.
• Avoid overly specific knowledge when developing items.
• Avoid textbook, verbatim phrasing when developing the
items.
• Avoid items based on opinions.
• Use multiple-choice to measure higher level thinking.
• Be sensitive to cultural and gender issues.
• Use case-based questions that use a common text to which a
set of questions refers.
Stem Construction Rules:
• State the stem in either question form or completion form.
• When using a completion form, don’t leave a blank for
completion in the beginning or
middle of the stem.
• Ensure that the directions in the stem are clear, and that
wording lets the examinee know
exactly what is being asked.
• Avoid window dressing (excessive verbiage) in the stem.
• Word the stem positively; avoid negative phrasing such as
“not” or “except.” If this cannot
be avoided, the negative words should always be highlighted
by underlining or capitalization:
Which of the following is NOT an example ……
• Include the central idea and most of the phrasing in the
stem.
• Keep the length of options fairly consistent.
• Avoid, or use sparingly, the phrase all of the above.
• Avoid, or use sparingly, the phrase none of the above.
• Avoid the use of the phrase I don’t know.
• Phrase options positively, not negatively.
• Avoid specific determinates, such as never and always.
• Make sure that there is one and only one correct option.
Distractor (incorrect options) Development Rules:
• Use plausible distractors.
• Incorporate common errors of students in distractors.
• Use familiar yet incorrect phrases as distractors.
• Use true statements that do not correctly answer the item.
• Avoid the use of humor when developing options.
• Distractors that are not chosen by any examinees should be
replaced.
Suggestions for Writing Good Multiple Choice Items:
• Present practical or real-world situations to the
students.
• Present the student with a diagram of equipment or procedure and ask for application, analysis
or
evaluation.
• Use pictorial materials that require students to apply
principles and concepts.
• Use charts, tables or figures that require interpretation.
For all my whining and complaining the multiple choice
question is here to stay until a better way evolves and right now I do not have
a better way for the licensing exam process. When I write practice exams I will include a
few questions that break the rules provided above just so the reader will have
experience with it. By the time anyone
trudges though 10 practice exams they should be able understand how to approach the multiple choice exam
format and since licensing exams are multiple choice exams so if you are going
to be licensed as a massage therapist you have to pass a multiple choice exam
whether I like it or not.
I do however have a
better way for the classroom- Rubrics.
A rubric is an explicit set of criteria used for assessing a
particular type of work or performance. A rubric usually also includes levels
of potential achievement for each criterion, and sometimes also includes work
or performance samples that typify each of those levels. Levels of achievement are often given
numerical scores. A summary score for
the work being assessed may be produced by adding the scores for each
criterion.
Here is a general example
CRITICAL THINKING RUBRICS
Based on a draft from Elaina Bleifield and the Paulus CT
Group
CATEGORY ONE: KNOWLEDGE AND COMPREHENSION (understanding the
basics)
4—The work consistently demonstrates clear, accurate,
detailed and comprehensive understanding of the relevant facts /data /
theories/ terms as well as the ability to organize the information for
application, presentation, documentation, and/orfurther examination.
3--The work demonstrates an adequate understanding of the
relevant facts / data / theories/ terms as well as the ability to organize the
information for application, presentation, documentation, and/or further
examination
2-- The work demonstrates an uneven and shaky understanding
of the relevant facts / data / theories/ terms as well as a limited ability to
organize the information for application, presentation, documentation, and/or
further examination.
1-- The work demonstrates an inadequate understanding of the
relevant facts / data / theories/ terms as well as a limited ability to
organize the information for application, presentation, documentation, and/or
further examination.
CATEGORY TWO: APPLICATION AND ANALYSIS (attaining the
concept)
4—The work demonstrates confident ability to work with the
key concepts / information / process / theory -- applying or extending them to
a wide variety of new problems or contexts, making predictions, recognizing
hidden meanings, drawing inferences, analyzing patterns and component parts,
communicating insightful contrasts and comparisons.
3--The work demonstrates adequate ability to work with the
key concepts / information / process / theory -- applying or extending them to
a variety of new problems or contexts, making predictions, recognizing hidden
meanings, drawing inferences, analyzing patterns and component parts,
communicating insightful contrasts and comparisons.
2-- The work demonstrates uneven and shaky ability to work
with the key concepts / information / process / theory --applying or extending
them with mixed success to new problems or contexts, making predictions,
recognizing hidden meanings, drawing inferences, analyzing patterns and
component parts, communicating insightful contrasts and comparisons.
1-- The work demonstrates extremely limited ability to work
with the key concepts / information / process / theory --applying or extending
them with very limited success to new problems or contexts, making predictions,
recognizing hidden meanings, drawing inferences, analyzing patterns and
component parts, communicating insightful contrasts and comparisons.
CATEGORY THREE: SYNTHESIZING AND EVALUATING (going beyond
the given)
4—The work demonstrates surprising/insightful ability to
take ideas / theories / processes / principles further into new territory,
broader generalizations, hidden meanings and implications as well – as well as
to assess discriminatively the value, credibility and power of these ideas
(etc) in order to decide on well-considered choices and opinions.
3-- The work demonstrates adequate ability to take ideas /
theories / processes / principles further into new territory, broader
generalizations, hidden meanings and implications as well – as well as to
assess discriminatively the value, credibility and power of these ideas (etc)
in order to decide on well-considered choices and opinions.
2-- The work demonstrates uneven and superficial ability to
take ideas / theories / processes / principles further into new territory,
broader generalizations, hidden meanings and implications as well – as well as
a limited ability to assess discriminatively the value, credibility and power
of these ideas (etc) in order to decide on well-considered choices and
opinions.
1-- The work demonstrates little ability to take ideas /
theories / processes / principles further into new territory, broader
generalizations, hidden meanings and implications as well – as well as a
limited and superficial ability to assess discriminatively the value,
credibility and power of these ideas (etc) in order to decide on
well-considered choices and opinions.
Here are some more examples
Well now that I have vented I need to get back to writing
multiple questions. UGH
No comments:
Post a Comment