Blog

Maritime Training Issues Blog

The latest from Marine Learning Systems.

Multiple Choice Test in Maritime Assessment

Multiple Choice Tests in Maritime Assessments

Jun 29, 2016 Murray Goldberg 4

Introduction

Multiple choice tests are one of the oldest assessment techniques in existence. Yet they are also one of the most highly maligned. Why, then, are they so common in maritime training? Are they effective? The answers to these questions are not as simple as one might hope. However, every maritime trainer needs to be aware of the strengths and weaknesses of multiple choice question (MCQs), and how to use them to greatest effect.

This article is the first of a series that looks at MCQs in maritime training. This first post examines the strengths and weaknesses of MCQs. Subsequent posts will offer some tips on how to create and use MCQs to take advantage of their strengths.

 

Criticisms of Multiple Choice Questions

Multiple choice question (MCQ) exams have been used for over 100 years to assess training effectiveness both within and outside the maritime industry. Even so, trainers and educational researchers love to hate them. There are many valid arguments frequently levied against MCQs:

One of the main arguments is that MCQs can’t test higher order thinking, but instead only test factual knowledge.

This is not technically correct; MCQs can be constructed to test higher order thinking, but it requires skill to construct such questions. And even in the case that such questions are constructed well, they are unlikely to be as effective at the assessment of reasoning ability as are other forms of assessment like written responses, oral responses, or demonstrations of skills.

 

Performance on an MCQ can generally only fit into one of two categories: correct or incorrect.

The problem is that a “black and white” analysis does not reflect the reality that knowledge and competencies are rarely either completely known or completely unknown. Candidates are much more likely to have a partial grasp of the knowledge being measured.

As before, MCQs can be constructed to test partial knowledge, but doing so requires some skill. And even if that effort is put in, they are unlikely to be as effective as other forms of assessment at revealing partial knowledge.

 

MCQs are frequently written ambiguously, are confusing, and can sometimes be guessed correctly.

These are all examples of a general class of problem with MCQs: simply said, they are hard to write well.

I can personally attest to this fact. At British Columbia Ferry Services Inc (BC Ferries) there is a testbank of over 13,000 questions which have largely been written by BC Ferries subject matter experts. Despite the high level of expertise and the great deal of effort put into the creation of these questions, it is not unusual at all for questions to require multiple cycles of use and review before they are considered to be well written and unambiguous. Constructing well-written MCQs is not as easy as one might imagine.

 

Why are MCQs Popular?

This is certainly not an exhaustive list of criticism of MCQs. There are others. With all of these drawbacks, one might wonder why they remain so popular.

One of the obvious reasons is that they are easy to grade. And as a former faculty member myself, I can tell you that I and most of my colleagues considered grading exams only slightly preferable to having one’s eyes gouged out with blunt sticks. This adds fuel to the fire of arguments against their use: not only do MCQs have many shortcomings, but their main reason for use is arguably because trainers are too lazy to grade answers to “better” assessment methods.

However, it is important to consider some of the benefits of MCQs. They actually have many, and these benefits need to be considered carefully. MCQs bring to the table benefits that other assessment mechanisms cannot offer. As one component of a comprehensive assessment strategy, MCQs are a valuable tool. However, their value can only be realized if they are used for what it is best at.

So – what are MCQs best at?

 

The Strengths of Multiple Choice Questions

They can Assess Factual Knowledge

One advantage of MCQs is that they are quite effective at testing well defined factual knowledge. This is important in the maritime industry because all skills and competencies are rooted in a foundation of knowledge.

While knowledge itself is not sufficient to ensure that a competency is held, it is necessary. That is, even if a trainee can demonstrate a skill under assessment conditions, if they do not possess the knowledge underlying that skill, then they cannot be expected to perform the skill safely under varying conditions. Knowledge must be tested separately as part of any comprehensive assessment strategy, and MCQs can be very effective at testing certain forms of knowledge.

 

They are Objective

Another significant advantage of MCQs is that they completely remove subjectivity in grading. In my view this is one of their greatest strengths. Variability in grading is a pervasive problem for any large organization. Regardless of how standardized the training may be, different trainers will have different ideas of what constitutes a “pass” or “fail”. This problem can be somewhat alleviated through the use of rubrics, but even these can never completely remove the subjectivity inherent in other assessment techniques. The grading of MCQs is 100% objective – a valuable contribution to an assessment program.

 

They Enable Frequent Formative and Summative Assessment

One of the main reasons for the popularity of MCQs, as mentioned above, is that they are easy to grade. This is especially true if they are done online where grading is automatic. This is obviously an advantage to the trainer. However, it is, in fact, an even more important advantage for the trainee.

The reason for this is that frequent and regular assessment is a cornerstone of effective training. Assessment comes in two flavors – formative and summative. Summative assessment is used at the end of training to determine whether someone has learned the concepts and therefore is fit for duty. It is, in general, for the benefit of the training organization.

Formative assessment is different. It exists to benefit the trainee and inform the training process. It can be used throughout training to provide feedback to the trainee as to how effectively the lessons are being learned. The more frequent the feedback through formative assessment, the earlier (and less costly) the training course correction are. The problem, of course, is that most forms of assessment are costly. Therefore, although formative assessment is critically important, it is rarely provided in sufficient quantity.

This is where MCQs are especially useful. MCQs, especially those delivered online, require no time to grade. They can be applied as frequently as desired as long as there are sufficient questions available. In the absence of an instructor who has sufficient free time to grade frequent formative assessment, MCQs are an exceedingly valuable tool as they enable frequent and regular formative assessment.

 

They are Easy to Randomize

Another feature of MCQ tests is that technology can be used to randomize the exams such that no two exams will be exactly the same. For those not familiar with MCQ question randomization, most learning management systems (LMS) have the ability to dynamically create exams by randomly selecting questions from an exam database. In the case of our LMS at Marine Learning Systems, we employ a form of intelligent randomization where the difficulty and coverage of questions on every exam is normalized. This helps ensure that all candidates receive exams of the same coverage and relatively equal difficulty even though the question mix is different.

The ability to randomize exams has two advantages over other kinds of assessment methodologies.

First, randomized examinations further enhance the opportunities for frequent and regular formative assessments. Randomized examinations allow trainees to continue to assess themselves, as each time they do so, they can receive a somewhat different examination.

Second, for summative assessments, randomized exams ensure that no two trainees receive the same exam. This helps prevent the opportunity for information about the exam to spread from one trainee to another.

 

They are Transparent: Analytics

Another wonderful advantage of technology-delivered MCQs is the ability for them to generate analytics which give training managers excellent visibility into the learning process.

Each time a trainee answers a MCQ online, a record is made of that response. This data, when analysed and presented, can provide insights into how well trainees – both as individuals and collectively – are learning. For example, which knowledge and concepts are not being learned well? What common misconceptions do our trainees have?

These and many more questions can be answered by MCQ analytics, yet they can be difficult at best (and impossible at worst) to obtain from most other types of assessment. If the goal is to identify gaps in knowledge early – before they lead to performance or safety issues, MCQ analytics can help tremendously.

 

Conclusion

So – are MCQs of value despite their obvious shortcomings? I say yes – absolutely.

Should they be used as the sole technique to assess seafarers? Absolutely not.

In fact, there is no single technique which could be used as the sole technique for assessment. Each assessment tool offers specific advantages. And while each tool has limitations, those limitations are accommodated by the strengths of the other tools. MCQs are no exception. Therefore, by combining assessment techniques in your overall assessment strategy, you can take advantage of the benefits of each.

The next article in this series is going to present some guidelines on how to create good multiple choice questions. While it is not difficult to do so, a little bit of advice goes a long way in ensuring the time you put into it yields good results.

If you would like to receive notification of the next article of this series (and subsequent maritime training articles) and have not yet done so, please subscribe to the blog below to to receive notifications.

Until then – thanks for reading!

Follow this Blog!
Receive email notifications whenever a new maritime training article is posted. Enter your email address below:

Interested in Marine Learning Systems?
Contact us here to learn how you can upgrade your training delivery and management process to achieve superior safety and crew performance.

Murray Goldberg

Murray Goldberg is the founder and President of Marine Learning Systems. He began research in eLearning in 1995 as a faculty member of Computer Science at the University of British Columbia. He went on to create WebCT, a highly successful LMS for higher education; serving 14 million students in 80 countries. Murray has won over a dozen University, National and International awards for his pioneering contributions to the field of educational technology. Now, in Marine Learning Systems, Murray is hoping to play a part in advancing the art and science of learning in the maritime industry.

Comments (4)
  1. Richard Teo    

    MCQs are not suitable for competency based learning and should not be encouraged as the assessment construct is not criterion referenced. It’s is imperative that an assessment tool must have rubrics that are rigorous and consistent for judgement to be made as to whether the demonstrable skill or set of skills making up the standard of competence has been attained. MCQs are not suitable except to measure the knowledge level and only if the correct answer to the question is aligned to the standard.

    1. Marine Learning Systems    

      Richard, you are absolutely correct. MCQs are very effective for measuring knowledge objectively, however they should never be the sole assessment technique used, especially given the importance of skills in the maritime industry. Demonstrative exams (“show me how to…”), with the help of rubrics like you mentioned, are a much more appropriate technique for skills assessment. Using a combination of assessment techniques is recommended as it combines the strengths inherent to each technique. Murray has written more about this in further detail in this past blog post.

    2. Murray Goldberg    

      I would take a somewhat more inclusing view on MCQ examinations. MCQ examinations, when done well, are an excellent tool for assessing knowledge. Moreover, they come with advantages such as the ability to offer more assessment (because of automatic grading), the ability to generate deep metrics, being highly objective, and other benefits that either are not part of other assessment techniques or are very difficult to achieve with other assessment techniques.

      It is also very important to remember that knowledge is a foundational requirement for any skill to be effectively performed under varying circumstances. So using a tool to assess (formatively and summatively) knowledge alongside practical skill assessments is an excellent technique. Jeff Joyce discusses this in his comment -and they at BC Ferries have had tremendous results implementing MCQ *as part* of a comprehensive assessment methodology.

      I am also unaware of why an MCQ-based exam can’t be criterion referenced, and would also argue that MCQs essentially self-define their rubric.

      Having said that, I completely agree that there are very good and very poor examples of both the application of MCQ exams and multiple choice questions themselves. But I don’t think MCQ exams are inherently flawed. Quite the contrary.

      Thanks everyone for these great comments.

  2. Jeff Joyce    

    This is an excellent summary of the pros and cons of multiple choice questions. Where I work (BC Ferries), we continue to fine-tune our MCQs, but we also hone our other assessment techniques. In most cases, our assessment process includes a MCQ exam, a series of practical demos, an oral exam, and a discussion/interview with someone in a senior position. This ensures multi-faceted testing, which inevitably leads to more objective testing and the realization by both successful and unsuccessful candidates that we are very serious about competency assessment prior to putting someone in an operational role. Fortunately, our LMS fully supports this series of assessment processes!

Comments are closed.

Scroll to Top