Centre for Teaching and Learning

Centre for Teaching and Learning
Centre for Teaching and Learning

Designing Remote Final Exams

Instructors with remote-delivery final exams may now face new challenges in order to accurately assess learning, manage accommodations and time zone differences, and address academic integrity concerns. This recorded webinar and resource describe general design considerations for remote exams and explain how to use the ICE model to design a common framework that ensures single and multiple exams are accurate assessments of student learning outcomes.

View the 25-minute webinar: Designing Remote Final Exams.

Sections:

General considerations about offering exams remotely
Using a Table of Specifications to Develop Exam Questions
The ICE Model
Using ICE in a Table of Specifications
Frequently Asked Questions

General Considerations About Offering Exams Remotely

Following are some general ideas compiled from a variety of institutions across Canada and the United States. Individual faculties, schools, and departments may have other guidelines; check the information for your faculty in these sites:

Faculty of Arts & Science
Faculty of Engineering and Applied Science
Faculty of Health Science
Faculty of Law
Smith School of Business
Faculty of Education
School of Graduate Studies

Timing and accommodations logistics

If you were planning a 3 hour exam, consider reducing it to 1.5 hours so that students with time accommodations will still complete within the scheduled exam time.

Offer a longer availability window (e.g., 48 hours) and once students begin, they’ll have the correct amount of time to complete the exam. This helps students who are now in different time zones, although keep in mind that students with multiple scheduled exams may not be able to take advantage of the extended window.

Write multiple exams for different groups of students. You might group according to time zone or according to amount of time received under accommodations (e.g. no additional time, time-and-a-half, double-time).

Academic integrity

Clearly communicate what’s allowed and not allowed to students (e.g., working with other students, sharing questions or information with other students, posting questions online, using particular materials, impersonating another student for the purposes of writing the exam). Appeal to elements of transparency, fairness, and the learning relationship between instructor and students. The University of Saskatchewan has a great email template available.

Consider having a question at the start of the exam where students agree to follow academic integrity rules.

Make a large question bank, and randomize distribution to each student so that students are less able to share questions/answers with each other. However, keep in mind that you shouldn’t use this question bank again in future years.

Limit the number of questions displayed per page and don’t allow students to go back to previous pages once they’ve submitted answers. This is not ideal from a pedagogical perspective, but it will reduce the ability of students to share questions and answers.

Remind students that we are all supposed to be social and physically distancing, so that’s one more reason they shouldn’t get together with each other in person to write the exam.

Using a Table of Specifications to Develop Exam Questions

Whether you’re writing a single exam with a deep question pool, a take-home exam, or multiple equivalent exams, a table of specifications can help you plan out the overall format and scope of the exam.

A table of specifications is a method for planning out the format and questions in your exam so that you are measuring exactly what you want to measure, both in terms of content and the way students engage with the content. It ensures that you will assess learning to the same level of complexity across multiple exams, meaning that the exams can be considered equivalent in terms of student grades. Even if you are only planning a single exam, a table of specifications can be helpful. For instance, if you are giving students a choice between two or more questions, it’s important to write questions that assessing student learning to the same level of complexity. Or if you are developing a pool of questions that will be assigned randomly to students, you’d want to make sure all questions in each pool are assessing the same learning.

Ideally, your exam planning will be done alongside a consideration of what was emphasized in the course itself. For instance, if students practiced a lot of identification-type questions throughout the semester, you would ask them to do the same thing on the exam. If you spent ten weeks on one topic and only two weeks on another, you’d expect to have the final exam topics distribute in a similar proportion.

The ICE Model

To get both the content and the way students are expected to engage with the content all into one exam, it’s helpful to align a table of specifications with a learning framework. Here we’ll use the ICE Model, developed by Sue Fostaty Young and Robert J. Wilson (2000). ICE stands for Ideas—Connections—Extensions. Ideas are basic facts or steps in a process; connections are relationships between facts or between facts and the learner; and extensions are hypotheses, applications, or extrapolations to new situations. The diagram below gives some examples of ideas, connections, and extensions for a variety of disciplines.

 

Ideas

Connections

Extensions

naming features of coniferous trees distinguishing between deciduous and coniferous trees designing a lesson plan for teaching coniferous and deciduous trees to a school group visiting a biological field station
listing properties of a particular element on the periodic table grouping different elements based on their properties predicting what properties will emerge when two elements are combined
describing a playwright’s characteristic style explaining how a playwright’s biography led them to a particular mode of theatrical presentation writing or directing a short original theatrical presentation that mimics a playwright’s style
explaining how water flows through sand comparing the way water flows through a variety of materials designing a system for safe underground storage of nuclear waste
following the steps in a process for doing a patient intake choosing between two approaches for assessing patients triaging patients in an emergency room by combining and selecting among a variety of intake strategies

Using ICE in a Table of Specifications

With the ICE model selected as a learning framework, now it’s possible to use a table of specifications to  combine content and what students need to do with the content into a single table to design an exam (PDF, 77KB).

General Table of Specifications: PDF version also available

A table of specifications has three main components:

Content column

  • the topics you’ve covered in the course and that you want to test. You can be as fine-grained or as general as you wish.

Ideas–Connections–Extensions columns

  • the total points you will assign to ideas-, connections-,  and extensions-type questions with each of those topics

Totals column and row

  • allows you to see how your plans distribute the assessment proportionally
  • does your weighting of ideas, connections, and extensions match the learning outcomes of the course? the learning activities you emphasized all semester? your hopes for this final assessment?
  • does the distribution of content match the distribution within the course?
  • make sure everything adds up to 100 in the bottom right corner of the table!

By planning everything in this table, it’s possible to then write multiple exams, or multiple questions within a particular segment, that are equivalent in terms of the assessment of student learning. The table can be adapted to suit your own needs as an instructor. In the two examples below, you’ll see that individual instructors can come up with quite different results.

              Business Example (PDF, 77KB)

Table of Specifications for Business: also available as PDF

In this example of a business exam, you can see that the exam is relatively balanced in terms of ideas, connections, and extensions overall, and in most content areas. However, note that ethics, for instance, is only extensions-type questions—perhaps not surprising since ethics often involves decision-making in novel situations.

              Music Theory Example (PDF, 77KB)

Table of Spcifications for Music Theory: also available as PDF

In this example of a music theory exam, you can see that the content areas are more fine-grained, and that the instructor has made notes for themselves about specific question types alongside the points distribution. The exam is almost exclusively ideas-type questions, but the instructor was fine with this because connections and extensions learning outcomes were assessed with other assignments (written reflection papers and a final project).

Frequently Asked Questions

Where can my students get support and guidance for online/remote learning?
Student Academic Success Services has a variety of resources for all students (undergraduate and graduate) involved in remote and online learning. They also offer online appointments and webinars on a variety academic strategies.
Which question type should I use?
  • what’s common in your discipline?
  • what types have students had practice with already?
  • are there certain formats that they will need to be good at in the future (e.g., multiple choice because their accreditation exam will use that style; essay because the next course in their program uses essay exams)?
How can I make it harder to find answers to my exam questions online?
  • write questions that require original/creative thought rather than straightforward definitions or identification (often these are considered to be higher-order skills on Bloom’s taxonomy)
  • ask students to check for errors in a given process/solution rather than asking them to tell you the process/solution
  • ask students to show work or justify answers
How can my students submit handwritten diagrams or processes for a remote exam?
  • ask students to draw, then take a picture of their drawing and upload
  • remember to allow extra time on these questions for students to complete the upload process
  • give students flexibility and choice to use programs/software that they are comfortable with, if possible
Should I use webcams or other online proctoring software to check on students during exams?
Most institutions are discouraging instructors from online proctoring of exams, since the software is costly, not all students have access to technology and high-speed internet, and online monitoring can put students under even more stress at a difficult time. Assume your exam is open-book and that students will communicate with each other, and (see next question)
Can I re-use the question banks I develop for my remote exam in future exams?
This is not recommended, since it’s possible that students will share questions or post question banks online. It’s unfortunate that this situation means that adjunct instructors, in particular, will see an increased workload without an increase in compensation.
I’m worried that my students and I may have technical difficulties during the exam itself.
  • have a test space before the exam time for students to practice uploading or accessing materials in onQ/other learning management systems
  • encourage students to test their equipment and processes before the exam begins
  • share contact information for ITS and the instructor in case of emergencies

References

On tables of specifications:

Fives, Helenrose and Nicole Barnes. 2018. “Table of Specifications.” The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation, ed. Bruce B. Frey. Thousand Oaks, CA: SAGE Publications. DOI: https://dx.doi.org/10.4135/9781506326139

On the ICE model:

Fostaty Young, Sue and Robert J. Wilson. 2000. Assessment and Learning: The ICE Approach.

Winnipeg: Portage and Main Press.

On the ICE model and tables of specifications:   

Fostaty Young, Sue. 2005. “Teaching, learning, and assessment in higher education: Using ICE to improve student learning.” In Proceedings of the Improving Student Learning Symposium, London, UK, 13, 105-115. London: Oxford Centre for Staff and Learning.