Instructors with remote-delivery final exams may now face new challenges in order to accurately assess learning, manage accommodations and time zone differences, and address academic integrity concerns. This recorded webinar and resource describe general design considerations for remote exams and explain how to use the ICE model to design a common framework that ensures single and multiple exams are accurate assessments of student learning outcomes.
- Webinar: Designing Remote Final Exams
- General considerations about offering exams remotely
- Using a Table of Specifications to Develop Exam Questions
- The ICE Model
- Using ICE in a Table of Specifications
- Frequently Asked Questions
with Robin Attas, Centre for Teaching and Learning
This session was recorded on Thursday, March 26, 2020 from 2:30 - 3:30pm using Zoom Video Communications
Instructors with remote-delivery final exams may now face new challenges in order to accurately assess learning, manage accommodations and time zone differences, and address academic integrity concerns. This webinar will describe general design considerations for remote exams and explain how to use the ICE model to design a common framework that ensures single and multiple exams are accurate assessments of student learning.
Following are some general ideas compiled from a variety of institutions across Canada and the United States. Individual faculties, schools, and departments may have other guidelines; check with your home department/faculty for more information on specific guidelines.
Timing and accommodations logistics
If you were planning a 3 hour exam, consider reducing it to 1.5 hours so that students with time accommodations will still complete within the scheduled exam time.
Offer a longer availability window (e.g., 48 hours) and once students begin, they’ll have the correct amount of time to complete the exam. This helps students who are now in different time zones, although keep in mind that students with multiple scheduled exams may not be able to take advantage of the extended window.
Write multiple exams for different groups of students. You might group according to time zone or according to amount of time received under accommodations (e.g. no additional time, time-and-a-half, double-time).
Clearly communicate what’s allowed and not allowed to students (e.g., working with other students, sharing questions or information with other students, posting questions online, using particular materials, impersonating another student for the purposes of writing the exam). Appeal to elements of transparency, fairness, and the learning relationship between instructor and students. The University of Saskatchewan has a great email template available.
Consider having a question at the start of the exam where students agree to follow academic integrity rules.
Make a large question bank, and randomize distribution to each student so that students are less able to share questions/answers with each other. However, keep in mind that you shouldn’t use this question bank again in future years.
Limit the number of questions displayed per page and don’t allow students to go back to previous pages once they’ve submitted answers. This is not ideal from a pedagogical perspective, but it will reduce the ability of students to share questions and answers.
Remind students that we are all supposed to be social and physically distancing, so that’s one more reason they shouldn’t get together with each other in person to write the exam.
Whether you’re writing a single exam with a deep question pool, a take-home exam, or multiple equivalent exams, a table of specifications can help you plan out the overall format and scope of the exam.
A table of specifications is a method for planning out the format and questions in your exam so that you are measuring exactly what you want to measure, both in terms of content and the way students engage with the content. It ensures that you will assess learning to the same level of complexity across multiple exams, meaning that the exams can be considered equivalent in terms of student grades. Even if you are only planning a single exam, a table of specifications can be helpful. For instance, if you are giving students a choice between two or more questions, it’s important to write questions that assessing student learning to the same level of complexity. Or if you are developing a pool of questions that will be assigned randomly to students, you’d want to make sure all questions in each pool are assessing the same learning.
Ideally, your exam planning will be done alongside a consideration of what was emphasized in the course itself. For instance, if students practiced a lot of identification-type questions throughout the semester, you would ask them to do the same thing on the exam. If you spent ten weeks on one topic and only two weeks on another, you’d expect to have the final exam topics distribute in a similar proportion.
To get both the content and the way students are expected to engage with the content all into one exam, it’s helpful to align a table of specifications with a learning framework. Here we’ll use the ICE Model, developed by Sue Fostaty Young and Robert J. Wilson (2000). ICE stands for Ideas—Connections—Extensions. Ideas are basic facts or steps in a process; connections are relationships between facts or between facts and the learner; and extensions are hypotheses, applications, or extrapolations to new situations. The diagram below gives some examples of ideas, connections, and extensions for a variety of disciplines.
|naming features of coniferous trees||distinguishing between deciduous and coniferous trees||
designing a lesson plan for teaching coniferous and deciduous trees to a school group visiting a biological field station
listing properties of a particular element on the periodic table
|grouping different elements based on their properties||predicting what properties will emerge when two elements are combined|
describing a playwright’s characteristic style
explaining how a playwright’s biography led them to a particular mode of theatrical presentation
|writing or directing a short original theatrical presentation that mimics a playwright’s style|
|explaining how water flows through sand||
comparing the way water flows through a variety of materials
|designing a system for safe underground storage of nuclear waste|
|following the steps in a process for doing a patient intake||
choosing between two approaches for assessing patients
triaging patients in an emergency room by combining and selecting among a variety of intake strategies
With the ICE model selected as a learning framework, now it’s possible to use a table of specifications to combine content and what students need to do with the content into a single table to design an exam (PDF, 77KB).
A table of specifications has three main components:
- the topics you’ve covered in the course and that you want to test. You can be as fine-grained or as general as you wish.
- the total points you will assign to ideas-, connections-, and extensions-type questions with each of those topics
Totals column and row
- allows you to see how your plans distribute the assessment proportionally
- does your weighting of ideas, connections, and extensions match the learning outcomes of the course? the learning activities you emphasized all semester? your hopes for this final assessment?
- does the distribution of content match the distribution within the course?
- make sure everything adds up to 100 in the bottom right corner of the table!
- By planning everything in this table, it’s possible to then write multiple exams, or multiple questions within a particular segment, that are equivalent in terms of the assessment of student learning. The table can be adapted to suit your own needs as an instructor. In the two examples below, you’ll see that individual instructors can come up with quite different results.
In this example of a business exam, you can see that the exam is relatively balanced in terms of ideas, connections, and extensions overall, and in most content areas. However, note that ethics, for instance, is only extensions-type questions—perhaps not surprising since ethics often involves decision-making in novel situations.
In this example of a music theory exam, you can see that the content areas are more fine-grained, and that the instructor has made notes for themselves about specific question types alongside the points distribution. The exam is almost exclusively ideas-type questions, but the instructor was fine with this because connections and extensions learning outcomes were assessed with other assignments (written reflection papers and a final project).
- what’s common in your discipline?
- what types have students had practice with already?
- are there certain formats that they will need to be good at in the future (e.g., multiple choice because their accreditation exam will use that style; essay because the next course in their program uses essay exams)?
- ask students to draw, then take a picture of their drawing and upload
- remember to allow extra time on these questions for students to complete the upload process
- give students flexibility and choice to use programs/software that they are comfortable with, if possible
Most institutions are discouraging instructors from online proctoring of exams, since the software is costly, not all students have access to technology and high-speed internet, and online monitoring can put students under even more stress at a difficult time. Assume your exam is open-book and that students will communicate with each other, and (see next question)
This is not recommended, since it’s possible that students will share questions or post question banks online. It’s unfortunate that this situation means that adjunct instructors, in particular, will see an increased workload without an increase in compensation.
- have a test space before the exam time for students to practice uploading or accessing materials in onQ/other learning management systems
- encourage students to test their equipment and processes before the exam begins
- share contact information for ITS and the instructor in case of emergencies
On tables of specifications:
Fives, Helenrose and Nicole Barnes. 2018. “Table of Specifications.” The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation, ed. Bruce B. Frey. Thousand Oaks, CA: SAGE Publications. DOI: https://dx.doi.org/10.4135/9781506326139
On the ICE model:
Fostaty Young, Sue and Robert J. Wilson. 2000. Assessment and Learning: The ICE Approach.
Winnipeg: Portage and Main Press.
On the ICE model and tables of specifications:
Fostaty Young, Sue. 2005. “Teaching, learning, and assessment in higher education: Using ICE to improve student learning.” In Proceedings of the Improving Student Learning Symposium, London, UK, 13, 105-115. London: Oxford Centre for Staff and Learning.