Queen's Learning Outcomes Assessment

Learning Outcomes Project
Learning Outcomes Project

Results from the longitudinal study

Figures extracted from Queen's Learning Outcomes Consortium report, see notes under each slide for explanation.
The full report on the study will be available through the Higher Education Quality Council of Ontario website (direct link available soon). 
  • Graphic showing 4 outcomes (critical thinking, problem solving, written communication, lifelong learning), 4 Tools (CLA+, CAT, TLO, VALUE), 4 Disciplines (Engineering, Drama, Physics, Psychology), over the 4 years of the project tracking undergraduate skill development.

    This four-year study was completed from 2013 - 2017. The study used four approaches to assessing cognitive skills, in multiple departments, over the four years. The project aimed to assess critical thinking, problem solving, written communication, and lifelong learning. Standardized instruments and surveys, program-wide rubrics, student and instructor interviews, and data linkage were used as assessment instruments in the study. 

  • Graphic showing the methods for the project. Identifying gaps in curriculum, creating common language for use in communication, consideration of costs, and development of protocol.

    The first step of the study was to define assessment constructs, and identify the tools best suited for assessment needs. The efficacy of the tools were investigated each year and refinements made when required. Next, we worked with instructors to explain assessment constructs and instruments being used, to determine which was suitable for inclusion of VALUE rubric scoring. Consultation and discussion lead to the development of common language for describing cognitive skills. Recruitment, testing, and course data collection were done in order for mapping of student skill demonstration to the assessment criteria to be completed. There was also investigation of costs and logistics throughout the project. 

  • Graphic showing the sample of students who completed the different tests in each year of study.

    Undergraduate students were recruited by research associates from specific courses in the participating departments (Psychology, Physics, Engineering, Drama). Student consent was sought in each year of the project. Many of the students completed the standardized tests and had their course assignment marked with the VALUE rubric, while for a range of reasons, some students only sat for the standardized tests or only had their course assignment marked using the VALUE rubric. 

  • Graphic displaying CLA+ test results over the four years of study.

    The CLA+ test allocates achievement standards of below basic, basic, proficient, advanced, and accomplished. The cut points for each of the standards are calculated using the CLA+ total score. The percentage of student at each level, including sample sizes in each year is displayed. The first year CLA+ total score mean was 1155.3 (SD 117.8), and the fourth year mean was 1211.5 (SD 116.9). When compared to the US average (80 participating institutions), Queens' fourth-year students finish just under a standard deviation higher (d= .90). 

  • Box plot displaying the CLA+ results by program of first and fourth year students

    First-year and fourth-year test takers were directly or propensity matched yielding an effect size of Cohen's d= .44. These results were determined with a small sample size and should be interpreted with caution. The highest fourth-year means were demonstrated by the Bachelor of Science students with the largest gains being demonstrated by the Bachelor of Arts students. 

  • Box plot displaying the CAT test results by program of first and fourth year students.

    First-year and fourth-year test takers were directly or propensity matched yielding an effect size of Cohen's d= .65. The CAT test yielded a first year mean of 19.5 (SD=5.5), and a fourth-year mean of 23.0 (SD 4.7). Compared to average means reported by the CAT provided Queen's fourth-year mean was .76 of a standard deviation better than the US average. 

  • Graphic showing inter-rater reliability when using the VALUE rubrics to assess critical thinking, problem solving, and written communication.

    The pre-calibration (independent scoring) agreement for first-year assignments was 64%, second and third-year both 68%, and in fourth year it was 57%. Following calibration the agreement for first-year assignments was 99%, second-year 93%, third year 96%, and through lengthy calibration, there as 10% post-calibration agreement in fourth-year. The greater initial spread in fourth-year scores required longer conversations, but ended up in higher final agreement. 

  • Graphic displaying the change from first year to fourth year in each of the VALUE rubric criteria for critical thinking, problem solving, and written communication. Graphic displays results for psychology, engineering, and drama students.

    The VALUE rubric scores showed that for critical thinking the first-year median was Benchmark 1, second and third-year median was Milestone 2, and the fourth-year median was Milestone 3. For problem solving and written communications, the median was Milestone 2 for first and second year, and Milestone 3 for third and fourth-year. The change from first to fourth-year were significant for all rubrics. 

  • Graphic displaying the comparative costs of the CLA+, CAT, and VALUE assessments.

    The costs displayed here are indicative of costs incurred within this study, and do not reflect research or staff costs. The costs are sensitive to $USD exchange rates, and the rates of pay for the markers and proctors. Full investigation of costs should be undertaken at the time and location of any future implementation.  

Slideshow compiled by Emily Hearns and Jenny Ge (Undergraduate students, Faculty of Education, Queen's University)