This four-year study was completed from 2013 - 2017. The study used four approaches to assessing cognitive skills, in multiple departments, over the four years. The project aimed to assess critical thinking, problem solving, written communication, and lifelong learning. Standardized instruments and surveys, program-wide rubrics, student and instructor interviews, and data linkage were used as assessment instruments in the study. 

The first step of the study was to define assessment constructs, and identify the tools best suited for assessment needs. The efficacy of the tools were investigated each year and refinements made when required. Next, we worked with instructors to explain assessment constructs and instruments being used, to determine which was suitable for inclusion of VALUE rubric scoring. Consultation and discussion lead to the development of common language for describing cognitive skills. Recruitment, testing, and course data collection were done in order for mapping of student skill demonstration to the assessment criteria to be completed. There was also investigation of costs and logistics throughout the project. 

Undergraduate students were recruited by research associates from specific courses in the participating departments (Psychology, Physics, Engineering, Drama). Student consent was sought in each year of the project. Many of the students completed the standardized tests and had their course assignment marked with the VALUE rubric, while for a range of reasons, some students only sat for the standardized tests or only had their course assignment marked using the VALUE rubric. 

The CLA+ test allocates achievement standards of below basic, basic, proficient, advanced, and accomplished. The cut points for each of the standards are calculated using the CLA+ total score. The percentage of student at each level, including sample sizes in each year is displayed. The first year CLA+ total score mean was 1155.3 (SD 117.8), and the fourth year mean was 1211.5 (SD 116.9). When compared to the US average (80 participating institutions), Queens' fourth-year students finish just under a standard deviation higher (d= .90). 

First-year and fourth-year test takers were directly or propensity matched yielding an effect size of Cohen's d= .44. These results were determined with a small sample size and should be interpreted with caution. The highest fourth-year means were demonstrated by the Bachelor of Science students with the largest gains being demonstrated by the Bachelor of Arts students. 

First-year and fourth-year test takers were directly or propensity matched yielding an effect size of Cohen's d= .65. The CAT test yielded a first year mean of 19.5 (SD=5.5), and a fourth-year mean of 23.0 (SD 4.7). Compared to average means reported by the CAT provided Queen's fourth-year mean was .76 of a standard deviation better than the US average. 

The pre-calibration (independent scoring) agreement for first-year assignments was 64%, second and third-year both 68%, and in fourth year it was 57%. Following calibration the agreement for first-year assignments was 99%, second-year 93%, third year 96%, and through lengthy calibration, there as 10% post-calibration agreement in fourth-year. The greater initial spread in fourth-year scores required longer conversations, but ended up in higher final agreement. 

The VALUE rubric scores showed that for critical thinking the first-year median was Benchmark 1, second and third-year median was Milestone 2, and the fourth-year median was Milestone 3. For problem solving and written communications, the median was Milestone 2 for first and second year, and Milestone 3 for third and fourth-year. The change from first to fourth-year were significant for all rubrics. 

The costs displayed here are indicative of costs incurred within this study, and do not reflect research or staff costs. The costs are sensitive to $USD exchange rates, and the rates of pay for the markers and proctors. Full investigation of costs should be undertaken at the time and location of any future implementation.