Please enable javascript to view this page in its intended format.

Queen's University

Whimsey - Some Modest Proposals

Department of Psychology
To: Unit and Department Heads
From: Vern Quinsey
Subject: Some modest proposals for evaluating academic excellence
Date: 07/11/2007

Dear Colleagues:


Research-intensive universities, such as Queen’s, must take the lead in developing sound measures of academic performance because the alternative is to have measures designed for us by politicians and bureaucrats. The latter measures, as we have seen in the UK, are unlikely to embody our core values and almost certain to lack academic rigour.

Below, I sketch a series of academic performance indicators that constitute an attempt to operationalize our shared conception of academic excellence. Although the approach taken here is psychological in nature because of my own academic background, I believe that most of the indicators proposed are quite general and new measures may be developed in other disciplines following the same principles.

There are three criteria that performance measures must meet. Indicators must be quantifiable, embody our core values (i.e., be correct in a political sense), and reflect outcome as opposed to process.

Number of Global Leaders of Tomorrow

The most direct measure of our academic performance in terms of our core values is the number of global leaders-of-tomorrow we produce annually. This number is meaningful in aggregate form—the number of global leaders produced by the university or by an individual department or unit—and in individual form—the number of global leaders produced by each individual faculty member. The latter is also appropriate for annual merit evaluations. Global leaders can be identified rigorously by counting Queen’s alumnae in current Who’s Who biographies. Note that Who’s Who in Canada will not suffice, only inclusion in Who’s Who in the World satisfies the global-leader criterion. Although this works for an institution-wide measure, at the departmental level, the value of each global leader would be assigned proportionally to each department in which the leader took courses. The same principle applies to individual faculty, who would receive credit for the leader in proportion to the number of courses, or fractions of courses, he or she taught the leader. An obvious exception occurs when the leader failed the course in question—in this case the faculty member who taught the course receives no credit.

Because our measure focuses on tomorrow’s leaders, counting today’s leaders is suboptimal, albeit undoubtedly correlated with the “gold standard”. However, this problem of measurement can be minimized by including only global leaders less than sixty years of age, on the grounds that those older don’t have that many tomorrows during which they will or even can be leaders and that those younger will, on average, obey Newton’s First Law. Some astute readers may have noticed a subtle problem that remains. How can an annualised measure of current performance take into account the time lag of varying lengths that inevitably occurs between students’ graduation from Queen’s and their appearance in Who’s Who? Fortunately, a backward application of regression techniques, well known to historians, allows one to adjust for time lags of varying lengths in the analysis of event histories.

Citation analysis

Citation analysis provides a better measure of research productivity than amount of grant support because grant support is a measure of process, not outcome. What we want to measure is not the number of papers published but rather the impact they have, operationalized in terms of how many times these papers are cited in the literature. There are, however, some issues with which one has to deal in performing these analyses. First, citation analysis only works at the individual article level in the sense that it is entirely inadequate to multiply the impact rating of a journal by the number of publications because in most fields, most journal articles in even the highest impact journals are seldom, if ever, cited.

It is also widely accepted that self-citations should be removed from the total number of citations (the result is sometimes referred to as net citations). I believe, however, that further refinements of citation analysis can increase its rigour, as well as better reflect our core values. Although the self-serving nature of self-citations is obvious, the inclusive self-serving nature of citations by relatives is not. Nepotistic citation is the dark underbelly of academic performance evaluation. Once again, this problem cannot be eliminated but it can be minimized by eliminating any citation by an author who has the same last name as any author in the work being cited. Although this procedure may be biased against people with common names like Smith, Wright, or Wong, individuals with these surnames get more than their share of citations in any case. In the final analysis, if one wants to make an omelette, one must break a few eggs. The full potential of citation analysis has yet to be realized. Perhaps surprisingly, citation analysis can be used to empirically investigate one of our core values, the attainment of gender equality. One can simply count the number of male and female authors that an individual faculty member cites and form a discrimination ratio. Of course, authors with gender-neutral names, like Kelly, Kim, and Toad-face, would not contribute toward this ratio. The beauty of this procedure is that it also works in reverse: One can tabulate the discrimination ratio of authors who cite the individual faculty member. Some faculty members, indeed some disciplines, might not care to make their work appealing to opposite gendered individuals.

I look forward to discussing these issues further.

Kingston, Ontario, Canada. K7L 3N6. 613.533.2000