Why is peer-review worse than coin-tossing?

Consider the following question:

If B is after A, and if C is after B, which one of the following statements is correct?

1. B is between A and C

2. C is between A and B

3. A is between B and C

4. A and B are after C

5. B and C are before A

The question is designed to test the ability to reason logically, and most people would choose option number 1. People less able to reason logically might pick any one of the options. Thus, the question appears to select against the less able.

    However, we are accustomed to letters of the alphabet being arranged in linear series, as in the text you are now reading. It is likely that both the person who set the question, and most people who read it, would assume this, probably quite unconsciously. However, if the letters were arranged in a circle, then all five of the options would be correct.

    A person with a more subtle mind would see that the arrangement of A, B and C is not specified. Instead of choosing the first option and quickly moving on to the next question, he/she would have to exclude the circle arrangement (because only one answer is correct) and perhaps other arrangements. His/her progress through the text paper would be slowed. Thus the question penalizes both those less able, and those more able, to reason logically.

    This is not just hypothetical. A study by the author of these web-pages found that, indeed, multiple-choice questions select against the more able students (Forsdyke 1978. A comparison of short and multiple choice questions in the evaluation of students of biochemistry. Medical Education 12, 351-356).

     Another, albeit anecdotal, example is by biochemist Jeffrey Toney in ASBMB Today (Dec 2017). The six year old boy was confronted with a pile of jig-saw pieces and told to put them together. Many of the pieces in the pile had fallen with the picture side down. The examiner did not tell him to be guided by the picture. He had a quick eye for shape and assembled the puzzle as a perfect rectangle with the picture face down. The teacher returned when he had finished and clicked her timer. Next he knew, his mother was in tears telling him he had failed the test!

     Most individuals in systems where research peer-review operates have already passed many selective gates (e.g. admission to university, graduate and doctoral degrees, appointment to a university or research institute). Thus, the less able are already likely to have been filtered out. All that is left for the peer-review gate to achieve is discrimination between those of average ability and the more able. This selection is against the more able (i.e. those who can think subtly about difficult problems). If the selection were merely by tossing a coin, the more able would at least have the same chance as being selected as those of average ability.

A Deep Integrity

So, if coin-tossing will not give us at least a 50:50 chance of spotting researchers of high ability, what will? Certainly, it is nothing to do with their appearance. Imagine that you suddenly found yourself by chance on a plane with your country's Olympic team. You would have little trouble picking out the small petite teenager as a likely gymnast, the tall, gangly, male as a likely high-jumper, and the heavy, thick-set male as a likely shot-putter or weight-lifter. 

    If you were similarly on a plane with your country's leading scientific researchers there would be no similar yard-stick. Researchers of talent come in all shapes, sizes, colours and sexes. The one common thread is a deep personal integrity that makes it impossible to engage in the marketing practices that the peer review system, as it currently operates, requires. Biologist Leigh Van Valen said it quite succinctly in 1976:

"The norm of our science remains dishonesty, because it is made necessary for the survival of creative research. Often one may either be honest, or continue in science, but not both."

      The Olympians who share your flight did not get selected to represent their country on the basis of what they proposed to do in the future. They were selected because they had demonstrated that they were the agilest, could jump the highest, and could lift the mostest! Similarly, if you were on a plane with top musicians you would know that each musician was there because he/she had repeatedly and consistently demonstrated his/her musical excellence. 

    Yet, of our researchers, while giving lip-service to track-record, we demand that they market a project that will be assessed by "peers." This requires that they discard projects that are difficult to market even though they believe they are the best projects, and propose projects that are easy to understand and are likely to receive matching funds from the pharmaceutical industry. Scientists of deep integrity can no more do this than the elfin gymnast could have lifted weights, or the weight lifter could have high-jumped! 

    Fortunately, there are some fine medical researchers of high integrity who somehow make it in our strange peer review funding system. But if many medical researchers are selected because of their skills in marketing, rather than in research, should we be surprised that, as pointed out by bioethicist, Jackie Smith (2003; Toronto Globe & Mail. 14th Jan):

"the genetic era is seeing legitimate and rogue scientists lose sight of human rights and dignity in a race for publicity, scientific glory and lucrative contracts"

And when these same people are called upon as the acknowledged "experts" to advise on matters that require high expertise, do we get the best advice?

Return to Peer Review Index

Return to HomePage

Established circa 2003 and last edited 13 Jan 2018 by Donald Forsdyke