Remco Heesen (Cambridge): “Statistical Biases in Peer Review”

Various biases are known to affect the peer review system, which is used to judge journal articles for their suitability for publication and grant proposals for their suitability for funding. These biases are generally attributed to cognitive biases held by individual peer reviewers. For example, gender bias in peer review is explained by the (explicit or implicit) gender bias of individual peer reviewers, as evidenced by the generally lower scores given to submissions authored by women. Here I introduce the notion of “purely statistical biases”: biases in peer review that arise even when individual peer reviewers are unbiased. This notion suggests that certain social groups or research programs may be disadvantaged by the peer review system even in the absence of cognitive biases. I use formal models to identify three possible mechanisms for purely statistical biases. The first mechanism relies on differences in information about authors available to decision makers. The second mechanism relies on differences in the underlying distributions of the “quality” of submissions. Finally, the third mechanism comes into play when reviewers judge submissions on multiple criteria: aggregating these judgments into a final decision leads to a third possible source of bias.

Remco Heesen is a Research Fellow in the Faculty of Philosophy at the University of Cambridge.