Michael Morreau (UiT: Arctic University of Norway): “From Diverse Grading standards to Collective Acuity”

Abstract: Juries, committees and expert panels appraise all manner of things. Often they do so by awarding grades: ordered evaluative predicates such as Excellent, Average and Poor, or the qualitative probability expressions Likely, Tossup and Unlikely.

Grades are as if tailor made to facilitate individual and collective judgement. For one thing they are coarse grained, enabling timely and accurate individual inputs. Also, they express absolute judgements, not comparative ones. This means we can aggregate the several grades assigned to any one thing by the different members of the group; and that, it turns out, is enough to avoid the problem about aggregating ordinal information that Arrow’s “impossibility” theorem reveals. There is another prominent  feature that may be expected to pull in the other direction, though. Different people have different thresholds or standards for assigning grades, and they have different ones on different occasions. That can lead to biases, equivocation and misunderstandings.

We’ll see that when there is uncertainty about the extent of diversity in grading standards, collective judgements based on grades can be misleading. In extreme cases, they are under reasonable assumptions completely meaningless; this follows as a corollary of Arrow’s theorem. What’s really surprising, though is that diverse grading standards are not just the liability they might seem. Under certain conditions they can in fact enable the group to make more fine-grained distinctions than the individual members can, and thus to discover objective rankings on which well-informed decisions depend. This acuity effect is explained by a simple mechanism and illustrated using a multi-agent computer simulation of a risk panel.

#LSEChoiceGroup