Judge

Over the course of many years of final year medical student examination, I've noticed that students' scores seem to be pretty consistent between examiners.

To explain how, allow me to run you through a typical examination. Each medical student goes through five examination stations, and is assessed by a pair of examiners in each station. Simply put, each student is examined by ten examiners. The basic idea behind meeting five pairs of examiners is using independent assessment as a bulwark against bias.

At the end of the (tiresome) day, most examiners will sit down and have a bird's eye view of the scoreboard. When it comes to proving the coherence between two examiners, the common understanding is seeing how close the marks are within each pair of examiners.

Now let's stop and consider if a coherent marking between two examiners can prove a fair system.

Looking back on the scoreboard, I'd been tempted to believe the close-enough pairs of marks. But I know that's not the whole picture. I can't say exactly why until I read an article published in Proceedings of the National Academy of Sciences of the United States of America.

That's a study investigating 1,112 judicial rulings in which experienced judges granted parole. Being the first case of the day or just after the judges' lunch break, in fact, the prisoners were far more likely to be granted parole. Alas, that's exactly the moment when the judges felt invigorated. As defined by these two daily food breaks of the judges, each day was split into three distinct "decision sessions." The percentage of favourable rulings dropped gradually from around 65 percent to nearly zero within each decision session and returned abruptly to around 65 percent after a break. There is no better example than this to tell us how we can colour the way human make decision. But that is not what the noble judges are supposed to do, you may have wondered. Why did the judges gravitate toward the prisoners' advantage immediately after a break? As it turned out, when judges make repeated rulings, they show an increased tendency to rule in favour of the status quo, default decision of not granting parole. Such natural tendency can then be overcome (call it bribe, if you wish) by taking a break to eat a meal.

As if that's not convincing enough, the researchers considered another explanation: the judges might have in their mind a reasonable proportion of favourable decisions, and once this "quota" is filled, then unfavourable decisions follow. When the researchers included a new variable that computed the proportion of favourable decisions up to that point in the day, they didn't have a whiff of statistical evidence to support such a claim.

Ahem. Sure enough, these results contradict the conventional wisdom that human can be expected to make fair and unbiased judgment. Not even the judges. Absurd as it might seem to us, I wasn't sure if the scores from the medical student examination will show the same cyclical pattern when plotted against the the time of the examiners' tea break and lunch break. Maybe, but only if such analysis doesn't offend either party too much.