Whenever there is judgment, there is noise

I shared a bit from Daniel Kahneman last year when I talked about complex language vs credible intelligence. I was recently listening to Daniel on the Hidden Brain podcast with Shankar Vedantam when Daniel raised another great observation.

His most recent book is titled “ Noise: A Flaw in Human Judgment “, and digs into how “noise” comes into many decisions made in our lives, with specific examples around insurance companies (with wildly different quotes) and sentencing lengths for various crimes. Even after factoring out race and other biases, which are certainly a problem themselves, there is still a wide range of sentences for the same crimes and that kind of disparity is known as noise.

He says it all comes down to this:

Whenever there is judgment, there is noise

While computer AI-based decisions don’t have noise, they still can’t come up with solid judgments in many cases so lots of things come back to humans, and humans create noise. The podcast and the book get into great detail on how to combat that, but it typically comes back to averaging out various human decisions. If you give a handful of people the same decision to make, and they make that decision independently of one another, the average of their answers is likely the best one.