Researchers show that three different methods for spotting bad decisions contradict each other — which one you pick determines what you think is actually broken

Three popular methods for identifying which decisions can be improved — looking for specific wrong beliefs, measuring how confident someone is, and finding patterns that fail in both the real domain and a test domain — give completely different answers about what's actually fixable. This matters because it means when a company or hospital runs a test to see if their workers are making bad choices, the method they pick determines what they'll conclude needs fixing.

We've spent years assuming that if you could just measure whether someone is making bad decisions, you'd know how to fix them. This paper shows the measurement method is not neutral — it's a choice that predetermines the answer. If you're a healthcare system testing whether your doctors are overtreating patients, or a bank testing whether loan officers are taking too much risk, the test you run will tell you which doctors or officers are 'improvable' based on which method you chose, not based on what's actually wrong. The implication is brutal: you can't separate the diagnosis from the diagnostic tool. Pick the wrong method and you're solving the wrong problem.