Annotators get better at their job while they're doing it — and it changes what AI learns from them

When people label data for AI training, they improve at the task as they work, especially if they're experts. This shift in their judgment quality means the data they produce early in the process differs from the data they produce at the end — which means AI trained on mixed-quality annotations learns from a moving target instead of a stable one.

Nobody treats data annotation as teaching — it's just treated as objective labeling. But this paper shows annotators aren't recording fixed truth; they're developing judgment as they go. The practical effect: if you're training an AI system on annotations collected over months, you're mixing data from annotators at different skill levels without knowing it. The competence shift is larger for experts than novices, which means your best annotators are the ones changing most over time, which means the AI trained on their work is learning from an inconsistent signal.