When AI trains on what people believe, it can trap them in worse beliefs

A mathematical model shows that AI systems trained on aggregated human beliefs and fed back to those same humans can degrade collective learning if the AI updates too fast. The mechanism matters: local AI systems trained on specific communities improve learning across the board, but replacing them with one global AI system trained on everyone's beliefs makes things worse for at least some people.

We are already building systems where AI learns from what humans say and believe, then feeds recommendations back to shape what those humans see next. This paper shows there is a speed threshold: if the AI is updating faster than the underlying human beliefs it's trained on, it starts amplifying whatever beliefs were in the training data instead of improving them. The practical implication is blunt: a single global AI system trained on everyone's beliefs will hurt learning outcomes for some groups, while smaller AI systems trained on specific communities or topics consistently help. As AI aggregation becomes standard infrastructure, the architecture choice matters more than the algorithm itself.