Researchers test whether BERT actually understands Italian grammar — it mostly doesn't

Computer scientists extracted linguistic patterns from BERT, a widely-used language AI, to see if it actually encodes the rules of Italian grammar in its internal layers. It turns out the model captures some grammatical structures but incompletely and unevenly across its layers, which means the 'understanding' claimed by language models is narrower and more fragile than it appears.

This is a narrow technical probe into how one model handles one language's grammar — it doesn't directly change what BERT can do or how it gets deployed. But it does add evidence to an emerging pattern: when you test language models against explicit linguistic theories rather than just benchmark scores, they turn out to be doing something more limited and more statistical than 'understanding.' The practical implication is slower and less obvious — it feeds into ongoing skepticism about whether these models genuinely extract language rules or merely memorize statistical associations that work well enough to pass tests. That skepticism shapes how companies and researchers build safety tests and how confident they should be in deploying these systems for high-stakes tasks like medical translation or legal document review.