Vision language models can't see what they can't name — and teaching them arbitrary words fixes it

The latest generation of AI image models that combine vision and language turn out to work mostly by translating pictures into words, which means they fail completely at visual tasks where the objects don't have names. Researchers tested this by showing the models pairs of images and asking them to find matching objects — the models nailed it when the objects were recognizable things (faces, shapes with names) and completely failed when they weren't (novel visual patterns with no linguistic anchor).

This is a clean failure mode: the models aren't actually limited by their ability to see or understand space — they're limited by their training process, which teaches them to convert visual information into text. The finding means current vision-language models are brittle in a specific, fixable way. What matters next is whether the field starts building models that can reason about images without forcing them through language first, or whether fixing the training pipeline turns out to be cheaper and better than rearchitecting the whole system.