Language models built with symbolic structure need near-perfect semantic tagging to beat baseline — and that's probably impossible

Researchers tested whether language models could work better if they incorporated predicted semantic structure (how words relate to each other grammatically and conceptually). They found that the semantic tagger has to be almost perfectly accurate to help at all, and that figuring out whether it's good enough requires looking at error distributions, not just overall accuracy scores.

This is a negative result that matters more than a positive one. The paper shows there's a hard floor on what you'd need to achieve to make this hybrid approach work — and that floor might be unreachable in practice. For the past few years, researchers have been betting that combining neural networks with symbolic structure would let language models be more interpretable and efficient. This work suggests that bet only pays off if your semantic tagger works at 95%+ accuracy. Most semantic taggers work at 85–92%. That gap is the entire story.