Researchers built a tool to compare AI safety policies — but which AI you ask gets wildly different answers

A group of researchers created an automated system that uses AI language models to compare pairs of AI safety policy documents against a shared set of activity categories, producing similarity scores and summaries. The catch: different AI models produce substantially different comparison results on the same document pairs, and their scores often diverge from what human experts judge to be true.

If you're trying to figure out whether two countries' AI safety policies actually overlap or conflict, you need a reliable way to read and compare them. Right now, you can't trust an AI model to do this consistently — the tool works better than nothing, but the results depend heavily on which model you pick, which means anyone using this approach is making invisible choices about what the comparison will find. This matters because policy coordination on AI safety is already hard enough without the measurement system being unstable.