AI models get worse at remembering things when their memory is compressed too much

Researchers found that when AI models compress their memory, it makes them worse at understanding new information. This means that making AI models smaller for faster use can make them less accurate.

AI models need to remember past information to make sense of new inputs. This memory, called the KV cache, takes up a lot of space. Companies try to compress this memory to make AI run faster and cheaper, especially on smaller devices. This paper shows that there is a trade-off: too much compression makes the AI less reliable, especially at a common compression level.