Enterprise AI applications that handle large documents or long-horizon tasks face severe memory constraints. As the context becomes longer, so does the KV cache, the area where the model’s working memory is stored.
A new technique developed by MIT researchers addresses this challenge with a faster compression method for KV caches. The technique, called Attention Matching, manages to compress the context by 50x with very little loss in quality.
Although it is not the only memory compaction technique available, Attention Matching stands out for its execution speed and impressive information-preservation capabilities.
KV Cache’s memory barrier
Large language models generate their responses sequentially, one token at a time. To avoid recalculating the entire conversation history from scratch for each predicted word, the model stores a mathematical representation of each previous token it processed, also known as a key and value pair. This important working memory is known as KV cache.
The KV cache scales with the length of the interaction because the model is forced to maintain these keys and values for all previous tokens in a given interaction. It consumes expensive hardware resources. "In practice, KV cache memory is the biggest limitation in projecting models in ultra-long contexts," Adam Zweiger, co-author of the paper, told VentureBeat. "This limits concurrency, forces smaller batches, and/or requires more aggressive offloading."
In modern enterprise use cases, such as analyzing large-scale legal contracts, maintaining multi-session customer dialogues, or running autonomous coding agents, the KV cache can grow to several gigabytes of memory for a single user request.
To solve this huge hurdle, the AI industry has tried many strategies, but these methods fall short when deployed in enterprise environments where extreme compression is necessary. A range of technical improvements include optimizing the KV cache by removing tokens deemed less important by the model or merging similar tokens into a single representation. According to the authors, these techniques work for mild compression but “degrade rapidly at higher reduction ratios”.
Real-world applications often rely on simple techniques, the most common method being to delete an old reference when the memory limit is reached. But this approach causes the model to lose old information when the context is longer. Another option is context summarization, where the system stops, writes a short text summary of the old context, and replaces the original memory with that summary. Although it is an industry standard, summarization is highly lossy and causes huge harm to downstream performance as it can remove relevant information from context.
Recent research has proven that it is technically possible to highly compress this memory using a method called cartridge. However, this approach requires training the latent KV cache model through slow, end-to-end mathematical optimization. This gradient-based training can take several hours on expensive GPUs just to compress one context, making it completely infeasible for real-time enterprise applications.
How attention matching is compressed without cost
Attention matching achieves high-level compaction ratio and quality while being orders of magnitude faster than gradient-based optimization. It bypasses the slow training process through clever mathematical tricks.
The researchers realized that to fully mimic how an AI interacts with its memory, they needed to preserve two mathematical properties while compressing the original key and value vectors into a smaller footprint. The first is “attention output”, which is the actual information that the AI extracts from its memory when interrogated. The second is “attention mass”, which serves as a mathematical weight that a token weighs relative to everything else in the model’s working memory. If compressed memory can match these two properties, it will behave exactly like the vast, original memory, even if new, unexpected user signals are added later.
"Attention matching is, in some ways, the ‘true’ purpose of doing latent context compilation, in that it directly aims to preserve the behavior of each attention head after compilation," Zweiger said. While token-dropping and related heuristics can work, explicitly matching attention behavior yields better results.
Before compressing the memory, the system generates a small set of “context queries” that serve as proxies for the types of internal searches the model performs when reasoning about a specific context. If compressed memory can accurately answer these context questions, it will later be successful in answering the user’s actual questions. The authors suggest various methods for generating these context queries, including adding a hidden hint to the document that tells the model to repeat the previous context, known as a “repeat-prefill” technique. They also suggest a “self-learning” approach where the model is prompted to perform some quick synthetic tasks on the document, such as aggregating all key facts or structuring dates and numbers into JSON format.
With these queries, the system chooses a set of keys to preserve in the compact KV cache based on signals such as the highest attention value. It then uses the keys and reference queries to calculate match values with a scalar bias term. This bias ensures that relevant information is preserved, allowing each retained key to represent the mass of multiple removed keys.
This formulation makes it possible to fit the values with simple algebraic techniques, such as ordinary least squares and non-negative least squares, completely avoiding computation-heavy gradient-based optimization. This is what makes attention matching much faster than optimization-heavy concatenation methods. Researchers also apply segmented concatenation to further improve performance on long contexts, processing contiguous parts of the input independently and combining them.
matching attention in action
To understand how this method performs in the real world, the researchers ran a series of stress tests using popular open-source models like Llama 3.1 and QUEN-3 on two different types of enterprise datasets. The first was quality, a standard reading comprehension benchmark using documents of 5,000 to 8,000 words. The second, representing a true enterprise challenge, was LongHealth, a highly dense, 60,000-token dataset containing the complex medical records of many patients.
The main finding was the ability of Attention Matching to compact the model’s KV cache by 50x without reducing accuracy, while taking only a few seconds to process documents. To achieve the same level of quality as before, the cartridges required hours of intensive GPU calculations per reference.
When working with dense medical records, standard industry solutions completely collapse. The researchers noted that when they tried to use standard text summarization on these patient records, the model’s accuracy dropped to the point where it matched a “no-context” baseline, meaning the AI performed as if it had not read the document at all.
Attention matching performs better than summarization, but enterprise architects will need to dial down the compression ratio for denser tasks than simple reading comprehension tests. As Zweiger explains, "The main practical trade-off is that if you are trying to preserve almost everything in the context of highly information-intensive tasks, you generally need a lighter compaction ratio to maintain strong accuracy."
The researchers also explored what happens in cases where absolute accuracy is not necessary but extreme memory savings do occur. They ran attention matching on top of a standard text summary. This combined approach achieved 200x compression. This successfully matches the accuracy of standard concatenation alone, but with a much smaller memory footprint.
One of the interesting experiments was to test online compaction for enterprise workflows, although they noted that this is a proof of concept and has not been rigorously tested in a production environment. The researchers tested the model on the advanced AIME mathematics reasoning test. They forced AI to solve a problem with a strictly limited physical memory limit. Whenever the model’s memory is full, the system stops, immediately compresses its working memory by 50 percent using Attention Matching, and continues thinking. Even after hitting the memory wall and having its KV cache shrink six times in a row, the model successfully solved the math problems. Its performance matched that of a model that offered massive, unlimited memory.
There are caveats to consider. At 50x compression ratio, Attention Matching is the clear winner in balancing speed and quality. However, if an enterprise attempts to push compression to the extreme 100x limit on highly complex data, the slower, gradient-based cartridge method actually outperforms it.
Researchers have released the code for attention matching. However, they note that this is not currently a simple plug-and-play software update. "I believe that latent concatenation is best thought of as a model-layer technique," Zweiger notes. "Although it can be applied on top of any existing model, it requires access to the model weights." This means that enterprises relying solely on closed APIs cannot implement it themselves; They need open-weight models.
The authors note that significant effort is still required to integrate this latent-space KV compaction into existing, highly optimized commercial inference engines. Modern AI infrastructure uses complex tricks like prefix caching and variable-length memory packing to keep servers running efficiently, and this new compaction technology will require dedicated engineering work to weave it seamlessly into those existing systems. However, there are immediate enterprise applications. "We believe post-ingestion compaction is a promising use case, where large tool call outputs or long documents are compressed immediately after being processed," Zweiger said.
Ultimately, Zweiger argues that the shift toward mechanical, latent-space compaction aligns with the future product roadmaps of major AI players. "We are seeing some enterprises themselves implementing certain models onboard providers, which is driving change." Zweiger said. "This is even more true for latent concatenation, where access to model weights is required. For example, OpenAI now exposes a black-box compaction endpoint that returns an opaque object instead of a plain-text summary."
<a href