As agentic AI moves from experiments to real production workloads, a quiet but serious infrastructure problem is coming into focus: memory. not counting. Not a model. Memory.
Under the hood, today’s GPUs don’t have enough room to hold the key-value (KV) cache that modern, long-running AI agents rely on to maintain context. The result is a lot of invisible waste – GPUs are redoing work already done, increasing cloud costs, and performance is suffering. This is a problem that is already visible in production environments, even if most people haven’t named it yet.
In the latest stop of the VentureBeat AI Impact Series, WEKA CTO Shimon Ben-David joined VentureBeat CEO Matt Marshall to unpack the industry’s emerging “memory wall,” and why it’s becoming one of the biggest blockers to scaling truly stateful agentic AI – systems that can remember and create context over time. Talks did not just solve the problem; This created an entirely new way of thinking about memory, through an approach that WEKA calls token warehousing.
gpu memory problem
“When we’re looking at the inference infrastructure, it’s not a GPU cycle challenge. It’s mostly a GPU memory issue,” Ben-David said.
The crux of the problem is how transformer models work. To generate responses, they rely on KV caches that store information relevant to each token in the conversation. The longer the context window, the more memory the cache consumes and the faster it grows. Ben-David said a 100,000-token sequence might require about 40 GB of GPU memory.
If the GPU had unlimited memory this wouldn’t be a problem. But they don’t do that. Even the most advanced GPUs are available on only about 288 GB of high-bandwidth memory (HBM), and that space also has to be accommodated by the model itself.
In the real world, in multi-tenant inference environments, this quickly becomes painful. Workloads like code development or processing tax returns rely heavily on KV-Cache for context.
“If I’m loading three or four 100,000-token PDFs into a model, that’s it — I’ve exhausted the KV cache capacity on HBM,” Ben-David said. This is known as the memory wall. “Suddenly, the estimation environment is forced to drop data," He added.
This means that GPUs are constantly throwing away context that they will soon need again, preventing agents from being stateful and maintaining interactions and context over time.
make hidden guesses
“We constantly see GPUs in approximate environments recalculating things they’ve already done,” Ben-David said. The system prefills the KV cache, starts decoding, then runs out of space and old data is discarded. When that context is needed again, the entire process is repeated – prefill, decode, prefill again. On a larger scale, this is a huge amount of wasted work. It also means wasted energy, extra latency, and a poor user experience – while reducing margins.
That GPU recalculation waste shows up directly on the balance sheet. Unnecessary prefill cycles can cost organizations approximately 40% of overhead, this estimate is creating a ripple effect in the market.
“If you look at the pricing of big model providers like Anthropic and OpenAI, they’re actually teaching users to structure their signals in a way that increases the likelihood of accessing the same GPU that their KV cache is stored in,” Ben-David said. “If you hit that GPU, the system can skip the prefill step and start decoding immediately, letting them generate more tokens efficiently.”
But this still does not solve the underlying infrastructure problem of extremely limited GPU memory capacity.
Solutions for Stateful AI
“How do you climb that memory wall? How do you overcome it? That’s the key to modern, cost-effective inference,” Ben-David said. “We see a lot of companies trying to solve this in different ways.”
Some organizations are deploying new linear models that attempt to create smaller KV caches. Others are focusing on handling cache efficiency.
“To be more efficient, companies are using environments that compute the KV cache on a GPU and then try to copy it from GPU memory or use a local environment,” Ben-David explained. “But how do you do this at scale in a cost-effective way that doesn’t strain your memory and doesn’t strain your networking? That’s something WEKA is helping our customers do.”
The AI memory bottleneck isn’t solved simply by throwing more GPUs at the problem. “There are some problems you can’t spend enough money to solve," Ben-David said.
Enhanced Memory and Token Warehousing, Explained
WEKA’s answer calls it Augmented Memory and Token Warehousing – a way to rethink where and how KV cache data resides. Instead of forcing everything to fit inside GPU memory, WEKA’s augmented memory grid extends the KV cache into a fast, shared “warehouse” within its neuralmesh architecture.
In practice, this turns memory from a hard bottleneck into a scalable resource – without adding inference latency. WEKA says customers see KV cache hit rates increasing to 96-99% for agentic workloads, as well as efficiency gains of up to 4.2x more tokens produced per GPU.
Ben-David put it simply: "Imagine you have 100 GPUs that are producing a certain amount of tokens. Now imagine those hundred GPUs acting as if they were 420 GPUs."
For large estimation providers, the result is not just improved performance – it translates directly into real economic impact.
“Just by adding that instant KV cache layer, we’re looking at some use cases where the savings will amount to millions of dollars per day,” said Ben-David.
This efficiency multiplier also opens up new strategic options for businesses. Platform teams can design stateful agents without worrying about running out of memory budget. Service providers can offer pricing tiers based on consistent context, with cached estimates delivered at dramatically lower costs.
what comes next
NVIDIA forecasts a 100x increase in projected demand as agentic AI becomes the dominant workload. This pressure is already trickling down from hyperscalers to everyday enterprise deployments – it’s no longer just a “big tech” problem.
As enterprises move from proof of concept to actual production systems, memory persistence is becoming a core infrastructure concern. Organizations that treat this as an architectural priority rather than an afterthought will see clear benefits in both cost and performance.
The memory wall is not something that organizations can easily afford to overcome. As agentic AI scales, this is one of the first AI infrastructure limitations that forces deep rethinking, and as Ben-David’s insights make clear, memory may also be where the next wave of competitive differentiation begins.
<a href