As agentic AI strikes from experiments to actual manufacturing workloads, a quiet however critical infrastructure drawback is coming into focus: reminiscence. Not compute. Not fashions. Reminiscence.
Underneath the hood, at this time’s GPUs merely don’t have sufficient area to carry the Key-Worth (KV) caches that trendy, long-running AI brokers depend upon to keep up context. The result’s lots of invisible waste — GPUs redoing work they’ve already completed, cloud prices climbing, and efficiency taking a success. It’s an issue that’s already exhibiting up in manufacturing environments, even when most individuals haven’t named it but.
At a latest cease on the VentureBeat AI Influence Sequence, WEKA CTO Shimon Ben-David joined VentureBeat CEO Matt Marshall to unpack the trade’s rising “reminiscence wall,” and why it’s changing into one of many largest blockers to scaling really stateful agentic AI — programs that may keep in mind and construct on context over time. The dialog didn’t simply diagnose the problem; it laid out a brand new manner to consider reminiscence totally, by way of an method WEKA calls token warehousing.
The GPU reminiscence drawback
“After we're wanting on the infrastructure of inferencing, it isn’t a GPU cycles problem. It's largely a GPU reminiscence drawback,” stated Ben-David.
The basis of the problem comes right down to how transformer fashions work. To generate responses, they depend on KV caches that retailer contextual info for each token in a dialog. The longer the context window, the extra reminiscence these caches eat, and it provides up quick. A single 100,000-token sequence can require roughly 40GB of GPU reminiscence, famous Ben-David.
That wouldn’t be an issue if GPUs had limitless reminiscence. However they don’t. Even essentially the most superior GPUs high out at round 288GB of high-bandwidth reminiscence (HBM), and that area additionally has to carry the mannequin itself.
In real-world, multi-tenant inference environments, this turns into painful shortly. Workloads like code growth or processing tax returns rely closely on KV-cache for context.
“If I'm loading three or 4 100,000-token PDFs right into a mannequin, that's it — I've exhausted the KV cache capability on HBM,” stated Ben-David. That is what’s often known as the reminiscence wall. “All of the sudden, what the inference surroundings is compelled to do is drop information," he added.
Meaning GPUs are continuously throwing away context they’ll quickly want once more, stopping brokers from being stateful and sustaining conversations and context over time
The hidden inference tax
“We continuously see GPUs in inference environments recalculating issues they already did,” Ben-David stated. Programs prefill the KV cache, begin decoding, then run out of area and evict earlier information. When that context is required once more, the entire course of repeats — prefill, decode, prefill once more. At scale, that’s an unlimited quantity of wasted work. It additionally means wasted power, added latency, and degraded consumer expertise — all whereas margins get squeezed.
That GPU recalculation waste exhibits up immediately on the stability sheet. Organizations can endure almost 40% overhead simply from redundant prefill cycles That is creating ripple results within the inference market.
“If you happen to have a look at the pricing of huge mannequin suppliers like Anthropic and OpenAI, they’re truly educating customers to construction their prompts in ways in which enhance the chance of hitting the identical GPU that has their KV cache saved,” stated Ben-David. “If you happen to hit that GPU, the system can skip the prefill section and begin decoding instantly, which lets them generate extra tokens effectively.”
However this nonetheless doesn't remedy the underlying infrastructure drawback of extraordinarily restricted GPU reminiscence capability.
Fixing for stateful AI
“How do you climb over that reminiscence wall? How do you surpass it? That's the important thing for contemporary, cost- efficient inferencing,” Ben-David stated. “We see a number of corporations attempting to unravel that in several methods.”
Some organizations are deploying new linear fashions that attempt to create smaller KV caches. Others are targeted on tackling cache effectivity.
“To be extra environment friendly, corporations are utilizing environments that calculate the KV cache on one GPU after which attempt to copy it from GPU reminiscence or use a neighborhood surroundings for that,” Ben-David defined. “However how do you do this at scale in an economical method that doesn't pressure your reminiscence and doesn't pressure your networking? That's one thing that WEKA helps our clients with.”
Merely throwing extra GPUs on the drawback doesn’t remedy the AI reminiscence barrier. “There are some issues that you simply can’t throw sufficient cash at to unravel," Ben-David stated.
Augmented reminiscence and token warehousing, defined
WEKA’s reply is what it calls augmented reminiscence and token warehousing — a solution to rethink the place and the way KV cache information lives. As a substitute of forcing the whole lot to suit inside GPU reminiscence, WEKA’s Augmented Reminiscence Grid extends the KV cache into a quick, shared “warehouse” inside its NeuralMesh structure.
In apply, this turns reminiscence from a tough constraint right into a scalable useful resource — with out including inference latency. WEKA says clients see KV cache hit charges soar to 96–99% for agentic workloads, together with effectivity positive aspects of as much as 4.2x extra tokens produced per GPU.
Ben-David put it merely: "Think about that you’ve got 100 GPUs producing a specific amount of tokens. Now think about that these hundred GPUs are working as in the event that they're 420 GPUs."
For big inference suppliers, the consequence isn’t simply higher efficiency — it interprets on to actual financial influence.
“Simply by including that accelerated KV cache layer, we're some use instances the place the financial savings quantity can be hundreds of thousands of {dollars} per day,” stated Ben-David
This effectivity multiplier additionally opens up new strategic choices for companies. Platform groups can design stateful brokers with out worrying about blowing up reminiscence budgets. Service suppliers can supply pricing tiers primarily based on persistent context, with cached inference delivered at dramatically decrease price.
What comes subsequent
NVIDIA tasks a 100x enhance in inference demand as agentic AI turns into the dominant workload. That stress is already trickling down from hyperscalers to on a regular basis enterprise deployments— this isn’t only a “massive tech” drawback anymore.
As enterprises transfer from proofs of idea into actual manufacturing programs, reminiscence persistence is changing into a core infrastructure concern. Organizations that deal with it as an architectural precedence somewhat than an afterthought will achieve a transparent benefit in each price and efficiency.
The reminiscence wall shouldn’t be one thing organizations can merely outspend to beat. As agentic AI scales, it is likely one of the first AI infrastructure limits that forces a deeper rethink, and as Ben-David’s insights made clear, reminiscence can also be the place the subsequent wave of aggressive differentiation begins.
[/gpt3]
