The AI Memory Wall
One knowledge worker running ten parallel agents, each holding a 100,000-token context, already burns through more than 100 GB of the fastest, most expensive memory money can buy just to keep those agents alive and coherent.
Scale that pattern across the hundreds of millions of knowledge workers, analysts, researchers, customer-success teams, and eventually regular users who will run similar setups daily, and the global requirement for this specific memory stops looking like a forecast. It starts looking like a hard physical limit the industry is not currently configured to meet.
Now let's talk about memory capacity and bandwidth, specifically the kind that must sit millimeters from the compute die.
How Inference Actually Spends Memory
Every time a large language model generates a token, two big things occupy memory:
Model weights (the fixed part)
These are the hundreds of billions (or trillions) of parameters that contain the model’s learned knowledge. Frontier models are already too big for one GPU, so the weights are sharded across multiple GPUs.
During the forward pass, activations and partial results move between GPUs over ultra-high-speed interconnects like NVLink. This traffic is heavy but relatively predictable. Once the model is loaded, the weights mostly just sit there.
The KV cache (the part that explodes)
This is the real culprit for long-context and parallel agent workloads.
In a transformer, the model uses self-attention to decide which earlier tokens matter for the token it is currently generating. Computing attention from scratch every single time would be insanely wasteful. So the system pre-computes and stores two vectors for every previous token, the Key and the Value. These live in the KV cache.
Every new token generated adds a small new slice to the cache.
The cache grows linearly with context length and with the number of parallel sessions (batch size).
Its size also depends on model architecture: number of layers, number of attention heads, head dimension, and numerical precision (usually BF16 = 2 bytes per value).
Simplified real-world example (mid-size model, Grouped Query Attention):
≈ 2 (K + V) × 32 layers × 8 KV heads × 128 head dim × 2 bytes ≈ 131 KB per token. For a 128K-token session → roughly 16–17 GB just for the cache. Add real-world overhead, larger models, or multiple concurrent sessions and you easily hit the ~20 GB figure commonly cited. Frontier models with wider attention or less aggressive compression push long sessions into the 40–100 GB+ range per active workload.
Crucially, this cache must live in HBM (High Bandwidth Memory).
3D-stacked DRAM placed directly on the same package or advanced interposer as the GPU/accelerator. HBM is the only memory fast enough and physically close enough to feed the matrix multiplications without the compute units starving. Regular DDR5 or even GDDR6X is too slow or too far away. And HBM is soldered. You cannot bolt on more later.
Production inference engines (vLLM, TensorRT-LLM, TGI, etc.) use clever tricks like paged attention to reduce memory fragmentation and allow non-contiguous allocation. These help, but they do not change the fundamental capacity math.
Why Agentic Workloads Are Uniquely Hungry
Casual chat is bursty and shallow. You send a message, get an answer, maybe follow up a few times, then the context is often summarized or reset. Peak simultaneous memory demand stays modest.
Agentic systems are the opposite. A useful agent loops:
Observe → call tools (search, code, APIs — results appended) → write scratchpad/plan → decide next action → repeat.
Each loop adds tokens. A single research or customer-success agent handling a non-trivial task can easily burn 50K–100K+ tokens in one continuous run.
Ten of those agents running in parallel for one knowledge worker already demands well over 100 GB of live HBM just for the KV caches, before counting weights or other overhead.
Algorithmic wins exist but are constant factors
Grouped Query Attention (GQA) and Multi-Latent Attention (MLA), used in models like Llama 3 and DeepSeek-V2/3, let multiple query heads share fewer key/value heads or project into a compressed latent space. These techniques deliver 4–8× reductions in KV cache size and bandwidth pressure.
But they are multipliers, not exponential solvers. The variables that actually compound are:
Adoption moving from “occasional single agent” to “routine parallel agents per person”
Context windows heading toward millions of tokens
Sessions becoming longer and more stateful as agents do real multi-step work
A 4–8× efficiency gain is excellent engineering. It does not beat a variable that keeps climbing.
Supply Problems
Only three companies manufacture high-end HBM at scale: SK Hynix, Samsung, and Micron.
Entering 2026, all three had their output sold out under multi-year contracts. New wafer capacity and advanced packaging lines (the CoWoS-family interposers needed to marry HBM stacks to logic dies) take 12–18 months to bring online even when everything goes perfectly. Meaningful relief is not expected before late 2027 into 2028.
On Micron’s June 25, 2026 earnings call, CEO Sanjay Mehrotra stated plainly that the company currently sees no line of sight on when supply catches demand. Tight conditions are expected to persist beyond calendar 2027, with only gradual improvement in 2028. Micron is currently able to fill roughly half to two-thirds of the HBM its customers are asking for.
It is a multi-year structural squeeze.
The same call also named the wave that turns a two-year squeeze into a decade-long one.
A humanoid robot, Mehrotra said, will need on the order of ten times the memory of an average L2+ autonomous vehicle. L2+ vehicles are already heading past 300 GB of DRAM (various types optimized for sensor fusion and inference) per car by 2026. Ten times that is roughly 3 TB of memory per humanoid robot. Per unit, not per fleet.
Why so much?
A humanoid operating in unstructured human environments must run high-resolution vision, tactile sensing, audio, dexterous manipulation planning, natural language, safety monitoring, and long-term memory of its environment and tasks, often concurrently. That is dramatically more complex than highway autopilot.
Chipmakers see this as the start of a sustained, multi-decade memory demand cycle beginning in the latter part of this decade.
Micron’s stock already ran ~830% over the trailing year and pushed the company through a $1T market cap at points. The move happened because the current shortage was visible and priced in. The open question is whether the next leg, software agents scaling first, then embodied robotics fleets, keeps extending pricing power because the physical gap keeps widening.
Where Crypto and Decentralized Systems Actually Fit
The heaviest inference, the live matrix multiplications with hot KV caches, will remain physically tied to scarce HBM for the foreseeable future. Physics (latency, heat, interconnect) does not allow easy decentralization of that layer.
What can be decentralized is everything around persistence, coordination, and long-term state.
Agentic systems become valuable precisely when they stop acting like amnesiac interns and start acting like persistent team members who remember previous work, user preferences, tool outcomes, and learned procedures across days, weeks, and different machines.
That is exactly where crypto-native designs create leverage:
Persistent memory layers let agents write structured intermediate thoughts, observations, and checkpoints to always-available decentralized stores instead of losing them when a session ends or a machine restarts. Only the currently relevant slice needs to be pulled back into expensive HBM when the agent wakes up.
Tokenized state primitives turn agent memory into ownable, composable, verifiable objects. One agent’s output can become another agent’s trusted input. Economic mechanisms (staking, payments, slashing) can incentivize uptime and data quality.
Omnichain memory protocols and decentralized KV-style stores function like a global, sharded, encrypted key-value layer purpose-built for agent workloads. Agents can read/write/query long-term memory without relying on centralized vector databases or suffering repeated cold-start costs (re-ingesting history and rebuilding context from scratch).
When HBM is rationed and expensive, the systems that minimize repeated cold-start waste and let agent knowledge compound reliably across runs will capture real usage as agent economies scale.
The coordination and persistence layer sitting on top of the scarce silicon is where crypto-native architecture can matter most.
Three things could slow the immediate pressure:
Agent adoption arrives slower than the linear-to-exponential models assume. Enterprises may stay in simple chat interfaces longer, or reliable multi-agent orchestration may prove harder in production than in demos.
Architectural compression beats the 4–8× already modeled. Major advances in KV cache quantization, intelligent eviction, sparse attention, state-space models (Mamba and successors), or hierarchical memory systems could reduce effective HBM needs per token more dramatically than expected.
Macro or capex shock. Hyperscalers and AI labs have guided for enormous AI-related capital expenditure (hundreds of billions cumulatively in 2026). A severe recession, energy regulation, or strategic pause could temporarily slow demand realization.
None of these change the underlying direction once agentic usage becomes the default workflow for knowledge work, and especially once physical robotics fleets begin scaling in the late 2020s and 2030s.
Even after applying constant-factor efficiency gains from better attention mechanisms, the product points to memory demand in a regime the industry is not currently tooled to supply at scale.
Disclaimer: This information is for educational purposes only and does not constitute professional financial or tax advice. Some content may be developed in collaboration with third parties, and we may hold positions in the assets mentioned. We strongly recommend conducting independent research and consulting with a qualified professional before making any financial or tax-related decisions.
Substack →
Telegram →
