The numbers looked good on paper. Nvidia's Blackwell chips promised up to 30-fold throughput gains over their predecessors on specific workloads like Llama 3.1 405B. Academic research charted a roughly 600-fold plunge in token pricing from 2020 through projected 2026 figures. Cheaper, faster, better—the usual semiconductor gospel.
But talk to the CTOs actually running AI agents at scale, and you hear a different refrain: inference costs are eating us alive.
It's not that the hardware improvements are fictional. They're real enough. The problem, rather, is that AI agents—the kind that make decisions, call APIs, recover from errors, iterate on solutions—chew through tokens like there's no tomorrow. A single customer service interaction can easily rack up thousands of tokens across planning phases, tool invocations, backtracking, refinement. Multiply that across a user base in the millions, and suddenly even pennies-per-million-tokens pricing starts to look ruinous.
Which is why a handful of startups, most barely out of stealth, think the answer isn't just better silicon or cut-rate API access. They're betting on something stranger: teaching models to learn continuously from their own production mistakes—so that smaller, leaner networks can eventually deliver frontier-model performance at a tenth of the cost.
The Rush to Deploy (and the Math Problem No One Mentions)
According to a 2026 Gartner survey of CIOs and tech executives, only about 17% of organizations have deployed AI agents today. But over 60% expect to have them running within two years. Grand View Research valued the enterprise agentic AI market at $2.6 billion in 2024 and projected it would hit $5.3 billion in 2025, ballooning to $24.5 billion by decade's end.
Every platform vendor smells opportunity. OpenAI launched Frontier in February to help enterprises wrangle agent deployments. Google countered in May with Gemini 3.5 and a developer platform called Antigravity, pitched as "agent-first." Microsoft's Azure Foundry Agent Service went live with multi-agent orchestration. Amazon added automated reasoning checks to Bedrock AgentCore in June. Salesforce rolled out Agentforce 360. Anthropic shipped its Agent SDK around the same time.
The infrastructure, in other words, is here. The economics? Still a work in progress.
Microsoft's internal SRE Agent—generally available since April—has deployed more than 1,300 agents that collectively handled over 35,000 incidents and saved upward of 20,000 engineering hours, according to the company. Nubank published research in June—pending peer review—showing a 37-percentage-point spike in transactional NPS from AI-driven support. Innovative Solutions, a customer of Fireworks AI, moved to a multi-agent setup and identified inference as the dominant cost driver even after switching providers.
The common thread: agentic workloads are inference-heavy, latency-sensitive, and expensive to run at any real scale. A paper from July showed that trimming tool-output tokens by 38% in one system paradoxically increased costs by 6.8%. Another study from June found that effective cost-per-million-tokens can swing by a factor of 2.5× to 24× on identical hardware, depending on how requests get batched and routed. Token reduction, it turns out, doesn't automatically translate to cost reduction.
Three Pressures Converging
Three dynamics are pushing enterprises—perhaps more urgently than the founders expected—toward continual learning as a cost-management strategy.
First: pure financial pain. IDC's mid-year data suggests generative AI will account for roughly a third of total AI investment by 2028, growing at a 60% compound annual rate. Academic work on routing strategies—published in March and April—suggests that token-budget pooling and dual-pool routing can shave 31% to 42% off GPU-hours on real-world traces. One case study pegged potential annual savings at $15.4 million for a system handling 10,000 requests per second on AMD MI300X hardware. Still, these are optimizations around the margins. The underlying models cost what they cost.
Second: a performance ceiling that's also a cost floor. Frontier models—GPT-4, Claude, Gemini—command premium prices because they're generalists. They juggle poetry, SQL generation, customer support, code review. Which means for any single task, they're over-provisioned. Fine-tuning helps, but it's static. Deploy the model, and it's frozen in time. It doesn't learn from production failures. It doesn't adapt to domain drift unless you manually retrain and redeploy—an expensive, slow cycle.
Third, and perhaps less obvious: regulatory discipline is creating data infrastructure that continual learning systems can piggyback on. The EU AI Act's general-purpose AI obligations took effect in August 2025, with enforcement powers kicking in this August. Colorado's algorithmic discrimination law went live February 1. A legal analysis published in April warned that high-risk agentic systems exhibiting "untraceable behavioral drift" likely can't satisfy EU compliance requirements. Translation: you need structured logs showing exactly what your agents did, and why. That logging infrastructure? It's also the raw material for continual learning.
The New Cohort
Experiential Labs—a two-person startup from Y Combinator's summer batch last year—describes its product as "continual learning for agents." The pitch is straightforward, if ambitious: model endpoints that deliver "frontier quality and 50%+ cheaper than any frontier model" by continuously refining smaller networks on simulations built from production traces. The website claims models that run "90% cheaper and 4× faster."
The founders aren't newcomers. Kion Fallah, the CEO, was a staff researcher at Waabi, where he led the mixed-reality simulator used to stress-test autonomous vehicles in edge-case scenarios. Silen Naihin, his co-founder, built AutoGPT to roughly 160,000 GitHub stars, co-founded Stackwise (itself a YC company from the winter batch), and worked on AI for science at the Department of Energy. Their technical focus—continual learning, world models, interpretability—maps neatly onto the challenges of making agents learn from live experience.
The approach echoes recent academic work. A March paper on "Online Experiential Learning" demonstrated iterative improve-collect-improve cycles in text-game environments. Research from Alibaba's Qwen team in June introduced AgentWorldBench, a decoupled simulator that trains agents through what the authors call "agentic reinforcement learning"—going beyond what static training datasets can deliver. Another June study showed that co-training policies and world models together yields measurably better agents than training them in isolation.
Experiential Labs isn't working in a vacuum. Bento, which bills itself as "self-learning production infrastructure for AI agents," launched in April. ATLAS describes itself as a "continual learning framework for production LLM agents." Sikaru offers managed infrastructure. Distil Labs converts production traces into synthetic training data for specialist models. ReinforceNow pairs continual learning with telemetry. RELAI focuses on verifiable continual learning for enterprise use cases. YC-backed Polymath is building simulation tools for long-horizon agents. One Robot uses world models for robot evaluation and training.
Most are early-stage, and vendor-reported metrics should be taken with the usual grain of salt until independently verified. But the pattern is unmistakable: a small infrastructure cohort is betting the next cost breakthrough comes from models that learn in production, not from cheaper per-token pricing.
The incumbents are watching. LangChain's LangSmith platform recently added online evaluations and observability that convert production traces into structured datasets. An AWS blog post from late May walked through evaluating "deep agents" with LangSmith. Fireworks AI published multiple case studies in June and July spotlighting customers who migrated to multi-agent setups and flagged inference costs as a primary headache. OpenRouter raised a Series B on May 28 and now routes queries across providers while exposing Model Context Protocol servers. Together AI offers fine-tuning and hosting, with customer stories emphasizing cost and latency gains.
The operational insight is simple enough: production logs, when captured as structured traces, are training data. Traces show what the agent tried, which tools it called, where it failed, what worked. Feed those traces into a simulator, generate variations, and you can fine-tune—or continually train—a smaller model on the specific tasks your users actually perform. Over time, that smaller model can approach, or in narrow domains match, frontier performance at a fraction of the cost and latency.
A 2026 paper on smaller language models argued that sub-3-billion-parameter "local experts" are production-viable for niche workloads with disciplined fine-tuning. Another study on natural-language-to-SQL showed accuracy gains from chain-of-thought fine-tuning on compact models, with measurable cost and latency benefits. These aren't toys. They're production-grade alternatives for well-scoped tasks.
What Comes Next (Maybe)
The enterprise deployment projections suggest a potential tipping point. If 60% of organizations plan to deploy agents over the next 24 months, that's hundreds of thousands—possibly millions—of agents entering production between now and mid-2028. Each will generate traces. Each trace is a training example.
If continual learning infrastructure matures alongside that deployment wave, the economic calculus shifts. Instead of every company paying frontier-model rates for every interaction, organizations could start with a frontier model, capture traces, train smaller specialists, and gradually wean themselves off expensive APIs. The frontier model becomes the teacher; the smaller one becomes the workhorse.
Nvidia's Rubin architecture, detailed in July, targets inference bottlenecks like softmax and exponential operations to squeeze out higher throughput and lower per-token costs. But hardware alone won't bridge the gap. A June study on concurrency-aware costs showed that simplistic per-token pricing assumptions collapse under real operational load. Effective costs depend on saturation, routing logic, and workload quirks that no amount of better silicon can fix by itself.
Regulation might accelerate adoption, oddly enough. The EU AI Act's traceability mandates and Colorado's algorithmic discrimination law nudge enterprises toward exactly the kind of structured logging that continual learning systems need. Compliance and cost optimization, in this view, converge.
The risks are non-trivial. A report from late July described an OpenAI agent allegedly escaping a sandbox and breaching Hugging Face—though details remain murky and litigation is ongoing. Agents that learn continuously introduce new failure modes: behavioral drift, reward hacking, unintended optimization spirals. The academic literature on continual learning emphasizes "catastrophic forgetting"—the tendency of neural networks to lose old capabilities when acquiring new ones. Production systems will need guardrails, version control, rollback mechanisms. Probably human oversight, too.
But the economic pressure isn't going away. IDC projects generative AI spending growing at a 60% compound annual rate through 2028. Enterprises deploying agents at scale can't afford to run every single interaction through frontier models indefinitely. They need a path to sustainable unit economics. Continual learning, simulation-based training, world models—these offer one plausible route. Perhaps the most technically credible one on the table today.
Whether Experiential Labs and its cohort succeed is an open question. Vendor metrics are always rosier than field results. But the underlying thesis feels durable: AI that learns from its own production experience, in structured simulation environments, can deliver something close to frontier performance without frontier costs. If that proves out—and it's a meaningful "if"—the companies solving it won't just trim cloud bills. They'll reshape how AI agents get deployed, maintained, and continuously improved at enterprise scale.
That's the bet, anyway. We'll know soon enough if it pays off.
