Jaber Jaber keeps publishing numbers that make infrastructure engineers do double-takes. His two-person company, RightNow AI, says it made DeepSeek V4 Flash run three times faster on identical hardware through kernel-level optimization alone—a claim based on the company's own benchmarks that awaits independent validation. No quantization tricks. No model retraining. Just surgical rewrites of how silicon talks to software.
The San Francisco startup, which emerged from Y Combinator in late 2026, now operates RunInfra—a platform serving six open-source models optimized for AI agents at prices that undercut closed APIs by substantial margins. Whether this represents a defensible business or merely repackaged open-source components that hyperscalers will swallow whole remains unclear. But the timing intersects with a fundamental shift in AI infrastructure economics that Gartner pegs at $66 billion in inference spending by 2027.
Deloitte reported in November 2025 that inference, the process of actually running trained models, would account for roughly two-thirds of AI compute in 2026. That marks an abrupt reversal from the training-dominated era that defined the previous eighteen months. As enterprises move beyond simple chatbots into multi-turn agent workflows, per-token pricing reveals only part of the cost story. The rest hides in architecture choices most developers never see.
When Every Function Call Replays the Entire Context
Agent workloads demolish traditional cost models in ways that catch finance teams off guard. Every function call replays context from scratch. Multi-turn loops run at effective batch sizes of one. Decode latency compounds across sequential tool invocations like interest on a payday loan.
RightNow published an analysis in August 2026 that quantified the spread with uncomfortable precision: "$0.09 and $290.12 are both the price of one million output tokens," Jaber wrote, identifying five cost levers that dwarf chip choice. Reasoning token volume swings cost by a factor of 14.3. Utilization patterns shift it ninefold. Concurrency configuration adds another 5.6× multiplier.
The firm's research, drawn from a July–August 2026 dataset, reached a conclusion that complicates vendor pitches: hosted APIs usually beat self-hosting on cost at typical 30% utilization rates. But data residency requirements, rate limits, model version pinning, and custom weights still push sophisticated deployments toward owned infrastructure. That's where kernel efficiency becomes the fulcrum for returns on investment.
Gartner said in mid-August that AI-optimized infrastructure-as-a-service spending would grow 96% in 2026. IDC pegged cumulative AI infrastructure buildout at nearly $1 trillion by 2029. All that capital hunts returns measured in tokens per second per dollar, not raw floating-point operations.
Selling Speed Without Losing Precision
RightNow's core pitch centers on what it calls "lossless inference"—full-precision serving tuned at the CUDA kernel and orchestration layers rather than through quantization. The company packages these optimizations as turnkey kits. A Kimi K3 "Turbo" configuration for eight NVIDIA B300 GPUs sold for $500 upfront plus $200 monthly maintenance when it launched last July.
RunInfra's catalog as of October lists models from DeepSeek, Zhipu AI (GLM 5.3 Flash), Alibaba (Qwen3.8 27B), and Recursal (Nemotron 3.5 Lightning 30B). Each comes with measured 24-hour cache-hit rates between 97.5% and 99.7%, according to company documentation.
Cached input tokens price as low as $0.01 per million on the pay-as-you-go tier. Session-aware routing keeps repeat calls close to cached context, a detail that matters more than it sounds when agents make dozens of sequential API calls. The platform exposes OpenAI- and Anthropic-compatible endpoints from a single prepaid balance. Enterprise customers get role-based access control, SOC 2 Type II compliance (attestation available under NDA), and service-level agreements up to 99.99%.
Third-party pricing trackers including LLMPricing.dev, Sovyron, and the Pi package registry now syndicate RunInfra's catalog, which suggests at least some traction among developers comparison-shopping inference providers.
The optimization gains lean heavily on upstream open-source advances, a dependency that cuts both ways for competitive positioning. The vLLM project's 2026 release notes detail a "Streaming Parser Engine" unifying tool-call and reasoning parsing across models, FlashAttention-4 FP8 key-value cache support for NVIDIA Blackwell GPUs, and key-value offloading across peer-to-peer secondary tiers. FlashInfer released version 0.7 on September 22 with production-grade fused mixture-of-experts paths and decode optimizations for sparse-attention architectures.
NVIDIA's TensorRT-LLM deprecated its Triton mixture-of-experts backend in September, giving a three-month migration window to newer CUDA Graph-based flows. A vLLM blog post co-authored with AgentX researchers in early September reported roughly 20% decode throughput gains at medium batch sizes from speculative-verification improvements, plus a 7% end-to-end boost from an upstreamed top-k sampling path.
Academic work published between April and June explored GPU/SmartNIC offload strategies (claiming 8.47× time-to-first-token reductions), disaggregated decode architectures overlapping remote attention with local feedforward layers, and concurrency-aware cost models treating agents as time-per-output-token-bounded rather than throughput-bound workloads. Whether these approaches survive contact with production systems at scale remains an empirical question.
RightNow also open-sourced AutoKernel in March, described on arXiv as an "Autonomous GPU Kernel Optimization Agent" workflow, and maintains OpenFang, a Rust-based "Agent OS" active on GitHub through the year. The strategic rationale for giving away core IP while selling hosted services invites the usual questions about moats and defensibility.
A Crowded Field Racing Toward the Same Opportunity
RunInfra enters a market thick with competitors chasing the agent wave from different angles. OpenRouter aggregates models behind a unified API with enterprise private endpoints. Together AI co-locates speech-to-text, large language models, and text-to-speech for real-time voice agents targeting sub-500-millisecond end-to-end latency, demonstrated in March. Fireworks AI offers serverless and dedicated open-weights inference across latency tiers. Cerebras and Groq pitch custom silicon, on-die SRAM architectures favored for decode-latency-sensitive loops.
Cloud incumbents package inference as managed services: NVIDIA's NIM microservices, AWS prescriptive guidance on agentic serverless architectures, Azure's agent-focused offerings. CoreWeave, Modal, Baseten, and Replicate compete on GPU rental pricing, with Modal publishing per-second rates for NVIDIA B200 and B300 chips earlier this year.
Atlas Cloud published a case study on September 20 with agent-platform vendor Lindy claiming roughly 90% lower inference cost by switching to DeepSeek V4 Flash, with 60% of input served from cache at over 3,000 requests per minute sustained. T. Rowe Price told AWS in September its self-service AI agent infrastructure costs "about $14 per seat per month" in combined infrastructure and token spend, targeting deployment of 10 to 15 agents by year-end.
Analyst houses see rapid adoption ahead, tempered by equally rapid cancellations. IDC projected more than half of enterprises using AI agents by 2027. Gartner's mid-2026 Hype Cycle for Agentic AI found 17% had deployed agents, with over 60% expecting deployment within two years. But the research firm forecast more than 40% of agentic projects would be canceled by end-2027 over cost, value, or risk concerns. That's not exactly a ringing endorsement of market maturity.
When Pricing Collapses and Regulators Circle
Inference pricing collapsed through 2025 and into this year with a velocity that surprised even bulls on AI commoditization. Artificial Analysis recorded a 75% price drop in the second quarter of 2025 alone for models scoring intelligence above 50 on its index, extending a multiyear decline trend. NVIDIA cited SemiAnalysis InferenceX data in claiming its GB300 NVL72 configuration could deliver tokens at roughly $0.123 per million at 116 tokens per second per user using Dynamo and TensorRT-LLM, updated within days of mid-October.
Regulatory pressure mounts in parallel. The European Union's AI Act general-purpose AI obligations took force August 2, 2025, with April 20 guidelines clarifying scope for open-source model providers: copyright policies, training-data summaries, and transparency obligations under Article 50. U.S. export controls tightened conditional licenses for NVIDIA H200 GPUs in January, with the Commerce Department moving in June to close loopholes on shipments to Chinese firms' overseas subsidiaries. The Federal Trade Commission issued subpoenas to major AI labs on October 15 over consumer harm incidents, signaling escalating oversight.
Hardware roadmaps promise cheaper silicon. NVIDIA's Blackwell-to-Rubin transition continues. AMD announced a "Helion Agentic AI Rack" on October 16 targeting November availability. But multiple 2026 sources emphasized software stack choices remain the dominant variable for agentic cost and latency even as chip prices fall. Deloitte's Technology, Media & Telecom Predictions argued the infrastructure gap between AI's "promise and reality" narrows through orchestration and kernel efficiency rather than raw compute alone.
The Open Question
RightNow's kernel-first approach bets that efficiency gains at the CUDA and serving-engine layers outpace chip commoditization for workloads where every millisecond of decode latency compounds across tens or hundreds of sequential tool calls. The company paused its consumer-facing RightNow Agent product in September, refocusing on hosted inference APIs. Guides remain online for developers integrating Claude Code, Codex, and other coding agents, updated as recently as October 12.
Whether a two-person team selling packaged vLLM configurations can defend margins against hyperscalers, custom-silicon vendors, and open-source distribution remains genuinely uncertain. TechRadar Pro wrote October 13 that "enterprise IT infrastructure isn't built for AI inference," citing gaps in orchestration, cooling, and cost visibility. Founders building agent loops want predictable per-task economics, not per-token averages that hide 14× spreads across workload patterns.
RightNow offers a narrow answer: faster open models, session-aware caching, and lossless tuning. The infrastructure layers are being rewritten in real time by dozens of well-funded competitors. The market will decide whether kernel optimization is a venture-scale wedge or a feature hyperscalers absorb within quarters. Jaber keeps publishing benchmarks. The numbers, at least for now, keep getting attention.
