ORO AI, a Y Combinator Fall 2026 startup, replaced ShoppingBench with ORO Bench on September 17, according to the company's changelog. The new open benchmark scores AI shopping agents on largely unseen tasks.
The research reviewed for this article contains no published performance figures for ORO Bench, so none are reported here. What the documentation does show is a benchmark designed to keep agents from memorizing their way to a high score.
How ORO Bench works
Daily races draw on largely unseen hidden tasks, according to ORO's research report. Those tasks come from releases that are replaced as agents adapt. Public practice tasks never appear in races.
Local practice runs happen in a separate world built through the same process. That world is not identical to the current qualifying tasks, ORO's documentation said, and existing suites are not silently rebound to different tasks.
The report also separates deterministic task compilation from a stochastic language-model shopper, meaning the tasks are fixed by design while the simulated buyer's behavior can vary. The company's public repository describes agents shopping on behalf of that simulated shopper inside sandboxed environments.
In practical terms, this is an evaluation and research platform. The research shows no launched consumer shopping assistant and no live merchant checkout integration.
Validators and rewards
ORO runs on Bittensor, a blockchain network that pays participants in tokens. Rewards flow through that network, and ORO's privacy policy said the company does not custody or transfer them and does not guarantee payment.
Validators act as the independent scorers. ORO's validator documentation lists a minimum stake weight of 40,000, though that is a threshold on stake weight, not a verified dollar cost. According to the architecture documentation, EnvPacks and delivery archives carry cryptographic digests that are checked before execution.
"Independent" has limits here. ORO operates the orchestration and aggregation infrastructure, and its terms reserve the authority to change competition rules and determine outcomes, the privacy policy said.
Winning a daily race also does not necessarily make a participant the leader on overall-score emissions, ORO's FAQ said.
Recent changes
Updated world-specific shopping environments and pass/fail scoring went live on October 22, per the changelog. Other October entries document fewer false hardcoding rejections, stricter security checks and more resilient evaluation handling. The most recent changelog entry located was dated November 1, so later changes may exist.
ORO's trajectory dataset on Hugging Face refers to 1,793 completed episodes. The company's site cautions that these are archived trajectories, not users, paying customers or deployed shopping agents.
From ShoppingBench to ORO Bench
ORO's competition began operating March 25, according to a company LinkedIn post. A March 26 changelog described the launch as a ShoppingBench-based subnet, along with an SDK and command-line tool, on-chain validators, a leaderboard, evaluation logs and a trajectory viewer.
ShoppingBench originated at Alibaba Group. Its paper was first submitted August 6, 2025, and covers more than 2.5 million real-world products. ORO did not invent it.
The newsletter notacompany listed "ORO AI launches at YC (Fall 2026)" on September 11.
Funding and competitors
Y Combinator's profile confirms ORO's Fall 2026 participation but discloses no investment terms. Dealroom lists a $125,000 seed round from Y Combinator in September 2026, and CB Insights lists a $500,000 convertible note dated September 1, 2026. Neither entry is independently confirmed, and ORO has not announced a round in the research provided. A $6 million April 2025 round raised by an unrelated ORO AI data project is not connected to this company.
Rival benchmarks are multiplying. EComAgentBench, posted June 16, 2026, has 662 tasks and hidden requirements spread across query, profile and clarification. CommerceBench, from Lazer, focuses on end-to-end commerce execution. E-Commerce Bench, an August 2026 paper, simulates 365 days of merchant operations. ARC, the Agentic Report Card, checks whether stores are ready for shopping agents and stops before payment.
ORO's repository is MIT-licensed. It says the name nods to gold in Spanish and Italian, and to where the founders met and live.
