Visa shopping tests find AI agents fail in 42.8% of runs
The benchmark tested models on their ability to uncover customers’ needs and decide whether a purchase was appropriate.
AI models failed in 42.8% of shopping test runs conducted for Visa’s new Agentic Commerce Benchmark, the company said. The assessment covered 15 tasks and tested 22 models developed by 12 labs.
The scenarios involved products including TVs, laptops and personal care appliances. Agents had to question customers to uncover requirements that were not initially disclosed, gather evidence supporting their selection and decline to purchase when no suitable product was available. Passing required identifying all customer requirements and reaching the correct purchasing decision.
Overall, 57.2% of runs passed. Individual models’ pass rates ranged from 28.9% to 82.2%. Each model attempted every scenario three times, but all three attempts were successful only 34.2% of the time. Among failed runs, 54% still resulted in a purchase, either before every requirement had been established or for a product that did not satisfy the customer’s needs.
Models that uncovered more undisclosed buying requirements performed better, Visa said. The five strongest on that measure passed 70.2% of runs, compared with 44.4% for the five weakest. Higher spending did not ensure success: the five costliest models per run passed 67.1%, versus 50.2% for the five cheapest.
Visa said the findings pointed to continuing reliability and consistency problems, rather than showing that agents were unready for commerce. It plans to expand the benchmark’s scenarios and product categories and add tests covering checkout, payments and ordering. Model providers have received the benchmark for feedback.
