LLM Evaluation for Ecommerce: How to Test What AI Picks

LLM evaluation in ecommerce means testing whether a language model completes real shopping tasks correctly: finding the right product, comparing it against others, and getting it into a cart. For a store there are two versions of this. You can test a chatbot you run on your own site, or you can test how outside AI agents treat your products. Whether people google, go to Amazon or ask a chatbot, AI now gives them a response with a product, and most e-commerce brands have no idea how to be that product.

My take: most evaluation talk is aimed at people building the model, and almost none of it at the person who has to sell through it. If you run a store, you don’t need to grade the model. You need to know why it picks someone else’s product, which is a different exam, and the store is the one sitting it.

TL;DR

• Evaluating a shopping agent means scoring the whole loop (search, clarification, recommendation, add to cart), not single answers.
• Current research shows even strong models fail a lot of shopping tasks, so your product data has to do more work than you’d hope.
• Agents read your product feed, catalog and structured data, not your homepage. Missing identifiers, vague prices and unquantified specs get your product dropped.
• For a store owner, the useful test is a head-to-head: the same shopper question, your product against a named competitor’s, repeated several times.
• Fix the data, publish, run it again. One scan is a snapshot, a series of runs is evidence.

Two meanings of LLM evaluation in ecommerce

The phrase covers two jobs that people mix up.

The first is evaluating an LLM that you deploy: a shopping assistant on your site that answers questions, suggests products and adds them to a cart. Here you choose the model, you control the prompt, and you can see every conversation. The question is whether the assistant is any good.

The second is evaluating how third-party AI agents treat your store. You don’t choose the model. The shopper does, often without knowing it. The agent never sees your homepage. It reads your product feed, your catalog and the structured data on product pages, then decides which product to put in front of the shopper. The question is whether your products survive that process.

Most of the academic work is about the first job. It is still worth reading for the second, because it shows what a shopping task looks like when you score it properly, and where models stumble.

What the benchmarks actually test

A researcher at Rezolve published a RecSys 2026 workshop paper that benchmarks 12 models as conversational shopping agents across the full loop of search, clarification, recommendation and add to cart. It scores 1,200 conversations (five runs across 20 scenarios) using a five-rubric LLM judge, plus behavioral metrics read from conversation traces. The rubrics cover intent understanding, clarification, recommendation quality, add-to-cart correctness and grounding. Grounding is the plain question of whether the agent’s claims match the product data it was given.

Algolia’s leaderboard takes a similar view from the search side. Each model runs inside a production retrieval agent and is scored on query rewriting, tool calls, retrieval and response synthesis, reading the response text, the tool calls and the retrieved records. So the model is judged as part of a system, not as a trivia contestant.

The AION paper goes further on rigor. It builds a deterministic shopping environment where each trial’s customer persona, difficulty and target cart are fixed in advance, then tests eight open-weight agents with 160 trials each and 44 metrics. Its useful point is that a pass or fail at the end hides what went wrong. Keeping the tool trace and environment state lets it separate under-action, over-purchase, unsupported product attributes and poor search. Those are four different failures that all look like “the agent got it wrong.”

EComAgentBench, from the EMNLP 2026 Industry Track, uses 662 tasks built on real Amazon products and reviews. Requirements are scattered across a visible query, a hidden profile and scripted clarification, and the agent must commit to a single product. Even the strongest of seven models reached only 57.1% overall accuracy, and results got worse as requirements moved from visible to hidden sources.

DeepShop, from the University of Amsterdam, tests web agents on shopping queries of rising difficulty. Its finding is that plain retrieval struggles with complex queries, and other methods stumble on filters and sorting preferences.

Read together, these papers say the same thing in different accents. Shopping is a multi-step job, agents fail in specific and diagnosable ways, and a model that sounds confident can still pick the wrong item. Which is roughly how a lot of us shop after midnight, so at least the machines are relatable.

Metrics that matter to a store

Researchers score models. Merchants have to score outcomes. A sensible mix looks like this.

Accuracy of what the agent says about your product. One journal paper on conversational shopping assistants defines product information accuracy for price, stock and features, price range accuracy, and feature description accuracy. If an agent quotes the wrong price or invents a feature, no ranking metric will save the sale.

Selection rate. Out of repeated runs of the same shopper question, how often does the agent pick your product over the alternatives? This is the number a store owner cares about most, and it is a rate, not a yes or no, because model answers vary from run to run.

Whole-conversation behavior. Zalando’s paper on evaluating production agents argues for scoring entire conversations rather than individual turns, because aggregate scores such as relevance tell you quality moved but not what failed or how to fix it. For a merchant this translates to: don’t just count wins, read why you lost.

Breadth, not only accuracy. A paper on multi-objective recommender evaluation warns that optimizing for accuracy alone produces systems that transfer poorly in the wild, and asks for measures like diversity, coverage and fairness too. On a store that means checking the whole catalog, not only your five hero products.

Fair caveat: none of these academic metrics measure revenue. They tell you whether an agent could do the job, not whether your bank balance noticed. You still need your own order data next to them.

What agents need from your data

The cheapest evaluation insight I know is this: agents compare records, not pages.

Ecommerce Guide describes four classes of field that decide whether an item can be compared at all: a stable identifier (GTIN, MPN or SKU), a current price with currency, an availability state, and the variant axes that separate one SKU from its siblings. Miss one and the item drops out of a constrained comparison. The same source points out that price has to be a single typed number. A display value like “from $39” gives the agent nothing to check against a $60 budget, so the item gets quietly dropped.

Specs are the other weak spot. A numeric constraint can’t be checked against an adjective. “Lightweight” is a feeling, “1.2 lb” is a value. Most catalogs are full of feelings.

This matches what we see in audits. A spirits marketplace doing about €70M a year had a fault in the data agents read on 88% of its 9,699 products. When agents were asked where to buy two bottles it stocks, it came up 0 times in 20. A Swedish skincare brand’s agent checkouts reached payment in 0 of 6 countries because a shipping setting asked for a phone number. I’m not naming either store. The point is that neither fault was visible on the website. A human shopper would never have noticed. The agent noticed immediately and left, with no complaint and no review.

If you want more on how the product page itself plays into this, our guide on AI-readable product pages goes through the markup, and the piece on how AI shopping agents recommend your products covers the selection logic.

How to test your own store, step by step

You don’t need a research lab. You need a repeatable routine. This is the one I’d run.

1. Write the shopper questions. Pick 10 to 20 real questions your customers would ask, with constraints in them: a budget, a size, a use case, a delivery country. Skip your brand name, because shoppers don’t type it.
2. Pick a named rival for each product. The agent is choosing between options, so you need an option to lose to. Use the competitor that shows up most often when you search the category yourself.
3. Ask more than one model, more than once. Models disagree, and the same model disagrees with itself. In our own simulations, on the same catalog, ChatGPT picked the product about 3x as often as Claude or Gemini (51.6% against about 18.5%). One model’s verdict is not the market’s verdict.
4. Record why it lost. A loss reason like “competitor lists a GTIN, yours doesn’t” is worth ten win-rate charts. Zalando’s argument for reading whole conversations applies here too.
5. Fix the data, then rerun. Change one class of problem at a time, such as identifiers, then prices, then specs, so you know what moved the result. The Zalando framework uses repeated runs to set a baseline and paired comparisons of isolated changes, and the logic holds at store scale.
6. Test the checkout, not only the answer. Winning the recommendation and then failing at payment is the skincare story above. Walk the path an agent would take: search, cart, shipping, payment.

For step 6 on Google’s side, our UCP Playground lets you test AI shopping agents against a store. UCP is Google’s open standard for agent checkout. Buying inside the AI answer is live today on Google AI Mode and Gemini in the US through UCP. ChatGPT, by contrast, links to the store’s own checkout. OpenAI has ACP, Google has UCP, Amazon has a wall.

If you want a quick look at how agents read your site’s tooling, there is also a free WebMCP shopping audit.

Evaluating a chatbot on your own site

If your real job is the first meaning, a branded assistant on your storefront, borrow the benchmark habits. Test multi-turn conversations, not one-off questions. Check each suggested product and variant against what the shopper said and against verified product data. Check that requirements survive follow-up questions. Check what happens when the item is out of stock. Those are the cases where assistants make things up.

There are plenty of examples of AI in ecommerce that show where on-site assistants earn their keep, and the rubric above tells you whether yours does.

One warning from me. A great on-site assistant doesn’t help the shopper who never visits your site. The outside agent has already chosen. So the data work in the previous section comes first, and the assistant second. I think that order is obvious once you say it out loud, and almost nobody follows it.

Keeping the results honest over time

Models change, catalogs change, competitors change their feeds. A score from March tells you little about November.

Treat evaluation as a loop. Run the scan, read the loss reasons, fix the data, publish, run it again. Keep the shopper questions fixed between runs, or you are comparing different exams. When you add a new product line, add its questions.

Also track what you can actually observe: visits from AI assistants in your analytics, and orders on your side. In our first pilot, DUA Flavours saw visits from AI assistants and direct traffic rise 12% from April to July 2026. That is a visits number, not a sales number, and I’d rather say so than let you assume otherwise. Our guides on measuring AI agent traffic and attribution tools go into the measurement side.

Perspective: why we test the store, not the model

We have run 36,797 simulated purchases as of October 2026, and the main thing they taught us is where the problem sits. It is rarely in the model. It is in the product feed, the catalog and the checkout the agent has to use.

That is why ecentic works on the store itself. We connect to Shopify, WooCommerce or a product feed, and have ChatGPT, Claude and Gemini compare each product with a named competitor’s. We show why the agent picks the other one, in plain English. When you click Apply, we rewrite the title, description and structured data and publish them to your store.

On Shopify, we push the fixes into Google Merchant Center and track visits from AI assistants. On WooCommerce, the free ecentic plugin lets agents search, fill a cart and check out through UCP. That checkout part is what plain feed tools don’t do.

I think a store that waits for the models to get better is making a bet it doesn’t control. The models will improve, and they will still read your data. Better to have data worth reading.

Xhurian Shaba

Run the first test on your own products

The quickest way to see your own numbers is the free scan. It covers your top 10 products, needs no card, and shows how ChatGPT, Claude and Gemini compare them with a named competitor’s, and why you lose when you do.

From there, the plans on our pricing page are Compete at $14.90 per month for up to 100 products, Scale at $44.90 per month for up to 1,000 products with one-click fixes, and Dominate at $149 per month, which adds Google Merchant Center sync and agent checkout setup. There is also pay-as-you-go at 3% of AI-attributed orders, capped at $500 per month. If you want the listing side first, see AI product listing optimization. We have platform pages for Shopify and WooCommerce too.

FAQ

What is LLM evaluation in ecommerce?

It is the process of scoring how well a language model handles shopping tasks such as search, clarifying questions, recommendations and add to cart. Good evaluations score whole conversations and check answers against real product data. For merchants, it also means checking how outside AI agents treat your products.

How do I know if AI assistants recommend my products?

Ask realistic shopper questions, with budgets and constraints, across several models and several runs. Count how often your product is picked against a named competitor’s, and note the reason when it isn’t. A single answer from a single chatbot is an anecdote, a repeated set of runs is a measurement.

What product data do AI agents need?

At minimum, a stable identifier such as a GTIN, MPN or SKU, a current price with currency as a single number, an availability state, and clear variant options. Specs should be measurable values, not adjectives. Without these, an agent can’t compare your item and tends to drop it.

How do I integrate AI data into an ecommerce store?

Start with the product feed and the structured data on your product pages, since that is what agents read. ecentic connects to Shopify, WooCommerce or a product feed, rewrites titles, descriptions and structured data when you click Apply, and publishes them to the store. Then rerun your test to see what changed.

Do benchmark scores tell me how my store will do?

Not directly. Benchmarks show how models behave on shopping tasks in general, and EComAgentBench’s top score of 57.1% shows there is plenty of room for error. Your own catalog and your own competitors decide your result, so test those.

Sources

• Benchmarking Models for Conversational E-Commerce (Rezolve, RecSys 2026 workshop)
• Algolia LLM Leaderboard: methodology
• Evaluating Open-Weight E-Commerce Agents with Environment-Grounded Verification
• EComAgentBench: Benchmarking Shopping Agents (EMNLP 2026 Industry Track)
• DeepShop: A Benchmark for Deep Research Shopping Agents
• Measuring and Improving AI Performance in Conversational Shopping Assistants
• E Pluribus Unum: Guidelines on Multi-Objective Evaluation of Recommender Systems
• On Evaluating and Improving Conversational Agents in Production (Zalando)
• Product Feeds for AI Agents: What Machines Need to Compare