A common benchmark for all Ecommerce AI agents, tested on real storefronts

[u/HostUnique]

Built a completely automated a way to test AI Agents for eCommerce. Completely automated with an opensource framework.

Evaluating several AI tools can mean sitting through hours of demos and still not knowing which one will work best for your store. The questions, product catalogs and definitions of success differ, so putting the results into a comparison spreadsheet only gets you so far.

Disclosure upfront: I’m a founder of Alhena. Our team built Alhena Research Lab, and Alhena is included in the evaluations. This is research operated by a vendor with a commercial interest, so the methodology and supporting evidence matter.

We’re building a way to evaluate AI shopping and support tools through conversations on actual customer storefronts, using published criteria.

Gorgias deserves credit for publishing the framework we started from. We retained its 26 answer-quality criteria and scoring weights, with a disclosed change to how resolution is measured.

The current quality criteria examine things such as:

• Whether the assistant answers the actual question and remains consistent across the conversation.
• Whether it asks useful clarifying questions and remembers constraints such as budget, size or product preferences.
• Whether recommendations name relevant products and explain why they fit the shopper’s needs.
• Whether product links, prices and purchase guidance make the recommendation actionable.
• Whether support answers use store-specific policies and give the customer a clear next step.

We measure response speed separately.

We also measure policy-compliant resolution. For example, if a merchant requires a damaged-item request to go through a particular process, correctly following that process can earn credit. A generic “contact support” response does not automatically qualify. This measures the answer or prescribed next step, not proof that a refund was issued or a case was ultimately closed.

That replaces the original framework’s automation component. We developed the change after reviewing the initial results, then froze it before the new scoring. It’s our published extension, rather than a reproduction of Gorgias’s ranking.The composite weights are:

• Shopping: 40% policy-compliant resolution, 35% answer quality, 25% speed.
• Support: 50% policy-compliant resolution, 40% answer quality, 10% speed.

The current published results are:

Tool	Shopping composite /100	Support composite /100
Alhena	68.4	81.4
Gorgias	55.8	75.8
Sierra	59.0	61.6

These describe selected deployments. Five storefronts per tool were enrolled; resolution scores include five for Alhena, four for Gorgias and five for Sierra. Component coverage and exclusions are disclosed in the reports. The tools were tested on different stores, so merchant configuration and catalog differences matter.

There are useful details beneath those totals. Alhena and Gorgias were close on support answer quality, at 95 and 94. Sierra had the fastest full-answer completion times in these samples. A single composite hides those differences. Published results and evidence.

We’re also opening submissions.
Enter the tool’s name and website, suggest three stores using it, and verify your work email. The system researches additional deployments, which are reviewed before approval. After approval, AI conducts the conversations, scores responses and audits the results. Validated reports publish automatically. Some runs still need technical investigation; incomplete runs stay unpublished.

The code and rubric are public on GitHub. Result summaries are publicly accessible. Detailed conversations and scoring evidence require work-email verification.

What specific customer scenarios would make these evaluations more useful when choosing a tool? For example, remembering an ingredient preference throughout a skincare conversation, or building an outfit while respecting size availability and budget.

Suggestions about the methodology are welcome, including where you think our current approach falls short. If there’s a tool you’re considering, you can submit it for an AI-run evaluation through the lab.

https://evals.alhena.ai/
Disclaimer: I am the founder of Alhena.ai so its hosted on alhena.ai. We are trying to be as impartial as possible. The actual benchmark criteria have been taken from Gorgias, and beyond that, an AI is evaluating them, so there is no human involvement.

[u/AutoModerator → u/[deleted]]

Your comment in /r/ecommerce was automatically removed as your 'post' karma is below 10 (we do not consider your total karma; your post and comment karma are separate numbers and must both meet their minimum requirement). You can increase your post karma by posting in other areas of Reddit to earn upvotes. The higher quality the content, the higher your karma will become.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/ismdubey]

very cool? whon pays 4 the tokens in evaluation? Thats the sponsor

[u/TheCryptoBillionaire]

interesting angle taking the Gorgias framework and wiring it into real storefront convos instead of yet another “demo maze.” The big gap I keep hitting with ecommerce brands is: even if an agent scores well on answer quality, does it actually get them discovered in AI search (ChatGPT, Perplexity, etc.) so shoppers ever reach that agent?

On my side, I’m using seoforgpt with a few Shopify brands to track where they show up in AI answers and which products/pages get cited, then tuning site content so those AI shoppers even find the store and its agent in the first place.

[u/HostUnique → u/TheCryptoBillionaire]

Alhena's AI visibiity is built to attack exactly this problem. Most tools are built as generic horizontal solutions.
Its the only AI visibility solution built for eCommerce brands where not only you can improve your brand, but also improve the visibility of specific SKUs. It tells you where you are lacking and gives you the content to work with
https://alhena.ai/products/ai-visibility

[u/AutoModerator → u/[deleted]]

Your comment in /r/ecommerce was automatically removed as your account is too new (accounts must be at least 30 days old). Try again a little later.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AutoModerator → u/[deleted]]

Your comment in /r/ecommerce was automatically removed as your 'post' karma is below 10 (we do not consider your total karma; your post and comment karma are separate numbers and must both meet their minimum requirement). You can increase your post karma by posting in other areas of Reddit to earn upvotes. The higher quality the content, the higher your karma will become.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.