Key takeaways
A score needs context. The questions, assistants and test date all matter when interpreting the result.
Knowing a product is not the same as recommending it. An assistant may describe a bag accurately without suggesting it to someone asking for advice.
A recommendation can be based on a mistake. The reasons given and the accuracy of product information need separate scrutiny.
The analysis should lead to specific actions. These might include clarifying sizes, improving a description or explaining how a product is made.
A score does not predict sales. Its relationship with commercial performance requires further analysis.
A collection director receives an analysis: one of the Maison’s bags has an AI desirability score of 69 out of 100.
The score is illustrative, but the questions it raises are practical. Why 69? Which products are recommended more strongly? What could be improved? And does the score say anything about how well the bag might sell?
In digital and e-commerce, we are used to working with metrics. A conversion rate, for example, can look strong or disappointing depending on the source of traffic, the market or the time of year.
An AI desirability score deserves the same scrutiny. Before judging the number, we need to understand how it was produced.
Three questions, three different tests
A prompt is simply the request given to an assistant. How it is worded determines what we can learn from the response.
Consider three examples.
1. Asking about a specific product
“What should I know about the Loewe Puzzle Edge small bag before buying it?”
This tests how the assistant describes a named product: which features it mentions, what it gets right and what it leaves out. It does not tell us whether the assistant would have suggested the bag without being asked.
2. Asking for recommendations without naming a brand
“I’m looking for an understated leather bag for everyday use. It needs to fit my phone, a small wallet and sunglasses, and be comfortable to wear crossbody. My budget is €3,000. What would you suggest?”
Here, the assistant chooses which products to recommend. Does the bag being assessed appear? Which alternatives are suggested, and why?
3. Comparing a shortlist
“I’m considering [bag A, exact size] and [bag B, exact size]. I want something lightweight, with a secure closure, that I can wear crossbody for several hours. Which would you recommend, and what are the trade-offs?”
This tests the reasons for choosing between identified products. If the assistant claims that one is lighter or has a more secure fastening, those claims then need checking.
A product can be described accurately in the first test, omitted from the second and preferred in the third. These are different findings.
These questions illustrate three types of test. They are not a reproduction of AUGMA’s testing protocol.
Why this bag rather than another?
A product’s appearance in a response is the first observation. The explanation helps us understand why it was suggested.
Consider this recommendation:
“Its zip closure keeps your belongings secure, making it a good choice for public transport.”
If the model being assessed does not have that fastening, the recommendation is partly based on an error. Both findings need to be recorded: the bag was recommended, and one of the reasons given was wrong.
The assistant might also confuse sizes, suggesting a small bag for carrying a laptop while describing the dimensions of a larger version.
An accurate description, meanwhile, may offer little help. “Quality leather” and “elegant lines” do not explain what makes a particular model distinctive or why it suits the customer’s request.
The analysis should allow teams to review the arguments, check the facts and, where sources are available, examine the information the assistant used.
The score summarises part of this work. It does not replace reading the responses.
How was the product tested?
A bag may be well suited to a working day and less suitable for travel. It may appear in recommendations for a classic design, then disappear when the customer asks for something less familiar.
The questions therefore need to reflect plausible shopping situations and cover a range of customer needs.
Other details matter too. Which assistants were tested, in what language and on what date? Did they have access to web search? If the bag was compared with other products, how were those alternatives selected?
Without this information, a finding from a few specific situations can easily be mistaken for a general assessment.
Responses can also vary when the same question is repeated. Several observations are needed to establish whether a finding holds.
To track change over time, the test conditions must remain sufficiently consistent. If the questions or assistants change, a movement in the score may reflect the test itself.
What should the team do next?
The analysis becomes useful when it identifies a problem the team can investigate.
Two sizes are repeatedly confused. Check how the variants are named and described, including the information published by retailers.
A construction detail is poorly explained. Is it described on the product page, shown in a photograph or explained in an accessible article? Or is it mainly known to the product team and store advisers?
A bag is rarely suggested for a use it suits. Examine the alternatives being recommended and the reasons given. Its absence does not prove that the product page is inadequate. It gives the team a specific question to investigate.
These findings call for different responses. An incorrect material description needs correcting. Confusion between models requires clarification. A poorly understood craft technique may need a more substantial explanation, supported by images.
The aim is to give teams a documented set of issues, with the relevant products, responses and sources attached.
Did the changes improve the responses?
Once the content has been revised, the questions need to be asked again.
Are the sizes still being confused? Are the features described more accurately? Is the bag recommended for uses it actually suits?
A higher score does not mean that everything has improved. Some errors may remain. Equally, resolving an important misunderstanding can help customers without moving the overall score very much.
Keeping the responses and recording when changes were made allows the team to examine these differences. Further rounds of testing can then show whether the improvement lasts.
What about sales?
An AI desirability score is not a revenue forecast. It describes what was observed in the tests.
To investigate its relationship with sales, the findings need to be considered alongside available commercial data: visits, product page views, additions to basket and purchases.
Some of the customer journey may remain unobserved. A customer might discover a bag through an assistant, search for its name on Google and buy it in a store. Website data will not necessarily connect those steps.
An increase in recommendations followed by an increase in sales does not establish cause and effect either. A campaign, renewed availability or a seasonal change may explain some or all of the movement.
Measuring recommendations adds information about how customers may discover products. The commercial effect needs to be investigated rather than assumed.
Why AUGMA
We are developing AUGMA to help teams understand how assistants describe a Maison and recommend its products.
The AI Desirability Index compares Maisons within a category. Product Check examines an individual product. Because these analyses assess different things, their scores are not directly comparable.
Both need to meet the same requirement: teams should be able to revisit the responses, understand the reasoning and identify information that needs attention.
For product, content, retail and e-commerce teams, this provides a shared basis for action. They can identify a misunderstanding, decide how to address it and check whether it persists in subsequent responses.
That is what AUGMA is designed to support: understanding what sits behind a score, deciding what to change and tracking what happens next.
Explore AUGMA and its approach to analysing AI recommendations: augma-ai.com