Demand for online shopping has never been higher.
The opportunities to improve search and discovery during this time are limitless — and as a result, many retailers are undertaking search vendor evaluations in earnest. But capitalizing on this opportunity begs the question:
“What can you do to properly evaluate search and discovery vendors? What questions can you ask to ensure you’re making the best purchasing decisions?”
By the end of this post, you will be equipped with all the knowledge and questions you need to bypass the hype and properly evaluate search and discovery solutions.
Let’s dive in:
Unique Linguistic Challenges
Linguistics suggests that words humans use most frequently see the greatest change. In our experience, there is no domain where this is more true than in ecommerce.
Stemming
Pepper and peppers may share the same stem, but they imply wildly different intent: one the spice and the other the vegetable (at least in a US context). Differentiating the meaning of similar words — including words with the same stem — is crucial to delivering effective search.
Here are some things to look out for when evaluating a vendor’s stemming abilities:
- Make sure your search vendor can accomplish basic stemming. Can they recall similar results for pizza and pizzas without manual tagging?
- Verify the vendor incorporates intelligent stemming. Can they distinguish shades of meaning between word stems? Do searches like “pepper” and “peppers” bring up different results without manual intervention?
Word Context
Beyond the stemming challenge, other linguistic headaches abound. It’s not simply a question of deciding what to show for single-word searches like “pepper” or “peppers.” Effective search also demands understanding how to rank matches on each word in many different contexts.
How should we rank things like Dr. Pepper, Bell Pepper & Sausage Supreme Pizza, Pepper Bacon, Salt & Pepper Chips, Red Pepper Soup, Lemon Pepper? In a different example, should we return Peanut Butter for “Peanuts?” What about Peanut Butter for “butter”?
All these cases require an understanding of user intent, which can only be captured with behavioral data. You can check out a recent webinar on some techniques to accomplish behavioral ranking, but ultimately we need an understanding of product attributes most and least associated with conversion for each search term and category (browse) page.
Here are some tips to make sure you make the right choice:
- Ask for a demo of the vendor’s search solution on your own catalog, with your user behavioral data. Verify rules and learnings used to generate the results.
- Verify that rules are learned automatically. Does the system require manual query rules, or does it learn these itself?
- Ask for proof that demos aren’t just manually curated through QA. Make sure the search system is not just a black box.
- Does the system incorporate behavioral learnings in its language modeling?
Retail Online = Challenges Multiplied
As commerce moves online, complexities multiply in uniquely challenging ways.
This presents a few questions you must ask the vendors you’re evaluating:
- Does the search vendor optimize results for local availability? If a particular item isn’t available locally, does the vendor have sophisticated recommendations tools to dynamically present the user with available alternatives?
- Does the search vendor optimize results based on what’s most compelling to users in different geographical locations? Different times of the day?
Personalization and “AI”: Missing in Action
Vendor claims about AI and machine learning are a dime a dozen.
But how do you know if the machine learning capabilities they promote actually work? It would be great if you could validate their claims in a 30-minute demo, but unfortunately that’s not enough.
So what can you do?
- Ensure the vendors you’re evaluating allow you to test their machine learning models with your data, before you make a purchasing decision.
- Validate the rules generated and updated from machine learning models. Are they responding to real user behavior? Can you see exactly why the models made the decisions they did (both on a broad basis and a user-by-user basis), or is the AI presented as a mysterious “black box”?
- Validate what data the models are drawing from to make decisions. Do they use:
- Behavioral data based on clickstream?
- In-session data?
- Historical affinity data?
- Regional and time-based data?
As a general rule, if a vendor uses the term “AI” to the exclusion of others, further investigation of their claims is highly recommended.
Performance & Stability
Because the demand for online shopping has never been higher, many retailers/vendors are struggling to respond to that demand at scale, causing performance to suffer and conversions as well.
Whereas a search page that takes 2+ seconds might be bad (but bearable) in apparel, it’s a death knell for other verticals like grocery. That’s because as cart sizes increase, performance becomes more and more crucial too. It is absolutely critical to ensure every visitor has a fast search and browse experience.
Here are some speed and performance considerations to validate when evaluating vendors:
- Is the vendor cloud-native? Some vendors started as hosted, single-tenant solutions and simply rebranded as cloud solutions.
- Do their systems scale up automatically to meet increased demand?
- Ask for performance reports under different load scenarios.
- Clarify that the quoted performance metrics aren’t just median times, but p95 or p99.
- Provide the vendor with a product catalog (they should be able to accept a simple CSV file) and check the performance times — look for autocomplete requests < 100ms and search request < 200ms.
Partnership Philosophy & Consultative Approach
Product discovery and search make up a huge part of the user journey, and the experience is not limited to just algorithms and merchandiser tools.
Search and browse success are determined as much by UI and UX as they are algorithms and machine learning. Ensure a vendor can help advise on important questions like:
- What does data indicate the optimal autocomplete presentation looks like?
- How many results should appear per page? Should we implement infinite scroll?
- How should we present our facet options?
Here are some pointers to check on:
- What is the vendor promising to do for your integration — will they take on responsibility for the delivery themselves or pass you to a systems integrator?
- Has the vendor provided advice on your search interface that’s backed by data?
- Will you have continuity in the team assigned to your integration, or will you work with whichever support team or engineer is fielding tickets that day?
- Will you have direct access to engineers, or will you need to be routed through a help desk?
Conclusion & Checklist
In conclusion, here’s a checklist of some of the most important things you’ll need to validate when evaluating vendors:
- Make sure your search vendor can accomplish basic stemming. Can they recall similar results for pizza and pizzas without manual tagging?
- Verify the vendor incorporates intelligent stemming. Can they distinguish shades of meaning between word stems? Do searches like “pepper” and “peppers” bring up different results without manual intervention?
- Ask for a demo of the vendor’s search solution on your own catalog, with your user behavioral data. Verify rules and learnings used to generate the results.
- Verify that rules are learned automatically. Does the system require manual query rules, or does it learn these itself?
- Ask for proof that demos aren’t just manually curated through QA. Make sure the search system is not just a black box.
- Does the system incorporate behavioral learnings in its language modeling?
- Does the search vendor optimize results for local availability? If a particular item isn’t available locally, does the vendor have sophisticated recommendations tools to dynamically present the user with available alternatives?
- Does the search vendor optimize results based on what’s most compelling to users in different geographical locations? Different times of the day?
- Ensure the vendors you’re evaluating allow you to test their machine learning models with your data, before you make a purchasing decision.
- Validate the rules generated and updated from machine learning models. Are they responding to real user behavior? Can you see exactly why the models made the decisions they did (both on a broad basis and a user-by-user basis), or is the AI presented as a mysterious “black box”?
- Validate what data the models are drawing from to make decisions. Do they use:
- Behavioral data based on clickstream?
- In-session data?
- Historical affinity data?
- Regional and time-based data?
- Is the vendor cloud-native? Some vendors started as hosted, single-tenant solutions and simply rebranded as cloud solutions.
- Do their systems scale up automatically to meet increased demand?
- Ask for performance reports under different load scenarios.
- Clarify that the quoted performance metrics aren’t just median times, but p95 or p99.
- Provide the vendor with a product catalog (they should be able to accept a simple CSV file) and check the performance times — look for autocomplete requests < 100ms and search request < 200ms.
- What is the vendor promising to do for your integration — will they take on responsibility for the delivery themselves or pass you to a systems integrator?
- Has the vendor provided advice on your search interface that’s backed by data?
- Will you have continuity in the team assigned to your integration, or will you work with whichever support team or engineer is fielding tickets that day?
- Will you have direct access to engineers, or will you need to be routed through a help desk?
