Introducing the Experiments Blog: Interesting Learnings from A/B Testing at Constructor

At Constructor, we run a lot of individualized A/B tests, specific to each of our customers, trying to optimize their business KPIs. It’s something we’ve done for years, and it's something unique we do out of the Search and Product Discovery vendors and technologies on the market. The primary goal of the practice is to increase our customers’ revenue, purchases, and related KPIs. 

We know that these tests have driven a lot of added revenue for our customers over the years. These were literal A/B tests, so when I say “know,” I mean that in a statistically significant way. But we realized they have some additional value as well: we now have a lot of learnings from the 1,000+ A/B tests we’ve run over the years across customers in various industries — primarily within ecommerce, but also commerce-adjacent industries along with content search. Plus, those learnings are generally interesting as data points of what sorts of techniques and strategies work well in some industries (but not in others), in different geographies, in varied user bases, and within other ways to slice the data.

In the past, we thought of this data as some of our secret sauce and shared very little outwardly about it, but lately, we’ve been having a change of heart. There are a few main reasons for this:

1. As part of our mission, we solve not only for the needs of today’s ecommerce retailers, but for the retailers of tomorrow. We believe it is our responsibility to play a role in building a vision for the future. Sharing our learnings is a small thing we can do to help the industry as a whole iterate and evolve. We think everyone can benefit, and ecommerce shopping experiences as a whole can become better.
   
   
2. The original research that currently exists on these topics is largely not statistical – it’s either opinion, or it’s individual data points about what worked on a single site, or what a handful of users said they preferred. There is certainly a place for this type of research, but as a company that strongly believes in data-backed decision making, we felt like sharing our learnings and the data behind them will provide important context, and hopefully save others from learning lessons we’ve already learned.
   
   
3. More selfishly, in terms of company policy, sharing these learnings means our competitors will likely copy them. That means we both get to influence more of the industry, and it also encourages us as a company to not rest on our laurels and encourages our mission to always find ways to make our service better for our customers. Both of these in turn mean ecommerce gets better and it gets better for a larger group of people.

The future of ecommerce will be shaped by those who innovate and push the boundaries of what’s possible. At Constructor, we’re excited to be part of that journey, and we invite you to join us in this evolving conversation.