What is eCommerce A/B Testing?
eCommerce A/B testing is a scientific method of evaluating a marketing variable compared to a control. This occurs when brands show a percentage of users either a control or variable version of a marketing asset. When properly executed, this exercise helps businesses evaluate a variable’s effectiveness related to the control.
eCommerce A/B testing mirrors the scientific method and helps businesses isolate individual elements that influence results. Once brands identify a winning set, they can apply their findings for remaining users and future marketing materials.
Overall, this eCommerce testing process creates an ongoing cycle of optimization and contributes to smarter marketing decisions.
eCommerce Testing Ideas
With advances in eCommerce and marketing technology, brands can a/b test almost any asset. However, we’ve seen that a few specific areas are particularly well suited for testing.
Calls to Action
Consider evaluating button color, size, font weight, capitalization, messaging and shape. Each factor can affect users differently, so be sure to take your time and evaluate ways to constantly improve.
Evaluate heading, button, form and image placement. Each of these elements contributes to the overall user experience and can either help or hurt the purchase process.
We frequently evaluate product detail and checkout pages as a part of a routine eCommerce AB testing process. These pages offer high conversion value for brands and should be a key focus for future success.
Testing the order and content of your product images can drastically influence consumer purchasing decisions. Consider evaluating lifestyle vs isolated product images on your product detail pages. Additionally, you can test if user-generated images perform better than branded imagery in various places throughout your store.
Testing individual or bundling pricing can help brands better understand how these elements directly influence conversion. If possible, we encourage brands to evaluate their pricing/conversion relationships and consider how they ultimately influences profitability. You may find that you can adjust pricing to increase conversion without influencing profitability.
Consider testing free shipping, discounts and added perks throughout your site. Comparing these elements directly can offer better insight into what resonates with your customers. You can then use this information to inform future marketing promotions.
Template design can drastically impact eCommerce email engagement. We recommend evaluating templates with identical subject lines and messaging to see what formats lend themselves best to different marketing promotions. Consider testing placement of your logo, banner imagery, CTAs and body messaging.
Subject Lines & Preview Text
This is one of the most straightforward email elements to testing. Evaluating different email subject lines can help you see what version gets a larger open or click rate. Analyze your results and then use the subject line that will get you the most interaction.
Test banner imagery, product highlights and other graphic elements within your email content to evaluate what motivates your users to convert. Although content will vary from email to email, marketers can use this information to identify trends and inform future marketing correspondence.
How to Run an A/B Test
Determine a Variable
When setting up your A/B test there are a few settings to determine. First, just like an experiment, you will need to determine your variable. The variable should be the only thing that is different between your two alternatives.
Set Distribution Percentages
After determining what you are testing, set your distribution percentages. We recommend sending Version A to about 15% of your audience and sending Version B to about 15% of your audience. After your results come back, the remaining 70% of your audience will receive the winning version.
Determine a Winning Metric & Duration
Next, you’ll need to set the winning metric. This is the standard you will use to determine your winner. In this email subject line test example, our winning metric is "opens by delivered." HubSpot allows you to choose from three winning metrics: opens by delivered, clicks by delivered, or clicks by opens.
You then need to set your test duration. We generally recommend longer test times to ensure your results are accurate.
Lastly, don’t forget to set your fallbacks. Just like in a regular experiment, things can go wrong. Your sample size may be too small to get accurate results or they may be inconclusive. As a result, it's best to set the version you want to send if you run into one of these problems.
Statistical Significance and Data Validity
Although we firmly believe in the benefits eCommerce a/b testing can offer, we caution you to ensure that sample testing sizes are large enough to produce statistically significant results. If your audience is not large enough, the testing will be invalid.
A/B testing is based on numbers and changes are only made if the data backs it up. When testing emails and eCommerce websites, you will typically need a large user base to create a representative sample size.
Overall, eCommerce A/B testing is a process that takes time, dedication and attention. If you need help developing a roadmap to navigate your A/B testing, we would be happy to help! Please reach out through the form below to get in touch.
Get in Touch
Related Blogs & Videos
Subscribe for industry insights and resources.