How to Create an A/B Testing Culture

Every business that has a website or web application that is detrimental to their business model should understand the power of A/B testing. The fact is reviewing conversion optimization or landing page best practices are great to start, but every business serves a different audience with different preferences, and the same thing that works for one website will fail on another. In order to get the insights, you need to test. There’s a right way and a wrong way to test, and if you’re serious about testing more as your business grows, here are the things you’ll need to do.

Get Website Traffic and Necessary Sample Sizes

The quick answer is that most websites need a minimum of 5,000 to 10,000 visitors per variation in order to start A/B testing. The long answer is in order to understand how much traffic you need, you must first understand significance and meaningful increase in order to calculate sample size.

When A/B testing you only want to implement a variation if the test shows there’s a significant increase in your goal conversion rate. Typically you want that significance to be 90% — 95%, which essentially means you’re 90% to 95% sure the new variation is an improvement over the control. However, in order to A/B test, you need website traffic in order to satisfy the necessary sample sizes of tests. Otherwise your tests won’t reach significance.

The sample size you need all depends on the conversion rate of whatever goal you have. You can use this sample size calculator from Optimizely if you know your conversion rates and have an idea of what you anticipate the meaningful increase to be. Generally speaking, the more you change something in your test, the higher the meaningful increase should be. For example, if you’re testing an entirely different homepage design, you might have a meaningful increase of 25% but if you’re just changing the color of a button, it might only be 10%.

Let’s put it all together in an example. Let’s say you’re an ecommerce store wanting to test a simplified version of your product pages. You reduce the number of colors and distracting elements and you estimate the meaningful increase to be 20%. You also know that visitors who view a product page have a conversion rate of 3.8%. Using Optimizely’s sample size calculator, we see that in order to achieve 90% significance, you’d need 9,100 unique visitors per variation.

For the record, it’s difficult to calculate sample size perfectly because you never really know what the meaningful increase of a variation will be, but use the sample size as a gauge. It will help you prioritize tests and make the right decisions when analyzing results.

Choose an A/B Testing Tool

Optimizely, VWO, and Google Optimize are three of the main players in the space. There are definitely many others that could be a good fit for your business but these three hit the three different tiers. Here is a matrix that breaks down all the different AB testing tools.

Optimizely is the more robust tool with lots of advanced functionality. It’s great for large websites with a high amount of unique visitors. If you have more than 250,000 unique visitors monthly, this might be the tool for you.

If you have between 50,000 and 250,000 unique visitors monthly, VWO is a great tool to look into. It has all the priority integrations and a clean user-friendly interface. They provide a lot of helpful content to help you scale your conversion optimization practice.

Anything less than 50,000 unique visitors, you’ll probably want to use Google Optimize. A new Google product, it’s free and integrates into Google Analytics. The visual editor is good and they also have a code editor. It’s very easy to get started with.

Collect Ideas from Across the Organization Every Month

You shouldn’t rely on one person or one team of people for A/B testing ideas, however you could designate someone to manage the conversion optimization practice. The fact is the sales team, customer service team, even finance team, may have valuable perspectives into what customers want and what drives revenue growth.

Establish a shared document or project management tool where users can add their ideas. Create a template that’s formatted for test ideas so that all requests are uniform. Test requests should answer the following:

1. What is your hypothesis?
2. Describe each variation or provide visual mockups
3. What part of the website will the test take place on?
4. What is your primary metric for success (KPI)?

Prioritize Those Ideas and Implement Based on Priority

Every business will have a different means of prioritizing their tests. Whether it be corporate goals, seasonality, updated projections, etc. But generally speaking the best way to prioritize is to collect the following data points for each test idea:

1. How much traffic will this test affect if the variation wins?
2. What is the current value of that traffic?
3. What do you estimate the increase in value to be if the variation wins?

Try to use monthly data for this calculation. For example, let’s say there were two test ideas:

Test 1: Effects 60,000 unique visitors a month. The current value of each visitor is $0.50 and the estimated increase in value from the variation is 10%. This means the test could net $3,000 dollars a month.

Test 2: Effects 5,000 unique visitors a month. The current value of each visitor is $4.00 and the estimated increase in value from the variation is 20%. This means the test could net $4,000 dollars a month.

So even though Test 1 affects more visitors, Test 2 has higher revenue implications and should be prioritized. Obviously it’s hard to estimate the increase in value from a variation. Sometimes small changes have large impact and large changes have small impact but overtime you’re estimations on the impact of test variations will improve.

Share Relevant Results

The insights that are learned from tests can be very valuable across an organization. If you saw success when you changed the messaging on your call to actions the sales team should be notified. If you tested a new image on your homepage banner that failed, collaborate with the marketing team to get ideas as to why, or maybe do some qualitative research to get unbiased opinions.

Some ideas for sharing are a monthly newsletter or email, a shared Google doc, or use a program like JIVE as a repository.

The Takeaway

In the end, A/B testing comes down to the planning you put into it. It does cost money to get you to the conclusions you want, so it’s important to strategize before jumping into any A/B testing tools. The questions above and the types of tests that have historically been most successful can help get you on the right path and move forward.

What would you say the AB testing culture is in your industry? Is there anything you would add to the list? Let us know in the comment section below.

Amanda DiSilvestro is the Editor-in-chief for Plan, Write, GO. She has been writing about all things digital marketing, both as a ghostwriter, guest writer, and blog manager, for over 10 years. Check out her blogging services to learn more!