Connecting with your customers. The right way.

A/B Testing Complete GUIDE

A/B testing is the kind of practice that everyone has heard of, few have actually tried it, in the proper manner. Now, don’t go trying to say you have. We’re not here to judge, but rather observe and comment upon these observations. And what we have seen among marketers is that those who have A/B tested are rather disappointed by it. It seems that A/B testing did not deliver, as after implementing the findings, numbers did not skyrocket. Sales have not increased substantially. So, what did they do in return? They simply gave up, pointing the finger towards A/B testing, making it the bad guy out of all this affair. Well, we are here to let you know that it’s not exactly true, for three very good reasons.


1Testing is not always meant to substantially increase sales. Sure, it can happen, as it has in the past. But it’s no rule. If you are entering the testing phase with these expectations, you will most likely be disappointed. Numbers may increase, but not in a staggering manner. Sometimes they might stay as they are. What you need to remember is that testing is made to identify problems and if possible increase sales. But it does not double, triple or quadruple them. It may increase them. Try to leave false expectations by the door and remember why you are A/B testing in the first place.

2 A/B testing, although simple at a first glance, is actually quite misleading. If you get wrong even the slightest details, everything can fall apart. It’s like a Domino game. Take out one piece and everything will crumble before your eyes. And if only your A/B testing would stop when getting something wrong, but it doesn’t. You are allowed to carry on and obtain incorrect results. You have no way of knowing if your findings are correct or incorrect, maybe only if these were too outrageous to believe. Otherwise, if you are dealing with a tiny error, you can’t know when looking at your results. You simply take it as it is, implement changes and then, nothing, if you are lucky. If not, the results could seriously damage your sales. So, sure, you say that A/B testing is no good.

3 A/B testing doesn’t bring quick results. Indeed, marketers are all about quick fixes. Only solid A/B testing cannot be done overnight. a lot of time, not to mention that you need to interpret the results you obtain. And by the way, so do the implementations that follow. You can’t go expecting to see actual results immediately after making the changes.
Now that we’ve cleared up a few of the issues marketers might stumble upon when deciding to test, perhaps we should go forward to the things that matter.

What is and how to do a bit A/B testing?

Generally speaking, A/B testing is all about taking two versions and testing them against each other. Copy, images, URL-s and virtually any element you think is worthy of a test can be verified against each other in the hope of finding which version functions better in terms of conversions. For instance, you could test payment methods within the same country. However, be aware of the fact that when testing you need to have one distinct element between your two versions, the one that is actually being tested and everything else needs to stay the same. If you change as much as one aspect apart from the one being tested, your entire campaign is lost. It will no longer be conclusive.


And now for the bigger issue, how can one test in the appropriate manner? We kept on talking about the importance of correct testing, but we have not yet said what this looks like. First of all, you need to set a goal. You are about to embark on a real testing adventure, so you need to know what it is you are hoping to find. Because testing can take a long time before bringing results you can use, you need to set goals and organize them based on importance. That’s why you can’t exactly close your eyes and put your finger on an element on your website and start testing. It just doesn’t work that way. You need tools that can analyze the behavior of your customers and tell you what it is that is going wrong on your website. For instance, Google Analytics can show you how your customer behaves on your page. And so can heatmaps (visual representations of your most popular and unpopular elements found on your website), for that matter.

Collect insights from these tools and decide which elements need to be tested. For instance, your calls-to-actions might not get the attention of your audience. Or the copy is too long or perhaps too short. Remember that any element can be tested and theoretically, any element can be improved. But you need to set priorities.

Let’s say that you have identified that the design of your CTAs buttons goes unnoticed by your customers. This is the first aspect you want to improve, therefore, it is also the first one you want to test. You need to set a hypothesis. If you change the color of your CTAs buttons, customers will notice them.

Give yourself sufficient time to perform the testing. And try to consider how you are going to measure the changes. In this case, it’s the CTR, but in others, it could be conversion or revenue growth. However, when testing CTA buttons, given that they might have the same design, you could go further down the sales funnel. In other words, knowing how many people are clicking a button may not be enough. You need your customers to go further than that. That’s how you know that the improvements you will be making are actually worthwhile.

To obtain solid results, you need to test over a longer period of time. This can vary, of course, from business to business. A week, for instance, is way too short for any business. You need at least one month of testing to obtain actual data. You can check it along the way, however. Considering that your testing will be done in a dedicated platform that allows you to see how the process is developing, you can keep an eye on your projects.

And one more thing. Testing one element and implementing the results might not get you as far as you would like to get. Testing, as a practice, has been designed to be continuously used. To put it differently, when the testing of one aspect ends, a new one begins and so on. You need to constantly improve your website, your landing pages, your emails, your copy, everything. It’s a neverending job.

Multivariate Testing

Yes, there is a way through which you can test multiple elements at the same time. We know you’ve been searching for it since you heard that A/B testing is just about taking one element at a time. Multivariate testing allows you to make better use of your time and resources, and it tells you why a variant functioned better than another.

However, there is an issue, here, you need to be aware of. Multivariate testing requires a lot of traffic. Before you get too excited about this type of testing, about the benefits it could bring, do consider checking traffic before anything else.

Testing is not just testing. Testing is prioritizing, testing is planning, testing is understanding, testing is implementing and most importantly, testing is doing it all over again.

Testing is patience and organization. Testing is not about having unreal expectations. Testing is not doing whatever you think is right. Testing is using dedicated tools to show you the right track.

But in the end, testing is improving and above anything else, testing is necessary.