Today, I’d like to give you an insight into an A/B test that I recently ran on my own blog, data36.com.
In the last few years, the Data36 blog has grown slowly but steadily. As of the time I’m writing this article (11 November 2019), the monthly traffic is well above 100,000 unique visitors. I feel honored to have this many people’s attention from all around the globe…
And this also puts me in an advantageous position:
Finally, I can run A/B tests on my blog with a big enough sample size!
In this article, I’d like to share the result of a landing page A/B test that I ran on data36.com not so long ago and that I found very, very interesting. (It showed something different than I initially expected.)
Let’s take a look at it!
Also, if you haven’t done so yet, I recommend going through these A/B testing articles, too:
- Statistical Significance in A/B testing (and How People Misinterpret Probability)
- A/B Testing Culture (within your Company)
Some context
Before I show you the details of the landing page A/B test itself, let me give you a little bit of context.
I have a 6-week online data science course called the Junior Data Scientist’s First Month. I have a few more products on Data36 but this one is my top priority. It’s a longer course (6 weeks long), it’s very comprehensive and it’s fairly expensive, too ($497). I’ve worked a lot on it and I’ve been continuously optimizing it ever since its first release, November 2017.
In early 2019, I took a few weeks to figure out how I could create a better landing page for the course.
I mean, the original version that I had at that time wasn’t bad… but it wasn’t great either. It had at least a few issues. I know that because I had 1-on-1 Skype meetings with almost all of the students who actually enrolled in it. So I already started this optimization project with a few important insights.
- I knew all the questions that students had but were not answered on my original landing page,
- I knew what confused them about my copy,
- I knew many of the doubts that they had before actually buying the course.
But I wasn’t sure whether I should address these on the landing page at all.
I was hesitating for one single reason. The original version was long already: 1,500 words long. It took ~7-8 minutes to read. (Plus it had a 5-minute long embedded video, too.) In today’s online world, you wouldn’t expect people to spend more time than that with your product. Or at least, I didn’t expect it.
(Note: If interested, here is this now-retired version. Note that since I don’t use it anymore, I removed the registration buttons.)
The suspicion that led to the landing page A/B test
However – thanks to the 1-on-1 sessions with the students – I had a hunch:
My suspicion was that not too many people would read through the whole landing page… but those people who really want to buy the course, they would. However long it is.
To get more evidence, I looked at my Google Analytics data and it more or less confirmed my theory. E.g. one of my analyses showed that those readers who ended up buying the course had spent ~18 minutes on average on the page before they decided to enroll.
Note: I know I use a stupid naming convention in GA. 😉
Okay. So people read. (More precisely: real potential customers read.) Great! But how much more information do they want?
That’s something I was never gonna figure out by myself.So I had to run a landing page A/B test.
The new version
It was time to create my “version B” and set up a proper experiment.
I wrote the new, extended version of my landing page. It became ~4,000 words long (that’s ~20 minutes to read) and I added 4(!) embedded videos to it (watching all those takes another 12 minutes). In general:
- I answered all the questions I could get from students,
- I addressed all their doubts,
- I made everything about the course super transparent.
But the new version became extremely long.
Okay, I had my A and B variations. (Here’s version B, in real life, by the way.)
The very last step was setting up the actual A/B test… And of course, waiting for the results.
The specifics of the experiment
By the way, here are the specifics of the landing page A/B test:
- I had two variations:
- Version A: the original 1,500 word-long page with 1 embedded video
- Version B: the new 4,000 word-long page with 4 embedded videos
- I split the visitors evenly between the two variations:
- 50% for version A
- 50% for version B
- The test ran for 3 weeks. (I needed 3 weeks to gather enough data for statistical significance. Having 100,000 unique visitors on the blog sounds nice — but the reality is that only a smaller portion of them visit this specific landing page.)
- Since I launch this course only 3 times a year and the registration period is only five days, I couldn’t use the number of purchases as my primary metric. So I had to look for the second best metric to evaluate this experiment: it was the number of waitlist subscriptions. (More accurately, it wasn’t the number but the conversion rate – so a percentage value – of waitlist subscriptions.)
- I was aiming for 99% statistical significance and for a minimum 50% conversion increase.
The results
After 3 weeks, I got the result.
And it blew my mind.
Variation B – the extremely long landing page – WON!
- +96%(!) increase in the number of waitlist subscriptions. In other words: I doubled my conversion rate!
- And the statistical significance was 99%+.
Note 1: If you are interested in the exact numbers, too, here they are:
>> version A ~3,300 visitors and 45 conversions
>> version B ~3,300 visitors and 88 conversions
It’s a smaller sample size but because of the huge difference in conversion rates it’s still a 99%+ statistical significance level.
Note 2: It’s also important to mention that later on, I followed up on this landing page A/B test and checked my actual primary metric, the number of purchases. When I launched my course, I saw that number double, too. So that confirmed the test results!
Without any false modesty I can proudly say: this was one of the most successful A/B tests I’ve run in the last few years.
The takeaway for me
I learned a lot from this experiment.
But my most important takeaway was that people do read – even for 30 minutes – before they buy an expensive product.
Remember, this is a 6-week, in-depth data science course that costs $497. This is a big investment for them, so they want to gather as much information as they can. And the best I can do is to make that available for them. It doesn’t matter if it takes 4,000 words and 4 videos. Making the course and its value super transparent and helping potential customers to make a good decision is much, much more important than the final length of the landing page!
This was a great finding for me. And I feel that I’ll be able to apply this to my future courses, too.
After all, I teach data science, so I can expect that the best and smartest readers of the blog will spend time analyzing my products and my offers before they decide to enroll.
The takeaways from this A/B testing case study for you
But that was only my takeaway.
Your takeaways from this A/B testing case study are very different.
Let me point it out because it’s important but maybe it’s not very intuitive on the first read.
- This A/B test case study doesn’t show that longer landing pages perform better than shorter ones. It shows that they perform better for my audience, for this specific product type and for this specific price category. To figure out what works for you, you have to run your own research, your own analyses and your own A/B tests.
- Also: Have you noticed how the A/B testing part was only the very last step of the whole process?
Remember, the original idea came because I talked to the students who took the course and I listened to them. And then I also tried to validate that idea with hard data. When I got to the experiment part and I created variation B, I was already fairly confident that it might perform great. The landing page A/B test was the final proof — but in itself (without the initial research) it would have been worth nothing.
Conclusion
Okay, that was it for today.
I hope you enjoyed this short A/B testing case study!
Oh, one more thing.
You might have a question. Something like: “wait a minute, isn’t it the great rule of A/B testing that you can only change one thing at a time between two variations? Tomi changed the whole landing page. And he still got great results? What’s going on here?”
In my next article, I address a few A/B testing myths and misconceptions. So continue there and find the answer for the above question. Here it is: click.
- If you want to learn more about how to become a data scientist, take my 50-minute video course: How to Become a Data Scientist. (It’s free!)
- Also check out my 6-week online course: The Junior Data Scientist’s First Month video course.
Cheers,
Tomi Mester
Cheers,
Tomi Mester
