A/B Test like a Data Scientist!
(online course)

Understand the science and the business logic behind A/B testing -- and achieve real, significant, long-term increases in conversion!

"Everything you need to know about A/B testing -- in an online video course."

90% of online businesses are doing A/B testing wrong!

This has happened to me a lot:

We were preparing a data science project, so we did an audit of previous projects. Talking to the client, she proudly mentioned that they run A/B tests from time to time. And that, in fact, they get pretty nice results.

Then we sat down to review those test results. And it almost always turned out that they made a mistake somewhere.

The promising uplifts in conversion rate were only the result of improperly implemented or wrongly evaluated tests. The 40% increases they saw were, in reality, hovering around 0%. (Or worst-case, they turned out to be 20% decreases in conversion.)

As I have written: 90% of online businesses are doing A/B testing wrong!

I created the A/B test like a Data Scientist! course so you can be the exception. I'll show you the basic concepts, best practices, and pitfalls you should avoid. I'll give you a framework, checklists and step-by-step guides. And I'll answer all the questions that an A/B testing course can possibly answer. So when you run your experiments in the future, they will be professionally executed, scientifically correct -- and most importantly, you will deliver real, long-term and outstanding results!

You'll A/B test like a Data Scientist!

Data Science + A/B testing

For running proper A/B tests, you don't have to be a data scientist. But you have to understand the scientific background of experiments.

Don't worry, it’s not as complicated as you’d think!

You have to master 3 main domains:

  • 1

    The business logic and the strategy. How can you make a real impact with A/B testing? How can you deliver real value (and real profit) for your online business?

  • 2
    The statistical and mathematical background. How can you avoid the typical statistical errors in your experiments (e.g. stopping and evaluating them too early)?
  • 3
    The research framework. Instead of testing random gut-feeling ideas (like many do), how you can prioritize your experiments and consistently keep the focus on the most promising, most impactful next steps?

Once you understand these, all you have to do is to learn a few more best practices and watch out for the typical mistakes.

And you'll be ready to run proper A/B tests with great results!

A/B test like a data scientist!

Tomi Mester - author of data36.com

Hi, I'm Tomi Mester, a data professional with 7+ years in the field. I've seen, managed and run hundreds of A/B tests. Besides, I was lucky enough to get to the industry's most important conferences and learn from the best professionals.

In the last few years, I collected the best methods (and improved them further in practice); I've seen all the typical mistakes; and I've tested all the best practices on real life projects.

Based on that experience, I created a framework and a knowledge base. If you learn, internalize and use it, you'll be able to run A/B tests in a professional manner, too.

And I put all this into an online course format.

How this A/B testing course works

The A/B Test like a Data Scientist! course is an online course. When you enroll, you'll get instant access to the course curriculum, where you will find:

  • 38 videos. (~3 hours in total) These form the backbone of the course. In these 5-10 minute long videos, I'll guide you through everything you need to know about A/B testing, step by step. (See the exact curriculum below.)
  • Step-by-step guides and checklists (in printable format). When you run your first experiments, you might need some extra support to execute everything properly. So I'll also give you a few "cheat sheets" that you can print and keep next to you for the first months. Following them, you won't overlook anything - guaranteed.
  • Extras in the Vault. I have a continuously expanding Vault section in the course curriculum, too. I’ll put extra learning materials there. In the first iteration of the course, this means further readings (my favorite articles from other professionals), case studies and more.
  • Action modules. And at the end of most modules, you'll also find lectures called: ACTION. Each of these action lectures contains a task that will help you to put the things you'll learn into practice. (After all, the ultimate goal of the course is that you'll be able to run successful A/B tests in real projects, right?)
ab testing course content

This is a compact yet comprehensive course, by design. In other words, there is everything in it that you'll need and nothing that you won't.

I intend it to be the only A/B testing course you need to take. You take this course, you listen to it, you internalize it -- and then you'll simply know everything that you have to know to run experiments properly and with great results.


Here's the exact curriculum of the course:


  • What​​​​​​​​​ is A/B testing exactly?
  • Why do yo​​​​u need it?
  • The correlation vs. causation issue
  • What can you A/B test?
  • EXTRA: Hand-picked A/B Testing case studies


  • Mindset question: Conversion Rate Optimization or Research?
  • When do you need to run an A/B test?
  • When don’t you need it?
  • When is it a really bad idea?
  • EXTRA: the "can I run this A/B test" checklist


  • The science behind A/B testing.
  • Typical mathematical mistakes.
  • Interpreting and understanding statistical significance and its true importance.
  • The right level of statistical significance (and how to calculate it)
  • EXTRA: online calculators (statistical significance and sample size)


  • STEP 1: How to prepare an A/B test properly. Initial research and prioritization. (It's at least ~80% of the job!) Success metric(s). Setting a hypothesis.
  • STEP 2: How to implement a test. Website A/B testing tool demo. About the "one-thing-at-a-time" myth and other typical mistakes.
  • STEP 3: How to run an A/B test. How long should it run? How big of a sample do you need? (+typical mistakes)
  • STEP 4: How to evaluate your results. How not to lie to yourself with statistics. What should you do when your new variation is worse than the original?
  • EXTRA: A/B Testing Hypothesis Form
  • EXTRA: The "can I start the A/B test?" checklist


  • Other A/B testing pitfalls.
  • The limitations of A/B testing
  • The art and science of A/B testing
  • EXTRA: Further reads and videos by other A/B testing experts


  • The right mindset to be successful with A/B testing.
  • Frequently asked questions and answers
  • Summary
ab testing course - statistical significance

You will get evidence-based answers to the most common questions

In the last few years, I’ve gotten a lot of questions on A/B testing. I'll answer all of them in the course. A few (but not all) of these:

  • "How long should I run a test?"
  • "How many visitors (how big of a sample size) do I need to make sure my results are real?"
  • "What's the appropriate level of statistical significance? (What is statistical significance exactly?)"
  • "What is it worth it to test? The copy? Colors? Buttons? Images? A whole landing page?"
  • "Which of my ideas should I test first? How do I prioritize?"
  • "What should I do if a test doesn't deliver the expected results?"
  • "What numbers should I analyze for evaluating the test results? Revenue or conversion rate? The sales of product A or product B? Micro-conversions or macro-conversions? What are the key metrics to look at?"
  • "Should I focus exclusively on A/B testing or can I run A/B/C/D/... tests, too?"
  • "Should I run multivariate tests?"
  • "Should I segment my users by device type (e.g. mobile vs. desktop)?"
  • "What is/are the best A/B testing tool(s)?"
  • "What should I do if my company doesn't have the resources to run experiments continuously? (E.g. if I'm the only one in the organization who runs tests?)"
  • "What should I do if my traffic is too low (and getting a large enough sample would take months)?"
  • And more...
ab tesztelés képzés

I've previously tested and fine-tuned the course materials in live workshops, too. So I know most of the questions that might be in your head and problems you'll face when starting out. I'll address them all in this A/B testing course!

This course is based on real life knowledge: you'll learn through the example of real business cases.

In the last few years, over 800 students have enrolled in my data science courses. And I have two golden rules: (1) keep it simple and (2) keep it practical!

So I always put a huge emphasis on explaining everything (even the most complex concepts) in a way that enables everyone to understand it, internalize it and then apply it in real life.

In this A/B testing course, you will see real-life examples. Most of the base concepts will be introduced through my (or through my friends' and colleagues') projects.

ab testing course ab test like a data scientist

Who is this course for?

I guarantee this: This is not another marketing BS course.

Here, we will focus on the practical parts of running experiments - and more importantly how you will be able to use them in real projects.

I recommend this course for people who don't expect A/B testing to be a fancy, one-size-fits-all marketing tool for overnight conversion boost. (Because that'd be a lie anyway.)

It's for online professionals who want to understand the science and the strategy behind A/B testing. Who see that this is a research method. And who are ready to improve their results methodically for real, long-term success.

Do you want to hear a proper "target audience," too? Fine. Here it is.

The A/B Test like a Data Scientist! course is for:

  • digital marketing professionals (SEM, SEO, PPC, Social, content marketing, etc.)
  • UX researchers and designers
  • online business professionals (for those who implement A/B tests as much as for those who lead CRO/Analytics/Web teams)
  • data analysts and data scientists who plan to work on similar projects in the near future
  • anyone who's interested - and wants to invest in furthering her career in data science or data-driven marketing

FAQ: "How deep do we go into A/B testing? Is this interesting for experienced online professionals, too?"

I got this question above from a course participant (before he registered). It might be a relevant question for many, so I’ve just copy-pasted the relevant part of the discussion here:

hi Tomi, quick question: How deep do we go into the topic? Is this interesting for experienced online professionals, too?

hey! Good question -- just in general: we start from scratch, and by the end of the course, we get far enough that everything regarding A/B testing will be completely clear. So I'd say that if someone has been running experiments for years, with great results, it's unnecessary for her to enroll. But for someone who just wants to get started with this field, or has been doing it for, say, half a year - but isn’t sure if she's doing it right, it's 100% that she will profit a lot from the course.

The A/B Test like a Data Scientist! course will take you here...

  • You will completely understand the theoretical and the practical fundamentals of A/B testing. You'll know when you need it and when you don't.
  • You'll be able to build, run and evaluate experiments professionally.
  • You'll know what you should watch out for -- and you won't ever make any of those common mistakes that many online professionals fall for.
  • You'll understand how experiments are able to support your business and how they can bring you higher conversion rates and more revenue.
  • You won't get confused by questions like "how long should the test run" or "how many visitors do we need." You'll understand the concept (and the importance) of statistical significance.
  • And of course, you'll know why it’s so crucial that A/B testing is not a stand-alone method -- but part of a comprehensive research framework.


Tomi Mester is a practicing data analyst and researcher for 7+ years.

He has worked for Prezi, iZettle and several smaller companies as an analyst/consultant.

He’s the author of the Data36 blog where he writes posts and tutorials on a weekly basis about data science, AB-testing, online research and data coding.

He's an O'Reilly author and presenter at TEDxYouth, Barcelona E-commerce Summit and Stockholm Analytics Day. More info about Tomi: check out the intro video. >>

Registration Details

Registration opens: 11 May 2020, 10pm (CET)

You will have lifetime access to all course materials and you will get all future updates, too.

Price: $297 (plus your country’s VAT if you live in the EU)

ps. If you are from Hungary, please email me before you register: tomimester@data36.com.


The registration for the next class (Class of May, 2020) opens on 11 May 2020, Monday.

If you'd like to be added to the waiting list, just enter your email address below, and you'll be the first to be notified when I open registration again.


I would never want you to be unhappy! If you are unsatisfied with the course, contact me (tomi@data36.com) in the first 30 days and I will give you a full refund.

Frequently Asked Questions

Is it a problem if I don't have too much initial knowledge on the topic?

No. We will start from scratch. It's useful, though, if you are familiar with the basics of online marketing.

What if I am unhappy with the course?

I would never want you to be unhappy! If you are unsatisfied with your purchase, contact me in the first 30 days and I will give you a full refund.

When does the course start and finish?

This A/B testing course is a completely self-paced online course - you decide when you start and when you finish. Also, you'll have lifetime access to all materials -- and you will get all future updates, too.

How much time does the A/B testing course take?

Watching the videos and going through the learning materials will take around 3 hours. If you go deeper into the extra materials and if you follow all recommended action steps, it might take ~8-10 hours in total.

How long do I have access to the course?

How does lifetime access sound? After enrolling, you have unlimited access to this course for as long as you like - across any and all devices you own.

Will I get an receipt/invoice?

Yes. Individuals get a receipt via email. Companies get an invoice. EU-based companies get VAT-invoices. If you need something even more special, just reach out to me (tomi@data36.com) and we will solve your administrative issues!