The Conversion Rate Optimization Framework I Use at Data36…

… is exactly the same one that many well-known startup and e-commerce companies are using. However, somehow this fundamental CRO framework hasn’t been published anywhere so far! (As far as I know, at least.) It’s time to change that! In this article I’m gonna share the Conversion Rate Optimization Framework that you should use every time you optimize your website, mobile app, or any other online product.

The principle: one analysis is no analysis!

If you are a data scientist, you are into analyses, A/B testing and similar stuff. On the other hand, if you are a UX researcher, you’re focusing more on qualitative methods like Usability Testing or Five Second Testing. But I will tell you what. Neither of these is useful without the other. Analysts forget from time to time that qualitative and quantitative methods should be used together!

There’s a model on how to combine qualitative and quantitative methods in a meaningful way. But first let’s sort out what kind of research methods we usually use in online research:

This is not a full list, of course. But these are the most useful methods. Should you use all of these all the time? No. But one thing’s for sure: you should use more than one of these when you investigate something. To be more conscious about it, let’s put them into a CRO framework.

The Conversion Rate Optimization Framework

I’ve had many conversations on this topic with many data scientists from many online businesses in the last few years and somehow it looks like every expert follows one very similar model. I’ve been using it for 5+ years, so I tweaked and optimized it a bit. This is how my Conversion Rate Optimization framework looks right now:

Phase 1: Historical data

The whole process starts with a qualitative research round (1) where you collect your first inputs. It’s a really important step. If you start with data analysis right away, you could accidentally miss some important things (the unknown unknowns) to look for in the ocean of data. It’s much better to collect ideas and “hunches” (or “suspicions”) first. And the easiest way to do this is by talking to the users. User interviews, usability tests, five second tests, exit-intent surveys, etc… As a result of this round, I usually have a list of 10-30 elements, prioritized by importance.

The next step is to go ahead and validate these suspicions with actual quantitative data (2). You can use the full data analytics arsenal: funnel analysis, segmenting, heatmapping, correlation analysis, etc… It will need some professional creativity and experience to pick the right methods for the job, but keep in mind here as well: one analysis is no analysis!

For instance:

If you have a hypothesis (e.g. the “To the Cart” button is not recognizable on your UI), then you should prove it in many ways.

• One is the input from the qualitative (1) round (e.g. two out of five test users couldn’t find the CTA button on UX tests.)
• Another way is to set up a funnel, so you can visualize where people stuck and churn. If they stuck somewhere: is that the point when they should click your CTA button?
• You can check your website heatmaps to see if people are clicking anywhere else but your “To the Cart” button. (Maybe it’s not the design of the buttons that caused the issue, but that your visitors just don’t find your offer interesting enough).
• You can segment the users and understand how those who clicked on the CTA buttons differ from those who didn’t.
• You can also try to find correlations that drive clicks (e.g. higher engagement on your blog posts drive more CTA clicks?)

You get the point.

One problem – many research methods.

Phase 2: Design the next version and test it.

Once you are done with the first two rounds, you will have validated problems to solve.

Now it’s time to arrange a brainstorming session (3) with designers, UX people, product managers, etc… As a result, try to come up with around 4 (or more) alternative solutions for one given issue. (Of course, a great idea can fix more issues at the same time.)

You have the new design ideas. Good!

Then it’s time for another round of qualitative research (4). Five Second Testing and Usability Testing will help you. The good thing is that you don’t have to spend your time with building the code part of your new drafts. These tests are totally doable with wireframes, prototypes and design mockups.

When you have some good-looking new versions that performed well on the qualitative test round, your next step will be to build them for real and release them for A/B testing (5).

And eventually the winner of the A/B test can go to production (6) for 100% of users.

Done!

You have a super-validated, super-lean new version of your product/website/app in a reasonable time. All the unnecessary analyses, designs and especially coding projects were just skipped – and at the same time you made sure that your new version is actually better for your users and for conversion purposes as well!

Summary of the CRO Framework

1. Qualitative Research
   1. User Interviews
   2. Usability Tests
   3. Five Second Testing
   4. Exit-Intent Surveys
   5. Etc…
2. Quantitative Research
   1. Funnel Analysis (to understand the big picture)
   2. Subfunnels (to understand the smaller parts)
   3. Heatmapping/Clickmapping for on-site analysis
   4. Segmentation
   5. Correlation analysis
   6. Etc…
3. Brainstorming session
4. Qualitative Research #2:
   1. Usability Tests
   2. Five Second Testing
5. A/B Testing
6. Release the best version

Disclaimer for the CRO Framework

The Conversion Rate Optimization framework is a model that works in ideal situations.

But there are always exceptions, right? E.g. if you are just starting up and you don’t have historical data at all, you have to skip the quantitative part. Or if you have a service/product that has no user interface (e.g. server hosting), there’s no way to run website heatmaps or five second tests. There are situations in which you can’t A/B test things (e.g. if you are a well-known brand, A/B testing your pricing can hurt your brand).

Try to apply this model as it is, but don’t be afraid to transform it for yourself. Be critical, skip parts that are not relevant to you and bring in other methods that fit better.

Oh. And one more thing. Did you realize that it is a circle? It’s not an accident. It represents the fact that you will be never done with optimization. There will always be room for improvement. You will always have new ways to do it better. So never stop analyzing – never stop working to understand your users and provide a better and better product or service for them.

Conclusion

The CRO framework is a very easy process to follow. I highly recommend it to every researcher: to data scientists as much as to UX people!

And once more: one analysis is no analysis!

Don’t draw conclusions from only one piece of the puzzle!

The more detailed the picture you can draw, the better you can solve your users’ problems!

Good luck with that!

• If you want to learn more about how to become a data scientist, take my 50-minute video course: How to Become a Data Scientist. (It’s free!)
• If you want to learn everything that you have to know about A/B testing (business elements, science elements, best practices, common mistakes, etc.) and become a real pro in building winning experiments, take my new online A/B testing course called A/B test like a Data Scientist!

Cheers,
Tomi Mester