The Online Research Framework I Use at Data36…

… is exactly the same that many well-known startup and e-commerce companies are using. However somehow this fundamental research framework hasn’t been published anywhere so far! (As far as I know at least.) It’s time to change it! In this article I’m gonna share the Online Research Framework that you should use every time, when you optimize your website, mobile app or any other online product.

The principal: One research is no research

If you are a data analyst or a data scientist, you are into analyses, A/B testing and similar stuff. On the other hand if you are a UX researcher, you’re focusing more on qualitative methods like Usability Testing or Five Second Testing . But I will tell you what. None of these are useful without the other. Analysts forget from time to time that qualitative and quantitative methods should be done together!

qualitative_quantitative_research_jinjang

There’s a model on how to combine qualitative and quantitative methods in a meaningful way. But first let’s list out, what kind of research methods we use in online usually:

Qualitative Quantitative
Usability Testing Segmentation
User Interviews Funnel Analysis
Five Second Testing Heatmapping/Clickmapping
First Click Testing Cohort Analysis
Card Sorting Correlation Analysis
Exit-Intent Surveys A/B testing

This is not a full list, of course. But these are the most common things. Should you use all of these all the time? No. But one thing’s for sure: you should use more than one of these, when you investigate something. To be more conscious about it, let’s put them into a framework.

The Online Research Framework

I had conversations on this topic with many data analysts from many startup/tech companies in the last few years (I’ll publish some of these conversations to the blog in an interview format soon) and somehow it looks that every expert follows one very similar model.
I’m using it for 5 years now, so I tweaked and optimized it a bit. This is how my online research framework looks right now:

online research framework

the Online Research Framework – 6 steps

Phase 1: Historical data

The whole process starts with a qualitative research round (1.) where you can collect your first inputs. It’s a really important step. If you’d start with data analysis right away, you could accidentally miss some important things to look after in the ocean of data. It’s much better to collect ideas and hunches first. And the easiest way to do this is talking to the users. User interviews, usability tests, five second tests, exit-intent surveys, etc… As a result of this round, I usually have a list with 10-30 elements – prioritized by importance.

Next step is to go ahead and validate these hunches with actual quantitative data (2.). You can use the full analytics arsenal: funnel analysis, segmenting, heatmapping, correlation analysis, etc… It will need some professional “creativity” and experience to pick the best methods, but keep in mind here as well: one research is no research!

For instance:

If you have a statement (eg. the buttons are not recognizable on your UI), then you should prove it in many ways. One is the input from the previous qualitative round (eg. 2 out of 5 users couldn’t find the CTA buttons on UX tests.) Another way is to set up a funnel – so you can understand, where people stuck. If they stuck somewhere: is that the point when they should click a button? Then you can check the heatmaps – to see if people are clicking anywhere else. (Maybe it’s not the design of the buttons that caused the issue, but simply your visitors just don’t find your offer interesting enough). Then you can segment the users and understand how those who clicked on the buttons differ from those who didn’t. You can also try to find some correlations, that drives clicks (eg. more engagement on your blog posts drive your CTA clicks?)

But you get the point, I guess. One problem – many data analyses.

Phase 2: Design the next version and test it.

Once you are done with the first two rounds, you will have validated problems to solve. Now it’s time to arrange a brainstorming session (3.) with designers, UX people, product managers, etc… As a result, try to have 3-4 (or more) alternative solutions for the same issue. I might not need to mention, that ideally one solution can fix more problems.

You have the new design ideas. Good! It’s time then for another round of qualitative research (4.). Five Second Testing and Usability Testing will help you. The good thing is, that you don’t have to spend your time with building the code part of your new drafts. These researches are totally doable with wireframes and design mockups.

When you have some good looking new versions, that performed great on the recently mentioned qualitative tests, your next step will be to build them for real and release them for A/B testing (5.) or multivariate testing. (How many test-versions you can have? It depends on the volume of your traffic. More info here.).

And eventually the winner of the A/B test can go to production (6.) for 100%.
Done! You have a super-validated, super-lean new version of your product/website/app/etc in a reasonable time. All the unnecessary analyses, designs and especially coding parts were just skipped – and at the same time you made sure that your new version is actually better for your users and for conversion purposes as well!

Summarizing the Online Research Framework again

  1. Qualitative Research
    1. User Interviews
    2. Usability Tests
    3. Five Second Testing
    4. Exit-Intent Surveys
    5. Etc…
  2. Quantitative Research
    1. Funnel Analysis (to understand the big picture)
    2. Subfunnels (to understand the smaller parts)
    3. Heatmapping/Clickmapping for on-site analysis
    4. Segmentation
    5. Correlation analysis
    6. Etc…
  3. Brainstorming session
  4. Qualitative Research #2:
    1. Usability Tests
    2. Five Second Testing
  5. A/B Testing
  6. Release the best version

Disclaimer for the Online Research Framework

The online research framework is a model, that works in ideal situations. But there are always exceptions, right? Eg. if you are just starting up and you don’t have historical data at all, you have to skip the quantitative part. Or if you have a service/product that has no User Interface (eg. server hosting), there’s no way to run heatmaps or five second tests. Then there are situations, where you can’t A/B test things (eg. if you are a well-known brand, A/B testing your pricing can hurt your brand).

Try to apply this model as it is, but don’t be afraid to transform it for yourself. Be critical, skip parts that are not relevant for you and bring in other methods that fits better.

online research framework

the Online Research Framework – 6 steps

Oh. And one more thing. Did you realize, that it is a circle? It’s not an accident. It represents the fact, that you will be never done with optimization. There will be always area for improvement. You will have always ways to do it better. So never stop research and analyze things – never stop to understand your users and provide a better and better product/service for them.

Conclusion

The online research framework is a very easy process to follow – I highly recommend it to every researcher. To data analysts/data scientists as much as to UX people!
And once more: one research is no research! The more detailed picture you can draw the better you can solve the problems of your users! Good luck with that!

I wrote a more detailed article earlier about how can you start with a good research plan and how you can fit this model into it. Here: Create a good data research plan (step-by-step).

If you want to be notified first about new content on Data36 (like articles, videos, handbooks, etc.), sign up for my Newsletter!

Cheers,
Tomi Mester

← Previous post

Next post →

2 Comments

  1. Khalifa

    Really insightful for someone like me that just start a career in data analyst. Thank you so much

Leave a Reply