Fake Door Testing: an “evil” but efficient testing method

Have you ever heard of fake door testing? If not:

Fake door testing is an easy yet very powerful method with which you can measure interest in a product (or a new feature) without actually coding and implementing the product itself.

Fake door testing step-by-step

The process consists of 4 steps:

1. You create a landing page where you explain the purpose of the service. It can be a fully featured landing page – but you can also create a short “beta” version.
2. You drive traffic to this landing page. The best way to do so is using paid advertisements. (E.g. Facebook, Linkedin, Twitter or Reddit ads.)
3. Users start to register… except they can’t yet.
   This is the trick! Once they click the registration button, instead of a registration form, you send them to a page that says, “Sorry, but this function/product/service is not yet available. But we will let you know when it’s ready.”
4. And of course, the whole point is that you measure the whole process: how many people visited your website, how many of them wanted to register, etc.

If enough people wanted to register, then you can get an idea of whether the product is worthwhile to develop – or not.

Done. Easy, yet powerful, just as I said.

But do you get now why it’s called fake door testing?
And why I said that it can be just as evil as it is efficient?

In the end, it will be your decision whether to take advantage of this method. In this article, I’ll give you a few more thoughts that will help you decide.

The advantages of fake door testing

1. You obviously save a lot of engineering and product development time. And with that, of course, money, too.
2. You can prove (or disprove) your idea very quickly.
3. If you’re doing it right, you can even collect the email addresses of the potential registrants – so you can notify them when it comes to the real launch. (You can also hire them for usability tests or for user interviews.)

Any of the above three steps can be key when it comes to launching a new product, a new business or a new feature.

However, I have to mention that fake door testing comes with a few obvious disadvantages as well.

The disadvantages of fake door testing

1) It’s not a very nice thing to do…

Fact: There will be visitors who will say that you “scammed” them, who will be annoyed and who will not use your product ever in the future.
In most cases, this can be considered an acceptable loss. Don’t forget that during the first testing phase you only reach out to a small proportion of your target audience – so, if you do things right, then the long-term success of your business (or product or feature) will not depend on this one little initial experiment.

2) It can decrease your credibility – especially if someone gets involved who will post about your fake door test in the media. For this reason, you have to be very careful with it if you are already in business and you want to use this method to validate a new feature.

E.g. You may have seen such messages on Facebook:

Note: I’ve never gotten an actual confirmation from a Facebook employee directly, but people say that when Facebook ran these pop-ups, they were doing fake door testing, too!

So keep these two disadvantages in mind before running your first experiment!

Do it like a pro!

Here are some tips and tricks for a perfectly executed fake door test:

1. Preferably, the users should not know that there is no product. So try to “lie like a pro”… Of course, you need to find the right balance between how unbiased you want your experiment to be (which requires convincing people that the product exists) and how scammy and guilty you want to feel at the end of the day.
   
2. Collect email addresses! This is good for a number of reasons. For one, your marketing costs are not going to waste. Secondly, it will be much easier to communicate to your subscribed potential registrants when it comes to the actual product launch.
   Plus, giving a subscription form is the perfect opportunity to explain to your audience that you don’t want to scam them, you have every intention to actually build this product, but you wanted to run an unbiased experiment.
3. Define the right metrics! Just like with every other experiment, you need easy-to-measure and easy-to-understand goals! (e.g. CTR, number of registered users, % of registered users, etc…)
4. Communication is key!
   Visitors who clicked in to a fake door test can react in different ways when you reveal yourself… So you have to be very careful with your words.
   Don’t write: “Haha you blew it, there is no product at all, and we got your email address.”
   Write: “Sorry, this product is not yet available. But we are working really hard on it and your registration gives us extra motivation. We promise that you will be the first to know about future developments!”
   Consult with your copywriters!
5. An even easier version of fake-door testing is advertisement-testing.
   This means that you set up a ~$100 budget on a social media ad campaign and you measure the CTR.
   If it’s high enough, then you’re good.
   With advertisement testing, though, you will face two problems: first, you can’t collect email addresses. Second, there is a chance to get false negatives: if the CTR is low, it doesn’t necessarily mean that your product is bad; maybe your targeting wasn’t correct.
   

Update: ad testing can now be done with “lead ads”, so you don’t have to give up on valuable email addresses. (Thanks to Chris Toba for the comment!)

Wanna give it a go?

I’m gonna be honest here: I still can’t decide if I like fake door testing or not.

I used it before, and there’s no doubt, it saved us a great amount of time and energy… However, recently, I decided not to use it (at Data36 at least), simply because I feel that now that I have a bigger audience, I can get the information I need with other, more user-friendly research methods — for instance with landing page A/B testing.

But it doesn’t mean that in certain use cases, at certain businesses, it wouldn’t be a legit testing method!

Good luck!

• If you want to learn more about how to become a data scientist, take my 50-minute video course: How to Become a Data Scientist. (It’s free!)
• Also check out my 6-week online course: The Junior Data Scientist’s First Month video course.

Cheers,
Tomi Mester

Cheers,
Tomi Mester