When we work with early stage start-ups, one of the most frequent questions we get asked is if it's too early to start thinking about A/B testing. Before starting Taplytics, we were app developers so we know all about the confusion you must be feeling. You have probably heard countless times that A/B testing is key to driving success but in the back of your mind you know you don't yet have the users to be able to reach significance. So I wanted to share a few key reasons you should absolutely start A/B testing now and how to interpret your data when you don't reach significance.
Validate Your Assumptions
Trying to gain traction is one of the toughest parts when your just starting out and I am sure you have had many discussions trying to figure out why you may not be hitting your numbers. A/B testing for a young app, or any app for that matter, allows you to be able to test completely different positioning and messaging of the app. When you’re designing and building your app, you make a lot of guesses along the way.
There are other methods to try to determine if a new feature will be a hit such as user interviews and mock prototypes. The struggle with these are a user may think they will use a product one way but once it's live in their hands, it might not be the case. The guesses you’re making are a good start, but having data to understand how your users are interacting with your app can help you understand what's working and what isn't.
What if you could try out a few completely different versions of your app at the same time? What if you could run completely different messaging about what the app can do, who’s it for and why your users should care? What if you could try out a completely different set of features and see what version results in more users not only trying it but also sticking around.
All of this is possible with mobile A/B testing
You can use visual editing to modify your copy, your images, and even your user interface. We’ve had some users completely re-skin their app. If you need to create your experiment with new code, maybe to try out different interface structures, recommendation algorithms or turn on/off features of the app, code based experiments are a great way to achieve this. Of course, you can also use your A/B testing results to measure the impact of the changes on your key metrics.
Now that you have these great reasons to A/B test you are probably still stuck on the fact that my experiments will never reach significance. So how can I make a decision from my data?
Look for Trends
Sometimes it's ok to ignore the statistical significance of your experiments. If you have a new app with only a few hundred downloads, you’re never actually going to hit full statical significance in any reasonable timeframe. Even if you haven't reached significance, when you are just starting out making decisions based on trends in your data is much better then changing features based on your gut feel. When you first launch an A/B experiment, your data will be very chaotic but should eventually settle. Once it does settle, you can start to look for trends to help you make a decision. As shown below this is a prime example of a time you can use trends in your data to make decisions.
What you want to be looking for is a flattening of the conversion lines into trends. Once you've seen it stabilize as demonstrated above, you can start inferring a trend out of the data. As shown above, there's a clear difference between the two variations and you should feel confident to which variation you should be moving forward with.
Where you want to be careful, however, is when the results are too similar or still changing, making it less clear which variation is going to win as shown below.
Keep track of the key goals you’ve set for you experiment and pay attention to wide divergences in conversion rates. These can be considered trends of how your app is performing and can give you enough of an indication if the interface is working as expected for your users.
Test Early, Test Often
It’s never too early for mobile A/B testing, even at early stages of your app’s growth. It can give you a competitive advantage over other apps in your category.
Knowing what your users are doing and how they react to your experiments can be valuable when trying to figure out which features to keep and what to change. Having the flexibility to make changes on the fly, without any app review delays can be the difference between a successful launch or losing your marketing momentum due to a small bug. Even if you are not reaching significance A/B testing has loads of benefits for your young app.
Using these tools is easy to install and for young startups we let you use them for free. So there’s no excuse not to be A/B testing from day one!