On Thursday Ari Grant and Kang Zhang of Facebook posted an article titled "Airlock - Facebook's mobile A/B testing framework" to Facebook's Engineering Blog. I wanted to take a few minutes to write about it this weekend, because it is a very good read for anyone who is interested in mobile development, whether they prefer the HTML or native routes.
There are a couple of things that I find very interesting and very telling about the article. First, the fact that Facebook is willing to take the time to talk about the process and the tool, but are not open sourcing the framework shows to me just how strategically important they feel that it is and will be to their business.
This to me is a marker that signals the true start of the mobile A/B testing landscape. To date, the complexity of mobile and the disparate, closed platforms that dominate it have caused most people to ignore the possibility and benefits of A/B testing. Even, for a company like Facebook it didn't become a pressing issue until over 70% of their users were on mobile.
To us at Taplytics this is crazy. If you are developing on the web everything is calculated and optimized and viewed in terms of hypotheses, significance levels and confidence intervals. On mobile, however, for the past 6 years we have been living in the era of the "artform" of mobile apps, where things are viewed in terms of gut feel and shooting from the hip. In the end, this very clearly shows just how immature the mobile market really is. But there is a silver lining available to anyone who is looking closely.
We live in a world where time after time people have proven that the well prepared, scientific approach is always better. And yet massive companies like Facebook, are just finally warming up to the idea of A/B testing and optimization when it comes to their mobile apps. So this leaves room for anyone, whether big or small, who takes advantage of the tools that are available to them, to be world beaters.
For us at Taplytics, while it would have been nice for Facebook to share more specific stats about their results, it is important that people are sharing stories like this either way. We believe that the benefits of A/B testing are clear and the numbers will show themselves soon enough. It is why A/B testing and optimization has won out so clearly on the web.
We think that through the use of the right tools, developers can overcome a lot of the challenges that stem from such a noisy market as the iOS and Android app stores to beat competitors that may be significantly larger.