In a lot of meetings I find I still get asked some variation of the question, "Why do I need to test?". This question can obviously come in many forms, but the crux of it is that in mobile, there are still a lot of people that do not think that mobile A/B testing is necessary or helpful. The people who are hesitant to test, generally fall into one of two groups.
The first group falls into the category that I like to call "spray and pray". These mobile developers feel that mobile A/B testing isn't necessary, because they have the App Store release cycle timed to a science and always have a new release ready to push for review when the last is approved. This group would clearly be the fastest movers on the market today, and they all wear it as a badge of honour.
The issue that arises with this strategy, though, is that you are building in a vacuum. You are always reacting to situations and data that are behind what you are building. This concept may not be entirely obvious, but if you are building to submit while the last version is still in the review process, you are building based on data that is happening on an app version that is two version points old. Because of this void of good data, it really would just be luck if you hit on the best design, structure, features or positioning for your app.
Now the second group falls into the category that I like to call "great expectations". This group is discernible in its grand visions, big hypotheses and long development cycles. Whereas the "spray and pray" group would be pushing releases at least bi-weekly, this group will be pushing updates to the App Store once a quarter.
The issue here is that while you have timely data on the features that are out in the wild, you are imposing unnecessary risk on your development process. The longer it takes to push new features, the harder it is to determine whether or not those are actually features that are desired by users.
In the end there is a reason why now clichéd terms like agile and lean have become so popular in web development. It's because these strategies effectively de-risk the development process. They rely on good, timely data, educated and informed decisions and nimble but not ADD development processes.
I'll be the first to say that the platforms themselves have been a major cause in the avoidance of A/B testing and optimization of mobile apps. The closed nature of the platforms and the organizational structure required to to build an app has made it necessary to take these suboptimal development strategies. It is clear that people won't change if it is difficult to do so. And even if it is generally accepted that testing and optimizing is better, as on the web, only those systems that make it extremely simple are the ones that actually get used.
I think that over time, tools like ours, will help people from these two groups to embrace a strategy of continuous testing and optimization. Not, because there is any secret magic to it, but because the data shows that understanding how your users interact with your app and giving them the design and features that elicit the reactions that you want, makes for far more successful development.
We encourage people to just test this strategy, because there really is nothing to lose. At the very least you just started to think more critically about your app's development and how it serves your users' needs. If you try this strategy using our platform, all you need to do is install one line of code and you're off to the races.
At the very least though, if you're already using analytics software like Mixpanel or Flurry, you should try to frame your development in a series of small testable hypotheses with releases as soon as you have legitimate data for your previous test. In this way you can start to understand the benefits of A/B testing using just a different mindset and tools that you probably already have installed.
And if you ever want help understanding how you can test this method better, or have questions relating to A/B testing and optimization you can always feel free to send me an email. I'm always happy to talk to curious app developers and just want to help.