Launchpad Toys ran a code-based experiment on the first time user experience in their Toontastic Jr. app using Taplytics. In just under three weeks they saw a 27% improvement in user interaction with a key part of their educational game and were able to push that winning variation live to all users instantly, saving them what would typically be a month of lost opportunity and lost revenue.
Launchpad Toys is a Y Combinator backed startup that creates digital toys and tools that empower kids to create, learn and share their ideas through play. They have developed and released the popular children's apps Toontastic (Apple iTunes Hall of Fame and New York Times iPad Apps of the Year), Toontastic Jr. Pirates, Toontastic Jr. Shrek, and Toontastic Jr. Kung Fu Panda. The Toontastic series of apps enable children to make their own cartoons and animated stories by playing with digital paper puppets while recording their voice. Since the launch of the apps, kids have made more than 10 million cartoons and spent nearly 400 years of playtime. They came to Taplytics to A/B test and hone the first time user experience in their app Toontastic Jr. Pirates and improve their in-app-purchase rates to increase revenue.
While Launchpad Toys saw a good uptake of their educational game with significant usage, they had a gut hunch that the app’s initial experience could be optimized further. Up until employing Taplytics, if Launchpad Toys wanted to try new things they had to submit an update to the App Store, with another update to distribute the winner. This could mean a month or more of missed opportunities while a losing variation was still taking up 50% of the traffic and reduced revenue. The hope was that A/B testing with Taplytics could help them to validate their hunches and enable them to choose winning variations instantly.
The first thing Launchpad Toys wanted to test was the way users select scenes and stories that they would use to create their animated cartoon. Toontastic Jr. originally had a spinner that automatically spun to a random selection of scenes on a storyline. The team’s hunch was that the average user assumed they couldn’t change and select their scenes, and that this automatic spinner was making a set choice automatically for them. In turn, if users did not see that they could select other scenes, families would not grasp the level of content to create from, offered in the app and would be less likely to unlock the whole app through in-app purchase.
Using Taplytics’ code-based experiments, Launchpad Toys easily set up a test to observe if there was a difference between the original configuration of spinning the scene picker automatically or leaving it to the user to try spinning themselves. To determine a winner for this test, Launchpad Toys set a goal around users who discovered the ability, starting to create their story.
In just under three weeks Launchpad Toys had a clear result. In the variation where scene selection did not spin automatically on launch, users made their own selection 27% more often. This and other improvements led to a 2X improvement in in-app revenue. The most important part for Launchpad Toys though wasn’t just the positive result, it was that they could then push the winning variation instantly to all their users. When they had done ad-hoc testing in the past the biggest issue they ran into was that it could take upwards of a month to either get the winning experiment out to all users or a negative variation pulled from their app and and get it through app review. In either case this could mean significant amounts of lost revenue and lost opportunity with users.
Co-founder of Launchpad Toys Thushan Amarasiriwardena had this to say,
“The most important part of Taplytics for us is the speed with which we can react to data. While this test was great and had a positive result, what really showed us the value of Taplytics was another test we ran. In that test, the variation we tried was clearly not working out and adversely affected sales. Taplytics’ analytics told us within a day of deployment that it had only a 5% chance of beating the baseline original version. We were able to cut the losing variation instantly and we had only one day of lost revenue, whereas with our previous testing methods, we could have been hemorrhaging money and a positive user experience for a month.”
For Launchpad Toys the benefit of using Taplytics came down to more than just a double digit increase in the usage of a key feature. The value that Taplytics really added was the ability to finally try things in their live apps without the fear of drastic consequences.
If you have had a similar experience or want to share your testing stories feel free to leave a comment below or get in touch with me at firstname.lastname@example.org.