“Massive variability makes mobile testing complex,” said Clint Sprauve, Sauce Labs’ mobile evangelist who spoke at Accelerate  2018 in Vienna earlier this month. In a session titled “Mobile Testing with Emulators, Simulators & Real Devices,” Sprauve said that organizations can address the complexity of device fragmentation, various app types, and geographically dispersed users with a mature mobile testing strategy. He demonstrated how Sauce Labs’ solution offers testing on real devices, as well as emulators and simulators. Addressing the complexity of mobile testing was a key theme at this year’s conference. Earlier in the conference, Eran Kinsbruner, director and evangelist at Perfecto, tackled a related topic when he presented “A Strategy for Continuous Testing for Mobile Apps.” According to both speakers, the key to successful mobile testing is developing a mature continuous testing practice. Success in any DevOps environment requires continuous testing. But mobile testing introduces unique challenges that make developing a mature continuous testing practice even more challenging. “Things can block the continuous testing engine which executes the feedback loop back to your developers,” said Kinsbruner, who described the four biggest challenges of testing mobile apps as:
  1. Test code: If you are not following guidelines, or maintaining your test code the way you do the production code, this will interfere with your engine. Things like notifications, pop-ups and keyboards can block and stop test automation and CI jobs.
  2. Test lab: If you do not have a test lab that executes for you, then you are going to have issues such as device disconnection, battery dying, or not getting on-time updates of OS. The ability to test effectively needs to be continuously maintained and updated for you.
  3. Market update readiness: Every month, new platforms are released. Every few weeks, OS updates are released. “Your engine has to be ready to validate these changes. If you’re not ready for that, you’re not serving your end users,” Kinsbruner explained
  4. Test data analysis: A lot of test artifacts are generated each cycle. You need a way to aggregate, filter, slice and dice all the test data in a way that will create “fast feedback and only relevant feedback to your developers.”
“These four buckets in most cases interfere with or make a difference between a poor performing continuous testing engine and a more advanced one,” Kinsbruner said. So how can organizations succeed with mobile testing amid all these challenges? Kinsbruner suggested measuring your organization’s DevOps maturity and striving toward elite standards. He suggested DORA’s 2018 State of DevOps Report as a good benchmark for organizations performing mobile testing. According to DORA, at an elite level, deployment frequency should be on-demand or multiple times per day. Additionally, both the lead time for changes and time to restore service must be less than one hour, while the change failure rate must be less than 15 percent.  

State of DevOps Report, 2018 – DORA

Both Kinsbruner and Sprauve stressed the importance of good test case design. When you’re testing mobile apps, test plans are inherently more complex because they require multi-dimensional test cases to account for the unique variables mobile introduces. This is especially important for testing across platforms and operating systems, as well as device orientation , location, background apps, network carrier latency and download speeds. Both Sauce Labs’ and Perfecto’s mobile testing solutions integrate with Tricentis Tosca’s’ Mobile Testing.