The Mobite Testing Pyramid Introduction Mobile testing involves more complexity when you compare itto web UL testing. A few examples of mobile testing challenges: OSissue Network issue Authentication issue Device incompatibility = f — As you can see, mobile testing sot only has to co: flavors (especially on Android), ete Layout issue ‘but nfo (heir artist? ‘the Ul aspects, but also the compatibility of hardware, network connectivity, operation system When most people start with testing for mobile, they tend to start with getting 2 few seal devices and perform exploratory testing manually. Overtime, the nun of devices will grow. because actual customer feedback comes in describing that specific devices. Does this sound familiar? 29 does not work properly in certain situations on In onder to reach fast feedback and lower amounts ofthe manual testing, a better usage of available tools is necessary Mobile Test Pyramid Based on my personal mobile testing experience, I came up with the mobile pyramid stratez Cohn (Succeeding with Agile’, 2009). Bur dis Ofcourse, it was inspired by the test pyramid of Mike ¢ quite cover the detail that is needed for a good mobile testing xa ‘The mot st pyramid has 3 levels: + Real deviee + Mobile simulators & emulators Desktop browsers (using mobile simulation) Real devices: real-life conditions ative API integration such as A @ Real devices: visual and usability testing Simulators/Emulators: functional end-user flows and st feedback on end vs with Browsers: system testing and responsive design In case of hybrid and web ar Responsive design and system Identical to the generic test pyramid, the broader the Laver in the pyramid the more tests you should have which cover a wider ran Here are the fbeus areas for each layer, along with their pros and cons as I see them, Desktop browsers: mobile resting on desktop browsers Focus area Description Functional system testing Isolated browser teats performing full fonctional validations Responsive des Resizing browsers and toggling user agents Cross-browser Use equivalent desktop browsers Overall visual layout No extensive visual checks because the rendering is different than deviess Pros + Fast execution: Mamer of milliseconds to launch a browser, also headless execution is possible Scalable: Easily set up 10+ browser instances per machi * Cross platform: Ability to use browsers on different operating systems ‘+ Mobile ssmulation uses the desktop browser engine. Mobile simulation in deskiop browsers i sill using the desktop browser. + No native integration: No native key'oards, incoming calls, ete + Just nota device. Incredibly fast, but still nt areal device Mobile simlatoes/enmulators: closer to the actual mobile experience NOTE: Only applicable for iOS and Android. Focus area Description Functional end-user flows Click paths throughout the application Native APT integration GPS injection, file arachments, incoming ealls ete Visuals (vanilla only) Use equivalent desktop browsers Overall visual layout _Emulators are limited to vanilla versions ‘Touch interactions Touch interactions such as swipe and tap comes closer to the user experience of a device than brow Pros + Eaay to setup: Sinmolators/enmulators are sy to setup, just download, install, and ni ‘Scalable: Virtualization means scalable and also running in parallel on one machine. Native API integration: Ability wo test native APIs such as incoming calle and GPS injection, Simulators of Intel-based emulators are fast: Simulators are fast, because they only have to simulate the software part. Emulators based fon the Intel architecture are fast. + Debugging possibilities: Fasy to debug simulator/emulators, already hooked up to the machine to access logs. Cons + Vanilla versions only: Manufacturer's eins are available, bur the device behavior i still based on what comes stock + No seal resource usage: CPUimemory usage of machine in case of simulators. Emulators wy to simulate the hardware + No real interoperability: Connectivity with NEC, Bluetooth, network connections. Slow ARM-based emulators: Emulators based on the ARM architecture are slow, which is the main architecture for Android devices. + Inaccurate color display im Light dark: Contrast brightness inaccurate in light dark environment Real devices: the real thing... Focus area Description Unabilisy ‘Validating usability such as acrual click areas, touch actions and voice over Pecformance CPU memory usage. batery, network strengths Native API integration Interruption (incoming calls, push notifications), esoures fighting (camera, GPS), NFC, Bluetooth Visuals Focus on devices which are aot available a simulatorsemulatars Manufacturer's sauce Real OS from manufacturers, eg. Samsung's TouchWiz and built-in browsers Pros + Native APIs in seal conditions: Ability to test native APIs not only with injections for automation, but also actual NEC touch for example Can be faster than emulators: Some the ARM-based emulators. i 1 age just faster than enmulatocs civ to the sinmalati a of hardware, especially compared to * Just the seal ching. Actual network conditions, battery/CPU memory usage, manufacturer's secret sauce on top of the OS * Costs: Real devices come with a price, usally you pay per deviee/eradle + New device means procurement: A new device is usually not available on-the-fly, even with cloud solutions. E g.when the new iPhone comes out, is not available immediately te procure. In the meantime, the 10S simulater would alzeady be available. + Development 0S build requited for automation. :O8 apps need to be signed with Development Distribution Ceruificate and Provisioning Profile for automation Conctasion, ‘There are loads of trade-offs when it comes to mobile testing. But by leaenin 1to-use all layers ofthe mobile testing pyramid to your advantage fast feedback that is requised in modem CUCD (leveraging desktop browsers, mobile simulators emulators, and real devices togethet) you can environments, I's just a matter of focusing on the right things in each layer for your context