Customer Visual

How to Break your App Best Practices in Mobile App Testing

Daniel Knott | @dnlkntt

14/13/16
4/13/16

Who am I ?

Daniel Knott
Software Test Manager/ Mobile Tester @AOE
Mobile Testing since the end of 2010
Before AOE, Team Lead QA @XING Mobile Team
Blogger and Speaker
http://www.adventuresinqa.com
http://www.linkedin.com/pub/daniel-knott/1a/925/993
https://www.xing.com/profile/Daniel_Knott
@dnlkntt

24/13/16

34/13/16
4/13/16

Company Profile

Intro

Current
Status

Mobile
Challenges
The Customer

Mobile Pyramid
Tools
44/13/16

Mobile Specific
Testing

Current Status

54/13/16
4/13/16
Author: Barry Caruth, https://www.flickr.com/photos/caruth/8401943928/

Current Status | Some Numbers

2+ millions apps available

100+ billion downloads
80% 90% of apps are deleted after the first
usage
56% of apps that requires a registration gets
deleted user checks device 150 times per
Average
day
45% are not well tested
64/13/16

KIFSU To Remember the important Things

K
Keep

74/13/16

I
It

F
Fast

Useabl
Simple
e

84/13/16
4/13/16

Mobile Challenges

Mobile Challenges | All known!?

94/13/16

http://opensignal.com/reports/2014/android-fragmentation/

Mobile Challenges | All known!?

104/13/16

http://opensignal.com/reports/2014/android-fragmentation/

Mobile Challenges | How to handle fragmentation

Know the target group

Group your devices, A B C
Test only on those devices
Test on real device
Dont use simulators or emulators

114/13/16

Mobile Challenges | How to handle fragmentation

Group 1, Prio A
High End
Devices
Dual/ Quad
Core CPU
RAM >1024MB
Retina, Full HD
Display >=
4,5
Based on your target group
124/13/16

Group 2, Prio B
Group 3, Prio C
Mid-range
devices
Dual Core CPU
RAM <1024MB
Screen size <=
4
Software not
older than 1
year

Small devices
Single Core
RAM < 512MB
Low screen size
and resolution
Older OS, older
browser

Mobile Challenges | More

Author: Davide Vizzini Limbte, http://www.flickr.com/photos/limbte/4007115435/

Author: Juan de Dios Santander Vela, http://www.flickr.com/photos/juandesant/209098053

Author: Rain Rabbit, http://www.flickr.com/photos/37996583811@N01/8033259053/

134/13/16

144/13/16
4/13/16

The Customer

Author: Michael Newman, https://www.flickr.com/photos/mzn37/122601628

The Customer | Better know them

Know the target group

Will downsize testing effort
Talk to your customer
Do usability testing with them
Author: Grzegorz Lobinski, http://www.flickr.com/photos/gregloby/3515990945/

Invite them to a beta testing program (Sonos,

FB, Twitter)

154/13/16

Mobile Specific Testing

164/13/16
4/13/16

Author: Theus Falcao, http://www.flickr.com/photos/theusfalcao/956376714

Mobile Specific Testing

Known functional testing is still required

Boundary values
Equivalence classes
Testing against acceptance criteria
Test the app as black box
Try to break it
Be creative and think in edge cases
174/13/16

Mobile Specific Testing

Author: Juan de Dios Santander Vela, http://www.flickr.com/photos/juandesant/209098053

Author: Ian Britton, http://www.flickr.com/photos/freefoto/728651045/

Author: @Doug88888, http://www.flickr.com/photos/doug88888/7149187135/

Author: Dan Levy, http://www.flickr.com/photos/danlevy/5607259711/

184/13/16

Author: Moyan Brenn, https://www.flickr.com/photos/aigle_dore/5626341059/

Mobile Specific Testing

Author: amateur_photo_bore, http://www.flickr.com/photos/andresv/233667770/

Author: Rain Rabbit, http://www.flickr.com/photos/37996583811@N01/8033259053/

Author, Daniel Sancho, http://www.flickr.com/photos/teclasorg/2852716491/

194/13/16

Mobile Specific Testing

Author: Surat Lozowick, http://www.flickr.com/photos/suratlozowick/4311105154/

Author: Surat Lozowick, http://www.flickr.com/photos/suratlozowick/4464587302/

Author: Theus Falcao, http://www.flickr.com/photos/theusfalcao/9563767145/

204/13/16

Mobile Specific Testing

Android
JouleUnit, to measure battery usage
Monkey, to stress test the app

iOS
Energy Usage in Instruments
UI AutoMonkey, to stress test the app
214/13/16

Mobile Specific Testing

Author: Molly Holzschlag, http://www.flickr.com/photos/mollyeh11/272434559/

224/13/16

Author: Quinn Dombrowski, http://www.flickr.com/photos/quinnanya/4718635484/

Mobile Specific Testing

Author: minxlj, http://www.flickr.com/photos/minxlj/3270996158/

Author: Ryan B. BulaPhotography, http://www.flickr.com/photos/nayralub/3763581778/

Author: Sterling Coffey, http://www.flickr.com/photos/n0ssc/7046915929/

234/13/16

Tools

Author: OZinOH, http://www.flickr.com/photos/75905404@N00/712614712

244/13/16
4/13/16

Test Automation Pyramid

Manual
Testing

UI Tests
Integration Tests
Unit Tests
254/13/16

Flipped Test Pyramid

Manual Tests
UI Tests
E2E

Unit Tests

264/13/16

Mobile Test Pyramid

Unit Tests
Beta Tests
End2End Tests
(UI Tests)

Manual Tests
274/13/16

Tools

Do test automation!
Dont buy huge test automation tools
Start simple and discover different tools
No tool is perfect
See which tool fits in your development
environment

284/13/16

Tools | Android

Robotium (ExtSolo)

Robolectric

Spoon

Espresso

Selendroid

Monkey

Calabash for Android

MonkeyTalk

Appium

Great mobile tools provided by the company square: http://square.github.io/

294/13/16

Tools | iOS

UIAutomation

ios-driver

Keep It Functional

Kiwi

MonkeyTalk

XCTest

Calabash for iOS

Appium

304/13/16

Security

314/13/16
4/13/16

Author: Alexandre Dulaunoy, http://www.flickr.com/photos/adulau/771254542

Security
Author: Anthony Catalano, http://www.flickr.com/photos/badwsky/34164244/

Author: Dennis Wong, http://www.flickr.com/photos/denniswong/2406135310/

uthor: orkomedix, http://www.flickr.com/photos/orkomedix/5211437958/

324/13/16

Security

Security testing is not easy

Get help from experts
Think twice about app permissions
Check the OWASP mobile security page
https://www.owasp.org/index.php/OWASP_Mo
bile_Security_Project

334/13/16

344/13/16
4/13/16

Summary

Mobile Specific Testing | Summary

Be creative and on the move while testing mobile apps

Combine your in house testing with crowd
testing
Be up to date! Know your target group
Keep the mobile features in mind
Keep the mobile test pyramid in mind
Do test automation and know the right tools

354/13/16

Av
ai
la
bl
e

so
on

Thanks for your attention!

More information about mobile testing

can be found in my book
Hands-On Mobile App Testing
which will be available very soon.

www.handsonmobileapptesting.com
364/13/16

in Germany

AOE GmbH
LuisenForum, Kirchgasse 6
65185 Wiesbaden
Germany
Phone: +49 6122 70 70 7 - 0
Fax:
199

+49 6122 70 70 7 -

E-Mail: sales@aoe.com
4/13/16
Web:374/13/16
www.aoe.com

Links & References

World Quality Report [http

://www.sogeti.com/en-us/looking-for-solutions/Services/Software-Control-Testing/TheWorld-Quality-Report-2013--2014
/]
Robotium [https://code.google.com/p/robotium/]
Spoon [https://github.com/square/spoon]
Robolectric [http://robolectric.org/]
Calabash for Android [https://github.com/calabash/calabash-android]
Selendroid [http://selendroid.io/]
ExtSolo [
http://docs.testdroid.com/_static/extSolodocs/com/bitbar/recorder/extensions/ExtSolo.
html
]
Espresso [https://code.google.com/p/android-test-kit/wiki/Espresso]
Monkey [http://developer.android.com/tools/help/monkey.html]
UI AutoMonkey [https://github.com/jonathanpenn/ui-auto-monkey]
384/13/16

Links & References

MonkeyTalk [https://www.cloudmonkeymobile.com/monkeytalk]
UIAutomation[https
://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/Instr
umentsUserGuide/UsingtheAutomationInstrument/UsingtheAutomationInstrument.html
]
Keep It Functional [https://github.com/kif-framework/KIF]
XCTest [
https://developer.apple.com/library/ios/documentation/ToolsLanguages/Conceptual/X
code_Overview/UnitTestYourApp/UnitTestYourApp.html
]
Calabash for iOS [https://github.com/calabash/calabash-ios]
ios-driver [http://ios-driver.github.io/ios-driver/?page=home]
Kiwi [https://github.com/allending/Kiwi]
Appium [http://appium.io/]
JouleJunit [https://code.google.com/p/jouleunit/]
Energy Usage Instruments [
https://developer.apple.com/library/ios/documentation/AnalysisTools/Reference/Ins
394/13/16
truments_User_Reference/EnergyUsageInstrument/EnergyUsageInstrument.html
]

404/13/16