Quality Assurance Testing

Tony Mack
PlanetLab Developers Meeting
Overview

• How do we validate/test our software?

o Developers test their own work, submit when they feel its
ready.
o Does it build?
o Developers have different testing methods/processes
(unit tests, manual tests, scripts, etc...)
o end to end system tests

•  Did we test everything?

o Backwards compatible?
o Did new development break old functionality?
Objectives

• Extensible testing system (test harness)

o Define a testing process (what are we testing? How do
we test it?)
o Configurable (Adding/Modifying/Updating test process
should be easy)
o Flexible (Test harness can run in different environments)
o Automated
o Developers can integrate their tests into the system

• Accounting
o Test results are logged. Can be reviewed at any time 
o Track long term debugging efforts/progress
Test Types
• New development
• Bug fixes
• Backwards compatibility
o Are we losing required functionality by
upgrading/replacing/deprecating
o Does current plc software support new node software?
o Does new plc software support old node software?
• Unit tests
o Fine grain module testing
• End to end tests
o Do previous operations still work (can we create a slice at
plc and log into it on a node)
• Long term tests
o Bugs that arise over time
• Stress tests
o What are the limits? How much of a beating can it take?
Current Test Harness
• Create a vserver and setup the testing environment (Optional)
• Download and install the nightly MyPLC (if build failed and no
nightly exists, abandon or test in a previously working
environment)
• Configure the PLC and add relevant test data to db
o Site, Nodes/People/Slices at site, Slices on nodes, Keys for
person accounts, NodeGroups, etc...
• Run the api uint test(check if all api methods operate as
expected and return what expected)
• Attempt to boot selected nodes (using qemu)
o Attempt to boot a 4.2 and 4.1
o If nodes come up in boot bootstate, check if NodeManager
is running and creating slices
• Execute node test scripts (cpu scheduler, planetflow, ...)
To Do

• Long term tests

• Upgrading a PLC thats already installed
• Specific NodeManger tests (slice attributes and resource
management)
• Multiple PLC's and federation
• Hardware diversity found in real nodes