Stress and volume

The stress and volume work was running a week ahead of time until we hit un-stubbed
testing.
The number of issues encountered in stubbed and un-stubbed testing was approximately
the same. This points to a problem in the turn around time for iterating through issues in
unstubbed testing.

Here are some things that slowed down the iteration

Shared Infrastructure

Problem:
The stress and volume database runs on a server shared with 7 other EC environments and
a similar number of spear environments. Tests could therefore only run overnight as at least
one other environment was usually in use, more frequently multiple other environments
were in use.
Impact:
● Only 1 test could be executed per day
● Our ability to re-launch tests that had failed early due to easily correctable oversights
was severely reduced.
● Network outages, IT enforced system re-starts and other out of hours maintenance /
down time events caused several failures either by directly failing the test, or making
results inaccessible.
Proposed Resolution:
Commission a stand alone database server for the stress and volume testing which is not
shared by any other environments. Run a second authentication server (signing auth) on
the new database server.

Benefit:
Stress and volume tests can be executed during normal business hours, allowing for:
● More test executions per day
● Faster iteration on issues that can be resolved without third parties
● Better support for test executers from other teams.

Problem:
Third parties were not engaged.
No VOTS resource was specifically budgeted to aid in the stress and volume testing,
The process for acquiring a resource to assist with an issue was slow and only secured the
resource for the minimum possible time required to provide a walk around / solution.
Impact:
● A Test failure due to an issue that required third party action cost 2 days minimum (1
day per test execution, minimum of 1 day before a resource is assigned and responds
to the issue, and then the relatively small amount of time to resolve the issue itself)
● Some sub-optimal solutions were taken as the third party teams were not resourced
to provide optimal ones.
● Repeated failures due to the same issue would frequently require multiple runs
through the resourcing process, incurring a 2 day delay each time.
Proposed Resolution:
 ● Acquire a VOTS/LANDATA resource/budget for the entirety of the stress and volume
process.
● Have co-location or a direct line between the EC resources and the VOTS resources
working on the stress and volume tests.
● Ensure resources are introduced at the commencement of the testing and the
responsibilities / expectations clearly set for both parties.
Benefit:
● Faster turn around time on issues that require third party involvement (down to either
1 day, or mere hours if independent infrastructure is used