Performance Testing End – To – End Through

Technology

DRAFT
July 2009
Contents
Purpose of Document..................................................................................................................................3
Components of Performance Testing..........................................................................................................3
Testing Approach.........................................................................................................................................5
Objectives of Testing Approach...............................................................................................................5

Components of Testing Approach...........................................................................................................5

Inputs of the Testing Approach...............................................................................................................5

Assumptions of Testing Approach...........................................................................................................7

Testing Approach – Details......................................................................................................................7

Step 1: Handheld RF Certification............................................................................................................8

Step 2: Truckmount Certification...........................................................................................................10

Step 3: Voice Certification.....................................................................................................................11

Step 4 – End-to-end RF Certification, Voice Certification and PC Load Testing......................................13

Other Considerations for Testing Approach – Lessons Learned from Francis Hughes...........................13

Risks – Not Preparing for Volume /Stress Testing.....................................................................................14

Benefits – Preparing for Volume/Stress Testing........................................................................................14
DC Activities - Dependencies.....................................................................................................................15
LCL DC Volume Details...............................................................................................................................16
Appendix...................................................................................................................................................17
Volume of Floor Devices........................................................................................................................17

2
Purpose of Document
The goal of a successful WMS implementation is to make sure to minimize any disruption of business
during implementation, at Go-Live and post Go-Live.

One of the ways in which companies can meet business and operations requirements is by conducting
Performance Testing, more specifically, Volume and Stress Testing.

The purpose of this document is to start thinking about WMS performance testing at the South Surrey
distribution center for the following considerations:

1. How to leverage testing at a Greenfield site to also begin validating testing for the larger LCL
DC’s
2. How to make sure that components of business processes occurring in the DC’s have been
tested under stress
3. How to make sure test scenarios have been well thought out on ways to break the system using
greater volumes of subsequent DC’s
4. How to leverage testing at South Surrey to better understand the larger DC’s
5. Benefits of testing real end-to-end versus simulated testing.
6. How to make sure end-to-end testing is conducted for devices, truckmounts and VoCollect.

Components of Performance Testing

The key to successful performance testing is to understand the business context that the organization
the business is operating in and the application usage – both anticipated and current levels – as well as
the architectural and systematic design, both at peak and steady state.

The components of testing are as follows:

3
* Volume Testing and Stress Testing are considered to be the two key areas within Performance Testing

Key areas to take into account during Performance Testing are as follows:

1. People - This includes man power to handle the processes involved within the DC, as well as the
people interaction with technology on the DC floor
2. Data Volume / Mix – This includes the volume of data moving through the WMS workflow, as
well output /reporting of data
3. Hardware – Equipment used in the DC (i.e., handhelds, VoCollect headsets, scanners, printers
and desktops etc.)

4
Testing Approach
The testing approach is imperative on where, when and how Volume and Stress testing will be
conducted. The approach that will be used will align our efforts toward testing the scalability of the
technology in order to proactively test whether or not requirements of the largest DC are met.

Objectives of Testing Approach

The Testing Approach will validate the following questions:

 Is VoCollect / Manhattan scalable enough to meet the demands of the larger DC’s?
 Is there a solution for capacity for daily spikes / blackouts?
 Are we in good shape for future implementations for the larger DC’s?

Components of Testing Approach

In order to validate the scalability of the WMS solution, the following components of volume will be
included in the testing approach:

 # of floor devices used within the main shift

 # of shifts that occur within a 24 hour period
 Other scenarios that cause spikes within a 24 hour period

Inputs of the Testing Approach

In order for Testing to be effective and valid, actual and relevant data from LCL DC’s will be used. This
data will serve as inputs to our Testing Approach.

The inputs are as follows:

1. Range of Low, Medium and High daily volume of business transactions as follows:

Figure 1:

Volume Metrics
Range of Volumes (Daily)
Business Transaction Device Used
Low Medium High
Receiving RF’s 2 36 125
Putaway RF’s 3 19 35
Replenishment RF’s 6 14 29
Picking Voice 10 110 240
Shipping RF’s 2 24 45

5
Note: RF’s include handhelds and truckmounts
Further details on specific activities can be found in the Appendix A - C.
Note: Metrics are based on data provided by LCL as relevant

Time line in order of DC Go-Lives and total number of users by technology (data provided by Dominique
Potvin and her team on. Refer to spreadsheet on page 16):
Figure 2

*Note: Although Vancouver illustrates the highest number, it is not indicative of the daily volume of business
transactions generated on a daily basis.

Figure 3

6
**Note: Although Vancouver illustrates the highest number, it is not indicative of the daily volume of business
transactions generated on a daily basis.

Assumptions of Testing Approach

Key assumptions of the Testing Approach are as follows:

1. End-to-end non-simulated testing with real data testing before go-live where applicable
2. Testing based on business process that occur at the DC as opposed to testing
equipment/application functionality
3. Test at the DC (first one will be at South Surrey)
4. Test well in advance of go-live

Testing Approach – Details

Testing will be a four (4) step process as follows:

Step 1 – Handheld RF Certification

Step 2 – Truckmount RF Certification

Step 3 – Voice Certification

Step 4 – RF and Voice Certification & PC Users

Key considerations for the above four (4) step process are as follows:

 Testing during these steps will focus on testing RF’s by daily volume ranging from Low to High for
each business transactions as illustrated in Figure 1 and Figure 2 above.
 Each level of the range starting from lowest to highest needs to be tested for since the highest range
represents the main shift at the DC.
 Testing for Low, Medium and High volume may be restricted to the number of users at each DC.
 If capacity issues arise during any of the sequential order of testing, they must be resolved to
provide the level of confidence required to move to the next highest range.
 Currently, combined numbers for Handheld and Truckmounts exist per DC (except for South Surrey),
hence larger volume testing during Steps 1 and Steps 2 will be based on these combined numbers.
 Since numbers by DC do exist for Voice, larger volume testing during Step 3 will be based on these
DC numbers.
 Hammer is an additional Load Runner Module that simulates the keying of the handheld device. We
currently do not own.

Step 1: Handheld RF Certification

RF Certification will involve three (3) testing steps as follows:

7
 a) Step approach to test daily number of lowest to highest RF utilization at DC (manual end-to-
end testing of Handhelds )
b) Use RF Hammer to calibrate simulation of real RF utilization from step a) to compare the
results of the volume of the two (2) curves and to understand how to adjust the curve as
necessary (simulated testing of Handhelds)
c) Step approach to test daily number of lowest to highest RF utilization + 20%

a) Step Approach

The following range will used as benchmark for RF utilization at South Surrey:

Note: Data used for South Surrey based on equipment purchase orders as at today (July 2009) and not by number of employee
by main shift using technology.

Manual testing on the Handhelds will be done at this step. Testing at each of the above levels will
provide confidence that Handheld volume requirements at South Surrey will be met.

b) RF Hammer - Handhelds

The purpose of testing with the RF Hammer is:

1) To make sure testing results from the curve load are similar to the results from step a)
2) To test to a volume high enough to meet the demands of the largest DC, in order to build
confidence to generate greater volumes

8
Note: As noted in the key considerations earlier, separate numbers for Handhelds do not exist by DC. Hence the numbers noted
in the above graph reflect the combined numbers for Handhelds and Truckmounts by DC. If however, more reflective numbers
do exist at LCL, they will definitely be considered as part of this stage of testing.

Simulated testing at this step will validate and build confidence that simulation of the RF technology is
able to meet the demand of real life. In addition, testing the RF Hammer using RF utilization will also be
done as deemed appropriate.

c) Step approach for largest DC + 20%

In order to account for future growth at the DC and further confidence, a 20% increase will be added to
the utilization of the largest DC’s from Step b) as follows:

Note: As noted in the key considerations earlier, separate numbers for Handhelds do not exist by DC. Hence the numbers noted
in the above graph reflect the combined numbers for Handhelds and Truckmounts by DC. If however, more reflective numbers
do exist at LCL, they will definitely be considered as part of this stage of testing.

Key metrics for the RF business transactions will include:

Inbound Cartons Outbound Cartons

Inbound Pallets Outbound Pallets
Inbound Trailers Outbound Trailers
Outbound Orders Outbound Lines
Please refer to Appendix (A and B), for metric details

9
Step 2: Truckmount Certification
Truckmount Certification will involve three (3) testing steps as follows:

a) Step approach to test daily number of lowest to highest Truckmount utilization at DC (manual
end-to-end testing of Truckmounts)
b) Use RF Hammer to calibrate simulation of real RF utilization from step a) to compare the results
of the volume of the two (2) curves and to understand how to adjust the curve as necessary*
c) Step approach to test daily number of lowest to highest RF utilization + 20%*

a) Step Approach

The following range will be used as benchmark for Truckmount utilization at South Surrey:

Note: Data used for South Surrey based on equipment purchase orders as at today (July 2009) and not by number of employee
by main shift using technology.

Manual testing at each of the above levels will provide confidence that Truckmount volume
requirements at South Surrey will be met.

b) RF Hammer - Truckmounts

As noted in Step 1, the purpose of testing with the RF Hammer is:

1) To understand how test results from the RF Hammer compare to the test results in step a) and
analyze how big the misalignment of the two curve loads are
2) To test to a volume high enough to meet the demands of the largest DC, in order to build
confidence to generate greater volumes

10
*Note: Although the RF Hammer is meant to test Handhelds and not Truckmounts, for purposes of
testing volume of the Truckmounts, the same numbers will be used as in Step 1 for the RF Hammer since
only combined numbers for RF usage currently exist for the other DC’s.

c) Step Approach for largest DC +20%

In order to account for future growth at the DC and further confidence, a 20% increase will be added to
the utilization of the largest DC’s from Step b).

Same numbers will be used as in Step 1 for the RF Hammer since only combined numbers for RF usage
currently exist for the other DC’s.

Step 3: Voice Certification

Voice Certification will involve two (2) testing steps as follows:

a) Step approach to test daily number of lowest to highest Voice utilization at DC (manual end-
to-end testing of VoCollect)
b) Step approach to test daily number of lowest to highest Voice utilization (manual end-to-
end testing of VoCollect using trained staff at South Surrey and extrapolation methods).

a) Step Approach

Similar to the Handheld and Truckmount Certification in Step 1 and Step 2, testing will be focused on
Voice by daily volume ranging from Low to High for the business transactions that use VoCollect.

Note: Data used for South Surrey based on equipment purchase orders as at today (July 2009) and not by number of employee
by main shift using technology.

Manual testing leveraging all staff trained in VoCollect as well as any cross trained labour at South Surrey
will be used to test each of the above levels. This will provide confidence that VoCollect volume
requirements at South Surrey will be met.

11
 b) Step approach by levels of South Surrey Staff and extrapolation

This step will manually test to a volume high enough to meet the demands of the largest DC, in order to
build confidence that greater volumes can be generated.

Similar to step a), manual testing leveraging all staff trained in VoCollect as well as any cross trained
labour at South Surrey will be used to test each of the above levels. This will be done in order to
additional VoCollect requirements for the larger DC’s.

Since the level of how much testing can be done at this step is limited to the number of trained
VoCollect staff at South Surrey, the remaining testing will be done using full extrapolation. This will need
to be revisited prior to the Vancouver go-live date in May 2010.

Key metrics for the Voice business transactions will include:

Avg. SKU’s picked daily Picking

Please refer to Appendix (C), for SKU metric details

12
 Step 4 – End-to-end RF Certification, Voice Certification and PC Load Testing
After each of the devices have been tested individually to account for DC requirements (Step 1 – 3), this
step will include PC Loading testing and combine the testing for Step 1, Step 2 and Step 3, to include
both RF and Voice Certification to test Low to High volumes of each technology simultaneously to
account for end-to-end testing.

Other Considerations for Testing Approach – Lessons Learned from Francis

Hughes
This section describes the other considerations for the testing approach, as well as lessons learned from
the recent WMS implementation roll-out at Francis Hughes.

DC Process Description of Key Activities Equipment Anomaly Complicating Factors

(Shift Cycle)
Receiving  Entering Shipment Numbers  Handhelds  Concurrent Applications or Background Processing :
 Scanning the Location  Truckmounts Inventory Snapshot Reporting
 Verification (ASN & Shipment)  Desktop UI Waving
 Scanning the ILN Routing – Trucks (Legacy or TMS)
Forklift  Putaway  Truckmounts Vendor Records Download
 Replenishment  Printers Store Records Download
 Picking Full Pallets  Desktop UI Store Orders Download
Database Stats
Voice Assembly  Picking Multiple Pallet Orders  Voice Orders  Concurrent or Integrated processes
 Picking Shorts  Printers dropping E.g., Shift change with break picklist
 End Pallet  Desktop UI  Volume Spikes
 Breaking Picklists Size of orders per day
Loading  Loading Trucks  Handhelds # orders per day (rerouting, seasonal spikes)
 Decking  Truckmounts # people per shift by role (e.g.,. 50+ pickers)
 Cross Docking  Desktop UI # shifts per day / week
 Assigning Trailers to Doors # inbound deliveries received
# outbound shipments required
Shift Change  Login  Voice
 Cutover Reporting and Database Queries
 Logout  Desktop UI

Volume and stress testing should be based on end-to-end business processes that occur at the DC
during each shift in the production environment, including: receiving, forklift, voice assembly, and
loading, as well as the shift change activities that occur at a point in time.

Within each process, there are several key activities which represent the interactions between the DC
equipment and the DC workforce. Each activity triggers the send/receipt of data through the system
and where volumes may spike by shift, week, or month and cause variance in system traffic which must
be accommodated based on accepted tolerances. The DC equipment used to perform these functions
includes: handhelds, truckmounts, voice consoles, printers and desktops all located on the DC floor.

13
Based on experience at the previous Francis Hughes roll-out the following performance anomalies were
uncovered as follows:

1. Across all processes, system performance decreased when one or more of the following
complicating factors were present: concurrent processes (e.g., waving, routing, queries) or
reporting functions (e.g., inventory snapshot) were running, several processes were combined (e.g.,
executing break picklist during a shift change), or there were spikes in daily, weekly, or monthly
inbound and outbound processing volumes. These complicating factors each represent a draw on
the system’s processing ability and occur throughout the 24 hour operation clock for the DC.

2. During the Cutover and Go Live period, additional reporting and queries were executed against the
system due to a requirement for closer performance monitoring. Additionally, throughout the
Cutover and Go Live period, data volume may have increased due to users incorrectly executing
functions, duplicating functions, or executing functions out of sequence. This caused an additional
processing draw that is not typical within the production environment.

Risks – Not Preparing for Volume /Stress Testing

The following are risks/implications of not taking into consideration Performance testing:

 Solution does not scales to meet business requirements

 Disruption to the business (cost increase)
 Not being able to anticipate and plan for larger DC implementations
 Mandatory overtime
 Diversion of orders to other DC’s
 Lost revenue at stores due to product stock out
 Late orders
 Order accuracy
 Loss of confidence
 Reactive to business process

Benefits – Preparing for Volume/Stress Testing

The following are benefits of taking into consideration Performance testing:

 Preparing for larger DC implementations

 Mitigate disruption to business
 Confidence that all aspects of DC operations will be functional at Go-live

14
  Focus attention to improving business operations and improving metrics
 Improve upon lessons learned
 Customer satisfaction
 Proactive to business process

DC Activities - Dependencies
Majority of the activities within the LCL Data Center are interrelated with technology (refer to red star in
figure below).

Refers to activities within the DC that interface data with users and technology.

It is important for the following components required in the DC be in place:

 People
 Hardware
 Data

These three components are crucial for Volume and Stress testing to build confidence required for
successful Go-Live as well as post Go-Live.

15
Examples testing for these components include the following:

- Impact of shift changes

- Impact of volume
- Impact of hardware failure
- Impact of order changes

Other considerations include:

- Contingency Planning
- Disaster Recovery
- DC Infrastructure
- Metrics (benchmarking)

LCL DC Volume Details

The attached spreadsheet provides details of each of the LCL DC’s in terms of size, # of people as well as
range of volume.

Scalability_WOS_Dat
a_Request_results v 3.xls

A plan to carry out volume and stress testing that takes into consideration the mid to high volume DC,
will help carry out the benefits as outlined earlier in this document.

16
Appendix

Volume of Floor Devices

Appendix A - Inbound Profile:

17
18
Appendix B: Outbound Profile:

19
Appendix C: Average SKU’s Picked per Day

20