INFORMAL REPORT: Failure Analysis

MEMORANDUM

TO: Charles Mathers, EVP, Trading

Carole Chatwick, EVP, Back Office
Taylor Petrovski, SVP, Risk Management
FROM: Sheryl Cochrane, AVP, Trading Systems Support
DATE: 19 September 2011
SUBJECT: Failure of Overnight Batch Processing

OVERVIEW

Provide any background Each night an automated batch program runs, which performs the following actions:
info readers may require
 Backs up all servers and databases
 Produces trader reports for start-of-day portfolio positions
 Produces back-office reports for settled trades
 Produces risk management reports
 Forwards trade information to corporate headquarters
Clearly identify the Yesterday morning, none of the reports had been produced when staff showed up for work.
problem being looked
into Upon further investigation, it was discovered that none of the back-ups had run, and trade
information had not been forwarded to our central office.

CAUSE OF BATCH PROGRAM FAILURE

Clearly state the cause The Trading Systems Support team investigated the failure, and determined that it was
of the failure
caused when the login failed for the automated process that initiates the batch program.

As part of the new security measures required by internal audit, the system now requires
users to update their password every three months. The system gave the process running
the batch program three warnings to update its password, but as the process is not run from
a terminal, but automatically kicked off each night at 1:00 am, the warnings were not
visible. After three warnings, the system denied access to the process running the batch
program.
 SOLUTION
End by providing the The password for the process that runs the batch program has been changed. To ensure this
solution, one or more
recommendations, or the does not happen again, we've introduced the following procedures and safeguards:
next steps needed to  The Data Centre has a new procedure to update the batch process password on the
identify the cause of first of every month.
failure  Any messages from the batch process are now routed to a terminal in the Data
Centre, so error messages will be visible.
 The new instruction has been added to the batch program, instructing it to page the
on-call support person if the program fails at any point.