Audit Program

Subject: Post Implementation Reviews

Version 1.0

Version Date: 15 October 2001

Author: David M Burbage

IT Auditor
Internal Audit Unit
Western Sydney Area Health Service
Locked Bag 7118
North Parramatta NSW Australia

e-mail: David_Burbage@wsahs.nsw.gov.au
Background

The audit will be undertaken as part of the 2000 / 2001 Audit Plan.

Scope and Period of Audit

To perform a post-implementation review

The audit reviewed outcomes of project/s (both computerised and non-computerised)

undertaken within the organisation, to verify whether the project/s achieved the desired
results and met the strategic outcome measures predicted by the organisation, within
the planned costs and schedules.

The review covered the period 1 January 1999 to 1 December 2000.

In order to determine the success of a project, the organisation needed to evaluate the
performance of the system after it is implemented. Through the Post-Implementation
Review (PIR), the organisation determined whether the project met its objectives and,
further, evaluates the development and management processes that supported the
project. The review also compares actual, total project expenditures to the project cost
estimates.

The PIR also determined whether the technology or change of work practice investment
is yielding the expected benefits to the processes, products or services supported by
the organisation.

Approach

The audit was conducted by :

* The circulation of a questionnaire to users, Project Managers, Data Centre Staff

and Systems Administrators of 5 projects to ascertain the level of acceptance,
ease of use, appropriateness of training, access to user documentation, etc.

* Interviews with project sponsors, project managers (internal and external where
available), systems administrators, data centre and ISD staff to ascertain the
status, maturity, robustness and overall capability of the project.

* Interviews with the project sponsor and other relevant staff to ascertain the
financial and other benefits realisation.
Post Implementation Review

Overall Audit Findings

Group Control Consideration

1. Contractor/s 1.1 Have contract details changed during the project

life-cycle?
1.2 Have cost and time parameters been met during the
project life-cycle?
1.3 Have professional standards and work standards
been met during the project life cycle?
1.4 Have all promised modules and functionality been
supplied and implemented?
1.5 Have contractors / sub contractors provided
adequate acquittance of claims, documentation of work
undertaken, outcomes, etc?
1.6 Have contracts been monitored?
1.7 Have contractors displayed adequate knowledge
and expertise in relation to the project?

2. Project Team 2.1 Have Team members been provided with an

effective working environment and intra-/ inter-/ discipline
support?
2.2 Have all disciplines been kept in the mainstream of
change and functionality at all times?
2.3 Has the team maintained good documentation
during the project life cycle?
2.4 Have issue logs been maintained, and have these
been ratified prior to sign off?
2.5 Were the project team adequately trained?

3. Users 3.1 Have users been trained adequately?

3.2 Have users accepted the system enthusiastically?
3.3 Have users concerns or problems been adequately
addressed?
3.4 Have user expectations been met satisfactorily?
3.5 Have user manuals been produced and are they
readily accessible?

4. Management 4.1 Has management displayed adequate support and

control over the project?
4.2 Have management revised policies and procedures
Post Implementation Review
or standards?
4.3 Have obligations of all parties been monitored and
validated?
Post Implementation Review

5. ISD 5.1 Has a monitoring process been established to

determine the efficacy of the system?
5.2 Has the system proved stable since go-live?
5.3 Is a service level agreement in place for the
system?
5.4 Have all parties been satisfied with the level of
service to date?
5.5 Has the system integrated effectively with other
systems?
5.6 Has liaison with the vendors been effective and
professional?
5.7 Has vendor support been adequate, effective and
timely?
5.8 Has security within the system been effective?
5.9 If the system has been a joint effort between two or
more vendors, have they worked effectively and united?
5.10 Have appropriate technical manuals been produced
relating to the system installed.

6. Project 6.1 Have payments relating to the project / system been

Sponsor based on contractual terms and conditions?
6.2 Have cost reports been viewed, verified and
validated?
6.3 Has the system changed substantially from what
was originally promised? Has this affected the Area=s
strategies, viability or work practices?
6.4 Have outstanding issues been resolved to the
satisfaction of the Area?
6.5 Have promised benefits been realised?
6.6 Has a method been implemented to capture
promised / anticipated savings?
6.7 If the project was implemented to support any
statewide strategy, has this been achieved?
6.8 Have performance agreements been substantiated
for contractors, vendors, etc?
6.9 Have milestones been signed off legitimately?
6.10 Has project documentation been retained and
maintained?
6.11

7. System 7.1 Has a system administrator been appointed?

Administrator 7.2 Have all users applied for, and received, access
WESTERN SYDNEY AREA HEALTH SERVICE
Post Implementation Review

Group Control Consideration

rights to the system?

7.3 Is there a hierarchical level of access for users,
super users, etc?
7.4 Has a criteria been set for enabling super user
status, etc?
7.5 Are issues, problems, errors, faults logged and
escalated, where necessary, to ISD or vendor
 WESTERN SYDNEY AREA HEALTH SERVICE
Post Implementation Review

Group Control Consideration

Risk Analysis

Objective Control Audit Test /

Risk Analysis
Questionnaire

1. The application as tendered was at 1, 2 1.1 - 1.7, 2.1 - 2.5, 3.2 - 3.4, 4.1, 2.1, 2.2, 3.1,
variance from what was functional at go live. 4.3, 5.5, 5.9, 6.3, 6.4, 6.7, 6.9, 6.10 3.2. 8.1, PA

2. User expectations have not been 2 1.3, 1.4, 2.2, 3.2 - 3.4, 4.2, 5.4, 3.2, 4.1 - 4.3,
realised. 5.7, 6.3, 6.4, 6.5, 6.7, 7.5 DC, EU, PA

3. Training has been inadequate. 1, 2 2.2, 2.5, 3.1, 3.5, 5.10, 5.11, 7.6 5.1 - 5.3, DC,
PA, EU

4. Manuals are inadequate and out of date. 1 3.5, 5.10 6.1 - 6.3, DC,
PA, EU

5. There is no ongoing administration of 1 3.3, 5.1, 5.6 - 5.8, 6.4, 7.1 - 7.5. 7.1 - 7.7, 8.2,
the system. PA, EU

6. The application is frequently unavailable 1, 2 1.4, 2.4, 3.3, 3.4, 4.3, 5.1 - 5.5, 9.1, 9.2, DC,
or does not work . 5.7, 6.4, 6.8, 7.5 PA, EU

7. The application requires constant 1 1.4, 2.2, 2.4, 3.3, 4.3, 5.1, 5.2, 5.7, 9.1 - 9.3, DC,
application of patches. 6.3, 6.4, 6.8, 6.10, 7.5 PA

8. No SLA was agreed to, or it is not 1, 2 3.4, 4.3, 5.1, 5.3, 5.4, 5.7, 6.4, 6.8, 10.1 - 10.3, DC,
operating. 6.10, 7.5 PA

9. The project has not been capitalised 3 1.2, 4.1, 6.1, 6.2, 6.6, 6.10 11., PA

10. The budget to actual costs were not 1, 3 1.2, 1.5, 1.6, 4.1, 6.1, 6.2, 12.1 - 12.4, PA
properly reconciled.

11. Savings outlined in the business case 1, 2, 3 1.2, 1.5, 1.6, 4.1, 6.1, 6.2, 6.5, 6.6, 12.4, 13.1, 13.2,
have not been realised. PA

Key:

DC - Data Centre Staff

PA - Project / System Administrator / Sponsor
WESTERN SYDNEY AREA HEALTH SERVICE
Post Implementation Review
EU - End User
 WESTERN SYDNEY AREA HEALTH SERVICE
Post Implementation Review

Audit Tests

Location / Unit Department

AUDIT TESTS Working Completed Audit

Paper Ref By & Date Result

1. Previous Management Letters

Determine if a previous audit has been performed
(internal / external) and that all recommendations
and proposed actions have been implemented.

2. Application Functionality
2.1 Review Business Case / project initiation
documentation / IPS and determine, by questionnaire
and interview with the Project Sponsor, whether the
application / system / project as installed /
implemented at go live stage varied adversely to the
inital proposal.
2.2 Review Change Control documentation to
ascertain level of changes to original specification

3. Pre Go-Live Issues

Review Project Steering Committee minutes / other
documents to ascertain the issues current at go-live.
3,1 Were all issues referred to Steering
Committee for go-live decision?
3.2 Have all issues been subsequently addressed
to the Area=s satisfaction?

4. User Expectation
By questionnaire:
4.1 Ascertain whether user expectations have
been met.
4.2 Ascertain whether ISD expectations have
been met
4.3 Ascertain whether project sponsor=s
expectations have been met
WESTERN SYDNEY AREA HEALTH SERVICE
Post Implementation Review
 WESTERN SYDNEY AREA HEALTH SERVICE
Post Implementation Review

5. Training
By questionnaire and Interview:
5.1 Users. Ascertain -
a. Whether users received training
b. Whether the training was timely, adequate and
pertinent
c. Whether users received take away workbooks
5.2 ISD. Ascertain -
a. Whether ISD staff (particularly data centre
staff) received training in the package.
b. Whether processes are in place to identify
deficiencies in training (e.g. constant calls from users
/ system administrators for advice on how to perform
basic functions).
5.3 System Administrator. Ascertain -
a. Whether System Administrators received
adequate and timely training.
b. Whether processes are in place to identify
deficiencies in training (e.g. constant calls from a
single user for advice on how to perform basic
functions).

6. Manuals
By questionnaire and interview:
6.1 Process users:
a. Do users have user manuals;
b. Are manuals kept up to date;
c. Are manuals readily available to all users.
6.2 Systems Administrators:
a. Do System Administrators have appropriate
technical manuals;
b. Are the manuals kept up to date;
c. Is access restricted to valid users only?
6.3 ISD / Data Centre:
a. Do staff have appropriate technical manuals;
b. Are the manuals kept up to date?

7. Administration
By questionnaire and interview ascertain:
7.1 Is there a designated administrator of the
system / process?
7.2 Are relevant records kept of:
 WESTERN SYDNEY AREA HEALTH SERVICE
Post Implementation Review

Location / Unit Department

AUDIT TESTS Working Completed Audit

Paper Ref By & Date Result
a. Down time;
b. Application issues, faults, errors;
c. User profiles;
d. Upgrades to process, access, documentation?
7.3 Are access rights reviewed and modified
where necessary?
7.4 Is there an ongoing review of the SLA to
determine need for revision?
7.5 Is the Administrator the custodian of the
project documentation, and is such documentation
stored securely?
7.6 Have performance statistics been maintained?
7.7 Have all users submitted an authorised
access agreement document?

8. Change Control
8.1 Have all changes from the original
specification been logged and authorised?
8.2 Have all changes to the application since go-
live been logged and authorised?

9. System Stability
9.1 Has the system proved robust and:
a. Has downtime been minimal;
b. Have users expressed concern over the
system (speed, durability, poor outcomes);
c. Are there substantial outstanding issues with
the vendors?
9.2 Are there ongoing problems with the
maintenance of the system (software / hardware /
peripherals)?
9.3 Have upgrades been provided in accordance
with contractual obligations?

10. Service Level Agreement

10.1 Is there a service level of agreement?
10.2 Have service levels been met?
 WESTERN SYDNEY AREA HEALTH SERVICE
Post Implementation Review

Location / Unit Department

AUDIT TESTS Working Completed Audit

Paper Ref By & Date Result
10.3 Has the SLA proved adequate?

11. Has the project been satisfactorily capitalised?

12. Cost reconciliation

12.1 Were budget to actual reconciliations
performed during the life of the project?
12.2 Were reconciliations presented to the Project
Steering Committee during the project life cycle?
12.3 Was a final reconciliation performed and
submitted to the Steering Committee at go-live time?
12.4 Are reconciliations performed regarding
running costs and to indicate the cost effectiveness
of the system?

13. Cost benefit savings identified in the business

case or other pre-project documentation:
13.1 Have been achieved and validated.
13.2 Have yet to be achieved due to the time frame
for such savings, and there is a mechanism in place
to capture such savings.

Questionnaire : DC - Data Centre Staff