Australian Evaluation Society Seminar

12 Nov 2015 12pm – 1:30pm

Performance Audit
Methods and Examples
Presenters
Jeremy Bean and Amelia Miller

1
 Performance Audit
Performance audits are carried out on the
efficiency and effectiveness of public sector
agencies, services and programs in
accordance with section 18 of the Auditor
General Act 2006

Two types:
• Narrow scope performance audits
• Broad scope performance audits

2
 Performance Audit

Performance auditing requires analytical

and creative skill
In contrast to financial auditing, it focuses on
the activity rather than the accounts
In contrast to compliance auditing, it relates
mainly to whether government initiatives are
achieving the government’s intentions

3
 Performance Audit vs Evaluation

Evaluation
• systematic assessment of the
appropriateness, effectiveness and/or
efficiency of a program, or part of a program
• includes assessment of government policy
• doesn’t have to result in an opinion
• reports to government or department

4
 Performance Audit vs Evaluation
Performance Audit
• examines the economy, the efficiency and
the effectiveness of government programs
and agencies
• does not assess government policy
• independent of government
• expresses an opinion
• reports to parliament

5
 Performance Audit
Basic Questions:
• Have the right things been done?
• If so, are things being done in the right
way?
• If not, what are the causes?

6
 Methodology and Processes

7
 Methodology and Processes

8
 Topic Generation and Selection
 The Auditor General may conduct
examinations and investigations at any
time and in such a manner as he sees fit
 Sources of ideas:
• Internally – staff
• Externally – Ministers, Agencies, the
public
• The AG’s own interests
 Some selected for further development

9
 Topic Generation and Selection
 Topics are chosen based on:
• topicality
• materiality
• Timeliness
• auditability
• interest from parliamentarians
• ability to add value
 Development of an initial Audit Objective
and Lines of Inquiry

10
 Examples of Topics
Student Attendance follow-on (recent)
In 2007 – initially something on schools –
“Literacy and Numeracy” – too focused on
pedagogy

Two audits and a follow-on:

• Student Attendance (2009)
• Behaviour Management (2014)
• Student Attendance Follow-on (2015)

11
 Examples of Topics
Delivering Services Online (current)
• 2013 idea focused on government use of
social media – expanded to digital
government
• Audit angle?
Narrowed to online services with emphasis
on transactions – case studies
• PAC inquiry into ICT procurement
• GCIO started in July 2015

12
 Proposal Development
 Background research
 Objective and Lines of Inquiry – most
important!
 Budget
There are no hard and fast rules on developing
lines of inquiry. To do it well requires:
• ability to assimilate large amounts of
information, often conflicting and biased
• ability to grasp bigger picture
• experience!

13
 Developing the audit
RAESP
Objective and Lines of Inquiry (initially)
To assess how well essential services are being
delivered to town-based and in remote Aboriginal
communities
1. Is there sound contracting and oversight of both
programs?
2. Have the improvement programs delivered
improved outcomes?
3. Is there good coordination between agencies
and organisations?

14
 Developing the audit
RAESP
Objective and Lines of Inquiry (after PAR)
To assess how well Housing delivers essential services to
remote Aboriginal communities through the RAESP. The
audit focuses on whether:
1. the RAESP is providing effective essential services to
remote communities in accordance with relevant
requirements, standards and guidelines
2. Housing actively manages the maintenance and repairs
of essential services in RAESP communities
3. Housing is integrated in its approach and coordinated
with other relevant agencies in delivering maintenance
and repairs of essential services in RAESP communities

15
 Developing the audit
Student Attendance Follow-on
Objective and Lines of Inquiry (initially)
To assess whether school attendance has improved,
review the Department of Education’s implementation of
the 2009 audit recommendations and broader changes in
attendance strategy and policy and assess progress in
addressing persistent non-attendance.
1. Has student attendance improved since 2009?
2. Has the department taken reasonable steps to address
the problems identified in 2009?
3. (What is happening with) the increase in persistent non-
attenders who are at severe educational risk?

16
 Developing the audit
Student Attendance Follow-on
Objective and Lines of Inquiry (after IFR)
To assess if student attendance had improved since our
2009 report
1. Does the Department of Education have
comprehensive information on student attendance?
2. Has the Department of Education taken reasonable
steps to address the issues identified in the 2009 audit?
3. Is the Department of Education addressing the needs of
key high-risk groups, including Aboriginal and low
socio-economic index status?

17
 Planning

18
 Fieldwork
Audits cover a very broad range of topics and are
unique. There is no “one size fits all”
• Document review
• Examine databases
• Visit sites, take photos

19
 Fieldwork
• Interview agency staff
• Interview stakeholders
• Conduct surveys
• Map processes

20
 Fieldwork

21
 Fieldwork
• Review websites, mobile sites
• Consult expert advice, external and internal
• Compare with other jurisdictions, benchmarks
• ANALYSE! – statistics, visualisation

22
 Example of fieldwork
Student attendance follow-on
• 6 schools in metropolitan area
• 2 schools in regional areas
• 4 education offices
• examination of databases
• review websites
• compare with other jurisdictions

23
 Example of fieldwork
Student attendance
ANALYSIS of statistics and visualisation

24
 Example of fieldwork
RAESP
• 27 remote communities
o 3 in the Goldfields
o 7 in the Pilbara
o 17 in the Kimberley
• 4 private contractors
• 2 government agencies
• Overlapping jurisdictions
• Several contracts, large databases, scientific
reports, social and cultural factors, anecdote

25
 Reporting: Writing is analysis

Reporting takes as long as the fieldwork stage.

Much analysis is done during drafting, especially the
non-quantitative stuff.

INTOSAI: Information analysis is an intellectual,

creative, and iterative process, which includes both
rational and irrational (creative) elements.
It always involves reflections and discussions,
brainstorming, and mostly non-quantitative
techniques such as content analysis, comparative
analysis, analysis with the aid of experts etc.

26
 Report Development
Where we choose what the story is. Many factors to
consider:
• What does the AG want to say?
• What will parliament / public be interested in?
• What really matters?
An audit needs to provide an opinion; it’s not just research
and we don’t do it in a vacuum.
Summary of Findings
Quality Assurance (QT4)

27
 Example of Report Development
Student Attendance Follow-on PA
• Same data and different outcomes
• Regular attendance (90% or more)

2009: 72% 2014: 70%

• Better Attendance, Brighter Futures Strategy

20 schools improved, 20 schools worsened

Is this Positive, Stable, or Negative?

28
 Example of Report Development
Factors include:
• Amount of variance
• Opinion of many people
• Agency Agreement

Findings:
• Stable attendance rates
• Negative finding with regards to Strategies

29
 Procedural Fairness
• Presenting findings to agencies
• Clarification and agreement of findings
• Sometimes change happens during the
audit…
• and just prior to publication!
• Agency response

30
 Example of agency response
Student attendance follow-on PA (2015)

31
 Findings and Recommendations
 Key Findings
 Recommendations to agencies
 Media and Publicity
 Public Accounts Committee follows up
 Follow-on Performance Audits
 Follow-up Performance Audits

32
 Tabling the Report and Close Out
• Presentation to parliamentarians and
interested stakeholders
• Media attention
• Stakeholder management – we’re
frequently in agencies and need to
maintain a good working relationship
• ORIMA surveys

33
 Post Project Review & Continuous Improvement
What went well

What could be improved

Budgets are often discussed

Identification of future topics

34
 Example of agency response
Parliamentary Education and Health Standing
Committee report (November 2012)

35
 Example of agency responses
Student attendance follow-on recommendation
(August 2015):
The Department of Education should provide better and
more transparent information on student attendance,
including in its Annual Report. This should include:
• setting and reporting on targets for attendance at
school level
• reporting student attendance by category of risk.

36
 Example of agency responses
Department of Education Annual Report 2015-16
(Oct 2015)

37
 Questions?

Thank you!
38