Bell and Delicate - Business Planning and Evaluation FMI Deck Final

BusinessPlanningandEvaluation
Puttingthepiecestogether!
Bell,Browne,Molnar&DelicateConsulting
WhoWeAreBBMDConsulting
FocusonPerformanceMeasurementandProgramEvaluation
ExtensivePerformanceMeasurementandProgramEvaluation
experiencewithfederal,provincialandmunicipalgovernments
(currentlysupportingthePerformanceMeasurementeffortsof
CanadianandOntarioGovernmentDepartments)
Pleasevisitwww.bbmd.ca fordetails
Evaluation Definitionand
Purpose
3.1IntheGovernmentofCanada,evaluationisthesystematiccollectionandanalysisofevidenceonthe
outcomesofprogramstomakejudgmentsabouttheirrelevance,performanceandalternativewaysto
deliverthemortoachievethesameresults.
3.2EvaluationprovidesCanadians,Parliamentarians,Ministers,centralagenciesanddeputyheadsan
evidencebased,neutralassessmentofthevalueformoney,i.e.relevance andperformance,offederal
governmentprograms.Evaluation:
supportsaccountabilitytoParliamentandCanadiansbyhelpingthegovernmenttocrediblyreport
ontheresultsachievedwithresourcesinvestedinprograms;
informsgovernmentdecisionsonresourceallocationandreallocationby:
supportingstrategicreviewsofexistingprogramspending,tohelpMinistersunderstandthe
ongoingrelevanceandperformanceofexistingprograms;
providingobjectiveinformationtohelpMinistersunderstandhow newspendingproposalsfit
withexistingprograms,identifysynergiesandavoidwastefulduplication;
supportsdeputyheadsinmanagingforresultsbyinformingthemaboutwhethertheirprogramsare
producingtheoutcomesthattheyweredesignedtoproduce,atan affordablecost;and,
supportspolicyandprogramimprovementsbyhelpingtoidentifylessonslearnedandbest
practices.
Reference:PolicyonEvaluation,April1,2009
SomeDriversofMeasurement
y TheTreasuryBoardofCanadas1997Report
entitledGettingGovernmentRight:
GoverningforCanadians beganthe
fundamentalshifttoaresultsbased
managementagenda.
y ResultsforCanadians AManagement
FrameworkfortheGovernmentofCanada
(March30th ,2000)intensifiedthepush
towardsaresultsbasedmanagementculture
inthefederalpublicsector.
y AnewTreasuryBoardofCanadaEvaluation
PolicyimplementedinMarch,2009.
1.BriefHistoryofPM
TodaysReality
1.BriefHistoryofPM
AmericanGovernmentexample
www.whitehouse.gov/omb/expectmore/
1.BriefHistoryofPM
WhyMeasurementisImportant
2.WhyPMisimportant
StrategicRoadmap
Where are we
now?
Where do we
want to be?
How do we get
there?
How do we
measure our
progress?
Where do we go
from here?
SWOT Analysis
Internal and external assessment

Client/stakeholder analysis
Quality assessment
Strategic issues
Mission and Values
Broad statement of organization's mission

Core values, actions to achieve mission
Staff and clients/stakeholders
Vision
Image of your organizations desired future
Outcomes and Outputs
Outcomes (direct, shared and ultimate)

Specific, measurable outputs to achieve outcomes
Strategies and Action

Plans
Gap analysis
Detailed action/work plans
Resource allocation
Performance Measures
Performance indicators
Methods to measure program performance
Accountability and responsibility
Monitor, Evaluate &

Report on Performance
Data
Systems to monitor progress

Compile performance information
Provide information for resource allocation
Continuous Improvement
Products and services better, faster, cheaper
Innovation
Develop and implement new products and services to

meet new and emerging demands
WhyOutcomeBasedPerformance
Measurement?
3.PMandPlanning/ReportingResponsibilities
PublicSector
PerformanceMeasurementFramework
Strategic Direction
(What we should be doing)
Program Delivery
(What we do)
Various Program Perspectives
WHY
Citizen Focus
Managing for Results
Public Service Values
Responsible Spending
Change
Objectives /
Initiatives
Sustaining
The
Business
Changing
The
Business
Effectiveness
Benefits / Consequences / Impacts
Behavioural change
PROGRAM OUTCOMES
WHO
Strategies
Plans
Mission
Objectives/
Initiatives
Performance Measurement
(How well we are doing)
Satisfaction
Reach
CLIENTS/STAKEHOLDERS
WHAT
Efficiency
OUTPUTS
-
HOW
Cost
Time
Quantity
Quality
PROCESSES/ACTIVITIES
INPUTS
ENABLERS
FINANCIAL CAPITAL
HUMAN CAPITAL
STRUCTURAL CAPITAL
Effective Resource
Management
Sustainable Workforce
Leadership and Values
Enabling Work
Environment
Employee Satisfaction
Productivity/Retention
RESOURCE ALLOCATION
INVESTMENT DECISIONS
PRIORITY SETTING
STRATEGIES
10
FederalGovernmentExample
ImplementedtheMRRS
(Management
ResourcesandResults
Structure)
Strategic Outcome
Program Activities
Sub-Activity Level
Sub-Sub Activity Level
11
WholeofGovernmentFramework
12
PerformanceMeasures
(HRMScorecard,July2005,ManagingForResults)
StrategicOutcomes
SupportingOutcomes
SafeCommunities
TimelyandAppropriateEmergencyResponse
CitizensFeelSafe
Buildings,propertiesandinfrastructureinHRMaresafe,healthyandwellmaintained
Reasonableamountspenttomaintainbuildings,propertiesandinfrastructure
Healthy,
Sustainable,
Vibrant
Communities
HRMisadesirableandattractiveplacetowork,learn,playand live
PeopleandGoodscanmoveeasilythroughoutthemunicipality
PreservationoftheEnvironment
AttractionofNewBusinesses,RetentionandGrowthofExistingBusinesses
DevelopmentisAppropriatelyPlannedandFiscallySustainable
TaxBurdenisBalancedandCompetitive
Excellencein
Governance
Citizensfeeltheyaremakingadifference/participatinginthefuturedirectionofHRM
CitizensaresatisfiedthatHRMvisionandprioritieshavebeenimplemented
CitizensbelieveHRMworkswithotherlevelsofgovernmenttoimproveservicedelivery
CitizensareconfidentinthegovernanceandmanagementofHRM
HRMhassoundfinancialmanagementpractices
Excellencein
ServiceDelivery
Customersaresatisfiedwiththelevelofservicesreceivedfrom HRM
CustomersaresatisfiedwiththequalityofservicesreceivedfromCouncilandstaff
Citizensfeelthemunicipalservicestheyreceiveareworththetaxestheypay
13
PerformanceMeasures
(DepartmentofManagementandBudget,FairfaxCounty,Virginia,2005)
Terminology
Definition
Examples
Input
Valueofresourcesusedtoproducean
output
Dollarsbudgeted/spent
Staffhoursused
Output
Quantityornumberofunitsproduced.
Outputsareactivityoriented,
measurable,andusuallyunder
managerialcontrol
Eligibilityinterviewsconducted
Librarybookscheckedout
Childrenimmunized
Prisonersboarded
Purchaseordersissued
Efficiency
Inputsusedperunitofoutput
Costperappraisal
Plansreviewedbyreviewer
Service
Quality
Degreetowhichcustomersaresatisfied Percentofrespondentssatisfiedwithservice
withaprogram,orhowaccuratelyor
Errorrateperdataentryoperator
timelyaserviceisprovided
Frequencyofrepeatrepairs
Averagedaystoaddressafacilityworkorder
Outcome
Qualitativeconsequencesassociated
withaprogram/service,i.e.,the
ultimatebenefittothecustomer.
Externalforcescansometimeslimit
managerialcontrol.Outcomesfocuson
theultimatewhyofprovidingaservice.
Reductioninfiredeaths/injuries
Percentofjobtraineeswhoholdajobformore
thansixmonths
Percentofjuvenilesnotreconvictedwithin12
months
Adoption/redemptionrateofimpoundedanimals
14
MeasurementConcepts
4.PMConcepts
15
CoreIssuestobeAddressedinEvaluation
Relevance
Issue#1:ContinuedNeedfor
program
Assessmentoftheextenttowhichtheprogramcontinuestoaddress
ademonstrableneedandisresponsivetotheneedsofCanadians
Issue#2:Alignmentwith
GovernmentPriorities
Assessmentofthelinkagesbetweenprogramobjectivesand(i)
federalgovernmentprioritiesand(ii)departmentalstrategic
outcomes
Issue#3:AlignmentwithFederal
RolesandResponsibilities
Assessmentoftheroleandresponsibilitiesforthefederalgovernment
indeliveringtheprogram
Performance(effectiveness,efficiencyandeconomy)
Issue#4:AchievementofExpected
Outcomes
Assessmentofprogresstowardexpectedoutcomes (incl.immediate,
intermediateandultimateoutcomes) withreferencetoperformance
targetsandprogramreach,programdesign,includingthelinkage
andcontributionofoutputstooutcomes
Issue#5:DemonstrationofEfficiency
andEconomy
Assessmentofresourceutilizationinrelationtotheproduction of
outputsandprogresstowardexpectedoutcomes
Toaddressvalueformoney,evaluationswillberequiredtoassessallcoreissuesidentifiedabove
(asappropriate,departmentsmaychoosetoaddressadditionalissuesintheirevaluations).
Reference:DirectiveontheEvaluationFunction,April1,2009
EvaluationandPerformance
Measurement
Formalevaluationsgenerallyconsisttwotypes:
Summative
Formative
Theyaregenerallyundertakenonacyclicalbasis(midwayand
atProgramendoffundingagreementusually2and5years)
PerformanceMeasurementisanongoingoperational
collectionofperformanceinformationutilizedtomodify
businessoperations
Performancemeasurementisanintegralpartofevaluation
lifecycleandfeedsthecyclicalevaluations
17
Source:Exhibit RMAFGuide,TreasuryBoardSecretariat,August2001
ThePerformanceContinuum
StagestoDevelopment&Implementation
8
SummativeEvaluationFundamentalProgramIssues
FormativeEvaluation ManagementIssues
Review/Assess/ModifyPerformanceMeasurementSystem
Measure&ReportagainstPerformanceInformation
Develop/ReviseInformationSystems&DataProcessing
EstablishAppropriateDataCollectionStrategy
DevelopPerformanceMeasures&Indicators
ProgramProfile LogicModelOutcomes KeyEvaluationIssues
PerformanceMeasurementUnderstanding
18
PlanningEvaluationCycle
Reference:ResearchMethodsKnowledgeBase,http://www.socialresearchmethods.net/kb/index.php
MeasurementProcess
4.PMConcepts
20
TheBasisofMeasurement the
LogicModel
Thecausalorlogicalrelationshipbetweenactivitiesand
outputsandtheoutcomesofagivenprogram,policyor
initiativethattheyareintendedtodeliver.Alsoknownasa
resultschain.
Inputs
Activities
Outputs
Direct
Outcomes
Shared
Outcomes
High Control
Ultimate
Outcomes
Strategic
Outcomes
Low Influence
RESULTS
4.PMConcepts
21
TheFrameworkforEvaluation
Asaframeworkforevaluation,theLogicModelpromptsorganizations
toexamineallfourcomponents:
1.Theclarityandthoroughnessoftheplansandprioritiesandhow
closelythesealignwiththeorganizationsmissionandvision
2.Theeffectivenessoftheprocesses,includingtheadequacyof
resources,efficiencywithwhichtheyareleveraged,andtheextentto
whichtheoptimalstakeholdersareengaged
3.Theextenttowhichspecificandmeasurableoutputsandoutcomes
aregeneratedinthenearandmiddleterm,accordingtodefined
timelines
4.Theextenttowhichspecificandmeasurableimpactsaregeneratedin
thelongertermandtheextenttowhichthesetrulyrealizethevision
andmissionoftheorganization
22
ImplementingMeasurement
Frameworks
4.PMConcepts
23
Definitions
Needtohaveacommonvocabularywithinthe
organizationonmeasurementtermsandconcepts.
Thismeansprovidingspecificdefinitionsof:
Activities
Outputs
3LevelsofOutcomes
Theobjectiveistohaveallparticipantssingingfromthe
samesongsheet,commonunderstandingand
interpretationofterms.
4.PMConcepts
24
Definition:Activities
Activities=whatwedo
Describesacollectionoffunctions(actions,jobs,tasks)
Activitiesrequireresources($andpeople)
Forexample:policydevelopment,partnership
management,research,manufacturing,advocacy.
4.PMConcepts
25
Definition:Outputs
Outputs=whatweproduce
Describesthedirectproductsandservicesgeneratedthrough
activities
Outputsareusuallytangibleandconcrete
Forexample:thepolicyitself,partnershipagreements,
researchreports,manufacturedproduct,changesto
legislationresultingfromadvocacy.
4.PMConcepts
26
Definition:Outcomes
Outcomes=whywedoit
Describestheeffects,benefitsorconsequencesachieved
throughthedeliveryofvariousactivitiesandtheiroutputs
Outcomesaredifferentiatedbythedegreeofcontrolthe
organizationhasovertheachievementoftheoutcomeand
thedegreeofinfluencetheorganizationhaswithother
intermediariesinachievingtheoutcomes.
4.PMConcepts
27
TypesofOutcomes
Categorizedaccordingtothedegreeofinfluence,asfollows:
Direct
Outcomes
Firstleveleffectsof,orimmediate
responsetotheoutputs,e.g.,changesin
degreeofcustomersatisfaction.
Shared
Outcomes
Secondorderofoutcomes,benefitsand
changesinbehaviour,decisions,policies
andsocialactionattributabletooutputs
todemonstratethatprogramobjectives
arebeingmet,e.g.,increased
employabilityasaresultofatraining
program.
4.PMConcepts
28
TypesofOutcomes
Ultimate
Theultimateorlongterm
consequencesOutcomesforhuman,
economic,civicorenvironmental
benefit,towhichtheorganization
and/orgovernmentcontributesto.
Allofwhichfeedthe
Strategic
Outcomes
4.PMConcepts
Thelongtermandenduring
organizationaland/orgovernment
widebenefitsorgoalsthattheProject,
ProgramorOrganizationis
contributingto.
29
PrinciplesofGoodOutcome
Statements
1. Outcomesarenounbaseddesiredendstates
2. Directionaloutcomestatementsdonotbelonginlogic
modelsoroutcomestatements.Directionisachieved
throughtargetsnotoutcomestatements
3. Avoidselfservingstatements.Afterall,thisisnottheend
statedesired
4. Clearcauseandeffectlinkagetothenextlevelofoutcomes
5. Nowigglewordsthataresubjecttointerpretation(e.g.
effective,efficient).
6. SIMPLE: Outcomesdescribedinawaythathelps
shareholders,publicandotherstorelatetothe
desiredstatethatisbeingdescribed
4.PMConcepts
30
Whatchangesintheworldresultfrom
Inputs
Activities
Outputs
Direct
Outcomes
Shared
Outcomes
High Control
Ultimate
Outcomes
Strategic
Outcomes
Low Influence
RESULTS
KNOWLEDGE
KNOWLEDGE
Awareness
Awareness
Understanding
Skills
Understanding
Skills
ATTITUDES
ATTITUDES&
BEHAVIOURS
Perceptions
Perceptions
Acceptance
Acceptance
Involvement
Compliance
Action
EARLY
/ MID-TERM
BEHAVIOUR
EFFECTS
Involvement
change
Societal
Compliance
economic
Socio
Action
benefits
SocialMarketingRole
SocialMarketingRole
4.PMConcepts
LONG
EARLYTERM
/ LATER
EFFECTS
EFFECTS
Societal
long term
and
change
enduring
Socio- economic
organizational
benefits
and/or
governmentwide benefits
or goals
31
Indicators,Targets&Standards
Indicatorsmeasureprogress,providinginformationonhowwellweare
doing,forexample:
Averageresponsetimeforcallsofgreatesturgency
%ofemergencycallstoFirerespondedtowithin5minutes
Targetsarespecificperformancegoalstiedtoindicators,againstwhich
actualperformancewillbecompared,forexample:
10minuteresponsetimeforcallsofgreatesturgency
90%ofemergencycallstoFirerespondedtowithin5minutes
Standardsaresimilartotargets,andrefertoaserviceleveltowhichan
organizationispreparedtocommit,forexample:
30minutesoritsfree
10%refundifasimilarproductisadvertisedatalowerprice
4.PMConcepts
32
HowtoDeterminetheCritical
Few CharacteristicsofEvaluation
Credible provideacredible,independentview
Specific eliminateambiguity,showrelevancetotheexpectedoutcome
Linked clear(causeandeffect)linkagestootherindicators
Reliable scientificallyandstatisticallysound,provideanappropriate
degreeofaccuracy.Measurethesamethingacrosstime(allowing
comparisons)andfordifferentgroups/regions
Available dataareeasilyaccessibleorthereisalowlevelofeffortto
collectandanalyze
Costeffective thecostsforcollectionarealignedwiththeoverallutility
oftheindicators
Understandable datacanbeeasilygraspedbyvariousaudiences
4.PMConcepts
33
PresentationFormats
There are no hard and fast rules concerning the format for reports
the presentation will be dictated by the indicators, and the
management information needs.
Itshouldbeseldomthecasethatonelooksatanindividualelement
ofanevaluationinisolation.Abalancedapproachismoreusefulto
getthefullpictureoftheperformanceofanorganization
Thebalancedapproachisalsolikenedtoanairplanecockpit.Rather
thangivingpilotsjustonetroublelight (theplaneiseitherokayor
notokay),abankofgaugesissuppliedsothatthepilotmaysee
trendsandwheretroubleisdeveloping
4.PMConcepts
34
CitizenPerspective
CitizenasClient
Public
Public
Services
Services
Citizenas
Beneficiary
Public
Public
Resources
Resources
Public
Public
Benefits
Benefits
Citizenas
Taxpayer
Cost
Efficiency
Effectiveness
Health&Safety
Economicwellbeing
Fairness&equity
Publicsecurity
4.PMConcepts
Timeliness
Accessibility
Reliability
Responsiveness
Fairness
35
ForMoreInformationorServices,
ContactUs
(613)5623468
info@bbmd.ca
www.bbmd.ca
36
Annex SupportingInformation
InternationalapproachestoPerformanceMeasurement
EvaluationStandards
PerformanceInformationAssessmentGrid
DataAnalysisTechniques
PerformanceFrameworks tyingitalltogether
4.PMConcepts
37
ApproachesToMeasurement
Method
Country
UnitedKingdomOperatingandFinancialReviewwww.opsi.gov.uk/
UnitedKingdom
NouvellesRgulationsconomique www.legifrance.gouv.fr
France
Australianlegislationforpublicsectororganisation
www.apsc.gov.au/publications02/performancemanagement.htm
Australia
ASXPrinciplesofGoodCorporateGovernanceandBestPractice
Recommendations
www.asx.com.au/supervision/pdf/asxcgc_marked_amended_principles_021
106.pdf
Australia
MERITUMGuidelines MeasuringIntangiblestoUnderstandandImprove
InnovationManagementwww.nfrcsr.org/international/practical_tools
toolkits/
European
Commonwealth
(6countries)
KeizaiDoyukaiwww.doyukai.or.jp/en/
Japan
TheGlobalReportingInitiativewww.globalreporting.org/Home
UnitedNations
Danishguidelinesonintellectualcapitalreporting
www.pnbukh.dk/site/files/pdf_filer/Intellectual_Capital_Statements_
_The_New_Guideline.pdf
Denmark
1.BriefHistoryofPM
38
PerformanceInformationAssessmentGrid
Priority
Degree of importance/relevance in terms of measuring efficiency (doing things well) and/or effectiveness
(doing the right things/having an impact)
High
It is essential in order to measure efficiency and/or effectiveness
Medium
It is useful in measuring efficiency and/or effectiveness
Low
It is of little use in measuring efficiency and/or effectiveness
Data Accessibility
The degree to which data are readily available either manually and/or through an automated system
High
Indicator data are immediately available
Medium
The appropriate data are not immediately available through an automated or manual system
but could be gathered relatively easily
Low
The data to support this indicator are not currently available or accessible and will be difficult
to access
Level of Effort
An estimate of the level of effort required to report reliably against the indicator on a regular basis
Low
Minimal effort to gather, analyze, interpret and report reliably on the indicator
Medium
It requires a moderate degree of effort to reliably report on the indicator
High
It is very resource intensive to reliably report on the indicator
4.PMConcepts
39
Qualitativeassessment
Narrativereview
Contentanalysis,quantifyingqualitativedata
Identifyingandverifyingemergentthemes
Groundedtheory
Flowdiagrams
4.PMConcepts
40
Quantitativeassessment
4.PMConcepts
Descriptivestatistics(frequencies,means,
etc.)
Multipleregressionandanalysisofvariance
Metaanalysis
Trendanalysis
Structuralequationmodeling
Costeffectivenessanalysis,casecosting,financial
analyses,etc.
41
Tyingitalltogether
Phase 1: 1-3 months
4.PMConcepts
Phase 2: 1-3 months
Phase 3: 1-3 months
42

Bell and Delicate - Business Planning and Evaluation FMI Deck Final

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Bell and Delicate - Business Planning and Evaluation FMI Deck Final

Uploaded by

Copyright:

Available Formats

BusinessPlanningandEvaluation

Internal and external assessment

Mission and Values

Broad statement of organization's mission

Image of your organizations desired future

Outcomes and Outputs

Outcomes (direct, shared and ultimate)

Strategies and Action

Monitor, Evaluate &

Systems to monitor progress

Products and services better, faster, cheaper

Develop and implement new products and services to

Various Program Perspectives

Sub-Sub Activity Level

ProgramProfile LogicModelOutcomes KeyEvaluationIssues

It is essential in order to measure efficiency and/or effectiveness

It is useful in measuring efficiency and/or effectiveness

It is of little use in measuring efficiency and/or effectiveness

Indicator data are immediately available

It requires a moderate degree of effort to reliably report on the indicator

It is very resource intensive to reliably report on the indicator

Phase 1: 1-3 months

Phase 2: 1-3 months

Phase 3: 1-3 months

You might also like