430 PDF

PRACTICAL METHODS FOR
EVALUATING COACHING
OthertitlesfromIES:
ExecutiveCoaching:InspiringPerformanceatWork
CarterA
IESReport379,2001.ISBN1851843086
BeyondtheScreen:SupportingeLearning
PollardE,WillisonR
IESReport425,2005.ISBN1851843558
andforHRNetworkmembers,thesepapersareavailablefromtheMemberswebsite:
ProvidingCoachingInternally:aLiteratureReview
CarterA
IESNetworkPaperMP43,2005.ISBN(noISBN)
BuildingCoachingCapability
CarterA
ChangingSkillMix:ARecipeforSuccess
TuohyS,ReillyP,HaydayS
Acatalogueoftheseandover100othertitlesisavailablefromIES,orontheIES
website,www.employmentstudies.co.uk
Practical Methods for
Evaluating Coaching
Alison Carter
Report 430
Publishedby:
INSTITUTEFOREMPLOYMENTSTUDIES
MantellBuilding
UniversityofSussexCampus
Falmer
BrightonBN19RF
UK
Tel. +44(0)1273686751
Fax +44(0)1273690430
http://www.employmentstudies.co.uk
Copyright2006InstituteforEmploymentStudies
Nopartofthispublicationmaybereproducedorusedinanyformbyanymeans
graphic,electronicormechanicalincludingphotocopying,recording,tapingor
informationstorageorretrievalsystemswithoutpriorpermissioninwritingfrom
theInstituteforEmploymentStudies.
BritishLibraryCataloguinginPublicationData
AcataloguerecordforthispublicationisavailablefromtheBritishLibrary
ISBN185184337X
PrintedinGreatBritain
The Institute for Employment Studies

TheInstituteforEmploymentStudiesisanindependent,apolitical,international
centreofresearchandconsultancyinhumanresourceissues.Itworkscloselywith
employersinthemanufacturing,serviceandpublicsectors,governmentdepartments,
agencies,andprofessionalandemployeebodies.Forover35yearstheInstitutehas
beenafocusofknowledgeandpracticalexperienceinemploymentandtraining
policy,theoperationoflabourmarketsandhumanresourceplanningand
development.IESisanotforprofitorganisationwhichhasover60multidisciplinary
staffandinternationalassociates.IESexpertiseisavailabletoallorganisations
throughresearch,consultancy,publicationsandtheInternet.
IESaimstohelpbringaboutsustainableimprovementsinemploymentpolicyand
humanresourcemanagement.IESachievesthisbyincreasingtheunderstandingand
improvingthepracticeofkeydecisionmakersinpolicybodiesandemploying
organisations.
The IES HR Network
ThisreportistheproductofastudysupportedbytheIESHRNetwork,through
whichMembersfinance,andoftenparticipatein,appliedresearchonemployment
issues.FullinformationonMembershipisavailablefromIESonrequest,orat
www.employmentstudies.co.uk/network/.
v
Acknowledgements
Theauthorisareindebtedtoallthecoaches,coachees,coachingprogramme
organisersandcasestudyparticipantsinvolvedintheresearch.Particularthanksgo
toTraceyConnage,JoFellows,LynneButler,GillElliot,KevinHaynes,Eddie
Gibbons,KarenIzodandWillemVanBaarsen.Thanksarealsoowingtothe
InternationalJournalofMentoringandCoachingforgivinguspermissiontousethe
Bristol&WestBuildingSocietycasestudy.
IESisgratefultotheIESManagementandEmployeeDevelopmentResearchNetwork
memberswhosponsoredourresearch.Atkeystagesoverthethreeyearperiodofthe
study,somemembersalsogavetheirtimetohelpshapetheresearchproject,testthe
emergingmodelofcoachingand/orcommentontheemergingissues.Memberswho
contributedtoourthinkingincludetheseminarparticipantsattwoIESResearch
NetworkconferencesinMarch2004andApril2006.
MentionmustalsobemadetocolleaguesatIESwhohavecontributedinspirationally
andintellectuallytothisproject.TheseincludePennyTamkin,RichardHayes,Jim
Hillage,KeithMattacksandPenelopeJohnson.SpecialthanksgotoMareKerrinfor
herliteraturereviewandtoHelenWolfeandJonnyGiffordforanalysisoftheLEAP
data.
vi
Contents
ExecutiveSummary ix
1 Introduction 1
1.1 Backgroundtotheresearchproject 1
1.2 Method 2
1.3 Structureofthisreport 2
2 DoesCoachingActuallyWork? 4
2.1 Iscoachinganeffectivetool? 4
2.2 Whatcanwelearnfrommentoring? 5
2.3 Whataboutreturnoninvestmentapproaches? 6
2.4 Issuesinevaluatingcoaching 8
2.5 Summaryofsuggestionsarisingfromtheliterature 13
3 OrganisationalExamplesofEvaluationPractice 14
3.1 Tmobileidentifyingpriorityareasforfuture 15
3.2 Corustrackingachievementofobjectives 16
3.3 NHSinWaleslearningaboutcoachingasatoolforculturalchange 18
3.4 Bristol&WestBuildingSocietysalesperformanceindicator 19
3.5 Governmentdepartmentdevelopingtools 21
3.6 DistributionCompany:perceptionsofbehaviouralchange 23
4 AFrameworkforCoachingEvaluation 26
4.1 Keydimensions 26
4.2 Theframework 27
5 ApplyingtheFrameworkinPractice 30
5.1 Overviewofthelocalgovernmentprogramme 30
5.2 Detailedreviewofhowtheframeworkwasapplied 33
5.3 Detailedreviewofevaluationmethodsused 35
5.4 Detailedreviewofpresentingtheevaluationfindings 39
5.5 Applyingtheframeworkforthesecondtime 40
vii
5.6 Applyingtheframeworkforthethirdtime 42
5.7 Lessonslearntaboutcoachingevaluation 43
5.8 Lessonslearnedaboutthedesignofcoachingprogrammes 44
6 Conclusions 46
6.1 Factorshelpingorhinderingevaluationpractice 46
6.2 Summaryoftipsforprogrammeevaluators 49
Bibliography 52
Appendix1:ExampleRecordofObjectives 55
Appendix2:ExampleMidpointTelephoneInterviewDiscussionGuide 58
LEAPDiscussionGuide:Telephoneinterview 58
IndividualOutcomes: 58
OrganisationOutcomes: 59
LEAPprogrammeprocesses: 60
ThankYou 61
Appendix3:ExampleVignette/SuccessStory 63
Appendix4:ExampleEndofprogrammeImpactQuestionnairefor
Participants 65
Appendix5:ExampleEndofprogrammeImpactQuestionnaireforSponsors 71
Appendix6:ExampleEndofprogrammeImpactQuestionnaireforCoaches 75
viii
Executive Summary
Asthecoachingindustrycontinuestogrowandentersthemarketmaturityphaseof
itslifecycle,thepressuretoshowthatcoachingworksandaddsvalueislikelyto
intensify.Coachingevaluationisanactivitythatalmostallcompaniesagreeis
important,butitisonethatisoftenneglectedintheperpetualrushtogetthingsdone.
Thisreportaimstodemystifycoachingevaluationandprovidecoachingprogramme
organiserswiththepracticalhelptheyneedinplanningtheirownevaluations.
Coachingasadevelopmenttoolisincreasinglyprominent.However,manycoaching
programmesareeitherevaluatedsuperficially(ifatall)oronlyatthereactionlevelof
thecoachees.Thereisalsoverylittlenonpartisanadviceoncoachingevaluation.
The IES study

Theaimsofthe20032006researchstudywereto:
examinetheevidenceaboutwhethercoachingisaneffectivetool
exploreissuesinevaluatingcoachinginaworkplacecontextandidentifywhat
factorshelp/hinder
provideillustrationsofhowleadingcompaniesareevaluatingtheircoaching
programmes
developandtestamodelofhowtoevaluatecoachingprogrammes,whichcanbeof
practicalusetocompaniesinplanningtheirownevaluations.
Findings
Theliteratureandourstudysuggestthatforcoachingtosustaincredibilitylevelsof
evaluationneedtoincrease.Suggestionsarisingfororganisationsinplanningyour
evaluationare:
ix
Adaptthetraditionalmodeloftrainingevaluationyouuseelsewhere.Oryoucan
usetheevaluationframeworkpresentedaspartofthisresearch.
Clarifywhytheevaluationisbeingconducted.Areyouseekingtoprove
something,improvesomething,orlearnsomething?
Berealisticaboutconstraints.Clarifyyourbudget,resourcesavailableandanytime
constraints,andconsidertheseinrelationtoyourpurpose.
Definesuccesscriteriabeforechoosingmeasures.
Beselectiveinyourevaluationmeasures.Collectdatatoshowwhethersuccess
criteriahavebeenachieved.Considerlookingforbenefitswellafterthecoaching
hasended.
Considertheperspectivesofdifferentaudiencesfortheevaluationandhowyou
willaccessarangeofviewpoints.
Makesureinadvancethatyourcoachesarewillingtouseyourevaluationtools
whenoperatinginyourorganisation.
Minimiseresistancetotheevaluationbylettingparticipantsandmanagersknow
beforethecoachingstartswhatevaluationmeasureswillbeusedandhowthey
willbeexpectedtocontribute.
Company illustrations
Oursixfeaturedcompaniescomefromavarietyofsectorsandcoveradiverserange
ofapproachestoevaluation.TheyincludeTMobile,Corus,aglobaldistribution
company,abuildingsociety,NHSinWalesandaUKgovernmentdepartment.
Thelessonslearnedfromthecompanyillustrationsaboutusingmethodsfor
collectingevaluationinformationwillbeofinteresttowouldbeprogramme
evaluators,andaresummarisedhere.
Business Results
Ifbottomlinebusinessresultsarewhatyouwant,focusingononekeybusiness
indicatorcanbeasimpleapproachyieldingstraightforwardresults.
Itisbettertoplanhowtoevaluatethecoachingbeforestartingtheprogrammeso
hardbaselinedatacanbecollected.Itisnotalwaysnecessarytocomplicatethingsby
calculatingreturnoninvestment(ROI).Measuringwhereitiseasiestprovides
reasonableevidence,meaningyoucanavoidtheexpenseofmeasuringbenefit
elsewherewhereitmaybedifficulttoidentifyanappropriatemeasure.Ifyoudogo
downtheRoIrouteyouwillneedsignificantfinancialresourcesandstatistical
competenceatyourdisposal.
x
Behavioural change
Multipleviewpointsandmultipledatacollectionmethodsareessentialwhenitcomes
tomeasuringperceptionsofbehaviouralchange.Facetofaceinterviewsenable
behaviourchangetobeexploredinmoredepth,althoughtelephoneinterviewscan
alsogeneratedetailedinformationandallowprobingandcanbealessexpensive
alternative.
Whendesigningimpactquestionnaireslookingatbehaviourchange,makesureyour
ratingsscaleallowsforthepossibilitythatchangesmaybeperceivedasnegative
ratherthanpositive.
Surveys
Attitudesurveysareasimpleandnonresourceintensivemethodtocollectreactions
tocoaching.Climatesurveyscanbeusefulinidentifyingchangesinsoftskillareas
suchascommunication,andareespeciallyrelevanttoorganisationsimplementing
coachingasastyleofmanagement.
Keepsurveyquestionnairesshorttogetabetterresponse.Responserateswillincrease
furtherwithremindersandchasing.Theycanbeimprovedbyaskingcoachesorline
managerstodistributeandcollectquestionnaires.
Becautiousininterpretingsurveyfindingswherethesizeofsurveyedpopulationis
small.
Control groups
Comparingtheresultsamongcoachedwithacontrolgroupofnoncoached
individualscanbeaveryeffectiveapproach.Beingabletocomparebeforecoaching
withaftercoachingresultscanalsobeseenbysomeasmorecrediblethanexamining
postcoachingdataalone.Ifyoudonthaveacontrolyouwillneedsomeformof
benchmarkforcomparativeanalysistoassesswhethertheactivityinquestionis
relativelyeffective.
An evaluation framework
IESproducedaprovisionalframeworkwhichwastestedoveratwoandahalfyear
periodonanindepthevaluationofthreecohortsofalocalgovernmentstrategic
coachingprogrammeforHRexecutives.Theframeworkwasthenrefined.
Figure1summarisesthetwodimensionsweproposeasbeingkeyforwouldbe
evaluators:threemainareasofevidencesoughtandfourmainlikelysourcesof
evidence.
xi
Figure 1: Key dimensions in framework
coach coachee
Individual level
Organisational level
organisation documents

Source: IES, 2006
Threesetsofkeyquestionswereidentifiedrelevanttotheareastoseekevidence
about.Evaluatorsneedtounderstandtheanswersbeforethecoachingbegins:
Whatdocoacheesexpecttogainfromthecoaching?Andhowwillweknowatthe
endifthesebenefitsarerealised?
Whatdoestheorganisationexpecttogainfromthecoaching?Andhowwillwe
knowattheendifthesebenefitsarerealised?
Whatinternalandexternalprocessesneedtobeinplacetoenablethecoaching
programmetodeliverthechangesexpected?Andhowwillweknowiftheyare
workingintimetochangethemiftheyarenotworking?
Theresearchalsoidentifiednumerousperspectivesthatmightberelevantaslikely
sourcesofevidence.However,itseemstherearefourmainsourcesthatitismost
helpfultoconsider:documents,egrecordsofobjectives,achievements,coaching
contracts;coachees;coaches,whetherinternalorexternal;organisationperspective,eg
linemanagers,sponsors,HR,staff.
Thethreemainareasandfourmainsourcescanbepresentedasasimpleframework,
asinFigure2.
The report
Thereportalsocontainsdetailsofhowtousetheframework,plusaselectionof
evaluationtoolsusedbyourearlyadopterorganisation,whichshouldofferideasand
inspirationforthosedevelopingtheirowntools.
xii
Figure 2: A framework for coaching evaluation
Likely Coachees Line Coaches Documents

sources of managers
evidence or
Evidence sponsors
sought at:
Individual
level
Organisation
level
Programme
processes
Source: IES, 2006
xiii

xiv
Institute for Employment Studies 1
1 Introduction
1.1 Background to the research project

Coachingisnowawidelyacceptedmethodforthedevelopmentofpeoplestalentsin
supportofcorporateobjectives.EightypercentoflargecompaniesinNorthAmerica
andtheUKusecoaching.UseisalsogrowingintherestofEuropeandAustralia.
Studieshighlightthepotentialforcoachingtobeeffectiveinincreasingperformance
butevidenceonactualimpactislacking.Despiteverypositiveanecdotalreports,most
organisationsstillhavenohardevidenceabouttheoutcomesfromtheircoaching
investmentseither.
Incompanyknowledgeandexpertiseaboutevaluatingtraininganddevelopment
activitiesingeneralhasbeengrowing.Butbackinearly2003therewasconcern
amongIESNetworkmembersthatonetoonecollaborativelearningmethods,like
coaching,justdidnotsitproperlywithinanyoftheevaluationmodelscurrentlyon
offer.Inaddition,somemembersfelttherewasnoproperguidanceavailabletothem
onhowtoevaluatecoachingwithintheirownorganisations.
TheaimsofthisIESresearchprojectwereto:
examinetheevidenceaboutwhethercoachingisaneffectivetoolforimproving
individualandorganisationalperformance
exploreissuesinevaluatingcoachingandidentifywhatfactorshelpand/orhinder
goodevaluationinaworkplacecontext
provideillustrationsofhowsomeleadingcompaniesareevaluatingtheircoaching
programmes
developamodelofhowtoevaluatecoachingprogrammes,whichcanthenbeof
practicalusetocompaniesinplanningtheirowncoachingevaluation
refinethemodelthroughapplyingitinareallifeworkplacecontext,andexplainit
inastepbystepmannerthatcanbeusedbyIESHRNetworkmembers.
2 Practical Methods for Evaluating Coaching
1.2 Method
Therewerefourmainaspectstotheresearchproject,startinginFebruary2003with
anextensiveexaminationoftheacademicandpractitionerevidencealreadypublished
aboutevaluatingcoaching.Fromthisweconcludedthatcoachingmayneedadifferent
evaluationapproachthanmoretraditionaldevelopmentmethods.
BetweenAprilandSeptember2003wecompletedcompanyillustrationsandpractical
examplesofhoworganisationsareevaluatingcoachingthroughdeskresearchandby
conductingindepthinterviewswitharangeofHRspecialists,coaches,coaching
schemeorganisersandcoacheeswithineightorganisationsfromsevendifferent
employmentsectors.
Fromthiswethendevelopedaprovisionalframeworkandapproachfororganisations
touseinevaluatingtheirowncoachingprovision.
Inordertotestthenewframework,fromNovember2003toApril2006weworked
collaborativelywithoneearlyadopterorganisationindesigninganew,flagship
coachingprogrammeunderpinnedbyacomprehensiveevaluationofthepilot
programmeusingthenewframework.Wethenusedtheframeworktoevaluatetwo
subsequentcohorts.Evaluationmethodsincludedindepthtelephoneinterviews,
impactquestionnairesandfocusgroupsinvolving74coachees,74management
sponsors,fiveexternalcoaches,13internalcoaches,programmeorganisersandother
interestedparties.Fromthiswerefinedthecoachingframeworkanddevelopeda
rangeoftoolsfororganisationstouseinevaluatingtheirowncoachingprovision.
Lessformally,theresearchershavebeenincontactwithtwoprivatesectororganis
ationsthathavealsousedtheprovisionalframework,withIESassistance,duringthe
periodofourformalstudy.
1.3 Structure of this report

Chapter2providesacommentaryontheexistingliteratureandexplorescoaching
elevationissuesinanorganisationalcontext.Inparticular,itasksthequestion:Is
coachinganeffectivetool?Thischapterwillbemostrelevanttothosewithaninterest
inthebroadstrategicissuesandtheorybehindevaluation.Thosereadersanxiousto
getonwithreadingaboutthenittygrittyofpracticemaywishtostartatChapter3
andreturnlatertoChapter2beforefinalisingtheirownevaluationplans.
Chapter3presentssixreallifecompanyexamplesofcoachingevaluationpractice.
Thisshouldgivereadersagoodillustrationofhoworganisationshavetackledtheir
evaluations.Followingeachcompanyexamplewehighlightparticularlearningby
wayoftipsforreaders.
InChapter4weintroducethenewcoachingevaluationframework,whichisfollowed
byadescriptionofacompanyexampleofusingthecoachingframeworkbyourearly
adopterorganisationinChapter5.Thisextendedexampleshouldenablereadersto
getaclearpictureofpreciselywhatthatorganisationexpectedtoachieve,which
evaluationmethodsitused,howitchosetopresentitsfindingsandasummaryof
lessonslearned.
Bywayofconclusion,wepresentasummarydiscussionofthefactorswhichseemto
helpandhindergoodevaluationpracticeandofferourfinaltoptipsforwouldbe
coachingprogrammeevaluators.
Thetitleofthisreportsuggestssomepracticaltoolkittypesolutionswillbeincluded
whichreaderscanextractandusefortheirownpurposes.Theappendicesarewhere
tolookfirstifthisisyourburninginterest.Theycontainaselectionoftheevaluation
toolsusedbyourearlyadopterorganisation.Theyarenotdesignedastemplates
becauseeachwasdevelopedforuseinaspecificcompanyandcoachingcontext.
Whattheydemonstrateistherangeoftoolsinuse,andtheyshouldofferideasand
inspirationfordevelopingyourowntools.
2 Does Coaching Actually Work?
Manyorganisationsfeelthatcoachingisbeneficial,buthavenohardevidenceto
supporttheiranecdotalinformation.Indeed,apreviousIESstudyfoundthatthearea
ofevaluationwasthesinglemostwidespreadsourceofconcerntocommissionersof
executivecoaching(Carter,2001).Theorganisationsinvolvedinthisearlyresearch
werekeenforIEStoassisttheminreviewinganddevelopingapproachesto
evaluatingcoaching.Thesponsorsofthisstudyaskedustoexaminetheexisting
researchevidenceaboutwhethercoachingisaneffectivedevelopmentmethod.
Theaimofthischapteristoexploreissuesinevaluatingcoachingwhicharisefrom
theexistingliterature.Inparticular,weexaminewhethertherearedifferent
considerationsinevaluatingcoachingfromthoseinevaluatingotherdevelopment
tools.
2.1 Is coaching an effective tool?

Arecentreviewofcoachingevaluation(Gray,2005)arguesthatthemanyclaimsfor
effectivenessofcoachingemanatefrominterestedparties,suchasindividualcoaches
orcoachingorganisations.
Thereislimitedevidenceontheimpactofcoachingasatool.Thisisespeciallytrue
whenitcomestotwoapplicationsofcoaching:usinginternalspecialistcoachesand
creatingacoachingstylemanagementcultureusuallyknownaslinemanageras
coach.Someofthepublishedstudiesintheseareashaveseriouslimitationsinthe
designandanalysisofthedatacollected.However,therearesomeinsightfulpapers
particularlyinathird,andlongerestablished,applicationofcoaching:usingexternal
executivecoaches.Eightstudieshavebeenidentifiedthatfocusontheefficacyof
executivecoaching.Theseinclude:
McGovernetal.(2001),whouseKirkpatricksmodelofevaluationtoassessthe
impactofcoachingonreactions,behaviouralchange,organisationaloutcomesand
returnoninvestment(RoI)
Oliveroetal.(1997),whoinvestigatedtheoutcomesofexecutivecoachinginaUS
publicsectoragency
JudgeandCowell(1997),whosurveyedcurrentexecutivecoachingpractices
Genger(1997),whoinvestigatedtheeffectivenessofexecutivecoachingpractices
throughquantitativeandqualitativemethods
Hall,OtazoandHolenbect(1999),whointerviewedexecutivesandcoaches
regardingexecutivecoachingpractice,effectivenessandfuturedirections
FosterandLendle(1996),whoinvestigatedtheeffectsofeyemovementdesensitis
ationreprocessing(EMDR)asatechniqueusedinexecutivecoaching
Laske(1999),whoexploredthetransformativeeffectsofexecutivecoachingonan
executivesprofessionalagenda
Garmanetal.(2000),whoexaminedpublicperceptionsofexecutivecoaching.
Insummary,thesestudiesprovidesomelimitedevidencethatexecutivecoachingis
effectiveinincreasingperformance(McGovernetal.,2001;Oliveroetal.,1997),is
viewedfavourablybyexecutives(Genger,1997)andhasthepotentialtofacilitate
developmentalchange(Laske,1999).However,ascanbeseenbythelist,theyallvary
inwhattheywereassessingintermsofeffectiveness,andallusedifferentapproaches
andmethodsforevaluation.Somehadquasiexperimentalapproachestogathering
informationoncoaching;othersweremoreinformalintheirapproach.Theydo,
however,provideanindicationofthepotentialforcoachingtobeeffectiveincertain
areasbut,apartfromMcGovernetal.(2001),offerlimitedguidanceonhowto
evaluatecoachingwithinorganisations.
Wenowlookatthepracticalissuesinsimilarapproachestocoaching(egmentoring)
inexploringwhatwecanlearnaboutthekeyissuesinevaluatingit.
2.2 What can we learn from mentoring?

Thereareundoubtedsimilaritiesinthenatureofthementoringandcoachingprocess.
Giventhis,therearelikelytobesimilarissueswhenitcomestoevaluation.An
extensiveliteratureexistsonmentoring,primarilybecauseithasbeenaroundfora
longerperiodanditisworthconsideringtheapproachesused.Whatcancoaching
learn?
Unfortunately,therearefewgoodinstancesoftheevaluationofmentoringorcoaching
whichaddresstheevaluationproblems.Gibb(1994)highlightsthefactthatthe
evidencesuggeststhatevaluatingmentoringusuallyconsistsofaskingmentorsto
assesstheusefulnessofthementoringrelationship(GibbandMegginson,1993).
Inothers,thefocusisonaskingthementeestoassessthevalueofthementoringfor
themselves.Whilethismayprovidesomeusefulinformation,itdoesnotamountto
anevaluationthatquantifiestheoverallvalueofmentoring.
Evaluationofmentoringhastendedtofocusontheevaluationofformalmentoring
schemes,particularlyforyoungpeople.Differentaspectsthatareconsideredinclude:
1. mentoringoutcomesidentifyingthenatureandachievementofmentoring
outcomeswhatisthevalueofmentoringinchangingknowledge,skillsor
attitudesofthementee?
2. thecontributionofmentoringtothebroaderinitiative
3. mentoringcosts.
Thesethreeaspectswouldappeartoprovideausefulframeworkbywhichlinkto
traditionalevaluationtheoriesandmethods.Gibb(1994)concludesbyrecognising
thattherearenoreadymademodelsforevaluatingmentoring,butthatconsidering
outcomes,effectsandcostsinaframeworkforevaluationisagoodstartingpoint.
Thesethreeareasareconsideredinthecoachingarenaalongwithanumberofother
issuesthatneedtobeborneinmind.
2.3 What about return on investment approaches?

Inoursearchofthepublishedliteraturewecouldonlyfindonereportedstudythat
hadusedanysystematicapproachbasedonamodelofevaluation.Intheexample
givenbyMcGovernetal.(2001),theyappliedKirkpatricksmodelsystematicallyto
evaluatecoachingatthereaction,learningbehaviouralandorganisationaloutcome
levels.TheyalsoincludedameasureofRoIbyapplyingPhillipswork(1997).In
addition,theydevelopedamethodologyforassessingthefactorscontributingto
coachingeffectivenessandineffectivenessandfoundthatthekeyfactortotheeffective
nesswasthequalityoftherelationshipbetweenthecoachandtheexecutive.
DuringtelephoneinterviewswithresearchersintheMcGovernstudy(2001),
participantswerehelpedtoidentifythetangiblebusinessbenefitsthatresultedfrom
coaching,andtocalculatethemonetaryvalueandtheirlevelofconfidenceinthat
estimate.Theywerealsoaskedtogiveapercentageofthebenefitthattheyattributed
tothecoachingandtheirlevelofconfidenceinthatpercentage.
Oneseniormanagerworkingoninterpersonalskillsandprojectmanagementskills
attributed50percentoftheimprovementsinherworkoutcomestothecoaching.She
was100percentconfidentinthisestimate.Sheestimatedthemonetaryvalueresulting
fromhercoachingtobeUS$215,000andwas90percentconfidentinthisestimate.
ThecostofthecoachingwasUS$15,000.
Adjustedbenefit=US$215,00050%100%90%=US$96,750
US$96 ,750 US$15 ,000

RoI (%) = 100 = 545%
US$15,000
Theresearchersbelievethatthisseniormanagerscompanygotback5.45timesits
investmentinthecoachingofthatoneindividual.
SincetheMcGovernstudy,consultancieshavebeentrippingoverthemselvestosell
companiesformulaeforcalculatingvalueorRoI.Attheirheart,mostofthese
approachessimplyseektohelpyouidentifyasignificantmarginofbenefitovercost:
Benefit () Cost of coaching

RoI (%) = 100
Cost of coaching
Thedifferencebetweenthevariousapproachesnowbeingmarketedishowtoestimate
thebenefitdata.Thisisareallytimeconsuminganddifficulttasktodoproperly.Itis
tediousstuffifyouarenotanumberspersonbutexcitingstuffifyouare!Isolating
causeandeffectinanytraininginterventionisalwaysdifficult,butevenmoresoif
thegoalsarecomplexandnoteasilyquantifiableastheyoftenareincoaching.Other
thingsthatarehappeningintheorganisationortotheindividualcanalsoinfluence
outcomes.
Asthecoachingindustrycontinuestogrowand,intheUSatleast,entersthemarket
maturityphaseofitslifecycle(MaherandPomerantz,2003)thepressureoncoaching
supplierstoprovethatcoachingworksandtoaddadditionalvalueislikelytointensify.
Thismayinpartexplainwhyauthors,professionalassociationsandcoachesareso
keentopersuaderesearchfoundationsandcompaniestoinvestinstudieswhich
documentthebenefitsofcoachinginvestmentbyquantifyinganRoI.Itiscertainlyin
thecoachingindustrysinterestsforanRoItobedemonstrated.However,IESResearch
NetworkmemberswespoketoadviserealcautioninusingthesekindsofRoIformulae
asthecoststoacompanyofanRoIstudymightoutweightheknowledgegainedby
thatcompany.
PersonalbeliefsarealsoafactoringoingdowntheRoIroute.Someboardmembers
willjustnotacceptthereliabilityoftheresultsbecause,asanystudentoffinancewill
tellyou,theyareonlyasgoodasthequalityofthesubjectivedatathatisfedinto
theminthefirstplace.Ofcourseyoucouldarguethatnotallperformanceindicators
aresubjectivelymeasuredbutbearinmindthattheirveryselectionmaybeviewedby
othersassubjective.
PersonalbeliefsaboutRoIaside,youcouldusebucketloadsoftimeandeffortonlyto
havetheresultthrownoutasawastebecausepeoplecanseewiththeirowneyes
improvementstosomeoneelsesskillset.Forothersthedegreeofmeasurement
capabilityrequiredisimpressiveandthisinitselfraisesthecredibilityofthetraining
function.ItsuptoeachcompanytoconsidertheprosandconsofRoIinreachingits
owndecisiononusingthiskindofapproach(seeTable2.1).
Table 2.1: Advantages and disadvantages of RoI Approaches
Advantages of RoI Disadvantages of RoI

applies a monetary value to business benefits can be time consuming and resource intensive
encourages participants and coaches to think still relies on subjective judgement
systematically about the business focus to their coaching
knowing organisational value is being measured may need to train participants in evaluation
encourage commitment from all parties guidelines
need to overcome participant resistance to
sharing their specific results
Source: IES, 2006
2.4 Issues in evaluating coaching

Wehaveconsiderableknowledgeandunderstandingofthetheoryofevaluationin
traininganddevelopment.Anydiscussionofevaluationwithinthiscontextdoesnot
developtoofarbeforemodelssuchasKirkpatricks(1997)areintroduced(Figure2.1).
Figure 2.1: Four levels of evaluation
Level 4
Organisational
performance
Level 3
Behavioural change
Level 2
Learning
Level 1
Reaction

Source: Kirkpatrick, 1996
Inreality,veryfeworganisationssystematicallyevaluatetraditionalformsoftraining
anddevelopment,sowhytheconcernoverevaluatingcoaching?Mostevaluationof
managementdevelopmenttakesplaceatthereactionlevel(level1)ofKirkpatricks
modelofevaluation(Tamkinetal.,2002)withoutmucheffortplacedatthelearning
orbehaviouralchangelevels.Ifthisisthenormofevaluationforotherformsof
traininganddevelopment,isthislevelofapproachalsoappropriatetocoaching
evaluation?Inpractice,andfromdiscussingtheissueswithIESResearchNetwork
members,thereseemstobearequirementandneedtobeabletodemonstratethe
contributionofcoachinginmoredetail,eitherattheleveloflearninggainedor
outcomesachieved,individuallyororganisationally.Asalessestablishedformof
developmentthattraininganddevelopmentdepartmentsareembarkingon,itismore
likelytobescrutinisedintermsofitsimpactondevelopmentthantraditionalformsof
developmentthatarewellestablished.Coachingisalsoconsideredarelatively
expensiveintervention.Traditionalformsofdevelopmentmayhavelimitedevidence
ofimpactonindividualandorganisationalbehaviourchange,buttheyaremore
acceptedandlesslikelytobechallengedintermsoftheirvalue.Anattemptat
addressingkeyevaluationissuesincoachingis,therefore,importantinestablishing
itscredibilityasasuccessfulintervention.Ifwetakeexistingmodelssuchas
Kirkpatrickandapplytocoaching,whatarethekeyissuesthatemerge?
2.4.1 Defining outcomes and evaluation measures
Evaluatingcoachingpresentsanumberofproblemsfortraditionalmodels.Coaching
isoftenlessformalthantraininginterventionsbasedononetoonesessionswhich
maynotberelatedtoanyspecificprogramme.Aparticularissuehereisthatcoaching
maystartwithoutformalaims,objectivesoroutcomeswhichwouldalmostcertainly
becentraltoanyformaltrainingcourse.Thislackofclearaimsandobjectivesatthe
startoftheprocesscausesrealdifficultiesforevaluationofthatprocess.Defining
outcomesofaninterventionindevelopmentandtrainingisthecornerstonetomost
evaluationmodelsasitprovidesananchorforyourevaluationmeasures(Figure2.2).
Thisinitselfisthesinglebiggestdifferenceoftheevaluationofcoachingfromother
approaches.
Thecredibilityofcoachingdependsonbeingabletohelpachievesetoutcomesmade
inthecoachingprocess.Yetthenatureofcoachingmeansthatitoperatesatavery
individuallevelwheredifferentpeoplewillgetdifferentthingsoutofit.Where
traditionalapproachestotrainingaimtohavealevelplayingfieldbytheendofthe
session,whereeveryoneknowsthesamethingtothesamelevel,coachingis
individuallyfocused.Thishasamajorimpactonevaluation,inthatnosetabsolute
criteriaforoutcomescanbeestablishedacrossanorganisation.However,thisdoes
notmeanthatnovaluationistakenascriteriacanbesetwithinabroaderframework.
Publishedaccountsofthecoachingprocessandbestpracticeguidancesuggesta
numberofcommonprocessesinvolvedincoaching,mostofwhichactuallydoinclude
settinggoalsorreachingasharedunderstandingofwhatwillbediscussed.Someof
thekeyfeaturesofcoachingarethatitis:goalspecific;actionandperformance
oriented;andobjective.Theseelementsalllendthemselvestosomeelementof
measurementframeworkandhenceevaluation.However,thesegoalsandobjectives
emergeaspartoftheprocess,ratherthanexistingatthebeginningaspartofthe
course.Datacollectedfrom60coachesbyGray(2005)indicatedthatindeedmost
Figure 2.2: Summary of traditional evaluation approaches
Identify
success criteria
Select
evaluation methods
Demonstrate
outcomes through
collecting evidence

Source: IES, 2006
coachesdodiscussaclearsetofgoalswithcoacheesattheoutsetofthecoaching
process.Oncegoalsaremadewithinthecoachingprocessthesethenleadtodefined
successcriteria(Carter,2001),whichcouldandshouldbelinkedtoevaluation
measures.Howthismightoperateinpracticeisexploredfurtherinthefinalsection
butthekeywouldbetodecide:
Whataretheseelementsorsuccesscriteriaandoutcomes?
Whichoftheseelementsaremeasurable?When?Bywhom?
Whatistheoverallvaluetotheindividualandtheorganisation?
2.4.2 Evaluating the bigger picture
Standalonecoachingsessionsaredifficulttoevaluateastheydonotappearatfirst
glancetolinktoanyparticularindividualoutcomeororganisationalcontribution.
Wherecoachingisoneaspectofamoretraditionalinterventione.g.leadership
developmentorculturalchangeprogrammesthatarebeingformallyevaluated,
outcomeswillhavebeenarticulatedfortheprogrammeandtheremaybesomeway
oflinkingevaluationtotheseoutcomes.Whiletheeffectonanindividualthroughself
reportmaybepositivethestrategicvaluefororganisationsaswellasindividuals
needstobeanswered.
Researchonevaluationofmentoringsuggeststhatdeterminingthecontributionof
mentoringtoachievingtheobjectivesofbroaderinitiativesisaproblem(Wilsonand
Elman,1990).Determininganddistinguishingtheeffectsofcoachingisdifficultin
itselfthentolinktheseeffectstotheachievementofspecificobjectivesinabroader
contextisevenmoredifficult.Manyotherfactorsalsoinfluenceprogresstowards
objectives.Althoughnodirectlinkmade,itispossibletoclarifytheexpectedlinks
betweencoachingandtheachievementofobjectiveswithinthebroaderinitiatives.
Forexample,relatingoutcomestothebroaderinitiativeorasMcGovernetal.(2001)
havedonebydemonstratingthechainofimpact.However,thequestionstillremains:
howdoyouevaluatethecoachingelement,inparticularfromtheorganisational
perspectiveratherthantheindividualsorcoachs?Thisleadsustoconsiderfrom
whoseperspectivetheevaluationistakingplace
2.4.3 Perspectives on evaluating outcomes

Figure 2.3: Adapting traditional evaluation for coaching
Coachee
perspective
Reaching a shared
Organisation
understanding
perspective
with coach
Identify
success criteria
Select
evaluation methods
Demonstrate
outcomes

Source: IES 2006
Theobjectivesandaimsoftheevaluation,particularlyintermsofwhichoutcomesare
measured,willalsodifferaccordingtothevariousstakeholdersinvolvedintheprocess.
Forexample,theorganisationmaywantanyevaluationtodemonstratetheimpactof
thecoachesonwhateverinitiativetheywerebroughtintoinfluence(egleadership,
culturalchange).Individuals(ontheotherhand)mayhaveadifferentsetofcriteria
withwhichtheyevaluatethesuccessofthecoaching,possiblyrelatedtothecriteria
agreeduponinthecoachingsessionratherthanthebroaderorganisationaims.While
theymayoftenbethesame,theymaynotalwaysbe.Anyevaluationneedstobe
explicitaboutwhosecriteriaandoutcomesitismeasuringand,wheresuccesscriteria
aredifferent,thatbothperspectivesareevaluated(seeFigure2.3).Thesameexample
maybegivenforanexternalcoachingcompanywhomaywanttodemonstratethe
influenceoftheircoachesonanorganisationinitiative,inadditiontoachieving
individualoutcomesandsuccesscriteria.Theytoowouldneedtoemployan
evaluationtoolthatevaluatesfrombothperspectives.Again,thisiswherecoaching
differs,asitmayhavedualobjectivesfromindividualandorganisationalperspective
which,giventhesetobjectivesintraditionaltraining,islesslikelytooccur.
2.4.4 Timing of evaluation and availability of information
Twootherissuesarisewherecoachingevaluationisdifferentfromtraditionalforms.
Thefirstisthetimingoftheevaluationandhowitaffectstheinformationgatheredon
theeffectivenessofthecoaching.Evaluationdirectlyafterthecoachingexperience
maynotbeappropriateasthebenefitsmayonlyaccruelateronwhenthecoachor
personcoachedfullyreflectsonthebenefitsthattherelationshiphasbroughtthem.In
thissenseitcanbeseenassimilartoevaluatingcareersguidance.Althoughthisis
truealsoofotherformsoftraining,wherebyyoumaynotgettouseyourskillsuntila
laterdate,itismorelikelythatyouwillbeabletoanticipateorrecognisewhere
trainingwillhelp.Withcoaching,theoutcomesofwhatislearntordiscussedmaynot
beimmediatelyobvious,eithertothecoachortothosecoached,unlesstheyarelinked
intoawiderprogramme(e.g.leadershipdevelopment,culturechange).Asthedefined
successcriteriamaynotbeagreeduponuntilthecoachingprocesshasbegun,itis
difficulttomeasurepreandpostcoachinginterventiontoassesstheimpact.The
secondissueisthatinformationwithwhichtoevaluatemaynotbereadilyaccessible
inacoachingsituation(egbecauseoftheconfidentialnatureofthecoachingcontext).
2.4.5 Qualities of the coach and coaching relationship
Afurthersignificantissueisthecontributionofthequalityandabilitiesofthecoach
totheoutcomes.Themoderatingeffectofthequalityandskillsofthecoachisperceived
tobemoreimportantaninfluenceincoachingthantheroleoftrainerinatraditional
trainingevent.Thisisbecauseofthesuggestionthattheeffectivenessofanycoaching
interventionisprimarilydeterminedbytheabilityofthecoachwhatwasitabout
themortherelationshipthatbroughtaboutthechangesandoutcomesobserved?
Therefore,anyevaluationmechanismwouldneedtoevaluatenotonlythechangesin
outcomevariables,orsuccesscriteriaasaresultofcoaching,butalsotakeameasure
ofthenatureoftherelationshipthathelpedtobringaboutthosechanges(Figure2.4).
Figure 2.4: Enabling factors to achieving success in coaching
Coaching
relationship
Coach Coachee
skills motivation to learn
ability learning style
style abilities
personality
support systems

Source: IES, 2006
Notallemployersmaychoosetotakeonthisadditionalperspective,althoughitcould
beusefulinidentifyinganycorporateenablersorbarrierswhichmaybehelpingor
gettinginthewayofachievingdesiredresults.Onceidentifiedthesebarrierscanbe
removedbytheorganisation.
2.5 Summary of suggestions arising from the literature

Coachingasadevelopmenttoolisbecomingincreasinglyprominent.However,many
coachingprogrammesareeitherevaluatedsuperficially(ifatall)oronlyevaluatedat
thereactionlevelieofthecoacheestotheprogramme.Thischaptersuggeststhatfor
coachingtosustaincredibility,theorganisation:
adaptsthetraditionalmodelsoftrainingevaluationitmayuseelsewhere
triestoavoidunnecessarilycomplicatedRoIformulaeunlessithassignificant
financialresourcesandstatisticalcompetenceatitsdisposal
considerstheperspectiveofdifferentaudiencesfortheevaluationandhowitwill
accessarangeofviewpoints
definessuccesscriteriabeforeselectingmeasures
selectsevaluationmeasuresthatcollectdataaboutwhethersuccesscriteriahave
beenachieved
getsfeedbackonenablingfactorsitcanchangenexttimee.g.styleofcoaches
selected.
considerslookingforbenefitswellafterthecoachingexperienceshaveended.
3 Organisational Examples of Evaluation

Practice
Ourstudysponsorsaskedustoprovidesomeillustrationsofhoworganisationsare
actuallyevaluatingtheircoachingprogrammes.Despitetheproblemsanddifficulties
facedbyorganisationsasoutlinedinthepreviouschapterinourstudywedid
encountermanyorganisationalexamplesofplannedactivitiesdesignedtoassistthe
organisationinmakingjudgementsabouthowthecoachingwasgoing,andtomake
decisionsaboutfuturecoachingprovisioninthelightofthesejudgements.
Inthischapterwepresentanumberofdifferentexamplesofappliedevaluationand
ourthoughtsortipsonwhatwecanlearnfromthem.Fromamongalltheexamples
weencountered,oursixchosencorporateillustrationsarenotpresentedtoreadersas
exemplarsofbestpractice,butratherasdiverseexamplesofpracticalapproaches
usedbyorganisations.
Thecompanyillustrationsonthefollowingpagescomefromarangeofsectorsand
coverarangeofdiverseapproachestoevaluation:
Tmobile,whousedattitudesurveysofmanagersandtheirstafftodetermine
priorityareasforfuturecoachingsupporttofurtherprogressthedevelopmentofa
coachingstyleofmanagement
Corus,whodevelopedstandardisedevaluationdocumentationforusebyexternal
executivecoachesandcoacheestoimproveconsistencyintheorganisations
administrationandtrackingofcoachingprocesses,andachievementagainst
objectives
NHSinWales,whosoughttounderstandwhether(andhowbest)external
coachingcouldbeimplementedtoaccelerateorganisationalculturechange
Bristol&WestBuildingSociety,whomeasuredimpactonsalesperformance
measurebetweengroupsoffinancialplanners,onlysomeofwhomreceived
coaching(fromSherwood,2004)
acentralgovernmentdepartment,whowereinitiallydisappointednottofinda
suitablehardbusinessmeasure.Insteadtheypiloteddocumentaryanalysisand
reactionquestionnairestotesttheeffectivenessoftheirongoingprogrammeandto
improvethedegreeofbehaviourchangeamongmanagers
aglobaldistributioncompanywhocomparedperceptionsofthedegreeof
behaviouralchangeusingpreandpostcoachingattitudesurveys.
3.1 T-mobile identifying priority areas for future
3.1.1 Context of the coaching programme
ThetelecommunicationscompanystartedrollingoutaprogrammecalledCoachto
InspirebackinAugust2001,designedtoenhancetheskillsofalllinemanages.The
programmeconsistedofsixjobrelatedcoachingmodulesegCoachingbyExample
andPlanningaCoachingSession.
3.1.2 Purpose of the evaluation
AninitialevaluationinApril2002aimedto:
assessreactionstotheprogramme
determinewhateffecttheprogrammehadmade
identifyadditionalhelpneededtosupportlinemanagersintheirroleasacoach.
3.1.3 Evaluation methods
Twentyfiveteammanagersand100customerservicestaffpersiteweresurveyed.
Respondentswereaskedtorespondtoaseriesofshortstatementsusingascaleof
stronglyagree,agree,disagree,stronglydisagree,dontknowornotapplicable.
Examplesofstatementsformanagersinclude:
Coachingispartofmydailyroutine.
Icoachirrespectiveofmyworkloads.
Ikeepuptodaterecordsofmycoachingactivities.
Thedevelopmentprovidedhashelpedmeimproveasacoach.
ThecompanyprovidesanenvironmentthatenablesmetocoachwhenIneedto.
Examplesofstatementsforstaffinclude:
Ibenefitfromcoachingatleastonceaweek.
Mylinemanagehasapositiveattitudetocoaching.
Mylinemanagercoachesmeirrespectiveofourworkloads.
ThewayIamcoachedgivesmeafeelingofsatisfactionandpride.
ThecoachingIreceivehelpsmeservemycustomerbetter.
Managerswereaskedanopenquestioninvitingsuggestionsforfurtherdevelopment
theywouldliketoimprovetheirperformanceasacoach.
3.1.4 Findings
Resultsshowedthat96percentofstaffrespondentsagreedtheyunderstoodthat
coachingwasimportantinenablingthecompanytomeetitsfuturevisionand85per
centagreedthattheirlinemanagerhadapositiveapproachtocoaching.Ninetyone
percentofthemanagerrespondentsagreedthatthedevelopmenthadhelpedthem
improveasacoach,butasyetamajoritydisagreedthatthecompanywasproviding
anenvironmentwhichenabledthemtocoachwhentheyneededto.Thiswashelpful
tothecompanyinprovidingaclearpointerofwherenexttofocusitsefforts.
3.1.5 Tips from T-mobile illustration
Attitude surveys are a simple and non-resource-intensive method to collect reactions to coaching.
Keep your survey questionnaire short like T-mobile did to get a better response.
Response rates will increase further with reminders and chasing.
3.2 Corus tracking achievement of objectives
Despiteitslongtraditionofinvestmentinmanagementdevelopmentprogrammes,
includingMBAs,inthepasttheDutchUKsteelmakerCorushadpossiblyfocused
moreresourcesonyoungtalentdevelopmentratherthanitskeyseniormanagers.So
in2003,Coruspilotedacoachingprogrammeforpersonaldevelopmentinvolving20
seniormanagersfromitsEuropeandivisionandthreeexternalcoaches.Sixinitialone
toonecoachingsessionsinthepilotweresupplementedbyastartupresidential
miniconferencewithallcoachees.Therewasalsoaonedaywrapupmeeting.
Theevaluationhadtwoaspects.First,toconsiderwhethercoachingasaprimary
deliverymechanismforopeninguppersonaldevelopmentwasanacceptable
approachforthiskeygroupofseniormanagers.Thesecondaspectwaswhetherthe
collectivepartoftheapproachwouldhaveanextraaddedvalueontopofthewidely
appliedindividualapproachofexecutivecoaching.
Akeyaspectoftheapproachwastogainagreementwiththeexternalcoaches(from
differentcoachingcompanies)abouttheneedtousestandardiseddocumentationto
collectdataaboutoutcomeswhichwouldbeusefultothecompanyformonitoringthe
process.Duringtheearlycoachingsessionsacoachingcontractincludinggoalsand
successcriteriawaswrittenbyeachseniormanagerandhis/hercoach.Partofthe
coachsjobwastohelpindividualstranslatetheirowncriteriaintoevaluation
measuresandtoagreemechanismstocollectevidenceofachievement.Thesetoowere
includedinthecoachingcontracts.Themainpurposewastosupportselfmanaged
development.
Telephoneinterviewswerealsoundertakenbythemanagementtrainingmanager
withtheseniormanagercoacheesandameetingwasheldwiththecoaches.These
tendedtocoverqualityassurancemonitoringissuessuchas:thenumberofmeetings
heldandtheirvenues;lengthofsessions;qualityofdiscussions;mainlearningsofar.
3.2.4 Findings
Feedbackfromseniormanagercoacheesthroughinterviewsandquestionnaires
indicatedthatcoachingasadeliverymechanismwaswellreceivedbythisgroup.
Gooduseofprecioustimeandthehighdegreeoftailoringtopersonallearningwere
particularlyhighlyvaluedaspectsofthecoaching.Theselfreporteddatainthe
coachingcontractallowedtheindividualstomanagetheirowndevelopmentbetter
alsoafterthecoachingactivityhadendedThecompanynotedtherichnessof
collectingindividualstoriesthroughthetelephoneinterviews.Withoutthese,the
companywouldhavebeenabletoidentifywhatwasworking,butnottowhatdegree
orwhy.
3.2.5 Tips from Corus illustration
Dont underestimate the power of a collective coaching activity. For Corus it brought to light the
degree of alignment between personal development and corporate culture.
Telephone interviews can provide detailed information and allow probing, and can be a less
expensive alternative to face to face interviews or focus groups.
Some coaches are attached to their usual evaluation methods so make sure in advance that your
coaches are willing to use your evaluation tools when operating in your company.
3.3 NHS in Wales learning about coaching as a tool for

cultural change
FollowingtheintroductionoftheWelshNationalAssemblyandotherchangestothe
politicallandscape,theCentreforNHSLeadershipinWalesidentifiedthatthetime
wasrighttobrokeradevelopmentprogrammeforCEOsanddirectorsacrossthe
publichealthcaresector.Theprogrammeaimedtokickstartthecreationofamore
innovativeorganisationalculturethatvaluednewideas,newthinkingandmanaged
risktaking.Twelvecoacheswithagoodknowledgeofthesector,selectedfromfive
differentexternalcoachingcompanies,providedfouronetoonecoachingsessions.
ThirtytwoCEOsanddirectorsparticipatedascoachees.
TheTavistockInstitutewasaskedtoconductevaluationonbehalfofalltheprogramme
sponsorsandpartners.Theevaluationapproachfocusedontwoquestions:
1. Hasthecoachinginterventiontakenplaceintheintendedway?
2. Whatcantheorganisationslearnfromtheintervention?
Avarietyofevaluationmethodswereused,manyofwhichwerebasedonself
reportbycoachees.Firstly,allcoacheeswereaskedtoprovidesomebaselinedataat
thestart.Thisincludedinformationabout:
thebiggestchallengestheyfacedintheirrole
thebiggestworkplaceissues
whichnetworkstheywerealreadyinvolvedin.
Aftereachcoachingsessioneverycoacheewasaskedtocompleteascoresheetto
generatedatausefulforqualityassurancepurposes.Aspectsscoredincludedtiming,
qualityofcoach/coacheerelationshipandcontent.Coacheeswerealsoaskedto
identifythemosthelpfulelementineachsession.Aftertheirfinalsession,allcoachees
wereaskedtocompleteaquestionnaireonmoregeneralissuessuchas:
motivationtojoiningprogramme
criteriausedinselectingwhichcoachtoworkwith
particularlyhelpfulmodels/inputs
whattheyhavedonedifferentlyasaresult.
Sixofthe32coacheeswerealsointerviewedfacetoface,toallowsomeoftheissuesto
beexploredinmoredetail.
Theperceptionsofcoacheswerealsocollected.Theywereaskedattheoutsetwhat
issuestheyimaginedmightbekey.Theideawastogobacktothecoachesagainafter
thefinalsessiontocomparethesewiththeactualissuesarising.Finally,
coach/coacheepairswereaskedtowritealearningvignettecoveringwhattheywere
workingonandhowsuccessfulthecoachingrelationshiphadproved.
3.3.4 Findings
Thefindingsrevealedthattheopportunityofferedbythecoachinghaddefinitely
beenperceivedandutilisedasamanagementlearningtool,ratherthanapersonal
developmentone.Coaching,asalearningmethod,wasalsoviewedpositively.
Coacheesvaluedtheindividualcareandattention,andtheacquisitionofeasyto
applymodels.Theprogrammewasalsothoughtparticularlytimelyintermsofinter
professionallearningandhadapositiveeffectonthepersonalmoraleandmotivation
ofcoachees.Theevaluationalsoraisedsomequestions.Theimplicationsofusing
knowncoacheswhowereformerlysenioremployeeswithinthesectormight
havesentmixedmessagestocoacheesiftheprogrammewasaboutretention.Coachee
commentsfromtheinterviewssuggeststhatusingrolemodelswhohaveleftan
organisationcanbeseenassomethingofaparadox.
3.3.5 Tips from NHS in Wales illustration
Response rates from feedback sheets or reaction questionnaires can be improved by asking
coaches or line managers to distribute and collect them.
It is difficult to evaluate the potential of coaching for cultural or other systemic change.
Face to face interviews enable behaviour change to be explored in more depth.
The production of learning vignettes after the final coaching session can serve as a useful
reminder to coachees of the need to apply learning in the workplace.
3.4 Bristol & West Building Society sales performance

indicator
InJune2003theheadoftrainingandrecruitmentatfinancialservicesproviderBristol
&WestBuildingSocietywasconsideringsupplementingexistingtrainingprogrammes
withonetoonefollowupcoachingsessionsaftertraininghadbeendelivered.As
wellashelpingtotransfernewlylearntskillsintotheworkplace,itwashopedthis
wouldsupportthecreationofacoachingculturethroughouttheorganisation.
Beforeproceedingthecompanydecidedthebestapproachwastoquantifythelikely
effectivenessbyrunningwhattheycalledatestpilot.Theperformanceofthesales
forcewasselectedasthebestoutputmeasureforasmallscaletestpilot.Theselling
skillstrainingcoursewastargetedasthebestinputactivitytoresearch.Theevaluation
strategywasdesignedbytheorganisationinpartnershipwiththeexternalcoaches.
3.4.3 Evaluation method
Thesalesperformanceof24financialplanningmanagerswasfollowedoverasix
monthperiod:foreachplanner,salesfiguresfromthethreemonthsfollowingthe
trainingcourse.Twelvemanagersattendedthecourse;sixhadnoexternalcoaching
butdidreceivetheusualfollowupsupportfromtheirlinemanager.Theremaining
12managersactedasacontrolgroupastheydidnotattendthetrainingcourseand
thereforehadnofollowupsupportfromacoachortheirmanagers.
3.4.4 Findings
Forthecontrolgroup,salesgrewbyfourpercent.Fromthis,thecompanyconcluded
that,haditnotdonetrainingorcoachingatall,otherinternalorexternalfactors
suchasmarketconditionsmeantbusinesswouldhaveincreasedbyfourpercent
anyway.Forthetrainingonlygroup,salesincreasedbyeightpercent,doublethe
increaseofthecontrolgroup.Forthegroupwhoreceivedtrainingpluscoachingsales
increasedby27percent.Thedramaticresultsledtheorganisationtovaluetheir
trainingprogramme,aswellasemphasisehowimportantsomekindofeffective
followupregimeistoembedlearning.
3.4.5 Tips from Bristol & West illustration
Focusing on one key business indicator was a simple pragmatic idea that yielded easy to
understand results.
There is no need to over-complicate things by calculating RoI when everyone can clearly see that
the increase in sales exceeded the cost of the coaching.
Measuring where it is easiest provides reasonable evidence, meaning you can avoid the expense
of measuring benefit for other groups coached where it might be less easy to identify an
appropriate measure.
Comparing the results among coached and non-coached can be a very effective approach.
Designing evaluation in partnership with external coaches before the coaching begins can work well.
3.5 Government department developing tools
During2003alargegovernmentdepartmentlaunchedamajorseniormanagement
andleadershipdevelopmentprogrammecoveringall300ofitsseniorcivilservants
andstaffmanagers.Avarietyoflearningmethodswasused,including360degree
feedback,sharedcrossorganisationallearninggroupsandoptionalworkshops/action
learningsets.Anexternalcoachprovidedthecriticalelementunderpinningthe
programme.Thefinalcohortsofparticipantsonthetwelvemonthprogrammewere
notexpectedtofinishuntil2005butbyMarch2004theearlycohortswereoverhalf
waythroughtheirprogrammes.
Eventhoughtheprogrammewasstillbeingrolledout,giventheresourcesbeing
investedintheprogramme,theorganisationwaskeentoactearlytoensurethatthe
programmehadagoodbusinessfocusandthatparticipantswerebeingsufficiently
supportedinmakingtheconnectionbetweentheirpersonaldevelopmentand
businessoutcomesrequired.
Qualityassuranceandreviewprocesseswerebuiltintotheprogrammefromthe
outsetbutnotevaluation.DuringMarchtoJuly2004IESconductedanexploratory
studywiththeaimofextendingandrefiningtheorganisationsthinkingin
constructinganevaluationstrategy.Theapproachagreedincludedworkingin
collaborationwithtwodirectoratesindevelopingandpilotingevaluationtools
suitableforfutureusebyotherdirectoratesinidentifyingimpactatindividualand
organisationlevels.
Anonymisedprogrammedocumentationwasanalysedfrom88participantsto
determinewhatkindsofindividualobjectivesparticipantswereworkingon.
Documentarysourcesincludeddevelopmentprofiles,actionplans,reaction
questionnairesandfocusgroupdatageneratedaspartofthesixmonthprogress
checkworkshop.
Inordertogetaclearviewofleadershipcapabilityrequirementsandobjectivesfor
theprogrammefromtheperspectiveofthedirectorates,organisationdocumentation
wasreviewed,directorsinterviewedandafocusgroupofstaffheld.Documentary
sourcesincluded:directoratebusinessplans,organisationscorecard,recentstaff
surveyresults,managementinformationandservicedeliverydataandskillsprofile.
Unfortunatelythedepartmentwasunabletodetermineasufficientconnection
betweenexistingbusinessmeasuresandtheobjectivesparticipantsactuallyappeared
tobeworkingon.Thismeanttherewasnopossibilityofusinghardbusiness
measuresattheinterimstage.Thereforetheprimaryfocusofthepilotevaluationwas
thedevelopmentandtestingofsoftmeasures.
Aseriesofimpactquestionnaireswasdevelopedtoassessperceptionsfrom
participants,theirlinemanagersandstaffofbehaviouralandperformancechanges.
Theselfassessmentparticipantquestionnairesweretheonlyonesofthese
questionnairesactuallydistributedinthepilot,goingtoallparticipantswithinthe
directorates.Issuesattheindividuallevelexploredbytheimpactquestionnaire
included:
selfawareness
teamdevelopment
motivation
personaleffectivenessandbehaviours
strategicskills
individualperformanceinservicedelivery.
Issuesattheorganisationlevelexploredbytheimpactquestionnaireincluded:
corporateleadershipeffectiveness
organisationculture
corporatereputation
communications
directorateperformanceinservicedelivery.
Asmallsampleofnineparticipantsontheearlycohortswasalsointerviewedindepth.
3.5.4 Findings
Theprogrammeappearedtobeworkingwellintermsofsecuringbehaviouralchange
attheindividuallevel.Forthoseleadersinnewrolesorwithcomplexnewtasksthe
programmeappearedparticularlybeneficial.Specificindividualoutcomesincluded
implementingtechniquesforlongtermplanning,moreproactivityinseeking
customerfeedbackandincreasedpersonalcapacitytoleadchange.
Thepilotevaluationwasnotabletodemonstrateimpactattheorganisationlevel.This
didnotmeanthattheprogrammewasfailing;itmaywellhavebeenthatthemidway
pointwastooearlytoidentifysuchanimpact.Thepilotdid,however,demonstrate
theusefulnessoftheimpactquestionnaireapproachandhowtheycouldbeusedto
linktofuturestaffsurveyexercises.Suchanearlypilotalsoallowedactionstobe
identifiedtofurtherstrengthenthebusinessconnectionsintotheprogrammeforthe
majorityofcohortsstilltocome.Theevaluationfindingsidentifiedactionsasneeded
infiveareas:
briefingallcoachesandgivingthemresponsibilityforensuringfutureparticipants
objectivessupportbusinesspriorities
usingrealbusinessimprovementtasksasprogrammetasks
introducingaformallinkbetweenactionplansdevelopedontheprogrammeand
annualpersonaldevelopmentplans
introducingsupportforparticipantslinemanagersandensuretheyareclearwhat
theirroleisinsupportingparticipants
encouragingsharingofdevelopmentneedsandlearningwithinlocalmanagement
teams.
Aftertheinterimevaluationwasfinishedtheorganisationsbusinessmeasureshad
beenupdatedandtheopportunitywastakentousetheseforevaluationpurposesto
coverallfuturecohortsoftheprogramme.
3.5.5 Tips from government department illustration
It would have been better to plan how to evaluate the coaching before starting the programme so
hard baseline data could have been collected.
Minimise resistance and increase response rates by letting all parties know before the coaching
starts what evaluation measures will be used and how they will be expected to contribute.
Be cautious in interpreting survey findings where the size of surveyed population is small.
Transfer of learning back into the workplace can be assisted by line manager involvement
throughout, ie discussing action plans, supporting implementation of goals and encouraging team
members to share objectives.
Effective evaluation is not just a one-off activity at the end; it can be really useful as an interim
stock-take of necessary improvements needed to a programme.
3.6 Distribution Company: perceptions of behavioural change
Aglobaldistributioncompanyintroducedacoachingprogrammeforlinemanagers
intheexpectationofimprovingstandardsofpeoplemanagementandmanager
behaviourintheworkplace.
Acomprehensiveinternalevaluationwasundertakenin2002inviewoftheheavy
investmentmadeincoachingandotherinitiatives.Asprogrammesuccesscriteria
focusedonachievingchangestomanagerbehaviour,theevaluationinvolved
collectingperceptionsofvariouspartiestodeterminewhethertherequiredbehaviour
changeswerehappening.
Preandpostcoachingmeasuresweretaken,includingrelevantstatementinstaff
attitudesurveyssenttoallstaffandmanagers.Attitudestatementsforstaffinboth
thebeforecoachingandaftercoachingattitudesurveysincludeditemssuchas:
Ireceiveregularandvaluablefeedbackfrommymanager.
Mymanagerencouragesmetodevelopmyskillsandknowledge.
Mymanagerconsultsmebeforemakingdecisionsthataffectme.
Facetofaceinterviewswerealsoundertakenbyinhousetrainingspecialistswitha
sampleofmanagers,theirbosses,staffandtheircoaches.
3.6.4 Findings
Interestingly,therewasagreementfromallonthosebehaviourswhereapositiveor
negativechange(inbehaviour)hadoccurred.Therewasalsoagreementaboutthe
extentofchangeamongallgroups,exceptamongtheindividualmanagers
themselves.Individualmanagersconsistentlyratedtheirbehaviouralchangeas
considerablygreaterthananybodyelsedid.Fromthisthecompanyinferredthatitis
importanttoseekfeedbackfrommorepeoplethanjustcoachees,otherwisethereisa
riskthatthecompanymaygetanexaggeratedlyrosypictureofbehaviouralchange.
3.6.5 Tips from Illustration
Multiple viewpoints and multiple data collection methods are essential when it comes to
measuring perceptions of behavioural change.
Being able to compare before coaching and after coaching results can be seen as more credible
than examining after-coaching data alone.
Climate or attitude surveys can be useful in identifying changes in soft-skill areas such as
communication.
When designing surveys looking at behaviour change make sure your ratings scale allows for the
possibility that changes may be perceived as negative rather than positive.
3.6.6 Summary of evaluation methods used
Inthischapterwehavepresentedanumberofdifferentexamplesofapplied
evaluationinreallifeorganisationsettings.Table3.1summarisestheevaluation
methodsusedinourcompanyillustrations.
Table 3.1: Summary of evaluation methods in company illustrations
T-mobile Corus NHS in Wales

Questionnaire surveys to the Telephone calls to coached Baseline info from the coached
coached Questionnaire to coached Post-session scoresheets from the
Questionnaire surveys to staff of Meeting with coaches coached
the coached Final questionnaire for the coached
Review of coaching contract
Review of achievement Interviews with sample of coached
reports Start and end questionnaires to
Individual stories coached
Learning vignettes from sample of
coachee/coached pairs
Bristol & West Distribution Company A large government department
Pre- and post-course sales Questionnaires pre- and Impact questionnaires to the
performance figures post- coaching to: coached
Use of control group to compare the coached Interviews with sample of coached
coached and non-coached the coaches Focus group of staff
boss of those coached Analysis of participant action plans
staff of the coached Review of business documents
Interviews with sample of Interviews with directors
all perspectives
Source: IES, 2006
Inthenextchapterwepresentanevaluationframeworkwhichotherorganisations
mightwishtouse.
4 A Framework for Coaching Evaluation
Ourstudysponsorsalsoaskedustopresentamodel,orframework,ofhowtoevaluate
coachingprogrammeswhichcouldbeofpracticalusetocompaniesinplanningor
reviewingtheirowncoachingevaluationstrategies.Havingreviewedtheresearch
aboutevaluatingcoachingprogrammesandconsideredwhatwefoundfromour
companyillustrations,IESperceivedthatorganisationswouldbebestadvisedto
considerthesituationfromuptofourdifferentperspectivesandlookatuptothree
differenttypesofoutcomes.Inthischapterwepresentasimpleevaluationframework.
4.1 Key dimensions

Basedontheresearchanditsownexperience,IESwasabletoidentifythreesetsof
keyquestionsthatwouldbeprogrammedesignersandevaluatorsneedto
understandtheanswerstobeforethecoachingbegins:
Whatdothecoacheesexpecttogainfromthecoaching?Andhowwillweknowat
theendifthesebenefitsarerealised?
Whatdoestheorganisationexpecttogainfromthecoaching?Andhowwillwe
knowattheendifthesebenefitsarerealised?
Whatinternalandexternalprocessesneedtobeinplacetoenablethecoaching
programmetodeliverthechangesexpected?Andhowwillweknowiftheyare
workingintimetochangethemiftheyarenotworking?
Forwouldbeevaluators,itmaybehelpfultoconsiderthesethreesetsofquestionsas
areastoseekevidenceabout.
Theresearchidentifiednumerousperspectivesthatmightberelevantaslikelysources
ofevidence.Howeveritseemstherearefourmainsourcesthatitismosthelpfulto
consider:
documentsegrecordsofobjectives,achievements,coachingcontracts
coachees
coaches,whetherinternalorexternal
organisationperspectiveeglinemanagers,sponsors,HR,staff.
InIESexperiencemanyorganisationsarenervousaboutrelyingontheperspectiveof
thecoaches,especiallywhenthecoachesareexternalcommercialproviders.Thismay
bebecausetheyperceivethecoacheshaveavestedinterestinseekingtodemonstrate
thattheircoachingwassuccessful.Ourexperiencesuggeststhatcoachescanadd
valuableinsightsintocoachingrelationshipprocessaspects,aswellofferathird
opiniononindividualoutcomes.Ontheotherhand,manyorganisationsareapparently
happytorelyexclusivelyontheperspectiveofcoachees.Aswesawinthedistribution
companyillustration,thistooisunwiseastheorganisationmaygetanexaggeratedly
rosypicturefromhappycoachees.Wethereforesuggestthatthemostusefulway
forwardistoensurethatyouconsiderallthreedifferenttypesofperspective.
Figure4.1summarisesthetwodimensionsweproposetobekeyforwouldbe
evaluators:areasofevidencesoughtandlikelysourcesofevidence.
Figure 4.1: Key dimensions in framework
coach coachee
Individual level
Organisational level
Programme processes
organisation documents

Source: IES, 2006
4.2 The framework

Thethreeareasandfourmainsourcescanbepresentedasasimpleframework,as
illustratedinFigure4.2.
Weperceivesimpleframeworksasusefulonlyiftheyenableuserstotreatthemasa
startingpointtodecidewheretheywanttolookandwhatthingstodo.Withthis
simpleframeworktherearetwelveblanksquarestoconsider.Youdonthavetofillin
allthesquaresifyoudontwantto.Theimportantthinginusingtheframeworkisto
considereachsquareinturnanddecidewhetherornotitwouldmakeauseful
contributiontoyourevaluationstrategy.Questionsfromthethreerowstoaskare:
Figure 4.2: A framework for coaching evaluation
Likely Coachees Line Coaches Documents

sources of managers
evidence or
Evidence sponsors
sought at:
Individual
level
Organisation
level
Programme
processes
Source: IES, 2006
Whataretheobjectivesthecoacheesarelookingat?
Whataretheorganisationobjectives?Whatisitthatseniormanagersreallywantto
see?
Whataretheenablingfactorsorprocessfactors?
Next,foreachofthecolumns,questionstoaskyourselfare:
Whatoutcomescanweexpectthecoacheestoidentify?
Whoaretherelevantorganisationrepresentativesforthisprogramme,andwhat
outcomesdoweneedthemtoidentify?
Whatoutcomesdowewantthecoachestohelpusidentify?
Askingandansweringthequestionsbasedonthisframeworkshouldhelpyoudecide
whatevidenceyouwanttoconsiderandwhichperspectiveyouwanttolookat.
Onceyouhavedecidedwhichsquaresyouareinterestedin,weenvisagedthenext
stepwouldbetoconsiderwhichindicators(egsalesperemployee)anddatasources
(egnetsalesincomefiguresandemployeenumbersfrompayroll)youwouldusein
ordertocollecttheinformationyouhavedecidedyouneedandthatmeetsyour
evaluationobjectives.
Table4.1providessomeexamplesofareassuitableformeasurement.Yourchoiceof
actualindicatorwithineachareawilldependonthetypeofbusiness,whatindicators
arealreadyusedwithinyourorganisationandwhatexistingdatasourcesyoucantap
into.
Table 4.1: Examples of areas suitable for measurement
Organisation level
Business performance based
Sales/turnover
Market share/new customer
Productivity
RoI
Cost-effectiveness/cost-benefit
Media citations
New products/services
Product/service quality
Task time/product development time/production time
People based
Staff absence/sickness
Customer satisfaction/complaints
Employee attitudes/corporate climate
Credibility/reputation/satisfaction among external stakeholders
Retention/motivation
Individual level
360-degree feedback ratings
Achievement of coaching objectives
Comparisons pre and post coaching
Skill level/knowledge level
Appraisal
Manager assessment/self-assessment
Job performance
Process indicators
Coachee, line manager and/or sponsor satisfaction with coaching
Performance and quality of coach
Learning transfer conditions/support
Cost per coachee
Administration arrangements/venues etc.
Source: IES, 2006
Oneofthethingswedonoticeisthatpeopleoftenforgettheirexistingmeasurement
methodslike360degreefeedbackortheannualstaffsurvey.Oftenthetendencyisto
assumeyouhavetoinventsomethingnew.IESwouldadvocateconsideringwhatis
alreadyinplace;itmakesbeforeandaftersnapshotsmucheasierand,ofcourse,it
savesonthedevelopmenttime/costsofhavingtoimplementanewsystem.
5 Applying the Framework in Practice
Thestudysponsorsaskedustotesttheevaluationframeworkinarealtimecoaching
setting.Wethereforelookedforasuitableearlyadopterorganisationwhichwas
willingtoworkwithusinusingtheevaluationframework.Itwasalsoimportantthat
theorganisationbewillingtoshareitsexperienceandoutcomes.Wewerefortunate
thatanumberofIESmemberorganisationsvolunteered.
Inthischapterweoutlinethelocalgovernmentcoachingprogrammeweselectedto
evaluateandpresentitinitiallyasafurthercompanyillustration.Followingthiswe
offerjointreflectionsfromtheresearchteamandourlocalgovernmentcollaborators
ontheexperienceofusingtheframeworkandthemethodsselected,andwepresent
thefindings.Oneofthereasonsweselectedthelocalgovernmentprogrammewas
becausetheywereclearthattheywantedacomprehensiveapproachtoevaluationto
underpinthedesignoftheirprogrammefromtheoutset.Weareluckythatthe
selectedprogrammeultimatelyranforthreephasesoveraperiodofthreeyears,
enablingustoexplorehowwelltheframeworkadaptedtochangingprogramme
contentandchangingcorporateplayers.
IntheappendicesyouwillfindsomeoftheevaluationtoolsIESdevelopedaspartof
thetestevaluation.
5.1 Overview of the local government programme
Backin2003achievingimprovementsinpublicserviceswasacornerstoneofcentral
governmentpolicyinEngland.Serviceimprovementinitiativeswereunderpinnedby
anumberofinfrastructureprojectsincludingbuildingHRcapabilityandleadership
capabilityinlocalcouncils.TheformerEmployersOrganisationforLocalGovernment
(EO)decidedtointroducetheLEAPHRCoachingserviceasonemethodofsupporting
HRdirectorsacrosslocalgovernmentinraisingtheirstrategiccapability.
FromNovember2003toSeptember2004theEO,inpartnershipwithIES,NorthWest
EmployersOrganisationandEastofEnglandRegionalAssembly,designedandrana
pilotprogrammeintheNorthWestandEasternregions.TwentyfourseniorHRstaff
andtheirsponsors(typicallyCouncilCEOs)wererecruitedfrom24differentcouncils.
Thepilotprogrammeconsistedoffourtwohour,onetoonecoachingsessionsand
twoonedayseminars.
Thepurposeoftheevaluationofthepilotwastodeterminewhethercoachingmight
contributetoimprovementsinstrategicthinkingandcorporateleadershipamong
seniorHRspecialists.Ifcoachingwashelpfulforthisgroup,thepilotevaluationwas
alsoexpectedtoidentifyanyadjustmentsnecessarybeforerolloutoftheprogramme
acrossEngland.
Thefirstkeydecisionmadewasthatthisevaluationwouldnotusehardbusiness
performancemeasures.Sincethechangessoughtatindividualandorganisational
levelsweremainlybehaviouralitmadesensetofocusontheperceptualmeasures
available.Itwasalsodecidedthatperceptionsonallaspectsoftheprogrammes
processesshouldbesought.AcopyoftheevaluationframeworkfortheLEAPpilot
programmeisshowninTable5.1andthisliststhemanyevaluationmethodsused.A
moredetaileddiscussionofthesemethodsisprovidedlaterinthischapter.
5.1.4 Findings
Thequestionnaireresponsesindicatedthatthepilothadbeenhighlysuccessfulin
meetingparticipantsandsponsorsneedsandexpectations.Therewereveryhigh
levelsofparticipantsatisfactionregardingwhattheyhadachievedthoughthe
programmeandstrongperceptualevidenceofimprovedstrategicthinkingand
individualperformance.Evidenceoforganisationalimpactwasthin,althoughthere
wasauniversallysharedbeliefthatthiswouldfollowovertime.
For80percentoftheparticipants,theLEAPcoachingprogrammewastheirfirst
exposuretocoaching,andwewereabletocaptureindirectquotationswhatthe
experienceofcoachingfeelslike.Thiswasveryusefulinpromotingtheprogrammeto
futurecohorts.
Asignificantfindingwasthehigherlevelsofpositiveresponseexhibitedby
participantswhosesponsorshadalsoreturnedquestionnairescomparedtothose
participantswhohadnosponsororwhosesponsordidnotreturnthequestionnaire.
Havinganovertlysupportivesponsor,itseems,makesadifferencetotheextentofa
participantsbeliefintheachievement.
32
Table 5.1: Evaluation framework for LEAP Pilot
Practical Methods for Evaluating Coaching

Areas of evidence Likely sources of evidence
sought
Documents Participants Sponsors Coaches Partners
Individual outcomes Objective records Mid-point telephone interview End of programme impact
Application forms End of programme impact questionnaire
Vignettes questionnaire One-year-on impact
Evidence collected by participants questionnaire
One-year-on follow up impact
questionnaire
Organisational Objective records Mid-point telephone interview End of programme impact
outcomes Vignettes End of programme impact questionnaire
questionnaire One-year-on impact
Evidence collected by participants questionnaire
questionnaire
Programme processes Mid-point telephone interview Review day focus Review day focus group
End of programme impact group End of programme review
questionnaire meeting
questionnaire
Source: Employers organisation/IES, 2003
Participantsandcoachesfromoneregionweresubjecttoacarefulmatchingprocess
whereasforpracticalreasonsthosefromtheotherregionwereallocatedonthebasis
ofgeographicalproximityalone.Onesurprisefindingwasthattherewasno
differenceinlevelsofgeneralsatisfactionandsatisfactionwiththecoachbetween
participantsfromdifferentregions.
Followingasuccessfulpilot,thereweretwofurtherphasesoftheLEAPprogramme.
Phase1ranfromFebruarytoDecember2005.Basedontheevaluationfindingsfrom
thepilot,twochangesweremadetotheprogramme.Firsttheseminarswerereplaced
withtwoadditionalonetoonecoachingsessionsresultinginsixtwohour,oneto
onecoachingsessionsbeingofferedtoeachcoachee.Thesecondchangewastogive
participantstheoptionofundertakinganupfront360degreefeedbackexercise.
Theevidenceofasuccessfulprogrammearisingfromtheevaluationencouragedthe
EOtoconsiderscalinguptheprogramme.Phase2ranfromApril2005toMarch2006
andinvolvedafurther25coachees.Phase2switchedtotheuseofinternalcoachesfor
thecoachingsessions.Inordertobuildupthecapabilityofthesectortocontinueto
delivercoachingitself,IESdesignedaninitialtrainingandongoingdevelopment
programmefor14seniorlocalgovernmentmanagerstooperateastheseinternal
coaches,supportedbycoachingsupervisorsbehindthescenes.Thecoachtraining
programmewasendorsedbyInstituteforLeadershipandManagement.
5.2 Detailed review of how the framework was applied
5.2.1 Answering the framework questions
What are the coachees objectives?
Itwasexpectedthatindividualcoachees(orparticipantsastheyarereferredtointhis
example)wouldeachhavedifferentobjectives,reflectingtheiruniqueoperating
contextsandtheirdifferingpositionsonthestrategiccapabilityimprovementjourney.
This,inturn,wouldmeantheywouldeachhaveslightlydifferentoutcomesinmind.
Theissuesfromtheindividualoutcomesrowontheframeworkthatwerenotedas
needingtobeaddressedwere:
participantexpectations
personaldevelopmentneeds
personaleffectiveness
strategicthinking
personalachievements.
What are the organisations objectives?
Sincetheparticipantscamefromdifferentcouncils,itwasalsoexpectedthat
organisationaloutcomesrequiredwouldbedifferent.However,anorganisational
needsanalysishadbeenundertakenbeforethedecisiontoofferacoaching
programme.Theissuesfromtheorganisationaloutcomesrowontheframeworkthat
werenotedasneedingtobeaddressedwere:
sponsorexpectations
HRfunctionprofile
HRfunctionleadership
corporateleadership
organisationalachievements.
TheEOitselfhadafurtherobjectivefortheprogramme.Ithopedthatsomeamong
thefirstcohortofparticipantswouldbeabletoactasrolemodels:willingtobe
identifiedasHRchampionsandhappytodescribewhatthestrategiccontributionof
HRlookslikeintheircouncil.
What are the enabling/process factors?
Theissuesnotedasrelevantfortheevaluationintermsoftheprocessfactorswere:
administration
pricing
qualityofcoaches
qualityofeventspeakers
qualityofvenues
changesrequiredbeforerollout.
What outcomes can we expect the coachees to identify?
Sincewewereevaluatingapilotandalltheparticipantshadsignedupknowingthe
programmewouldbeevaluated,theEOfeltthatitwasreasonabletoexpect
significantinvolvementfromparticipants.Thus,weexpectedcoacheestoassistin
identifyingallrelevantissues(listedabove)forindividual,organisationaland
processes,plusanychangetheyperceivedintheirbehaviourasaresultofattending
theprogramme.
What outcomes can we expect the coaches to identify?
Weweremindfulofwhatwefeltneededtobekeptconfidentialbetweencoachand
coachee,butconcludedthatthecoachesperspectiveonachievements,behaviour
changeandprogrammedesignwouldbeusefuljustforthepilot.However,themost
significantcontributionfromthecoacheswouldbetakingresponsibilityforensuring
thatparticipantscompleteddocumentationwhichwouldthenbeusedaspartofthe
evaluation.Decidinginadvancetousewrittenrecordsofobjectivessetenabledusto
agreeinadvancewithparticipantsandcoacheswhatcouldbesharedaspartofthe
evaluation(andatwhatlevelofdetail).
Who are the relevant organisation representatives for this programme, and what
outcomes can we expect them to identify?
Eachparticipanthadasponsorfromtheircouncil.Thesesponsorswerefelttooffer
thekeyperspectiveonanyindividualbehaviourchange,anyimprovementsinHR
functionandcorporateleadershipandanyotherorganisationaloutcomes.
Thefourpartnerorganisationsinvolvedindesigning,marketingandadministering
theprogrammewerealsofelttohaveausefulcontributiontomaketotheevaluation
ofthepilot,especiallyinrelationtoidentifyingprogrammeprocesses.
Inthisparticularevaluationcontextfourperspectiveswereconsideredessential(ie
coach,coachee,sponsorandpartner),ratherthanthethreeperspectiveswetypically
encounteredinourcompanyillustration.
5.3 Detailed review of evaluation methods used
Record of Objectives
Mostdevelopmentplanning/appraisalsystemsandtrainingprogrammesuse
standarddocumentationtorecordparticipantobjectives,althoughsomeofthe
companiesweencounteredwhoemployexternalcoachesrelyonthecoacheskeeping
theirownrecords.Othercompaniesmeetsomeresistancefromparticipantswhenitis
suggestedlaterthatsuchformswouldbeusefulforevaluation,astheyareperceived
ascontainingconfidentialdata.
TheEOrequiredallcoachesandparticipantstouseastandardform,tobecompleted
bytheendofthefirstonetoonecoachingsession,anditwasmadeclearfromthe
outsetthatformswouldbeseenbyprogrammeevaluators.Coachesweregiventhe
responsibilityofsubmittingacopyofformsforalltheirparticipants.Thishadthe
advantageoflesspeopletochasethanifallparticipantshadbeenresponsible.Acopy
oftherecordofobjectivesformisincludedatAppendix1.Justthreecolumnswere
usedtoencourageparticipantstoarticulate:
whattheyhopedtogetoutoftheprogrammeietheirobjectives
howtheywillknowtheyhaveachievedtheirobjectivesietheirsuccesscriteria
bywhatmeanstheywilldemonstratethatachievementieevaluationmethod.
Mid-point telephone interviews
Becausetheinitialprogrammewasapilot,theEOwaskeentogetsomefeedbackon
howthingsweregoingmidwaythroughtheprogramme.Thiswaspartlydrivenby
financialconsiderations:onlyifitlookedliketheprogrammewasoncourseto
achieveindividualandorganisationaloutcomeswouldabidformoniesbemadeto
rolloutthefullprogrammethefollowingyear.(Collectingdataatthemidpointalso
enableschangestoprogrammeprocessestobemade,ifnecessary,beforethe
programmefinishes;hadaparticularproblemwithacoach,participantorvenue
emerged,therewouldhavebeentimeforchangestobemade.)
Telephoneinterviewswithparticipantswasthemethodselectedatthemidpointas
theseenabledalltherelevantissuestobeexplored.Itwasalsoexpectedthatthe
informationgeneratedaboutindividualexpectationsandoutcomeswouldassistin
developingtheimpactquestionnaires.
Questionsaskedinthetelephoneinterviewsrelatedtoindividualoutcomesfocused
largelyonparticipantsperceivedpersonalbenefitsandbehaviouralchangessince
beginningtheLEAPprogramme.Participantswereaskedtoconsiderwhatthekey
personalbenefitshadbeen,andhowthesehadbeentranslatedintobehavioural
change;whoorwhatthischangehadaffected;howithadbeensustained;andthe
extenttowhichtheyfeltitwasattributabletotheLEAPprogramme.Followingon
frompersonaloutcomesandachievements,participantswerethenaskedtoconsider
howtheyfelttheLEAPProgrammehadaffectedtheirorganisation,job,HRfunction
andtheirLocalAuthority.Questionsinthissectionfocusedonperceivedsupport
withintheorganisation,thevalueoftheHRfunction,theuseandbenefitoftheLEAP
programmetotheHRfunctionandwiderauthority,plustheextenttowhich
perceivedbenefitscanbeattributedtotheprogrammeitself.Acopyofthetelephone
interviewdiscussionguideisincludedasAppendix2.
ThetelephoneinterviewswerecarriedoutduringMay2004aftertwoonetoone
sessionsandoneseminarhadbeenexperienced.Twentytwooutof24participants
tookpart.Participantswereencouragedtospeakfreelyandcandidlyabouttheir
experiencesandthoughtsinordertoconsolidateandenhancetheirownlearning
fromtheprogramme,aswellasinformanddirectfutureelementsandactivities.
Theresultsfromthetelephoneinterviews,togetherwithananalysisofthewritten
recordsofparticipantobjectives,wereusedtocompileaninterimreport.Theinterim
reportprovidedasummaryofkeythemesandissuesthatemerged,andwaswritten
forthefourpartnerorganisationsasaninterimviewofprogress.
End of programme impact questionnaires
TheEOwasverykeenthatendofprogrammequestionnairesbeusedandthat
somewhereinthequestionnaireshouldbesomeperceptualmeasureofimpact.This
wasbecauseperceptualmeasuresareveryusefulincapturingthekindofobservable
behaviourchangewhichlayattheheartoftheindividualandorganisationaloutcomes
soughtfortheLEAPprogramme.
Questionssoughttoaccesseachindividualsviewsinthefollowingareas:
perceptionsofcoachingasamethodofpersonalandprofessionaldevelopment
expectationsof,andobjectivesfortheLEAPHRcoachingprogramme
theobservedimpactoftheLEAPHRcoachingprogrammeonpersonaleffectiveness,
HRfunctionleadershipandstrategicthinking/corporateleadershipskills
theextenttowhichparticipantswouldrecommendtheprogramme
qualitativeassessmentofaspectsofthecoachingprocess,includingcoachsstyle,
strengthsandweaknesses,andparticipantspersonalcomparisonswithother
formsofpersonaldevelopment
sponsorassessmentoftheimpactoftheprogrammeonthisindividual,theHR
function,andtheorganisationasawhole
suggestionsforchangeanddevelopmentoftheLEAPHRcoachingprogramme.
TwoversionsoftheimpactquestionnairesweredistributedinAugust2004:oneto
participantsandonetosponsors.Somequestionsweredesignedusingasixpoint
attituderatingsscalesothatresponsesfromparticipantsandtheirsponsorsonthe
individualandorganisationaloutcomescouldbedirectlycompared.IESalso
distributedathirdversionofthequestionnairetothecoaches,althoughthiswas
outsidethescopeoftheEOactivity.
Twokeyareasofpersonalimpactwithpotentialindividualoutcomeswereidentified
fromtheprevioustelephoneinterviews,andarangeofquestionnairestatementswere
developedrelatingtoeach.Participantswereaskedtoindicatetheextenttowhich
theyagreedordisagreedwiththestatements,allofwhichassessedpotentialchanges
inthewayparticipantswereworkingorbehavingasaresultoftheLEAPHRcoaching
programme.
Inrelationtoorganisationaloutcomesaseriesofstatementsweredeveloped,again
basedonlikelyoutcomesidentifiedthroughthemidpointinterviewdata.Thesame
sixpointattitudescalewasusedinaskingparticipantsandsponsorstoagreeor
disagreewiththestatements.
Participantswerealsoaskedaseriesofopenendedquestionsregardingtheirfeelings
aboutbeingcoached,andelementsofthecoachingthatworkedmostandalsoleast
well.Itwashopedthattheirresponseswouldhelpimprovethewayweanswerthe
questionsfromanyfutureparticipants,WhatiscoachinglikeandwhatcanIexpect?
Fullresponseswerereceivedfrom20participantsandtensponsorsindicatinghigh
responseratesof83percentand56percentrespectively.Theresultsfromthetwo
questionnaires,togetherwithananalysisofparticipantsrecordsofachievement
supportedbyevidenceofachievementcollectedbyparticipants,wereusedtoproduce
anendofpilotreport.
Production of vignettes
TheEOdecidedfromtheoutsetthatshortvignetteswouldbeproducedasanoutput
oftheevaluationprocess.Thevignettesareillustrativeexamplesoftheoverallimpact
oftheLEAPHRcoachingprogrammeonsomeindividualparticipants.Intendedtobe
seenassuccessstories,theideawastousetheminfuturepromotionalmaterialasthe
programmewasrolledout.
ThreevignetteswereproducedattheendofthepilotinSeptember2004with
participantspermissionandhavebeenusedwidelysincethen,includingontheEOs
website.Theywerealsousedaspartofthepresspacks,andsubsequentlyasback
groundtoarticlesinlocalgovernmentpublicationsandPeopleManagement(Warren,
2005).AnexampleofthevignettesisshowninAppendix3.
Itmaybethatinothersectorsindividualsmightbereluctanttobeidentifiedin
vignettes.ParticipantsintheLEAPprogrammewere,onthewhole,willingtobe
publiclyidentified.Itmaybethattheprogrammeorganisershelpedbyensuringfrom
theoutsetthatparticipantsknewthatifthepilotprogrammewassuccesstheywould
beexpectedtoactasHRchampionsthroughoutthesector,encouragingotherHR
specialiststoembedmorestrategicwaysofworking.Perhapsimplicitinthisisan
assumptionofbeingwillingtostepintothespotlight.Itisworthspeculatingwhether
thepilotparticipantsmayhavebeenmorehighlymotivatedtomakethemostofwhat
theprogrammeofferedsincetheyknewtheirprogresswasbeingsopublicly
monitored!
Focus groups at review day
InitiallytheEOconsideredusingasecondtelephoneinterviewwithparticipantsat
theendofthepilottocaptureexperiencesoftheprogramme.Thiswasquickly
replacedbytheideaofbringingallpartiestogetherforareviewandevaluationday
inSeptember2004.Partners,coachesandparticipantsmetinseparatelyfacilitated
focusgroupstoexploreissuesandproblemswiththeprogrammeandtodiscusswhat
theyhadgainedfrombeinginvolved.Laterthethreegroupsgottogethertoshare
theirviewsandidentifyareasforfutureaction.
Follow-up impact questionnaires
InordertoassessfurtherthelongtermimpactoftheLEAPpilotprogrammeashort
followupquestionnairewassenttoallofthepilotgroupparticipantsandtheir
sponsors.Theprimaryaimwastolookathowtheprogrammehadaffectedparticipants
atanindividuallevelandtheircouncilsatanorganisationalleveloneyearon.The
maindifferencefromtheendofprogrammequestionnaireisthatthefollowup
questionnairewasshorteranddidnotcoverprogrammeprocesses.
Onlyfourparticipantsandeightsponsorsreturnedcompletedquestionnaires,a
responserateof16percentand32respectively.Theverylowresponseratesmay
havebeeninfluencedbyrelocationofparticipants,holidays(thequestionnaireswere
distributedinAugust)andsurveyfatigue(thiswasthefourthevaluationmethod
requestedofparticipants).
Thesmallsamplenumbersmadeindepthanalysisinappropriate.Itwas,however,
interestingtohearthat,forsomeparticipants,thelevelofsatisfactionandpersonal
changeasaresultofthepilotprogrammehadledthemtobecomeinvolvedinthe
subsequentrolloutoftheprogramme,eitherassponsorsorintrainingtobecoaches
themselves.
5.4 Detailed review of presenting the evaluation findings

Usingmultiplemethodsforcollectingevaluationdoesmeanyoushouldexpectto
generateaconsiderableamountofinformation.Thiscanoftenresultinatime
consumingandcomplexprocesstomakesenseofitall,soitisimportantnottouse
upallyourtimeallocationincollectingthedata.Thevastmajorityoforganisations
havethecapabilitytodothisworkthemselvesand,wheretheyalsohavethecapacity,
theymaywellprefertoundertakeallaspectsofcoachingevaluationinhouse.Ifyou
douseexternalresourcesitisusefultodowhattheEOdidandbefullyinvolvedin
thedevelopmentoftheevaluationframeworkanddatacollectionmethods.
IESbroughttogethertheissuesidentifiedthroughallthedatacollectionmethodsinto
onereport.Inpresentingtheresultsfromtheendofprogrammequestionnaires,the
intentionswereto:
categoriseandgiveexamplesofparticipantoriginalobjectives
comparetheaverage(mean)scoresreturnedbyparticipantsandsponsorsaskedto
assesschangesinworkingorbehavingasaresultoftheprogramme
usedirectquotestoofferfurtherinsights.
AnexampleofpresentingacomparisonofaveragescoresisshownasFigure5.1.It
wasimportanttoexplaininthereportthatbothgroupshadbeenaskedtoindicatethe
extenttowhichtheyagreedordisagreedwithaseriesofstatements.Theresponse
optionswere1disagreedverystronglythroughto6agreedverystrongly.Thusan
Figure 5.1: Comparing average scores in LEAP
More effective at the strategy 4.50

and planning aspects of role
and work 4.30
More pro-active in seeking 4.40

feedback from internal
customers of HR/OD function 3.90
More effective in realigning 4.40

work of my HR/OD function to
the organisation 4.20
More effective in managing the 3.80

expectations of stakeholders 4.20
Puts more effort into 4.30

developing a vision for people
management and sharing this 4.40
Extends networks inside and 4.00

outside the organisation 3.70
4.00
A better corporate player
4.30
1 2 3 4 5 6
Participant's response Sponsor's response
Source: IES, 2004
average(mean)scoreof3.5couldbeconsideredthemidpointrepresentingneither
disagreementoragreement.Anaverage(mean)scoreoflessthan2.5couldbe
consideredverynegativeandanaverage(mean)scoreofmorethan4.5couldbe
consideredverypositive.
ExamplesofdirectquotesareshowninFigure5.2.Ratherthansimplylistpagesof
quotes,thisexample,takenfromtheprogrammeprocessessectionofthereport,
demonstrateshowtheycanbepulledtogethertoillustrateapoint.For80percentof
theparticipantstheLEAPcoachingprogrammewastheirfirstexposuretocoaching,
soweuseddirectquotestoreportintheirownwordsonwhattheyhadmadeofthe
experienceofcoaching.
5.5 Applying the framework for the second time

TheevaluationplansforLEAPphase1weremuchsimplerthanforthepilotandthe
costofimplementationwaslower.Thisisinlinewithwhatyoumightexpectafteran
underthemicroscopeexaminationofanewprogramme.Astheprogrammeformat
settlesintoaroutine,theprogrammeevaluationlikewisebecomeslessonerous.
Figure 5.2: Key themes about the experience of being coached
Coaching compares well with

Being coached can feel other personal development
Challenging and refreshing tools
Exhausting, exhilarating, exciting Compares favourably. This was my
and a real confidence booster first experience, I am very much
sold on this
Safe because it is confidential and
supportive Proactive and continuous, not a
one off event which is forgotten!
Strange at first, intense and tiring
Much better, more personal, more
Very privileged as it provides an directed
opportunity to offload problems,
work through issues, feel One of the best experiences, the
enthusiastic about the job, look to lessons will stick with me for a
the future long time
The coaching
experience
Successful elements Less successful elements

Being able to explore what I Benefits only felt over long term
wanted from work and life with
It can be quite challenging and
non-judgmental coach
you have to be prepared to be
Being able to be completely open and take what is said
honest with no fear of comeback constructively
No hidden agenda, undivided Sessions too close sometimes,
attention, setting personal needed more info in advance
objectives
One to one relationship, felt free
to be honest and open
Time to reflect and clarify areas
of strength and for improvement

Source: IES, 2006
TheevaluationframeworkforLEAPphase1isincludedasTable5.2.Asitshows
thereislessemphasisonprogrammeprocesseswithjustonemethodselected.There
wasarelianceonparticipantstoidentifyanyareaswherestandardsofdelivery
lapsed.Thetworowsofoutcomeshavebeencombinedsincetheperceptualmethods
proposedwerethesameforbothindividualandorganisationaloutcomes.Thisis
likelytobeacommonoccurrencewhereallmeasuresareperceptual.Itislesslikely
whenbusinessmeasuresareselected.
Table 5.2: Evaluation framework for LEAP Phase 1
Areas of evidence Sources of evidence

sought
Documents Participants Sponsors
Individual outcomes and Records of End of programme impact End of programme
organisational outcomes objectives questionnaires impact questionnaires
Mid-point impact questionnaire
Programme processes End of programme
questionnaire
Youwillnoticethatthemidpointtelephoneinterviewsaregone,replacedbyaless
resourceintensiveimpactquestionnaire.Oneoptionconsideredwastodevelopapre
coachingquestionnaireinsteadofoneatthemidpoint.Howeveraprecoaching
questionnairewasruledoutalongwiththeoptionofdoingnothing.Thiswaspartly
becauseofthebenefitstoparticipantsinthepilotofsomepauseatthemidpoint.It
seemshavingamidpointevaluationfocusesthemindonwhathasbeenachieved
andhowbesttomakeuseoftheremainingsessionstoachieveremainingornew
objectives.
5.6 Applying the framework for the third time

Sincetheevaluationofphase1wascosteffectiveandsuccessful,theevaluationplans
forphase2wouldhaveundoubtedlylookedthesamehadtheprogrammeresourced
thecoachesinthesameway.However,becausephase2includedanadditional
streamofactivitytotraininternalcoaches,thismeantthecoachingframeworkneeded
tochangetoreflectthenewcircumstances.Theevaluationframeworkforphase2is
showninTable5.3.
Copiesofparticipant,sponsorandvolunteercoachendofprogrammeimpact
questionnairesareincludedatAppendices4,5and6respectively.
Table 5.3: Evaluation framework for LEAP Phase 2
Areas of evidence Likely sources of evidence

sought
Documents Participants Sponsors Volunteer coaches
Individual outcomes Records of End of programme End of programme End of programme
and organisational objectives impact impact impact
outcomes questionnaire questionnaire questionnaire
Focus group
Coaching programme End of programme End of programme Focus group
processes questionnaire questionnaire
Coach training Focus group
processes
5.7 Lessons learnt about coaching evaluation
5.7.1 Consider the cost-benefit of each evaluation method
Theliteratureencouragesustolookforbenefitswellafterthecoachingexperiencehas
ended.Trackingparticipantsexperiencesandbehavioursovertimeisreallytheonly
wayofmeasuringtheimpactofinterventions(likecoaching)withalatenteffect.Itis
alsoimportantformeasuringthesustainabilityofeffect.Withoutthistracking,any
measuredimmediateorshorttermbenefitmaybepartialandthereforeanunder
estimateormaynotbesustainedandthereforemaybeanoverestimate.
InpracticetheEOfoundthecostsandpracticalitiesmadethisdifficult:peoplemove
oninorganisations;maintainingcontactdetailscanbetricky;andpeoples
willingnesstocooperatemaylessenovertime.
5.7.2 Get coaches to use a common tool for recording objectives
TheEOwasnotlookingtoidentifyhardmonetarybenefitsfromthisprogramme,so
itwasnotnecessaryforparticipantstoexpressalltheirobjectivesintermsofadded
valuetothelocalgovernmentsectoringeneralortheircouncilinparticular.
Nevertheless,inpracticethebriefingtocoachesontheuseofacommonrecordof
objectivesprovedverybeneficial.Itensuredthatcoachesandparticipantsalikekept
theireyeontheoutcomestheyexpectedtoachieve.Italsoensuredthatobjectives
includedamixofpersonalandorganisationalones.Ifcoacheesagreeonlypersonal
objectivesornoobjectivesatall,noevaluationoforganisationalbenefitwillbe
possiblelater.
5.7.3 There is a need for a comprehensive evaluation tool
IESundertooksomeadditionaldataanalysis.Incomparingasamatchedtriadimpact
questionnaireresponsesfromparticipants,sponsorsandcoacheswewerestruckby
thepositiveresponsesreceivedacrossallimpactitemsforallgroups.Thisdemonstrates
sharedperceptionsofsuccessfuloutcomesresultingfromthecoachingprogramme.
Wheredifferencesinperceptionexist,thesewereminimalandrelatedonlytothe
degreeofresponse.Therewerenoconflictingviewsbetweenthethreegroups.
Factoranalysiswasalsoperformedshowingthatdifferentelementsoftheprogramme
areratedtodifferingdegreesofpositiveresponsebyeachgroup,highlighting
differencesinobjectivesandmethodsofassessingtheprogramme.Thisreinforcesthe
appropriatenessofimplementingacomprehensiveevaluationtool,takingintoaccount
avarietyofviewpointsinanumberofways.Theimplicationsforevaluatingfuture
coachingprogrammesmayincludeakeenappreciationofthedifferingviewpointsof
stakeholdersandparticipants.
5.7.4 Consider the scale on impact questionnaires
Forallthreeevaluationroundsweusedasevenpointscalerangingfromvery
stronglyagreetoverystronglydisagreetomeasurerespondentsviewsonaseriesof
statements.Longerscaleslikethesecanbeusefulindiscriminatingresponseswhere
theyareallatoneendofthescale.Howeverinthiscasewefoundnomeasurable
differencebetweenverystrongly(dis)agreeandstrongly(dis)agree.Giventhis,wewould
certainlyhavechangednexttimetousingafivepointscaleinsteadrangingfrom
stronglyagreetostronglydisagree.
5.8 Lessons learned about the design of coaching

programmes
5.8.1 Organisation sponsors are important even for those at director level
AsignificantfindingfromtheLEAPprogrammewasthehigherlevelsofpositive
responseexhibitedbyparticipantswhosesponsorshadalsoreturnedquestionnaires
comparedtothoseparticipantswhohadnosponsororwhosesponsordidnotreturn
theirquestionnaire.Itseemsthathavinganovertlysupportivesponsormakesa
differencetotheextentofaparticipantsbeliefintheirachievements.Thisreinforces
theimportanceofemployerengagementinthesuccessofcoachingprogrammes.
5.8.2 A careful matching process doesnt always make a difference
Thepractitionerliteraturestronglypromotestheideathatcoachingschemeorganisers
shouldcarefullymatchcoachandcoacheepairsinordertomaximisethechancesof
buildingtherapportandtrustuponwhichsuccessfulcoachingrelationshipsdepend.
Ourfindingsindicatednodifferenceinlevelsofgeneralsatisfaction,orlevelsof
satisfactionwiththeircoach,betweenthoseparticipantsfromtheoneregionwhere
coachesandparticipantsweresubjecttoacarefulmatchingprocessandthosefrom
theotherregionwherecoacheswereallocatedonthebasisofgeographicalproximity
alone.Itmaybethatourfindingswereinfluencedbythehighpercentageofthose
beingcoachedforthefirsttime.Neverthelessourresultschallengeperceivedwisdom
aboutmatching.
5.8.3 360-degree feedback results are useful at the collective level
Undertakinga360degreeexercisewasanoptionalextraforparticipantsinLEAP
phases1and2.Therewasahightakeupwithparticipantsreportingtheexerciseas
helpfulinidentifyingthatsomeoftheobjectivestheywantedtoworkonwerealso
endorsedbyothers.Anindividualsresultswereconfidentialtothecoacheeandtheir
coach.Howeverindividualresultswereanonymisedandcombinedacrossall
participantsineachphasetoproduceacollectivepicture.Thiswasveryusefultothe
EOinsupportingwiderresearchandshapingperceptionsofissuesforseniorHR
specialistsacrossthelocalgovernmentsector.
5.8.4 Use of internal coaches
EvenfortheseasonedHRdirectorswetrainedasinternalcoacheswithinthesector,
thesupportofanexperiencedcoachingsupervisor,intheearlydaysatleast,was
consideredessential.
Wecomparedlevelsofparticipantsatisfactionattheendoftheprogrammebetween
thosecoachedbythelessexperiencedinternalcoachesusedinLEAPphase2andthe
moreexperiencedexternalcoachesusedinthepilotandphase1.Ourfindings
indicatednosignificantdifferenceinlevelsofgeneralsatisfactionorlevelsof
satisfactionwiththeircoach.Thismaybeasurprisetosomeexperiencedand/or
externalproviders.
Unfortunatelywewerenotabletocomparelevelsofparticipantsatisfactionatthe
midpointofthethreecohortsastherewasnomidpointimpactquestionnaireduring
phase2.Itmighthavebeeninterestingtoseewhetherornottherewasanyevidence
ofmoreexperiencedexternalcoachesbeingperceivedbyparticipantsasmaking
progressmorequickly.
Thefindingthattherewasnodifferencebetweentheexternalcoachesinthepilotand
phase1andtheinternalcoachesinphase2maybeareflectionoftherigorous
selectioncriteriaweappliedforaccesstothecoachtrainingprogramme.Itmayalso
beinfluencedbythefactthattheinternalcoachesintheLEAPprogrammecannotbe
consideredinternalintheusualsenseastheywerenotemployedbythesame
organisationastheircoachees.Perhapsamoreaccuratetermforthemwaswithin
sectorcoaches.
6 Conclusions
Sincewebeganthisresearchbackin2003coachingasadevelopmenttoolhasbecome
evenmoreprominent,especiallyintheUK.TheCIPDsannualtrainingand
developmentsurvey(CIPD,2005)indicatesthatcoachingbylinemanagersisusedby
84percentofUKrespondingorganisationsandcoachingbyexternalpractitionersis
usedby64percent.Thereisgoodnewsforcompaniesinthatnowthereismore
practicaladviceavailabletothemonhowtoimplementcoachingsuccessfullyintheir
organisations(ClutterbuckandMegginson,2005),andonhowtomakeabusiness
caseforcoachinginvestments(Jarvisetal.,2006).TheUKhasalsofollowedtheUSin
callingformoreprojectsusingRoImethodologies(Kearns,2006).Worryinglythough
verylittleevaluationisyetbeingreportedastakingplacebycompanies(CIPD,2004)
andverylittlenonpartisanadviceoncoachingevaluation.
Inthisfinalchapterwereflectontheresearchwhilediscussingthefactorswhichseem
tohelpandhindergoodcoachingevaluationpractice.Bywayofconclusionwethen
pulltogetherallthelearningfromtheliterature,companyillustrationsandourearly
adoptercasestudytoofferourfinaltipsforwouldbecoachingprogramme
evaluators.
6.1 Factors helping or hindering evaluation practice

Duringourdiscussionwithemployersweformedtheviewthatcoachingevaluation
isanactivitythatalmostallcompaniesagreeisimportant,butitisonewhichisoften
forgottenintheperpetualrushtogetthingsdone.Findingthetimetoevaluateisa
biggerbarrier,itseems,thanhavingthecapabilitytodoit.
Wherecoachingevaluationistakingplaceitrarelylooksliketheneutralandobjective
exercisethatpractitionersfeelacademicsdescribeandpromote.Inpracticecoaching
evaluationseemstofallintooneofthreecategories,basedonwhattheorganisationis
doing:theyareseekingtoprovesomething,improvesomethingorlearnsomething.
6.1.1 If you are searching for proof
Althoughmuchofthepractitionerliteratureurgescompaniestogodowntheprove
somethingroute,amongourcompanyillustrationsinChapter3onlyBristol&West
BuildingSocietysucceededinthisapproach.Thecentralgovernmentdepartment
initiallyhadhopedtodosoaswell,butdidnthavethebaselinedatainplacetoallow
themtodoso.
TheBristol&Westillustrationwasasmallscaleexampleofintenttoidentify(prove)
whetheranincreaseinsales,akeyorganisationalperformancemeasure,occurred
whencoachingwasintroducedtosupportsalesstaffaftertheirattendanceonformal
trainingcourses.Weconsiderittobesuchaninterestingexamplebecausewe
understandtheorganisationdidnotattempttoevaluatethelikelyimpactforany
otherstaffgroup,althoughcoachingwasbeingconsideredformanysuchgroups,or
inanyothercontextthanpostcoursesupport.Theapproachwasfortheorganisation
tofocusitsenergywhereitwasbelievedeasiesttomeasureorganisationalimpact(if
therewasany).Theideawasthentousethesefindingsasreasonableevidence(not
thesamethingasclaimingproof)thattheremightbebenefitforothergroupsor
contexts.Thissuccessfullyavoidedunnecessaryadditionalexpenseandthecomplexity
ofmeasuringelsewherewhereitmightbelessstraightforwardtoidentifyan
appropriateorganisationalmeasureofimpact.
OnemightwonderwhytheBristol&Westevaluationteamstoppedshortofacost
benefitanalysisoraRoIexercise.Ourperceptionisthattherewasnoneed:everyone
couldclearlyseethattheincreaseinsalessignificantlyexceededthecostofthe
coaching.
Academicsarewellawarethatproblemsofcausalitycanbeviewedasabigbarrier
toevaluation,especiallyforthoseemployersthatmightbeseekingtoprove
something.Thereisnologicalcausationbetweenimprovedbusinessresultsandthe
factthattherehasbeencoachingandsuchlinksareinterpretative.Manyvariables
affectbusinessresultsincludinglinemanagementactionsandcompetitioninthe
market.Justbecausesaleswentupby27percentinBristol&Westdoesnotmeanit
wasbecauseofthecoaching.Indeed,someacademicsmightarguethattheymayhave
goneupbymorehadtherebeennocoaching.Thefourpercentincreaseinsales
amongthecontrolgroupisfairlystrongevidenceinthisparticularcase.Howeverit
wouldhavebeenevenstrongerifithadbeensupplementedbyqualitativedatafrom
participantsexplaininghowthecoachinghadmadeadifference.
TheinhouseplusexternalcoachpartnershipdoingtheevaluationintheBristol&
Westillustrationdidnotconcernthemselvesasmuchwithcausalityasresearchers
tendto.Thesearchforprooffortheheadoftrainingandrecruitmentandherboard
ofdirectorswasatestofreasonablenessofthechainofimpactbeingclaimed.A
scientificlevelofproofwasnotrequired.Theuseofacontrolgroup,comparing
financialplanningmanagerswhoreceivedcoachingwiththosewhohadnot,was
certainlyagoodideainunderstandingtheissuesofcausation.Wewouldcommend
theorganisationforthisstep.Interpretationwasstillneededthough.Forinstance,one
couldarguethatthegroupwhowerenotofferedcoachingortrainingandthegroup
notofferedaddoncoachingmayhavefeltlessfavourablytreated,andmighthave
hadlowerlevelsofmotivationthanthecoachedgroup.This,onecouldargue,may
explainthepoorerperformanceofsomegroups.Thisreinforcesthepointthat,while
claimingbusinessresultsarenotproofinastrictlyscientificsense,companiescan
removetheweightofthecausalitybarrierbyadoptingthepragmaticattitudeof
Bristol&West.Thekeyquestionforwouldbecoachingprogrammeevaluatorsis
thusredefinedasDowebelievethecoachinghelpedachievethebusinessoutcome?
6.1.2 If you are looking to make improvements
Thelocalgovernmentcasestudyisanexampleofthemostcommonintentwe
encountered:toimproveaflagshipdevelopmentprogrammeutilisingotherlearning
methodsbutpredominantlyonetoonecoaching.Thecasestudyinvolvedanin
depthevaluationusingmultipledatacollectionmethodstoidentifybehavioural
changesthatoccurredinindividualleadersonapilotcoachingprogramme.Theidea
wastousethisinformationtoassesswhetherthedesiredbehaviouralchangeswere
beingrealised.Itwasalsoexpectedthatanynecessaryincrementalimprovementsto
theprogrammewouldbeidentified,andthesemightincreasetheextentofbehavioural
changeforindividualleadersinfuturecohortsontheprogramme.Twoofourcompany
illustrationsinChapter3,thedistributioncompanyandthecentralgovernment
department,werealsolookingforprogrammeimprovementswiththeirprimary
pointofinterestbeingbehaviouralchange.
Insteadofevaluatingdirectbusinessoutcomes,coachinginthelocalgovernmentcase
studywasmeasuredthroughsubjectivejudgementsonwhetherparticipantshad
improvedtheireffectiveness.Thisiswhatacademicswouldcallperceptualevaluation.
Itis,ofcourse,perfectlypossibleforinhousepractitionerstocarryoutthiskindof
measurementinastandardisedwayandnothavetohireexternalresearchers.
Structuredquestionnairescanbeadministeredtoparticipantsbeforeandafterthe
coachingtocomparetheirpreandpostcoachingperceptionoftheirowneffectiveness.
360degreefeedbackquestionnairescanalsobringintotheevaluationprocessthe
perceptionsoftheirlinemanagers,customers,suppliers,peersandownstaff.
Employerswecontactedduringourresearchhadmixedviewsaboutperceptual
coachingevaluation.Ontheonehandtheyfounditmucheasiertoconceptualisethan
measuringbusinessoutcomesbecausetherewerefewermethodologicalbarriersor
hindrancestoovercome.Onecouldstillargueaboutcausality,butifallpartiesagreed
thattherequiredbehaviourswerebeingdisplayed,asinthelocalgovernmentcase
study,employerstellusthatevidencethatthedesiredoutcomehadbeenachieved
madetheextenttowhichcoachingwasresponsibleirrelevant.Perhapsthisistrueif
theonlypointofinterestisbehaviouralchange.However,mostorganisationsare
makinginterventionchoicesupagainstlimitedresourcestoallocate.Theytypically
wanttoknowthattheyhaveinvestedwisely(ietheyhavegotareturnonthat
investment),oratleastthatsometangible(aswellasanyintangible)benefitsoccurred
asaresultoftheintervention.Thismayrequirethestakeholderstothinkandact
differently.Thismayalsodemandthatchangeagents(coaches)drivetheconversation
towardsconsidering,letaloneachieving,thatpossibleend.
Butwhileperceptualevaluationwaseasiertoconceptualise,employersperceiveditto
bemuchhardertoimplement.Insistencefromparticipantsandcoachesthatcollected
datashouldbetreatedasconfidentialhinderedattemptstogetagoodresponserate,
andcastdoubtsonthegeneralisabilityoftheresults.
Theappendicesofthisreportwillbeparticularlyhelpfultothoseinterestedin
perceptualmeasures.
6.1.3 If the focus is on learning
InboththeTmobileandNHSinWalescompanyillustrationsinChapter3wewould
categorisetheprimaryintentasbeingtolearnaboutthepotentialofcoachingasatool
fororganisationallearningandculturechange.Inbothcaseswewoulddescribethe
illustrationsasbeingprimarilyaboutcollectingreactionstothecoaching.TheNHSin
Waleshiredanexternalevaluatortoexploretheissuesarisingforindividualleaders
participatinginacoachingprogramme.However,aninhouseteamcouldjustas
easilycollectandexplorereactionsasaresultofusingquestionnaires,shorttelephone
interviewsorfocusgroupinterviews.
Responseratesfromreactionnaires,asquestionnairestoassessparticipantsreactions
aresometimesknown,canbeimprovedconsiderablybyaskingcoachesorline
managerstodistributeandcollectthem.Sometimesformsareevencompletedaspart
ofthefinalcoachingsessioninthepresenceofthecoach.However,ourexperience
suggeststhedatamaynotthenbeasreliableascoacheesmayfeelprotectiveoftheir
coachorunderpressuretogivepositiveresponses.
6.2 Summary of tips for programme evaluators
6.2.1 Planning your evaluation
Acceptthatthetraditionalmodeloftrainingevaluationyoumayuseelsewhere
mightnottransfereasilytocoachingevaluation.Youmayhavetoadaptitoryou
canusetheevaluationframeworkdevelopedaspartofthisresearch.
Clarifywhytheevaluationisbeingconducted.Areyouseekingtoprovesomething,
improvesomethingorlearnsomething?
Clarifyyourbudget,resourcesavailableandanytimeconstraintsandconsider
theseinrelationtoyourpurposeegdoyouhavetimetoexploresustainableimpact
orRoIifthisisyourpurpose?
Definesuccesscriteriabeforeselectingmeasures.Selectevaluationmeasuresthat
collectdatatoshowwhethersuccesscriteriahavebeenachieved.Considerlooking
forbenefitswellafterthecoachinghasendedbutbeawarethatinpracticethiscan
bedifficult.
Considertheperspectivesofdifferentaudiencesfortheevaluationandhowyou
willaccessarangeofviewpoints.Makesureinadvancethatyourcoachesare
willingtouseyourevaluationtoolswhenoperatinginyourorganisation.
Designingevaluationinpartnershipwithexternalcoachesbeforethecoaching
beginscanworkwell.Minimiseresistancetotheevaluationbylettingparticipants
andmanagersknowbeforethecoachingstartswhatevaluationmeasureswillbe
usedandhowtheywillbeexpectedtocontribute.
Itisdifficulttoevaluatethepotentialofcoachingforculturalorothersystemic
change.
Effectiveevaluationisnotjustaoneoffactivityattheendoftheprocessbutitcan
bereallyusefulasaninterimstocktakeofnecessaryimprovementsneededtoa
programme.
6.2.2 Methods for collecting information
Considerthecostbenefitofeachevaluationmethodieassessthepotentialvalueof
thefindingswiththerisksingeneratingthefindingsandthecostofeachevaluation
methodunderconsideration.
Attitudesurveysareasimpleandnonresourceintensivemethodofcollecting
reactionstocoaching.Climatesurveyscanbeusefulinidentifyingchangesinsoft
skillareassuchascommunication,andareespeciallyrelevanttoorganisations
implementingcoachingasastyleofmanagement.Keepsurveyquestionnaires
shorttogetabetterresponse.Responserateswillincreasefurtherwithreminders
andchasing.Theycanbeimprovedbyaskingcoachesorlinemanagerstodistribute
andcollectquestionnaires.Becautiousininterpretingsurveyfindingswherethe
sizeofsurveyedpopulationissmall.
Facetofaceinterviewsenablebehaviourchangetobeexploredinmoredepth,
althoughtelephoneinterviewscanalsogeneratedetailedinformationandallow
probingandcanbealessexpensivealternative.Multipleviewpointsandmultiple
datacollectionmethodsareessentialwhenitcomestomeasuringperceptionsof
behaviouralchange.Whendesigningimpactquestionnairesthatlookatbehaviour
changemakesureyourratingsscaleallowsforthepossibilitythatchangesmaybe
perceivedasnegativeratherthanpositive.
Dontunderestimatethepowerofacoupleofimpressiveindividualanecdotal
storiesorlearningvignettesinpersuadingseniorcolleaguesthatcoachingwasa
worthwhileinvestment.
Ifbottomlinebusinessresultsarewhatyouwant,focusingononekeybusiness
indicatorcanbeasimpleapproachyieldingstraightforwardresults.Itisbetterto
planhowtoevaluatethecoachingbeforestartingtheprogrammesohardbaseline
datacanbecollected.Itisnotalwaysnecessarytocomplicatethingsbycalculating
RoI.Measuringwhereitiseasiestprovidesreasonableevidence,meaningyoucan
avoidtheexpenseofmeasuringbenefitelsewherewhereitmaybedifficultto
identifyanappropriatemeasure.IfyoudogodowntheRoIrouteyouwillneed
significantfinancialresourcesandstatisticalcompetenceatyourdisposal.
Comparingtheresultsamongcoachedwithacontrolgroupofnoncoached
individualscanbeaveryeffectiveapproach.Beingabletocomparebeforecoaching
withaftercoachingresultscanalsobeseenbysomeasmorecrediblethanexamining
postcoachingdataalone.Ifyoudonthaveacontrolyouwillneedsomeformof
benchmarkforcomparativeanalysistoassesswhethertheactivityinquestionis
relativelyeffective.
Bibliography
CarterA(2001),Executivecoaching:Inspiringperformanceatwork,IESReport379
CarterA(2004),Practicalmethodsforevaluatingcoaching,TrainingJournal,January
CIPD(2004),Aguidetocoachingandbuyingcoachingservices,CharteredInstitutefor
PersonnelandDevelopment
CIPD(2004),CIPDAnnualtraininganddevelopmentsurvey,CharteredInstitutefor
CIPD(2005),CIPDAnnualtraininganddevelopmentsurvey,CharteredInstitutefor
CIPD(2006),LearningandDevelopmentSurvey2006,CharteredInstituteforPersonneland
Development
ClutterbuckD,MegginsonD(2005),MakingCoachingWork,CharteredInstitutefor
FosterS,LendleJ(1996),Eyemovementsensitisationandreprocessing:fourcasestudies
ofanewtoolforexecutivecoachingandrestoringemployeeperformanceafter
setbacks,ConsultingPsychologyJournal:PracticeandResearch,Vol.48
Garman,AN,WhistonDL,ZlatoperKW(2000),Mediaperceptionsofexecutive
coachingandtheformalpreparationofcoaches,ConsultingPsychologyJournal:
Practice&Research,52
GengerC(1997),Coaching:Theoryandpractice,unpublishedmastersthesis,Universityof
SanFrancisco
GibbS(1994),Evaluatingmentoring,Education&Training,Vol.36,No.5
GibbS,MegginsonD(1993),InsideCorporateMentoringSchemes;ANewAgendaof
Concerns,PersonnelReview,Vol.22,No.1
GrayD(2004),PrinciplesandProcessesincoachingevaluation,InternationalJournalof
MentoringandCoaching,EMCC,Vol.II,Issue2,December
HallD,OtazoK,HolenbeckG(1999),Behindcloseddoors:Whatreallyhappensin
executivecoaching,OrganizationalDynamics,Vol.27
JarvisJ,LaneDA,FilleryTravisA(2006),ThecaseforCoaching,CharteredInstitutefor
JudgeW,CowellJ(1997),Thebravenewworldofexecutivecoaching,BusinessHorizons,
Vol.40,No.4
KearnsP(2006),Doescoachingwork?,TrainingJournal,June,pp.414
KirkpatrickD(1977),Evaluatingtrainingprograms:evidencevs.proof,Trainingand
DevelopmentJournal,Vol.31
KirkpatrickD(1983),Fourstepstomeasuringtrainingeffectiveness,Personnel
Administrator,Vol.28,No.11
LaskeO(1999),Anintegratedmodelofdevelopmentcoaching,ConsultingPsychology
Journal:PracticeandResearch,Vol.51
MaherS,PomerantzS(2003),Thefutureofexecutivecoaching:Analysisfromamarket
lifecycleapproach,InternationalJournalofCoachinginOrganizations,US,Volume
1(2)
McGovernJ,LindemannM,VergaraM,MurphyS,BarkerL,WarrenfeltzR(2001),
Maximisingtheimpactofexecutivecoachingbehaviourchange,organization
outcomesandreturnoninvestment,TheManchesterReview,Vol.6,No.1
OliveroG,BaneD,KopelmanR(1997),Executivecoachingasatransferoftrainingtool:
Effectsonproductivityinapublicagency,PublicPersonnelManagement,Vol.26,No.4
PhillipsJ(1997),Returnoninvestmentintrainingandperformanceimprovementprograms,
GulfPublishingCompany
SherwoodJ(2004),Doescoachingactuallywork?,InternationalJournalofMentoringand
Coaching,EuropeanMentoringandCoachingCouncil,Vol.II,Issue2,December
TamkinP,YarnallJ,KerrinM(2002),Kirkpatrickandbeyond:Areviewofmodelsoftraining
evaluation,IESReport392
WarrenC,(2005),QuantumLeap,PeopleManagement,March
WilsonJ,ElmanN(1990),Organizationalbenefitsofmentoring,AcademyofManagement
Executive,Vol.4,No.4
Appendix 1: Example Record of Objectives
InstituteforEmploymentStudies
Recordofobjectives
nameofparticipant: nameofcoach:
nameoforganisationsponsor(whereapplicable):
startdate:
objectivesforthe successcriteria evaluationmethod

programme (e.g.observation,feedback,
selfreport,learninglog)

NORTH WEST EMPLOYERS
objectivesforthe successcriteria evaluationmethod

programme (e.g.observation,feedback,
selfreport,learninglog)

ThisrecordisforyouandyourIEScoachtohelpyoumakethemostofthe
programmebytailoringsessionstoyourpersonalrequirements.Onecopywillbe
heldatIESforpilotprogrammeevaluationpurposes,andthendestroyed.Youmay
wishtoshareacopywithyourorganisationsponsororcolleagues,butthisisentirely
optional.
Appendix 2: Example Mid-point Telephone

Interview Discussion Guide
LEAP Discussion Guide: Telephone interview
Introduction:
Hello,mynameis.andImcallingfromtheInstituteforEmploymentStudies.Youwillbe
awarethattheLEAPHRcoachingprogrammeyouarehalfwaythroughisapilotand
thereforethereisanevaluationbeingundertaken.Iamundertakingtelephoneinterviewswith
alltheparticipantsinbothpilotregionsaspartofthatevaluation.
Iwastoldthatyouwouldbeexpectingmycall,Ihopethatthisisstillaconvenienttimetoask
youafewquestions.Itwillonlytakefortyfiveminutesatmost.
ItisimportantforyoutounderstandthatIamnotevaluatingyou.IamevaluatingtheLEAP
programmeandwhetherthereareanypositiveoutcomesarisingfromtheprogrammeforeither
individualsortheirauthorities.
Pleasespeakasfranklyasyoucan.Youropinionisalearningopportunitybothforyouin
termsofclarifyingwhatyouaregainingfromtheprogramme,andfortheorganisersin
meetingyourneedsfortheremainderoftheprogramme.
Individual Outcomes:
CanIstartbyaskingsomequestionsaboutanyoutcomesforyoupersonally,ienotyour
authority.First:
1. Whichelementsoftheprogrammesofarhavebeenmostusefultoyoupersonally?
Inwhatwaywasituseful?
Whatdidyoulearn?ExploreknowledgeaboutHR,strategicthinking,personal
learningaboutself?
Whatimpactdidthislearninghaveonyou?iewhatareyounowdoingdifferently?
Explorebehaviouralchange
2. Pleasecanyougivemeanexampleofhowyouaredoingthingsdifferently?
Whatdidyoudo?
Howdidothersreact?
Howwasthisdifferentfromtheirpreviousreactions?
3. Canyougivemeanotherexample?
Hasthischangeinthewayyouaredoingthingsbeensustained?(Ifnot,whynot?)
Whathasbeentheeffectonthosearoundyou?
Whathasbeentheeffectonyourjob?
4. Whichelementsoftheprogrammehavebeenleastusefultoyou?
Why?
Whatcouldbechangedintheprogrammetomakeitmorevaluabletoyou?
5. IfIwastoaskyoutosummarise,whatwouldyousayhasbeenyourbiggest
personalachievementsofarsinceyouhavebeenontheprogramme?
Towhatextentdidtheprogrammecontributetothis?Specificallywhatpercentage
(%)oftheachievementwouldyouattributetobeingontheprogramme?
Organisation Outcomes:
Idliketotalkabitnowaboutthecontextinwhichyouwork,ieyourHRteamandyour
authority.
6. Howsupportivewouldyousayyourorganisationistowardsyourongoing
personalorHRprofessionaldevelopment?
Whoissupportive/unsupportive?
Whatevidenceisthere?
Whatkindofsupportwouldyoulikefromyourorganisation?
7. HowhighlywouldyousayyourorganisationratestheHRfunction?
Whoratesithighly/lesshighly?
Whatevidenceisthere?
8. DoyoubelievethattheprogrammesofarhasbeenusefultotheHRfunctionasa
whole?
Inwhatway?Canyougivemeanexample?ProbeforchangeinHRfunction
profileorpeoplemeasures
Whoapartfromyouhasbenefited?
HowwouldyoudefinetheHRfunctionbenefit?Orexpectedbenefit?
9. Doyoubelievethattheprogrammesofarhasbeeninanywayusefultoyour
authorityasawhole?
Inwhatway?Canyougivemeanexample?Probeforchangeinpeople
managementpracticeorpeoplestrategy.
WhoapartfromyouoryourHRfunctionhasbenefited?
Howwouldyoudefinetheorganisationbenefit?Orexpectedbenefit?
10. IfIwastoaskyoutosummarisewhatwouldyoupointtoasthebiggest
achievementforyourHRfunctionoryourauthoritysofarsinceyouhavebeenon
theprogramme?
Towhatextentdidtheprogrammecontributetothis?Specificallywhatpercentage
(%)oftheachievementwouldyouattributetotheprogramme?
LEAP programme processes:

Ivecoveredtwoofthethreemostimportantaspectsfrommypointofview.Thefinalsetof
questionsrelatestoyourcoachandthewaytheprogrammehasbeendelivered.Firstly:
11. Canyoubrieflydescribe,inacoupleofsentences,yourfeelingsaboutthe
programmeoverall?
Wasteoftime?Useful?Worthwhile?
12. Howwouldyourateyourcoachingsessions,usingascaleof110with1being
lowand10beinghighiehighlyeffective?
Skillsofyourcoach
Howcomfortableyoufeelwithyourcoach?
Contentofsessions?
Careandfocusofthecoachonyourneeds?
Amountofstretchandchallengefromcoach?
Yourabilitytoapplythecoachingsessionswithinyourwork?
Yourcommitmenttoapplyingyourlearning?
Timingofsessions?
Venue(s)ofsessions?
13. Whatcouldbedonetoimprovethecoachingsessions?
Bythecoach?
Byyou?
14. Howwouldyouratesomeotheraspectsoftheprogramme,onceagainusinga
scaleof110with1beinglowand10beinghigh?
Costofprogramme?
Administrationofprogramme?
Firstevent?
Supportfromyourorganisation?
Supportfromyourregion?
15. Lookingaheadtotherestofyourprogramme,doyouhaveanyothersuggestions
forchangesthatyouwouldliketheorganiserstoknowabout?
16. Doyourequireanyfurthersupport?
Whoshouldprovidethis?Yourregion,EmployersOrganisation,yourcoach?
17. Arethereanyotherissuesyouwouldliketoraisebeforewefinish?
Thank You
Appendix 3: Example Vignette/

Success Story
JanSounessHeadofHR,WarringtonBoroughCouncil
operationalactivitiesanddidnotfocus
sufficientlyonstrategicmatters.Jans
personallearninghasfocusedondeveloping
theabilitytostepbackandenableothers
withintheteamtocomeforward,which
ultimatelywasseenaschangingtheculture
ofthefunction.
Amongherobjectives,Janwasseekingto
developthecapacitywithintheHRCommunity
withintheauthoritybydevelopingthe
consultancyskillsofstaffwithinthedivision.
JanfeltthattheProgrammehadchallenged
Thisvignette,producedbytheInstitutefor themindsetwithintheHRfunctionand
EmploymentStudiesfortheEmployers helpedtoinstilconfidencetoattempttasks
Organisationforlocalgovernmentseeksto differentlyandmorecreatively,withthe
providesomeillustrativeexamplesofthe acceptancethatsometimesthismaynotwork.
overallimpactoftheHRCoaching
SincebeginningtheProgramme,workshops
Programmeonsomeoftheindividual
withintheHRfunctionhavehelpedto
participants.Theseareintendedtobe
identifythechallengesthatneedtobefaced,
successstoriesandillustratetheexperiences
andalsohighlightedthestrengthsoftheteam
ofparticipantsatthemidtermstageofthe
andareaswheretheycanandindeedare
Programme.Eachoftheparticipantsnamed
makingsignificantstrategiccontributionsto
hasgivenpermissionfortheirexperiencesto
theauthorityasawhole.
bemadeavailableforpublicitypurposes.
OneofJansmostsignificantpersonal
ForJanSouness,theHRCoachingProgramme
achievementswasthesecuringofsupportto
wasidentifiedasprovidingavital
influenceandchangetheseniormanagement
opportunitytolookatthesupportthatis
structureandworkmorecollaborativelyina
neededtoachieveobjectivesandthelong
strategicmanner.Jannotedthatthe
termvision,bothinrelationtoherroleand
Programmewasseenasgivingthisprocess
thatoftheHRfunctionasawhole.
moregravitasandshewasabletoattribute
PriortoparticipatingintheProgramme,Jan approximately25percentofthissuccessto
feltthatthefunctionalltooreadilyjumpedinto theProgramme.
Appendix 4: Example End-of-programme

Impact Questionnaire for Participants
LEAP PHASE 2: PARTICIPANTS QUESTIONNAIRE
How do I complete this questionnaire? You can either complete the questionnaire on screen and
email your response to jonny.gifford@employment-studies.co.uk or you can print off the
questionnaire and send your response by post to Jonny Gifford, IES, Mantell Building, Falmer,
Brighton, BN1 9RF.
Please note: if you complete and post a printed questionnaire, it is important that you write your
name on the questionnaire, so that we know that you have responded. Please return your
completed questionnaire by 15 April at the latest.
If you have any queries, please contact Jonny Gifford on tel. 01273 873701 or Gwen Leeming on
tel. 01273 678186.
The information provided in this questionnaire will be fully anonymised.
Part one: How has the LEAP coaching programme impacted on you
1. Pleaseindicateyourprimaryobjective(s)inembarkingontheLEAPcoaching
programme:
Assistingwithpersonaldevelopmentneeds
Assistinginyourowncareerplanning
SupportforchangingHR/ODfunction
Supportfororganisationalinitiatives
Other(pleasespecify)
2. Howwelldidtheprogrammehelpyoumeetthisobjective/theseobjectives?(Please
tickoneboxonly)
Notatallwell Notverywell Reasonablywell Verywell Extremelywell
3. Pleaseputatickinoneboxforeachstatementtoindicatehowyouworkdifferently
asaresultoftheLEAPcoachingprogramme.
Very Very
HavinggonethroughtheLEAPcoaching Strongly Strongly
strongly Disagree Agree strongly
programme: disagree agree
disagree agree
I give my own development more priority
I am more aware of the impact of my
behaviour on others
I more aware of my strengths and
weaknesses
I more aware of my motivations
I use a wider range of leadership styles for
different circumstances
I am more effective in dealing with difficult
people and situations
I tackle the management of the HR team
better
Very Very
HavinggonethroughtheLEAPcoaching Strongly Strongly
strongly Disagree Agree strongly
programme: disagree agree
disagree agree
I have improved my personal profile as a
senior HR/OD manager
I am more effective at developing strategy
and planning
I manage my own team better
I have increased my personal effectiveness
I am more confident in my ability to lead
change
I am more effective in managing
stakeholders expectations
I put more effort into developing and
sharing a vision for people management
I have extended my networks inside and
outside the organisation
I make a greater effort to find innovative
approaches
I make a greater effort to find flexible
ways of working
I am better at encouraging managed risk
taking
I am a better corporate manager
4. PleasecommentontheimpacttheLEAPcoachingprogrammehashad:
Onyou: ........................................................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................
Onyourfunction: .......................................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................
Onyourorganisation:................................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................
Part two: The coaching processes
5. Wasthisyourfirstexperienceofworkingwithacoach? yes no
6. Thecoachingprogrammeofferedsixsessionsoveraperiodofsixmonths.Please
commenton:
Thenumberofsessions(pleasetickoneboxonly)
toomany aboutright toofew
Thefrequencyofthesessions(pleasetickoneboxonly)
toooften aboutright notoftenenough
Additionalcomments: ...............................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................
7. WouldyourecommendcoachingsessionstootherHR/ODmanagers?
Pleaseshareyourreasons: ........................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................
8. WearekeentoimprovetheadvanceinformationthatwegivetoHRexecutiveson
whatcoachinginvolves.
Pleasecommentontheexperienceofbeingcoached:
.......................................................................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................
Howdoescoachingcomparewithotherdevelopmentopportunitiesyouhave
experienced:
.......................................................................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................
Isthereanyotherinformationthatyouthinkfutureparticipantsmightbenefitfrom
knowinginadvanceabouthowcoachingworks?
.......................................................................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................
9. Howusefuldidyoufindthe360degreefeedbackforthecoachingprocess?(Pleasetick
oneboxonly)
veryuseful partlyuseful neutral notuseful
Pleasecomment: .........................................................................................................................
.......................................................................................................................................................
10. Didyou,oranyofyourcolleagueshaveanydifficultyusingthe360degreetool?
yes no
11. Doyouhaveanysuggestionsforimprovingthe360degreefeedbackelementofthe
LEAPcoachingprogramme?
.......................................................................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................
12. HowwouldyouliketofollowuptheLEAPcoachingprogramme?(Pleasetickonebox
only)
Furthercoachingformyself
Networkingevents
Traintobeacoach
Usecoachinginownorganisation
Other(pleasespecify)...................................................
13. HowwouldyouratetheoveralladministrationoftheLEAPcoachingprogramme?
(Pleasetickoneboxonly)
verypoor poor quitepoor quitegood good verygood
14. DoyouhaveanyadditionalcommentsabouttheLEAPcoachingprogramme?
.......................................................................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................
Thank you for completing this questionnaire

Please return this questionnaire to: Jonny Gifford, The Institute for Employment Studies,
Mantell Building, University of Sussex, Falmer, Brighton, BN1 9RF


Impact Questionnaire for Sponsors
LEAP PHASE 2: SPONSOR QUESTIONNAIRE
Your colleague is nearing completion of a six-month coaching programme (LEAP) and has named
you as their sponsor. The programme is being run by the Employers Organisation for local
government (EO) in partnership with the Institute for Employment Studies (IES), which is also
evaluating the programme.
As part of the evaluation, we would like you to complete this short questionnaire on changes in
your colleagues awareness, behaviour and effectiveness since starting the programme. We would
be grateful if you could return your completed questionnaire by 31 March.
How do I complete this questionnaire? You can either complete the questionnaire on screen and
email your response to jonny.gifford@employment-studies.co.uk or you can print off the
Brighton, BN1 9RF.
Please note: if you complete and post a printed questionnaire, it is important that you write your
name on the questionnaire, so that we know that you have responded.
If you have any queries, please contact Jonny Gifford on tel. 01273 873701 or Gwen Leeming on
tel. 01273 678186.
1. Howlonghaveyouworkedwiththecolleagueinquestion? yrs mths
2. Pleaseputatickinoneboxforeachstatementabouthowyoubelieveyourcolleague
worksdifferentlyasaresultoftheLEAPcoachingprogramme.
HavinggonethroughtheLEAP Very Very
Strongly Strongly Cannot
coachingprogramme,Ibelieve strongly Disagree Agree strongly
disagree agree comment
thatmycolleague: disagree agree
Gives his/her own development
more priority
Is more aware of the impact of
his/her behaviour on others
Is more aware of his/her
strengths and weaknesses
motivations
Uses a wider range of
leadership styles in different
circumstances
Is more effective in dealing
with difficult people and
situations
Tackles the management of HR
team better
Has improved his/her personal
profile as a senior HR/OD
manager
Is more effective at developing
strategy and planning
Manages his/her team better
HavinggonethroughtheLEAP Very Very
coachingprogramme,Ibelieve strongly Disagree Agree strongly
thatmycolleague: disagree agree
Has increased his/her personal
effectiveness
Is more confident in his/her
ability to lead change
Is more effective in managing
stakeholders expectations
Puts more effort into
developing and sharing a vision
for people management
Has extended his/her networks
inside and outside the
organisation
Makes a greater effort to find
innovative approaches
Makes a greater effort to find
flexible ways of working
Is better at encouraging
managed risk taking
Is a better corporate manager
3. Onreflection,whichaspectsofyourcolleaguesbehaviour:
HavechangedmostnoticeablysincestartingtheLEAPcoachingprogramme?
.......................................................................................................................................................
.......................................................................................................................................................
Werealreadyparticularstrengths?
.......................................................................................................................................................
.......................................................................................................................................................
4. Whatfurtherchangeswouldyoumostliketoseeinyourcolleague?
.......................................................................................................................................................
.......................................................................................................................................................
5. DoyouhaveanyotherfeedbackabouttheimpacttheLEAPcoachingprogrammehas
hadonyourcolleagueoryourorganisation?
.......................................................................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................
.......................................................................................................................................................

Please return this questionnaire to: Jonny Gifford, The Institute for Employment Studies, Mantell
Building, University of Sussex, Falmer, Brighton, BN1 9RF


Impact Questionnaire for Coaches
LEAP PHASE 2: COACHES QUESTIONNAIRE
How do I complete this questionnaire? You can either complete the questionnaire on screen
and email your response to jonny.gifford@employment-studies.co.uk or you can print off the
Brighton, BN1 9RF.
Please note: if you complete and post a printed questionnaire, it is important that you write
your name on the questionnaire, so that we know that you have responded. Please return your
completed questionnaire by 15 April at the latest.
If you have any queries, please contact Jonny Gifford on tel. 01273 873701 or Gwen Leeming
on tel. 01273 678186.
Please assign the two participants you have coached to Participant 1 or Participant 2 (we do not
need to know their names). Then take a moment to think about each of them in turn and
indicate the extent to which you agree with the statements below about the impact of the LEAP
coaching programme on them.
Part one: How has the LEAP coaching programme impacted on your participants
Participant 1
1. PleaseputatickinoneboxforeachstatementabouthowyoubelieveParticipant1
Havinggonethroughthe Very Very
LEAPcoachingprogramme, strongly Disagree Agree strongly
IbelievethatP1: disagree agree
Gives his/her own
development more priority
Is more aware of the impact
of his/her behaviour on
others
motivations
leadership styles in
situations
Tackles the management of
HR team better
Has improved his/her
personal profile as a senior
HR/OD manager
Is more effective at
developing strategy and
planning


Manages his/her team
better
Has increased his/her
personal effectiveness
Is more effective in
managing stakeholders
expectations
developing and sharing a
vision for people
management
Has extended his/her
networks inside and outside
the organisation
Makes a greater effort to
find innovative approaches
find flexible ways of
working
managed risk taking
Is a better corporate
manager
2. WhatarethefivemostnoticeableimprovementsthatyouhaveseeninParticipant1:
a. .......................................................................................................................................
b. .......................................................................................................................................
c.........................................................................................................................................
d. .......................................................................................................................................
e.........................................................................................................................................
Participant 2
3. PleaseputatickinoneboxforeachstatementabouthowyoubelieveParticipant2

Gives his/her own
development more priority
Is more aware of the impact
of his/her behaviour on
others
motivations
leadership styles in
situations
Tackles the management of
HR team better
Has improved his/her
personal profile as a senior
HR/OD manager
Is more effective at
developing strategy and
planning
Manages his/her team
better
Has increased his/her
personal effectiveness
Is more effective in
managing stakeholders
expectations
developing and sharing a
vision for people
management
Has extended his/her
networks inside and outside
the organisation
find innovative approaches
find flexible ways of
working
managed risk taking
Is a better corporate
manager
4. WhatarethefivemostnoticeableimprovementsthatyouhaveseeninParticipant2:
a. ......................................................................................................................................
b. ......................................................................................................................................
c. ......................................................................................................................................
d.......................................................................................................................................
Part two: Programme processes, delivery and evaluation
5. Thecoachingprogrammeofferedsixsessionsoveraperiodofsixmonths.Please
commenton:
Thenumberofsessionsoffered(pleasetickoneboxonly)
toomany aboutright toofew

Thefrequencyofthesessions(pleasetickoneboxonly)
toooften aboutright notoftenenough
Additionalcomments: ........................................................................................................
.................................................................................................................................................
6. WearekeentoimprovethematchingprocessofcoachestoHRexecutivesin
future.Isthereanyotherinformationthatyouthinkfutureparticipantsmight
benefitfromknowinginadvance,regarding:
Howcoachingworks?
.................................................................................................................................................
.................................................................................................................................................
Yourcoachingstyle?
.................................................................................................................................................
.................................................................................................................................................
7. Asacoach,whatfurtherassistanceorsupportwouldyouliketoreceivefrom
participants,theEOortheirregions,orIES?
.................................................................................................................................................
.................................................................................................................................................
8. DoyouhaveanyadditionalcommentsabouttheLEAPcoachingprogramme?

Please return this questionnaire to: Jonny Gifford, The Institute for Employment Studies,
Mantell Building, University of Sussex, Falmer, Brighton, BN1 9RF

430 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

430 PDF

Uploaded by

Copyright:

Available Formats

PRACTICAL METHODS FOR

The Institute for Employment Studies

The IES HR Network

The IES study

Likely Coachees Line Coaches Documents

Source: IES, 2006

1.1 Background to the research project

2 Practical Methods for Evaluating Coaching

1.3 Structure of this report

Institute for Employment Studies 3

4 Practical Methods for Evaluating Coaching

2 Does Coaching Actually Work?

2.1 Is coaching an effective tool?

Institute for Employment Studies 5

2.2 What can we learn from mentoring?

6 Practical Methods for Evaluating Coaching

2.3 What about return on investment approaches?

Institute for Employment Studies 7

US$96 ,750 US$15 ,000

Benefit () Cost of coaching

8 Practical Methods for Evaluating Coaching

Table 2.1: Advantages and disadvantages of RoI Approaches

Advantages of RoI Disadvantages of RoI

Source: IES, 2006

2.4 Issues in evaluating coaching

Institute for Employment Studies 9

2.4.1 Defining outcomes and evaluation measures

10 Practical Methods for Evaluating Coaching

Figure 2.2: Summary of traditional evaluation approaches

2.4.2 Evaluating the bigger picture

Institute for Employment Studies 11

2.4.3 Perspectives on evaluating outcomes

12 Practical Methods for Evaluating Coaching

2.4.4 Timing of evaluation and availability of information

2.4.5 Qualities of the coach and coaching relationship

Figure 2.4: Enabling factors to achieving success in coaching

skills motivation to learn

ability learning style

Institute for Employment Studies 13

2.5 Summary of suggestions arising from the literature

14 Practical Methods for Evaluating Coaching

3 Organisational Examples of Evaluation

Institute for Employment Studies 15

3.1 T-mobile identifying priority areas for future

3.1.1 Context of the coaching programme

3.1.2 Purpose of the evaluation

3.1.3 Evaluation methods

16 Practical Methods for Evaluating Coaching

3.1.5 Tips from T-mobile illustration

Response rates will increase further with reminders and chasing.

3.2 Corus tracking achievement of objectives

3.2.1 Context of the coaching programme

3.2.2 Purpose of the evaluation

Institute for Employment Studies 17

3.2.3 Evaluation methods

3.2.5 Tips from Corus illustration

18 Practical Methods for Evaluating Coaching

3.3 NHS in Wales learning about coaching as a tool for

3.3.1 Context of the coaching programme

3.3.2 Purpose of the evaluation

3.3.3 Evaluation methods

Institute for Employment Studies 19

3.3.5 Tips from NHS in Wales illustration