6/27/2015 www.medscape.

com/viewarticle/845632_print

www.medscape.com

What'sMeasuredMatters:MeasuringPerformance
inAnaesthesia
F.KiernanD.J.Buggy
BrJAnaesth.2015114(6):869871.
Deliveringvalueformoneyisanincreasinglyinsistentdemandinhealthpolicy,drivenbyfinancialconstraint,
increasedpatientexpectations,anageingpopulation,andmoreexpensivetechnologies.Healthperformance
measurementiscentraltoensuringthatthehealthcareweprovideisofvaluetoboththosewhopayforit,andthose
whouseit.Itimproveshealthsystems,bycollectinginformationonpopulationhealth,therebyallowingthe
appropriateallocationofresourcesbasedontheselevelsofhealth,andprovidesthestimulustoimprovethequalityof
healthcareprovided.HealthPerformanceMeasurementisanecessarypartofensuringthatthehealthsystemis
accountabletoitscitizens. [1]

Theconcernsvoicedovertheroleofperformancemeasurementinhealthcare,rarelyrelatetotheideaofthatclinical
datashouldbecollectedinordertoensureaminimumstandardofcare.Instead,clinicianconcernsarefocusedon
thechoiceofindicatorsthatisusedtoassessthisqualityofcare,andhowtheresultsshouldbeused.Oursurgical
colleagueshavebeenattheforefrontofdebatesregardingthepublicreportingofresultsofsurgicalperformance
measurement.Measuringclinicianperformanceisnowcommonplaceinmanyhighincomecountries, [1]andthe
publicreportingofanaesthesiarelatedoutcomemeasuresissuretofollow.

Partofourhesitationinmeasuringandcomparingtheperformanceofanaesthetistsasphysiciansandhealthcare
providers,isbecauseofalongstandingbeliefthathighqualitydatacannotbecollectedinlargeenoughamounts,to
allowaccuratecomparison. [2]Thecollectionofeithersmallamountsofhighqualitydata,orlargeamountsofpoor
qualitydata,isknowntoleadtoinaccuraciesandpotentialgaming. [3]However,ourabilitytocollectaccurateand
appropriatemeasuresofperformancehaschangedwiththeadventofBigData.BigDataallowsustocollectdata
fromheterogeneousdatasets,integratethisdata,andusepredictiveanalyticstodeterminemostefficientand
effectivemeansofcare.Asaresultofmajoradvancesindigitaltechnology,wecananalyseandcomparecostand
clinicaleffectivenessdataacrossjurisdictionsandovertime,performaccurateriskadjustment,anddelivertheresults
tothosewhousethem.Theobstaclestohighqualitydatacollectionarenolongertechnological.

Perhapsthetworemainingbarrierstoaccurate,relevantdatacollectionareuncertaintyoverchoiceofindicatorsthat
bestreflectpatientoutcome,andoptimumuseofthatdata.Whenincentives,eitherfinancialorreputational,are
attachedtoanindicator,itisperceivedasmoreimportantthanunmeasuredaspectsofcare.Theindicator
subsequentlyreceivesadisproportionateamountofattentionfromtheprovider,aphenomenonknownas'what's
measurediswhatmatters'. [4]Evidencefromprimarycarestudieshasshownadecreaseinqualityforpatientswith
asthmaandheartdisease,whosecarewasnotassociatedwithanincentive. [5]Thishighlightsthatdelivering
appropriatecarerequiresthatwemeasurewhatmatterstopatients,ratherthanmerelythedatathatitiseasiestto
collect.Whiledataonprocessindicatorsinanaesthesia,(e.g.theadministrationofantiemetics,applicationof
thermoregulatorydevices,theperformanceofanairwayassessmentetc.),maybeeasytocollect,theymatterless
topatientsthanthetrueoutcomemeasuresofanormalrecovery,nomorbidityandabsenceofvomitingandpain.
Indeed,qualitativedataonpatientreportedoutcomemeasures,havedemonstratedthatpostoperativevomitingis
rankedasthehighestconcernforpatientsundergoingambulatorysurgery,moresoeventhanpostoperativepain. [6]It
wouldseemthatemesismightbethe'HolyGrail'ofoutcomemeasuresithasbeenvalidatedasarelevantpatient
outcome,andiseasytocollectforalargepopulationofpatients.However,totakeasingleexampleofhow
performancemeasurementdatamaybeabused,reportingvomitingoutcomesonapatientgroupundergoing
ambulatorysurgeryonlytakesintoaccountthislowriskgroup,whowouldbeexpectedtohavelowmortality,low
morbidity,andnohospitalstay.Inreality,asystematicreviewof108commonlyusedanaesthesiaqualityindicators,
foundthatonly40%weretrulyvalidated. [7]

http://www.medscape.com/viewarticle/845632_print 1/4
6/27/2015 www.medscape.com/viewarticle/845632_print

Ifoutcomedatainsteadfocusesonthehighriskgroups,thenmortality,morbidity,delayeddischarge,andunplanned
ICUadmissionwouldseemtobesuitablealternativemeasuresofperformance.Analysisofpostoperativeoutcomes
fortheelderlysuggeststhattheriskofmyocardialinfarction,stroke,delirium,andpulmonarycomplicationscanbe
decreasedthroughqualityimprovementmethods. [8,9]However,anaturalhesitationinusingtheseoutcome
measures,restsonevidencethattheymaynothaveastrongrelationshipwiththequalityofcareprovided.Indeedthe
'signaltonoise'ratioformortalityhasbeenshowntobetoolowforittobeusedasamarkerofquality. [10]
Furthermore,ashealthismultidimensional,andinhospitalcareismultidisciplinary,mortalitycannotberelatedtothe
performanceofanindividualanaesthetist,orteamofanaesthetists.Indeedmortalityisoftenrelatedtofactorsbeyond
thecontrolofclinicians,includingthelevelofeducationofnursingcolleagues, [11]andthemanagementstructureof
thehospital. [12]Inaddition,thereisalackofconsensusevenamonganaesthetistsregardingtheappropriatenessof
thesemeasures.Whileunplannedadmissionstointensivecareaftersurgery,havebeendescribedbysomeasbeing
aninappropriatewayofexaminingquality, [13]othershavesuggestedthattheyareanaccuratemeansofmeasuring
performanceandimprovingquality. [14]

Thesecondobstacleconcernshowthesedataareused.Berwick'sframeworkforqualityimprovementdescribeshow
thepublicreportingofoutcomedataleadstoimprovementsinqualityfromoneoftwomethodseitherpatients
selectbetterprovidersofcare,orthedataprovidesinformationonareasofunderperformance,leadingtoastimulus
forimprovementfromthedeficientproviders. [15]Thisassumes,however,thatthecareprovidedisactuallyamenable
toimprovementsinquality.TheOECDestimatesthatonlyhalfofdeathsfromischaemicheartdiseasecanbe
consideredamenabletohealthcare,whiledeathfromhypertensivediseasescanbepreventedbyimprovementsin
qualitywithintheentirehealthcaresystem. [16]Attachingahighpoweredincentivetotheseoutcomemeasures,could
resultinperverseoutcomes.TheBMAdemonstratedthatattachingincentivestowaitingtimesinEmergency
Departmentshadanegativeeffectoncareinotherareasofthehospital,includingthecancellationofemergency
lists. [17]Misrepresentationofdataisaknownfeatureofhealthcaresystemsthatrelyonthepublicreportingofdata.
[18]WhileGaming,(definedastheinfluenceofmortalitybydeliberatelyalteringvariablesotherthanclinicalquality),

hasrarelybeendescribedinanaesthesia,41%ofthereportedreductionsinsurgicalmortality,intheNewYork
CardiacSurgeryReportingSystem(CSRS)werebecauseofgamingratherthanimprovementsincare. [19]'Cream
skimming',(i.e.therefusaloftreatmenttohighriskpatients,whoaremorelikelytohavepooroutcomes),isanother
potentialmisuseofhealthcareperformancemeasurement,withtheCSRSalsofindingthat62%ofcardiacsurgeons
refusedtotreathigherriskpatients,becauseofreputationalrisksfrompublicreportingofpotentialadverseoutcomes.

Providinginformationonperformancemeasures,withoutadequateexplanationofthecontext,isaflawedmeansof
providingaccountablehealthcare,andindeedislittlemorethanarewardpunishmentsystemforclinicians.
Comparisonsbetweenthepublicandprivatesectorshavefoundthatpublicationofrawhealthcareperformance
measurementindices,hasaparticularlyperverseeffectonthepublicsector,byadverselyaffectingtheprofessional
attitudeofproviders, [20]leadingtoadestructionofthepatientdoctorrelationship.

Furthermore,whiletheprovisionofdataintheformofaleaguetableisaresponsetothebeliefthatprovidingpatients
withmoreinformationwillhelpthemmakebetterdecisions,inreality,informationasymmetryinhealthcare,(i.e.the
discrepancyinknowledgethatexistsbetweenpatientsandclinicians)meansthatpatientsareunlikelytobeableto
distinguishbetweenrelevantandirrelevantinformation.Particularattentionshouldbepaidtotheneedsofthosewho
areleastlikelytouseleaguetablestomakechoicestheelderly,recentimmigrantsandthosewithlower
educationallevels.Cliniciansshouldnotonlybeinvolvedindeterminingtheindicatorusedtoassesstheir
performance,butalsothemannerinwhichthisiscommunicatedtothepublicandtheirpatients.

Ultimately,highqualitydatacollection,performancemeasurement,andappropriatereportingofthatperformance
measurementarenecessarytoensurecontinuedimprovementinaccountable,equitablehealthcareoutcomes.In
addition,aneagernesstoreportthesemeasurespubliclymaydemonstratethatthehealthcaresystemiscommitted
toacultureofaccountabilityandtransparency.However,improvementsinthehealthcaresystem,bothinqualityand
resourceallocation,shouldbebasedonclinicalassessments,andaccuratedatacollection,ratherthanpolitical
decisions.'What'smeasuredmatters',thereforeweneedtopaycloseattentiontoindicatorsthatmattertomostto

http://www.medscape.com/viewarticle/845632_print 2/4
6/27/2015 www.medscape.com/viewarticle/845632_print

patients,butwhicharealsopreventable.Efficientdatacollectionmustaccuratelyreflectperformance,andmust
includemeasurestodecreaseperverseoutcomessuchasupcoding,orrecordingpatientsasbeinghigherriskthan
theircomorbiditiessuggest,andcreamskimming.

Thecareprovidedbyanaesthetistsiscomplex,asareourpatients.Thereforetheanalysisofourcareshouldnotbe
consideredtobeanylesscomplex.Whileprocessleveldatamaybeeasytocollect,itprovideslittleinformationon
theeffectoftreatmentonpatients.Intheeraof'bigdata',wecanmovetowardsthecollectionofoutcomemeasures
usingaccurateinformation,enablingustoassessoutcomesthatarepotentiallypreventable,andthatmatterto
patients.

References

1. DerickBrinkerhoff.AccountabilityandHealthSystems:Overview,Framework,andStrategies.2003.Available
fromhttp://www.who.int/management/partnerships/accountability/AccountabilityHealthSystemsOverview.pdf
(accessed13December2014)

2. ScholleSH,RoskiJ,AdamsJL,etal.Benchmarkingphysicianperformance:reliabilityofindividualand
compositemeasures.AmJManagCare200814:8338

3. HolmstromB,MilgromP.MultitaskPrincipalAgentAnalyses:IncentiveContracts,AssetOwnership,andJob
Design.JLaw,EconomicsandOrganization,OxfordUniversityPress,19917:2452,SpecialI

4. BevanG,HoodC.What'smeasurediswhatmatters:targetsandgamingintheEnglishpublichealthcare
system.PublicAdm200684:51738

5. CampbellSM,ReevesD,KontopantelisE,SibbaldB,RolandM.EffectsofPayforPerformanceontheQuality
ofPrimaryCareinEngland.NEnglJMed2009361:36878

6. MacarioA,WeingerM,CarneyS,KimA.Whichclinicalanesthesiaoutcomesareimportanttoavoid?The
perspectiveofpatients.AnesthAnalg199989:6528

7. HallerG,StoelwinderJ,MylesPS,McNeilJ.Qualityandsafetyindicatorsinanesthesia:asystematicreview.
Anesthesiology2009110:115875

8. NeufeldKJ,LeoutsakosJM,SieberFE,etal.OutcomesofEarlyDeliriumDiagnosisAfterGeneralAnesthesia
intheElderly.AnesthAnalg2013117:4718

9. SieberFE,RyanBarnettS.PreventingPostoperativeComplicationsintheElderly.AnesthesiolClin201129:
8397

10. ScottIA,BrandCA,PhelpsGE,BarkerAL,CameronPA.Usinghospitalstandardisedmortalityratiosto
assessqualityofcareproceedwithextremecaution.MedJAust2011194:6458

11. AikenLH,SloaneDM,BruyneelL,VandenHeedeK,etal.,fortheRN4CASTconsortium.Nursestaffingand
educationandhospitalmortalityinnineEuropeancountries:aretrospectiveobservationalstudy.Lancet2014
383:182430

12. WestMA,BorrillC,DawsonJ.Thelinkbetweenthemanagementofemployeesandpatientmortalityinacute
hospitals.IntJHumResourManage200213:1299310

13. PiercyM,LauS,LohE,ReidD,SantamariaJ,MackayP.Unplannedadmissiontotheintensivecareunitin
postoperativepatientsanindicatorofqualityofanaestheticcare?AnaesthIntensiveCare200634:5928

http://www.medscape.com/viewarticle/845632_print 3/4
6/27/2015 www.medscape.com/viewarticle/845632_print

14. VlayenA,VerelstS,BeekeringGE,SchrootenW,HellingsJ,ClaesN.ExploringunplannedICUadmissions:
ASystematicReview.JBIDatabaseSystRevImplementRep20119:92559

15. BerwickD.DisseminatingInnovationsinHealthCare.JAMA2003289:196975

16. EllenNolte.Theconceptof'avoidable'mortality.2009.Availablefromhttp://www.oecd.org/health/health
systems/43894416.pdf(accessed5November2014)

17. HealthPolicyandEconomicResearchUnit.BMAsurveyofA&Ewaitingtimes.2005.
http://www.collemergencymed.ac.uk/code/document.asp?ID=3156(accessed10November2014)

18. SmithP.Ontheunintendedconsequencesofpublishingperformancedatainthepublicsector.IntJPublic
Adm19952/3:277310

19. GreenJ,WintfieldN.Reportcardsoncardiacsurgeons:assessingNewYorkstate'sapproach.NEnglJMed
1995332:122932

20. NoordegraafM,vanderMeulenMJ.ProfessionalPowerPlay.OrganizingManagementinHealthCare.Public
Adm200886:105569

BrJAnaesth.2015114(6):869871.2015OxfordUniversityPress

Copyright2007TheBoardofManagementandTrusteesoftheBritishJournalofAnaesthesia.PublishedbyOxford
UniversityPress.Allrightsreserved.

http://www.medscape.com/viewarticle/845632_print 4/4