Professional Documents
Culture Documents
Critical Appraisal of Quantitative Research: November 2010 Sarah Lawson Research & Learning Support
Critical Appraisal of Quantitative Research: November 2010 Sarah Lawson Research & Learning Support
Learningobjectives
Understandtheprinciplesofcriticalappraisalanditsrole inevidencebasedpractice Understandthedifferentlevelsofquantitativeevidence Beabletoappraisequantitativeresearchandjudgeits validity Beabletoassesstherelevanceofquantitativeresearch totheirownwork Knowaboutresourcesavailabletohelpthemtocritically appraiseresearch Beabletocriticallyappraisequantitativeresearchasa group
Whatisevidencebasedpractice?
Evidencebasedpracticeistheintegrationof individualclinicalexpertise withthe bestavailableexternalclinicalevidence from systematicresearch and patientsvaluesandexpectations
Theevidencebasedpractice(EBP) process.
Decisionorquestionarisingfroma patientscare. Formulateafocusedquestion. Searchforthebestevidence. Appraisetheevidence. Applytheevidence.
EBPinpractice
dependinguponspeciality,between50and80per centofall'medicalactivity'isevidencebased. www.shef.ac.uk/scharr/ir.percent.html
Whydoesevidencefromresearchfail togetintopractice?
75%cannotunderstandthestatistics 70%cannotcriticallyappraisearesearchpaper
UsingresearchforPractice:aUKexperienceofthebarriersscale. Dunn,V.etal.
Whatiscriticalappraisal?
Weighingupevidencetoseehowusefulitis indecisionmaking Balancedassessmentofbenefitsand strengthsofresearchagainstitsflawsand weaknesses Assessresearchprocessandresults Skillthatneedstobepracticedbyallhealth professionalsaspartoftheirwork
WhatcriticalappraisalisNOT
Negativedismissalofanypieceofresearch Assessmentofresultsalone Basedentirelyonstatisticalanalysis Onlytobeundertakenbyresearchers/ statisticians
Whydoweneedtocritically appraise?
Itusuallycomesasasurprisetostudentsto learnthatsome(thepuristswouldsay99%of) publishedarticlesbelonginthebinandshould notbeusedtoinformpractice Greenhalgh 2001
Whydoweneedtocritically appraise?
studieswhichdon'treporttheirmethodsfully overstatethebenefitsoftreatmentsbyaround25% Khanetal.ArchInternmed,1996; Maheretal,Lancet1998. studiesfundedbyapharmaceuticalcompanywere foundto be4timesaslikelytogiveresultsthat werefavourabletothecompanythanindependent studies Lexchin etal,BMJ,2003
Sourcesofbias
poorcontrolgroup/controldosage surrogateoutcomes ignoredropouts modifytriallength misusebaselinestatistics statisticsoverload
HowdoIappraise?
Mostlycommonsense. Youdonthavetobeastatisticalexpert! Checklistshelpyoufocusonthemost importantaspectsofthearticle. Differentchecklistsfordifferenttypesof research. Willhelpyoudecideifresearchisvalidand relevant.
Researchmethods
Quantitative Usesnumbersto describeandanalyse Usefulforfinding preciseanswersto definedquestions Qualitative Useswordstodescribe andanalyse Usefulforfinding detailedinformation aboutpeoples perceptionsand attitudes
Levelsofquantitativeevidence.
Systematicreviews. Randomizedcontrolledtrials. Prospectivestudies(cohortstudies). Retrospectivestudies(casecontrol). Caseseriesandreports Opinionsofrespectedauthorities.
SystematicReviews.
Thoroughsearchofliteraturecarried out. AllRCTs (orotherstudies)onasimilar subjectsynthesisedandsummarised. Metaanalysistocombinestatistical findingsofsimilarstudies.
RandomisedControlledTrials (RCTs)
Normaltreatment/placeboversusnew treatment. Participantsarerandomised. Ifpossibleshouldbedoubleblinded. Intentiontotreatanalysis
Cohort studies
prospective groups (cohorts) exposure to a risk factor followed over a period of time compare rates of development of an outcome of interest Confounding factors and bias
AppraisingRCTs
Recruitment and sample size Randomisation method and controls Confounding factors Blinding Follow-up Intention to treat analysis Censoring
Appraisingsystematicreviews.
Wasathoroughliteraturesearchcarriedout? Publicationbias paperswithmoreinteresting resultsaremorelikelytobe:
Submittedforpublication Acceptedforpublication Publishedinamajorjournal PublishedintheEnglishlanguage
Publicationbias
AllSSRItrialsregisteredwithFDA 37studieswereassessedbyFDAaspositive 36ofthesewerepublished. 22studieswithnegativeorinconclusive resultswerenotpublishedand11were writtenupaspositive. Turneretal.NEJM,2008.
Reviewsingeneralmedicaljournals
50reviewsin4majorjournals19856 Nostatementofmethods Summaryinappropriate Currentsystematicreviewsdonotroutinely usescientificmethodstoidentify,assessand synthesiseinformation (Mulrow,1987)
Istheresearchquestionfocused?
Areresultssignificant?
Howwasdatacollected? Whichstatisticalanalyseswereused? Howprecisearetheresults? Howaretheresultspresented?
Intentiontotreatanalyses
Analysingpeople,attheendofthetrial,inthe groupstowhichtheywererandomised,even iftheydidnotreceivetheintended intervention
Statisticalanalyses
Oddsratios,absoluteandrelative risks/benefits,hazardratios/relativehazards Thelikelihoodofsomethinghappeningvs the likelihoodofsomethingnothappening Numberneededtotreat(NNT) Thenumberofpeopleyouwouldneedto treattoseeoneadditionaloccurrenceofa specificbeneficialoutcome
OddsRatioDiagrams.(Blobbograms or
ForestPlots.)
OddsRatioDiagrams
Lineofnoeffect nodifferencebetween treatmentandcontrolgroup Result(blob)totheLeftofthelineofnoeffect =Lessoftheoutcomeinthetreatmentgroup. ResulttotheRightoftheline=Moreofthe outcome. BUT Istheoutcomegoodorbad?
Cardiacdeaths less=good
Smokingcessation more=good
ConfidenceIntervals.
Longerconfidenceinterval=less confidentofresults widerrange. Shorterconfidenceinterval=more confident narrowerrange. Crosseslineofnoeffect/nosignificance= Inconclusiveresults.
Confidenceintervals
PValues.
Pstandsforprobability howlikelyisthe resulttohaveoccurredbychance? Pvalueoflessthan0.05meanslikelihoodof resultsbeingduetochanceislessthan1in20 =statisticallysignificant. Pvaluesandconfidenceintervalsshouldbe consistent
NumberNeededtoTreat
Thenumberofpeopleyouwouldneedto treattoseeoneadditionaloccurrenceofa specificbeneficialoutcome. Thenumberofpatientsthatneedtobe treatedtopreventonebadoutcome.
Areresultsrelevant?
CanIapplytheseresultstomyownpractice? Ismylocalsettingsignificantlydifferent? Arethesefindingsapplicabletomypatients? Arefindingsspecific/detailedenoughtobe applied? Werealloutcomesconsidered?
Thegoodnews!
Someresourceshavealreadybeen criticallyappraisedforyou. Anincreasingnumberofguidelinesand summariesofappraisedevidenceare availableontheinternet.
Summary.
Searchforresourcesthathavealreadybeen appraisedfirst,e.g.Guidelines,Cochranesystematic reviews. Searchdownthroughlevelsofevidence,e.g. systematicreviews,RCTs. Usecheckliststoappraiseresearch. Howcantheseresultsbeputintopractice?
Questions?
39