Show

Home > CHAPTER 6: INCORPORATING QUALITATIVE EVIDENCE IN OR ALONGSIDE

EFFECTIVENESS REVIEWS > 6.4 ASSESSMENT OF QUALITATIVE RESEARCH
6.4 ASSESSMENT OF QUALITATIVE RESEARCH
The application of quality criteria to qualitative research is widely debated, although
many accept the need for clear and transparent approaches for judging the quality or
credibility of research. For example, it has been noted that the distinguishing mark of all
good research is the awareness and acknowledgement of error and, that what flows
from this is the necessity of establishing procedures which will minimise the effect such
errors may have on what counts as knowledge.26 It is less clear whether consensus can
be reached over an agreed set of principles for judging quality. Qualitative researchers
from different disciplines and from different theoretical backgrounds may have different
criteria for assessing the quality of a study.27 Some argue that quality cannot be
determined by following prescribed formulas 28 or that it is fruitless to try to set standards
for qualitative research as such. 29 Others, accepting the need for structured procedures,
argue for more rigorous use and reporting of analytical approaches which improve
reliability and validity.30 Others have suggested there are general questions that can be
asked to judge validity and reliability in qualitative research, but that these are not
readily codified.31 It has also been argued that quality assessment should take account of
theory in the design of the research, analysis, and interpretation of the data.32
6.4.1 How should quality be assessed?
Despite lack of consensus about quality assessment a number of different tools and
techniques are now available. Over one hundred sets of proposals on quality in
qualitative research have been identified,33 a subset of which have been
reviewed,34 including five that were developed specifically for use in systematic
reviews.35, 36, 37, 38, 39 The majority of tools available are generic, and to date there have been
few attempts to develop method specific approaches. This is despite arguments that
different qualitative methods need to be appraised in different ways.40
Some issues in using structured approaches were illustrated in a recent study. 41 Two
structured methods the Critical Appraisal Skills Programme (CASP) tool, 42 and the
Quality Framework34 were systematically compared with an approach based on
unprompted judgement (where experienced qualitative researchers relied on their own
expertise to make judgements of quality). Each approach was used to assess twelve
qualitative studies investigating support for breastfeeding. Agreement between
researchers and between methods was slight, and importantly researchers disagreed on
the quality of the studies, whether papers were actually reporting qualitative research
and whether the study was relevant to the review question. Because answering
questions about quality is largely a subjective process involving judgement, it may lead
to differences both between researchers and methods.
In addition, the Quality Framework was criticised for its length and complexity, which is
likely to impact on its use in future systematic reviews. The authors identified a need for
continued debate and empirical research into the use of quality assessment. Similarly,
the authors of a recent review who attempted to apply two different quality frameworks,
concluded that further methodological work is needed to produce clear guidance about
how quality appraisal should be undertaken.32

An innovative approach, developed to appraise qualitative studies for inclusion in a set of

reviews focusing on peoples experiences and perspectives, uses generic methodological
quality criteria tailored to the specific review question. 43 It is designed to help
researchers assess to what extent studies may have distorted, misrepresented or simply
missed peoples experiences and perspectives. The authors have published a series of
reports that outline how the approach has been applied in practice.44, 45
A structured review of reports published between 1988 and 2004, appraising and
synthesising qualitative studies in health and health care is available. 46 The authors
found that over 60% of the 42 reviews included either explicitly stated that quality
appraisal was not carried out or failed to report any appraisal of studies. Interestingly
where quality appraisal was used, in all but one case the instrument or criteria were
modified, suggesting that available methods are difficult to apply in practice. Others have
opted to construct their own criteria for assessing rigor as part of the review process.47
Box 6.1 lists some of the different appraisal tools that have been developed explicitly for
use in systematic reviews and/or have been used for that purpose (this is not a
comprehensive list). Researchers interested in carrying out quality assessment, might
consider using one or more of these tools.
Box 6.1: Appraisal tools
Popay,
Rogers
Williams (1998)39

Critical
Skills
(1998)42

&Primary
question
relates
to
the
appropriateness of the
methods used. This is
followed by a detailed
assessment
of
methodological
soundness.
Appraisal10 questions relating to
Programmerigour, credibility and
relevance.

Quality Framework18 questions relating to

(2003)34
9 key areas: findings;
design; sample; data
collection;
analysis;
reporting;
reflexivity
and neutrality; ethics
and auditability.
Prompts
forGeneric set of prompts
appraising
relating to aspects of
qualitative researchreporting and aspects
(2004)33
of study design and
execution.
Long
&
GodfreyA
tool
to
explore
(2004)38
descriptive
and
evaluative elements of
a study. 34 questions
across 4 key areas:
phenomenon
studied
and context; ethics;
data collection, analysis
and
potential
researcher bias; policy
and
practice

implications.

Walsh
&
(2006)48

DowneSet of prompts relating

to 8 key areas: scope
and purpose; design;
sampling
strategy;
analysis;
interpretation;
reflexivity;
ethical
dimensions; relevance
and transferability.

6.4.2 How should quality assessment be used?

Quality assessment has been used to establish a quality threshold below which studies
will be excluded, or to distinguish between studies in terms of overall
contribution.32, 36, 49 There is no consensus as to which approach is preferable. Quality
assessment can also be used to gain an understanding of the relative strengths and
weaknesses of the body of evidence and taken into account during the process of
synthesis.
Some have reported that better quality studies appear to make stronger contributions to
the synthesis19, 49 or that weaker studies contribute nothing substantially different from
the stronger studies.32 Sensitivity analysis has been used to explore the relationship
between the quality of qualitative studies and contribution to review findings. 50 The
analysis was based on 62 primary studies from five reviews, and suggested that studies
judged to be of low quality contributed little to the overall review findings. This appears
to be the first attempt to apply sensitivity analysis to the question of quality in
qualitative research and further assessment is required. However, the findings are
consistent with the more descriptive accounts offered about study quality and overall
contribution to synthesis.19, 32, 49
6.4.3 When should quality assessment be carried out?
The use of quality assessment is further complicated by debate around when it should be
carried out. The need for appraisal of studies before the synthesis has been
queried.51 The authors of one qualitative synthesis reported that the necessity of prior
quality appraisal was a moot point. 36 They did go on to comment however that the
appraisal process was a useful prelude to the synthesis because it helped to screen out
inappropriate and poor quality studies. Clearly, if quality assessment is to be used to
establish a quality threshold, then assessment will need to take place before the
synthesis.