EVALUATION OF EVIDENCE

Ephrem M (MPHE, Assistant professor)

 Validity of epidemiological studies

 Validity refers to the degree of closeness between a

measured value and the true value of what is being
measured.

 There are two types of validity, internal and external.

A. Internal validity - is the degree to which the results of

the study are correct for the particular group of people
studied

b. External validity (generalizability) - is the extent to

which the results of the study apply to people not in it.
 Cont…
 The existence of statistical significant association does not in
itself constitute a proof of causation.

 The observed association could be real or false (artifactual)

A) The association may be the result of chance
B) The association may be the result of bias
C) The association can be the result of a confounding effect

First assess an observed association or finding is validated for

bias, chance and confounding then assess other supportive
evidences
1. Bias
1. Bias
 is any systematic error in the design, conduct, or analysis of
a study that results in a distorted estimate of what the study
is attempting to measure.

 It is undesirable

 It can’t be ‘adjusted for’

 Useful to consider in any study design

 Essential to consider in critical appraisal

Bias
It is a systematic error introduced into the study
design.

Two major forms

1. Selection Bias: refers to any error that arises in the
process of identifying the study subjects.

2. Information Bias: refers to any systematic error

introduced during measurement of information on
either exposure or outcome variables
 1. Selection bias
 Selection bias affects the representativeness of the study
subjects, either as a result of sample selection, or as a result of
non-response or loss to follow-up.

 In descriptive studies there is selection bias if the occurrence of

the condition being studied differs significantly between the
study population and the reference population.

 In analytic studies selection bias introduced, if selection of

study subjects follows different criteria, (eg. cases/ controls).

 Selection bias is introduced in any of study designs. (but more

of case-control and surveys)
Types of Selection Bias
1. Non-response Bias: Self selection/ Volunteer bias/ Compliance
bias

 Those who agree to be in a study may be in some way different

from those who refuse to participate
– Volunteers may be different from those who are enlisted.

– If study subjects fail to volunteer in a study (< 20%)

2. Healthcare access bias: when the patients admitted to an
institution do not represent the cases originated in the
community.
3.
‘Selection bias’ and cross-sectional surveys

• ‘Non-response bias’ is the major concern

• Unrepresentative sampling

• Some people listed may be not reached

• Replacement of selected study participant

• It might affect observed prevalence rates

 Selection Bias in case-control studies

Source population

?? ??

Cases Controls
Criteria for selection of cases and controls should be
similar except for the outcome variable?

...Selection bias occurs if selection of cases or

controls is dependent on their exposure status
Selection Bias: solution
• Cross-sectional surveys
– Maximise response rates

– Think about sampling techniques (SRS, Systematic S.)

– Gather information about ‘missing (non-respondent)’

groups, don’t replace study subjects

– Be careful about inferences

• Case control studies

– As above

– Careful selection of control groups, and should have a

similar criteria as in selection of cases
 2. Information Bias
• This refers to bias which arises during the data collection process,
• In analytical studies, usually one factor is known and another is
measured
• e.g. in case control studies, the ‘outcome’ is known and the
‘exposure’ is measured and in cohort studies, the exposure is
known and the outcome is measured

• Error observed in measurement/ information obtained in the study

could be:
– Error due to difference in participants (biology)
– Error due to ‘observers’
• Bias is introduced when measurement (ascertainment) of
exposure/ outcome or both is not well done
Types of Information Bias
1. social-desirability bias is a type of response that respondents to
answer questions in a manner that will be viewed favorably by others

2. Interviewer Bias: an interviewer’s knowledge on the exposure and

outcome may influence the structure of questions and the manner of
presentation which may influence the response

3. Recall bias: those with a particular outcome or exposure may remember

events more clearly or amplify their memories

4. Observer Bias: Observers may have preconceived expectations of

what they should find in an examination
 Cont…
4. Hawthorne effect: People act differently if they know they are
being watched, An effect first documented at Hawthorne
manufacturing plant

5. Misclassification bias; Errors are made in classifying either the

disease or exposure status

6. Lose to follow up: Those that are lost to follow up or who

withdraw from the study may be different from those who are
followed for the entire study

7. Surveillance Bias: The group with the known exposure or

outcome may be followed more closely or longer than the
comparison group.
Information bias: case control studies
• ‘Recall bias’

– Cases selectively more likely to remember and

disclose their exposure status than controls

– Mothers having a child with diarrhoea more to

remember (recall) their children’s diet in the last three
days than women having children with out diarrhoea
case control studies.....
Observer bias’
– If observers know about the caseness or non-
caseness, their measurement of the exposure could
be biased

– Smoking and lung cancer

• Interviewer 'tries harder' to elicit smoking exposure in
patients with lung CA cases than among patients with
out lung CA
 Information bias: cohort studies and
trials
• Focus is on outcome rather than exposure

• Problems when outcome is ambiguous (i.e. in a

behaviour measurement)

• e.g. Knowledge of intervention group in a trial may

influence outcome
 Ways of minimizing information bias
1. Blinding/Masking.
• A blinded study (Masked study) is a study in which observer(s)
and/or subjects are kept ignorant of the group to which the
subjects are assigned, as in experiment, or of the population from
which the subjects come, as in a non experimental study.

– Blinding interviewers to case status

– Blinding participants (and /interviewers) to study hypothesis

• Blinding is of greatest importance when the outcome is

subjectively determined.
 Ways of minimizing information bias cont.
2. Set up strict guidelines for data collection: same standard
procedures, instruments, questionnaires, interviewing techniques
etc should be used for data collection in both comparison groups
– Train observers or interviewers to obtain data in the same
fashion

– It is preferable to use more than one observer or interviewer,

but not much, since they cannot be trained in an identical
manner

3. Classification of study subjects according to their outcome &

exposure status should be based on the most objective &
accurate methods available
4. Randomly allocate interviewer during data collection
assignments

5. Build in methods to minimize loss to follow-up

2. Confounding effect
2. Confounding effect
• The word came from Latin, ‘confundere’ meaning ‘to
mix up’

• Confounding bias is distortion of the estimated effect of

an exposure on an outcome, caused by the presence of an
extraneous factor associated both with the exposure and
the outcome,

Exposure Outcome

Confounder
 The role of confounding cont…

To bring a confounding effect, that confounding variable

must fulfill each of the following criteria
1) the variable must be associated with the exposure and,
independent of that exposure, be a risk factor for the
disease.
2) The distribution (frequency) of the confounding
variable should vary between the groups that are
compared
3) Confounder must not be an intermediate link in a
causal pathway between exposure and outcome
Confounding
Example
Grey hair Death

???
(the confounder) is strongly and independently associated
both with the outcome (dying) and with the exposure (grey
hair)

If left uncontrolled, the confounder would have produced a

spurious association between exposure and disease
 Effect of a confounder
• could be large

• may produce over or underestimate the true effect

– Positive confounding = Risk ratio or risk difference is increased
from the true value by effect of the confounding variable
– Negative confounding = Risk difference or Risk Ratio are
brought closer to the null by the effect of the confounding
variable

• may change the apparent direction of the effect

Interaction (effect modification)
• Two or more factors acting together to cause,
prevent or control a disease

• The effect of two or more causes acting

together is often greater than would be
expected on the basis of summing the
individual effects.

• Example
– Smoking and asbestos dust Vs Lung cancer.
• Collinearity?

• Mediators ?
 Interaction
• Factor A having RR= 2.0 to develop disease D

• Factor B having RR= 1.7 to develop disease D

• Combination of factors A&B have a RR of 4.5 to

develop disease D

• Expected RR if summative effect would be 3.7

 How to control Confounding effect

In the Study Design:

 Randomization

 Restriction

 Matching In the Analysis:

 Stratification

 Multivariable Adjustment
In the Study Design:

1. Randomization - Rarely possible except in RCT

2. Restriction

-restrict the admissibility criteria for subjects and limit entrance

into the study to individuals who fall within a specific category

or categories of the confounder.

-It is restricting to certain population (gender, certain

age group)

For example, if sex and race are potential confounding factors, the

study could include only nonwhite men or only white women

3. Matching

technique that selects subjects so that the distribution of

potential confounders is similar in both groups

E.g Suppose you suspect that age and sex are

confounding factors for a specific association. If a 30 year

old, woman is selected as a case, a 30 year old woman

should be selected for the control group. The only difference

should be with respect to their disease status.

 Matching…..

can be used in any design but most often used in

case/control studies where ‘n’ is smaller

 matching can be expensive and time consuming

 can limit the ability of the study to investigate the

matching factors themselves

 only controls confounding of matching factors

In the Analysis:
1. Stratification: Involves the evaluation of the association within

homogenous categories or strata of the confounding variable.

Controls confounding and helps describe effect modification

A+ A-

B D B D

 Observing RR and RD at stratified levels of a 3rd

variable (A) describes joint effects and controls

confounding
stratification
• Stratification: Involves the evaluation of the association within homogenous

categories or strata of the confounding variable.

 Controls confounding and helps describe effect modification

• Step 1: Do analysis crudely

• Step 2: Do analysis after stratifying (in the presence and absence of the potential
confounder)

• Step 3: Compare the results for difference between stratified and combined

• Interpretation of result:
– if no (much) difference between stratified and combined then the potential
confounder is not a real confounder.

– If the combined effect is higher than their additive effect then there is an
interaction
Example
• Association thought: Alcohol Vs MI
• Possible confounder: cigarette smocking

– Step 1: Do analysis crudely

Combined Outcome

Exposed
+ -
+ 100 50

700 1000
OR= 2.9
Example
• Step 2: Do analysis after stratifying (in the presence and
absence of the potential confounder)

Stratified
Smokers Non-smokers
Outcome Outcome
+ - + -
+
Expos
70 20
+
Expos
30 30

200 500 500 500

OR= 8.7 OR= 1.0

Example
– Step 3: Compare the results for difference between
stratified and combined

– Interpretation of result:
• if no (much) difference between stratified and combined
then the potential confounder is not a real confounder.

• the 10% rule is a good rule of thumb for assessing

whether there is confounding present

Combined OR = 2.9 Stratified

Presence of Conf. : OR = 8.7
Absence of conf. : OR = 1.0
Conclusion
There is much difference, thus there is
confounding effect
 5. Multivariate analysis
• Multivariate analysis allows for the sufficient estimation of
measures of association while controlling for a number of
confounding factors simultaneously

• The most common way that many factors are controlled for
simultaneously is through the use of a multiple regression model.

• Multiple linear regression is most appropriately used when the

dependent outcome variable is continuously distributed (e.g levels
of BP).
• In many epidemiologic studies, the outcome of interest is a binary
variable, such as diseased Vs non diseased. Multiple logistic
regression analysis.
3.The role of chance
 3. The role of chance

• we can draw inferences about the experience of an entire

population based on an evaluation of only a sample.

• When an inference is made from a sample, it may be

inaccurate because of the role of chance

• So, assessing for chance is an important step in making an

inference about general population

• One of the major determinants of the degree to which

chance affects the findings in any particular study is sample
size.
 The role of chance cont…

• the smaller the sample , the more variability there will

be in the estimates and the less likely the findings will
reflect the experience of the total population.

• the larger the sample on which the estimate is based,

the less variability and the more reliable the
inference.
 Cont….
• Evaluating the role of chance is through:

a. Hypothesis testing (test of statistical

significance)
or
b. Estimation of confidence interval
 The role of chance cont…
• By convention, in medical research, if the P value is
less than or equal to 0.05, the association between the
exposure and disease is considered statistically
significant.

• Any value of P less than 0.05 indicates the role of

chance is very minute.

• if the P value is greater than 0.05, chance can not be

excluded as a likely explanation, and the findings are
stated to be not statistically significant at that level .
 The role of chance cont…
 confidence interval (CI) is far more informative measure than
P value to evaluate the role of chance

 The CI represents the range within which the true magnitude

of effect lies with a certain degree of assurance.

 If the null value is not included in the interval, then the

corresponding P value is less than 0.05 and the association is
statistically significant.

 For example, in evaluating the relationship of smoking with

bladder cancer in men, instead of simply reporting that those
who smoke had a statistically significant increased risk
(RR=1.9) of bladder cancer compared with those who did not,
the 95 percent CI of (1.3-2.8)
 The role of chance cont…
• The larger the study sample, the more stable the estimate,
and the narrower the CI.

• The wider the CI, the greater was the variability in the
estimate of the effect and the smaller the sample size.

• CI of (1.3-1.8) indicates a much smaller degree of

variability than one of (1.2-7.6) and is much more
informative about the true magnitude of the RR associated
with a particular exposure.

• Thus, the P value and CI together provide the most information

about the role of chance.
Stage 2: Bradford-Hill Criteria
• Strength of association
• Temporality
• Consistency
• Dose-response relationship
• Biological plausibility
• Coherence
• Specificity
• Experimental evidence
• Analogy
 Causation or association – smoking
and lung cancer
• Strength of association
– Lung cancer rates far higher for smokers than non-smokers
• Temporality
– Smoking precedes onset of disease
• Consistency
– Different types of studies produce the same result
• Biological plausibility
– Theory that smoking causes tissue damage that over time
could cause cancer is highly plausible
• Coherence
– Theory ‘makes sense’ given current knowledge
 Causation or association – smoking
and lung cancer
• Dose-response relationship
 Data showed linear relationship between amount smoked
and incidence
• Experimental evidence
 Animal experiments using tobacco tar produced cancer in tissue
• Specificity
– Lung cancer is predicted by incidence of smoking

• Analogy
– Induced smoking in animal experiments showed increased
lung cancer
 Consistency
• Repeated observation of an association in studies conducted on
different populations under different circumstances
• If studies conducted by….
– different researchers
– at different times
– in different settings
– on different populations
– using different study designs
……all produce consistent results, this strengthens the argument for
causation

• e.g. The association between cigarette smoking and lung cancer

has been consistently demonstrated in a number of different
types of epidemiological study (ecological, case-control, cohort)
Is there a causal relationship between fluoride in water and bone
fractures?
• 18 studies have investigated the association between
hip fractures (outcome) and water fluoride level
(exposure)
– 30 separate statistical analyses

• 14 analyses produced a ‘positive association’

• 13 analyses produced a ‘negative association’
• 3 ‘no association’

The inconsistency of these results casts doubt on the

hypothesis that there is a causal relationship between
fluoride in water and bone fractures
 Strength of the association
• “Measures of association”
– used to quantify the strength of the association between an
exposure and outcome
– e.g. Relative risk, odds ratio

• Strong associations are more likely to be causal than

weak associations
– The larger the relative risk (RR) or odds ratio (OR), the
greater the likelihood that the relationship is causal

• Weak associations are more likely to be explained by

undetected biases or confounders
 Temporality
• This refers to the necessity for the exposure to
precede the outcome (effect) in time
• Any claim of causation must involve the cause
preceding in time the presumed effect
• Easier to establish in certain study designs
– Prospective cohort study

Easiest to establish in a cohort study

Normal
Lack of temporality rules out causality
Exposure
lung
Cancer
Outcome
TIME
 Dose-response relationship
• Dose-response (‘biological gradient’)
– the relationship between the amount of exposure (dose) to a
substance and the resulting changes in outcome (response)

• If an increase in the level of exposure increases the risk

of the outcome
– this strengthens the argument for causality
R
R I
R I S
R I
I
S
S K
S
K K K
0 cigs/day < 5 cigs/day 5 - 20 cigs/day > 20 cigs/day
 (Biological) Plausibility

• Plausibility refers to the biological

plausibility of the hypothesised
causal relationship between the
exposure and the outcome
– Is there a logical and plausible biological mechanism
to explain the relationship?
“A high dose of caffeine could constrict a mother’s blood vessels
reducing the blood flow to the placenta” (Biological Plausibility)

< 200 mg caffeine/day

 STUDY DESIGN
Relative ability of different types of
study to ‘prove’ causation
NB: Assuming study well-designed & conducted & bias etc. minimised

Type of Study Ability to ‘prove’

causation
1) Randomised STRONG
Controlled Trial
2) Cohort Study Moderate
3) Case-control study Moderate
4) Cross-sectional study WEAK
5) Ecological study WEAK
 Is this association causal?

Does consumption of French fries by preschool

children cause breast cancer?
Strength
Consistency
Temporality
Dose response
Biological plausibility
Study design
 Is this association causal?

Does consumption of French fries by preschool

children cause breast cancer?
Strength Weak: OR = 1.27
Consistency No
Temporality Yes
Dose response No
Biological plausibility Yes
Study design Case Control
Is this association causal?
 Is this association causal?

Does consumption of French fries by preschool

children cause breast cancer?
Strength Weak: OR = 1.27
Consistency No
Temporality Yes
Dose response No
Biological plausibility Yes
Study design Case Control
Is this association causal?
 Is this association causal?

Does cigarette smoking cause lung cancer?

Strength Strong: OR, RR = 4 - 20
Consistency Yes
Temporality Yes
Dose response Yes
Biological plausibility Yes
Study design Ecological, C/S, CC, Cohort

Is this association causal?

 Is this association causal?

Does cigarette smoking cause lung cancer?

Strength Strong: OR, RR = 4 - 20
Consistency Yes
Temporality Yes
Dose response Yes
Biological plausibility Yes
Study design Ecological, C/S, CC, Cohort

Is this association causal?

Thank you