Toward the Collection of Essential Data

and Use of Selected Outcome Measures

in Shared Mental Health Care

Jack Haggarty MD, FRCPC

Director
Lakehead Psychiatric Hospital and Thunder Bay Regional Health Science
Centre. Fort William Clinic Shared Mental Health Service

David Haslam MD, MSc, FRCPC

Director
Mental Health Consultation and Evaluation in Primary Practice
University of Western Ontario
 Thanks to:
• Nick Kates and Anne Marie Crustolo
 Agenda
• Framework for determining the health of a
shared mental health care service
• Strategy:
(a) Essential data collection
(b) Outcome measure implementation
(Severity of illness, psychiatric diagnoses,
disability, satisfaction instruments)
• Discussion - toward consensus on common
approaches and instruments
Canadian Institute for Health Information
Health Indicators Framework

Acceptability Accessibility Appropriateness Competence

Continuity Efficacy Effectiveness Safety

www.cihi.ca
 What is an indicator?
A measurement tool, screen or flag that is used
as a guide to monitor, evaluate and improve
the quality of patient care, clinical support
services and organizational functions that
affect outcomes.1

Canadian Counsel on Health Services

Accreditation, 1996
 Clinical and Outcome Assessments Schedule

FORM OR Acronym Time of 1st Follow- Third Psych. Last

SCALE Referral up Visit Visit Visit Visit
(each)

Referral X
Form

Severity of TAG X
Mental
Illness
Symptom PHQ X X
scale
Function WHO- X X
Scale DAS

GAF X X
 Properties Needed by Measures
Used in Clinical Practice
• Validity
• Reliability
• Responsiveness to change
• Interpretability
• Appropriateness and acceptility
 Questions to be asked
when selecting outcome measures
• Who completes the measure? What effect
will this have?
• How long does it take to complete?
• Do staff and patients find it easy to use?
• Who will need to be trained and informed
about the measure?
• Is it financially affordable (ideally available
in public domain)?
 Assessing Severity of Illness
• A brief method of identifying the severity
of the mentally ill
• suitable for everyday use (‘feasible’)
 Global Assessment of Function (GAF)

• Clinician’s judgement of overall level of

functioning
• Guide treatment need and planning (DSM-IV)
• Rating - clinical dx and sx stronger predictor of
than social or occupational functioning1
• Minimally associated with treatment outcome1
• No robust association btwn GAF and clinican
interview or patient self report1

1. Moos RH. Psychiatr Serv 2002;53, 730-7.

Threshold Assessment Grid (TAG)

Slade M et al. Threshold Assessment Grid (TAG): the

development of a valid and brief scale to assess the
severity of mental illness. Soc Psychiatry Psychiatr
Epidemiol 2000;35:78-85.

TAG website: www.iop.kcl.ac.uk/prism/tag

 Threshold Assessment Grid
• Assesses severity of person’s mental health
problems
• Referrals to “routine” community mental
health service
• One page assessment
- 7 domains
- 4 to 5 point scale for each domain
• Second page provides evidence-based
TAG
criteria for each domain
 Clinical and Outcome
Assessment Schedule
FORM OR Acronym Time of 1st Follow- Third Psych. Last
SCALE Referral up Visit Visit Visit Visit
(each)

Referral X
Form

Severity of TAG X
Mental
Illness
Symptom PHQ X X
scale
Function WHO- X X
Scale DAS X
X
GAF
 Diagnostic assessment
Self-administered tool that:
• help identify and diagnose patients with
mental illnesses commonly encountered in a
primary care practice
• allow for reporting of associated stressors
• practical
• reasonable performance characteristics
• easy to average literacy level
• useful in both initial management decisions
and monitoring treatment outcome
General Health Questionnaire (12 and 26)1

• Self completed (3 to 4 minutes)

• Validated\reliable, “current state”, predictive validity
• Identifies common underlying elements
• Focuses: inability to carry out normal functions;
appearance of new distressing phenomena
• “Caseness” varies according to threshold
• FP (physically ill) and FN (“compared to normal”)

1. Goldberg D P et al Psychol Med 1997;27,191-7.

Patient Health Questionnaire (PHQ)

• Instrument based on psychiatric

disorder criteria (reference symptom
count)
• Not dependent on threshold of
symptom severity
• Offset potential for symptom count not
reliable indicator of impairment by
having scale mode
 Patient Health Questionnare
(PHQ)
Article:
Spitzer RL et al Validation and utility of s
self-report version of PRIME-MD: the PHQ
primary care study. JAMA;1999:282:1737-
1744.

Website:
http://www.cmecenter.com/primemdtoday
 Patient Health Questionnaire
Performance as a Case Finding Tool
Can it screen for several psychiatric illnesses?
• Yes (multidimensional questionnaire)
- Somatic disorder
- Depression
- Panic disorder
- Generalized anxiety disorder
- Eating disorder
- Alcohol abuse
• Screening questions trigger more extensive
diagnostic questioning (DSM IV)
 Patient Health Questionnaire
Allows for the reporting of important non-diagnostic data?
• Yes
- menstruation, pregnancy, and childbirth history
- stressors (including abuse) and severity of associated
distress
- summary measure of global functional impairment
(i.e. level of difficulty with work, home, school)

PHQ
 PHQ Operating Characteristics1
(n=585)
PPV(%) (95% CI) NPV(%) (95% CI)

Any mood d\o 61 (52-70) 94 (92-96)

Major dep d\o 73 (58-84) 94 (92-96)
Any anxiety d\o 63 (52-72) 97 (95-98)
Panic disorder 90 (76-96) 98 (97-99)
Probable ETOH 63 (48-76) 97 (96-98)
abuse/dependence
Any eating d\o 89 (71-96) 96 (94-97)

1. Spitzer et al JAMA 1999;282:1740

 Clinical and Outcome Assessments Schedule

FORM OR Acronym Time of 1st Follow- Third Psych. Last

SCALE Referral up Visit Visit Visit Visit
(each)

Referral X
Form

Severity of TAG X
Mental
Illness
Symptom PHQ X X
scale
Function WHO- X X
Scale DAS X
X
GAF
Considerations on ‘Disability’

1. Diagnosis fails to predict what we wanted to know

– Service needs
– Level of care
– Work performance
– Social integration
2. Dx and Disability can predict
– Service utilization
– Improvement after treatment
– Return to work
– Work performance
– From WHO-DAS Training Manual WHO. 2000
Considerations on ‘Disability’

3. Value for health care and policy decisions

– Identifying needs
– Matching treatments with interventions
– Measuring outcomes and effectiveness
– Setting priorities
– Resource allocation

– From WHO-DAS Training Manual WHO. 2000

 Options to Measure Function
• Medical Outcome study:SF 36, SF 12
• Cumulative Illness Rating Scale (CIRS)
• Sheehan Disability Scale: Anxiety D/O focus
• Quality of Life Measures (Q of L)…
• Review of Literature Primary Care and Outcome
– Limited by predominantly symptom based>function
– If function: time in bed, SF 12, SF 36, CIRS, lost work
days,

• For further review: Outcome Measurement in psychiatry. 2002.

W.W. Ishak et. al. Am Psych Press.
General Information
Welcome to the Home Page for the World Health Organization Disability Assessment Schedule II (WHODAS II). This
site provides information relevant and useful to researchers, clinicians, and administrators who are interested in
Download WHODAS II
learning about and using this instrument for assessing levels of functioning.
Download I-Shell - WHODAS The WHODAS II has been under development by WHO for several years. Final versions are expected to be released
Manuals &
Software* in 2001. Currently, the WHODAS II is available in eleven versions and sixteen languages. Available versions include
self-administered, interviewer-administered, and proxy-reported.
The WHODAS II provides a profile of functioning across six activity domains, as well as a general disability score. This
Upload WHODAS
information can be used to
data*
List of Centres •Identify needs
•Match patients to interventions
Translations •Track functioning over time
•Measure clinical outcomes and treatment effectiveness
Ongoing Field Trials
Return to this page often to receive the latest information and updates regarding the WHODAS II, including semi-
structured versions of the instrument and publications.
Scoring

Frequently Asked
Questions

Contact information

Related Links

WHO Home Page

Send your comments and feedback about this site to whodas@who.int

* Available only to WHODAS-II Centers and requires a user-id and password.

 WHO-DAS
• Used in the physically ill
• Rheumatology-Ann Rheum Dis 2003:62.140-145
• Pulmonay Rehab-Chest 2002:122.948-954
• SMI-Acta Psych Scand 2002:105.196-201. And 2000:102.26-31
•

• Cost effectiveness- Medical Care Vol.41.2:208-217

• Developing Countries-Soc Psychiatry Psychiatr Epid (1997) 32:
387-390
 WHO-DAS Six Domains
• Understanding and communicating
• Getting around
• Self care
• Getting along with others
• Life activities
• Participation in society
 WHO-DAS Versions.
Self-Administered
• 36 Item. All domains, overall score

• 12 Item. Helpful esp. when domain specific

information is not required. OUR VERSION

• Other as INTERVIEW and PROXY versions

 WHO-DAS II – Our Use
Considerations
• Developed Intn’l Experts ‘Common metricè of
ADM’ disorders
• 1995 attempt to classify Measure ‘reliable’ and
‘valid’
• Other options limited. ie. SF 36…too long..other
• Built on >300 Q of L, ADL, functional measures
WHO-DAS II – Our Use
Considerations:
Multi-phase field testing for lang., cultural concepts,
value etc. of DISABILITY….WHO-DAS, DAS-S,
DAS-II (present phase).
• Only International instrument, validated-languages,-
age, - Dx types, Brief version available
• No cost yet ‘state-of-the-art’ instrument
• Kept using- simple wording, practical information
 WHO-DAS II – Our Use
• Considerations
• WHO Classification system forms core domains
• Discriminates symptoms from other domains of
importance ie. work, cognition, self-care,
interpersonal relations (esp 36 version)
• “stable factor structure, replicable across countries
and population groups, high discriminatory ability,
good test-retest (kappa .65-.78), high correlation
with SF12, 36, London Handicap Scale, and
WHOQOL.”
– The WHO Disability Assessment Schedule (WHODASII) and its relation with ICF
WHO Document. Brisbane Meeting Heads of WHO Centres. Oct 2002.

• Ate lunch with Dr. Wayne Katon. Univ. Washington

 First Year Findings
Improved mean WHO-DAS scores pre and post
n=30

pre post
3
2.5
2
1.5
1
0.5
0
H1: S2: S3:new S4:joining
overall home learning activites
health respon p=.007 p<.001
p<.001 p<.001
 First Year Findings
Improved mean WHO-DAS scores pre and post
n=30

pre post
2

0
S5: S6: S11: S12: W2:
emotional concent. maintain. daily interfere
affect friendships work with life
p<.001 p=.002 p=.031 p=.001 p=.001
 First Year Findings

• WHO-DAS data which did not significantly

improve
– standing
– walking
– washing
– dressing
– dealing with stangers
 Paired Samples T-Test
Comparison of Pre- and Post-Treatment Responses to WHODAS
Shared Mental Health Care Service
July 2001 – May 2003
T1 T2
Domain n t df p
mean SD mean SD
H1 87 2.82 .91 2.09 .76 7.214 86 .001
S1* 91 1.90 1.18 1.73 1.07 1.745 90 .084
S2 91 2.42 1.10 1.62 .84 7.470 90 .001
S3 89 1.78 1.07 1.34 .66 4.002 88 .001
S4 85 2.40 1.37 1.66 1.05 5.807 84 .001
S5 89 3.04 1.14 1.98 1.08 9.372 88 .001
S6 91 2.08 1.01 1.38 .73 6.554 90 .001
S7 87 1.82 1.20 1.48 .90 3.197 86 .002
S8* 90 1.19 .58 1.06 .28 2.642 89 .010
S9 91 1.22 .53 1.05 .23 3.012 90 .003
S10 89 2.09 1.24 1.54 .91 4.824 88 .001
S11 87 2.03 1.29 1.34 .89 5.301 86 .001
S12 83 2.52 1.09 1.65 .99 7.322 82 .001
H2 75 2.80 1.17 1.87 .91 7.072 74 .001
H3 71 17.06 10.21 9.58 11.00 5.339 70 .001
H4 68 5.40 8.72 2.51 6.44 3.310 67 .002
H5 69 10.62 9.94 5.36 9.07 4.376 68 .001

•All domains are significant at the 99% Confidence Interval, except S1 which is significant at the 90% level.
•S1-Standing for long periods. S8-Washing your whole body.
Comparison of Pre- and Post-treatment WHODAS Scores for All Cases Closed
July 2001 - May 2003

2.82
3 2.42 2.40
2.09
1.90 1.73 1.78 1.66
2 1.62 1.34
1

0
H1 S1 S2 S3 S4

First Final
Comparison of Pre- and Post-treatment WHODAS Scores for All Cases Closed
July 2001 - May 2003
3.04
3
1.98 2.08 1.82
2 1.38 1.48 1.19 1.06 1.22 1.05
1
0
S5 S6 S7 S8 S9

First Final
Comparison of Pre- and Post-treatment WHODAS Scores for All Cases Closed
July 2001 - May 2003

17.06
18
9.58 10.62
12
5.40 5.36
6 2.51

0
H3 H4 H5

First Final
 Clinical and Outcome Assessments Schedule
FORM OR Acronym Time of 1st Follow- Third Psych. Last
SCALE Referral up Visit Visit Visit Visit
(each)

Satisfaction 1. VSQ* X
Scale 2. CSQ* X X
(if
3. GP on expect Month
nurse & to be 3, 6
psych. last then
visit) q12
month
 VSQ - Satisfaction with Explanation of Treatment
July 20, 2001 - June 12, 2003
(N = 308)

60.7%
80%
30.2%
40% 8.4% 0.6% 0.0%

0%
Excellent Very Good Good Fair Poor
 VSQ - Satisfaction with Technical Skills
July 20, 2001 - June 12, 2003
(N = 313)
70.6%
80%
22.4%
40%
6.4% 0.6% 0.0%

0%
Excellent Very Good Good Fair Poor
 VSQ - Satisfaction with Personal Manner
July 20, 2001 - June 12, 2003
(N = 312)

120% 80.8%

80%
15.1%
40% 4.2% 0.0% 0.0%

0%
Excellent Very Good Good Fair Poor
 Clinical and Outcome Assessments Schedule
FORM OR Acronym Time of 1st Follow- Third Psych. Last
SCALE Referral up Visit Visit Visit Visit
(each)

Assess. X
and
Intervent.

Psych. X
Consult.

Psych. X X
Follow-Up

Tx X
Outcome
 CIHI Health Indicators Framework
Acceptability Accessibility Appropriateness Competence
Satisfaction • Time to first • TAG • adequate skill
questionnaire contact, • # of visits set
• patients counseling, • feed back,
•GPs psychiatrist retreat
•psychiatrist • Crisis • fidelity items
access • academic
• Phone avail. detailing
• Indirect care
Continuity Efficacy Effectiveness Safety
• f\u clinicians • PHQ • TAG
• disposition •WHO-DAS II • Treatment
after episode of outcome
care