Adm Policy Ment Health (2014) 41:625–635
DOI 10.1007/s10488-013-0506-4
ORIGINAL ARTICLE
Composing a Core Set of Performance Indicators for Public
Mental Health Care: A Modified Delphi Procedure
Steve Lauriks • Matty A. S. de Wit •
Marcel C. A. Buster • Onyebuchi A. Arah •
Niek S. Klazinga
Published online: 15 June 2013
 Springer Science+Business Media New York 2013
Abstract Public mental health care (PMHC) systems are
responsible for the wellbeing of vulnerable groups that
cope with complex psychosocial problems. This article
describes the development of a set of performance indi-
cators that are feasible, meaningful, and useful to assess the
quality of the PMHC system in Amsterdam, the Nether-
lands. Performance indicators were selected from an
international inventory and presented to stakeholders of the
PMHC system in a modified Delphi procedure. Charac-
teristics of indicators were judged individually, before
consensus on a core set was reached during a plenary
discussion. Involving stakeholders at early stages of
development increases support for quality assessment.
S. Lauriks
M. A. S. de Wit
M. C. A. Buster
Department of Epidemiology, Documentation and Health
Promotion (EDG), Municipal Health Service Amsterdam
(GGD Amsterdam), Amsterdam, The Netherlands
S. Lauriks
O. A. Arah
N. S. Klazinga
Department of Public Health, Academic Medical Centre,
University of Amsterdam, Amsterdam, The Netherlands
S. Lauriks (&)
Department of Epidemiology, Documentation and Health
Promotion (EDG), Municipal Health Service Amsterdam
(GGD Amsterdam), Nieuwe Achtergracht 100, Afd. EDG,
Room B4.32b, 1018 WT Amsterdam, The Netherlands
e-mail: slauriks@ggd.amsterdam.nl
O. A. Arah
Department of Epidemiology, UCLA Fielding School
of Public Health, Los Angeles, CA, USA
O. A. Arah
UCLA Centre for Health Policy Research, Los Angeles,
CA, USA
Keywords Public health
Mental health services

Quality indicators
Health care
Consensus

Delphi technique
Introduction
In addition to its functions in universal and selective
prevention of psychosocial deterioration, the public
mental health care (PMHC) system provides support for
individuals with severe and complex psychosocial prob-
lems who are characterized either by not actively seeking
help for their psychiatric or psychosocial problems, or by
not having their needs met by private (regular) care
services (Bransen and Wolf 2001). The current study
focused on this individual care level of PMHC. The
activities of the PMHC system ranges from guided
referral, which includes signaling a problematic situation
and reporting the client to the appropriate provider, to
the provision and coordination of specialist and multi-
dimensional care. Stakeholders generally include mental
health and addiction care services, municipal institutions
(i.e. the municipal health service, and the welfare and
employment service), non-profit or goodwill organiza-
tions providing shelter (e.g. the Salvation Army), and
judicial agencies (i.e. the police department, the public
prosecutors office, and criminal rehabilitation services).
However, the specific organizations and their activities
within the PMHC system are determined locally under
direction and responsibility of municipal government.
Thus, information on PMHC quality is primarily relevant
to local providers (partners) to support their quality
improvement efforts, and to municipal governmental
bodies to inform decisions on funding and provide public
accountability.
123
626
In recent years, the application of performance indica-
tors (PI) to assess the quality of care has become common
practice in several health care sectors. PI have been used to
support the quality improvement efforts of providers,
inform choices of consumers, and assist health insurers in
contracting decisions. Mental health service systems have
lagged behind general health systems in terms of quality
measurement and the use of information technology
(Institute of Medicine 2006; Croft and Parish 2011). In an
effort to address these deficits in the Dutch mental health
care sector, several trade organizations worked with the
health care inspectorate (IGZ) and health insurance com-
panies to develop a core set of PI for mental health and
addiction care providers (Steering committee on Trans-
parency in Mental Healthcare 2006). In addition, a number
of collaborations of mental health providers develop and
exchange knowledge on routine outcome monitoring
(ROM) aimed at the promotion of transparency in (private)
mental health care and assessment of treatment effective-
ness (van der Feltz-Cornelis et al. 2010). These national
initiatives reflect an international trend to develop data sets
that could inform improvements in mental health services
by routinely measuring outcomes (Bickman 2012). In the
public health care sector and many other public human
services, a major objective of the use of PI is to provide
public accountability. In 2005, a collaboration of three
national public health organizations launched a project
aimed at the selection of PI from existing sets, to stan-
dardize the assessment of quality of municipal health ser-
vices (Beter Voorkomen 2008). The project group found
that the provision of PMHC is highly specific to the local
care system, and could not select one unambiguous set of
PI for PMHC (Beter Voorkomen 2008). The municipal
health service of Amsterdam (GGD Amsterdam) and
Department of Public Health of the University of
Amsterdam recognized a need for structural quality
assessment rising from a policy reform that entailed the
transfer of funds for the public mental health care system
from national to local government. These institutes col-
laborate on a project to develop PI for the local PMHC
system to meet the challenges of transparency, account-
ability and service improvement (Klazinga et al. 2007).
As part of this project, the current study set out to
develop a core set of PI for the PMHC system in the city of
Amsterdam, the Netherlands and was aimed at answering
the following question: which PI are meaningful and fea-
sible for assessment of quality of the PMHC system in
Amsterdam, are supported by local stakeholders, and are
representative of diverse dimensions of the system? Start-
ing with a large pool of PI identified from international
literature, we conducted a systematic selection method and
a structured consensus-building procedure to select a core
set of PI.
123
Adm Policy Ment Health (2014) 41:625–635
Methods
To compose the PI-set, we used a three-step method; pri-
mary selection process, evaluation of PI by individual
stakeholders, and evaluation of PI by the group of stake-
holder representatives.
Primary Selection Process
The selection process was executed in two phases. First, PI that
could potentially be applied to asses the performance of the
local PMHC system in the municipality of Amsterdam were
selected from a large pool of PI for PMHC published in
international literature (Lauriks et al. 2012). The pool of PI was
condensed by merging PI that assess similar aspects, domains,
or dimensions of PMHC into a more general indicator. For
instance, several PI in the pool measure the number of days
between discharge from inpatient treatment and the first out-
patient contact, only varying in the appropriate number of days.
These PI were merged into one indicator, leaving the appro-
priate number of days to be determined at a later stage.
Selection was then based on two criteria: (1) the indi-
cator assesses the performance of outpatient PMHC ser-
vices or systems. In some nations, PMHC includes
inpatient treatment other than crisis-intervention, such as
inpatient detox or psychiatric treatment. In the Netherlands,
these functions are primarily the responsibility of regular
(private) mental health care and addiction care services; (2)
the indicator is based on administrative data that is col-
lected during the care process. PI that required new data
collection with e.g. questionnaires or surveys were
removed from the selection at this stage.
Second, three researchers with expertise in the field of
PMHC and quality assessment, evaluated each indicator on
the fit in a previously developed conceptual framework for
PMHC in the Netherlands. This framework consists of a
matrix of functions and target groups of the PMHC system.
It identifies six functions of PMHC (i.e. monitoring, sig-
naling, reporting, referral, care provision, and care coor-
dination), for client groups defined by the cause of
vulnerability (i.e. psychiatry and addiction, living circum-
stances, and harmful social relationships) (Lauriks et al.
2008). The researchers evaluated the fit of the PI individ-
ually and discussed their differences. Only when all three
researchers were convinced that the PI fit the conceptual
framework, the indicator was added to the selection.
Evaluation by Individual Stakeholders
The modified Delphi procedure used in the current study was
based on methods applied in other studies to develop PI for
primary care and mental health care (Campbell et al. 2002;
Hermann et al. 2004). All major service providers and
Adm Policy Ment Health (2014) 41:625–635
municipal policy services involved in the PMHC system
were asked to delegate one administrator and one care pro-
fessional to represent the service in a 2-round modified
Delphi procedure. Each participating representative was
instructed on the rating system and procedure during an
introductory interview.
In round one of the procedure individual representatives
were asked to rate three expressions of meaningfulness, three
expressions of feasibility, and the overall usefulness of the PI
on a 9-point scale. Meaningfulness was expressed in:
• ‘importance’, a rating of significance of the aspect of
care measured by the indicator;
• ‘relevance’, a rating of the gap between the actual and
desired level of quality of the aspect of care measured
by the indicator;
• ‘outcome’, a rating of the relation of the measured
aspect of care to desired health outcomes.
Feasibility was expressed in:
• ‘specification’, a rating of the clarity of what the
indicator intends to assess;
• ‘burden’, a rating of the expected administrative burden
of collecting data for the indicator;
• ‘data’, a rating of the availability and reliability of the
data needed for the indicator.
With ‘overall usefulness’ the representative rates the rele-
vance and necessity of including the indicator in the core set.
To attain ratings of meaningfulness and feasibility, the
mean rating of the three expressions was calculated. The
mean rating on feasibility, meaningfulness and overall
usefulness of two representatives from the same stake-
holder (i.e. administrator and care professional) was cal-
culated to ensure equal weight of all stakeholders in the
panel appraisals of the PI.
PI were directly selected in the core set when the median
meaningfulness panel rating was C7 with a dispersion
(standard deviation from the median, indicating agreement)
B1.5, together with a with a median overall usefulness panel
rating C7. Furthermore, the median feasibility panel rating
should be C5.5, or at least one representative should have
rated the feasibility of the indicator C7. When the median
meaningfulness, overall usefulness, or feasibility rating were
\5.5 (and none of the stakeholders had rated the feasibility
of the indicator C7), the indicator was directly removed from
the set after round one. The remaining PI were placed on the
agenda of round two of the procedure.
Evaluation by the Group of Stakeholder
Representatives
Before attending the second round meeting, all represen-
tatives received the results of round one in a personalized
627
overview containing their own individual and panel ratings
of feasibility, meaningfulness and overall usefulness, and
the individual and panel appraisals of all PI. The meeting
was presided by an independent chairman who monitored
time and ensured an open discussion. A member of the
research team explained the components of each individual
indicator, the quality dimension it intended to measure, and
the panel ratings of the expressions of meaningfulness,
feasibility and overall usefulness. The representatives
where invited to discuss these properties before making an
open vote for selection of the indicator in the core set.
When the majority of the representatives in the panel voted
in favor of the indicator, it was selected in the core set.
Results
A total of 330 PI were selected from the pool (n = 1,480)
of unique PI published in international literature. The
included PI assess distinct aspects of PMHC, outpatient
PMHC services or systems, and utilized administrative data
sources. The requirement of not collecting new data was
the primary reason for removing PI at this stage of the
selection process: 561 PI (37.9 %) in the pool were solely
based on questionnaires or surveys. Another 128 PI were
removed as they assessed the performance of inpatient- and
private health care services. In addition, a considerable
reduction in the number of eligible PI was realized by
merging similar PI.
Next, three members of the research team with detailed
functional knowledge of the local PMHC system and
expertise in public mental health services research, selected
56 PI that fit the conceptual framework of the PMHC
system. These PI assess the performance on one or more of
the functions of the PMHC system (i.e. monitoring, sig-
naling, reporting, referral, care provision, and care coor-
dination) for one or more of the client groups defined in the
conceptual framework and were presented to the stake-
holders’ representatives.
Sixteen organizations were identified as the primary
stakeholders in the PHMC system and approached to del-
egate representatives. One municipal service responded
that their function in the adult PMHC system was very
limited and referred to the other, already approached,
municipal services. One criminal rehabilitation service was
unable to designate representatives within the time con-
straints. Fourteen stakeholders were therefore represented
in the consensus procedure. Stakeholders included the
Municipal Health Service, the regional police department,
the department of the public prosecutor, the Municipal
Welfare and Social Service, the Municipal Housing, Care
and Societal policy service, mental health- and substance
abuse care providers (n = 2), criminal rehabilitation
123
628 Adm Policy Ment Health (2014) 41:625–635

Table 1 Characteristics of members of the stakeholder panel maximum feasibility ratings of the PI in round one of the
No. panelists
procedure, by further steps in the selection process (i.e. PI
(n = 24) directly selected in core set, PI to be discussed in round
two, and PI directly removed after round one), and ordered
Stakeholder perspective
by median overall usefulness.
Mental health/substance abuse care provider 4 Thirteen PI were directly selected in the core set. The
Rehabilitation/probation service 4 median meaningfulness ratings ranged from the minimum
Police department 1 for selection of 7.0 for ‘Permanent housing’ and ‘Homeless
Public prosecutor’s department 1 with income’, to 7.4 for the ‘Vocational status’ indicator.
Municipal health service 2 Meaningfulness dispersion ranged from 0.8 (‘Time
Municipal policy department for housing, 1 between referral and intake’ and ‘Stable vocational status’)
public health and society
to 1.4 (‘Service coverage difficult to engage clients’). Nine
Municipal employment and welfare service 2
of the selected PI had a median feasibility rating C5.5. The
Shelter and housing services 6
lowest maximum feasibility rating for the remaining PI that
Day-time support and vocational services 3 were selected was 7.5 (i.e. ‘Time between referral and
Occupational background intake’).
Manager 4 Fourteen PI were directly removed from the core set.
Team leader 3 Five PI were rated not meaningful and not useful by the
Policy officer 8 panel, eight more PI were removed based on low feasi-
Case manager 3 bility, and removal of one indicator (‘Time between dis-
Quality control officer 1 charge and outpatient contact’) was based solely on the low
Regional coordinator 1 median rating of the overall usefulness.
Psychiatrist 1 The remaining 29 PI were placed on the agenda to be
Rehabilitation officer 2 discussed in round two of the consensus procedure.
Psychiatric nurse 1 Fifteen of the 24 members of the stakeholder panel
Years of experience in current function attended the meeting. The representatives of criminal
0–5 years 11 rehabilitation services, police department, and public
5 years or more 13 prosecutors office were unable to attend leaving these
Functional group stakeholders unrepresented at the meeting. The represen-
Administrator 14 tatives of these stakeholders were asked to assess the out-
Care professional 10 comes of round two separately, paying specific attention to
PI concerned with arrests and judicial contacts.
The stakeholder panel agreed on 17 PI in addition to the
services (n = 2), shelter and housing services (n = 3), and 13 PI that were selected in round one. Thus, 30 PI were
day-time support and vocational services (n = 2). The selected in the core set, of which the descriptions, numer-
stakeholders designated a total of 24 representatives that ators, and denominators are presented in Table 3.
formed the consensus panel. Characteristics of the panel The PI in the core set cover process and outcome
members are shown in Table 1. domains of the PMHC system and four dimensions of
The local authority and provider perspectives were PMHC performance, i.e. accessibility (the ease with which
represented by the three municipal departments and the PMHC services are reached), continuity (the extent to
mental health-, addiction-, vocational-, and homeless care which PMHC is smoothly organized over time), appropri-
service providers. The police department, criminal reha- ateness (the degree to which provided PMHC services is
bilitation services and public prosecutor’s department relevant tot the clinical needs given the current best evi-
represented the nuisance control perspective in the proce- dence), and effectiveness (the degree of achieving desirable
dure. Representatives had diverse functions within their outcomes given the correct provision of PMHC services to
organizations ranging from psychiatrist to quality control all who could benefit). Four distinct categories of desirable
officer and were generally highly experienced, with over outcomes of the PMHC system were identified: (reduction
50 % of the panel having five or more years experience in of) justice system involvement; (improved) housing;
their current occupation. Divided into functional groups, (improved) vocation and earning; and (improved) health.
slightly more administrators than care professionals took The panel concluded that one of three PI that assess
seat in the stakeholder panel. follow-up after inpatient care should be selected in the core
Table 2 shows the median overall usefulness, median set and further research on the construct and criterion
meaningfulness and dispersion, median feasibility, and validity is needed to decide which one (i.e. ‘Outpatient

123
Adm Policy Ment Health (2014) 41:625–635 629

Table 2 Round one panel ratings for overall usefulness, meaningfulness, and feasibility of PI by further selection process and ordered by overall
usefulness
PI description Useful Meaningful Meaningful Feasible Feasible
median median dispersion median maximum

PI directly selected in core set after round one

Overall service coverage homeless 8.0 7.2 1.3 5.3 8.0
Homeless with income 8.0 7.0 1.0 6.0 8.2
Stable vocational status 7.8 7.3 0.8 6.0 8.2
Time between referral and intake 7.8 7.2 0.8 5.2 7.5
Improved housing 7.5 7.3 1.0 5.7 7.5
Mental health service coverage homeless 7.5 7.1 1.0 5.8 8.0
Drop-out 7.5 7.1 0.9 6.0 7.8
Mental health status 7.5 7.1 1.1 5.8 7.0
Vocational status 7.3 7.4 0.9 5.3 7.7
Assisted housing 7.3 7.3 1.3 6.0 8.5
Vocational reintegration project 7.0 7.3 1.1 5.6 7.3
Service coverage difficult to engage clients 7.0 7.2 1.4 4.9 8.0
Permanent housing 7.0 7.0 1.1 5.6 7.7
PI to be discussed in round two
Comprehensive evaluation 8.0 6.8 1.3 5.9 8.2
Homeless receiving care 8.0 6.7 1.1 6.0 7.8
Homeless registered with care providers 7.5 6.9 1.3 6.1 7.5
Start treatment within 98/105/147 days after registration 7.3 6.7 0.8 5.6 7.3
Lost to follow-up 7.0 6.4 1.1 5.6 7.7
Case management services 7.0 6.2 1.1 5.8 7.3
Screening for dual diagnosis 7.0 6.0 1.3 5.4 7.3
Improvement in vocational status 6.8 7.2 1.1 5.3 7.5
Homeless participating in vocational activities 6.8 7.2 1.0 5.8 7.2
Daily functioning 6.8 6.6 0.9 5.0 7.3
Outpatient contact within 7, 14, or 30 days 6.8 6.5 1.3 5.6 8.3
Population receiving care 6.8 6.4 0.8 5.6 7.7
Assigned case manager 6.8 6.3 1.2 6.0 7.3
Physical health status 6.8 6.1 0.9 6.0 6.7
Length of treatment for substance-related disorders 6.5 6.1 1.2 5.8 7.5
Change in the nr. of arrests in the last 30 days 6.5 5.8 1.2 5.0 8.7
New client index 6.5 5.8 1.1 5.4 8.3
Arrests within the last 30 days 6.5 5.7 1.2 5.3 8.7
Homeless with vocational activities [24 h/week 6.3 6.8 1.7 5.5 7.3
Ambulatory care before admission 6.3 6.0 1.3 5.8 7.3
Time between first and second contact 6.3 5.9 0.9 5.6 7.5
Readmission within 7,14, 28 or 30 days 6.3 5.5 1.5 5.4 7.2
Monthly contact during 6 months 6.0 5.9 1.4 5.9 8.3
Assertive community treatment 6.0 5.9 1.2 5.8 7.2
Clients with mental disorders in gainful employment 5.8 6.4 1.0 5.3 7.7
Judicial contacts 5.8 5.7 1.2 5.0 7.7
Time spend in jail 5.8 5.7 1.4 5.0 7.7
Change in time spend in jail 5.8 5.6 1.0 4.8 7.2
Average cost of ambulatory care 5.5 5.6 1.3 5.7 7.3
PI directly removed after round one
Days on the street 6.8 7.1 0.9 5.0 6.8
Median duration untreated disorders 6.5 7.0 1.4 5.0 7.0

123
630 Adm Policy Ment Health (2014) 41:625–635

Table 2 continued
PI description Useful Meaningful Meaningful Feasible Feasible
median median dispersion median maximum

Service reach/utilization 6.5 6.3 1.3 5.1 6.8

Evaluation of drug use 6.5 5.8 1.2 5.3 6.7
Independent activities 6.3 7.2 1.3 4.8 6.8
Social relations of people with mental disorders 6.0 6.6 1.1 4.8 6.5
Evaluation of general physical health 6.0 5.6 1.2 5.1 7.0
Somatic screening 5.5 5.5 1.1 5.0 6.8
Time between discharge and outpatient contact 5.3 6.2 1.1 5.5 7.7
Homeless with an increase in income 5.0 5.4 1.1 5.8 7.7
Escape from involuntary admission 5.0 5.1 1.8 6.3 8.0
Mortality of PMHC clients 5.0 5.0 1.1 5.5 8.0
Standardized mortality rate 5.0 4.8 0.8 5.3 7.0
Life years lost 4.8 5.0 1.3 5.1 7.5

contact within 7, 14, or 30 days’ or ‘Monthly outpatient
contact during 6 months’ or ‘Lost to follow-up’).
The indicator described as ‘Assertive community treat-
ment’ was divided in two PI as the panel concluded that
ACT is an intervention for a specific group and although it
is an important intervention in PMHC, it targets only a
subgroup of the PMHC-population. Thus, two PI are
selected in the core set, one indicator described as ‘High
service users in intensive care teams’, a measure of the
appropriateness of the provision of specific interventions
(i.e. ACT, Function ACT and Critical Time Intervention 1),
and one indicator described as ‘Clients with assigned case
manager’, a measure of the appropriate provision of case
management services.
Discussion
A core set of 30 PI for the local PMHC system in
Amsterdam was composed. Representatives of all the
major stakeholders of the PMHC system including the
police department and public prosecutors’ office, municipal
services, private providers, and goodwill organizations
have reached consensus on the meaningfulness, feasibility
and usefulness of these PI in a 2-round modified Delphi
1
Assertive Community Treatment (ACT) is an intensive mental
health program model in which a multidisciplinary team of profes-
sionals services patients who do not readily use clinic-based services,
but who are often at high risk for psychiatric hospitalization (Bond
et al. 2001). Function ACT is a rehabilitation-oriented clinical case-
management model based on ACT, that combines individual case
management with a team approach (Remmers van Veldhuizen 2007).
Critical time intervention employs strategies similar to those used in
ACT primarily to maintain the continuity of care during a critical
period transition from inpatient to community care (Herman and
Mandiberg 2010).
procedure. Every indicator in the core set assesses a
meaningful aspect of PMHC performance, provides
stakeholders with useful information in the performance of
the system, and data collection is feasible given the current
information infrastructure.
Several gaps in PMHC quality assessment were identi-
fied. One function specified in the initial framework, i.e.
‘monitoring’ (Lauriks et al. 2008) is not assessed with the
selected PI. Furthermore, some dimensions of quality that
were mentioned by stakeholders as relevant for the con-
struct of quality are not covered by the PI in the core set.
For instance, measures of efficiency and sustainability of
the PMHC system are not selected. The current method
entailed selecting PI from a pool of PI that have been
published in literature. By design, no new PI were devel-
oped. Thus, although the core set covers most of the
functions and quality dimensions of the local PMHC sys-
tem, additional PI still need to be developed. In addition,
the current study focused on the provider perspective of
PMHC system quality. However, the client perspective
should be represented to assess the quality of mental health
care comprehensively (Druss et al. 1999). Therefore, the
current set of PI is complementary to patient-reported
outcome measures and consumer perspective PI developed
as part of other, service level, quality assessment initiatives
[e.g. (Steering committee on Transparency in Mental
Healthcare 2006; van der Feltz-Cornelis et al. 2010).
The majority of professionals and administrators in the
stakeholder panel were not specialized in system-level
quality assessment. Differences in level of knowledge or
experience with indicators could have introduced bias in
the responses (Turpin et al. 1996). Clear instruction before
the individual round, for instance on what constitutes a
valid and reliable indicator and the dimensions of quality in
health care in general, and group leadership during the

123
Adm Policy Ment Health (2014) 41:625–635
Table 3 Core set of performance indicators for the local PHMC system
PI descriptiona
Process indicators—accessibility
Overall service coverage homeless
Service coverage difficult to engage
clients
Homeless registered with care providers
Mental health service coverage homeless
New client index
Process indicators—continuity
Outpatient contact (1) within 7, 14, or
30 days OR (2) monthly contact during
6 months OR (3) lost to follow-up
Time between referral and intake
Time between first and second contact
Start treatment within 98/105/147 days
after registration
Ambulatory care before admission
Drop-out
Process indicators—appropriateness
Length of treatment for substance-related
disorders
Comprehensive evaluation
High service users in intensive care team
Clients with assigned case manager
Vocational reintegration program
Outcome indicators—effectiveness—justice system involvement
Arrests within the last 30 days
Change in the nr. of arrests in the last
30 days
631
Numerator Denominator
# #
homeless within the catchment area of the PMHC homeless persons within the catchment
system that receives care from C1 providers area
# #
new clients of a service provider that were diverted difficult to engage clients within the
to care through an outreach program catchment area
# #
homeless clients registered with a PMHC provider clients registered with a PMHC provider
# #
homeless persons with a severe mental illness (SMI) of homeless persons with severe mental
that receives assertive community treatment (ACT) illness (SMI)
or intensive outreach treatment
# #
new clients receiving care from a PMHC provider clients receiving care from a PMHC
provider
# #
clients discharged from involuntary mental health or clients discharged from involuntary
addiction care services that (1) had C1 outpatient mental health or addiction care services
contact within 7, 14 or 30 days after discharge OR
(2) had C1 outpatient contact per month in the
6 months after discharge OR (3) had no outpatient
contact in the 2 months after discharge
#
R time (days) between referral and intake clients that is referred to a PMHC
provider
#
R time (days) between intake and first treatment new clients receiving care from a PMHC
contact provider
# #
clients for whom the time between registration and clients that started treatment, per
start of treatment was B98 days for outpatient treatment modality
treatment, B105 days for inpatient treatment, and
B147 days for assisted housing
# #
involuntary admissions in a mental or addiction care involuntary admissions in a mental or
service for which an outpatient contact within the addiction care service
7 days prior to the admission has been registered
# #
clients that terminated treatment for whom clients that terminated treatment
‘unilateral termination by client’ is registered as the
reason for termination
# #
clients that starts treatment for a drug-related clients that starts treatment for drug-
disorder and stays in treatment C90 days related disorders
# #
clients for whom the status and care needs with clients with a valid intake
regard to housing, drug use, social functioning,
vocational activities, finance, mental health, and
physical health are evaluated at intake
# #
persons with SMI that are high service users (C2 persons with SMI that are high service
inpatient treatments or C4 crisis interventions) that users (C2 inpatient treatments or C4
receives care provided by an intensive care team crisis interventions)
i.e. an ACT team, a functional ACT (FACT) team,
or a critical time intervention (CTI) team
# #
clients that is assigned a case manager at the start of clients that receives care provided by a
the treatment program PMHC service
# #
eligible clients that participates in a vocational clients that is eligible for an vocational
reintegration program reintegration program
# #
clients with a valid intake that were not arrested by clients with a valid intake and second
the police in the 30 days preceding the intermediate evaluation
evaluation
Difference between the # of arrests in the 30 days #
clients that was arrested C1 in the
preceding intake and the # of arrests in the 30 days 30 days preceding intake and have a
preceding the second evaluation valid second evaluation
123
632 Adm Policy Ment Health (2014) 41:625–635

Table 3 continued
PI descriptiona Numerator Denominator
# #
Judicial contacts clients that had C1 contact with the police or justice clients with a valid first and second
department between first and second evaluation evaluation in the reporting period
#
Change in time spend in jail Difference between the R time (days) spend in jail in clients that spend time in jail in the
the 30 days preceding intake and R time (days) 30 days preceding intake and have a
spend in jail in the 30 days preceding 6 months valid 6 months evaluation
evaluation
Outcome indicators—effectiveness—housing
# #
Supported housing clients with SMI that live independently with clients with SMI that can live
assistance that is in accordance with their needs independently with some assistance
# #
Improved housing clients that was homeless at intake, whose housing clients that was homeless at intake with a
status had improved in the month (or quarter) valid second evaluation
preceding the second evaluation at 3, 6, or
12 months after intake. Housing status was ranked
(from low to best) street—night shelter—temporary
housing \6 months—permanent housing
[6 months
# #
Permanent housing clients that was homeless at intake, who lived in clients that was homeless at intake with a
permanent housing in the month (or quarter) valid second evaluation
preceding the second evaluation at 3, 6, or
12 months after
Outcome indicators—effectiveness—vocation/earning
# #
Vocational status clients with an vocational status at the second clients with a valid first and second
evaluation per type of vocational status (social evaluation
participation, vocational activation, support to gain
employment, gainful employment)
# #
Improvement in vocational status clients in the age of 18–65 whose vocational status clients in the age of 18–65 that had no
has improved at second evaluation. Possible vocation at intake with a valid second
improvements are: no vocation—vocational evaluation
activation—support to gain employment—gainful
employment
# #
Stable vocational status clients with an vocational activity at intake that kept clients with a valid intake and second
its vocational status at second evaluation. evaluation in the reporting period that
has an vocational activity at intake
# #
Homeless with income homeless clients that have an income (welfare or homeless persons with a valid evaluation
gainful employment) in the month (or quarter)
preceding the most recent evaluation
Outcome indicators—effectiveness—health
# #
Physical health status clients that has physical health problems at intake clients that has physical health problems
whose physical health has improved or who is in at intake and a valid second evaluation
good physical health at second evaluation. Physical
health has been assessed by a trained physician at
both evaluation moments
# #
Mental health status clients that has mental health problems at intake clients that has mental health problems at
whose mental health has improved or who is in intake and a valid second evaluation
good mental health at second evaluation. Mental
health has been assessed by a psychiatrist,
psychologist, or nurse trained in psycho diagnostics
at both evaluation moments
# #
Daily functioning clients whose daily functioning has (a) improved, clients with a valid intake and second
and (b) stabilized assessed with a reliable evaluation
instrument (e.g. HoNOSb or SSM-Dc)
All PI are reported over a period of 1 year from January 1 to December 31
a
Contact, intake and treatment refer to face-to-face contact moments with professionals of any care providing organization within the local
PMHC system unless otherwise specified
b
Health of the Nation Outcome Scale (Mulder et al. 2001)
c
Self-Sufficiency-Matrix Dutch version (Lauriks et al. 2010)

123
Adm Policy Ment Health (2014) 41:625–635
panel meeting, were offered in order to reduce these dif-
ferences. An alternative would be to ask the representatives
to read the literature on the PI and base their rating on
scientific evidence as well as on their professional opinion
and experience, such as with the RAND method (Fitch
et al. 2001). However, willingness to participate in the
project and the response rate would likely be considerably
lower with the increased required investment. As partici-
pation of all, or at least as many as possible, different
stakeholders involved with local PMHC was paramount, a
modified Delphi procedure was considered to be more fit-
ting. Similar procedures have been used successfully to
develop PI for early psychosis treatment services (Add-
ington et al. 2005), primary mental health care (Wariach
et al. 2010), and long-stay hospital and community mental
health units (Killaspy et al. 2011). Next to the response
rate, the procedure has other advantages as well. Specifi-
cally, the system orientation was enforced with the mem-
bers of the panel as they became more aware of their
interdependence in the provision of integrated care. The
collective and common perspective of the PMHC system as
a whole was further developed. Furthermore, by including
the stakeholders in the early stages of PI development,
support for the implementation and use of the PI to assess
the quality of the local PMHC system was gained. Con-
vincing stakeholders of the value of quality assurance and
improvement has been identified as one of the key chal-
lenges to capture aspects of process and outcome (Wang
et al. 2006; Zayas et al. 2011). Stakeholders appreciated the
opportunity to have some control over which measures the
municipality could use to assess the quality of the system
they were part of. Possible suspicions with regard to being
held accountable based on meaningless PI could be sub-
dued by the procedure (Valenstein et al. 2004). The pro-
cedure and design of the study had a number of
disadvantages, which could have influenced the results.
The current set of PI was developed to supply system-level,
technical performance information for local governmental
and service administrators, and professionals who provide
public mental health care services and are therefore part of
the PMHC system. By focusing on these groups, several
other functional groups, who are involved in either the
collection of data for the PI or in the use of information
provided by the PI, were not involved in the procedure.
Three groups in particular were not represented in the
stakeholder panel. First, as discussed above, clients were
not represented in the procedure. This information is
therefore complementary to service-level, subjective per-
formance information that is available, or should be made
available through future research. Second, researchers may
offer a different view on the meaningfulness and feasibility
of PI. Although the primary selection process was con-
ducted by researchers, and researchers informed the panel
633
discussion by providing background information and
explanations with specific PI, this group had limited
influence on the final selection of PI in the core set. Third,
data managers of PMHC providers could have appraised
the PI differently, particularly with regard to the aspect of
feasibility of PI, data managers could have a more in-depth
knowledge of data availability and -reliability. Although
the professionals and administrators were expected to have
some information on which data is collected within their
organization, and we accounted for individual representa-
tives having information on the data availability in the
selection criteria, the feasibility of PI is (only) based on
appraisals of data availability, reliability and administrative
burden. Therefore, future research should include a more
detailed mapping of the information infrastructure of the
PMHC system to establish the validity of the appraisals of
feasibility of PI found in the current study.
The inclusion of (only) local Amsterdam stakeholders,
and selecting PI (partly) based on their feasibility, could
limit the implementation of the PI in other local PMHC
systems. Other municipalities may have different data
sources available to them. Although this core set of PI for
PMHC could be highly informative for PMHC profes-
sionals and policymakers in other municipalities and
regions both in the Netherlands and other countries, the
feasibility of the PI in the current core set should be
reassessed before implementation in others local PMHC
systems. Developers could use the 56 PI proposed here as a
starting point for their own consensus procedure with local
stakeholders and not only reassess the feasibility but gain
support for the use of PI in general. Variants of, if not the
same, PI have been proposed and implemented to assess
the performance of several international, national and local
PMHC and mental-, and substance abuse services and
-systems. Garcia Armesto et al. (2008) provided an over-
view of the mental health care information systems in 18
OECD countries to explore the possibilities for measuring
the quality of mental health care and identify potential
indicators to be included in OECD’s HCQI set, the authors
conclude that data for (variants of) a number of indicators
included in the current set for the local PMHC system
would be immediately available, or even already being
collected, in 5–11 countries other than The Netherlands
(Garcia Armesto et al. 2008). A number of large federal
and national initiatives have proposed (variants of) several
PI that are included in the current core set as well.
These initiatives include those by the Canadian Federal/
Provincial/Territorial Advisory Network on Mental Health
(McEwan and Goldner 2001), the US Department
of Health and Human Services (Trutko and Barnow 2003),
the NMHWG Information Strategy Committee Perfor-
mance Indicator Drafting Group (2005), the English
National Health Service and Healthcare Commission
123
634
(Griffiths 2007). An example of a more local effort that
proposed similar PI on outcomes such as justice system
involvement, and improvements in housing and vocational
status, is published by the California Mental Health Plan-
ning Council (2010), who, based on a stakeholder consul-
tation process, recommend a set of PI to be used to measure
the performance of the Mental Health Services Act and
aspects of the broader public mental health system at the
individual, state, county and community level, for planning
and quality improvement purposes.
The current study provides a core set of meaningful,
useful and feasible PI for the local PMHC system that is
supported by the stakeholders that form that system. Next,
the project will focus on establishing the criterion validity
of the individual PI, pilot implementation of the PI to asses
the state of the local PMHC system, and development of
new PI that could fill the gaps in PMHC performance
assessment that the current study has identified to provide
comprehensive PMHC performance information for con-
tinuous PMHC improvement and local care planning.
Acknowledgments We thank the health care providers, goodwill
organizations and municipal services for their contribution. This
project is part of the Academic Collaborative Public Health program
of the Municipal Health Service Amsterdam (GGD Amsterdam) and
the Academic Medical Centre of the University of Amsterdam—
Department of Public Health, funded by ZonMw. OAA was supported
by a Veni career Grant (#916.96.059) from the Netherlands Organi-
zation for Scientific Research (NWO).
Conflict of interest None.
References
Addington, D., McKenzie, E., Addington, J., Patten, S., Smith, H., &
Adair, C. (2005). Performance measures for early psychosis
treatment services. Psychiatric Services, 56(12), 1570–1582.
Beter Voorkomen (2008). Quality programme prevention. Indicators
for public accountability [Dutch]. htttp://www.ggdkennisnet.nl/
thema/kwaliteitsinstrumenten/ publicaties/publicatie/171. Accessed
January 2012.
Bickman, L. (2012). Why can’t mental health service be more like
modern baseball? Administration and Policy in Mental Health
and Mental Health Services Research, 39, 1–2.
Bond, G. R., Drake, R. E., Mueser, K. T., & Latimer, E. (2001).
Assertive community treatment for people with severe mental
illness. Critical ingredients and impact on patients. Disease
Management and Health Outcomes, 9(3), 141–159.
Bransen, E., & Wolf, J. (2001). Public mental health care for socially
vulnerable people: Report and study of the current practice in
PMHC for the socially vulnerable [Dutch]. Utrecht: Trimbos-
instituut.
California Mental Health Planning Council. (2010). Performance
indicators for evaluating the mental health system. Sacramento:
California Mental Health Planning Council.
Campbell, S. M., Braspenning, J., Hutchinson, A., & Marshall, M.
(2002). Research methods used in developing and applying
123
Adm Policy Ment Health (2014) 41:625–635
quality indicators in primary care. Quality and Safety in Health
Care, 11, 258–364.
Croft, B., & Parish, S. L. (2011). Care integration in the patient
protection and affordable care act: Implications for behavioral
health. Administration and Policy in Mental Health and Mental
Health Services Research,. doi:10.1007/s10488-012-0405-0.
Druss, B. G., Rosenheck, R. A., & Stolar, M. (1999). Patient
satisfaction and administrative measures as indicators of the
quality of mental health care. Psychiatric Services, 50(8),
1053–1058.
Fitch, K., Bernstein, S. J., Aguilar, M. D., Burnand, B., LaCalle, J. R.,
Lázaro, P., et al. (2001). The RAND/UCLA appropriateness
method user manual. Santa Monica: RAND.
Garcia Armesto, S., Medeiros, H., & Wei, L. (2008). Information
availability for measuring and comparing quality of mental
health care across OECD countries. Paris: Organisation for
Economic Cooperation and Development; OECD Technical
Papers, No.20.
Griffiths, H. (2007). Mental Health. In P. Whitty (Ed.), The Better
Metrics project version 8. London: The Healthcare commission.
http://www.healthcarecommission.org.uk.
Herman, D. B., & Mandiberg, J. M. (2010). Critical time intervention:
Model description and implications for the significance of timing
in social work interventions. Research on Social Work Practice,
20, 502–508.
Hermann, R. C., Palmer, H., Leff, S., Schwartz, M., Provost, S., Chan,
J., et al. (2004). Achieving consensus across diverse stakeholders
on quality measures for mental healthcare. Medical Care, 42,
1246–1253.
Institute of Medicine. (2006). Improving the quality of health care for
mental health and substance abuse conditions. Washington, DC:
National Academic Press.
Killaspy, H., White, S., Wright, C., Taylor, T. L., Turton, P.,
Schützwohl, M., et al. (2011). The development of the quality
indicator for rehabilitative care (QuIRC): A measure of best
practice for facilities for people with longer term mental health
problems. BMC Psychiatry, 11, 35.
Klazinga, N.S., Verhoeff, A.P., Brussel, G. van, Pijpers, F., Das, C.,
Arah, O.A., & Lombarts, M.J.M.H. (2007). Development and
implementation of population based performance measures in
the public health system of Amsterdam. ZonMw research
proposal Annex 4, Academic Collaborative GGD/AMC, network
for epidemiology. Amsterdam: GGD Amsterdam.
Lauriks, S., Buster, M. C. A., de Wit, M. A. S., Arah, O. A., &
Klazinga, N. S. (2012). Performance indicators for public mental
health care: A systematic international inventory. BMC Public
Health, 12, 214.
Lauriks, S., Buster, M. C. A., de Wit, M. A. S., Weerd, S. van de, &
Tigchelaar, G. (2010) Self-sufficiency matrix: Dutch translation.
http://www.selfsufficiencymatrix.org. Accessed January 2012.
Lauriks, S., de Wit, M. A. S., Buster, M. C. A., ten Asbroek, A. H. A.,
Arah, O. A., & Klazinga, N. S. (2008). Performance indicators
for public mental health care: A conceptual framework [Dutch].
Tijdschrift voor Gezondheidswetenschappen, 86, 328–336.
McEwan, K., & Goldner, E. M. (2001). Accountability and perfor-
mance indicators for mental health services and supports: A
resource kit. Ottawa: Health Canada.
Mulder, N., Loos, J., Wierdsma, A., & Poodt, H. (2001). Health of the
nation outcome scale: Dutch translation. http://www.trimbos.nl.
Accessed January 2012.
NMHWG Information Strategy Committee Performance Indicator
Drafting Group. (2005). Key PerformanceIndicators for Austra-
lian Public Mental Health Services. ISC Discussion Paper No.6
Australian Government Department of Health and Ageing,
Canberra.
Adm Policy Ment Health (2014) 41:625–635
Remmers van Veldhuizen, J. (2007). FACT: A Dutch version of ACT.
Community Mental Health Journal, 43, 421–433.
Steering Committee on Transparency in Mental Healthcare. (2006).
Basic set of performance indicators mental healthcare and
addiction care 2006 [Dutch]. Utrecht.
Trutko, J., & Barnow, B. (2003). Core performance indicators for
homeless-serving programs administered by the US Department
of Health and Human Services: Final report. Office of the
Assistant Secretary for Planning and Evaluation, US Department
of Health and Human Services.
Turpin, R. S., Darcy, L. A., Koss, R., McMahill, C., Meyne, K.,
Morton, D., et al. (1996). A model to assess the usefulness of
performance indicators. International Journal for Quality in
Health Care, 8(4), 321–329.
Valenstein, M., Mitchinson, A., Ronis, D. L., Alexander, J. A., Duffy,
S. A., Craig, T. J., et al. (2004). Quality indicators and
monitoring of mental health services: What do frontline provid-
ers think? American Journal of Psychiatry, 161, 146–153.
635
van der Feltz-Cornelis, C. M., Volker, D., & de Heer, E. (2010).
Routine outcome monitoring: Developments in mental health
care [Dutch]. Utrecht: Trimbos-instituut.
Wang, M. C., Hyun, J. K., Harrison, M. I., Shortell, S. M., & Fraser, I.
(2006). Redesigning health systems for quality: Lessons from
emerging practices. Joint Commission Journal on Quality and
Patient Safety, 32(11), 599–611.
Wariach, P., Sarojini Saklikar, R., Aubé, D., Jones, W., Haslam, D., &
Hamill, K. (2010). Quality measures for primary mental
healthcare: A multistakeholder, multijurisdictional Canadian
consensus. Quality and Safety in Health Care, 19, 519–525.
Zayas, L. E., McMillen, J. C., Lee, M. Y., & Books, S. J. (2011).
Challenges to quality assurance and improvement efforts in
behavioral health organizations: A qualitative assessment.
Administration and Policy in Mental Health and Mental Health
Services Research. doi:10.1007/s10488-011-0393-5.
123