European Journal of Operational Research 266 (2018) 221–237

Contents lists available at ScienceDirect

European Journal of Operational Research

journal homepage: www.elsevier.com/locate/ejor

Decision Support

High-fidelity whole-system patient flow modeling to assess health

care transformation policies
Ali Vahit Esensoy∗, Michael W. Carter
Centre for Healthcare Engineering, University of Toronto, 5 King’s College Road, Toronto, ON M5S 3G8, Canada

a r t i c l e i n f o a b s t r a c t

Article history: Health systems are continuously seeking ways of transforming their capacities and processes to provide
Received 15 June 2016 care in novel ways that are aligned with emerging best practices and patient choice while remaining re-
Accepted 14 September 2017
sponsive to fiscal pressures. These transformation policy options call for interventions across multiple sec-
Available online 22 September 2017
tors and patient cohorts, and expect benefits to be realized across the care system, driving policy analysts
Keywords: to take a whole-system point of view in their assessments. This paper presents a system dynamics simu-
OR in health services lation for the assessment of healthcare transformation policies involving alterations to patient pathways
Systems dynamics and service levels. The model takes a whole-system, strategic perspective, and is designed to evaluate the
Simulation direction and magnitude of patient flow changes resulting from transformation policy implementations.
The strategic simulation model is developed through a collaborative process with decision-makers across
the health system. It has a simple model structure while providing detailed breakdown of cross-sector
flows through the use of patient-level clinical and demographic data. A use case is presented for the as-
sessment of the impact of Ontario’s proposed stroke best practices. The results indicate significant patient
flow gains from the implementation of this policy, which are contingent on significant investments in the
community care sector.
© 2017 Elsevier B.V. All rights reserved.

1. Introduction care integration regions called Local Health Integration Networks

Similar to many jurisdictions in the world, Ontario’s health
system is under pressure to meet the increasing burden of dis-
ease mostly driven by the aging population while simultaneously
controlling health care spending (Drummond & Burleton, 2016;
Falk, Mendelsohn, Hjartarson, & Stoutley, 2016; Martin, 2016). The
province’s response to this growing pressure was to seek systemic
changes through a series of health system transformation plans
that focus on how health is promoted, managed and maintained
(MacLeod, 2007; MOHLTC, 2012a; MOHLTC, 2015; MOHLTC, 2015).
The common thread running through these broad transformation
policies is the focus on better use of existing resources across the
continuum of care for chronic and complex patients. The plans
seek to shift care activities among the provider sectors such as
acute, rehabilitation, and home and community care to move de-
livery from higher to lower cost modalities to contain the growth
in overall health spending, with the added benefit of bringing care
closer to home. Examples of policy levers considered for imple-
mentation of such systemic change include the introduction of
∗
Corresponding author.
E-mail addresses: ali.esensoy@utoronto.ca (A.V. Esensoy), carter@mie.utoronto.ca
(M.W. Carter).
(LHINs) (Ronson, 2006) and focused provider networks for high
cost patient cohorts (MOHLTC, 2012b), clinical best practice guide-
lines administered by bundled payment models (Stewart, 2016)
and large investments to create capacity in home and community
care for seniors (MOHLTC, 2007).
This broad-spectrum transformation agenda highlights the im-
portance of system capacity planning for both the Ministry of
Health and Long-term Care (Ministry or MOHLTC) and the LHINs
as the regional health authorities responsible for coordinating care.
Increasingly planners need to evaluate policy options from a sys-
temic perspective, where investments in one sector are expected
to result in decreased utilization in others. Furthermore, policy op-
tions are becoming more complex and harder to evaluate as they
incorporate capacity and patient flow levers in multiple sectors at
once.
The previous phase of this research, conducted with MOHLTC
and LHIN planners, and local health care providers captured the
whole-system patient flow complexities of transformation policies
in a qualitative model (Esensoy & Carter, 2015). It was concluded
that while qualitative modeling is an effective facilitator of policy
design and evaluation, the system planning process would greatly
benefit from quantitative analysis to size the potential patient flow
and resource implications of policies considered. In this regard,

https://doi.org/10.1016/j.ejor.2017.09.019
0377-2217/© 2017 Elsevier B.V. All rights reserved.
222 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237
Ontario is well positioned for the development of system-wide
planning models with its population-wide, patient-level data on
health service use, clinical and functional assessments, length of
stay and disposition that are accessible to decision-makers across
the health system through business intelligence systems such as
IntelliHealth Ontario (Health Analytics Branch, 2012).
This paper presents a whole-system simulation model of a LHIN
developed in collaboration with the MOHLTC. Its purpose is to aid
understanding of and quantify cross-sector patient flow dynamics
in the context of system-level interventions such as new models
of care, practice guidelines and large scale capacity changes. The
model is intended to accommodate a broad range of policy levers
and is designed to support strategic decisions for health system
planning by estimating the size and direction of intended and un-
intended consequences of such transformation initiatives. Specific
attention was paid to include clinical characteristics as cohort vari-
ables where possible, to make sure policies aimed at clinical sub-
populations can be represented within the whole-system model.
The model is envisioned as a strategic tool to assess the health
care system’s responses to interventions that apply across sectors,
rather than a tactical tool to make provider-level planning deci-
sions. Given that fourteen LHINs of Ontario exhibit significant dif-
ferences (Office of the Auditor General of Ontario, 2015) the model
was built over highly granular data to bridge system planning with
local patient and provider realities thereby increasing its relevance
to local planners.
The paper is organized as follows. Section 2 provides an
overview of whole-system modeling in healthcare, followed by
Section 3 that presents the methodology used in creating the sim-
ulation model. Section 4 applies the model through a case study
on stroke best practices implementation in a LHIN. Finally, Section
5 reviews lessons learned through this research and proposes next
steps.
2. Whole-system modeling in healthcare
Hans, van Houdenhoven, and Hulshof (2012) framework of
healthcare planning decisions describes strategic planning as ad-
dressing structural decision making to translate an organization’s
mission into the health care delivery process. Whole system mod-
els are defined as a class of strategic planning models in health-
care operational research that aim to model processes that cannot
be adequately represented without capturing the feedback effects
across the care continuum (Brailsford, 2007). Hulshof, Kortbeek,
Boucherie, Hans, and Bakker (2012) review of planning models in
healthcare found that there are a limited number of examples of
strategic models in the literature, and even fewer models that con-
sider cross-department or cross-organizational interactions to cap-
ture the whole care process.
The system dynamics (SD) simulation model developed for Eng-
land’s Department of Health by Royston, Dost, Townshend, and
Turner (1999) is the first model with a whole-system perspective.
The qualitative model was used to study the effects of capacity al-
location decisions across the continuum of care on the wait times
for elective procedures and discharge waits in acute care hospi-
tals. This model was intended to be a learning tool to explore
the relationships between various sectors of the healthcare system
and was not used as a decision support tool to size the effects
of policy changes. To explore policies to address long wait times
in emergency departments (ED) in the UK, Lane, Monefeldt, and
Rosenhead (20 0 0) developed an SD model to explore the effects of
bed closures on ED wait times. The simulation model explored the
interactions of ED and scheduled elective treatment sub-systems
as they compete for ward beds. Analysis of planning scenarios
showed that ED waits and elective cancellations should be simul-
taneously used as performance indicators for evaluating bed ca-
pacity decisions. Expanding the emergency department wait dy-
namics beyond the scope of hospital departments, Brailsford, Lat-
timer, Tarnaras, and Turnbull (2004) built an SD simulation model
for ED planning in Nottingham, UK, encompassing ambulance ser-
vices, ED, social services, and primary care. The model was used to
test scenarios around managing the demand for ED and the effects
of separating resources within the ED for certain patient cohorts,
concluding that demand management was the most effective lever
in addressing ED pressures. In their work similar to Royston et al.
(1999) model, Wolstenholme, Monk, McKelvie, and Smith (2008)
used SD to model local health authorities in England to study the
implications of home and intermediate care capacity on flows in
and out of hospitals. This comprehensive model contains primary
care, acute care separated into medical and surgical beds, interme-
diate care and post-acute care in long-term care homes, and home
care and continuing care. Stocks and flows were subdivided into
seniors-oriented age bands and depending on the sector and type
of resource used (e.g. medical or surgical beds in acute care). The
resulting simulation model was used as a learning tool for policy-
makers to investigate the behavior of the complex healthcare sys-
tem with attention to outcomes such as number of patients treated
in acute care settings, service wait times and delays in discharges.
This work is foundational in that it is the first pragmatic exam-
ple of a whole-system model designed for whole-system planning
at the local level. Lastly, Rashwan, Abo-Hamad, and Arisha (2015)
used an SD simulation model to explore policy directions for re-
ducing discharge delays faced by seniors in the Irish health care
system, addressed from a national perspective. The authors specif-
ically assessed interventions aimed to offset the increasing acute
care demand from the aging population. Policies assessed included
reducing hospitalizations and improving the capacity and through-
put of post-acute services. The simulation model results show that
given the aging population of Ireland, national policies aimed at
increasing post-acute bed capacity are likely to be temporary solu-
tions.
While the simulations from the whole-system modeling litera-
ture capture a range of care sectors, they are limited to age and
sex breakdowns for the patient flows among them. This limits the
ability to explore policies aimed at specific patient groups, such as
changes in models of care and bundled/integrated care scenarios.
The research presented in this paper compliments this literature
by implementing a high fidelity whole-system model that can be
used to assess a broad range of policy options and their effects on
system-wide patient flows. Specifically, this work aims to use com-
prehensive, patient-level data as the foundation of a highly flexible
simulation to model flows for a broad and modifiable range of pa-
tient cohorts. The intended flexibility is in line with the analytics
requirements generated by the transformation-oriented policy en-
vironment in Ontario, confirmed through the learnings from the
qualitative modeling phase of this research.
3. Simulation model
The simulation model’s development builds on the researchers’
existing collaboration with the Ministry and the Central East LHIN
(CE LHIN) (Esensoy & Carter, 2015). Specifically, the research objec-
tive for this phase was to address the modeling requirements iden-
tified in the qualitative phase of this research, including the ability
to translate care model or capacity interventions into model inputs
for a broad range of patient types at sector interfaces. Furthermore,
this strategic model needed to have a three to five year prediction
horizon and have a whole system perspective through the inclu-
sion of acute, post-acute and community services. These objectives
raise three key design considerations for modeling. Firstly, control-
ling the model boundaries when modeling the “whole” system is
critical in order to balance the maximal coverage of care sectors
 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237
while controlling the inflation of model complexity. Related to this
first consideration is the goal of keeping the structure of the model
as simple as possible to ensure that the model can be explained,
used and maintained with relative ease. The last consideration was
to leverage LHIN-specific data to populate this simple structure in
the lowest level of granularity possible in order to achieve maxi-
mum flexibility in model parameterization, specifically to represent
key patient characteristics.
The literature review in Section 2 highlighted that SD is the
preferred method for producing strategic patient flow simulations.
As the modeling concepts are built on stocks and flows, there
is very little overhead in creating basic patient flow artefacts in
SD simulations, which simplifies the model structure considerably.
Furthermore, SD simulations are quite forgiving in situations where
data availability and quality may be limited and varied across
model components (Brailsford, 2008) and are quite amenable for
producing strategic level models where multiple sub-systems are
represented together at a high level (Marshall et al., 2015a, 2015b).
The following sections describe the model details with respect to
the overall system and sector structures. The implementation de-
tails of the simulation model with a full set of specifications is
available from the authors.
3.1. System-level structure
Simulation boundaries differentiate between components of the
system that are explicitly represented within the model, and those
that are assumed to be independent of the states of the model
variables and are treated as exogenous inputs (Sterman, 20 0 0). The
qualitative model collaboratively built with health system decision-
makers and care providers was the first step in establishing the
overall whole-system boundary for the simulation model. This first
phase identified five care sectors, defined as collections of similar
care providers aligned with different stages of care for patients, for
which patient flow interactions were critical for strategic decision-
making: acute care, rehabilitation and complex continuing care,
long-term care homes, home and community care, and informal
care. Rehabilitation in this context refers to services that restore
functional ability of patients. Complex continuing care is a long-
term inpatient service for patients requiring intensive care beyond
that which is provided in home and long-term care settings.
Informal care is the collection of functional and cognitive sup-
ports provided by unpaid care partners such as family and friends.
In Ontario there are significant data challenges for modeling the
informal care sector, including but not limited to the definition
of informal care, sizing the capacity of this sector, the flow pat-
terns in and out of informal caregiving and the duration of care. As
such, the structure or parameters created in the quantitative model
could not be validated with real world data. Providing a hypothet-
ical structure and parameter set for this sector is in direct con-
flict with the modeling motivation of a comprehensive, data-based
foundation for strategic scenario analyses for health system patient
flow. Given these limitations, informal care was not included in the
simulation.
For the remaining sectors, it was decided to separate emergency
department from acute care as a separate sector, due to the differ-
ences in the time scales of patient length of stay (LOS) and flow
delays. Complex continuing care and rehabilitation were also mod-
eled as separate sectors to reflect the different roles these sectors
play in the healthcare system. The list below summarizes the fi-
nal set of six endogenous sectors and how the different resources
within the LHIN are aggregated to form them. From a strategic per-
spective, the expert panels found that collapsing separate providers
into one sector was a reasonable approach. For example, all acute
care beds in the LHIN were modeled as a single resource. All other
223
sectors in the LHIN and all sectors outside of the LHIN were treated
exogenously, with care demands arriving from these sectors.
i. Emergency Department (ED) – all hospital emergency depart-
ment beds.
ii. Acute care – aggregation of all hospital medical, surgical, medi-
cal/surgical combined, intensive care unit, obstetric and mental
health beds.
iii. Long-term Care Homes (LTCH) – combination of all long-stay and
short-stay beds funded by the province.
iv. Home and Community Care (HCC) – all Community Care Access
Centre (CCAC) managed long-stay, rehabilitation and short-stay
home care client spaces, modeled as separate services within
HCC.
v. Complex Continuing Care (CCC) – all CCC-designated beds in
acute care hospitals.
vi. Inpatient rehabilitation (rehabilitation) – all hospital
rehabilitation-designated beds.
The flow network depicted in Fig. 1 establishes a whole-system
feedback structure as the admission and discharge volumes of the
sectors are linked to each other. There are three types of flows
shown in this diagram, as described below.
A unidirectional flow happens when one sector sends
patients to another, but does not explicitly receive patients
back from that sector. For example, patients will leave the
emergency department for an inpatient bed, but the model
does not capture patient cohorts presenting at an emergency
department straight from an inpatient facility.
Bidirectional flows are connections between sectors
that can go both ways. For example, patient cohorts may
flow from inpatient to rehabilitation and vice versa.
Parallel flows happen when patients in one sector can
simultaneously use the resources in another. One example
is patients in long-term care temporarily using emergency
department services.
At each sector, there are flows that allow patient cohorts to en-
ter and exit this sector network. ED, acute, rehabilitation, LTCH and
HCC sectors can admit patients from the community, and there-
fore bring demand into the network. It is important to note that
demand is not generated from a population health model and as
such does not explicitly model prevalence of health conditions in
the public. Demand in the model comes from health system events
generated by the population as a whole. Patient cohorts can exit
the network at each sector through death, by being discharged to
non-simulated sectors such as home without formal care, or by be-
ing discharged outside the LHIN. Parallel flows can temporarily cre-
ate duplication of cohorts as they occupy the source sector and ED
simultaneously. This is generally a reasonable assumption as in the
event of an emergency episode, resources in the patients’ original
provider are held reserved for a period of time. If the emergency
episode results in a long term absence from the source sector, the
resources are freed up, and this is reflected as a discharge record
in administrative data and captured in the discharge rates of the
sectors.
In parameterizing this flow network, it was assumed that ex-
ogenous demand calculated from historical data stays constant
over the modeling horizon; however, the simulation model allows
the user to vary these constant arrival rates. This decision is in line
with the model’s focus on cross-sector flows over demand predic-
tion and allows us to avoid having to re-forecast demand every
time patient cohorts are changed. However, it also means that de-
mand is likely to be underestimated especially towards the end of
224 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237
Fig. 1. Endogenous sector structure of the simulation model (CCC: complex continuing care, ED: emergency department, HCC: home and community care, LTCH: long-term
care home, Rehab: inpatient rehabilitation).
the simulation horizon. Related to this, the model does not sim-
ulate the aging of patients in the flow network to simplify the
model structure. While tracking the demographic shifts would be
important for long-term system simulations, for a short to medium
term modeling horizon these shifts may not result in significant ef-
fects. For example, Ontario’s senior population is expected to grow
at an average annual rate of 3.5% over the next 25 years, resulting
in the 65+ age group’s population share growing at an average an-
nual rate of 0.38 percentage points per year (Ontario Ministry of
Finance, 2014). While the growth rate of this cohort is much faster
than the 1.3% average annual population growth for Ontario, their
share of the province’s population would only grow by about 1.9
percentage points over a five-year period. This was seen as an ac-
ceptable trade-off between model complexity and accuracy.
The cross-sector flow network is implemented as a series of ad-
mission and discharge flows among the sectors. Similar to other
whole-system models discussed in this paper, the flows within this
network are split into six age and sex cohorts: female, under 65;
female, 65–74; female, 75+; male, under 65; male, 65–74; male,
75+. Unlike the other whole-system models discussed, within each
sector, patient flows are broken down into sector-specific clinical
sub-cohorts. This allows for policy analysis to target patient co-
horts by age, sex and clinical condition. With the inclusion of the
source and destination care sectors, this breakdown makes it pos-
sible to identify a specific cross-sector flow cohorts such as the co-
hort of hip replacement surgeries in acute care for patients aged
65 years and above, who were discharged to long-term care.
Clinical breakdown in a whole-system model is not straightfor-
ward. In healthcare data sources, clinical conditions are generally
independently defined within each sector’s dataset. For example,
inpatient visits may capture a diagnosis related to that encounter,
while home care assessments could capture functional limitations.
There are population risk groupers that reconcile this information
to establish person-level risk profiles (CIHI, 2015; Hopkins Health-
Care ACG® System, 2017). However, these tools abstract diagnostic
and clinical to categorize the population in a small number of mor-
bidity groups, which may not always align with target populations
of policy interventions. In keeping with the flexibility design prin-
ciple for this simulation model, each sector’s flows were parame-
terized with a user-adjustable breakdown of clinical conditions. For
admissions from exogenous sectors, flows were broken down into
clinical sub-cohorts using historical clinical condition mixtures for
each age and sex cohorts. The formulation for this in variable HCin
is described in Appendix A (18).
For flows among endogenous sectors, there are two possible
volume transfer options that can be used in the model. The de-
fault option is for discharge flows to be aggregated by age and
sex sub-cohorts, to be broken down into clinical sub-cohorts at
the accepting sector by the historical cohort mixture for patients
accepted from the source sector. The user is able to alter this mix-
ture, using the HCin variable. For example, under this bulk transfer
method, all patients leaving acute care for rehabilitation would be
aggregated into demographic cohorts at the time of acute care dis-
charge, losing their acute care clinical grouping labels. Once ad-
mitted to the rehabilitation sector, the patients in each age/sex
cohort would be distributed into rehabilitation clinical groupings
based on the historical distribution of rehabilitation clinical group-
ings for each age/sex cohort admitted from acute care. The sec-
ond option is where patient cohorts are explicitly mapped between
the admitting and discharging sectors, by some cohort characteris-
tic. For example, it is possible to indicate in the simulation that
the stroke cohort in acute care is mapped to the stroke cohort
in the rehabilitation sector. Under this approach, all cohorts ex-
cept for the stroke cohort would be bulk mapped between the
acute and rehabilitation sectors, and the acute care stroke cohort
would directly transfer over to rehabilitation, retaining its age/sex
distribution. This is achieved by using the Mi variable described
in Appendix A (22). Having both cross-sector transfer approaches
 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237 225

Fig. 2. Sector stock and flow structure archetype, with full variable specification in Appendix A (ALC: Alternate Level of Care).

allows the patient pathways through the system to be declared in A sector’s admission rate is determined by the current resource
a highly flexible manner that accommodates a broad range of co- utilization, and its target utilization level. A sector-wide admit
hort definitions without the need for change in the underlying data rate is calculated to seek this utilization goal and it is distributed
or the cross-sector structure of the model. among the incoming flows by allocating to the source sectors. Al-
location shares can be adjusted by the user to reflect scenarios
where sectors may provide higher priority access to certain sec-
3.2. Sector structure
tors. Demand from exogenous sources is estimated based on the
historical arrival rate by patient cohort.
The qualitative model developed in the previous research phase
Once admitted, cohorts remain in the care stock for a pre-
was used as the starting point for the sector-level structure. The
determined time to complete their treatment and move into the
similarity of the causal relationships in the causal loop diagram
ALC stock where they remain until they are admitted to their next
(CLD) were assessed with respect to their involvement in flow con-
destination. If sectors do not collect reliable patient-level ALC days
trol, and the results were used to create a simple stock-and-flow
data such as HCC and LTCH in our case, and if this metric is nec-
structure for an archetypical sector in the simulation model. As
essary, the care and ALC portions of the LOS can be separated out
shown in Fig. 2, this structure contains the key relationships un-
by trimming the historical LOS by a fixed percent for each cohort
covered in the CLD and identifies how policy levers can be in-
and destination combination. All LOS parameters can be changed
terfaced with the flow network as simulation model inputs. This
before and during the simulation. Discharge rate to exogenous sec-
archetype model was used as the blueprint for all the care sectors
tors is determined by the historical rates for each cohort and desti-
in the simulation model further helping simplify the model struc-
nation combination, and are modifiable. Discharges to endogenous
ture. Further details on the sector archetype formulation can be
sectors are dependent on the admission rates set by those sectors
found in Appendix A.
in the model. For both endogenous and exogenous destinations,
Each sector in the model has time varying capacity that can
different cohorts can be prioritized for discharge by the user by
be adjusted by the user. This capacity is defined based on funded
allocating a higher percentage of the discharge flows to them.
beds for all sectors except for HCC. For this sector capacity is de-
This generic formulation can be used to add other sectors into
termined by the funded hours for case management, nursing, oc-
the model, where additional data analysis would be required to
cupational therapy, physical therapy and personal support services.
estimate the transition, LOS and other parameters for the cohort
The user is able to change the capacity level of each resource in-
breakdown relevant for that sector. Exogenous admissions and dis-
dependently before or during a simulation run.
charges of the other sectors in the model would then be connected
Patients in a sector utilize this capacity during active care, and
to the newly created sector.
while they are awaiting discharge or when designated Alternate
Level of Care (ALC) as it is referred to in Ontario. In our model, the
awaiting discharge stock is tracked for ED, acute, rehabilitation and
CCC sectors. For HCC and LTCH sectors, the discharge delay concept 3.3. Implementation
does not apply, but it is kept as a buffer for endogenous discharge
backlogs. For HCC utilization is calculated as the level of consump- The simulation is implemented as an SD model in Ventana
tion of the funded services. Consequently, the model is parame- Systems Inc. (2012), with data and parameter calculations han-
terized with hours of resource used per day per patient for each dled through Microsoft Excel (Microsoft Corporation, 2015) work-
HCC cohort and the admission rate for each cohort is linked to the books. This separation of parameters and simulation structure al-
availability of these five resources at the required quantity. There- lows for simple adjustments of patient cohorts and their character-
fore, for HCC, the highest utilized resource dominates the others as istics without making changes to the underlying simulation code.
the bottleneck resource, and controls the admission rate into the The parameters in the model were estimated from 2009 and 2010
sector. health system data for the Central East Local Health Integration
226 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237
Fig. 3. Simulation model data and information flows (ALC = Alternate level of care, LHIN = Local Health Integration Network, LOS = length of stay).
Network and model calibration is performed against 2011 and 2012
observations.
Fig. 3 provides a summary of information flows for the sim-
ulation setup. The user is able to change how this patient-level
data is aggregated in Excel to create new patient cohort group-
ings, which is then used to calculate cohort characteristics such
average LOS, disposition, and cross-LHIN referral rates. All model
parameters populated using historical data are dependent on this
cohorting and are calculated prior to simulation start. Patient co-
hort characteristics are calculated from patient-level extracts ob-
tained from IntelliHealth Ontario (MOHLTC, 2014) a business intel-
ligence tool that contains patient-level clinical and administrative
data for Ontario. The baseline cohort data sources and the group-
ings used in the simulation are provided in Appendix B. The user
is able to use the clinical groupings in Appendix B for targeting
interventions to flows for specific cohorts and demographics. The
clinical groupings of the simulation can be changed prior to the
simulation run if necessary. For example, different case mix groups
in the acute care sector can be merged to create a broader condi-
tion group. The baseline capacity for the simulated sectors, where
the user is able to use output from the Healthcare Indicators Tool
(Health Analytics Branch, 2012) to group certain types of provider
or resource types into capacity categories, resulting in aggregate
annual capacity estimates for each sector. Once these parameters
are passed to the simulation model, they can be modified during
a simulation run. However, cohort and capacity groupings remain
static. All data sources used by the model are readily accessible by
planners in the MOHLTC, LHINs and health service providers.
At simulation start, all stocks are populated with initial patient
volumes. For those sectors that have assessment-based datasets
(CCC, rehabilitation, LTCH and HCC), these values were calculated
from point-in-time volume of active patients. For emergency and
acute care sectors that have discharge-based datasets, the initial
patient volumes were estimated through a two-step process where
the simulation was run with static baseline parameters for a five-
year warm-up to obtain steady state estimates.
The simulation time window is between October 1st, 2010 and
March 31st, 2015 for 1641 days. Within this time window, the sim-
ulation starts with initial stock levels and warms-up within the
two simulated months. 2011 and 2012 resuls overlap with ob-
served data to allow for model input calibration against observed
values. The model runs in continuous time with the simulation
clock ticking in three-hour intervals with daily results collection,
where the simulation outputs at each three-hour tick are averaged
to represent the state of the simulation variables for that day. This
time step was chosen as it was half the minimum LOS for the ED
sector, which has the shortest LOS in the simulation model. The
runtime of the simulation is typically just under 25 seconds on a
system with 2.8 gigahertz Intel Core i7 processor and 16 gigabytes
of memory.
3.4. Validation and use
Whole-system models, like the one discussed in this paper, are
meant to be strategic analysis tools for ambiguous and complex
policy questions. It is a given that different policy questions will
require distinct cohort configurations and will evaluate options us-
ing flow outcomes unique to their objectives. Furthermore, inter-
ventions considered in transformation policies rarely have histor-
ical data to validate the model against. As such verifying that the
entire complement of model outputs closely replicate historical pa-
tient flow observations is not necessarily realistic.
Within the decision-making cycle, whole-system models play
the role of policy exploration tools as opposed to predictive fore-
casting models. As such, validation of strategic-level models is an
iterative process aimed at developing trust in the simulation by its
intended users (Barlas, 1996; Oliva, 2003). Brailsford et al. (2004)
suggest using both white-box (reviewing model in detail with
the user) and black-box (checking model output accuracy against
 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237
observed values) validation techniques to build confidence in the
model. Therefore, for this model we propose a three-step approach
to checking the validity of the simulation prior to using it for pol-
icy analysis:
Step 1: Work closely with the users to translate the policy op-
tions into model inputs and outputs, and collect hypothe-
ses around expected outcomes. This is a white-box process
where the clients are exposed to the model structure, the
underlying data and both the model and data limitations.
This process should result in the co-creation of scenarios to
be analyzed by the model and the performance indicators
that matter to the policy analysts.
Step 2: Calibrate the model to maximize the accuracy of the
model against the model outputs selected in step 1. This is
the black-box process where model inputs can be calibrated
to minimize error.
Step 3: Populate the model with the policy options and compare
outcomes against the system behavior hypotheses from step
1. At this stage, it is possible that model behavior is rejected,
or that the clients revise their policy outcome hypotheses
in light of the simulation analysis. This is the phase where
users are able to derive the most value out of the model as
they are able to use the model as a laboratory for learning
and testing their hypotheses (Graham, 2002). If necessary,
model parameters and structure can be revised to ensure be-
havioral validity.
This was the validation process used in the application of this
model for the analysis of stroke best practices policy, as presented
in the next section. From the beginning, the qualitative and quanti-
tative phases of model development were carried out in close col-
laboration with the end users, and they were aware of the assump-
tions and limitations of the model. Once the policy question was
determined, it was deconstructed through a facilitated session and
expected outcomes from implementation were gathered from the
literature and policy analysts. The policy analysis scenario was then
spelled out in terms of model inputs and performance indicators
that the simulation can produce and validated through the experts
prior to analysis using the model. Preliminary results were shared
with both policy and stroke experts for validation prior to conduct-
ing further exploratory analyses around the policy. This facilitated
validation and use cycle is a pragmatic approach to operational-
izing whole system models due to the complexity of the policy
questions traditionally explored in such analyses. Furthermore, as
many policy and clinical experts can be skeptical of the new quan-
titative models representing healthcare processes they are familiar
with 1, validity could be rejected prematurely without a facilitated
process. Not surprisingly, it can be seen that all of the literature
cited in Section 2 involved facilitated model use.
4. Case study – stroke best practices implementation
4.1. Stroke best practices overview
Stroke is the sudden loss of brain function due to blood flow
interruption to the organ, and can be caused by blood vessel
blockages due to clots (ischemic) or the rupture of blood vessels
(hemorrhagic). When a blood vessel blockage is temporary, the
ischemic stroke is classified as a transient ischemic attack (TIA)
(OSN, 2015b). Note that for the remainder of this paper the is-
chemic category will refer to both ischemic stroke and TIA pa-
tients together. Annually, in Ontario there are around 25,0 0 0 stroke
events, resulting in over 15,0 0 0 acute and 3500 inpatient reha-
bilitation admissions (Hall, Linkewich, Khan, & Levi, 2015; OSN,
2015a). The health system pathways for stroke patients are char-
acterized by transition delays and suboptimal referrals. Discharge
227
delays account for a third of the stroke patients’ stay in acute care
on average and only 60% stroke patients who are inpatient rehabil-
itation candidates in fact receive this care (Hall et al., 2012).
To address system-wide challenges with this cohort, Ontario’s
Rehabilitation and Complex Continuing Care Expert Panel estab-
lished stroke best practices (SBP) in 2011 based on the four rec-
ommendations made by the Ontario Stroke Network’s Stroke Ref-
erence Group:
1. Acute care: Timely transfer of appropriate patients from acute
facilities to rehabilitation, with five- and seven-day average LOS
targets for ischemic and hemorrhagic stroke, respectively.
2. Inpatient rehabilitation: Provision of greater intensity therapy
in inpatient rehabilitation (rehabilitation), with three hours of
therapy seven days a week, with an average LOS target of 28
days.
3. Outpatient rehabilitation: Timely access to outpatient or
community-based rehabilitation (OPCBR) for appropriate pa-
tients with three visits per week over an eight to 12-week pe-
riod.
4. Access: Ensure that all rehabilitation candidates have equitable
access to the rehabilitation they need.
The implementation of these evidence-based SBP recommen-
dations are expected to provide improved patient flow for stroke
patients across the system, along with an overall reduction in
bed utilization by this cohort in acute, inpatient rehabilitation
and complex continuing care sectors. The SBP effectively propose
a new stroke model of care at the health system-level, affecting
acute, inpatient, home and complex continuing care sectors and
introducing a much expanded OPCBR service. The expected pa-
tient pathway changes are presented in Fig. 4. For ischemic stroke
patients, the pathway is simplified by restricting post-acute care
to home, with 13% of them receiving OPCBR (Mayo et al., 20 0 0;
Meyer, Callaghan, Kelloway, & Hall, 2012). Hemorrhagic stroke pa-
tient pathways balance the level of care across the patient acuity
levels by diverting low acuity patients from inpatient rehabilita-
tion to OPCBR while moving home-bound complex continuing care
patients into inpatient rehabilitation. This pathway aims to reduce
acute LOS by sending hemorrhagic stroke patients to inpatient re-
habilitation earlier, and inpatient rehabilitation LOS by increasing
the intensity of rehabilitation received and moving the later stages
of this care to the community by providing OPCBR to all patients
discharged from this sector.
SBP is a cross-sector patient flow intervention, and therefore an
ideal policy scenario for the model. The case study presented here
was conducted in collaboration with the Implementation Branch of
the MOHLTC to study the effects of the implementation of the SBP
within a LHIN in Ontario. It builds on Meyer et al. (2012) calcu-
lations on the impact of SBP for the province, and extends it by
incorporating cross-sector flow feedbacks of the system to deter-
mine the extent to which SBP can achieve the targets it set out,
and estimates the whole-system patient flow effects of the policy.
4.2. Simulation setup and calibration
To configure patient flows in the simulation for the analysis of
SBP, we first aligned the stroke cohorts in the model to standard
Ontario definitions. In acute care, this involved creating two new
cohorts in acute care for hemorrhagic and ischemic stroke using
inpatient case mix group (CMG) codes (CMGs are the Canadian
version of Diagnosis-Related Groups; Fetter, Shin, Freeman, Averill,
& Thompson, 1980) based on MOHLTC’s resource indicator stan-
dards (Health Analytics Branch, 2016), where the hemorrhagic co-
hort contained both hemorrhagic and unspecified strokes, and as
discussed in Section 4.1, ischemic and TIA cohorts were combined
228 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237

Fig. 4. Stroke best practices changes to hemorrhagic and ischemic stroke cohorts flows.

Table 1
Model output validation for SBP analysis.

Sector Indicator Units Measure MPE (%)

Acute care Sector discharges Patients/month Annual total −5.5

Inpatient rehabilitation referrals Patients/month Percent of all discharges 2.3
CCC referrals Patients/month Percent of all discharges 1.3
Inpatient care ALOS Days/patient Annual average 6.3
Inpatient discharge delay Days/patient Annual average 3.9
Hemorrhagic stroke discharges Patients/month Annual total 3.7
Hemorrhagic stroke inpatient LOS Days/patient Annual average −0.6
Hemorrhagic stroke inpatient discharge delay Days/patient Annual average 10.0
Ischemic stroke discharges Patients/month Annual total −1.0
Ischemic stroke inpatient LOS Days/patient Annual average 9.3
Ischemic stroke inpatient discharge delay Days/patient Annual average 4.9
Inpatient rehabilitation Sector discharges Patients/month Annual total 4.3
Sector care ALOS Days/patient Annual average −0.5
Stroke discharges Patients/month Annual total −2.1
Stroke ALOS Days/patient Annual average 0.5
Complex continuing care Sector discharges Patients/month Annual total −4.4
Sector care ALOS Days/patient Annual average 4.6

under the ischemic stroke group. Inpatient rehabilitation data al-
ready had a stroke cohort defined through rehabilitation client
group (RCG) codes. It was confirmed that this cohort was the same
as Meyer’s et al. stroke group defined by the rehabilitation patient
group codes, which are derived from RCGs. The model’s existing in-
put levers were sufficient to translate the changes to LOS and dis-
position into simulation parameters. The only new variable added
to the simulation was a stock to keep track of OPCBR referrals in
the SBP scenario.
After mapping the cohorts, the simulation was run to calibrate
the model and establish a base case scenario. The model was pa-
rameterized with clinical and administrative data from 2009 to
2010 and run to end of 2012. We assessed model accuracy with
2011 and 2012 data to compare actuals with model results for sec-
tor discharge volumes, cross sector referral patterns, average LOS
and discharge delays. Mean percent error (MPE) metric was used
to compare the model estimates with actuals on a monthly basis. A
significant discrepancy was identified for the rehabilitation sector
throughput in our validation comparisons, stemming from a nearly
20% increase in average LOS in the sector over the comparison pe-
riod. The inpatient rehabilitation LOS was adjusted by +20% to ac-
count for this change. Validation results for the calibrated model
are provided in Table 1.
The MPE results for the selected indicators show that generally
the model has a slight under-estimation bias of discharge volumes,
and an over-estimation bias for lengths of stay. This is an expected
outcome due to the static nature of arrival rates possibly under-
estimating sector demand over the comparison period, as noted in
Section 3.1. For an aggregated model such as this, the bias indica-
tors exhibit an acceptable level of variation and in line with the
10% target from similar literature (Lane et al., 20 0 0).
The SBP simulation scenario was established by changing dispo-
sition and LOS parameters of the stroke cohorts in acute care and
inpatient rehabilitation sectors, and establishing OPCBR estimates
for those patients who are referred to this service. The scenario
parameters are summarized in Table 2. All LOS and disposition pa-
rameters use Ontario calculations from Meyer et al. (2012), with
the exception of two, which were recalculated with the rehabili-
tation patient group (RPG) mix of inpatient rehabilitation for the
model LHIN. The diversion rate of rehabilitation-bound acute hem-
orrhagic patients to home with care with OPCBR was determined
to be 5.5%, lower than Meyer’s et al. (2012) provincial estimate of
6.6% due to a lower portion of RPG 1160 patients in CE LHIN in-
patient rehabilitation settings compared to the province. Expected
ALOS for inpatient rehabilitation for stroke was calculated to be 32
days instead of the provincial 30-day estimate. Acute care diver-
sions of SBP were only applied to those stroke patients bound for
 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237 229

Table 2
SBP simulation scenario inputs. (OPCBR: outpatient/community-based rehabilitation).

Sector Cohort Definition ALOS Disposition

Acute care Hemorrhagic stroke CMGs 025 and 028 7 days • 5.5% of rehab
referrals diverted
to home with care
with OPCBR
• 29% of CCC
referrals diverted
to rehabilitation
• 13% of home and
home with care
referrals also
provided OPCBR

Ischemic stroke CMGs 026 and 027 5 days • All CCC and
rehabilitation
referrals diverted
to home with
OPCBR
• 13% of home and
home with care
referrals also
provided OPCBR

Rehabilitation Stroke
OPCBR Outpatient rehabilitation
Community rehabilitation
Speech language pathology
RCGs 10.1, 10.2, 10.3, 10.4 and 10.9 32 days All home and home
with care discharges
also provided OPCBR
88% of all OPCBR referrals 70 days All patients discharged
home
12% of all OPCBR referrals
Service added to 50% of all OPCBR referrals

Table 3
SBP scenario sensitivity analysis details. Note that all distributions are triangular (CCC: complex continuing care, LOS: length of stay, OPCBR:
outpatient/community-based rehabilitation).

Sector Scenario input Target Distribution parameters

Acute care Hemorrhagic stroke expected care LOS 7.0 days (7.0, 7.0, 8.9)
Ischemic stroke expected care LOS 5.0 days (5.0, 5.0, 7.7)
% of hemorrhagic stroke rehabilitation referrals diverted to home with care with OPCBR 5.5% (0.0, 5.5, 5.5)
% of hemorrhagic stroke CCC referrals diverted to rehabilitation 29% (0, 29, 29)
% of ischemic stroke rehabilitation referrals diverted to home 100% (0, 100, 100)
% of ischemic stroke CCC referrals diverted to home 100% (0, 100, 100)
Rehabilitation Stroke average care LOS 32 days (32, 32, 36)

rehabilitation and CCC. The stroke population that was discharged
to other sectors after acute care were assumed to continue doing
so at the baseline rate. To establish average OPCBR LOS, we used
the Ontario Stroke Network’s Stroke Reference Group’s recommen-
dation of 8–12 weeks discussed in Meyer et al. (2012) and took the
midpoint of 70 days.
While Meyer et al. (2012) based LOS and disposition effects of
SBP policy implementation on the analysis of provincial data, they
made a number of assumptions based on expert judgment to cal-
culate these estimates. The SBP scenario was therefore simulated
with sensitivity analysis to represent both the expert opinion na-
ture of the LOS and diversion estimates, and that being a new ser-
vice SBP will not be implemented perfectly everywhere resulting
in some patients who continue under the old baseline model. The
details of this sensitivity analysis is provided in Table 3. These sce-
nario inputs were represented using mode at bound triangular dis-
tributions. For LOS inputs, one bound was set to the baseline value
of the input and the other bound and the mode set to the SBP ex-
pert estimate. The diversion inputs were given a zero lower bound
and the SBP target for mode and upper bound. This bounded the
sensitivity range between current state and ideal SBP state with
more weight given to the SBP targets.
To estimate the effects of implementing the stroke policy, the
base case and the SBP scenario were simulated separately for years
2012–2015. The SBP scenario inputs were assumed to take effect at
the start of the simulation run with the results calculated from 50
sensitivity runs. The performance indicators selected capture the
policy’s effectiveness through LOS measurements for the stroke co-
hort and its effects for the overall system though assessing sector
throughput and capacity utilization changes. The simulation results
were collected to estimate the annual mean value of these compar-
ison indicators over the three-year period.
The entire case study was encapsulated in scenario analysis
dashboards developed for policy analysts using the primary inputs
and key performance indicators identified. The scenario setup (Fig.
5) and outcomes analysis dashboards (Fig. 6) enabled interaction
with the whole-system model without the need for exposing the
stock and flow view of the model.
4.3. Results and discussion
The implementation of SBP is expected to increase the through-
put of stroke patients through acute care and inpatient rehabilita-
tion sectors by reducing their LOS through increased care intensity
and by directing cohorts to community settings with lower dis-
230 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237

Fig. 5. Stroke policy input dashboard.

Fig. 6. Stroke policy scenario comparison dashboard.

charge delays. The effectiveness of the policy in shortening of the
LOS along the stroke pathway is summarized in Table 4. It can be
seen that SBP does in fact improve the LOS for the stroke cohort,
especially for ischemic patients. Targets for overall LOS are not
fully met due to discharge delays still experienced by stroke sub-
cohorts discharged to sectors that do not benefit from SBP such as
LTCH.
These results show that SBP’s LOS and cross-sector diversion
changes can result in significant patient flow improvements across
the stroke pathway, which translates into capacity recovered for
other patient cohorts and increased overall sector throughput.
Table 5 presents a summary of the implications of SBP implemen-
tation. When SBP is in effect, bed occupancy of the stroke co-
hort drops by 10.0 beds in acute and 11.1 beds in rehabilitation.
On average, this gain corresponds to 0.631% and 5.72% of the bed
 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237 231

Table 4
Length of stay comparison for stroke patients between base case and SBP implementation. (ALC: alternate level of care, LOS: length
of stay, SBP: stroke best practices).

Indicator SBP target average days Base case SBP case Difference
Average 95% CI [L, U] Average 95% CI [L, U] Average (SBP – base)

Acute care hemorrhagic 7 11.5 10.1 −1.40

Total average LOS [11.4, 11.6] [9.47, 11.0]
Acute care ischemic total 5 10.6 7.15 −3.45
Average LOS [10.5, 10.7] [6.27, 8.32]
Acute care hemorrhagic 0 2.92 2.47 −0.45
Average ALC LOS [2.88, 3.05] [2.34, 2.61]
Acute care ischemic average 0 2.28 1.30 −0.98
ALC LOS [2.25, 2.41] [1.17, 1.49]
Inpatient rehabilitation 32 39.2 37.3 −1.90
Stroke average LOS [39.1, 39.3] [35.9 39.2]
Inpatient rehabilitation 0 5.83 4.56 −1.27
Stroke average ALC LOS [5.58, 6.01] [4.41, 4.65]

Table 5
Resource utilization comparison between base case and SBP implementation at sector level for all patient cohorts. All figures are calculated as the difference between the
SBP case and the base case simulations. (ALC: alternate level of care, CCC: complex continuing care, N/A: not applicable).

Sector Reduction in bed utilization during care for stroke cohort Reduction in ALC bed utilization for all cohorts Sector admission rate change
Average beds/day 95% CI [L, U] Average beds/day 95% CI [L, U] Average patients/day 95% CI [L, U]

Acute care 10.0 22.9 4.15

[6.19, 13.5] [17.8, 26.0] [3.34, 4.90]
Rehabilitation 11.1 12.7 0.525
[5.95, 15.8] [6.74, 18.2] [0.263, 0.750]
CCC N/A 0.0 0.002
[−0.150, 0.0560] [0.0 01, 0.0 04]
Long-term care homes N/A N/A 0.126
[0.105, 0.147]
Home care N/A N/A 3.56
[2.70, 4.31]

capacities of these sectors respectively. SBP’s stroke cohort capac-
ity gains and the re-allocation of stroke patients from institutional
to shorter wait community destinations result in reduced ALC bur-
den and overall increased sector throughput. As the sectors are
inter-connected, the gains made by the stroke cohort are magni-
fied through system-wide flow improvement feedback, further re-
covering 22.9 beds in acute and 12.7 beds in rehabilitation sectors.
This implies that the improved systemic flow frees up 2.06 beds
in acute and 1.14 beds in rehabilitation for every bed freed up by
the stroke cohorts in those sectors. Additionally, this flow improve-
ment goes beyond the stroke cohort. For example, in acute care
the average ALC days for patents waiting for cardiac rehabilitation
placement are reduced by 1.17 days, 95% CI [0.590, 1.66].
For the LHIN the SBP implementation would require the accom-
modation of 460 OPCBR enrollments annually 95% CI [423, 501],
translating to an average of 88.8 active patients per day, 95% CI
[81.6, 96.6]. This case load change would require the LHIN to in-
crease speech language pathology/therapy visits by 145%, and oc-
cupational and physical therapy visits by 23% over the baseline
funded capacity. The patient flow gains of SBP are dependent on
these investments in OPCBR, which are the downstream enablers
of early discharges from acute and rehabilitation settings. The im-
portance of this investment can be demonstrated through the se-
lective implementation of the LOS and diversion components of
SBP. Fig. 7 compares the three different levels of SBP implemen-
tation: LOS Only – Only LOS changes for stroke, Diversion Only –
only cross-sector referrals changes, and full SBP implementation.
It can be seen that clinical LOS improvements only provide limited
gains, whereas cross-sector referral changes contribute significantly
to the improvements expected from the policy. As such, OPCBR in-
vestments will be critical to realize potential gains from SBP. What
is also important to note here is that both LOS and referral com-
ponents, when applied together provide greater gains than their
individual parts, highlighting the dynamic feedback of the flow
system.
The analysis shows that SBP is in fact a policy with significant
potential to improve patient flow for stroke patients and through
spill-over effects, other cohorts within the LHIN. The whole-system
analysis indicates that the policy’s success rests on the effective
implementation of the diversion policies and creation of commu-
nity capacity for OPCBR. This significant increase in community
support is a considerable barrier for the implementation of this
policy. While the finding that significant investments are needed
in OPCBR was not surprising to the policy analysts, the model was
able to not only quantify the case load but also present the reduc-
tion in utilization in other sectors as a result of expanding commu-
nity capacity. This evidence was welcomed by the policy analysts
as a crucial planning finding.
It should be noted that there are certain caveats to these con-
clusions. The policy effects presented in this analysis are likely op-
timistic estimates for three reasons. In assessing SBP outcomes,
we did not incorporate changes to stroke incidence. Gains made
by the stroke cohort could potentially be used towards accommo-
dating the increasing demand by the same group, reducing the
spillover effects of the policy. Furthermore, capacity and through-
put gains predicted may be inflated due to the scope of the
model, where certain sectors such as mental health are not en-
dogenously simulated. The model assumes arrival rates from ex-
ogenous sectors to be capped at historical rates and directs in-
creased admission rates to those patients waiting in endogenous
sectors, creating a throughput improvement feedback loop for en-
dogenous sectors. In reality, gains in capacity could be used to
improve admission rates from these exogenous sectors, reducing
the magnitude of the system-wide feedback loop by spreading
it across more sectors. Furthermore, while the analysis assumed
that any recovered bed capacity would be allocated according to
232 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237

Fig. 7. Box plot showing acute inpatient alternate level of care bed gains under different levels of SBP implantation (LOS = length of stay, SBP = Stroke Best Practices).

Fig. 8. Detailed stock and flow diagram of the care sector archetype. (ALC: alternate level of care).

historical priorities in the model it is possible that LHINs and in-
dividual providers may choose to prioritize admissions to these
beds differently or even retire the recovered capacity to save
costs.
5. Discussion and conclusion
The modeling described in this paper shows that high fidelity,
whole-system simulations can be built using Ontario’s data assets,
and be used as effective analysis tools for exploring transformation
policies. The model presented in this paper provides a simulation
formulation to include clinical characteristics in whole-system sys-
tem dynamics simulation models. This extension of the literature
is important as healthcare policies frequently target specific pa-
tient cohorts. Furthermore, through the case study of a care model
change for a specific clinical condition, we highlight the impor-
tance of considering such policies from a whole-system perspec-
tive to effectively estimate performance limitations brought about
by the constraints in the system, and to discover both positive and
negative unintended consequences on patients and providers out-
side of the policy scope. The modeling approach and the sector
archetype presented in this paper are likely applicable to other
health systems that capture patient flow data among care sec-
tors. The model would be especially valuable to single payer health
 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237 233

Table 6
Sector archetype variables and specifications.

Variable Description and specification

t Simulation time where t0 = simulation start time, tf = final simulation time. (1)
n Set of endogenous sectors in the model. (2)
n ∈ N, where N = {ed, ac, rh, cc, lt, hc},
where ed = emergency department, ac = acute inpatient, rh = inpatient rehabilitation,
cc = complex continuing care, lt = long-term care homes, hc = home and community care
Note: n = 0 is used to refer to “this sector” for which parameter and equations
specifications are provided for in this section.
e Set of exogenous sectors and dispositions in the model.
e ∈ E, where E = {deceased, eac, erh, ecc, elt, ehc, home without care, other, supportive
housing},
where eac = exogenous acute inpatient, erh = exogenous inpatient rehabilitation,
ecc = exogenous complex continuing care, elt = exogenous long-term care homes,
ehc = exogenous home and community care
i All endogenous or exogenous sectors to admit from or discharge to in the model. (4)
i ∈ I, where I = N ∪ E
c Patient clinical cohorts for this sector. (5)
c ∈ C for all endogenous sectors n, and C = {edc, acc, rhc, cxc, lhc, hoc},
where edc = emergency department cohorts, acc = acute inpatient cohorts, rhc = inpatient
rehabilitation cohorts, cxc = complex continuing care cohorts, lhc = long-term care home
cohorts, hoc = home and community care cohorts
C is further elaborated in Appendix B.
r Patient clinical cohorts for other endogenous sector n in the model. (6)
r ∈ Cn
d Demographic breakdown cohorts for stocks and flows. (7)
d ∈ {Female < 65, Female 65–75, Female > 75, Male < 65, Male 65–75, Male > 75}
j Resource type j that provides care capacity in the sector. (8)
j ∈ {case management hours, nursing visits, occupational therapy visits, physical therapy visits,
personal support visits} for i = hc and j = beds for all other i
Kj (t) Amount of resource j available, in resource units. (9)
Kj (t) is initialized from data and is a user modifiable parameter ≥ 0
Rcjn Units of resource j consumed by cohort type c per day for endogenous sector n, in (10)
resource units.
initialized f rom data as a modi f iable parameter ≥ 0, f or n = hc
Rc jn = {
1, f or all other n
RCj (t) Consumption of resource j at time t, in resource units. (11)

RC j (t ) = Ccd (t ) × R0
c d
CAc (t) Capacity of sector for cohort c defined as the minimum patients of that cohort that can be (12)
accommodated by the resources given the sector cohorts’ requirements defined in Rcjn . In
active patients per day.
min K j (t )Zc j
C Ac (t ) = ( Rc jn )
j
where Zcj is percent of resource j allocated to cohort type c per day for endogenous sector
n.
initialized f rom data as a modi f iable parameter ≥ 0, f or n = hc
Zc j = { (13)
1, f or all other n
An example may help with the interpretation for CAc (t). Assume that for home care, a
cohort requires 3 nursing hours, 2 personal support worker hours and 1 case management
hour. One patient can be admitted to home care if all three resources are at least at these
levels. Assume there were 10 units of each resource available, and all of these resources
are available to this cohort. Then the number of patients of this cohort that can be
admitted is min(10/3, 10/2, 10/1) = 10/3
U(t) Utilization of sector n at time t, in percent. (14)
RC (t )
U (t ) = CAj(t )
T(t) Utilization target of sector n, in percent. (15)
τ (t) is initialized by the user and is a modifiable parameter ≥ 0
A(t) Admission rate into this sector at time t, in patients per day. (16)
( (T (t ) − U (t )) · CA(t )) + D(t ), U (t ) < T (t )
A(t ) = { D (t ), U (t ) = T (t )
0, U (t ) > T (t )
HAid Historical arrival rates from sector i and demographic d, in patients per day (17)
HAid is initialized from data and is a modifiable parameter ≥ 0
HCin A matrix of form xdc and size |d| x |c| containing historical cohort c breakdown for (18)
demographic d in this sector coming from sector i into sector n. In percent.
HCin is initialized from data and is a modifiable parameter ≥ 0, where
|c |

xdc = 100 f or ∀ d
c
ARi (t) Percent of admission rate A(t) allocated to sector i at time t. (19)
ARi (t) is initialized from data and is a user modifiable parameter ≥ 0 for all i, where

ARi (t ) = 100
i
NXAn Percent of admission rate for an endogenous sector n redirected to its exogenous (20)
counterpart. Used to track out-of-LHIN admissions for endogenous sectors. In percent.
NXAn is initialized from data and is a user modifiable parameter, where 0 ≤ NXn ≤ 100.
XAcde (t) Number of patients of cohort c and demographic d admitted from exogenous sector e at (21)
time t in patients per day
X Acde (t ) = min( (A(t ) · ARe (t )) + (NX An · ARn (t )), H Aed ) × HCcde
(continued on next page)
234 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237

Table 6 (continued)

Variable Description and specification

Mi A matrix of form xrc and size | r | x | c | containing the mapping of cohorts r coming from (22)
sector i to this sector’s cohort c, where 0 ≤ xrc ≤ 1 and
|c |

xrc = 1 f or ∀ r
c
PRnrd (t) Patients discharged from endogenous sector n cohort r and demographic d, in patients per (23)
day.
For n = ED
P R(ED)rd = (Crdn (t ) + Wrdn (t ) ) × Enrd ,
where Enrd is the emergency visit rate per patient of cohort r and demographic d in
endogenous sector n.
For all other n (24)
P Rnrd = f (Wrdn (t ), ARn (t )),
where P Rnrd is equivalent to NDncd (t) specified in (36)
and where Wrdn (t) is equivalent to Wcdi (t) specified in (32).
NAcdn (t) Number of patients admitted from endogenous sector n with cohort c and demographic d (25)
at time t, in patients per day.
P Rnrd × Mi , for cohorts r from sector n specified in Mi
N Aicd (t ) = {
(P Rnrd ) × HCin ,
r for all other cohorts r from sector n
ICcd Number of patients with cohort c and demographic d in care at time to , in patients. (26)
ICcd is initialized from data and is a modifiable parameter where Icd ≥ 0 for ∀ c, d
Ccd (t) Number of patients with cohort c and demographic d in care at time t, in patients. (27)
tf   
Ccd (t ) = ∫ ( X Acde (t ) + N Acdn (t ) − CCcdi (t ))dt + ICcd
t0 e n i
Lcd Historical average length of stay for cohort c and demographic d, in days per patient. (28)
Lcd is initialized from data and is a modifiable parameter where Lcd ≥ 0 for ∀ c, d
HS A matrix of form xcdi and size | c | x | d | x | i | containing the market shares of sectors i (29)
for cohort c and demographic d pairs for this sector, where 0 ≤ xcdi ≤ 1 and
|i|
xcdi = 1 f or ∀ c, d pairs
i
HS is initialized from data, and is user modifiable. It is possible to direct patients to both
endogenous sectors N and their counterparts in E to track out-of-LHIN discharges for
endogenous sectors. In fact, these out-of-LHIN redirects are calculated at the time of the
initialization of HS
CCcdi (t) Number of care completions for cohort c and demographic d with target discharge to (30)
sector i, in patients per day.
CCcdi (t ) = CcdL (t ) × HS
cd
IWcdi Number of patients of cohort c and demographic d waiting for discharge to sector i at t0 (31)
IWcdi is initialized from data and is a modifiable parameter where IWcdi ≥ 0 for ∀ c, d, i
Wcdi (t) Number of patients of cohort c and demographic d waiting for discharge to sector i in (32)
service s at time t
tf
Wcdi (t ) = ∫ (CCcdi (t ) − X Dcde (t ) − N Dcdn (t ))dt + IWcdi
to
HDcde Discharge delay for cohort c and demographic d in waiting for exogenous sector e, in days (33)
per patient.
HDcde is initialized from historical data and is a modifiable parameter where HDcde ≥ 0 for
∀ c, d, e
XDcdsi (t) Exogenous patient discharges to sector e cohort c and demographic d at time t, in patients (34)
per day.
X Dcde (t ) = HWDcdi ((tt))
cde
RAn (t) Admission rate of endogenous sector n from this sector at time t, in patients per day. (35)
RAn (t ) = An (t ) · αn0 (t ),
where αn0 (t ) is the percent of admission rate An (t) of sector n allocated to this sector at
time t
NDcdn (t) Endogenous patient discharges to sector n cohort c and demographic d at time t, in (36)
patients per day.
N Dcdn (t ) = min(RAn (t ) × (DAcdn (t ) + SAcdn (t )), Wcdn (t )) where
cdn (t )
DAcdn (t ) =  W W (t )
is the share of discharge rate to sector n of patients in cohort c (37)
c d n cdn
and demographic d, proportional to their share in Wcdn , and is user modifiable

SAcdn (t ) = max(RAn (t ) − N Dcdn (t − 1 ), 0) × DAcdn is the slack variable that allocates (38)
c d

any unused discharge bandwidth from the previous time step, to ensure that N Dcdn (t )
c d
is as close to RAn (t) as possible
Ds (t) Discharge rate from service s at time t, in patients per day (39)
 
D(t ) = X Dcde (t ) + N Dcdn (t )
∀ ∀

systems where data capture is almost at the population level and
policy interventions can be enforced throughout the continuum of
care.
The case study in this paper is complementary to stroke litera-
ture in operational research. While there is a range of operational
research models published on stroke patient pathways through
the care system (Barton, McClean, Garg, & Fullerton, 2010; Bayer,
Petsoulas, Cox, Honeyman, & Barlow, 2010; Gillespie et al., 2016;
McClean, Barton, Garg, & Fullerton, 2011; Monks, Pearn, & Allen,
2015; Pitt, Monks, Crowe, & Vasilakis, 2016; Sundberg, Bagust, &
Terént, 2003; van der Zee & van der Vorst, 2007) the SBP anal-
ysis is unique in capturing the stroke dynamics within a whole-
 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237
system context together with other patient cohorts. This approach
allows treating the stroke cohort and the resources it uses not as
an independent care pathway but a connected component of the
broader health system. As such this case study addresses some of
the opportunities for operational research applied to stroke care
by Churilov and Donnan (2012), specifically the rehabilitation and
social community care problem areas from capacity planning and
public policy perspectives.
The types of policy questions that invoke whole-system think-
ing tend to be complex, with numerous uncertainties and edu-
cated guesses attached to them. In that light, the model presented
in this paper should be treated as a part of the policy analysis
toolkit along with other tools such as qualitative models, demand
and supply forecasts, and economic analyses. Therefore, success-
fully adopting the model into decision-making in this manner re-
quires modelers to be a part of the policy analysis team to fa-
cilitate the model’s parameterization from the range of evidence
available and to help policy analysts interpret the system behav-
iors that emerge from the model. Using dashboards as described
in Section 4.2 can help modelers and policy analysts co-develop
problem statements, policy scenarios and agree on key indicators
to compare them. The modeler can configure the simulation model
to the needs of the policy analysts and the analysts can use the
dashboard outputs to confirm model behavior. Once the dashboard
designs are confirmed by the policy analysts, they can use the
interactive dashboards for evaluating scenarios, and bring in the
modeler as necessary for modifications to the dashboards and in-
terpretation support. For example, the design and implementation
of the dashboards for the SBP analysis took two in-person meet-
ings with the policy analysts over five days, including data analysis
and parameter calculation. Minimal customization of the simula-
tion model was required since it contained almost all of the pa-
tient cohorts, flow levers and performance indicators by default.
This analysis process demonstrates that whole system models, un-
der the collaborative development conditions, may facilitate faster
turnaround for analytics to support policy analyses.
As a part of the collaboration with the Ministry, the model
with a user guide, dataset codes and case studies and the re-
lated dashboards were transferred to its Health Analytics Branch
in June 2015. While the model is not yet adopted as a policy tool
at the Ministry, it is being considered for expansion to the provin-
cial level. The knowledge transfer to the analysts in the branch
had challenges perhaps partly due to not involving them earlier
in the research project. These challenges have highlighted the im-
portance of co-development of such models not only between the
operational researcher and the users, but also with those who are
ultimately responsible for operating and maintaining the simula-
tion. While co-development with these stakeholders is not always
a priority for commissioners of simulation models, our experience
highlights that it needs to be a priority if health systems are inter-
ested in embedding strategic analytics into their decision-making
environment.
Stroke represents a relatively large patient cohort for the health
system, with around 1500 inpatient visits per year for the case
study LHIN. At this volume, the SD approach of aggregate flows
235
with average LOS and transition characteristics will support strate-
gic analyses. However, for rarer conditions with much lower an-
nual case counts, aggregate flows may hide individual variability
among patients that might have policy-relevant flow consequences.
Furthermore, given that patient flows are represented in through
mutually exclusive diagnostic conditions most responsible for the
care encounter at each sector, the model is not able to represent
comorbidities. This feature, coupled with the aggregation of flows
as opposed to modeling individual patient transitions does not al-
low the model to represent the accumulation of clinical conditions
as patients move through the health system. For planning scenar-
ios where rare conditions or patients’ change in frailty and co-
morbidities are important a more granular approach such as agent
based simulation models may be more appropriate (Marshall et al.,
2015a, 2015b).
There are some key next steps for improving the model. Firstly,
wait lists for services were not included in the flow network. While
improving delayed discharges and overall sector throughputs are
key performance considerations, wait list management is also a
critical aspect of system capacity planning. Ontario explicitly cap-
tures wait lists for surgical procedures, diagnostic imaging and
long-term care homes. Surgical and LTCH wait lists are potentially
the most reasonable expansion opportunities for the next itera-
tion of the model. Furthermore, primary care (general practitioner
services) and mental health services were excluded from endoge-
nous sectors in this iteration. While inpatient mental health can
be incorporated into the flow network as an endogenous compo-
nent, primary care is a complex sector. It is a critical component
of community-oriented system transformation initiatives, yet con-
ceptualizing its capacity and flows is not readily evident especially
because patients are generally rostered into primary care prac-
tices rather than being admitted to services along a care path. One
way to incorporate primary care into the model structure could
be through tying certain cross-sector patient flows to the inten-
sity and capacity of certain services offered in primary care, such
as screening and referral services. Finally, informal care capacity
and supports can also be incorporated as a demand and pathway
modifier in the cross-sector flow network once more rigorous data
collection is available for this relatively unquantified aspect of the
health system.
Acknowledgments
This research was supported by a research grant from the
Health System Strategy Division of the Ontario Ministry of Health
and Long-term Care.
Appendix A. Sector structure archetype formulation
This appendix provides formulation details of the sector
archetype of the model. Fig. 8 below provides further details on
the archetype structure visually, and builds on Fig. 2 in the body
of the paper. Table 6 below the figure presents the definitions and
formulations of the archetype elements.
236 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237

Appendix B. Baseline patient cohort groupings

ED Acute LTCH HCC CCC Rehabilitation

C edc acc lhc hoc cxc rhc

Source Canadian Canadian Institute for Canadian CCAC Home Canadian NRS
Institute for Health Information Institute for Care Database Institute for
Health Discharge Abstract Health Health
Information Database Information Information
National Continuing Continuing
Ambulatory Care Reporting Care Reporting
Care Reporting System System
System
Cohort Canadian Triage Canadian Institute for Referral Source CCAC Service Resource Rehabilitation
variable and Acuity Health Information Program Utilization Client Group –
Scale – CTAS Case Mix Groups Group – RUG RCG
Cohort 1. CTAS 1 1. Blood-Lymphic 1. Acute 1. Post-Acute 1. Clinically 1. Amputation
divisions 2. CTAS 2 2. Burns 2. CCC 2. In-Home Complex Care 2. Brain Injury
3. CTAS 3 3. Circulatory 3. Community Long Stay 2. Cognition 3. Cardiac
4. CTAS 4 4. Digestive 4. HCC 3. 3. Extensive 4. Fracture, Hip
5. CTAS 5 5. Endocrine 5. Other Rehabilitation 4. Physical 5. Fracture,
6. CTAS 6. Ear/Nose/Throat 6. Rehab 5. Special Care Other
Unclassified 7. Eye 7. Supportive or 6. Special 6. Medical
8. Female Reproductive Assisted Living Rehabilitation, 7. Neurological
9. Hepato-Pancreas High Intensity 8. Orthopaedic,
10. MH 7. Special Other
11. MH-Dementia Rehabilitation, 9. Other
12. MSK Low Intensity 10. Pulmanory
13. MSK-Knee Repl. 8. Special 11. Hip
14. MSK-Hip Repl. Rehabilitation, Replacement
15. Multi-Unspecified Medium 12. Knee
16. Neuro Intensity Replacement
17. Neuro-Hemorrhagic 9. Special 13. Stroke
Stroke Rehabilitation,
18. Neuro-Ischemic Ultra-High
Stroke and Transient Intensity
Ischemic Attac 10. Special
19. Other Rehabilitation,
20. Pregnancy Birth Very-High
21. Renal-Urinary Intensity
22. Respiratory
23. Skin-Breast
24. Trauma-Poisoning

Falk, W., Mendelsohn, M., Hjartarson, J., & Stoutley, A. Fiscal sustain-
References ability & the transformation of Canada’s healthcare system: A shift-
ing gears report. Mowat Centre for Policy Innovation. [Online]. Avail-
able:. (2016). https://mowatcentre.ca/wp-content/uploads/publications/
Barlas, Y. (1996). Formal aspects of model validity and validation in system dynam-
38_fiscal_sustainability_and_transformation_canada.pdf Accessed: 24-May.
ics. System Dynamics Review, 12(3), 183–210.
Fetter, R. B., Shin, Y., Freeman, J. L., Averill, R. F., & Thompson, J. D. (1980). Case mix
Barton, M., McClean, S., Garg, L., & Fullerton, K. (2010). Modelling costs of bed occu-
definition by diagnosis-related groups. Medical Care, 18(2) pp. i–53, Feb.
pancy and delayed discharge of post-stroke patients. In Proceedings of the 2010
Gillespie, J., McClean, S., Garg, L., Barton, M., Scotney, B., & Fullerton, K. (2016). A
IEEE workshop on health care management (WHCM) (pp. 1–6).
multi-phase DES modelling framework for patient-centred care. The Journal of
Bayer, S., Petsoulas, C., Cox, B., Honeyman, A., & Barlow, J. (2010). Facilitating stroke
the Operational Research Society, 67(10), 1239–1249 May.
care planning through simulation modelling. Health Informatics Journal, 16(2),
Graham, A. K. (2002). On positioning system dynamics as an applied science of
129–143 Jun.
strategy. In Proceedings of the 2002 international system dynamics conference.
Brailsford, S. C., Lattimer, V. A., Tarnaras, P., & Turnbull, J. C. (2004). Emergency and
Hall, R., Khan, F., O’Callaghan, C., Kapral, M., Hodwitz, K., Fang, J., et al. (2012).
on-demand health care: Modelling a large complex system. Journal of the Oper-
Ontario stroke evaluation report 2012. Institute for Clinical Evaluative Sci-
ational Research Society, 55(1), 34–42 Jan.
ences [Online]. Available: http:// www.ices.on.ca/ ∼/media/ Files/ Atlases-Reports/
Brailsford, S. (2007). Tutorial: Advances and challenges in healthcare simu-
2012/Ontario- stroke- evaluation- report/Full%20Report.ashx [Accessed: 20-Dec-
lation modeling. In Proceedings of the 2007 winter simulation conference
2015].
(pp. 1436–1448).
Hall, R., Linkewich, B., Khan, F., & Levi, J. (2015). Ontario and LHIN 2013/14
Brailsford, S. (2008). System dynamics: What’s in it for healthcare simula-
– stroke report cards and progress reports. Institute for Clinical Evaluative
tion modelers. In Proceedings of the fortieth conference on winter simulation
Sciences 15-Jun-2015. [Online]. Available: http:// www.ices.on.ca/ Publications/
(pp. 1478–1483).
Atlases- and- Reports/ 2015/ Ontario- Stroke- Report- Cards Accessed: 20-Dec.
Churilov, L., & Donnan, G. A. (2012). Operations research for stroke care systems: An
Hans, E. W., van Houdenhoven, M., & Hulshof, P. J. H. (2012). A framework for
opportunity for the science of better to do much better. Operations Research for
healthcare planning and control, Handbook of healthcare system scheduling: 168
Health Care, 1(1), 6–15 Mar.
(pp. 303–320). Boston, MA: Springer, US.
CIHI, “Population risk adjustment grouping project,” cihi.ca, (2015). Canadian In-
Health Analytics Branch (2016) “Readmissions within 30 days for selected CMGs,”
stitute for Health Information. [Online]. Available: https://www.cihi.ca/en/pia_
health.gov.on.ca, 14-Jul-2014. Ontario Ministry of Health and Long-Term Care,
prag_en.pdf [Accessed: 11-Mar-2017].
Health System Information Management and Investment Division Health Ana-
Drummond, D., & Burleton, D. Charting a path to sustainable health care in Ontario:
lytics Branch. [Online]. Available: http://www.health.gov.on.ca/en/pro/programs/
10 proposals to restrain cost growth without compromising quality of care. TD
ris/docs/readmission_within_30days_selected_cmgs_en.pdf [Accessed: 14-May].
Economics Special Reports. Toronto (ON): TD Bank Financial Group; 2010. [On-
Health Analytics Branch. Health analyst’s toolkit (2012). Ontario Ministry of Health
line]. Available:. (2016). https://www.td.com/document/PDF/economics/special/
and Long-Term Care, Health System Information Management and Investment
td- economics- special- db0510- health- care.pdf [Accessed: 24-May].
Division Health Analytics Branch. [Online Available: http://www.health.gov.
Esensoy, A. V., & Carter, M. W. (2015). Health system modelling for policy develop-
on.ca/english/providers/pub/healthanalytics/health_toolkit/health_toolkit.pdf Ac-
ment and evaluation: Using qualitative methods to capture the whole-system
cessed: 24-May-2016].
perspective. Operations Research for Health Care, 4, 15–26 Mar.
 A.V. Esensoy, M.W. Carter / European Journal of Operational Research 266 (2018) 221–237
Hulshof, P. J. H., Kortbeek, N., Boucherie, R. J., Hans, E. W., & Bakker, P. J. M. (2012).
Taxonomic classification of planning decisions in health care: a structured re-
view of the state of the art in OR/MS, 1(2), 129–175. http://doi.org/10.1057/hs.
2012.18.
Johns Hopkins HealthCare ACG® System (2017). [Online Available: https://www.
hopkinsacg.org/ Accessed: 11-Mar-2017].
Lane, D. C., Monefeldt, C., & Rosenhead, J. V. (20 0 0). Looking in the wrong place
for healthcare improvements: A system dynamics study of an accident and
emergency department. The Journal of the Operational Research Society, 51(5),
518–531.
MacLeod, H. (2007). “Health results team third annual report 2006–07: A focus on
results and sustainability,” Queen’s Printer for Ontario.
Marshall, D. A., Burgos-Liz, L., IJzerman, M. J., Crown, W., Padula, W. V., Wong, P. K.,
et al. (2015a). Selecting a dynamic simulation modeling method for health care
delivery research-part 2: Report of the ISPOR dynamic simulation modeling
emerging good practices task force. Value Health, 18(2), 147–160 Mar.
Marshall, D. A., Burgos-Liz, L., IJzerman, M. J., Osgood, N. D., Padula, W. V., Hi-
gashi, M. K., et al. (2015b). Applying dynamic simulation modeling methods in
health care delivery research-the SIMULATE checklist: Report of the ISPOR sim-
ulation modeling emerging good practices task force. Value Health, 18(1), 5–16
Jan.
Martin, R. L. Building better health care: Policy opportunities for Ontario.
09-Apr-2014. Institute for Competitiveness & Prosperity [Online]. Available:.
(2016). http://www.competeprosper.ca/work/working_papers/working_paper_20
[Accessed: 24-May 2016]. ISBN: 978–11–9927065–009–99.
Mayo, N. E., Wood-Dauphinee, S., Côté, R., Gayton, D., Carlton, J., Buttery, J.,
et al. (20 0 0). There’s no place like home: An evaluation of early supported dis-
charge for stroke. Stroke, 31(5), 1016–1023 May.
McClean, S., Barton, M., Garg, L., & Fullerton, K. (2011). A modeling framework that
combines Markov models and discrete-event simulation for stroke patient care.
ACM Transactions on Modeling and Computer Simulation (TOMACS), 21(4), 25–26
Aug.
Meyer, M., Callaghan, C. O., Kelloway, L., & Hall, R. (2012). The impact of moving to
stroke rehabilitation best practices in Ontario. Toronto: Ontario Stroke Network.
Microsoft Corporation. (2015). Microsoft excel. Redmond, WA: Microsoft Corporation.
MOHLTC. (2014). IntelliHEALTH ONTARIO. Toronto no. SAS BI.
MOHLTC, “Backgrounder: Aging at home strategy,” ontario.ca, (2007). [On-
line]. Available: https://news.ontario.ca/archive/en/2007/08/28/McGuinty-
Government- Transforming- Community- Living- To- Help- Seniors- Live- Independe.
html [Accessed: 10-Dec-2015].
MOHLTC, “Patients first: Action plan for health care,” health.gov.on.ca, (2015). [On-
line]. Available: http://www.health.gov.on.ca/en/ms/ecfa/healthy_change/docs/
rep_patientsfirst.pdf [Accessed: 10-Dec-2015].
MOHLTC, “Ontario’s Action Plan For Health Care,” health.gov.on.ca, (2012a). [On-
line]. Available: http://www.health.gov.on.ca/en/ms/ecfa/healthy_change/docs/
rep_healthychange.pdf [Accessed: 10-Dec-2015].
MOHLTC, “About health links,” news.ontario.ca, (2012b). [Online]. Available: https://
news.ontario.ca/mohltc/en/2012/12/about- health- links.html [Accessed: 14-May-
2016].
237
MOHLTC, “Patients first – A proposal to strengthen patient-centred health care
in Ontario,” health.gov.on.ca, (2015). [Online]. Available: http://www.health.gov.
on.ca/en/news/bulletin/2015/docs/discussion_paper_20151217.pdf [Accessed: 07-
May-2016].
Monks, T., Pearn, K., & Allen, M. (2015). Simulation of stroke care systems. In Pro-
ceedings of the 2015 winter simulation conference (WSC) (pp. 1391–1402).
Office of the Auditor General of Ontario. (2015). Annual Report 2015. Office of the
Auditor General of Ontario auditor.on.ca [Online]. Available: http://www.auditor.
on.ca/ en/ content/ annualreports/ arreports/ en15/ 3.08en15.pdf [Accessed: 24-May-
2016].
Oliva, R. (2003). Model calibration as a testing strategy for system dynamics models.
(Dec. European Journal of Operational Research, 151, 552–568.
Ontario Ministry of Finance. (2014). Ontario population projections, 2013–2041: On-
tario and its 49 regional municipalities, counties and districts. Ontario Ministry of
Finance.
OSN. (2015a). Stroke stats and facts. OSN ontariostrokenetwork.ca [Online]. Available:
http://ontariostrokenetwork.ca/information- about- stroke/stroke-stats-and-facts/
Accessed: 23-Dec-2015].
OSN. (2015b). Information about Stroke. OSN ontariostrokenetwork.ca,[Online]. Avail-
able: http://ontariostrokenetwork.ca/information- about- stroke/ [Accessed: 24-
Dec-2015].
Pitt, M., Monks, T., Crowe, S., & Vasilakis, C. (2016). Systems modelling and sim-
ulation in health service design, delivery and decision making. BMJ Quality &
Safety, 25(1), 38–45 Jan.
Rashwan, W., Abo-Hamad, W., & Arisha, A. (2015). A system dynamics view of the
acute bed blockage problem in the Irish healthcare system. European Journal of
Operational Research, 247(1), 276–293 May.
Ronson, J. (2006). Local health integration networks: Will ‘made in ontario’ work?
Healthcare Quarterly, 9(1), 46–49 Jan.
Royston, G., Dost, A., Townshend, J., & Turner, H. (1999). Using system dynamics to
help develop and implement policies and programmes in health care in Eng-
land. System Dynamics Review, 15(3), 293–313 Sep.
Sterman, J. (20 0 0). Business dynamics: Systems thinking and modeling for a complex
world with CD-ROM. McGraw-Hill Education.
Stewart, M. (2016). Quality-based procedures. Health Achieve. [Online]. Avail-
able: http://www.healthachieve.com/2014/Presentations%202014/Health%
20Information%20Management_Michael%20Stewart.pdf . Accessed: 08-May.
Sundberg, G., Bagust, A., & Terént, A. (2003). A model for costs of stroke services.
Health Policy, 63(1), 81–94 Jan.
van der Zee, D., & van der Vorst, J. (2007). Guiding principles for conceptual model
creation in manufacturing simulation. In Proceedings of the thirty-ninth confer-
ence on winter simulation: 40 years! The best is yet to come (pp. 776–784).
Ventana Systems Inc. (2012). Vensim. Harvard, MA: Ventana Systems Inc..
Wolstenholme, E., Monk, D., McKelvie, D., & Smith, G. (2008). Influencing and in-
terpreting health and social care policy in the UK. In E. B. H. Qudrat-Ullah,
H. Qudrat-Ullah, J. M. Spector, & P. I. Davidsen (Eds.), Understanding complex
systems, no. 9 (pp. 155–183). Springer Berlin Heidelberg.