Environmental Science and Policy 111 (2020) 63–73

Contents lists available at ScienceDirect

Environmental Science and Policy

journal homepage: www.elsevier.com/locate/envsci

Common institutional design, divergent results: A comparative case study of T

collaborative governance platforms for regional water planning
Emily Bella,*, Tyler A. Scottb
a
Nicholas School of the Environment, Duke University, United States
b
Department of Environmental Science and Policy, University of California, United States

ARTICLE INFO ABSTRACT

Keywords: Environmental governance challenges often span geographic and sectoral boundaries, requiring collaboration
Collaborative governance between diverse stakeholders and multilateral decision making. To facilitate such efforts, policymakers and
Regional water planning public managers create and support platforms that provide a structured framework for promoting collaborative
Natural language processing governance. As the collaborative platform model is often centrally initiated – e.g., when state or national actors
create a system of local collaborative resource management platforms – cross-case comparison is particularly
important for understanding how a common design leads to variance in procedures and outputs in local contexts.
Many collaborative platforms leave a “paper trail” of documents such as meeting records and plans. This analysis
compares 10 identically designed and simultaneously initiated regional water planning platforms in the State of
Georgia. Drawing on 106 meeting reports, we apply topic modeling to these meeting documents to generate
replicable and scalable measures of how participant actions and interest representation unfold over time.
Specifically, we measure topical focus on water planning issues over time, and compare these process-phase
measures between councils and against the content of the resulting plan developed by each planning council.
While existing literature has focused on how institutional design features such as representation and decision-
rules shape procedural outcomes and outputs, we observe considerable variation in procedural behavior and
plan outputs despite the fact that all 10 platforms share a common design. The consulting firm selected to direct
each local platform is shown to be associated with both the topical focus of each regions’ planning discussion and
the BMPs selected in regional plans. This comports with recent evidence pointing to the important – and largely
overlooked – role that technical consultants play in environmental governance and regulatory processes.

1. Introduction platform design, simultaneously implemented across local regions, re-

Although collaborative governance sometimes emerges through
organic, “bottom-up” processes (Koontz and Newig, 2014; Guerrero
et al., 2015), increasingly the environmental governance literature re-
cognizes that public agencies use collaborative governance tools stra-
tegically to further policy goals (Emerson and Nabatchi, 2015a; Newig
and Koontz., 2014). One prominent approach is for federal, state, or
regional policymakers to create, convene, and support a system of local
platforms (such as regional forums, work groups, and planning coun-
cils) that provide a structured framework for promoting collaboration
(Ansell and Gash, 2018). Policymakers and convening agencies need to
structure local platforms in such a way that they adhere to regulatory
mandates and achieve central policy aims, but also allow for local in-
fluence and stakeholder participation. This paper explores a simple
question in this regard: To what extent does a common collaborative
sult in consistent processes and outputs? And as a follow-up question,
what factors help to explain observed differences?
Our observational research design takes advantage of an empirical
case where a convening authority (the state of Georgia [United States])
concurrently initiated ten regional integrated water resource manage-
ment (IWRM, see Grigg, 2019) planning processes occurring from 2009
to 2011. Existing studies of distributed collaborative platform models
compare individual platforms at different time points and/or within
highly varied policy and regulatory contexts (e.g., Schneider et al.,
2003; Ulibarri, 2015b). In this instance, the ten IWRM processes were
initiated simultaneously, given similar resources, tasks, and timelines,
and conducted under a common regulatory authority. Each planning
process was documented in meeting summaries, which allows us to
measure and compare patterns of discussion within each region over
time. Using automated content analysis of these documents, we assess

⁎
Corresponding author.
E-mail addresses: emily.bell@duke.edu (E. Bell), Davis.tascott@ucdavis.edu (T.A. Scott).

https://doi.org/10.1016/j.envsci.2020.04.015
Received 24 June 2019; Received in revised form 13 April 2020; Accepted 24 April 2020
Available online 04 June 2020
1462-9011/ © 2020 Elsevier Ltd. All rights reserved.
E. Bell and T.A. Scott
each deliberative process and how this corresponds to subsequent
planning outputs.
In the following section, we discuss how internal (i.e., platform-
specific) and external (i.e., region-specific) dynamics are theorized to
shape collaborative environmental governance processes. Next, we
provide an overview of the Georgia regional planning case and the data-
collection processes. We then describe the automated content analysis
approach we use, and how this method fits within recent efforts to
develop objective, replicable, and scalable measures of collaborative
governance processes and outputs. Our analysis: (1) shows how colla-
borative platform processes vary as a function of contextual drivers
such as shared consultants; and (2) compares topical emphases ob-
served in each planning process to the content of the regional water
plans developed by each planning council. The concluding section re-
flects on the theoretical and methodological implications of this study
for collaborative governance scholarship, and outlines future directions
for data collection and content analysis that can further our under-
standing of collaborative institutional design.
2. Drivers of collaborative governance processes
Collaborative governance refers to institutional processes and struc-
tures that support coordination and cooperation among managers and
policy decision makers from different organizations, administrative le-
vels, and sectors (Emerson and Nabatchi, 2015b). Practitioners use
collaborative governance to address transboundary or trans-sector en-
vironmental policy issues such as water resource management (Scholz
and Stiftel, 2010) and land use and transportation planning (Lubell
et al., 2010), where the scale and scope of policy problems and existing
authorities are mismatched.
Public managers sometimes collaborate directly with other stake-
holders, but also use a variety of strategies to foster collaboration
among stakeholders themselves (Koontz et al., 2004). Drawing from the
technology world, Ansell and Gash (2018) coin the idea of collaborative
platforms to describe institutional structures that promote and support
collaboration (Ansell and Gash, 2018). Support might include tangible
benefits such as administrative support, facilitation, and technical re-
sources, and rules that use incentives or require consultation and en-
gagement (Scott and Thomas., 2017). Often, collaborative platforms are
used to locally implement centrally mandated policies and programs.
For instance, the state of California’s Sustainable Groundwater Man-
agement Act requires local water agencies and users to develop basin-
level management plans (Kiparsky et al., 2017), the U.S. Environmental
Protection Agency’s (EPA) National Estuary Program engages regional
stakeholders at 28 major estuaries to design and implement conserva-
tion and management plans (EPA, 2015), and the U.S. Federal Energy
Regulatory Commission (FERC) supports collaborative licensing and
operational planning processes for privately owned hydropower facil-
ities nationwide (Ulibarri, 2015a).
While these types of collaborative approaches are normatively
popular, much remains unresolved about how and why different in-
stitutional designs shape outputs and outcomes (Newig et al., 2018).
Institutional design is critical for the distributed collaborative platform
model to ensure program fidelity and coherence across local platforms
while still allowing for local input and flexibility (Kiparsky et al., 2017).
Design features such as decision rules, size, representation, and lea-
dership structure and frame how platforms function (Choi and
Robertson., 2014; Dobbin and Lubell, 2019; Parker et al., 2010). An
important question in this regard is what variation policymakers should
expect within a distributed collaborative platform model. In other
words, what level of consistency in process and outputs might be ex-
pected when higher-level policymakers deploy the same collaborative
platform design in multiple local contexts?
Existing evidence points to a variety of external and internal factors
that shape collaborative platform implementation. First, heterogeneity
in local environmental, institutional, and demographic features means
64
Environmental Science and Policy 111 (2020) 63–73
that each platform faces a unique array of challenges (Hughes and
Pincetl., 2014). Thus, platform processes and outputs may differ to the
extent that stakeholders face heterogeneous management challenges
(Koontz and Newig, 2014) and different institutional contexts (Lubell
and Lippert., 2011). To understand internal drivers of collaborative
processes, recent work has focused on how stakeholder participation
shapes collaborative processes and outputs. Newig et al. (2018) posit
that the nature and strength of public participation in a collaborative
platform shapes the extent to which a given local platform exhibits
unique and/or divergent results. In other words, a common platform
design will tend to beget common results until or unless the involve-
ment of local actors exerts a unique influence. Platform implementation
choices also shape outputs. Focusing on the same set of Georgia IWRM
cases considered herein, Scott and Carter (2019) find a strong corre-
spondence between generated plan content and the private consultants
contracted to facilitate each local regional process.
Many comparative studies of collaborative platforms have relied
upon content analysis of platform outputs and post hoc surveys of par-
ticipants (Koebele, 2019b; Biddle and Koontz., 2014; Ulibarri, 2015b).
Tracing how collaborative processes unfold over time has proven more
elusive. However, many collaborative platforms generate procedural
documents such as meeting summaries, public comments, and other
textual data that provide a longitudinal record of how a given process
unfolds. These texts can be used to shed light on attendance, discourse,
power dynamics, adaptation, and various other procedural dynamics
(Scott et al., 2020; Koski et al., 2018; Purdy, 2012; Emerson and
Gerlak., 2014). Due to the difficulty of accessing these procedural data
and of scaling up content analysis, much of the process-oriented col-
laborative governance research has focused on single cases, with Hui
et al. (2018) being a notable exception.
This paper takes advantage of automated content analysis tools,
namely topic modeling, applied to procedural documents from ten iden-
tically designed, simultaneously implemented collaborative platforms to
compare process trajectories. Our analysis seeks to evaluate the extent to
which distributed platforms exhibit consistent behavior, and to understand
what contextual drivers are associated with different trajectories and
outputs. In the next section, we provide an overview of the Georgia re-
gional planning case that enables a qualitative, comparative study.
3. Background: regional water planning in the State of Georgia
This study focuses on collaborative regional water planning pro-
cesses in the State of Georgia. After decades of mounting pressure from
population growth, drought, and increased competition for water re-
sources (Coastal Georgia Council, 2011), the Georgia General Assembly
approved a State Water Plan in 2008. The plan divided the state into ten
planning regions and mandated that each develop a Regional Water
Plan to account for local – but interconnected – water management
issues (Georgia Water Council, 2008). The plan also authorized the
state governor to appoint a body of 40 stakeholder representatives to a
planning council for each region. The objective of the council ap-
pointments was to provide opportunities for representation of “local
governments, water users, and other water-related interests in each
planning region” (Georgia Water Council, 2008). In practice, this meant
that participants included water managers such as local governments
and water utilities, representatives of industries such as agriculture,
manufacturing, and energy production, and development and con-
servation interest groups. Councils were tasked with three primary
areas of focus: (1) water resource assessment and monitoring, (2)
forecasts of water supply, demand, and quality, and (3) identification of
management practices to meet social needs and to protect water re-
sources (Georgia Water Council, 2008).
To support each platform in the planning process, the Georgia
Environmental Protection Department (EPD) hired an environmental
consulting firm for guidance and facilitation (Fig. 1). Bidding and selection
occurred on a region-by-region basis, resulting in the hiring of four
E. Bell and T.A. Scott
Fig. 1. Georgia Water Planning Regions and consulting designations.
separate firms, with each firm responsible for either two or three of the ten
planning regions. Consulting agency representatives followed a common
set of state-established guidelines to direct each regional process.
From 2009 to 2011, stakeholders in each region met every two months
to ultimately generate their respective plans by a common deadline.
Appointed council members and planning consultants convened every
meeting throughout the duration of the planning process. State re-
presentatives and technical experts (e.g., hydrogeologists, foresters, etc.)
also intermittently participated by giving presentations, answering ques-
tions, and providing other means of support. For the duration of the
process, consultants followed state requirements by facilitating discussion
between council members to establish terms that would be included in a
memorandum of agreement between the planning council, the Georgia
EPD, and the Georgia Department of Community Affairs (DCA).
Consulting firm representatives also led group exercises, surveying parti-
cipant concerns and knowledge about regional trends. This helped to
frame objectives of the planning process and familiarize stakeholders with
regional water capacities and needs. In subsequent meetings, consulting
responsibilities entailed updating council members and other stakeholders
with state-developed regional assessments, monitoring data for water use
and discharge, and provided projections for future energy, municipal, in-
dustrial, and agricultural use.
During the planning process, each region’s council appointed sub-
committees of council members to choose best management practices
(BMPs) from a state-generated list of options for inclusion in the draft plan.
These subcommittees also held responsibility for developing plan subsec-
tions, which followed a predefined structure that had been developed by
the Georgia EPD.1 Subcommittee actions were embedded in an iterative
1
All plans, for example, included standard sections on topics such as current
water resources, forecasts and future water needs, goals to address water needs,
and implementation strategies.
65
Environmental Science and Policy 111 (2020) 63–73
deliberation process – BMP and Plan language recommendations to the
council often resulted in subsequent subcommittee review and con-
sideration. Where subcommittees could not reach consensus on BMPs or
language to include in the draft, the council served as an arbiter. Over
time, this strategy moved the draft plan content close to the participants’
collective goals until the council finally approved the draft plan by con-
sensus to submit for EPD approval.
The consistent state-designed institutions and timeframe across all
water planning regions creates an opportunity to systematically com-
pare topics of deliberation between processes and over time. We use
summary documents of these meetings to assess the topical focus and
trajectory of each deliberation process. In the next section, we describe
and deploy the automated content analysis approach we use and how
this method fits within recent efforts to develop objective, replicable
and scalable measures of collaborative platform processes.
4. Methods: automated content analysis
To evaluate the dynamics of each collaborative process, we apply
topic modeling2 to meeting summary documents. By modeling topical
focus on a meeting-by-meeting basis, this approach provides a detailed
account of specific time points in the planning process (Hui et al.,
2018). The method of topic modeling we use is latent
Dirichlet allocation (LDA) (Blei et al., 2003). LDA is a common topic
modeling method and has been employed in a variety of environmental
governance research applications (e.g., Grubert and Siders, 2016;
Altaweel et al., 2019). Applications include hazardous waste problem
definition in Congressional hearings (Nowlin, 2016), media portrayals
of natural resource controversies (Blair et al., 2016; Sachdeva et al.,
2
Topic models are statistical models used in natural language processing to
detect abstract topics that appear in a corpus, or collection of documents.
E. Bell and T.A. Scott
Fig. 2. Topic model of stemmed text from Georgia RWP reports, 2009–2011.
2018), and public perceptions about water conservation (Gao et al.,
2018).
In brief, a topic model represents the content of documents within a
corpus based upon the relative prevalence and co-occurrence of words and
phrases in each document (Grubert and Siders., 2016). The aim is to word
and phrase content on a reduced set of dimensions, or topics, that re-
present latent concepts or themes. LDA is a Bayesian model that estimates
the probability distribution of documents across latent topics, and the
probability distribution of these latent topics across words or n-grams3
(Blei et al., 2003). The number of latent topics is prespecified, and the
model then associates words with topics – which allows us to identify the
nature of a given topic – and the topic(s) of a given document.
To train our LDA model, we examined n-grams present in more than
three documents, but no more than 68% of the documents in the
corpus. The objective of these cutoffs is to exclude words that are so
ubiquitous they will be found no matter the topic (e.g., filler words such
as the and and, and words which are generic to this case context such as
water or council), as well as words which appear too infrequently to
estimate their topical distribution.
For LDA, there is no single agreed-upon approach to selecting the
number of topics to analyze. To address this, we tested: (1) the “elbow”
heuristic, which attempts to find a topical cutoff point using k-means
clustering (Evangelopoulos et al., 2012; Kosinski et al., 2016); and (2) a
multiple measure fit index operationalized within the ldatuningpackage
in R (Nikita, 2016). However, both were inconclusive (see Appendix A).
Thus, we turned to an iterative approach, adjusting the number of to-
pics until the groups of elements reflected logical and separable topics
(Feuerriegel et al., 2016). We find that fitting three latent topics gen-
erated groupings that provide the most interpretable results and appear
empirically grounded in the regional water planning process.4
3
An n-gram refers to an n-length portion of a longer string. A bigram, for
example, could be any pair of words, such as stream buffer or interbasin transfers;
trigram examples might include terms such as water use efficiency or non-point
source pollution. We used unigrams, bigrams, and trigrams.
4
LDA is stochastic, meaning that the different results can occur each time the
model is fit. To ensure replicability, it is necessary to also set a seed (i.e., the
starting value for R’s random number generator). To assess the degree of pos-
sible variation, we ran ten models, each with different seeds from our original
model. To examine consistency, we used three different measures. First, we
used average descriptor set difference, which shows us the extent to which top
66
Environmental Science and Policy 111 (2020) 63–73
4.1. Data sources and processing
Our primary data consists of 106 meeting reports generated by ten
different IWRM processes in the 2009–2011 planning period. These
data are published on the Georgia Water Planning website.5 Profes-
sional facilitators contracted by the Georgia EPD both guided meetings
and wrote memos summarizing each meeting that occurred in each
region. All memos followed a similar structure, beginning with meta-
data on the sender, receiver, date, and location. The procedural docu-
ments also contain data showing stakeholder attendance, participant
actions, and interest representation. Next, these documents outlined
meeting agendas, discussion and activities that transpired for each
meeting item, and a roster of attendees which categorized participants
by their role in the planning process (e.g., water council members,
public attendees, state agency representatives, etc.). All report content
was written in prose (as opposed to shorthand), enabling a systematic
analysis of meeting events across documents and regions.
Prior to analysis, we used a series of standard techniques to pre-
process the text, including tokenization, named entity recognition
(NER), stemming, and removal of stop words. Through tokenization, we
generated base units of analysis from raw text input, such as words,
numbers, symbols and punctuation, allowing us to filter out non-
(footnote continued)
terms in the topics overlap (we measured a score of 0.18), where 0 would be
perfect overlap (more stability in the topics) and 1 would be no overlap (the
least stability). Second, we used topic-term stability to see the proportion of
overlap between two topics to the total number of terms in the topic when we
use different seeds for the model. This produced a score of 0.69 (1 being perfect
overlap), suggesting that that majority of the terms in two topics – even with
different seeds – tend to overlap. Finally, with partition stability, we were able
to assess the extent of agreement on the topics the documents covered. This
measures, for example, whether the frequency of topic emphasis we observe in
the temporal plot is accurate. A score of 1 indicates perfect stability in what the
document text will be labeled (e.g., top emphasis in Topic 1 vs. Topic 2, etc.).
The partition stability produced a score of 0.58, indicating how well the topic
labels on documents in one model (with one seed) predict the topic labels on
documents produced by a model with another seed. At random chance, this
would be generally a much smaller score (e.g., 0.02), so this score aligns with
an expected score for a well-fit LDA model. For further reference on these ap-
proaches, please see (Belford et al., 2018).
5
https://waterplanning.georgia.gov/
E. Bell and T.A. Scott
Fig. 3. Topical emphasis by region and meeting.6
essential tokens. Next, we used NER to identify references to real-world
referents such as people or places. In our case, NER was useful for fil-
tering out these entities by removing tagged NER tokens for people,
places, and organizations. Because each IWRM process involves dif-
ferent local places and local stakeholders, named entities create in-
herent content differences between process documents that cloud our
ability to model substantive differences of interest. Thus, we removed
person and place tokens to specifically analyze and compare planning
discussions of water management issues and practices.
We then stemmed the words to examine only the root of the tokens
(e.g., actors, acting and acted would become act). This ensured max-
imum detection of words with a common meaning. Finally, we removed
“stop words” (common words that do not convey meaning or context)
such as the, and, and a in order to remove noise and improve the pre-
cision of our analysis. To account for frequent words common in front
matter (e.g., titles, page headers, and review of prior meetings), we
excluded terms such as council and meeting as additional stop words.
5. Results
The results section proceeds as follows: First, we present topic
model results across processes and over time and discuss the validity of
these findings. Second, we review the regions’ respective planning
outputs. We then summarize planning outputs and assess how topic
model results reflecting deliberation in different planning processes are
associated with differences in planning outputs.
67
Environmental Science and Policy 111 (2020) 63–73
5.1. Topical emphases in IWRM planning processes
Fig. 2 shows the top 10 terms associated with each of the three
estimated latent topics fit to the text of all observed meeting reports. In
this figure, the y axis refers to the elements for each topic that appeared
most frequently, and the “beta” term on the x axis refers to the loga-
rithmized parameter of each element (word) in a topic, or the model’s
value for that word—higher weights reflect a stronger element asso-
ciation with a given topic.
Elements of the first topic are primarily standard terms and proto-
cols for regional planning. Unigrams and bigrams, including procedure,
moa (“memorandum of agreement”), oper procedur (“operating proce-
dure”), and epd response (i.e., “Georgia Environmental Protection
Division response”) were common terms in several of the regional
planning documents.
The second topic includes terms relating to water quality and
quantity. For example, nutrient and standard could refer to nutrient
standards established in accordance with the Clean Water Act (CWA),
requiring states to regulate pollutant quantities in water bodies.
Standards were also central to planning discussions of provision topics,
such as variation in water withdrawal permitting, which could impact
assimilative capacity in streams. Another common point of discussion in
technical expert and council review was the sustainable yield of regional
6
All regions concurrently participated in the planning process but reports for
some meetings were not available.
E. Bell and T.A. Scott Environmental Science and Policy 111 (2020) 63–73

Table 1 For example, Topic 2 was virtually absent in meeting report content for
Descriptive water quality and quantity statistics by region.a the ALT, CGA, and SSA regions (which contracted CDM Smith), was
Region Average Percentage of Total miles no or Total miles very slightly more evident in the CNG, MOC, SUO, and UOC regions (using
surface aquifers where exceeded good assimilative CH2M Hill and JJG Consulting), and showed comparatively much
water gap demand exceeds assimilative capacity greater frequency over time in the LFO, MCH, and UFL regions7 (Black
(MGD) safe yield capacity and Veatch Consulting). This comports with a prior finding in the case
ALT 62.60 NA 339 1334
of Georgia water planning by Scott and Carter (2019) that consultant
CGA 38.10 NA 383 861 firms are a key influence in collaborative planning processes. Like Topic
CNG 11.64 NA 67 1019 2, the third topic – containing a collection of regional planning and
LFO 227 33% NA NA water quantity elements – shows some variation linked to regions
MCH 365 0% NA NA
whose planning meetings were facilitated by the same consulting firm.
MOC NA NA 43 506
SSA 87.6 0% 120 522 The degree of emphasis of Topic 3 also varies spatially – only planning
SUO NA 0% 267 626 regions that were spatially contiguous with the MNGWPD showed at
UFL 227 25% 8.80 NA least moderate-to-high levels of frequency in emphasis in the planning
UOC NA 0% NA NA
period. This pattern alone does not equate to inferences about the
planning process, but indeed directs attention to natural resource
1NA = either no data were available, or results were presented in visual
(mapped) format. management challenges that may warrant further attention in future
a
Plans discussed future quality and quantity projections, but these often did research.
not present numerical estimates.

5.3. Variation in topical emphasis

water, or the volume of groundwater resources that could be withdrawn
without negative implications. In the meetings, technical experts often
led discussion by explaining model results for current and projected
regional water status. During these processes, council members also
examined current regulatory frameworks, namely the US Army Corps of
Engineers Water Control Plan, designed to promote water management
safety, conservation, and compliance with federal legislation for major
water control infrastructure (USACE, 2016). The term control also sur-
faced in relation to erosion and sedimentation control during har-
vesting, and proposed EPA requirements to control nutrients (i.e., ni-
trogen and phosphorus), which was of key interest following a federal
lawsuit under the Clean Water Act. Following discussion, consensus was
a common term that appeared in discussions in deciding best man-
agement practices to include in the regions’ respective water plans.
The third topic is less clearly defined (with lower beta measures and
greater element diversity) than the first and second topics. Nonetheless,
the third topic does exhibit several elements linked to water quantity
issues, as well as regional planning goals and decisions. For example,
minimum, close, and node often appeared with regard to topics such as
minimum flow regime for regions’ rivers and whether there was a gap
to close between water-use needs and availability; these nodes would be
points of data collection along the river for expert water resource as-
sessments. Given water conditions during that period, there were sev-
eral occasions in planning meetings in which council members asked
technical experts about the potential for interbasin transfers to alleviate
future potential water shortages. Finally, the word district often sur-
faced when council members in regions adjacent to the Metropolitan
North Georgia Water Planning District (MNGWPD) discussed the future
implications of urban growth and how this would impact their own
water resources.
5.2. Topical emphases over time
Fig. 3 plots estimated topical distributions by meeting and region for
the 2009–2011 planning period. Two key trends in this schematic are
the temporal prominence of topics at specific points in the planning
process, and place-based variation. Emphasis of the first topic – regional
planning and protocols – peaks early on in the planning period for all
regions, albeit to different extents. Topic 1 may have surfaced earlier on
in the process due to discussion about the memorandum of agreement
required between planning councils, the Georgia EPD, and the DCA to
establish operating procedures and rules for meetings.
Temporal patterns for the second topic – covering aspects of water
quality and quantity – varied across regions by consultation leadership.
Despite some common regional trends in topical emphasis, there
remains the question of why some planning processes would exhibit
practically no emphasis for some of the topics. For example, why would
Water Quality and Quantity – a very important topic in this planning
context – have virtually no mentions over time in the Altamaha (ALT),
Coastal Georgia (CGA), and Suwannee Satilla (SSA) region meeting
reports?
To address this question, we compared the ALT regional features
with the CNG and MCH regions which showed moderate, and high le-
vels of temporal emphasis for Topic 2, respectively. The MCH – as
discussed earlier falls within the Apalachicola-Chattahoochee-Flint
RiverBasin, which has been characterized by several decades of com-
peting water uses, severe drought, and associated water-quality chal-
lenges linked to the assimilative capacity of water bodies such as lakes
and streams (Vaughan, 2001; Leitman, 2008). Planning in this region
discussed model outputs for planning resource assessments to discuss in
inter-regional joint meetings in the Basin that were organized by the
EPD (Cash, 2010). Complex hydrogeology in the CNG region creates
highly-localized interactions between ground and surface water sys-
tems, which motivated discussion about accessing new reservoirs to
reduce storage costs for meeting future demand and improving assim-
ilative capacity during low-flow times8 (Brownlow, 2009). The ALT
region council members and consultants examined options to address
future low-flow periods and storage needs (Brown and Zitsch, 2010),
but this precluded discussion about reservoir access or issues of com-
peting demand.
5.4. Topical emphasis of meetings and regional water plan content
With LDA we highlight latent topical dimensions not readily evident
in reported meeting deliberations, but these alone do not tell us about
planning decisions. In this section, we manually review state approach
regional water plans (RWPs) developed by each planning council, and
compare topical emphases from our model with plan output. We then
discuss key insights from these comparisons.
7
The LFO, MCH, and UFL regions each fall largely within the ACF River
Basin.
8
Low-flow periods can impact the assimilative capacity of water bodies, or
their ability to receive wastewater without detrimental impacts on aquatic
wildlife or humans consuming the water (GA EPD, 2009).

68
E. Bell and T.A. Scott Environmental Science and Policy 111 (2020) 63–73

Fig. 4. Shared best management practice selections between regional water plans.

5.5. Comparing RWP emphasis on water quality and quantity with best surface water gaps in MGD and showed relatively little emphasis in the
management practices planning meeting content. Unlike the quantity measurements, assim-
ilative capacity appeared to have no clear patterns in relation to re-
We anticipated that Topic 1 – logistics of the planning process – gional topic emphasis.
would not manifest as planning decisions in and of itself, so we com-
pared the latent water quality and quantity topic (Topic 2) to regional
5.6. Water quality and quantity BMPs
plans to evaluate patterns between process and output. Specifically, we
focused on RWP assessments of i) water quality and quantity resources
Next, we compared the temporal emphases of the water quality and
(Section 2) and ii) future resource capacity and needs (Section 5). By
quantity topic to BMPs selected by each region for their respective plans.
comparing temporal emphases of the topic model with the plan sec-
Specifically, we evaluated the extent to which regions with similar topical
tions, we could assess similarities and differences for key areas of
emphasis also shared common BMPs. Fig. 4 maps shared water quality and
concern. For projected surface and groundwater quality and quantity,
quantity BMP selections between planning regions. The thickness of the
these included (i) assimilative capacity,9 (ii) aquifer capacity and “safe”
edge connecting two regions reflects the number of BMPs selected in
or “sustainable” yield,10 and (iii) surface water availability, as recorded
common between those two regions. For instance, ALT, SSA, and CGA
at different planning nodes.11
regions have 17 identical BMPs in their respective plans.
Descriptive measurements of regional water quality and quantity
Comparing the observed topical emphases from meeting delibera-
(Table 1) showed similar patterns to topical emphasis in the meeting
tions with the selected BMPs, we saw strong similarities among regions
reports. The LFO, MCH, and UFL regions—which had the highest
guided by common contracted consulting firms. Fig. 4 shows, for ex-
average gaps between surface water supply and demand in average
ample, that the ALT, SSA, and CGA regions included a high number of
MGD—also showed high frequency and emphasis for Topic 2. Alter-
shared water quantity and quality BMPs in their respective plans.
nately, the ALT, CGA, and CNG regions had the lowest measured
Common representatives from CDM Smith Consulting supported and
facilitated the collaborative decision-making process for the ALT, SSA,
and CGA regions. Likewise, the LFO and UFL regions also share a re-
9
Assimilative capacity refers to the amount of pollutants that can be released latively high number of BMPs, as do the CNG and UOC regions – these
in a watershed while still meeting water quality standards; issues such as fecal pairs received planning assistance from the Black and Veatch and
coliform, dissolved oxygen levels, and pH imbalance may be sufficient to cause CH2M Hill consulting firms, respectively. Given this, one could infer
a water body not to meet standards and be placed on the 303(d) list of impaired that consultants not only influence planning process dynamics, but have
waters to be addressed by total maximum daily load (TMDL) standards (MOC
some influence on plan outputs. However, there are also cases where
Regional Water Planning Council, 2011).
10 neighboring regions without the same hired consultants chose many
Sustainable yield refers to an amount of groundwater extraction that can be
sufficiently recharged or stored. identical BMPs (e.g., CNG and MCH regions). Given the regional clus-
11
Planning nodes are stations located along rivers that collect longitudinal tering of firms (e.g., CDM Smith served all three regions in the south-
data from measurements of river flow; these data can inform computational east/coastal part of the state), it is difficult to tease out the relative
models of the severity and duration of gaps between supply and demand for influences of common geographic context and common consultant
future projections (ALT, 2011). guidance.

69
E. Bell and T.A. Scott
6. Discussion and conclusion
Policymakers who fund and initiate collaborative decision-making
processes face many uncertainties about how the process will unfold
and what the results will be. This analysis shows how a simple auto-
mated content analysis method, topic modeling, can be applied to
meeting summary documents to compare the procedural dynamics of
multiple collaborative platforms and link these dynamics to subsequent
outputs. Despite a shared institutional design and strong contextual
similarities between the ten cases analyzed, we observe major differ-
ences both in how platforms proceed with IWRM planning and the
management plans they generate. Interestingly, these differences do not
appear to correspond to policy conditions such as projected water de-
mands or pollution issues. Instead, the environmental consulting firm
hired to support each council is a stronger predictor both of which
topics participants discuss and which BMPs a council selects. While the
normative view of collaborative environmental governance often fo-
cuses on stakeholder inclusion and participatory decision-making, the
role and influence of the ‘consultocracy’ (e.g., documented in other
policy sectors by Jupe and Funnell, 2015; and Gunter et al., 2015) re-
mains an understudied issue. In what follows, we conclude by first re-
flecting on these findings, and second on the promise and problems of
using automated content analysis to research collaborative governance.
6.1. Theoretical reflection
As discussed previously, a fundamental institutional design chal-
lenge for collaborative governance platforms is achieving the con-
sistency and regularity necessary to satisfy procedural aims and ensure
program fidelity while still allowing flexibility necessary for success in
different local contexts (e.g., Kiparsky et al., 2017). Often, the literature
has focused on how representation (who is invited to participate) and
decision-rules (how collective decisions are made) shapes procedural
outcomes and outputs (Choi and Robertson., 2014; Koski et al., 2018;
Brown, 2002; Siddiki et al., 2015). However, in the case of Georgia
IWRM planning, we observe considerable variation in procedural be-
havior and plan outputs. Considering that all ten cases considered
herein share a common institutional design and many contextual si-
milarities, it is noteworthy that we observe many different discussion
trajectories.
Instead, the consulting firm selected to direct each local platform is
shown to be strongly associated with both the topical focus of each
regions’ planning discussion and the BMPs selected in regional plans.
This comports with recent evidence pointing to the important role that
expert technical consultants play both in Georgia water planning spe-
cifically (Scott and Carter., 2019) and in environmental governance
writ large (Edelenbos et al., 2011; Sardo and Weitkamp., 2017).
The observed correspondence between plans and consultants in this
exploratory analysis points to several testable hypotheses going for-
ward. One particularly interesting line of future inquiry relates to the
role that experts play in generating knowledge within collaborative
platforms. Emerson and Nabatchi (2015a) describe collaborative gov-
ernance as having emergent properties, meaning that the collective
enterprise can create or result in collective conditions that do not exist
within or cannot be created by the constituent parts. Knowledge and
knowledge creation in particular have been shown to play a key role in
building emergent capacity for environmental governance (van der
Molen, 2018). When consultants are viewed as co-producers of
knowledge within a collaborative platform, the particular expertise and
experience of the technical consultants involved might be hypothesized
to play a more influential role in how collaboration unfolds than if
consultants are viewed more as facilitators. Coupling more detailed
information about the technical consultants supporting a collaborative
platform – e.g., past projects, training and education, firm emphases –
with collaborative platform process and output measures is necessary to
test this influence.
70
Environmental Science and Policy 111 (2020) 63–73
Further, while technical context remains relatively constant in our
analysis – since all platforms are engaged in IWRM planning – we also
expect that the relative influence of expert consultants is likely to vary
depending on the informational needs of platform participants and the
technicality of the problem(s) at hand. For instance, in the Georgia
IWRM case appointed council members were primarily representatives
of local governments and interest groups, and not necessarily technical
experts. In a different context where participants have more technical
issue expertise and experience, consultants might play a more facil-
itative role. Similarly, different environmental governance issues pre-
sent different levels of technical complication and informational needs.
To the extent that collaborative platform decisions pertain to science
and engineering questions, rather than issues such as stakeholder co-
ordination, value disputes, or distributional choices, we hypothesize
that the relative influence of technical consultants on collaborative
processes and outcomes will vary accordingly.
6.2. Methodological reflection
Collaborative environmental governance platforms often generate
considerable “paper trails” (e.g., meeting summaries, scoping and draft
documents, comments and reviews, etc.). This paper presents an ex-
ample of how automated text analysis techniques can be applied to
analyze and extract information from these documents. While the
analysis above is descriptive in nature, it serves to demonstrate a
scalable and replicable approach for measuring how collaborative
platforms function over time. LDA is anything but novel, but the extent
of its application in this research area is limited.
Obviously, the immediate appeal of automated content analysis is that
automation makes it much easier for a researcher to assess and characterize
document content. Perhaps more promising, however, is automated con-
tent analysis methods provide a means for scaling up comparative research
designs. Topic modeling and similar tools provide a means of consistently
measuring and comparing texts across cases and time. For instance, the
LDA model fit to the Georgia IWRM cases for 2008–2010 can now be
applied to meeting summaries from subsequent plan update processes or
IWRM planning cases in an entirely different place to generate objective
measures of how content differs in these cases and contexts.
Traditional qualitative coding methods are much more difficult to
replicate and extend in this fashion, particularly between different re-
searchers. In contrast, automated content analysis is highly scalable
since additional data inputs can be readily incorporated, and in most
cases, replication is largely a matter of deploying a common fitting
algorithm. Thus, while automated content analysis does not obviate the
need for detailed case knowledge, one can easily see the important role
that topic modeling and similar tools can play in large scale research
networks. It is within such networks that multiple research teams could
conduct longitudinal studies of collaboration and coordination in
complex governance systems.
Our exploratory analysis also demonstrates several challenges when
using topic modeling in empirical environmental governance research.
First, topic modeling is often unsupervised, meaning that the user does
not know a priori what words and phrases will “hang together” nor the
optimal number of groups of those objects. A key consideration for
researchers using LDA, then, is how to assess the construct and content
validity of topics selected. Measured topics are typically defined in-
ductively by evaluating the distribution of words associated with dif-
ferent topics and the distribution of topics within documents. Defining
and distinguishing topics is easiest when the topics differ more.
Tutorials and model tests often use corpuses such as the 20 Newsgroups
data set (Rennie and Lang., 2008), which contains news articles per-
taining to diverse topics such as science, religion, and sports. These
categorical distinctions make it much easier to rely on contextual clues
to define topics than in the case of the IWRM meeting documents we
analyze (because all topics are water-related and conceptually inter-
related).
E. Bell and T.A. Scott Environmental Science and Policy 111 (2020) 63–73

This challenge of clearly measuring and distinguishing subtopics
within a conceptual domain is not limited to topic models. In the course
of this analysis we also attempted to classify document content using
pre-trained semantic similarity measures fit to a massive open-domain
corpus (see Mikolov et al., 2013).12 This approach likewise struggled to
provide meaningful and clearly interpretable topical distinctions be-
cause all content was (correctly) identified as being within the general
context of water. Domain-specific semantic representation models (e.g.,
Hamilton et al., 2016) offer to improve measurement of subtopics
within a specific policy domain. However, training a model on a par-
ticular corpus affects how well the model transfers to other domains
(e.g., different governance scales or related issues). For example, if a
model is trained to examine local water management processes, its
ability to scale to other policy issues, geographic extents, or adminis-
trative levels may vary depending on whether contexts have greater or
fewer topics, different topics, or elements that comprise these topics.
Thus, there are tradeoffs and benefits associated with both domain-
specific and open-domain content analysis approaches. In any case,
developing domain specific models for particular environmental gov-
ernance applications is also a promising avenue for future work – given
the specialized terms and technical issues associated with different
environmental issue areas, content analysis models customized for a
given domain likely offer more specificity and nuance.
Finally, while text mining and natural language processing (NLP)
tools enable large-scale, objective measurement of certain aspects of
collaborative platforms, these tools perhaps offer the most promise as
part of mixed methods designs. Surveying or interviews with
stakeholders that participated in the Georgia regional planning process,
for example, could speak to key subjective factors such as participant
learning (Koebele, 2019b), issue coordination (Koebele, 2019a), and
power dynamics (Purdy, 2012; Dobbin and Lubell, 2019). Likewise,
consequential interactions such as “hallway conversations” at meetings
and discussions that occur outside of formal group events and meetings
are not shown in procedural documents, so automated content analysis
alone can potentially miss key elements of the story. Nonetheless, au-
tomated content analysis and other NLP tools offer a means by which to
answer the call for larger comparisons and longer analyses to better
understand the causal pathways and outcomes of collaborative en-
vironmental governance (Thomas and Koontz, 2011; Koontz and
Thomas, 2006; Newig et al., 2018).
CRediT authorship contribution statement
Emily Bell: Conceptualization, Investigation, Software,
Visualization. Tyler A. Scott: Supervision, Data curation,
Conceptualization.
Declaration of Competing Interest
The authors declare that they have no known competing financial
interests or personal relationships that could have appeared to influ-
ence the work reported in this paper.

Appendix A

See Figs. A1 ans A2

Fig. A1. An elbow analysis to determine the number of appropriate topics to include in an LDA model. This analysis presents no elbow, however, so hyperparameters
required informed judgement.

12
While a full discussion of semantic similarity models is beyond the scope of this paper, in brief, we used the word2vec (Mikolov et al., 2013) word embeddings
model. Word embeddings represent words and phrases as vectors in a multi-dimensional context space. Dimensional values are estimated based on the co-occurrence
of words with surrounding words, with the underlying logic being that words that occur nearer to one-another in text are more contextually similar. Whereas topic
models model content using a “bag of words” approach, word embeddings are estimated by predicting a target word based on the nearest surrounding words (the
“continuous bag of words” approach) or by predicting surrounding words given the target word (the “skip-gram” approach) (see Mikolov et al., 2013). In applying
word embeddings to the Georgia IWRM corpus, the shortcoming we thus encountered was that water issues such as water quality, water demand, and water returns
are of course contextually similar. Thus, attempting to measure how much a given meeting document related to water quality versus water demand, for instance,
resulted in unmeaningful (and sometimes nonsensical) results.

71
E. Bell and T.A. Scott
Fig. A2. An LDA Tuning model, which draws on four different algorithms for detecting the ‘best’ number of topics. This is supposed to be determined from the point
at which the functions level out, but there is no clear indication of this.
Appendix B. Supplementary data
Supplementary material related to this article can be found, in the online version, at doi:https://doi.org/10.1016/j.envsci.2020.04.015.
References
ALT, 2011. Altamaha Regional Water Plan. 2011. June 10, 2019. https://
waterplanning.georgia.gov/document/publication/2011-altamaha-regional-water-
planpdf/download.
Altaweel, M., Bone, C., Abrams, J., 2019. Documents as data: a content analysis and topic
modeling approach for analyzing responses to ecological disturbances. Ecol. Inform.
51 (May), 82–95.
Ansell, C., Gash, A., 2018. Collaborative platforms as a governance strategy. J. Public
Adm. Res. Theory 28 (1), 16–32.
Belford, M., Mac Namee, B., Greene, D., 2018. Stability of topic modeling via matrix
factorization. Expert Syst. Appl. 91 (January), 159–169.
Biddle, J.C., Koontz, T.M., 2014. Goal specificity: a proxy measure for improvements in
environmental outcomes in collaborative governance.”. J. Environ. Manage. 145
(December), 268–276.
Blair, B.D., Weible, C.M., Heikkila, T., Evensen, D., 2016. Comparing human and auto-
mated coding of news articles on hydraulic fracturing in New York and Pennsylvania.
Soc. Nat. Resour. 29 (7), 880–884.
Blei, D.M., Ng, A.Y., Jordan, M.I., 2003. Latent dirichlet allocation. J. Mach. Learn. Res.:
JMLR 3 (January), 993–1022.
Brown, A.J., 2002. Collaborative governance versus constitutional politics: decision rules
for sustainability from Australia’s South East Queensland forest agreement. Environ.
Sci. Policy 5 (1), 19–32.
Brown, R., Zitsch, K., 2010. Letter to coastal Georgia regional water planning council.
2010. In: Council Meeting 7 - Summary. November 17.
Brownlow, R., 2009. Coosa-North Georgia Council Meeting #4 Summary. CH2M Hill.
Cash, T., 2010. Letter to council members. In: Joint Water Planning Council Meeting.
February 11, 2010.
Choi, T., Robertson, P.J., 2014. Deliberation and decision in collaborative governance: a
simulation of approaches to mitigate power imbalance. J. Public Adm. Res. Theory 24
(2), 495–518.
Coastal Georgia Council, 2011. Coastal Regional Water Plan.
Dobbin, K.B., Lubell, M., 2019. Collaborative governance and environmental justice:
disadvantaged community representation in California Sustainable groundwater
management. Implementing SGMA: results from a stakeholder survey. Policy Stud. J.
(December). https://doi.org/10.1111/psj.12375.
Edelenbos, J., Buuren, A., Schie, N., 2011. Co-producing knowledge: joint knowledge
production between experts, bureaucrats and stakeholders in dutch water
72
Environmental Science and Policy 111 (2020) 63–73
management projects. Environ. Sci. Policy 14 (6), 675–684.
Emerson, K., Gerlak, A.K., 2014. Adaptation in collaborative governance regimes.
Environ. Manage. 54 (4), 768–781.
Emerson, K., Nabatchi, T., 2015a. Collaborative Governance Regimes. Georgetown
University Press, Washington, D.C.
Emerson, K., Nabatchi, T., 2015b. Collaborative Governance Regimes. Georgetown
University Press.
EPA, 2015. Overview of the National Estuary Program. 2015. December 6, 2019.
https://www.epa.gov/nep/overview-national-estuary-program.
Evangelopoulos, N., Zhang, X., Prybutok, V.R., 2012. Latent semantic analysis: five
methodological recommendations. Eur. J. Inf. Syst. 21 (1), 70–86.
Feuerriegel, S., Ratku, A., Neumann, D., 2016. Analysis of how underlying topics in fi-
nancial News affect stock prices using latent dirichlet allocation. In: 2016 49th
Hawaii International Conference on System Sciences (HICSS). Ieeexplore.Ieee.Org.
pp. 1072–1081.
GA EPD, 2009. Clean water. Presented at the Environmental Compliance Workshop.
https://archive.epa.gov/region4/ead/news/web/pdf/cwa_corrections_ga.pdf.
Gao, Y., Church, S.P., Peel, S., Prokopy, L.S., 2018. Public perception towards River and
water conservation practices: opportunities for implementing urban stormwater
management practices. J. Environ. Manage. 223 (October), 478–488.
Georgia Water Council, 2008. Georgia Comprehensive State-Wide Water Management
Plan.
Grigg, N.S., 2019. IWRM and the Nexus approach: versatile concepts for water resources
education. J. Contemp. Water Res. Educ. 166 (1), 24–34.
Grubert, E., Siders, A., 2016. Benefits and applications of interdisciplinary digital tools for
environmental meta-reviews and analyses. Environ. Res. Lett. 11 (9), 093001.
Guerrero, A.M., Bodin, O., McAllister, R.R.J., Wilson, K.A., 2015. Achieving social-eco-
logical fit through bottom-up collaborative governance: an empirical investigation.
Ecol. Soc. 20 (4). https://eprints.qut.edu.au/124910/.
Gunter, H.M., Hall, D., Mills, C., 2015. Consultants, consultancy and consultocracy in
education policymaking in England. J. Educ. Policy 30 (4), 518–539.
Hamilton, W.L., Clark, K., Leskovec, J., Jurafsky, D., 2016. Inducing domain-specific
sentiment lexicons from unlabeled corpora. Proceedings of the Conference on
Empirical Methods in Natural Language Processing. Conference on Empirical
Methods in Natural Language Processing 2016 (November) 595–605.
Hughes, S., Pincetl, S., 2014. Evaluating collaborative institutions in context: the case of
regional water management in Southern California. Environ. Plann. C Gov. Policy 32
(1), 20–38.
Hui, I., Ulibarri, N., Cain, B.E., 2018. Patterns of participation and representation in a
E. Bell and T.A. Scott
regional water collaboration. Policy Stud. J. https://doi.org/10.1111/psj.12266.
Jupe, R., Funnell, W., 2015. Neoliberalism, consultants and the privatisation of public
policy formulation: the case of Britain’s rail industry. Crit. Perspect. Account. 29
(Supplement C), 65–85.
Kiparsky, M., Milman, A., Owen, D., Fisher, A., 2017. The importance of institutional
design for distributed local-level governance of groundwater: the case of California’s
Sustainable groundwater management act. Water 9 (10), 755.
Koebele, E.A., 2019a. Cross-coalition coordination in collaborative environmental gov-
ernance processes. Policy Stud. J. https://doi.org/10.1111/psj.12306.
Koebele, E.A., 2019b. Policy learning in collaborative environmental governance pro-
cesses. J. Environ. Policy Plan. 21 (3), 242–256.
Koontz, T.M., Newig, J., 2014. From planning to implementation: top-down and bottom-
up approaches for collaborative watershed management. Policy Stud. J. 42 (3),
416–442.
Koontz, T.M., Thomas, C.W., 2006. What do we know and need to know about the en-
vironmental outcomes of collaborative management? Public Adm. Rev. 66 (s1),
111–121.
Koontz, T.M., Carmin, J.A., Steelman, T.A., Thomas, C.W., 2004. Collaborative
Environmental Management: What Roles for Government? RFF Press,
Washington, D.C.
Kosinski, M., Wang, Y., Lakkaraju, H., Leskovec, J., 2016. Mining big data to extract
patterns and predict real-life outcomes. Psychol. Methods 21 (4), 493–506.
Koski, C., Siddiki, S., Sadiq, A.-A., Carboni, J., 2018. Representation in collaborative
governance: a case study of a food policy council. Am. Rev. Public Adm. 48 (4),
359–373.
Leitman, S., 2008. Lessons learned from transboundary management efforts in the apa-
lachicola—chattahoochee—flint Basin, USA. Transboundary Water Resources: A
Foundation for Regional Stability in Central Asia. Springer, Netherlands, pp.
195–208.
Lubell, M., Lippert, L., 2011. Integrated regional water management: a study of colla-
boration or water politics-as-usual in California, USA. Int. Rev. Adm. Sci. 77 (1),
76–100.
Lubell, M., Douglas Henry, A., McCoy, M., 2010. Collaborative institutions in an ecology
of games. Am. J. Pol. Sci. 54 (2), 287–300.
Mikolov, T., Chen, K., Corrado, G., Dean, J., 2013. Efficient Estimation of Word
Representations in Vector Space. arXiv. cs.CL. arXiv. http://arxiv.org/abs/1301.
3781.
MOC Regional Water Planning Council, 2011. Middle Ocmulgee Regional Water Plan.
Molen, F., 2018. How knowledge enables governance: the coproduction of environmental
governance capacity. Environ. Sci. Policy 87 (September), 18–25.
Newig, J., Koontz, T.M., 2014. Multi-level governance, policy implementation and par-
ticipation: the EU’s mandated participatory planning approach to implementing en-
vironmental policy. J. Eur. Public Policy 21 (2), 248–267.
Newig, J., Challies, E., Jager, N.W., Kochskaemper, E., Adzersen, A., 2018. The en-
vironmental performance of participatory and collaborative governance: a frame-
work of causal mechanisms. Policy Stud. J. 46 (2), 269–297.
Nikita, M., 2016. Ldatuning. https://cran.r-project.org/web/packages/ldatuning/
73
Environmental Science and Policy 111 (2020) 63–73
vignettes/topics.html.
Nowlin, M.C., 2016. Modeling issue definitions using quantitative text analysis. Policy
Stud. J. 44 (3), 309–331.
Parker, K.B., Margerum, R.D., Dedrick, D.C., Dedrick, J.P., 2010. Sustaining watershed
collaboratives: the issue of coordinator-board relationships. Soc. Nat. Resour. 23 (5),
469–484.
Purdy, J.M., 2012. A framework for assessing power in collaborative governance pro-
cesses. Public Adm. Rev. 72 (3), 409–417.
Rennie, J., Lang, K., 2008. The 20 Newsgroups Data Set. http://qwone.com/∼jason/
20Newsgroups/. .
Sachdeva, S., Emery, M.R., Hurley, P.T., 2018. Depiction of wild food foraging practices
in the media: impact of the great recession. Soc. Nat. Resour. 31 (8), 977–993.
Sardo, A.M., Weitkamp, E., 2017. Environmental consultants, knowledge brokering and
policy-making: a case study. Int. J. Environ. Policy Decis. Mak. 2 (3), 221–235.
Schneider, M., Scholz, J.T., Lubell, M., Mindruta, D., Edwardsen, M., 2003. Building
consensual institutions: networks and the national estuary program. Am. J. Pol. Sci.
47 (1), 143–158.
Scholz, J.T., Stiftel, B., 2010. Adaptive Governance and Water Conflict: New Institutions
for Collaborative Planning. Routledge, Abingdon, UK.
Scott, T.A., Carter, D.P., 2019. Collaborative governance or private policy making? When
consultants matter more than participation in collaborative environmental planning.
J. Environ. Policy Plan. 21 (2), 153–173.
Scott, T.A., Thomas, C.W., 2017. Unpacking the collaborative toolbox: why and when do
public managers choose collaborative governance strategies? Policy Stud. J. 45 (1),
191–214.
Scott, T.A., Ulibarri, N., Scott, R.P., 2020. Stakeholder involvement in collaborative
regulatory processes: using automated coding to track attendance and actions. Regul.
Gov. 14 (2), 219–237.
Siddiki, S.N., Carboni, J.L., Koski, C., Sadiq, A.-A., 2015. How policy rules shape the
structure and performance of collaborative governance arrangements. Public Adm.
Rev. 75 (4), 536–547.
Thomas, C.W., Koontz, T.M., 2011. Research designs for evaluating the impact of com-
munity-based management on natural resource conservation. J. Nat. Resour. Policy
Res. 3 (2), 97–111.
Ulibarri, N., 2015a. Collaboration in federal hydropower licensing: impacts on process,
outputs, and outcomes. Public Perform. Manag. Rev. 38 (4), 578–606.
Ulibarri, N., 2015b. Tracing process to performance of collaborative governance: a
comparative case study of federal hydropower licensing. Policy Stud. J. 43 (2),
283–308.
USACE, 2016. Engineering and Design: Water Control Management. 1110-1112-240.
https://www.publications.usace.army.mil/Portals/76/Publications/
EngineerRegulations/ER_1110-2-240.pdf.
Vaughan Jr., M., 2001. Multi-year drought in apalachicola-chattahoochee-Flint River
Basin. In: Proceedings of the 2001 Georgia Water Resources Conference. April 26 and
27, 2001, Athens, Georgia., Edited by Kathryn J Hatcher. Georgia Institute of
Technology. . https://smartech.gatech.edu/handle/1853/43562.