Linking Network Structure To Collaborative

Journal of Public Administration Research And Theory, 2017, 163–181
doi:10.1093/jopart/muw041
Article
Advance Access publication June 28, 2016
Article
Linking Network Structure to Collaborative

Governance
Nicola Ulibarri,* Tyler A. Scott†
*University of California; †University of Georgia
Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Abstract
How do social networks differ between highly collaborative and less collaborative forms of gov-
ernance? Drawing on a prior study that characterized the level of collaboration for three federal
hydropower relicensing processes, we develop exponential random graph models of meeting
attendance and participation networks. We find that the highly collaborative relicensing pro-
cess had lower overall density and a propensity for relatively fewer and stronger interactions.
Reciprocity is highest in the high-collaboration process, indicating that it is characterized by
mutual interactions. In the low-collaboration process, patterns of connections between any three
members of the network displayed a more unidirectional structure, suggesting asymmetrical
interactions between active versus passive members of the network. By linking network struc-
ture to collaborative dynamics, this study helps elaborate potential mechanisms of successful
collaboration.
Introduction of collaborative governance’s long-term performance

Public decision-makers and managers increasingly are scarce, it has been shown to positively impact the
use collaborative, networked forms of governance to resources managed (Kelman, Hong, and Turbitt 2013;
address complex public problems. In a collaborative Scott 2016).
process, public agencies work jointly with other public Accompanying collaborative governance’s pro-
and private organizations to develop or implement pol- liferation are calls to better evaluate its performance
icies and programs. The process entails building trust (Emerson and Nabatchi 2015b; Emerson, Nabatchi,
between participants; developing shared understand- and Balogh 2012; Gray 2000; Innes and Booher 1999;
ing of the problem via deliberation and negotiation; Koontz and Thomas 2006; O’Leary and Bingham
and developing the resources, capacity, and leadership 2003; Newig and Fritsch 2009; Thomas and Koontz
to support engagement (Ansell and Gash 2008; Bryson, 2011; Thomson, Perry, and Miller 2008, 2009).
Crosby, and Stone 2006; Bryson, Crosby, and Stone Collaborative governance’s outcomes are theorized
2015; Emerson and Nabatchi 2015a). Through col- to emerge from interpersonal and interorganizational
laborative governance, public agencies can overcome dynamics like principled engagement among partici-
long-standing conflict and build stakeholder trust and pants (Emerson and Nabatchi 2015a). Thus, in order
acceptance of decisions (Emerson et al. 2009; Scott to better understand what collaborative governance
2011). It allows participating organizations to share does, it is important to consider how process features
resources (Berardo 2014) and integrate diverse types of relate to collaborative outputs and outcomes (Ulibarri
information into decisions (Beierle and Cayford 2002; 2015a).
Connick and Innes 2003; Dale and Armitage 2011; However, assessing interorganizational collabora-
Korfmacher and Koontz 2003). Although analyses tion poses numerous analytical challenges. Analyzing
collaboration within policy networks often requires
Address correspondence to the author at nulibarri@gmail.com. intensive data collection and analysis methods such
© The Author 2016. Published by Oxford University Press on behalf of the Journal of Public Administration Research and Theory, Inc. 163
All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.
164 Journal of Public Administration Research and Theory, 2017, Vol. 27, No. 1
as interviews, document analysis, and process tracing networks. Then, we specify the exponential random
(Connick and Innes 2003; Margerum 2011; Ulibarri graph models (ERGMs) used to model networks and
2015b) or survey instruments and statistical modeling test our hypotheses. Finally, we assess our findings and
(Berardo 2014; Berardo and Scholz 2010; Schneider discuss their implications.
et al. 2003; Lubell, Henry, and McCoy 2010; Scott
and Thomas 2015; Thomson, Perry, and Miller 2009;
Ulibarri 2015a). The effort required to collect and ana- Background
lyze these data present a significant research design We examine this question using the case of the Federal
barrier. Perhaps more importantly, these modes of Energy Regulatory Commission’s (FERC) process for
data collection do not readily facilitate cross-case com- licensing hydropower facilities. To obtain a license for
parisons, longitudinal analysis, or replication. Survey continued operation of a hydropower dam, electric
instruments or interview scripts are typically custom- utilities, water districts, and other dam owners/opera-
ized for the network context at hand, for instance, using tors (hereafter referred to as “utilities”) undertake a
a specific type of “name generator” (Henry, Lubell, and 5-year process to identify potential project impacts,

McCoy 2012) to elicit responses concerning network develop and interpret studies to quantify those
partners. Although it is technically feasible to imple- impacts, and develop a license application contain-
ment longitudinal surveys, pragmatic issues related ing operating requirements to mitigate those impacts.
to response burden and survey fatigue loom large. It FERC then assesses these proposed requirements to
is perhaps emblematic that the best known and most determine the final contents of a license. Traditionally,
extensively used longitudinal collaborative environ- the application was developed unilaterally by the util-
mental governance network data are a two-period ity, with feedback from resource agencies and other
sample from 1999 and 2001 (see Berardo and Scholz interested stakeholders occurring after the utility sub-
2010; Scholz, Berardo, and Kile 2008). mitted its license application. Since the late 1990s,
A readily available source of network data for however, a series of process reforms have restructured
policy and management actions is the paper trail of the process to integrate stakeholder feedback from the
meeting minutes and public comments containing earliest scoping phases (Kosnik 2010). At minimum,
objective data on who participants are and how they FERC hydropower relicensing requires a series of pub-
interact. However, the extent to which readily observ- lic meetings to discuss scoping, study development,
able actions like meeting attendance reflect underlying study results, and draft and final license applications.
collaborative dynamics is unclear. This article takes However, many utilities opt to exceed these require-
advantage of a unique opportunity to examine the ments, working closely with federal and state agencies,
relationship between objective participation metrics local governments, tribes, nonprofit organizations, and
and collaborative dynamics. the public throughout the process.
Having already established the underlying level of In Ulibarri (2015b), document analysis and a par-
collaboration in three planning processes using docu- ticipant survey were used to measure the degree to
ment analysis and survey data (see Ulibarri 2015b), which collaboration occurred among stakeholders for
we use statistical network analysis to assess whether a series of recent hydropower relicensing processes.
observed network linkage patterns (e.g., co-attend- Collaboration was conceptualized following Emerson,
ance at group meetings) reflect the occurrence of key Nabatchi, and Balogh’s (2012) “Integrative Framework
collaborative dynamics. Thus, we make a theoretical for Collaborative Governance,” which focuses on three
contribution to the public administration literature dynamics. The first, principled engagement, reflects
by speaking to the relationship between participatory the use of face-to-face deliberation and negotiation to
metrics and collaboration among network organiza- develop shared problem definitions and decisions. The
tions; we also aim to make an empirical contribution second, shared motivation, captures the extent to which
by demonstrating how policy makers and managers participants trust one another and believe the process
might assess the efficacy of a collaborative governance is legitimate. The third, capacity for joint action, meas-
process in real time by monitoring attendance and ures the leadership, structure, and resources necessary
participation. to support collaboration. We developed measures to
In what follows, we first describe the cases selected operationalize these three dynamics, drawing on a sur-
for this analysis and the process by which each case vey of relicensing participants (to measure perceived
was determined to exhibit low, medium, or high col- levels of trust, efficiency, and co-creation) and analy-
laboration. We then describe the literature concern- sis of meeting minutes, public comments, technical
ing policy networks and collaborative governance reports, and other documents developed during the
and develop hypotheses regarding what we should relicensing (to qualitatively and quantitatively assess
observe in low-collaboration versus high-collaboration who participated, how they interacted, and whether
Journal of Public Administration Research and Theory, 2017, Vol. 27, No. 1 165
the broader structure supported collaboration). These particularly interested in what might be expected in
were then used to measure collaboration in eight each case. In other words, what should a network
recently completed relicensing processes across the around hydropower relicensing look like when col-
country. For more information, see Ulibarri (2015b). laboration is high versus when collaboration is low?
In this analysis, we consider three distinct relicens- The policy network literature informs our expectations
ings, shown in Table 1. All three processes took place regarding what structural characteristics should—or
after FERC’s shift to a more collaborative relicensing should not—be prominent in processes with a given
approach. These three cases were selected because level of collaboration, as different network configu-
they represent three distinct levels of collaboration. In rations distinctly affect the ability of a collaborative
Washington (high collaboration), stakeholders were network to solve the problem it was created to solve
engaged via deliberation and negotiation throughout (Berardo 2014; Bodin and Crona 2009).
the relicensing process, such that both large and small
decisions incorporated the full set of interests in the Network Density
relicensing. In Missouri (medium collaboration), the Network density refers to the average value of a ran-

process was designed to be collaborative and inclusive, domly chosen network tie. Simply put, denser networks
with regular meetings, neutral facilitation, and interest- have more ties and/or higher value ties, so the aver-
based negotiation. However, stakeholders did not trust age value of a tie is greater. Bodin and Crona (2009)
one another, so once discussions entailed developing hypothesize that, in general, a greater number of ties
the actual management regime, stakeholders no longer presents greater opportunities for joint actions by
engaged jointly and most large decisions were made increasing communication and fostering the develop-
by subsets of stakeholders. In Georgia (low collabo- ment of norms of reciprocity and trust. This hypothesis
ration), stakeholders were engaged only to the degree reflects the broader notion that trust and social capi-
required by FERC regulation and most decisions were tal among actors facilitate collective action (Axelrod
made unilaterally by the utility. 1997; Ostrom 2000; Pretty and Ward 2001; Putnam
2000); further, several analyses of natural resource
governance networks demonstrate a positive relation-
Hypotheses ship between network density and joint action (Hahn
The levels of collaboration shown in Table 1 stem from et al. 2006; Sandström and Carlsson 2008).
survey instruments and document analysis applied to
H1:
High-collaboration processes will exhibit
each case. What we ask here is to what extent statis-
greater network density than low-collaboration
tical network analysis of process participation tracks
processes.
with these more detailed findings. Statistical network
analysis is a common methodological tool kit applied Our network data (described below) consist of discrete
to institutional collective action and collaborative gov- count data reflecting the total number of interactions
ernance research (Berardo 2014; Berardo and Scholz between two actors within observed relicensing meetings.
2010; Berardo, Heikkila, and Gerlak 2014; Henry Coded network ties are thus counts, rather than a binary
2011; Henry, Lubell, and McCoy 2011; Jasny 2012; metric, allowing us to explore the extent to which interac-
Jasny and Lubell 2015; Lee, Lee, and Feiock 2012; tions are concentrated within a relatively small subset of
Leifeld and Schneider 2012; Lubell et al. 2012; Scott network actors or whether interactions are more evenly
2015; Scott and Thomas 2015). Using network analysis distributed across the network. Specifically, network den-
allows us to characterize interorganizational network sity can be increased in two ways: either by increasing the
structure and to test hypotheses concerning individual number of total ties in the network or by increasing the
network behaviors. value of ties that are already greater than zero.
Because this article seeks to compare network Recall that in essence, network density is expected
structures and observed collaboration level, we are to increase collaboration because each network tie
Table 1. Level of Collaboration by Case
Washington Missouri Georgia
Principled engagement High Medium Low

Shared motivation High Medium-low Low
Capacity for joint action High Medium Medium-low
Overall collaboration High Medium Low
Note: Overall collaboration is aggregated from principled engagement, shared motivation, and capacity for joint action. See Ulibarri (2015b)
for more information.
presents an opportunity for communication and joint redundant—and thus inefficient—in terms of informa-
action (Bodin and Crona 2009). However, two net- tion sharing, network structures that are minimally
works can exhibit similar density statistics and yet be redundant can impede performance when credibility is
structured very differently. Networks in which interac- at a premium (Lazer and Friedman 2007). Reciprocity
tions are highly concentrated among a limited subset can enhance the credibility of actors’ commitment to
of actors, with more isolated actors and fewer paths of one another and thus support collaboration (Berardo
communication, can have high overall density due to and Scholz 2010).
these dominant relationships even if there is a lack of
H3:
Higher collaboration processes will exhibit a
overall connectivity. Accordingly, one might anticipate
stronger tendency for reciprocal ties than low-
that networks characterized by a stronger bimodal dis-
collaboration processes.
tribution of high-value ties and empty ties might not
facilitate joint action.
Network Cohesion: Efficiency, Credibility, and
Specifically, overall density (tie presence and magni-
Inclusiveness
tude) and nonzero density (tie presence) will differ to
Although reciprocated ties might be important for fos-

the extent that a few actors tend to dominate dialogue
and deliberation (which will increase overall density tering joint action, densely concentrated ties within a
but not nonzero tie density) versus more broadly par- network can facilitate or inhibit joint action, depend-
ticipative processes (which will increase overall density ing upon the problem context (Bodin and Crona
and nonzero tie density). Thus, we hypothesize that 2009). Bodin and Prell (2011) define network cohesion
relicensing processes shown to be more collaborative as a relative concept, wherein two networks of identi-
will differ from less collaborative relicensing processes cal density might be more or less cohesive to the extent
in terms of having more observed connections between that ties are more widely distributed versus highly con-
actors (i.e., more ties with a value of at least one centrated. On one hand, low-cohesion networks with
[“nonzero” ties]) but not necessarily in terms of the dense patterns of connectivity are duplicative and thus
average tie value (which is driven by both the number potentially inefficient in terms of gathering and sharing
of nonzero ties and the magnitude of each tie value). information. More widely distributed ties (i.e., a net-
work with greater cohesion) facilitate greater access to
H2:
High-collaboration processes will exhibit a external knowledge and resources, thereby supporting
lesser tendency to have empty ties than low-col- collective action (Bodin and Crona 2009; Newman and
laboration processes. Dale 2007; Sandström and Carlsson 2008), whereas
a lack of network cohesion can inhibit broader col-
Network Reciprocity: Mutual Ties laboration and foster dueling coalitions (Borgatti and
Low-level structural characteristics within a network, Foster 2003).
such as the patterns of ties between network actors, On the other hand, in certain situations, lower
greatly influence network-level outcomes (Provan and network cohesion can buffer decision-making from a
Kenis 2008). For instance, reciprocated ties (i.e., a tie “constant influx of less relevant information” (Bodin
from A → B and from B → A) can serve to make coop- and Crona 2009, 368) and foster informational diver-
eration more feasible by increasing credibility among sity by allowing different knowledge to develop in dif-
actors (Berardo and Scholz 2010). Governance pro- ferent clusters within the network (Bodin and Crona
cesses that involve complex systems—in the case of 2009). Further, smaller groups of close-knit actors
hydropower relicensing, the system includes energy can be more efficient at decision-making (Provan and
production, aquatic ecosystems, and water resources— Kenis 2008).
require a high degree of information exchange among This tension reveals an efficiency-inclusivity trade-
relatively specialized actors (Crona and Bodin 2006). off (Provan and Kenis 2008) between the strategic
Patterns of mutual exchange can strengthen relation- benefits of changing the scale or level of government
ships between these actors (Berardo and Scholz 2010; actions to involve non-state actors on one hand (Newig
Putnam, Leonardi, and Nanetti 1993). and Fritsch 2009) and the time, effort, and costs of
The collaborative governance literature strongly coordination and cooperation on the other (Margerum
emphasizes the need for trust and commitment among 2011). Thus, collaborative governance institutions
stakeholders for collaborative efforts to be success- must navigate trade-offs between involvement (pro-
ful (Margerum 2002, 2011; Wondolleck and Yaffee viding access to resources [Schneider et al. 2003],
2000). Particularly in situations where actors do not increased perceived legitimacy [Bryson, Crosby, and
largely agree on goals, trust and information cred- Stone 2006; Dietz and Stern 2008; Margerum 2011],
ibility are critical for sustaining collective action and diverse knowledge that can improve policy [Beierle and
reducing defection (Berardo 2014; Leach and Sabatier Cayford 2002; Sirianni 2009], and scale advantages
2005). Thus, although reciprocated ties might be [Feiock 2013; Feiock and Scholz 2009; Gerlak, Lubell,
and Heikkila 2012; Karkkainen 2002]) and exclusion search costs are a more significant transaction cost bar-
(facilitating quick action [Imperial 2005; Margerum rier than credibility issues in networks constituted by
2011; Wondolleck and Yaffee 2000], reducing exter- fragmented authority. The case at hand, hydropower
nal decision costs [Feiock 2013], and excluding actors relicensing, exhibits a great deal of fragmented author-
who lack sufficient expertise [Day and Gunton 2003; ity because it involves local, state, and federal actors
Lasker and Weiss 2003] or resources [Wondolleck in different policy subfields of energy, fish and wildlife,
and Yaffee 2003] to contribute meaningfully). These water resources, and land use. In light of such frag-
dueling forces place a premium on collaborating with mentation, we anticipate that network effectiveness
the right people and advantage cases where there are (in terms of whether the relicensing network is able to
fewer “wrong” people involved. successfully collaborate and produce quality outputs)
One way that networks exhibit tendency toward more requires greater overall connectivity and network-wide
or less cohesiveness is through transitivity. Transitivity cohesion (as opposed to more redundant structures
reflects the “a friend of a friend is my friend” axiom, in that maximize credibility and commitment).
that networks with high transitivity are those where if
H4:
High-collaboration processes will exhibit lower

actor A is connected to actors B and C, then actors B and
triad closure bias than low-collaboration processes.
C are more likely to be connected to one another as well
(Kilduff and Tsai 2003; Lubell et al. 2012). This ten-
dency for transitive structures to arise is known as triad Data
closure bias, because it means that network two paths As described above, instead of measuring collaborative
(e.g., k → i → j on left side of Figure 1) tend to close into networks using respondent-generated data, this article
triangles (right side of Figure 1). Thus, high transitivity tests the linkages between observed participation and
is reflected in dense, interconnected network patterns collaborative dynamics. We code the relevant network
(Berardo and Scholz 2010) that reinforce direct ties in each case to be the full set of individuals who are
between participants (Desmarais and Cranmer 2012a). recorded as having attending at least one meeting at
In contrast, a network of similar density but with lower some point in the relicensing process. Ties between
triad closure bias is characterized by more widely dis- individuals are then coded based upon recorded com-
persed connections and greater overall cohesion (e.g., ments recovered from meeting minutes.
if triad closure bias is lower, actor k in Figure 1 is rela- Records of meeting attendance and meeting minutes
tively more likely to choose actor m over j). for the three cases were obtained from http://elibrary.
Berardo and Scholz (2010) discuss how the particu- ferc.gov. All meetings were catalogued by purpose,
lar attributes of a collective action dilemma can shift location, and date. Because two of the cases held many
the optimal balance between dense, more redundant meetings, a subsample of meetings was selected for
network structures and more dispersed, cohesive tie further analysis. For Washington, every 10th meeting
patterns.1 Scholz, Berardo, and Kile (2008) show that chronologically was selected (resulting in 34 of 336
total meetings). For Missouri, approximately half of the
meetings had minutes available, and every other meet-
ing of those with minutes was selected (for 32 of 138
meetings). For Georgia, which held a comparatively
small number of meetings, every meeting with minutes
was included in the sample (for 9 of 11 meetings).
For sampled meetings, all attendees were cata-
logued, along with their organization and organiza-
tion type (utility, federal agency, state agency, local
government, NGOs, business, tribe, consultant/lawyer,
facilitator, or individual). To identify how attendees
participated, meeting minutes were analyzed using
QSR NVivo 10 qualitative analysis software, catalogu-
Figure 1. Two-paths and transitive triad network structures.
ing every instance where an individual (1) made a pres-
1 Berardo and Scholz (2010) specifically demonstrate that actors are entation or (2) participated in a discussion.
more likely to choose “bonding” structures that maximize credibility and Emerson and Nabatchi (2015a) characterize “prin-
minimize defection in higher risk cooperation dilemmas and more likely cipled engagement” within collaborative governance
to choose “bridging” structures that maximize information transmission
as an interactive process of discovery and deliberation.
and network cohesion in lower risk coordination dilemmas. Although
our analysis does not specifically use the bridging and bonding capital The extent to which principled engagement occurs
framework, the Berardo and Scholz “Risk Hypothesis” nonetheless is a key collaborative dynamic that drives benefi-
demonstrates the importance of problem context. cial outcomes (Ulibarri 2015a, 2015b). The meeting
minutes we code provide direct evidence of dialogue does potentially affect our analysis. ERGMs rest on
and deliberation among stakeholders, demonstrating the assumption that all ties are observed; by sampling,
which participants present or discuss key issues. If a we knowingly omit some ties. However, these omitted
given participant gives a presentation or substantively ties are missing randomly (given our sampling proto-
engages in discussion at a given meeting, we code a col) and therefore should represent the full network
tie value of 1 from the discussant to all other meeting we would observe were we to code all ties. Moreover,
attendees.2 If a meeting attendee does not participate the unsampled meetings are not substantively differ-
in discussion, ties from said attendee to other meet- ent from those included in the sample—whether by
ing attendees are given a value of 0. Because of the topic, location, or timing over the course of the reli-
one-way nature of this interaction (one individual is censing—so we have no reason to assume that a sub-
talking to—not necessarily with—another individual) stantially different pattern of attendance or dialogue
these are directed ties that originate at the discussant took place at those meetings. Finally, unsampled meet-
and terminate with audience members. Thus, in each ings—and resultant missing edges—are less problem-
network, a tie from one individual to another is not atic in this cross-sectional analysis since we are simply

necessarily reciprocated (i.e., A → B does not equal summing observed interactions across observed meet-
B → A).3 ings. In doing so, we are not making an assumption
Since deliberation among participants can occur at about what did or did not happen at other meetings
more than one meeting, tie values between individu- (as we would be in a longitudinal framework where
als are not limited to 0 and 1. After coding meeting- ties might be assumed to form or dissolve over time),
specific ties, we then sum the meeting-specific tie values but rather simply assessing overall patterns of dia-
across all meetings. For instance, the tie value between logue across observed meetings.
individuals A and B (A → B) is equal to the number Two additional notes are merited about potential
of times that individual A has engaged in substantive limitations underlying our data. First, given the dif-
discussion at a meeting in which individual B was in ferent number of meetings in each case, we might
attendance. These tie values summarize the extent to expect to see differences in network structure that
which each organization had the opportunity to learn stem purely from differing opportunities to engage.
about the perspectives and goals of other network With fewer meetings, we would expect to have larger
organizations within the collaborative relicensing pro- attendance; with more meetings, people might be
cess. Table 2 summarizes descriptive network statistics more selective about when they attend, potentially
for each relicensing. increasing the prevalence of zero-valued ties in
Although we sampled the meetings to make the our sample. However, these differences should not
research endeavor feasible (as coding hundreds of affect our other network measures. A tie represents
meeting would take a substantial amount of time), it a unidirectional flow of engagement, not simply co-
attendance, so the overall statistics should capture
differences in the internal dynamics of each meeting
Table 2. Network Summary Statistics regardless of who shows up. Second, the extent to
No of Actors Total Edges which the meeting minutes captured every partici-
pant action varied across the three cases. The Georgia
Washington 146 1,241 minutes provided word-for-word transcriptions, and
Missouri 157 5,119 Missouri provided thorough summaries with a name
Georgia 84 1,297
attached to every idea. Washington had some simi-
larly detailed minutes, but other minutes (particularly
2 We do not assume that a positive tie value implies collaboration during settlement negotiations) instead tracked only
between the two parties, as participants might be voicing disagreement the organization or mentioned a topic that was dis-
with one another. Rather, a positive tie value simply represents an act
of engagement.
cussed without attribution. Because not every partici-
3 An alternative for examining co-participation would be to just use pant action was captured in the Washington minutes,
meeting attendance, as presumably there are verbal (and nonverbal) they likely undercount tie values relative to actual lev-
interactions that occur at meetings (e.g., hallway conversations) els of engagement.
that contribute to network function. However, this poses a modeling
challenge in that if attendance were used as a basis for coding
inter-actor ties, then every attendee at a given meeting would be
coded as having a tie to every other attendee. This makes estimation Modeling Networks
problematic, since the network then consists of complete subgroups
Since this research concerns collaboration among
(in which all possible ties exist) and empty subgroups in which no ties
exist. Going forward, we hope to generate data that can be used to network actors, we use a statistical network analy-
examine participation in more detail, but using meeting minutes to code sis to analyze links between individual actors as the
deliberation is an important step in this direction. dependent variable. In a network, the presence of one
network tie influences the presence of others, thus 2015). This parameter reflects the expected value
violating the standard independence assumptions of of a tie between I and j based upon the value of all
most regression models (Robins, Lewis, and Wang ties observed in the network. Network density is
2012). We use ERGMs that explicitly model interde- computed as:
pendence among network ties (Lubell et al. 2012) by
modeling every tie conditionally based upon all other g (y ) = ∑ y , (1)ij
observed ties (Lusher, Koskinen, and Robins 2013). (ij )∈Y
Appendix A discusses the mechanics of ERGMs in

detail. where yij is the value of an observed tie from individ-
Although ERGMs are commonly applied to binary ual i to individual j and Y is the set of all network
networks where a tie is simply present or absent, recent members. The sum term acts as an intercept, in that it
methodological advances enable the use of generalized reflects the expected value of a randomly selected tie
ERGMs (GERGMs) that model valued network ties across all actors. Given that we code tie values based
(Desmarais and Cranmer 2012b; Krivitsky 2012). As upon observed deliberation at meetings, it is important

described previously, we represent each tie as the sum to consider that average tie value is potentially subject
of all unidirectional engagement across meetings from not only to what happens at meetings but the num-
one participant to another. As these summary values ber of meetings that are observed. Network density
are count data, a Poisson-reference GERGM is used increases as ties take on higher values (which can only
to model the overall network (see Krivitsky, 2012; happen as more meetings are observed) and decreases
Krivitsky and Butts, 2013, for a detailed explanation as more participants are observed (because density is a
of the Poisson GERGM specification). ratio of total tie values to number potential actor pairs,
GERGMs facilitate inference on a network of or dyads). Thus, in order to test H1 using a consistent
interest by comparing the observed network to a dis- basis of comparison across the three cases, we control
tribution of simulated networks that have similar char- for the number of meetings each actor attends in esti-
acteristics. Appendix A provides a fuller discussion of mating the sum parameter. This serves to produce den-
this process (and presents model goodness-of-fit analy- sity estimates that differ from the raw ratio of the sum
sis). Essentially, we model the observed network as a of all observed ties over total dyads (because the sum
function of endogenous structural characteristics and parameter estimate is the predicted value of a tie from
exogenous attributes of participating organizations actor i to actor j controlling for the number of meet-
and then compare the estimated parameters to a dis- ings that actor i attended) and accounts for sampling
tribution of parameters estimated from a set of simu- differences so as to enable more accurate comparison
lated networks weighted according to similarity to the of network density. We thus test Hypothesis 1 by com-
observed network. Empirical parameters that fall in paring the sign and magnitude of the sum parameters
the extreme of either tail of the simulated parameters in GERGM models that also include number of meet-
distribution are considered “significant,” since it is ings attended as a covariate.
unlikely that a given network structure occurred sim- Hypothesis 2 asserts that high- and low-collabora-
ply due to random variation. tion networks will be of similar overall density but
An example of a structural term that can be fit that high-collaboration networks will have fewer non-
in a GERGM is a two-path, wherein two actors existent ties. To test Hypothesis 2, we add a nonzero
(Organization A and Organization C) are linked via a term that reflects the total number of ties greater than
path of ties, first from Organization A to Organization zero observed in the network (i.e., total number of
B (A → B) and second from Organization B to edges):
Organization C (B → C). Of course, a primary chal-
lenge is linking these types of structural terms to the g (y ) = ∑ I (y ij )
≠ 0 (2)
theoretical hypotheses outlined above. We draw upon (ij ) ∈Y
several recent analyses linking theoretical network
concepts with network measures (Bodin and Crona to the baseline models used to test Hypothesis
2009; Borgatti 2005; Burt 2005; Henry and Vollan 1. Having controlled for overall network density, the
2014; Scholz, Berardo, and Kile 2008) and recent sign and magnitude of the nonzero term thus reflects
ERGM scholarship (Desmarais and Cranmer 2012a, the tendency for a network to have more ties between
2012b; Krivitsky 2012; Lusher, Koskinen, and Robins actors (independent of magnitude). The estimated
2013) to link the four hypotheses developed above nonzero coefficient represents the additive log change
with expected model results. in the expected value of a tie between nodes i and j,
To represent network density (H1), we use the sum given that the number of nonzero ties held by node i
parameter in the R ergm package (Handcock et al. increases by one.
To compare reciprocity and transitivity across the intuition here is that in networks with high transitivity,
three relicensing networks, we fit a third series of mod- the final value for this term should be higher on aver-
els that include terms for mutual ties and transitive tri- age since more triangles tend to be “closed.”
ads. The mutual term (Handcock et al. 2015) models Table 3 summarizes how each term is computed.
the prevalence of reciprocated ties between nodes for
nonbinary network ties:
Results and Analysis
g (y ) = ∑ min (y , y ) (3)
ij ji
Although basic network density can be computed
(ij ) ∈Y
algebraically simply by dividing the number of total
by recording the minimum observed tie value between possible ties by the total summed value of all ties to
each pair of actors. This reflects how the observed tie compute average tie value, as noted above, it is impor-
value from node i to node j influences the expected tant to account for the number of meetings that a
value of a tie from node j to node i. If network reci- given actor attends. We thus estimate the sum param-
procity is high, then this statistic will be higher because eter (described above) for each case within a GERGM

yij and y ji show a tendency to increase in concert and model that includes a term for the number of observed
meetings that a given actor has attended. These results
(
thus raise min yij , y ji . ) are shown in Table 4.
Transitivity is modeled using the transitiveweights In each model, the sum parameter functions in a
parameter in the ergm packages (Handcock et al. similar fashion to the intercept in a regression model
2015), which models triad closure bias for nonbinary by providing a baseline tie value estimate before factor-
network ties. The formula: ing in other parameters. Parameters estimated for the
Poisson-reference GERGM models shown in Table 4
g (y ) = ∑ min (y , max ij k ∈N (min (y ik , ykj ))) (4) (and below in Tables 5 and 6) can be interpreted as hav-
ing an additive effect on the natural log of the expected
(ij ) ∈Y
value of a tie between any two network organizations.
is based upon the generalization of transitivity terms That is, we can exponentiate a parameter (exp (β )) to
used in binary ERGMs for valued tie data provided identify its multiplicative relationship to expected tie
by Krivitsky (2012). Essentially, this term first identi- value. Although we provide more detailed interpre-
fies the minimum value along each two-path by which tations in the remainder of this section, two general
nodes i and j are connected; that is, if A → B = 2 and B heuristics hold: negative coefficients serve to reduce the
→ C = 3, then min(AB, BC) = 2 . Next, it identifies the expected value of a tie (affected by the network struc-
maximum value out of all observed two-paths between ture represented by the coefficient), whereas positive
nodes i and j (e.g., A → B → C). Finally, it identifies coefficients increase the expected tie value, and coef-
the minimum value between the former result and ficients that are larger in magnitude (either positive or
the actual tie value from node i to node j. The basic negative) evidence a larger marginal effect.
Table 3. Relating Hypotheses to Terms in ERGM Package
Network Concept Term in ERGM Packages Term Structure
H1 Network density sum g (y ) = ∑y ij

(ij ) ∈Y
H2 Nonzero tie density nonzero g (y ) = ∑ I (y ij ≠0 )
(ij ) ∈Y
H3 Reciprocity mutual g (y ) = ∑ min (y , y ) ij ji

(ij ) ∈Y
H4 Transitivity transitiveweights g (y ) = ∑ min (y , max (y ij k∈N ik , ykj ))
(ij ) ∈Y
Table 4. Baseline Models with Density Parameter and Control for Attendance
Case (Collaboration Level) Washington (High) Missouri (Medium) Georgia (Low)
Sum −3.62*** (0.04) −3.18*** (0.03) −3.03*** (0.05)

Meetings attended 0.12*** (0.002) 0.19*** (0.001) 0.28*** (0.01)
***p < .01, **p < .05, *p < .1.

Table 5. Density and Nonzero Ties with Control for Attendance
Sum −1.92*** (0.08) −2.05*** (0.13) −2.09*** (0.22)

Nonzero −1.67*** (0.09) −1.09*** (0.04) −0.85*** (0.06)
***p < .01, **p < .05, *p < .1.
Table 6. ERGM Parameter Estimates for Three Relicensing Networks
Sum −2.99*** (0.08) −3.72*** (0.13) −3.92*** (0.21)

Nonzero −2.44*** (0.09) −1.23*** (0.04) −1.27*** (0.06)

Mutuality 1.72*** (0.09) 0.11*** (0.04) 0.76*** (0.07)
Transitive triads 1.91* (0.03) 1.84*** (0.12) 2.41*** (0.21)
***p < .01, **p < .05, *p < .1.
For instance, by exponentiating the sum param-

eter (exp ( −3.62) = 0.03) , we find that the expected
tie value between any two randomly selected indi-
viduals in the Washington relicensing network is 0.03.
For the Georgia case, the baseline expected tie value
between any two individuals is only slightly greater,
0.04 (exp ( −3.18) = 0.04) . Missouri exhibits a slightly
higher baseline density, with an expected tie value
prior to factoring in other parameters approximately
0.05 (exp ( −3.03) = 0.05) .
Turning to Hypothesis 2, concerning the extent to
which network interactions are concentrated among a
few individuals or more widely dispersed, we can first Figure 2. Proportion of all possible ties observed at a given value
visualize the relative proportions of zero-valued and for each network.
nonzero ties across each network. Figure 2 shows how
each network compares in terms of the proportion of would typically be expected by the Poisson reference
all possible ties ( N *[ N − 1]) for that network that are distribution. In other words, a negative coefficient for
of a given value. As shown in Figure 2, the Georgia net- the nonzero parameter pulls down the baseline tie value
work appears to have a reduced prevalence for nonzero expectation to account for a higher prevalence of ties
ties relative to the other networks, followed by the with a value of zero. For the three models in Table 5, we
Missouri network and then the Washington network. observe that all three networks have a negative value for
To quantify this prevalence, Table 5 presents a model the nonzero coefficient, indicating that all three cases
for each case that incorporates the nonzero statistic have a relatively strong prevalence for zero-valued ties.
that accounts for the extent to which each network However, the Washington case has the largest nega-
has an excess of dyads with a value of zero (i.e., no tive value, showing that of the three relicensing pro-
tie) relative to the Poisson reference distribution, net cesses, it has the lowest nonzero tie density (i.e., more
of the other model terms. The intuition for this term is zero value ties). The parameter value of −1.67 is inter-
that a network can exhibit bimodality such that many preted as reducing the expected value of a tie between
pairs of nodes have no tie (i.e., tie value equals zero) two randomly selected individuals in the network by
whereas other node pairs have higher value ties; in this 81% (exp ( −1.67 ) = 0.19) . The Georgia and Missouri
case, simply fitting a sum parameter (which averages relicensing networks are again very similar for this
tie value across all potential ties) does not accurately parameter: for Georgia the nonzero parameter reduces
represent what we observe in the network. the expected value of a tie by 66% (exp ( −1.09) = 0.34) ,
A negative, large magnitude coefficient indicates whereas for Missouri the nonzero parameter reduces the
that there are more dyads with no observed tie than expected value of a tie by 57% (exp ( −0.85) = 0.43) .
Lastly, we turn to Table 6, which presents the results are based upon meeting attendance and participation,
for network models that include all terms of interest. one would expect that meeting attendance is positively
The high-collaboration case (Washington) exhibits by associated with tie values independent of any under-
far the highest degree of mutuality between network lying network drivers. As expected, each meeting an
participants. As described above, the mutuality param- actor attends significantly increases the expected tie
eter models the extent to which the tie value observed value between that actor and every other network
between nodes A and B (yAB) corresponds to the tie participant, although the magnitude of this increase
value from B to A (yBA). In other words, it models the is small: 5% (exp (0.02) = 1.05) for Washington,
reciprocal tendency of a network. In the Washington 16% (exp (0.15) = 1.16) for Georgia, and 15%
network, for each one-unit increase in yij the predicted (exp (0.14) = 1.15) for Missouri. The magnitude this
value of yji increases by 458% (exp (1.72) = 5.58) . parameter is small in each model partly because it
This might at first seem like an outlandish increase, models the general increase in expected tie value to any
but bear in mind that the baseline expected tie value network member; thus, although meeting attendance
between any two randomly selected nodes is very, very should increase the expected value of a network tie,

low; that is, a 458% increase from 0.1, for instance, is it does necessarily provide fine-tuned guidance about
still only 0.46. More importantly, the large magnitude specific ties.
of this predicted increase reflects the informational
gain for predicting yji that is provided by knowing the
Discussion
value of yij. If we know that yij > 0, then it makes sense
that our expectation for yji is markedly different than Our first hypothesis (Table 7) was that high-collabo-
the baseline value because nodes i and j are not just ration processes would exhibit greater participation
any two randomly selected nodes but rather nodes network density than low-collaboration processes.
that already have some form of relationship. In any However, in the restricted ERGM model containing
case, the tendency for reciprocated ties is lessened in only terms for density and meetings attended (Table 4),
the Georgia case, as a one-unit increase in yij predicts the highest collaboration case (Washington) is shown to
a 114% increase (exp (0.76) = 2.14) in the value of have the lowest density. At first glance, this runs counter
yji. The mutuality term is even weaker in the Missouri to the expectation of Hypothesis 1 that high-collabora-
case, as a one-unit increase in yij predicts just a 12% tion processes will be evidenced by stronger network
increase in tie value (exp (0.11) = 1.12) . density relative to low-collaboration processes.
The transitive triads parameter (the transitive- However, Table 5 (as well as Figure 2) indicates that
weights statistic in the ergm package, Handcock et al. there is more to this story, as the Washington network
2015) can be interpreted as predicting how a one-unit also exhibits the strongest proportion of zero-valued
increase in the strongest two-path through which node ties, which serves to pull down the overall density
i already reaches node j (e.g., A → B → C) increases the estimate. Hypothesis 2 holds that high-collaboration
expected value of a direct tie between nodes i and j (yij, processes will exhibit greater nonzero tie density
e.g., A → C). This statistic is smallest for the medium- than low-collaboration process but will not differ in
collaboration Missouri case and largest for the low-col- terms of average tie value. This expectation is also not
laboration Georgia case. For the Georgia relicensing, a borne out in our results, as the high-collaboration pro-
one-unit increase in the strongest two-path between two cess (Washington) has the lowest nonzero tie density
( ( ))
nodes—measured as maxk ∈ N min yik , ykj —increases (i.e., a higher proportion of dyads with no tie),
expected value of a tie between nodes i and j (yij) by
1013% (exp (2.41) = 11.13) . Although we return to Table 7. Hypothesized Differences Between High- and
this finding in the context of our hypotheses below, Low-Collaboration Processes
clearly the Georgia case exhibits a very strong triadic clo- H1: High-collaboration relicensing processes will exhibit
sure tendency. In the Missouri relicensing, triad closure greater network density than low-collaboration
strength is smaller than Georgia, but still highly signifi- processes
cant, with a one-unit increase in the strongest two-path H2: High-collaboration relicensing processes will exhibit
between two nodes increasing the expected value of a tie a lesser tendency to have empty ties than low-
between nodes i and j (yij) by 530% (exp (1.84) = 6.30) collaboration processes
. Finally, in the high-collaboration Washington case, a H3: Higher collaboration relicensing processes will exhibit
one-unit increase in the strongest two path between two a stronger tendency for reciprocal ties than low-
nodes predicts a 575% increase in the value of a direct collaboration processes.
tie between said nodes (exp (1.91) = 6.75) . H4: High-collaboration relicensing processes will
exhibit lower triad closure bias than low-
Each model also includes a control for the number
collaboration processes
of meetings each individual attended. Since tie values
followed by medium-collaboration Missouri, with the greatest degree of mutuality by far (although mutual-
low-collaboration Georgia having the lowest density ity is also higher in low-collaboration Georgia than
of nonzero ties. In summary, the case with highest col- in medium-collaboration Missouri). Even though
laboration exhibits the lowest overall participation relicensing process participants come from many dif-
density and has many more isolated actors. ferent organizations, the strength of reciprocal ties
What might account for the unexpected result in the Washington case evidences that many partici-
that the most collaborative network is also the most pants engaged in repeated, two-way dialogue with one
exclusive of the three nominally collaborative relicens- another. Such interactions can help create strong con-
ing processes? Bodin and Crona (2009) hypothesize nections that foster shared understanding and enable
that because network interactions represent potential joint action (Emerson and Nabatchi 2015a).
opportunities for developing common ground and ini- The final structural attribute, transitivity, is shown
tiating joint action (i.e., dialogue and discourse create to be strongest in the network with the lowest observed
more opportunities for collaboration to occur), net- collaboration (Georgia). This supports Hypothesis 4’s
works with more ties should be the most collabora- expectation that high-collaboration processes would

tive. However, others suggest that the most successful exhibit lower triad closure bias. As to why this occurred,
collaborative ventures (in terms of reaching agreement recall that we measure a tie as a unidirectional link
and producing high-quality collective outputs) are between a meeting attendee who gave a presentation
instances where a small (relative to the overall net- or participated in a discussion to all other attendees.
work) group of actors collaborate with one another Thus, triadic structures such as that shown in Figure 1
to leverage resources and influence policy making reveal patterns of interaction in which some partici-
(Lubell 2004a; Ansell and Gash 2008). As discussed pants tend to dominate deliberation, whereas others
above, inclusion of a greater number of network actors are passive listeners. The relative prevalence of transi-
increases the time and effort required to reach deci- tive triads in the Georgia case indicates that dialogue
sions and take action (Imperial 2005; Margerum 2011; was more unidirectional. This stands in contrast to the
Wondolleck and Yaffee 2003) and raises external high degree of mutual dialogue in the Washington case,
decision costs (Feiock 2013). Thus, high-functioning which perhaps evidences a more robust discussion
collaborative decision-making forums might not be among participants since attendees who are speaking
the most inclusive in terms of incorporating all net- are also receiving information from others.
work actors into decision-making but rather institu- A final observation is on the similarity in network
tions where a subset of key network actors are heavily structure between the low-collaboration case and the
involved (i.e., stakeholders who offer particularly rele- medium-collaboration case, despite seemingly differ-
vant information, hold key resources, or are otherwise ent levels of interaction arising from the initial docu-
critical for achieving successful outcomes). ment analysis. In low-collaboration Georgia, there was
Put in terms of our empirical findings, it is possible very little interaction between participants relative to
that networks having slightly lesser overall density, and the other two cases. The utility held only 11 public
a greater proportion of dyads having no tie (as observed meetings. Although a diverse network of organiza-
in the Washington case), evidence this phenomenon at tions attended meetings, formal presentations by the
work, revealing relicensing networks that are sufficiently utility company and a few other stakeholders repre-
collaborative to leverage resources and get things done sent the preponderance of dialogue. Thus, there was
but not so much that they invite disagreement or delay.4 very little authentic deliberation (Innes and Booher
This does not mean that density necessarily hinders col- 2010) between participants. In contrast, the medium-
laboration but rather indicates that network density is collaboration Missouri process looked a lot more like
insufficient to distinguish among processes or decision collaborative engagement. The utility hired a well-
forums that are all nominally “collaborative.” known facilitator to design the process, and partici-
This interpretation is further supported by the pants used deliberation and interest-based negotiation
results of the mutuality parameter used to test to jointly define problems and develop technical stud-
Hypothesis 3 (that high-collaboration processes will ies. However, when the participants began to develop
exhibit a stronger tendency for reciprocal ties). Indeed, actual management recommendations, participants
across the three cases, the Washington network has the reverted to positional negotiation and the final rec-
ommendations reflected only a subset of stakeholder
4 A less theoretically interesting possibility is that the unexpected interests (see Ulibarri 2015b, for more details on each
lack of ties between many of the participants in high-collaboration
process). Thus, the paper trails for these two processes
Washington reflects the differences in how closely the meeting
minutes identified each speaker across the three cases. Washington look very different, with one case having minimal con-
had the least detailed minutes of the three cases, so the observed data sultation and the other with more extensive engage-
likely undercount tie values relative to the other cases. ment. These differences were not picked up by the
network analysis; the network results instead more for a large number of cases would require substantial
closely reflect the overall level of collaboration (which manpower, using a larger sample size would allow for
includes very low levels of trust from the Missouri trends to be ascertained with more statistical certainty.
survey, drawing the final ranking downward). This Second, examining the evolution of collaborative gov-
suggests that the network analysis was effective in cap- ernance networks over time is a critical way to build
turing the overall level of collaboration but may not upon current work. Neither collaboration nor net-
mirror individual dynamics as effectively. work structures are static phenomena but vary over
the course of a decision-making process. With frequent
meetings occurring over multiple years, participants
Conclusion have the opportunity to get to know one another very
In this study, we asked whether network structure well, learn to cooperate, and possibly even adopt one
metrics reflect the quality and extent of collaborative another’s beliefs (Leach et al. 2013; Lubell 2004b). The
decision-making. Using meeting attendance and partici- network and associated collaborative actions might
pation records from three FERC hydropower relicens- start by building connections across relatively dispa-

ing processes, we tested metrics for network density and rate actors but evolve toward a closely bonded group.
network cohesion and then related them to the under- Additionally, recent scholarship speaks to how the
lying level of collaboration in each case. Drawing on plethora of decision forums within modern policy sub-
meeting minutes allows for a novel method of coding systems requires network actors to allocate their time
one-way network ties between presenters or discus- toward forums where they are best able to exert their
sants and audience members. This is a way to advance influence and garner benefits (Lubell 2013; Lubell,
beyond the commonly used affiliation method by focus- Henry, and McCoy 2010; Lubell, Robins, and Wang
ing on the more certain interactions between audience 2014). The complexity incentivizes affiliation with
members and those who present at meetings rather than forums that contain like-minded actors (Gerber, Henry,
assuming interactions among all of those in attendance. and Lubell 2013; Henry, Lubell, and McCoy 2011).
Our findings serve to demonstrate how patterns of Given this, it is possible that peripheral actors will
individual-level interaction relate to the level of col- drop out of collaborative decision forums by virtue of
laboration on a more general basis. Specifically, in their peripheral status, increasing network concentra-
the high-collaboration case, we observe more concen- tion not through increased interaction but by reducing
trated interactions, more two-way communication, the number of participants.
and a lesser tendency for a few actors to dominate con- Thus, adding a temporal component, both in assess-
versation, whereas in the low-collaboration case, we ing how collaboration changes over time (e.g. via pro-
observe higher overall involvement but less two-way cess tracing) and how the networks change over time
communication. These results speak to the complexity (e.g. via temporal ERGMs), offers to deepen under-
of collaborative governance in policy networks in that standing of how networks and collaboration interre-
inclusion and widespread involvement (often touted late. In particular, if they evolve at different rates, it
as theoretical advantages of collaborative governance) may suggest that one is driving changes in the other,
must be balanced against time and capacity constraints an important finding for process designers seeking to
and the need to reach agreements. encourage effective collaboration. Recent theoretical
One intent of this analysis was to provide network- (Emerson and Nabatchi 2015a) and methodological
based metrics of collaborative dynamics that can be (Ingold and Leifeld 2016; Leifeld et al. 2015) contribu-
scaled up or used to compare across cases more read- tions provide a road map for longitudinal analysis of
ily than the more qualitative or case-specific measures collaborative governance and networks going forward.
in the literature. In our previous analysis, principled
engagement was found to be the dynamic that most Funding
strongly influenced the content and quality of hydro- This work was supported by the Bill Lane Center for
power licenses (Ulibarri 2015a, 2015b); connecting the American West and the Emmett Interdisciplinary
these findings to network metrics reveals that triadic Program in Environment and Resources at Stanford
closure bias and mutuality potentially represent useful University.
network-based measures of principled engagement for
analyzing collaboration and its outcomes.
References
This study was subject to methodological limita-
Ansell, Chris, and Alison Gash. 2008. Collaborative governance in theory
tions that provide fruitful avenues for future research.
and practice. Journal of Public Administration Research and Theory
First, as an exploratory study, we assessed only three 18:543–71.
cases by identifying trends among our data. Although Axelrod, Robert M. 1997. The complexity of cooperation: Agent-based mod-
measuring collaboration and collecting network ties els of competition and collaboration. Princeton, NJ: Princeton Univ. Press.
Beierle, Thomas C., and J. Jerry Cayford. 2002. Democracy in practice: Public Gerber, Elisabeth R., Adam D. Henry, and Mark Lubell. 2013. Political homo-
participation in environmental decisions. Washington, DC: Resources for phily and collaboration in regional planning networks. American Journal
the Future. . of Political Science 57:598–610.
Berardo, Ramiro. 2014. Bridging and bonding capital in two-mode collabora- Gerlak, Andrea K., Mark Lubell, and Tanya Heikkila. 2012. The promise
tion networks. Policy Studies Journal 42:197–225. and performance of collaborative governance. In Oxford Handbook of
Berardo, Ramiro, Tanya Heikkila, and Andrea K. Gerlak. 2014. US Environmental Policy, ed. Sheldon Kamieniecki and Michael E. Kraft,
Interorganizational engagement in collaborative environmental manage- 413–34. Oxford, UK: Oxford Univ. Press.
ment: Evidence from the South Florida Ecosystem Restoration Task Force. Gray, Barbara. 2000. Assessing inter-organizational collaboration: Multiple
Journal of Public Administration Research and Theory 24:697–719. conceptions and multiple methods. In Cooperative strategy: Economic,
Berardo, Ramiro, and John T. Scholz. 2010. Self-organizing policy networks: business, and organizational issues, ed. David Faulkner and Mark De
Risk, partner selection, and cooperation in estuaries. American Journal of Rond, 7–32. New York, NY: Oxford Univ. Press.
Political Science 54:632–49. Hahn, Thomas, Per Olsson, Carl Folke, and Kristin Johansson. 2006. Trust-
Bodin, Orjan, and Beatrice Crona. 2009. The role of social networks in natural building, knowledge generation and organizational innovations: The role
resource governance: What relational patterns make a difference? Global of a bridging organization for adaptive comanagement of a wetland land-
Environmental Change: Human and Policy Dimensions 19:366–74. scape around Kristianstad, Sweden. Human Ecology 34:573–92.
Bodin, Örjan, and Christina Prell, eds. 2011. Social networks and natural Handcock, Mark S., David R. Hunter, Carter T. Butts, Steven M. Goodreau,
resource management: Uncovering the social fabric of environmental gov- Pavel N. Krivitsky, and Martina Morris. 2015. statnet: Software tools

ernance. Cambridge, UK: Cambridge Univ. Press. for the statistical analysis of network data. The Statnet Project. http://
Borgatti, Stephen P. 2005. Centrality and Network Flow. Social Networks www.statnet.org. R package version 2015.11.0, CRAN.R-project.org/
27:55–71. package=statnet.
Borgatti, Stephen P., and Pacey C. Foster. 2003. The network paradigm in Handcock, Mark S., David R. Hunter, Carter T. Butts, Steven M. Goodreau,
organizational research: A review and typology. Journal of Management and Martina Morris. 2008. statnet: Software tools for the representa-
29:991–1013. tion, visualization, analysis and simulation of network data. Journal of
Bryson, John M., Barbara C. Crosby, and Melissa Middleton Stone. 2006. The Statistical Software, 24:1–11.
design and implementation of cross-sector collaborations: Propositions Henry, Adam D. 2011. Ideology, power, and the structure of policy networks.
from the literature. Public Administration Review 66:44–55. Policy Studies Journal 39:361–83.
———. 2015. Designing and implementing cross-sector collaborations: Henry, Adam D., Mark Lubell, and Michael McCoy. 2011. Belief systems and
Needed and challenging. Public Administration Review 75:647–63. social capital as drivers of policy network structure: The case of California
Burt, Ronald S. 2005. Brokerage and closure: An introduction to social capital. regional planning. Journal of Public Administration Research and Theory
Oxford: Oxford Univ. Press. 21:419–44.
Connick, Sarah, and Judith E. Innes. 2003. Outcomes of collaborative water Henry, Adam D., Mark Lubell, and Mike McCoy. 2012. Survey-based meas-
policy making: Applying complexity thinking to evaluation. Journal of urement of public management and policy networks. Journal of Policy
Environmental Planning and Management 46:177–97. Analysis and Management 31:432–52.
Crona, Beatrice, and Orjan Bodin. 2006. What you know is who you know? Henry, Adam D., and Björn Vollan. 2014. Networks and the challenge of
Communication patterns among resource users as a prerequisite for co- sustainable development. Annual Review of Environment and Resources
management. Ecology and Society 11:7. 39:583–610.
Dale, Aaron, and Derek Armitage. 2011. Marine mammal co-management Imperial, Mark T. 2005. Using collaboration as a governance strategy lessons
in Canada’s Arctic: Knowledge co-production for learning and adaptive from six watershed management programs. Administration & Society
capacity. Marine Policy 35:440–49. 37:281–320.
Day, J. C., and Thomas I. Gunton. 2003. The theory and practice of collabora- Ingold, Karin, and Philip Leifeld. 2016. Structural and institutional determi-
tive planning in resource and environmental management. Environments. nants of influence reputation: A comparison of collaborative and adver-
31:5–19. sarial policy networks in decision making and implementation. Journal of
Desmarais, Bruce A., and Skyler J. Cranmer. 2012a. Micro-level interpreta- Public Administration Research and Theory 26:1–18.
tion of exponential random graph models with application to estuary net- Innes, Judith E., and David E. Booher. 1999. Consensus building and com-
works. Policy Studies Journal 40:402–34. plex adaptive systems. Journal of the American Planning Association
———. 2012b. Statistical inference for valued-edge networks: The generalized 65:412–23.
exponential random graph model. PloS One 7:e30136. ———. 2010. Planning with complexity: An introduction to collaborative
Dietz, Thomas, and Paul C. Stern. 2008. Public participation in environmental rationality for public policy. New York, NY: Routledge.
assessment and decision making. Washington, DC: National Academies Jasny, Lorien. 2012. Baseline models for two-mode social network data. Policy
Press. Studies Journal 40:458–91.
Emerson, Kirk, and Tina Nabatchi. 2015a. Collaborative governance regimes. Jasny, Lorien, and Mark Lubell. 2015. Two-mode brokerage in policy net-
Washington, DC: Georgetown Univ. Press. works. Social Networks 41:36–47.
———. 2015b. Evaluating the productivity of collaborative governance Karkkainen, Bradley C. 2002. Collaborative ecosystem governance: Scale,
regimes: A performance matrix. Public Performance & Management complexity, and dynamism. Virginia Environmental Law Journal
Review 38:717–47. 21:189–244.
Emerson, Kirk, Tina Nabatchi, and Stephen Balogh. 2012. An integrative Kelman, Steven, Sounman Hong, and Irwin Turbitt. 2013. Are there mana-
framework for collaborative governance. Journal of Public Administration gerial practices associated with the outcomes of an interagency service
Research and Theory 22:1–29. delivery collaboration? Evidence from British crime and disorder reduc-
Emerson, Kirk, Patricia J. Orr, Dale L. Keyes, and Katherine M. Mcknight. tion partnerships. Journal of Public Administration Research and Theory
2009. Environmental conflict resolution: Evaluating performance 23: 609–30.
outcomes and contributing factors. Conflict Resolution Quarterly Kilduff, Martin, and Wenpin Tsai. 2003. Social networks and organizations.
27:27–64. Thousand Oaks, CA: Sage.
Feiock, Richard C. 2013. “he institutional collective action framework. Policy Kolaczyk, Eric D. 2009. Statistical analysis of network data: Methods and
Studies Journal 41:397–425. models. New York, NY: Springer.
Feiock, Richard C., and John T. Scholz. 2009. Self-organizing federalism: Koontz, Tomas M., and Craig W. Thomas. 2006. What do we know and need
Collaborative mechanisms to mitigate institutional collective action dilem- to know about the environmental outcomes of collaborative manage-
mas. Cambridge, UK: Cambridge Univ. Press. ment? Public Administration Review 66:111–21.
Korfmacher, Katrina Smith, and Tomas M. Koontz. 2003. Collaboration, Pretty, Jules, and Hugh Ward. 2001. Social capital and the environment. World
information, and preservation: The role of expertise in farmland preserva- Development 29:209–27.
tion task forces. Policy Sciences 36:213–36. Provan, K. Keith G., and Patrick Kenis. 2008. Modes of network governance:
Kosnik, Lea. 2010. Balancing environmental protection and energy production Structure, management, and effectiveness. Journal of Public Administration
in the federal hydropower licensing process. Land Economics 86:444–66. Research and Theory 18:229–52.
Krivitsky, Pavel N. 2012. Exponential-family random graph models for valued Putnam, Robert D. 2000. Bowling alone. New York, NY: Simon & Schuster.
networks. Electronic Journal of Statistics 6:1100–28. Putnam, Robert, Robert Leonardi, and Raphaëlle Nanetti. 1993. Making
Krivitsky, Pavel N., and Carter T. Butts. 2013. Modeling valued networks with democracy work: Civic traditions in modern Italy. Princeton, NJ:
Statnet. The Statnet Development Team. https://statnet.csde.washington. Princeton Univ. Press
edu/ (accessed April 15, 2016). Robins, Garry, Jenny M. Lewis, and Peng Wang. 2012. Statistical network
Lasker, Roz D., and Elisas Weiss. 2003. Broadening participation in commu- analysis for analyzing policy networks. Policy Studies Journal 40:375–401.
nity problem solving: A multidisciplinary model to support collaborative Sandström, Annica, and Lars Carlsson. 2008. The performance of policy net-
practice and research. Journal of Urban Health 80:14–47. works: The relation between network structure and network performance.
Lazer, David, and Allan Friedman. 2007. The network structure of exploration Policy Studies Journal 36:497–524.
and exploitation. Administrative Science Quarterly 52:667–94. Schneider, Mark, John Scholz, Mark Lubell, Denisa Mindruta, and
Leach, William D., and Paul A. Sabatier. 2005. Are trust and social capital the Matthew Edwardsen. 2003. Building consensual institutions: Networks
keys to success? Watershed partnerships in California and Washington. and the national estuary program. American Journal of Political Science

In Swimming upstream: Collaborative approaches to watershed manage- 47:143–58.
ment, ed. Paul A. Sabatier, Will Focht, Mark Lubell, Zev Trachtenberg, Scholz, John T., Ramiro Berardo, and Brad Kile. 2008. Do networks solve
Arnold Vedlitz and Marty Matlock, 233–58. Cambridge, MA: MIT Press. collective action problems? Credibility, search, and collaboration. The
Leach, William D., Christopher M. Weible, Scott R. Vince, Saba N. Siddiki, Journal of Politics 70:393–406.
and John C. Calanni. 2013. Fostering learning through collaboration: Scott, Charity. 2011. A case study in collaborative governance: Health care
Knowledge acquisition and belief change in marine aquaculture partner- law reform in Georgia. Conflict Resolution Quarterly 28:441–63.
ships. Journal of Public Administration Research and Theory 24:591–622. Scott, Tyler A. 2015. Does collaboration make any difference? Linking collab-
Lee, Youngmi, In Won Lee, and Richard C. Feiock. 2012. Interorganizational orative governance to environmental outcomes. Journal of Policy Analysis
collaboration networks in economic development policy: An exponential and Management 34:537–66.
random graph model analysis*. Policy Studies Journal 40:547–73. ———. 2016. Analyzing policy networks using valued exponential random
Leifeld, Philip, Skyler J. Cranmer, Bruce A. Desmarais, and Maintainer graph models: Do government-sponsored collaborative groups enhance
Philip Leifeld. 2015. xergm: Extensions for exponential random graph organizational networks? Policy Studies Journal 44:215–44.
analysis. R Package version 1.5.0. http://143.107.212.50/web/packages/ Scott, Tyler A., and Craig W. Thomas. 2015. Do collaborative groups enhance
xergm/xergm.pdf. interorganizational networks? Public Performance & Management
Leifeld, Philip, and Volker Schneider. 2012. Information exchange in policy Review 38:654–83.
networks. American Journal of Political Science 56:731–44. Sirianni, Carmen. 2009. Investing in democracy: Engaging citizens in collabo-
Lubell, Mark. 2004a. Collaborative environmental institutions: All talk and rative governance. Washington, DC: Brookings Institution Press.
no action?” Journal of Policy Analysis and Management 23:549–73. Thomas, Craig W., and Tomas M. Koontz. 2011. Research designs for evalu-
———. 2004b. Resolving conflict and building cooperation in the national ating the impact of community-based management on natural resource
estuary program. Environmental Management 33:677–91. conservation. Journal of Natural Resources Policy Research 3:97–111.
———. 2013. Governing institutional complexity: The ecology of games Thomson, Ann Marie, James L. Perry, and Theodore K. Miller. 2008. Linking
framework.” Policy Studies Journal 41:537–59. collaboration processes and outcomes. In The collaborative public man-
Lubell, Mark, Adam D. Henry, and Mike McCoy. 2010. Collaborative insti- ager: New ideas for the twenty-first century, ed. Rosemary O’Leary and
tutions in an ecology of games. American Journal of Political Science Lisa B. Bingham, 97–120. Washington, DC: Georgetown Univ. Press.
54:287–300. ———. 2009. Conceptualizing and measuring collaboration. Journal of Public
Lubell, Mark, Garry Robins, and Peng Wang. 2014. Network structure and Administration Research and Theory 19:23–56.
institutional complexity in an ecology of water management games. Ulibarri, Nicola. 2015a. Collaboration in federal hydropower licensing:
Ecology and Society 19:23. Impacts on process, outputs, and outcomes. Public Performance &
Lubell, Mark, John T. Scholz, Ramiro Berardo, and Garry Robins. 2012. Management Review 38:578–606.
Testing policy theory with statistical models of networks. Policy Studies ———. 2015b. Tracing process to performance of collaborative governance:
Journal 40:351–74. A comparative case study of federal hydropower licensing. Policy Studies
Lusher, Dean, Johan Koskinen, and Garry Robins. 2013. Exponential Random Journal 43:283–308.
Graph Models for Social Networks: Theory, Methods, and Applications. Wondolleck, Julia M., and Steven L. Yaffee. 2000. Making collaboration work:
New York, NY: Cambridge Univ. Press. Lessons from innovation in natural resource management. Washington,
Margerum, Richard D. 2002. Evaluating collaborative planning: Implications DC: Island Press.
from an empirical analysis of growth management. Journal of the ———. 2003. Collaborative ecosystem planning processes in the United
American Planning Association 68:179–93. States: Evolution and challenges. Environments 31:59.
———. 2011. Beyond consensus: Improving collaborative planning and man-
agement. Cambridge, MA: MIT Press.
Newig, Jens, and Oliver Fritsch. 2009. Environmental governance: Appendix A
Participatory, multilevel, and effective? Environmental Policy and Model Specification and Goodness-of-Fit
Governance 19:197–214.
Newman, Lenore, and Ann Dale. 2007. Homophily and agency: Creating Model Specification
effective sustainable development networks. Environment, Development A GERGM assumes that the observed network (i.e.,
and Sustainability 9:79–90.
pattern of ties) represents a sample from a hypotheti-
O’Leary, Rosemary, and Lisa B. Bingham. 2003. The Promise and perfor-
mance of environmental conflict resolution. Washington, DC: RFF Press.
cal distribution of possible network realizations. To
Ostrom, Elinor. 2000. Collective action and the evolution of social norms. The draw inferences about the underlying network phe-
Journal of Economic Perspectives 14:137–58. nomena, a GERGM compares the observed network
to the distribution of possible network graph con- estimation technique that samples from the distribu-
figurations, with each hypothetical graph weighted tion of possible network configurations. Sampling is
based upon similarity to the observed configuration. weighted based upon similarity to the observed net-
Comparing the observed network to the distribution of work in terms of descriptive characteristics such as
possible networks provides a probabilistic assessment density and structural attributes such as the number of
of whether a particular driver is more or less prevalent triangular-patterned ties in order to generate a distri-
in the observed network (by showing whether a given bution of network graphs that are a suitable basis for
structure occurs more or less in the observed network comparison.
than if ties are distributed at random; Kolaczyk 2009; Basically, the MCMC procedure iteratively pro-
Lusher, Koskinen, and Robins 2013). poses a single tie change to the network and then
For nontrivial networks, a vast number of possible compares the likelihood of the pre-change and post-
network configurations exist, and it is infeasible to ana- change network graphs. Every time the proposed
lyze all possible graphs. Thus, GERGMs use a Markov change increases the likelihood of the simulated net-
chain Monte Carlo (MCMC) maximum likelihood work graph, the MCMC process makes the proposed

Figure A1. Tracing MCMC parameter estimates for Washington network model.
Figure A2. Tracing MCMC parameter estimates for Missouri network model.

Figure A3. Tracing MCMC parameter estimates for Georgia network model.
Figure A4. Density of MCMC parameter estimates for Washington network model.
change and then repeats the same steps. When the Krivitsky (2012) specifies the likelihood function for a
proposed change decreases the likelihood (shown zero-modified Poisson-reference GERGM as:
below) of the network graph, the MCMC process
only elects to make the proposed change a fixed per-
centage of the time (e.g., 50%; Lusher, Koskinen, and
Prθ ;h,η, g (Y = y ) ∝ ∏ exp (θ y 1 ij + θ 2 1yij >0 ) y ! (A1)
ij
(i , j ) ∈ Y
Robins 2013). This iterative procedure searches the
parameter space to (hopefully) converge on a station- with the reference distribution represented by h(y):
ary distribution of network graphs that resemble the
observed network. h (y ) = 1 / ∏y ij ! (A2)
(i , j ) ∈ Y
Social networks are often sparse in that actors who
interact often interact multiple times and many other Equation A1 specifies the model sans any additional
actors have zero ties. This means that the dyad-wise model terms as the observed baseline distribution
tie distribution is zero inflated relative to a stand- of network ties. Additional terms are then added
ard Poisson distribution (Krivitsky and Butts 2013). to the model to account for endogenous structural

Figure A5. Density of MCMC parameter estimates for Missouri network model.
Figure A6. Density of MCMC parameter estimates for Georgia network model.
characteristics and exogenous attributes of participat- Butts 2013).5 Each model in this article is fit via a four-
ing organizations. parallel sampling chains. Every chain discards the first
10,000 networks to ensure that samples used for esti-
Model Goodness-of-Fit mation purposes are drawn from a stable distribution
When fitting a GERGM, it is important to ensure that (Lusher, Koskinen, and Robins 2013). After this “burn
the MCMC estimation process does not exhibit degen- in” period, each chain takes 160,000 samples (with
eracy; a degenerate MCMC cascades to a completely each sample being a different possible network graph).
full or completely empty model (i.e., all high-value
ties or all no-value ties) and weights these unrealis- 5 Note that the goodness-of-fit plots shown in Appendix A refer to
tic network graphs too highly (Handcock et al. 2008; the unrestricted models presented in Table 6; we also performed
Kolaczyk 2009; Lusher, Koskinen, and Robins 2013). goodness-of-fit analysis for the restricted models shown in Tables 4
and 5. Each model mixes adequately, exhibits consistent, unimodal
Figures A1–A3 show traceplots for each param-
estimation within and across MCMC chains, and faithfully replicated
eter across iterations of each MCMC chain to dem- the structural characteristics of the observed networks. Because the
onstrate that the model is mixing sufficiently and not unrestricted models in Table 6 are more difficult to fit, we focus on their
straying outside of the parameter space (Krivitsky and goodness-of-fit in this appendix.
So that all draws are not taken from the same small converge and that the distribution of estimates gener-
area, each chain is thinned by discarding 1,000 indi- ated by each chain are unimodal and approximately
vidual perturbations between each sample. Figures A1, normally distributed. Figures A4–A6 show density
A2, and A3, respectively, thus trace the maximum like- plots for each model parameter for Washington,
lihood parameter estimates for each MCMC chain and Georgia, and Missouri, respectively.
parameter in the corresponding model. The seemingly One advantage of the parallel modeling approach
random, back-and-forth movement of each chain dem- we use, in which each model uses four separate MCMC
onstrates that the model is searching throughout the chains, is that comparing chain-specific distributions
parameter space, and the lack of an upward or down- demonstrates model robustness. Figures A4–A6 dem-
ward trend shows that each model does not become onstrate that the four parallel chains converge around
degenerated and trend toward a completely empty or consistent estimates for each model parameter. This
completely full network. is an important improvement upon many published
Along with mixing adequately and sufficiently ERGMs that do not use parallel processing. Although
searching the parameter space, it is also important serial processing still allows the analyst to establish

that the parameter estimates for each MCMC chain whether the single MCMC chain produces a unimodal
Figure A7. Distribution of simulated network statistics based upon Washington model.
Figure A8. Distribution of simulated network statistics based upon Missouri model.

Figure A9. Distribution of simulated network statistics based upon Georgia model.
distribution, comparing the results of several chains fit observed statistics. The unrestricted Washington
in parallel shows the extent to which model results are model (Figure A7) is the poorest fit in this regard, as
consistent and replicable. the model struggles to simulate networks that closely
Finally, in the discussion of model goodness-of-fit resemble the observed network. This is not the case
above in the main body of the manuscript, we noted for the restricted models with only sum, nonzero,
that GERGM inference is based upon comparing the and meetings attended parameters (shown in Tables
observed network to the distribution of simulated 4 and 5).
networks generated by the model. That is, if we simu- The conundrum here is that dropping model terms
late a large number of networks based upon estimated to increase the goodness-of-fit with regard to simula-
model parameters, we should expect that the simu- tion makes comparison to other network models more
lated networks generally resemble the observed net- problematic, since adding or subtracting parameters
work. To test this, we simulate 100,000 hypothetical changes each estimate (see Tables 4–6). Since our anal-
networks based upon each best fit model presented ysis focuses on comparison across the three cases, this
above and compare the distribution of network statis- issue is paramount. Moreover, given that (1) the good-
tics found in the simulated networks to the observed ness-of-fit metrics presented above (Figures A1 and
values in Washington, Georgia, and Missouri. A4) speak to adequate MCMC mixing and consistency
Figures A7–A9 plot the density of simulated val- in parameter estimation within and across MCMC
ues and show where the observed statistics fall within chains and (2) the worst fitting parameter in Figure A7
these distributions using a dashed line. In each case, is the Meetings Attending parameter, which is not of
the observed statistic is near the middle of the sim- substantive importance to the results in any case, we
ulated values and the simulated values appear to believe that it is still appropriate to make comparisons
be approximately normally distributed around the using the three models.

Linking Network Structure To Collaborative

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Linking Network Structure To Collaborative

Uploaded by

Copyright:

Available Formats

Journal of Public Administration Research And Theory, 2017, 163–181

Linking Network Structure to Collaborative

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Introduction of collaborative governance’s long-term performance

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Table 1. Level of Collaboration by Case

Washington Missouri Georgia

Principled engagement High Medium Low

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Appendix A discusses the mechanics of ERGMs in

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Table 3. Relating Hypotheses to Terms in ERGM Package

Network Concept Term in ERGM Packages Term Structure

H1 Network density sum g (y ) = ∑y ij

H3 Reciprocity mutual g (y ) = ∑ min (y , y ) ij ji

Case (Collaboration Level) Washington (High) Missouri (Medium) Georgia (Low)

Sum −3.62*** (0.04) −3.18*** (0.03) −3.03*** (0.05)

***p < .01, **p < .05, *p < .1.

Table 5. Density and Nonzero Ties with Control for Attendance

Case (Collaboration Level) Washington (High) Missouri (Medium) Georgia (Low)

Sum −1.92*** (0.08) −2.05*** (0.13) −2.09*** (0.22)

***p < .01, **p < .05, *p < .1.

Table 6. ERGM Parameter Estimates for Three Relicensing Networks

Case (Collaboration Level) Washington (High) Missouri (Medium) Georgia (Low)

Sum −2.99*** (0.08) −3.72*** (0.13) −3.92*** (0.21)

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

***p < .01, **p < .05, *p < .1.

For instance, by exponentiating the sum param-

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

Downloaded from https://academic.oup.com/jpart/article/27/1/163/2629285 by guest on 10 March 2022

You might also like

Sum −3.62* (0.04) −3.18* (0.03) −3.03*** (0.05)

p < .01, p < .05, p < .1.

Sum −1.92* (0.08) −2.05* (0.13) −2.09*** (0.22)

p < .01, p < .05, p < .1.

Sum −2.99* (0.08) −3.72* (0.13) −3.92*** (0.21)

p < .01, p < .05, p < .1.