Safety Science 83 (2016) 59–73

Contents lists available at ScienceDirect

Safety Science
journal homepage: www.elsevier.com/locate/ssci

Evaluation of a safety culture intervention for Union Pacific shows

improved safety and safety culture
Michael Zuschlag a,⇑, Joyce M. Ranney a, Michael Coplen b
a
Office of Safety Management and Human Factors, US Department of Transportation John A. Volpe National Transportation Systems Center, 55 Broadway, Cambridge, MA 02142, USA
b
FRA Human Factors Division, Office of Research, Development, and Technology, 1200 New Jersey Avenue, SE, Mail Stop 20, Washington, DC 20590, USA

a r t i c l e i n f o a b s t r a c t

Article history: The Federal Railroad Administration (FRA) sponsored a multiyear pilot demonstration of Clear Signal for
Received 1 February 2013 Action (CSA), a safety culture intervention implemented with Behavioral Science Technology Inc., at a
Received in revised form 26 August 2015 Union Pacific (UP) service unit. CSA combines peer-to-peer feedback, continuous improvement, and
Accepted 1 October 2015
safety-leadership development. The US Department of Transportation John A. Volpe National
Transportation Systems Center conducted an independent program evaluation of the pilot, using qualita-
tive and quantitative measures. The evaluation found that, over two years, the site experienced signifi-
Keywords:
cant improvements in safety outcomes, operations, and safety culture, including an 80% drop in at-risk
Peer-to-peer feedback
Railroad
behaviors, a 79% decrease in engineer decertification rates, a 81% decline in the rate derailments and
Changing At-Risk Behavior (CAB) other incidents, and better labor–management relations. Comparison locations showed no improvements
Continuous improvement in the decertifications or derailments. The success of the pilot, in addition to successes UP had earlier with
Safety culture CSA-type processes, encouraged UP to expand these processes throughout its transportation department.
Safety leadership The success of this pilot and other similar pilots led to the development and adoption of the FRA’s
Railroad Safety Risk Reduction Program in the Rail Safety Improvement Act of 2008, and the implemen-
tation of similar safety-culture programs by other carriers.
Ó 2015 Elsevier Ltd. All rights reserved.

1. Introduction counteract systemic upstream causes of accidents and injuries

1.1. Background
Despite continued efforts by the Federal Railroad Administra-
tion (FRA), management, and unions, safety systems in the railroad
industry have stagnated (Coplen, 1999). The current safety systems
are imbedded in the industry’s ‘‘safety culture.” In this article
safety culture is defined as the factors that determine an organiza-
tion’s (labor and management) commitment, style, and proficiency
in ensuring safety that result from safety-related beliefs, values,
attitudes, competencies, and behavioral patterns (Reason, 1997,
2003). The railroad safety systems have progressively elevated
safety performance until 1986, when improvements stalled (FRA,
2001, 2008). Therefore, it is possible that the safety culture may
be limiting further safety improvements (Ranney and Nelson,
2003).
The key aspect of an effective safety culture is a ‘‘trust culture,”
where the organization’s members trust each other (Reason, 2003).
This trust is necessary to open rich communication on safety
issues, allowing an organization to identify and ultimately
⇑ Corresponding author. Tel.: +1 617 494 3250.
E-mail address: michael.zuschlag@dot.gov (M. Zuschlag).
(Reason, 1997, 2003; DeJoy, 2005). However, railroad culture has
several characteristics that limit such trust, especially between
labor and management (Coplen, 1999). The culture has a
command-and-control management style, reactive tendencies,
and inclinations to inflict punishment for accidents and injuries,
inclinations that arise from a rule-and-discipline approach to
safety (Gamst, 1982) and litigious incentives. In practicing com-
mand and control, managers tend not to elicit input (including
safety-related information) from workers but instead to issue
orders. Reactive tendencies discourage proactively collecting infor-
mation on conditions or trends that may lead to accidents or inju-
ries. Instead, labor and management react to each injury or
accident as a separate incident. Traditionally, management tends
to blame accidents and injuries on rules violations, while labor
tends to blame workplace deficiencies or management pressure
for productivity. Managers characteristically respond to injuries
and accidents by disciplining workers, including firing them, for
safety rule infractions. Injured workers often sue the company
for financial compensation under the 1908 Federal Employers Lia-
bility Act (FELA). Fear of discipline and lawsuits breeds distrust and
chills cooperation and communication between workers and man-
agers, stifling the sharing of safety information.

http://dx.doi.org/10.1016/j.ssci.2015.10.001
0925-7535/Ó 2015 Elsevier Ltd. All rights reserved.
60 M. Zuschlag et al. / Safety Science 83 (2016) 59–73
To improve safety, the FRA Human Factors Division is exploring
new approaches that counteract these cultural tendencies. These
approaches achieve this by incorporating the following features,
which are characteristic of a positive safety culture (Reason,
1997; Phimister et al., 2004):
 Nondisciplinary: Seeking to improve safety without punishment
or blame through protective elements such as worker
anonymity.
 Proactive: Collecting data on at-risk behaviors and conditions to
prevent associated accidents or injuries before they occur, and
thus reduce the incentives for workers and managers to blame
each other.
 Systems-safety-analysis orientation: Gathering and using rich
objective data to identify underlying organizational factors in
safety.
 Cooperative: Engaging stakeholders within both management
and labor.
 Sustainable: Including mechanisms for long-term sustainment.
These features improve safety by creating an environment
where individuals freely exchange information upward, down-
ward, and laterally across the organizational hierarchy, providing
the open communication necessary to solve safety problems.
This paper presents an evaluation of one such approach, the
FRA’s Clear Signal for Action (CSA) applied to a transportation
department. With funding and sponsorship from the FRA, Behav-
ioral Science Technology Inc. (BST) actively designed and imple-
mented the demonstration pilot. The US Department of
Transportation John A. Volpe Center, also with sponsorship from
the FRA, independently conducted a formative and summative
evaluation (Rossi et al., 1999). This article presents a summary of
the summative evaluation.
1.2. Clear Signal for Action (CSA)
1.2.1. CSA implementation
CSA integrates three approaches that have been applied previ-
ously to improve safety proactively:
 Peer-to-peer feedback (PPF), where workers observe each other,
and exchange feedback about the safety of their behavior, con-
ditions, and organizational factors (Geller, 2001; Krause, 1995).
 Continuous improvement (CI), where workers and managers
cooperatively gather and analyze data to identify systemic
causes of observed at-risk behaviors and conditions, and then
implement corrective actions to address the causes
(Harrington, 1987; Juran, 1964; Krause, 1995).
 Safety-leadership development (SLD), where managers are
trained to promote proactive safety practices such as PPF and
CI (Krause et al., 1999).
Fig. 1 illustrates a combined theory of action and theory of
change (Funnel and Rogers, 2011) for CSA.
Detailed CSA activities are listed in the box headed Implementa-
tion, and their theoretical outcomes are depicted in the two columns
of boxes designated Proximal Outcomes and Distal Outcomes. Prox-
imal outcomes result directly from implementation activities, while
distal outcomes are mediated by proximal outcomes. The arrows
indicate the effects of prior activities and outcomes on subsequent
ones, with influence moving primarily in a left-to-right direction.
Within the Implementation box, activities are grouped according
to their primary association with PPF, CI, or SLD.
1.2.1.1. Peer-to-peer feedback (PPF). To initiate the PPF component,
a local CSA process steering committee, composed of workers and
sometimes several managers, develops a checklist of safe and at-
risk worker behaviors and working conditions based on analyses
of injury reports and other sources of safety information (Krause,
1997). The steering committee recruits, trains, and coaches work-
ers to be ‘‘peer observers,” who observe the safety of their cowork-
ers (overtly, with their permission) then conduct with them
anonymous nonconfrontational feedback sessions devoid of any
disciplinary connections. The feedback includes both acknowledg-
ing any observed safe behavior, and discussing any observed at-
risk behavior. By focusing on the behavioral and conditional ante-
cedents of accidents, CSA seeks to proactively prevent accidents
before they occur.
1.2.1.2. Continuous improvement (CI). Within the CI component,
workers are trained to interview their coworkers during the feed-
back sessions about the coworkers’ explanations for any observed
at-risk behaviors or conditions. Thus, a feedback session has feed-
back from the observed peer to the observer, in addition to from
observer to peer. The observing worker records on the checklist
all data on the behaviors and the explanations. The steering com-
mittee aggregates and objectively analyzes these data through
root-cause problem-solving to identify the systemic causes for bar-
riers to enhancing safety. Potential systemic causes include organi-
zational policy, training, tool design, environmental conditions,
procedures, and cultural aspects. The steering committee executes
corrective actions against barriers that it can remove, for example,
through feedback to workers during PPF sessions. If the barriers
require actions beyond the authority of the steering committee,
such as new equipment purchase or procedures changes, a joint
labor–management barrier removal team reviews and prioritizes
the barriers, then develops corrective actions, which management
executes. Data-gathering continues after a corrective action is
deployed, to allow its effectiveness to be evaluated.
1.2.1.3. Safety-leadership development (SLD). Within the SLD com-
ponent, managers are trained in effective nondisciplinary, proac-
tive techniques for enabling employees to work safely, including
but not limited to supporting safety-related activities such as feed-
back sessions and barrier removal. These SLD processes are con-
ducted parallel to existing disciplinary processes. SLD activities
are not a substitute for addressing rules violations.
1.2.2. Integration of behavioral and safety culture approaches
When used alone, PPF approaches have often placed too little
emphasis on the influence of upstream managers, systems, and
policies and procedures on at-risk behavior and conditions, result-
ing in negative reviews from several unions (Spigener and Hodson,
1997; Howe, 1999; Frederick and Lessin, 2000). Thus, recent vari-
ants of PPF, such as CSA, have acknowledged that behavioral-
oriented safety interventions can complement culture-oriented
safety interventions such as CI and SLD. (DeJoy, 2005). These new
variants integrate PPF with CI, utilizing the peer-to-peer sessions
as opportunities to collect the data needed by CI. SLD encourages
managers to implement corrective actions that need management
support and otherwise targets ‘‘latent” factors in accidents and
injuries that are further back in the chain of causation, such as
safety climate and culture (Reason, 1997). SLD trains organiza-
tional leadership to eliminate these causes since it has the
resources and authority to alter the direction of the organization.
It also trains managers to provide the necessary resources and
the integration of CSA into other safety programs so it becomes
institutionalized. SLD can therefore accelerate changes initiated
by PPF and CI and make them lasting characteristics of the organi-
zation’s safety culture.
By combining PPF, CI, and SLD, responsibility for safety is
distributed among workers and managers. PPF activities are
 M. Zuschlag et al. / Safety Science 83 (2016) 59–73
Fig. 1. Theory of action for CSA, showing activities and theoretical outcomes (PPE: personal protective equipment).
predominantly or fully workers’ responsibility since peers make
observations and provide feedback. Because SLD changes organiza-
tional leadership, it is fully managers’ responsibility. CI is a joint
responsibility, but because management is accountable for ensur-
ing that the systemic corrective action process is effective, man-
agers have more responsibility than workers (Deming, 2000;
Walton, 1986). Involving both parties in safety helps to embed a
safety process into the culture by giving each party a stake in its
successful outcome. In contrast, when a safety process is solely
the responsibility of either labor or management, it will tend to
have a more limited impact on the non-responsible party. CSA is
thus an integration of behavioral-oriented and culture-oriented
safety interventions (DeJoy, 2005). PPF improves safety and culture
from the bottom up via the activities of workers, SLD improves
safety and culture from the top down via the activities of man-
agers, and CI represents the meeting of the top-down and
bottom-up processes, where workers and managers share informa-
tion on safety problems and solutions.
1.2.3. Theoretical outcomes on safety performance and culture
The theory of action for CSA shown in Fig. 1 specifies the out-
comes on the following:
 Worker practices: The at-risk and safe behaviors identified in the
PPF checklist and targeted by feedback sessions. Examples
include walking with eyes on the path and following crew-
safety communication procedures.
 Systemic conditions: Aspects of the physical and organizational
environment that impact safety, such as facilities, tools, train-
ing, and disciplinary policies. Examples include switches that
are in good operating condition, availability of proper tools
and personal protective equipment (PPE), and training in safe
worker practices such as the proper way to align a coupler.
 Management practices: On-the-job manager behaviors that take
the form of proactive, nondisciplinary promotion of safe behav-
iors, conditions, and processes. Examples include coaching
employees on safe behavior and encouraging participation in
safety processes. These practices are in addition to implementa-
tion activities such as consistently providing the necessary bud-
get, equipment (e.g., computers for data analysis), and physical
space for training and operations.
61
 Safety culture: The factors that determine an organization’s
(labor and management) commitment, style, and proficiency
in ensuring safety that result from safety-related beliefs, values,
attitudes, competencies, and behavioral patterns (Reason, 1997,
2003). This evaluation is specifically concerned with attitudes
and behavioral patterns related to labor–management trust
(Reason, 1997, 2003), the lack of which has theoretically limited
safety advances in the railroad industry (Coplen, 1999).
 Occurrences: Safety events associated with injuries; fatalities;
damage to equipment, for example, from derailments; and asso-
ciated close calls, for example, safety-rule violations such as
engineer decertifications.
As depicted by the arrows in Fig. 1, the implementation directly
promotes the targeted safety practices of both managers and work-
ers through its SLD and PPF components, respectively. Workers’
practices change as a result of both the feedback sessions and
the training required to perform them (Ranney et al., 2010). The
CI process directly improves behavioral and systemic conditions
identified in data analysis.
There are reciprocal effects between specific practices and
safety culture. Changes in safety attitudes encourage workers and
managers to change their practices. Conversely, changes in prac-
tices may change attitudes (Krause, 1995). Safer practices also
improve the culture by building better labor–manager relations.
As managers see workers encouraging each other to work more
safely, their trust that workers will perform tasks safely increases
and they perceive less of a need for discipline. With managers
engaging in more proactive, non-disciplinary approaches to safety,
workers are assured of management’s commitment to their safety.
As discipline becomes less frequent and management’s commit-
ment to safety becomes more apparent, workers trust managers
more.
Improved systemic conditions also improve labor–management
relations within the safety culture by demonstrating manage-
ment’s commitment to safety. Improved labor–management rela-
tions can reciprocally improve systemic conditions by fostering
labor–management and cross-trade cooperation in improving
safety conditions.
Safer practices and improved systemic conditions reduce occur-
rences, as does the safety culture through increased dialogue about
62 M. Zuschlag et al. / Safety Science 83 (2016) 59–73

safety. Collectively, these changes result in improved worker and The newer workers were somewhat less distrustful of
manager safety practices beyond those specifically targeted by management.
implementation.
In summary, CSA improves safety culture simultaneously with
1.3.2. CAB structure and initiation
safety performance through ‘‘boot-strapping” reciprocal causation.
CAB leadership and day-to-day operations were executed by
two full-time ‘‘facilitators”: an engineer from BLET and a conductor
from UTU. The steering committee comprised an additional eight
1.3. CAB: An implementation of CSA
workers, who met approximately once a month. A newly arrived
superintendent (head of the service unit) supported the implemen-
1.3.1. Context and motivations
tation, in part due to seeing CSA-like Total Safety Culture success-
Between 2005 and 2008, the FRA sponsored a CSA demonstra-
fully implemented in a mechanical department in the UP Central
tion pilot in the transportation department on the San Antonio Ser-
Region. The superintendent appointed a senior service-unit man-
vice Unit of the Southern Region of Union Pacific Railroad (UP).
ager to serve part-time as the local management sponsor for
Transportation departments in railroads comprises road opera-
CAB, acting as chief liaison between the steering committee and
tions, which are driving trains long distances on main line track
management.
between terminals, and switching operations, which are sorting
BST’s model of CSA includes customizing it to fit with local con-
cars in yards. Behavioral Science Technology Inc. (BST), a company
texts. For example, the stakeholders at San Antonio chose not to
that has implemented CSA-like programs in a broad range of indus-
focus at first on traditional industrial safety threats, such tripping
tries, designed and instructed the implementation of the San Anto-
hazards and pinch points. Instead, CAB initially focused on behav-
nio CSA process, which local stakeholders named Changing At-Risk
iors to improve alertness and teamwork for locomotive cab opera-
Behavior (CAB).
tions on the road. Its focus was limited to practices related to high-
Like all the service units of UP, the San Antonio Service Unit
workload situations, such as operating under constraining signals,
comprised hundreds of transportation workers, most working out
a condition that UP calls Cab Red Zone (CRZ). Fourteen months
of a central hub (in this case, the city of San Antonio), with others
after its origination, CAB expanded to include safety in yard-
working out of peripheral terminals up to 150 miles away (e.g.,
switching operations, using a different checklist of behaviors. In
Laredo and Del Rio). CAB covered approximately 1100 workers;
the current paper, the implementations are distinguished as
the size of the workforce remained unchanged throughout the
CAB–CRZ and CAB-Switching. The evaluation of CAB thus com-
evaluation period, and turnover was low.
prised two phases: a first phase, from August 2005 through
During the CAB implementation, UP and the FRA started a sim-
September 2006, which included only CAB–CRZ, and a second
ilar CSA demonstration pilot on the Livonia Service Unit (Coplen
phase, from October 2006 to about January 2008, which included
and Ranney, 2009) and UP, acting independently from the FRA, ini-
CAB-Switching and the continuation of CAB–CRZ. The evaluation
tiated a PPF process on the Houston Service Unit. BST provided
also included a four-year baseline period prior to the first phase,
consultation services for Livonia, and, to a lesser extent, Houston.
for which data indicated the site’s state prior to CSA
Livonia and Houston were also in the Southern Region, although
implementation.
their safety processes were limited to switching operations, while
The CAB process began in August 2005 with the initiation of
CAB covered both switching and road operations.
regular peer-to-peer feedback sessions. Often the employees
UP was interested in partnering with FRA with these demon-
observed their peers while engaged in their normal work (e.g., a
stration pilots because they had been working with the three ele-
conductor may observe an engineer across the cab of a locomo-
ments of CSA – PPF, CI, and SLD – since 1988. In 1993, UP had
tive). However, many employees also dedicated entire days to con-
established a successful CSA-type program entitled Total Safety
duct only PPF sessions (e.g., ride in several different cabs per day,
Culture (Geller, 2001) in its mechanical department. After seeing
observing both engineer and conductor). For each observation,
it succeed there, UP wanted to see if it could work in transportation
the observer and peer completed the feedback exchange at a safe
where most accidents occur. UP was particularly focused on the
and convenient break in the work (e.g., while the train waited at
Southern Region because a series of high-profile accidents there
a siding for another train to pass). Employees would choose to
had increased FRA scrutiny of the region’s safety. The severity of
observe primarily based on available opportunities, especially for
these particular accidents may have justified the scrutiny, but
CAB–CRZ. CRZ conditions may or may not occur during a trip,
the Southern Region’s levels of safety through that time were actu-
and the ability of an employee to observe the crew of a particular
ally steady or improving (Zuschlag et al., 2012), and UP’s overall
train depended on the ability to arrange rendezvouses with the
injury rate was lower than most US railroads (FRA, 2006).
train. However, especially for CAB-Switching, the steering commit-
Local managers and workers, along with their unions, the
tee also chose to focus PFF sessions on locations where they felt it
Brotherhood of Locomotive Engineers and Trainmen (BLET) and
was needed most. The criteria for choosing locations did not
the United Transportation Union (UTU), worked together to imple-
change over the course of the evaluation period.
ment CAB. The recent serious accidents in the region raise both
manager and worker awareness of safety, and encouraged the
two parties to cooperate for improvements despite the historic dis- 1.3.3. CAB evolution and challenges
trust between them. Labor support for CAB was also abetted by an Approximately halfway through the evaluation period, the
influx of new workers throughout the corporation prior to the eval- superintendent was promoted and left the service unit. In UP, this
uation period. While average UP years-of-service for transportation tenure of one to two years is fairly typical for a railroad superinten-
workers at SASU was 16 years, the influx created a bimodal distri- dent, and was observed in other service units of the UP Southern
bution. One side of the distribution, comprising two-thirds of the Region. CAB facilitators briefed the newly appointed superinten-
workers, averaged 7 years of service, while the other side averaged dent on the process, and he continued to strongly support the
33 years of service. The worker’s age distribution, likewise, was implementation. At about the same time, the local management
bimodal, with one side averaging 35 years of age and the other sponsor was also promoted and left the service unit. His lieutenant,
averaging 53 years of age. The influx of relatively new workers who had previously earned credibility with the workers for his
facilitated support for CAB because older workers had developed support of CAB among other safety work, became the new local
a deep distrust of the management over decades of employment. management sponsor.
 M. Zuschlag et al. / Safety Science 83 (2016) 59–73
By the end of the evaluation period, over half the workforce had
been trained to conduct PPF sessions, a rate somewhat greater than
expected at the start of implementation. The higher rate was
achieved because the superintendent arranged that additional
training sessions be held when rail traffic was low and manpower
demands were reduced (e.g., in January of each year). Originally
the steering committee conducted training only at a terminal in
the city of San Antonio, but over the course of the evaluation it
expanded training to the peripheral terminals. During this time,
three of the original steering committee members were systemat-
ically replaced by other workers, consistent with the BST consul-
tant’s advice for terms limits and rotation. Two of the new
members were from the outlying terminals to provide balanced
representation.
To support CAB across the service unit, the superintendent set a
fixed monthly ‘‘budget” of worker-hours for CAB training, PPF ses-
sions, analysis, and other CAB activities. In an unusual move for a
railroad, the superintendent delegated responsibility for allocating
the budget among CAB activities to the facilitators demonstrating
commitment to the program and trust in the facilitators. Normally,
a manager (e.g., the management sponsor), rather than a worker,
would receive such responsibility. Through the first year of the
implementation, management was pleased with the way in which
the facilitators allocated the budget.
However, as the implementation expanded to peripheral ter-
minals and to switching operations in the second year of imple-
mentation, the facilitators had increasing difficulty completing
all activities. The rate of PPF sessions, a metric of implementation
strength, was falling far below the planned level. The facilitators
requested a larger budget commiserate with the expansion of
CAB, but, due to external economic and organizational pressures,
the superintendent could barely maintain the existing budget.
The facilitators appealed to the superintendent’s superior, the
transportation vice president for the region. The typical railroad
management response to such a crisis would be to exercise ‘‘com-
mand and control,” and takeover budgeting responsibility from
the workers. However, the vice president instead issued the
facilitators a challenge: for every worker-day of CAB administra-
tion the facilitators eliminated, the vice president would increase
the budget by one-and-a-half worker-days. The steering
committee rose to the challenge and reduced administration
costs by 20%, earning a 30% budget increase. The combined
increase in efficiency and budget size allowed an increase
in the rate of PPF sessions and the expansion to the entire
service unit.
1.3.4. Implementation strength
For the sake of relative brevity, this paper excludes a formal
evaluation of implementation, and instead focuses on the evalua-
tion of the outcomes, especially the effects of CSA on safety and
safety culture. Details of the context, pre-existing mechanisms
(Pedersen et al., 2012), and the implementation evaluation are in
a full government report on the demonstration pilot (Zuschlag
et al., 2012). Only a brief summary is provided herein.
The implementation evaluation, using quantitative and qualita-
tive measures, found CAB to be sufficiently strong to proceed with
the outcome evaluation (Rossi et al., 1999). Management acted as
‘‘safety climate engineers” (Simard and Marchand, 1994, 1997)
providing the necessary resources for the process and enlisting
labor officials and workers to support the program. Worker sup-
port for the process grew over the evaluation period in part due
to knowledge that the all-worker steering committee maintained
control of the feedback session data to ensure no negative reper-
cussions for participating workers.
At the end of the evaluation period, the workers had partici-
pated in approximately 4800 PPF sessions. While this corresponds
63
to more than four PPF sessions per worker, the total number of PPF
sessions remained below what had been planned at the start of the
implementation. However, by other measures, the implementation
was very strong. For example, as related above, management was
motivated to make CSA successful and supplied the worker-hours
and other resources to support it. In addition workers were trained
faster than planned. These characteristics, combined with the com-
pletion of the program activities (e.g., data analysis and corrective
action execution), implied a strong implementation (Egan et al.,
2009).
Trust between management and labor at the site was problem-
atic, a condition that augurs poorly for safety process success
(Reason, 1997). However, the parties were inclined to cooperate
more than normal for a railroad due to joint concern over the
recent serious accidents. The distrust was also counteracted by
the role behaviors of the leadership (Pedersen et al., 2012). Both
the superintendent and regional vice president resisted the rail-
road inclination to exert managerial control over the process,
which would have likely led to worker distrust and disenchant-
ment. Instead, the leadership empowered the workers by giving
them control over the process and budget, encouraging them to
find their own solutions to problems. This concrete and visible
way of going above and beyond the normal procedures demon-
strated commitment to the program and a show of good faith
which likely invested the workers in the process, helped grow
the trust in the leadership.
In summary, the implementation, while not perfect, was rea-
sonably strong and complete. Any lack of observable outcomes
for safety or safety culture cannot be attributed to implementation
failure, and, conversely, any observed outcomes may be reasonably
attributed to the CSA implementation (Egan et al., 2009).
1.4. Evaluation question
This paper is summative evaluation of a pilot demonstration of
CSA for outcomes for safety or safety culture, comprising a
‘‘bottom-line” assessment of the effectiveness of CSA at the site
(Rossi et al., 1999). For this purpose, the evaluation question was,
‘‘What are the effects of the CSA process on safety and safety cul-
ture?” The evaluation looked for improvements in:
 Worker practices.
 Occurrences, specifically incidents of material damage and
engineer decertifications.
 Safety culture, specifically the presence of a ‘‘trust culture” in
labor–management relations.
Additional evaluation questions and the formative evaluation
are addressed in Zuschlag et al. (2012).
2. Evaluation design, procedures, and data analysis
Randomized controlled trials (RCT) were infeasible for this eval-
uation, as is often the case for studies concerning occupational
safety programs (Pedersen et al., 2012). Instead, the evaluation
used a mixed-methods design, with both quantitative and qualita-
tive methods (Creswell, 2003). A mixed-methods design has
advantages over RCT for assessing the effectiveness of organiza-
tional change efforts, such as safety culture interventions, which
are embedded in a complicated system. For example, CAB operated
in the context of complications such as customized program imple-
mentations and leadership turn-over. The evaluation period for
CAB spanned approximately two and a half years (August 2005
to about January 2008), providing ample time for extraneous
events. By combining qualitative and quantitative methods for
64 M. Zuschlag et al. / Safety Science 83 (2016) 59–73
multiple data sources, the mixed-method design can account for
the impact of context and extraneous events that influence the
ability of the program to produce outcomes. Specifically in this
evaluation, we used qualitative methods to peer inside the ‘‘black
box” of the program and directly observe the influence of the con-
text on the mechanism comprising the program (Pawson and Tilly,
1997; Pawson, 2002). In addition, when a detailed theory of change
drives selection of the measures, the mixed-methods design rivals
randomized control trials in an ability to make plausible inferences
of causation (GAO, 2009). This summative evaluation used a con-
current nested research strategy (Creswell, 2003): the quantitative
data were considered the primary measure for improvements in
safety or safety culture, while the qualitative data provided expla-
nations and insight into the processes associated with the quanti-
tative results. Qualitative data also provided an opportunity to
capture unexpected outcomes.
Qualitative measures of outcomes, utilizing a case-study
methodology (Yin, 2009), included primarily open-ended inter-
views with workers and managers and were used for explanatory
purposes. Quantitative measures included:
 PPF feedback session data.
 Corporate safety data supplied by UP.
 Close-ended attitude and behavior surveys of workers and
managers.
In addition to the above measures for the outcomes of CSA, the
evaluation included measures for an implementation evaluation
and the assessment of the role of initial conditions and events in
establishing CSA at the site (as summarized above in Section 1.3
and detailed in Zuschlag et al., 2012). These measures included
field notes, process artifacts, and project records as additional qual-
itative data sources for assessing the context and mechanisms. By
using a theory of change, combining qualitative and quantitative
methods, and performing an implementation evaluation to assess
if an implementation failure occurred, the method is thus consis-
tent with a realistic model of evaluation (Pedersen et al., 2012;
Pawson et al., 2005).
2.1. Feedback session data
In accordance with BST training and materials, the CAB steering
committee collected the checklists completed during feedback ses-
sions and scored the data on each checklist to record which behav-
iors were performed safely and which were at-risk. For purposes of
evaluation, the overall percentage of at-risk behaviors for the ser-
vice unit for each month of the evaluation period provided an indi-
cation of that month’s safety of worker practices targeted by CAB–
CRZ and CAB-Switching. Autocorrelations of the data (Cohen et al.,
2003) were not significant, indicating that the at-risk percentages
were independent month-to-month, and that ordinary regression
analyses, rather than a time-series analyses, were suitable. If CAB
is effective at improving CRZ or switching practices, then the per-
centage of at-risk behaviors should, on average, decrease over the
months of the evaluation period.
An analysis of data from training videos assessed the consis-
tency of worker recordings of at-risk behaviors. CAB–CRZ trainers
showed one of two videos at the end of peer-observer training.
Each scripted video portrayed a typical mix of safe and at-risk
behaviors for five CRZ events. Trainees used their checklists to
record safe and at-risk behaviors seen in the videos. The same
videos were used for the coaching of many peer observers some-
time after training, but during coaching each peer observer saw a
different video than that seen during training.
Having members of the same population rate the same videos
allowed calculation of the consistency of the ratings, as shown in
Table 1.
Reliability is necessary in order for actual changes in practices to
be statistically detectable with CAB feedback session data. Time
drift represents a systematic tendency for the same practices to
be rated as safer (positive drift) or more at-risk (negative bias) in
later than in earlier training. Retest drift represents a tendency for
the same practices to be rated as safer (positive) or more at-risk
(negative) relative to how long ago an individual worker was
trained.
Complete ratings of training videos were collected from 108
peer observers at the end of training and from 13 peer observers
during subsequent coaching. Reliability was significantly higher
than 80% (83.06%, t(107) = 4.56, p < 0.0001); this is above the min-
imal standard for inter-judge reliability that is often used in eval-
uation research (Rossi et al., 1999). Time drift was not significant
(r = 0.0312, n = 108, p = 0.749), indicating that peer observers
trained at the beginning of the evaluation period rated the same
sessions as equally at-risk as those trained at the end of this period.
However, the analysis for retest drift showed that peer observers
rated the videos as more at-risk during coaching than during train-
ing (average difference, 7.50%; t(106) = 2.05; p < 0.0427), sug-
gesting that experienced peer observers became stricter over
time. The proportion of experienced peer observers increased
throughout the evaluation period since observers trained in the
beginning stayed to conduct feedback sessions at the end. This sug-
gests the possibility that the true average at-risk scores may be
lower than reported for later periods in the evaluation and that
any improvement in practices that may be found would in fact
be greater than indicated.
Overall, the feedback session ratings appeared to be adequate or
perhaps even conservative for measuring positive changes in
practices.
2.2. Corporate safety data
UP provided corporate safety data from its own tracking sys-
tems to allow the calculation of changes in the rates of relevant
safety occurrences since the start of CAB. The number of occur-
rences in the dataset was too few and the information on them
too sparse to allow useful sub-categorization of the occurrences
in order to analyze for changes in the character of the occurrences.
Interviews with managers revealed no evidence of changes in the
character of occurrences at any site.
2.2.1. Engineer decertifications
The railroad industry defines an engineer decertification as an
occurrence in which a locomotive engineer loses FRA authorization
to run trains due to a serious safety violation. In this evaluation,
decertifications served as a leading indicator of catastrophic road
accidents, which can easily result from the associated safety
violations.
Decertifications are a relatively objective measure avoiding the
measurement problems associated with minor injuries (Pedersen
et al., 2012). Automated electronic devices aboard the trains and
along the tracks detect most of decertifications the evaluation ana-
lyzed, making such occurrences resistant to over-reporting and
under-reporting. The decertification process involves an investiga-
tion including the railroad management, the union, and the FRA,
and includes documented evidence and a hearing, where all parties
crosscheck the evidence and conclusions. This crosschecking by
parties with different interests further reduces the chance of bias.
The data were limited to three types of decertifications that are
associated with CRZ practices because each often results from a
 M. Zuschlag et al. / Safety Science 83 (2016) 59–73
Table 1
Measures of consistency for CAB feedback session data.
Measurement Calculation
Reliability Percentage of agreement between a peer observer and the
standard observation for each CRZ event in the video as
determined by the steering committee
Time drift Trend in average at-risk scores over time, as indicated by
correlation of the scores with the date that the video was
rated in training
Retest drift Difference in average at-risk scores between training and
coaching
loss of crew attention under CRZ conditions. These types are: pass-
ing a stop signal, moving outside an authorized stretch of track,
and exceeding train or track speed limits by more than 10 mph.
Decertification data were further filtered to occurrences in which
the formal investigation of the decertification found the engineer
to be at fault. This comprises the vast majority of the decertifica-
tion data. If CAB–CRZ is effective in reducing occurrences, CRZ-
related decertifications should decrease, beginning in the first
phase, after the initiation of CAB–CRZ.
2.2.2. Incidents of material damage
The railroad industry defines ‘‘incidents” as occurrences involv-
ing operation of on-track equipment that result in property dam-
age (e.g., see FRA, 2011). These occurrences include collisions,
fires, and derailments, with the latter accounting for approxi-
mately 84% of the incidents in the data. The incident data in this
evaluation are resistant to under-reporting because by definition
they involve physical hardware damage and its repair, and this is
difficult-to-impossible to hide or ignore. For example, in the case
of defining a derailment, a wheel of a train is either on the track
or off. If it is off, then special equipment must be ordered to get
it back on before the train can be moved. A given transportation
crew might not report damage from a collision on its shift, but then
the subsequent crew or maintenance personnel will discover the
damage and report it. While there may be disciplinary
repercussion for causing a derailment, it cannot be fixed without
management intervention.
UP provided both FRA-reportable and non-reportable incidents
for this evaluation, the difference being the cost of the incidents.
The FRA states that any incident that costs the carrier more than
a certain amount (for example, $7700 in 2006) must be reported
to the government and is hence a reportable incident. The inclu-
sion of non-reportable incidents substantially increases the sample
size and thereby the statistical power of the analyses. UP provided
data only on incidents attributed in a UP investigation to an error
by a transportation worker. Incidents attributed to physical faults,
such as a rail breaking, were not included in the data. An incident
investigation involves union officials and management represent-
ing all major departments. The FRA audits the investigation process
to check its accuracy, and may conduct its own investigation for
more serious incidents. Thus, the checks and balances are similar
to that in a decertification investigation.
The vast majority of incidents are associated with switching
rather than with CRZ operations, so if CAB-Switching is effective
in reducing occurrences, incidents should decrease, beginning in
the second phase, after the initiation of CAB-Switching.
2.2.3. Analysis strategies for corporate safety data
2.2.3.1. Comparison data. As this is a field study, strict experimental
control is not possible, and there is always the possibility of con-
founds with the safety process being the true cause of changes in
safety. However, there can be an increase in the confidence in
the analysis of the effectiveness of the safety process by
65
performing a quasi-experiment and contrasting the data showing
an apparent safety effect with no-treatment ‘‘comparison data”
that, in theory, should not be affected by the process (Rossi et al.,
1999; Shadish et al., 2002).
The evaluation team worked with UP management to select
comparison data such that the associated operations were similar
to those of the treatment (e.g., type of work, number of workers,
style of management). For this paper, comparison data were
obtained for other service units or yards that did not implement
the safety process at the same time. For CAB–CRZ, decertification
performance at the San Antonio Service Unit was compared with
that at other service units in UP’s Southern Region: Fort Worth,
Houston, and Livonia. According to UP management, all four ser-
vice units shared the following characteristics:
 Approximate size of territory and number of workers.
 Age and tenure of the workers.
 Unions representing the workers.
 Type of work and traffic.
 Corporate and regional organizational culture and manage-
ment, including the same regional vice president.
 Role behaviors of local management regarding safety (manage-
ment at all sites were interested in cooperating with workers
and starting CSA or similar programs, but only San Antonio
implemented CSA for road work during the evaluation period).
 Level of FRA scrutiny (owing to recent high-profile accidents).
 Definitions and reporting procedures for safety occurrences.
In addition, the service units all had approximately the same
baseline level of decertifications, varying between 1.2 and 1.5 per
200,000 worker-hours. San Antonio had the lowest baseline rate,
but the differences among the service units was not significant
(multiple Poisson occurrence rate comparison (Nelson, 1982)
v2(3) = 1.67, p = 0.644). Qualitative data from the implementation
evaluation revealed no unusual events or changes to any of the
sites during the evaluation period (e.g., no catastrophic accidents).
This lack of a difference in changes is most crucial for the validity
quasi-experiment since an effect of treatment is indicated by the
relative performance of the treatment site to the comparison sites
over time. In summary, the comparison sites were a suitable con-
trast for San Antonio.
For CAB-Switching, comparison with the other three service
units in the region was problematic because two of these service
units had similar safety programs for switching that coincided with
CAB-Switching, and therefore they were not used. Instead, compar-
isons within the San Antonio Service Unit were made among the
three locations with the largest yards in the service unit: Eagle
Pass, Laredo (comprising two yards), and the San Antonio Complex
(comprising three yards in the city of San Antonio).
At the end of the evaluation period, different locations had dif-
ferent strengths of implementation of CAB-Switching due to differ-
ent start dates and rates of training. Managers at all three locations
wanted to initiate CAB-Switching as soon as possible—there was
equal interest to improve safety at all yards. However, logistics
forced the different strengths of implementation. Laredo was a
peripheral terminal, requiring travel of the San-Antonio-based
trainers; thus the steering committee deferred its implementation
until the end of the evaluation period, resulting in a weak imple-
mentation at that time. Eagle Pass was also a peripheral terminal,
but had relatively few workers (approximately 20) who were all
trained in just a few visits in under two months at the start of
CAB-Switching. With all workers trained from the beginning of
CAB-Switching, Eagle Pass had the strongest implementation. The
San Antonio Complex required no trainer traveling, but its size
was comparable to Laredo, so San Antonio’s training took much
66 M. Zuschlag et al. / Safety Science 83 (2016) 59–73
longer than Eagle Pass’s. By the end of the evaluation period, about
half its workers at the San Antonio Complex were trained, giving it
an intermediate implementation strength at that time.
In addition to having different number of workers, the three
locations had different baseline occurrence rates (v2(2) = 17.25,
p = 0.0002). The San Antonio Complex had the highest rate
(12.15 incidents per 100,000 car-moves), but this was not signif-
icantly different than the rate at Eagle Pass (9.69, two Poisson
occurrence rate comparison (Nelson, 1982) F(50, 196) = 1.204,
p = 0.188). The rates for the San Antonio Complex and Eagle Pass
were each significantly higher than the rate for Laredo (5.93, F
(98, 196) = 2.008, p < 0.0001, and F(98, 48) = 1.601, p = 0.036,
respectively).
Thus, as a group, the three locations were more similar than dif-
ferent: two out of three were approximately the same size, and two
out of three had approximately the same baseline occurrence rate.
In other areas, all three locations shared the same characteristics as
listed above for the four service units of the Southern Region. In
addition, the three locations shared the same superintendent and
service-unit-level of management. Most crucially, the qualitative
data from the implementation evaluation revealed no unusual
events or changes to any of the locations during the evaluation per-
iod. The three locations were thus suitable for contrasting against
each other.
Data were gathered for the baseline years prior to, in addition to
during, the CAB implementation, constituting a pre-post design
with no-treatment comparisons. With such comparison data avail-
able, the chief statistical analysis is the relative performance of the
treatment and comparison data from baseline through interven-
tion—that is, the presence of a statistical interaction. If the changes
in safety are different for the treatment and comparison data, the
implication is that some factor associated with only the treatment
data, such as the safety process, is specifically affecting the treat-
ment data. On the other hand, if there are no differences in safety
changes, there is a strong possibility that a single factor other than
the treatment is responsible for changes in both treatment and
comparison data.
2.2.3.2. Gap times. The occurrence data were analyzed as gap times
(Cook and Lawless, 2010; Maguire et al., 1952) to determine the
occurrence rate changes associated with CAB. A gap time (also
known as inter-arrival time) is the time between a pair of adjacent
occurrences. ‘‘Time” is expressed in units that represent the site’s
exposure to the risk of an occurrence. In the railroad industry,
the unit of exposure for incidents is typically car-moves (the num-
ber of cars moved through a location), while the unit of exposure
for decertifications is typically worker-hours at the site. Because
the gap times are calculated as the elapsed exposure rather than
calendar time, the data are corrected or effectively normalized
for different levels of exposure at different times and places (e.g.,
different size yards).
The timeline in Fig. 2 shows the gap times between five hypo-
thetical decertifications. For instance, the gap between the April
20th and April 23rd decertifications is 24,000 worker-hours, and
the decertification between the April 23rd and April 28th decerti-
fications is 40,000 worker-hours, and so forth.
Gap times are the mathematical inverse of a rate. For example,
if 24,000 worker-hours were completed since a previous decertifi-
cation, then there was 1 decertification per 24,000 worker-hours
for that time period, equal to a rate of 1/24,000  200,000 or
8.33 decertifications per 200,000 worker-hours.1 Thus, if gap times
increase, then the rate must decrease, and vice versa.
1
The constant of 200,000 is included to be consistent with the railroad convention
of representing rates as roughly the frequency per 100 workers working 40 h a week
for one year.
Analyses of gap times and other statistical analyses of elapsed
time until an event are routine in medical research (Lee and
Wang, 2003; Bradburn et al., 2003) and reliability engineering
(Department of Defense, 1996). For example, mean time between
failures is a familiar statistic from reliability engineering. In this
evaluation, safety occurrences are treated no differently than fail-
ures in a population of replaceable components (Nelson, 2003).
In contrast, analysis of railroad safety data traditionally represents
data points as the occurrence rates for a convenient block of time
(e.g., the monthly decertification rate) rather than gap-times. How-
ever, for relatively sparse frequencies of occurrences, such as found
at a single railroad service unit, analyses of such block-time data
have lower statistical power than analysis of gap times. Block-
time analysis can also invalidate the normality and homoscedastic-
ity assumptions of parametric statistical analyses (Zuschlag et al.,
2012). Gap time data are thus preferred for this data set.
2.2.3.3. Survival analysis. Most inferential statistical analysis
requires that the gap times be statistically independent and iden-
tically distributed. All gap time data were checked for indepen-
dence by calculating the lag-1 and lag-2 autocorrelations (Cohen
et al., 2003). The data were checked for being identically dis-
tributed by inspecting Weibull probability plots (Nelson, 1982).
When gap times are statistically independent and identically
distributed, survival analysis techniques are suitable for inferential
analysis (Cook and Lawless, 2010). Survival analysis comprises a
suite of statistical techniques to analyze the effects of variables
on the times until an occurrence, which may include analysis of
gap times. Survival analysis is preferred over conventional least
squares techniques such as analysis of variance because, compared
to conventional analyses, survival analysis:
 Is more capable of addressing the approximately exponential
distributions typical of gap times.
 Produces more accurate parameter estimates due the use of
maximum likelihood estimation.
 Allows inclusion of a ‘‘right-censored” gap time, being the first
occurrence that occurs after the end of the evaluation period.
This evaluation employs two forms of survival analysis to ascer-
tain the relationship between the CAB safety process and the
occurrences rates: Cox regression and Weibull regression (Lee
and Wang, 2003; Nelson, 1982). The Weibull regression fits a
specific parametric distribution to the data, while the Cox regres-
sion derives a nonparametric distribution from the data.
Like any regression, Cox and Weibull regressions produce coef-
ficients, bi, representing the magnitude of the effect for each pre-
dictor variable. Specifically, exp(bi) is the ratio of the ‘‘hazard” or
chance of an occurrence at any given moment for each integer
increment of the predictor variable (Bradburn et al., 2003). With
a predictor variable representing the presence of CAB, one can cal-
culate the percent change in the chance of the occurrences associ-
ated with CAB (Hosmer et al., 2008).
2.3. Practices and safety-culture survey
A forced-choice survey included measures of practices and
safety culture, specifically self-reported CRZ work practices and
labor–management relations. Table 2 describes the scales, along
with their sources and supporting research, and the outcome
related to each.2
The scales were evaluated for inter-item reliability with use of
Cronbach’s alpha because this is the first time that such scales have
been used in the railroad industry. These scales were added to 11
2
Please contact the first author for more detailed information about the scales.
 M. Zuschlag et al. / Safety Science 83 (2016) 59–73
Table 2
Forced-choice survey scales and their relation to outcomes.
Scale Description
Unsafe behaviors – CRZ Self-reported extent to which employees
follow CRZ rules and proceduresa
Labor–management relations Perceived trust and friendliness between labor
and management, with particular attention to safety
a
Conceptually based on Hofmann and Stetzer (1996). Managers reported on the practices of their subordinates, whereas workers reported for themselves. Yard employees
skipped this scale.
Table 3
Numbers of respondents for the CRZ scale for each phase and respondent type.
Respondent typea
Phase Workers
First 179 (19%)
Second 86 (9%)
a
Return rates are in parentheses.
proprietary scales in an organizational climate survey used by BST
for evaluating its implementation sites. Service-unit management
distributed the survey by mail to workers and managers on two
occasions: early in the first phase, in December 2005, and after
the end of the second phase, in April 2008. Table 3 lists numbers
of respondents and return rates. Qualitative data suggested that
the lower worker response rate for the second phase was due the
new superintendent (see Section 1.3.3) coordinating less with the
unions on administering the survey than his predecessor (while
continuing to strongly support CAB). The distributions of the self-
reported years of service and ages of the respondents from each
administration were compared with the actual respective distribu-
tions from data provided by the service unit’s human resources
department. There were no significant differences, implying that
respondents from each administration were representative of the
workers and managers at the site.
If CAB has an effect as predicted, improvements in average scale
scores should be seen from the first to the second phase.
2.4. Interviews
Interviews provided qualitative data concerning perceptions of
the outcomes (Patton, 2002). In this evaluation the interviews
served primarily to illuminate the process of any observed quanti-
tative results. Respondents were asked about outcomes attributed
to CAB and probed on the role CAB played in any reported changes
to safety and labor–management relations.3 The interviews were
confidential and were conducted one interviewee at a time. To allow
comparisons as CAB was implemented, interviews were conducted
approximately annually, for a total of three times: in the winters
of 2005 and 2006 (‘‘initial” interviews), 2006 and 2007 (‘‘midterm”),
and 2007 and 2008 (‘‘final”). The interviews were open-ended and
semistructured; respondents were allowed to reply freely, and
3
An interview guide may be obtained by contacting the first author.
67
Fig. 2. Example of gap times.
No. of items Item source Outcome
6 UP code of operating rules, CRZ practices
checked by UP subject-matter experts
6 Dastmalchian et al. (1989) Safety culture
questions changed to follow each respondent’s unique thoughts.
Thus, respondents were not all asked the same questions.
To obtain the richest information on outcomes, purposive sam-
Managers
pling was used to identify the interviewees (Crano and Brewer,
2002). That is, sampling was deliberately weighted toward select-
16 (32%)
26 (52%)
ing respondents with rich knowledge to provide insight. The
respondents therefore tended to be those very knowledgeable
about CAB and the workforce reactions to it. Nonetheless, efforts
were made to include representatives of various viewpoints. Inter-
viewees included workers and managers, BLET and UTU members,
yard and road workers, and workers with various levels of involve-
ment in CAB, such as steering committee members and workers
trained or not trained in CAB. Participants included respondents
who were both supportive and skeptical of CAB.
While representative sampling is preferred for answering nar-
row research questions, purposive is preferred for obtaining broad
insight about a phenomenon (Crano and Brewer, 2002; Patton,
2002). As mentioned above the interviews served to illuminate
the processes responsible for any observed quantitative outcomes,
therefore purposive was more appropriate.
Table 4 shows the types and numbers of respondents. Facilita-
tors and the local management sponsor were interviewed on all
three occasions. The top three rows represent labor members.
Therefore at the initial interviews there were 14 employees and
five managers. CAB facilitators and top management helped to
identify prospective respondents, selecting people who were
respected, credible, and neutral rather than pro-labor or pro-
management. Other than those directly involved in CAB, the inter-
viewees were different for each phase. Stakeholder involvement
throughout the lifecycle of the program, such as through annual
interviews and participation in respondent selection, has been
shown to be a critical factor in meaningful use of evaluation find-
ings (Johnson et al., 2009).
The evaluation team analyzed interview data by breaking the
responses into comments, then identifying and sorting the com-
ments into themes representing the most frequent comments. To
accomplish this two or three team members sorted comments sep-
arately into themes and then reviewed each other’s work to arrive
at a consensus on the themes (Miles and Huberman, 1994; Patton,
2002).
Comparing interview data across the evaluation period pro-
vided detailed insight into changes in management practices, sys-
temic conditions, and safety culture, the latter also measured by
the forced-choice survey. For this paper, safety culture is defined
to include safety-related norms, competencies, beliefs, values,
68 M. Zuschlag et al. / Safety Science 83 (2016) 59–73

Table 4 to the monthly percentages. Following the trend for CAB–CRZ

Respondents to periodic interviews included 19 at initial, then 18, then 16 at final. scores, PPF sessions by CAB peer observers showed a decreasing
Respondents No. of respondents tendency for at-risk switching practices from the beginning of
Initial (Winter Midterm (Winter Final (Winter of CAB-Switching (r = 0.569, n = 17, p = 0.0157). Based on a linear
of 2005–2006) of 2006–2007) 2007–2008) regression of these data, the percentage of at-risk practices
CAB steering 4 4 4
decreased, from 5.13% in August 2005 to 1.05% in January 2008.
committee In other words, the rate of at-risk practices at the end of the eval-
members uation period was approximately one-fifth that at the start of CAB-
Trained CAB peer 5 4 3 Switching.
observers
Workers untrained 5 3 3
in CAB 3.1.2. Survey: Unsafe Behaviors–CRZ scale
The Unsafe Behaviors–CRZ scale had a Cronbach’s alpha of
Management
Frontline/ 3 3 4 0.837, indicating acceptable reliability (Rosenthal and Rosnow,
middle 1991).
Upper 2 2 1 A two-by-two analysis of variance (ANOVA) evaluated changes
Corporate/ 0 2 1
in unsafe CRZ behaviors, with respondent type (worker and man-
regional
ager) and phase (first and second) as independent variables. The
Total 19 18 16
reported use of CRZ practices was significantly more frequent for
the second phase (M = 3.988) than for the first phase (M = 3.749,
F(1, 294) = 5.739, p = 0.017). There was no significant effect of
respondent type (F(1, 294) = 0.049, p = 0.825) or interaction
between respondent type and phase (F(1, 294) = 1.079, p = 0.300),
indicating that workers and managers saw the same degree of
improvement in workers’ behavior.

3.1.3. Summary findings for worker practices

Evidence from both the feedback session data and the survey
indicate that the worker practices targeted by CAB improved with
the introduction of the process. Feedback session data suggest that
at-risk practices fell to a fraction of their rates at the beginning of
the implementation. Survey data indicate that workers and man-
agers both saw these improvements.

3.2. Occurrences

Fig. 3. Monthly percent at-risk scores for CAB–CRZ and CAB-Switching feedback
sessions.
attitudes, relations, and interaction patterns (Reason, 1997). Thus,
interview responses relevant to changes in any of these, singly or in
combination, were used to explain the process of safety culture
change under CSA.
3. Findings
3.1. Worker practices
3.1.1. Feedback session data
3.1.1.1. CAB–CRZ. Fig. 3 depicts the average at-risk scores for CAB–
CRZ feedback sessions between August 2005 and December 2007.
The dark blue straight line represents the least-squares best-fit lin-
ear relations to the monthly percentages.
Throughout the evaluation period, PPF sessions by CAB peer
observers showed decreasing tendency for at-risk CRZ practices
(r = –0.797, n = 29, p < 0.0001). Based on a linear regression of these
data, the percentage of at-risk practices decreased, from 7.14% in
August 2005 to 1.05% in December 2007. In other words, the rate
of at-risk practices at the end of the evaluation period was less
than one-sixth the rate at the beginning.
3.1.1.2. CAB-Switching. Fig. 3 shows average at-risk scores for CAB-
Switching feedback sessions from the start of CAB-Switching in
October 2006 through December 2007, with the light magenta
straight line represent the least-squares best-fit linear relations
The 1-lag and 2-lag autocorrelations for the gap times of decer-
tifications and incidents were all low (|r| < 0.2) and on average not
significantly different from zero, indicating that gap times appear
to be adequately independent for inferential statistical analysis.
Weibull probability plots indicated the data were from a single dis-
tribution (Nelson, 1982). Gap times were thus suitable for survival
analysis.
3.2.1. Decertifications
To evaluate the outcome of CAB–CRZ on occurrences, the San
Antonio Service Unit was compared to the other three Southern
Region service units combined. A Cox regression was performed
on the gap times of CRZ-related decertifications since the start of
the first phase. Predictor variables were date, service unit (San
Antonio versus the others), and the date-by-service-unit interac-
tion. This regression found a significant interaction (v2(1) = 4.68,
p = 0.030), indicating that, as time progressed since the start of
CAB–CRZ, the San Antonio rate of decertifications changed signifi-
cantly more than the other service units.
Table 5 shows the results of Cox regressions of date on the
decertification gap time for each service unit separately. The per-
cent changes in chance of decertifications were calculated from
the regression coefficients. A positive percent represents a percent
increase in the chance of a decertification at any given moment
while a negative percent represents a percent decrease in the
chance.
The San Antonio Service Unit, the site that implemented CAB,
showed a significant 79-% decrease in chance of decertifications
through the first and second phases, from September 2005 (when
CAB started) through the end of available data in February 2008.
The other three service units of the Southern Region (Fort Worth,
 M. Zuschlag et al. / Safety Science 83 (2016) 59–73
Table 5
Cox regressions of date on gap times of decertifications for phases and each service
unit.
Phase Statistical Service unit
value
San Fort Houston
Antonio worth
Baseline Change in 0% 13% 5%
chance
n 45 50 54
p 0.997 0.817 0.924
First and Change in 79% 0% 22%
second chance
n 42 72 74
p 0.014 1.000 0.543
Houston, and Livonia) showed no significant change in the chance
of decertifications. During the four-year baseline period, no service
unit showed a significant trend in decertifications.
3.2.2. Incidents
To evaluate the outcome of CAB-Switching on occurrences, the
major yards of the San Antonio Service Unit were divided into
three locations, each corresponding to a different strength of
implementation due to different start dates and rates of CAB-
Switching training:
 Eagle Pass: a yard with a strong CAB implementation.
 San Antonio Complex: three yards with moderately strong
implementations.
 Laredo: a pair of yards with relatively weak implementations.
Fig. 4 shows the incident rates at the three locations during the
Second Phase and before the Second Phase (Baseline and First
Phase).
A Weibull regression was performed on the gap times of inci-
dents. Predictor variables were phase (baseline and first phase ver-
sus second phase), location, and the phase-by-location interaction.
This regression found a significant location by phase interaction,
indicating that the three yard locations had different changes in
rates (v2(2) = 6.16, p = 0.046). Both Eagle Pass and the San Antonio
Complex had significant decreases in their incident rates (v2(1)
= 7.54, p = 0.006, and v2(1) = 4.13, p = 0.042, respectively), while
Laredo, which had the weakest implementation, did not signifi-
cantly change (v2(1) = 0.01, p = 0.909). Eagle Pass, which had the
strongest implementation, experienced an 81-% drop in the rate
of incidents over a 1-year period, which was significantly greater
than the modest 32-% drop at the San Antonio Complex (v2(1)
= 4.19, p = 0.041), and also significantly greater than the change
at Laredo (v2(1) = 6.06, p = 0.014). The change at the San Antonio
Complex was not significantly different from the change at Laredo
(v2(1) = 1.20, p = 0.2733).
In baseline and the first phase, the incident rate for Eagle Pass
and the San Antonio complex were not significantly different from
each other (F(50, 196) = 1.204, p = 0.188), but were each signifi-
cantly higher than the rate of Laredo (F(98, 48) = 1.601, p = 0.036
and F(98, 196) = 2.008, p < 0.0001). In the second phase, the rate
for Eagle Pass was significantly lower than the rates of Laredo (F
(8, 40) = 2.186, p = 0.049) and San Antonio (F(8, 74) = 3.097,
p = 0.0045), while San Antonio and Laredo were not significantly
different (F(44, 92) = 1.349, p = 0.129).
Prior to the second phase, no location showed a significant
trend in incident rates, except that Laredo had an unexplained
spike in incidents straddling the transition between the first and
second phase. Fisher exact tests of the frequencies of reportable
and non-reportable incidents found no significant change in the
69
Livonia
83%
37
0.965
193%
37 Fig. 4. Rates of human factors incidents per 100,000 car moves at site yards. Dashed
0.109 lines represent no significant difference.
proportion of reportable and non-reportable incidents for any site
since the start of CAB-Switching (lowest p = 0.259). There is no evi-
dence that the rate of the more costly incidents changed differently
than the rate of the less costly incidents.
3.2.3. Summary findings for occurrences
Occurrences decreased in the presence of CSA. From the begin-
ning of CAB–CRZ until the end of the evaluation period, there was a
gradual decline in the chance of engineer decertifications most clo-
sely related to the practices that CAB–CRZ promoted. This decline
appears to have begun at the same time as CAB–CRZ rather than
having continued a trend preceding it. Other service units of the
Southern Region, which had no CSA process for CRZ practices, did
not experience a decline, implying that the cause was unique to
the San Antonio Service Unit, the only unit with a CAB–CRZ pro-
cess. The decrease implies a reduction in serious safety violations,
such as passing a stop signal, which are often precursors to catas-
trophic road accidents. Thus, a reduced chance of decertification
implies a reduced probability of such accidents.
Human factors incident rates dropped significantly at Eagle Pass
and the San Antonio Complex after CAB-Switching began at those
locations. Rates did not significantly change at Laredo, which did
not start CAB-Switching until the final months of the second phase.
The decreases in incident rates were not the product of a general
improving trend that began prior to CAB-Switching. While the
improvement at Eagle Pass was significantly better than that of
the San Antonio Complex and Laredo, the improvement of the
San Antonio Complex was not significantly better than that of Lar-
edo. This lack of significance is not surprising given the relatively
short evaluation period (about one year) for CAB-Switching.
For both decertifications and incidents, the chance of occur-
rences associated with CSA fell to about a fifth of that before imple-
mentation of CSA.
3.3. Safety culture
In this section quantitative survey results are followed by the
related interview data to illuminate the apparent processes behind
the scale score changes.
3.3.1. Survey: Labor–Management Relations scale
The Labor–Management Relations scale had a Cronbach’s alpha
of 0.809, indicating modest reliability.
A two-by-two ANOVA evaluated changes in the Labor–Manage-
ment Relations scale, with respondent type (worker and manager)
and phase (first and second) as independent variables. The analysis
showed that labor–management trust was significantly better in
the second phase (M = 2.881) than in the first phase (M = 2.324, F
(1, 303) = 11.123, p = 0.001) for both workers and managers. The
70 M. Zuschlag et al. / Safety Science 83 (2016) 59–73
mean improvement on the scale was not insubstantial (accounting
for 10% of the variance), but remained below the 3.0 midpoint of
the 5-point Likert response stem used by the scale. This suggests
that while safety culture improved, some labor–management
issues remained. Managers generally saw relations as being signif-
icantly better than did workers for both phases (Ms = 3.472 com-
pared with 2.378, F(1, 303) = 56.296, p < 0.001). This is a
commonly found difference in perspective that may reflect labor
and management’s different bargaining roles (K. Bell, BST, personal
communication, August 26, 2013; M. Mangan, BST, personal com-
munication, August 27, 2013). There was no significant interaction
between respondent type and phase (F(1, 303) = 0.050, p = 0.823),
indicating that workers and managers saw the same degree of
change, and that there was no change in the gap between workers
and managers.
3.3.2. Interviews: Labor–Management Relations
In this section concerning the interviews, italic text designates
themes from the qualitative analysis of the interviews. All inter-
view quotes shown in this paper were representative of typical
responses for the corresponding theme, rather than unusual or
extreme responses. These interview themes suggest changes in
attitudes and behavioral patterns related to trust producing
changes in commitment, style, and proficiency in ensuring safety
(Reason, 1997).
From the initial interviews through the midterm and final inter-
views, respondents reported improvements in relations between
labor and management, echoing the quantitative results found by
the survey. The interviews suggested some factors that may have
contributed to the improvements in labor–management relations
indicated by the survey.
In the initial interviews, workers were asked to describe their
chief safety concerns. Among the top concerns were the conditions
of facilities and equipment, as indicated by the worker quoted
below:4
Worker, Initial Interviews
There is debris all over the yard, the gondola cars are overfull,
and then there are gigantic pieces of metal lying around. There
are holes two feet deep from work done by Maintenance of
Way. You could break a leg if you fell in one of these holes.
Consistent air conditioning for the train cabs was also men-
tioned as an equipment problem:
Worker, Initial Interviews
AC [air conditioning] in cabs – this is a problem. It has
improved, but not to where it should be. . .. There is a big differ-
ence when one gets off a train with AC than one without it. Lots
of the fleet are not equipped.
Management is responsible for conditions of facilities and
equipment. Thus, workers’ safety concerns implied they perceived
poor organization support for safety. Workers felt management
practices inadequately promoted safety.
As part of CAB, the barrier removal team identified barriers
related to the conditions of facilities and equipment, which man-
agement removed. Workers appear to have noticed the difference.
Most workers and managers reported improvements in facilities
and equipment since the initial interviews. Locomotive air condi-
tioning, yard clean-up, and building repair in particular were
mentioned:
4
The other top concern was fatigue, especially as related to the work schedule.
Workers would later report improvements regarding this issue, thanks to more
reliable train arrival information, but these advances could not be directly linked to
the CAB process.
Worker, Final Interviews
I have seen a lot of improvements in the facilities, in providing
employees with more stuff, such as locker rooms, more stuff in
the engine, such as AC in it. That has totally changed in the last
three years. I’ve seen the facilities are now clean.
Management practices such as barrier removal may have con-
tributed to improving safety at the site. However the change in
management practices likely had the additional effect of improving
worker perceptions of management. In the interviews, workers
(and managers) reported seeing that management commitment
to safety had improved since the baseline period:
Worker, Initial Interviews
Management has a great safety attitude if it benefits them. They
push cars through even if they are not safe. Then they strong-
arm employees with safety when they want to. All management
at all levels is the same.
Worker, Final Interviews
I see that 80 percent management have commitment . . .but . . .
lower management can only execute what upper management
says.
As the worker’s quote above from the final interviews indicates,
commitment may have been improved, but workers still see issues
in translating that commitment into action. This may account for
labor–management relations improving yet still remaining rela-
tively low as measured by the survey.
Management’s commitment toward safety implies that rules
are enforced consistently and that managers can be relied on for
safety. Some workers and many managers reported greater fairness
by management and greater trust between workers and managers, an
improvement in behavioral patterns since the initial interviews:
Worker, Final Interviews
I have a lot more trust with my managers [than I did a few years
ago]. I can go to all of them. I know that a lot of the older guys
still don’t trust managers. I know that my managers want me to
work safe.
Managers reciprocated the changes in worker perceptions:
Manager, Final Interviews
When I first started, the mentality between managers and trainmen
was that [managers] considered trainmen as oxen that have to be
whipped to get them going. . .. Over the years, management’s
thoughts have changed. The workers are smart people and have
smart things to say, too. . .. There are things that they could teach me.
With workers seeing managers as having greater commitment
to safety, and workers and managers each trusting the other more,
it is understandable that labor–management relations scale scores
would improve.
Greater mutual trust facilitates more communication and coop-
eration (Reason, 2003). Furthermore, the joint labor–management
barrier removal team provided a venue to exercise communication
and cooperation. Consistent with these factors, respondents
reported improved communication and cooperation between
management and labor on safety:
Worker, Initial Interviews
A worker was told to pull 100-plus cars out under certain con-
ditions that broke the train in two. It was a manager’s idea to
not cut away a smaller set. Could have avoided it but he
wouldn’t listen to the worker’s idea.
Worker, Final Interviews
I have experienced one incident myself, where managers
approached me after letting them know about problems in
 M. Zuschlag et al. / Safety Science 83 (2016) 59–73
getting a switch lined up. The four managers went there to
investigate; they talked to us and said thank you for bringing
it to our attention
Such openness in turn was associated with more reports of
workers initiating communication with managers:
Manager, Final Interviews
It used to be that the employees wouldn’t even talk to a man-
ager without their union local chair, but now I can deal with
the employees directly. Before, they wouldn’t talk to us even
on the smallest issue, but now the conversations are more open
and frequent.
Manager, Final Interviews
I think CAB has opened up the relationship between manage-
ment and workers. It may be the one avenue that opened doors
that hadn’t been available before.
3.3.3. Summary findings for safety culture
Quantitative survey data indicate that labor–management rela-
tions improved with a successful CSA implementation. Consistent
with previous research (see DeJoy, 2005), qualitative data suggests
that the increase in the labor–management relations scale score
could be related to the apparent improvements in safety commu-
nication and cooperative behavioral patterns mentioned in inter-
views. These improvements were, in turn, related to the
improved perceived management commitment to safety that was
associated with the barrier removal identification subsequent
management improvement of facilities and equipment. CSA thus
may improve labor–management trust by providing a means for
ongoing cooperation and communication. Specifically, the
barrier-removal process provides sustainable opportunities for
workers and managers to address safety through cooperation
rather than bargaining. The quantitatively measured improvement
in labor–management relations, bolstered by the explanatory
power of the qualitative analysis, indicate a shift in the safety cul-
ture toward a ‘‘trust culture” (Reason, 2003), a key feature of an
effective safety culture that is traditionally lacking in the railroad
industry (Coplen, 1999). The improvement in labor–management
relations is consistent with predictions for safety processes like
CSA that feature a nondisciplinary, proactive, cooperative,
systems-safety-analysis orientation (Reason, 1997).
4. Discussion
4.1. Outcome summary
The theory of change, presented in Section 1.2.1., usefully
guided the outcomes to measure. Quantitative results from process
metrics, surveys, and corporate safety data, depicted on Fig. 1, all
point toward safety-related improvements from implementing
CSA. Interview data did not contradict the quantitative findings,
but rather provided consistent explanations for the quantitative
findings. To summarize:
 Worker practices targeted by the CSA process improved with
the program’s introduction. Process metrics indicated that the
commission of at-risk behaviors dropped by about 80%.
 The chance of engineer decertification gradually declined by
79% after the introduction of CSA at the service unit, whereas
the chance of decertification at comparison service units did
not decline.
 The rate of incidents of material damage dropped by 81% at a
yard with a strong CSA implementation, and by 32% at yards
with a moderately strong implementation, whereas there was
71
no significant rate change at yards with a relatively weak CSA
implementation.
 Labor–management relations, specifically trust, improved at the
demonstration site, indicating the development of a more effec-
tive safety culture.
In summary, CAB has demonstrated the potential for CSA to
improve safety and safety culture in railroad transportation
departments.
4.2. Evaluation limits
4.2.1. Context of the CAB process
Despite the comprehensive nature of the evaluation, there are
some limitations to take into account when drawing conclusions.
The findings were from a specific service unit at a specific point
in history. This site had the following features, which may be rele-
vant to the generalizability of the results:
 The number of employees at the treatment site ranged from 20
(for CAB-Switching) to 1100 (for CAB–CRZ).
 The pilot was conducted on a US freight railroad, which has cer-
tain operational and organizational characteristics (e.g., two
crew members in a CAB, which allows them to observe each
other; FELA, which normally discourages labor–management
trust).
 The US freight railroad industry in general was experiencing
growth, owing to increase demands for coal (McPhee, 2005).
 Labor and management were relatively motivated to cooperate
on safety due to recent high-profile accidents and an influx of
new workers, although this was counteracted by a long history
of labor–management distrust and a US legal system that pro-
vides incentives to clash over safety (Zuschlag et al., 2012).
 Top managers were inclined to empower employees with con-
trol over the safety process, even when the process encountered
difficulties.
 There were no unusual changes in leadership, organizational
structure, or other highly disruptive events.
 The baseline level of safety was relatively high, consistent with
the theoretical potential of CSA to move railroads to the next
level of safety.
Whether or not the same results can be observed at other sites
and times depends on the similarity of the sites, the implementa-
tion support, the management’s role behavior and motivation, the
level of success in adapting the CSA process to conditions at the
locality, and the selection of safety measures that are appropriate
for the evaluation (e.g., decertifications, rather than catastrophic
road accidents) and resistant to over-reporting or under-
reporting. Similar results might be more likely at similar US freight
sites than at passenger or freight sites outside the US.
4.2.2. Rigor and confounds
As a field study that could not randomly assign participants to
conditions, this evaluation nonetheless achieved a high degree of
rigor through the use of mixed research methods, pre-post quasi-
experimental design with no-treatment comparison, and systematic
employment of a theory of change (GAO, 2009). It is nearly inevitable
that confounds or events will occur simultaneously with safety-
process implementation, rendering ambiguous the apparent cause
of any observed safety improvements. In the case of CAB, however,
regional, corporate, or industry-wide confounds with time are
implausible alternative explanations for improvements in occur-
rences because such improvements would also be observed at com-
parison sites. The treatment and comparisons sites were similar in
size, initial safety level, type of work, organizations (for both labor
72 M. Zuschlag et al. / Safety Science 83 (2016) 59–73
and management), worker tenure, leadership and leadership role
behaviors, motivation for safety, regulator oversight, and definitions
of safety occurrences. Yet, this evaluation found safety occurrence
improvements uniquely at the treatment sites.
This leaves local confounds as alternative explanations for
safety improvements at the demonstration site. The full report of
this demonstration (Zuschlag et al., 2012) documents all relevant
confounds and analyzes their potential as alternative explanations.
However, any field setting has multiple factors occurring and influ-
encing each other, and it is ultimately misleading to search for one
feature that is responsible for all change. Anyone interested in
replicating the outcomes of CAB at another site ought to appraise
the unfolding events at San Antonio, from the initial context
through the constant evolution of CAB, and consider how to
achieve or exploit equivalent conditions and processes at the
intended site. For example, CAB likely benefited from the formative
evaluation, where the ongoing evaluation was used to improve
CAB’s implementation. Thus, to maximize the chances of successful
implementation, CSA processes should include equivalent activi-
ties, specifically, stakeholder involvement in:
 Collection and analysis of data on implementation performance
and outcomes.
 Feedback on implementation performance to improve the over-
all implementation.5
4.2.3. Implications of baseline occurrence rates
It is natural for safety interventions to be applied first at sites
where safety is worst. However, confounding an intervention with
the worst baseline safety complicates the interpretation of the
results of a pre-post quasi-experiment, such as used in this evalu-
ation. The treatment site may have the worst safety at baseline for
multiple reasons, including due to some unusual random transi-
tory factors. Thus, improvements may appear at follow-up due to
regression to the mean when some of the unusual transitory fac-
tors are probabilistically likely to become absent (Rosenthal and
Rosnow, 1991).
Also, organizational reactions to a poor safety record can compli-
cate interpretations. Precisely because the treatment site has the
worst safety record, organizations may formally or informally initi-
ate multiple simultaneous interventions on the site resulting in con-
founds with the intervention. For example, in addition to initiating
CSA, a manager may increase the frequency of operational safety
testing of employees, or the FRA may conduct more thorough safety
inspections. More insidiously, organizational reactions can yield an
illusory improvement: a manager with the worst safety record may
be under pressure to improve the apparent record, which may trans-
late into under-reporting (Pedersen et al., 2012).
None of these complications were plausibly operating for the
occurrence results reported in this paper. The safety measures in
this evaluation, decertifications and incidents of material damage
(mostly derailments), are resistant to under-reporting, unlike
minor injuries. Qualitative data collected as part of the implemen-
tation evaluation indicated that the only safety interventions at the
treatment sites were those connected to CAB. In the case of decer-
tifications, the treatment site, which was the San Antonio Service
Unit, had approximately the same chance of decertifications at
baseline as the comparisons sites. Qualitatively, the treatment
and comparison sites were under the same FRA scrutiny following
some high-profile accidents, and management at all sites were
interested in improving safety (e.g., two out three comparison sites
started CSA-related processes for their yards during the evaluation
period). Thus, improvements in decertifications at the treatment
5
BST’s process included precisely these activities, and the evaluators’ formative
activities are best regarded as a supplement to what was happening anyway.
site cannot be attributed to either regression to the mean or
organizationally-induced confounds. It is most plausibly attributed
to CAB–CRZ, which was unique to the San Antonio Service Unit.
In the case of incidents, Eagle Pass showed the greatest improve-
ment but at baseline its occurrence rate was significantly higher
than the comparison site Laredo, which may cast suspicion on the
results. However, the rate for Eagle Pass at baseline was not signifi-
cantly different than the rate for the San Antonio Complex, yet Eagle
Pass had a significantly lower rate than the San Antonio Complex at
follow-up. If there were regression to the mean or organizational
confounds, then the San Antonio Complex should have improved
at least as much as Eagle Pass. Indeed, because the San Antonio Com-
plex is larger, it should arguably have had more organizational moti-
vation to improve its safety record than Eagle Pass because the San
Antonio Complex had a greater raw number of incidents (in fact,
qualitative data indicated that all yards in the service unit were
motivated to improve safety). Finally, with regard specifically to
regression to the mean, Eagle Pass had the lowest occurrence rate
at follow-up –significantly lower than Laredo. Its occurrence rate
did not merely approach the mean over time; it fell significantly
below the mean. This is inconsistent with regression to the mean.
The most plausible and parsimonious interpretation is that increas-
ingly strong CAB-Switching implementations led to greater
improvements in incident rates.
4.3. Conclusions and recommendations
This demonstration pilot shows the potential for CSA to
improve safety performance and culture in the US freight railroad
industry. However, safety programs cannot be expected to have
impact unless they are implemented successfully. This article
focused on the impact of the Union Pacific demonstration pilot at
San Antonio, Texas, while a separate report covers the substantial
amount of data collected and analyzed concerning the strength
of the implementation (Zuschlag et al., 2012). However, questions
still remain. Therefore, future research could fruitfully focus on
explaining the key elements of a strong implementation to help
managers learn how to leverage the money spent on safety.
The FRA is committed to finding and promoting new safety
approaches that reduce risk through focusing on unsafe conditions
and practices that might lead to occurrences rather than taking the
traditional approach of focusing on occurrences themselves (FRA,
2008). CSA is one such risk-reduction method that is apparently
workable for the railroad industry.
Furthermore, CSA appears to have the ability to transform the
safety culture away from the characteristics that limit safety
improvements and toward proactive, non-disciplinary, cooperative,
systems-safety approaches. Indeed, there is reason to believe that
such transformation is underway beyond the demonstration site.
Based on the success of CAB in San Antonio and TSC in UP’s mechan-
ical departments, UP elected to expand its TSC safety process to the
transportation departments at all of its locations (Grimaila, 2007).
The success of CSA at UP and similar pilot implementations also
led to the development and adoption of the FRA’s Railroad Safety
Risk Reduction Program in the Rail Safety Improvement Act
(2008). These pilots also encouraged the implementation of similar
safety-culture programs by other carriers, including Amtrak, Tor-
onto Transit, and BNSF.
Acknowledgments
This evaluation would not have been possible without the coop-
eration of a large number of managers and employees at Union
Pacific Railroad.
The authors would especially like to thank Brian Gorton,
Michael Byars, and Kelvin Phillips for their considerable assistance,
 M. Zuschlag et al. / Safety Science 83 (2016) 59–73
along with Lance Fritz, Joe Santamaria, Roby Brown, Michael
Mitchell, Mark Barnum, Ted Lewis, Shane Keller, Ronald Tindall,
Greg Burger, John Dunn, Russell Elley, Mike Araujo, Paul Dillon,
Carl Eddington, José Gutierez, Wil Hardiman, Chad Jistel, Oscar
‘‘Doctor” Mayfield, Fernando Nanez, Pat Pino, Martin Vacca, Mario
Valadez, John Paul Schuster, and Andy Wright. Thanks also to
George Wollard and Jay Finney of Behavioral Science Technology
Inc. (BST) for providing education and insights into the implemen-
tation of methods like Clear Signal for Action in the railroad indus-
try. Along with Kelly Johnson and others at BST, George also
gathered data for us from a survey customized to our require-
ments. Jonny Morell from Fulcrum Corporation and Demetra Collia
from the Bureau of Transportation Statistics provided additional
technical assistance, and Christopher Nelson of the RAND Corpora-
tion provided insight into realist program evaluation. Special
thanks to Wayne Nelson for instructing and advising on gap time
analysis and survival analysis. Shuang Wu of Computer Sciences
Corporation assisted in the processing and analyses of the data.
Katherine Blythe, Alison Stieber, Cassandra Oxley, and other
MacroSys staff provided editorial assistance.
The work was performed under an interagency agreement
between the FRA Human Factors Division and the Office of Safety
Management and Human Factors of the US Department of Trans-
portation John A. Volpe National Transportation Systems Center.
References
Bradburn, M.J., Clark, T.G., Love, S.B., Altman, D.G., 2003. Survival analysis Part III:
Basic concepts and first analyses. Br. J. Cancer 89, 605–611.
Cohen, J., Cohen, P., West, S.G., Aiken, L.S., 2003. Applied Multiple Regression/
Correlation Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates,
Mahwah, NJ.
Cook, R.J., Lawless, J.F., 2010. The Statistical Analysis of Recurrent Events. Springer
Science + Business Media, New York, NY.
Coplen, M.K., 1999. Compliance with Railroad Operating Rules and Corporate
Culture Influences. Results of a Focus Group and Structured Interviews. Federal
Railroad Administration, DOT/FRA/ORD-99/09, DOT-VNTSC-FRA-97-7.
Coplen, M.K., Ranney, J., 2009. Safe Practices, Operating Rule Compliance, and
Derailment Rates Improve at Union Pacific Yards with STEEL Process – A Risk
Reduction Approach to Safety. Research Results. Federal Railroad
Administration, RR09-08. <http://www.fra.dot.gov/eLib/Details/L04248>
(10.10.14).
Crano, W.D., Brewer, M.B., 2002. Principles and Methods of Social Research, second
ed. Lawrence Erlbaum Associates, Mahwah, NJ.
Creswell, J.W., 2003. Research Design: Qualitative, Quantitative, and Mixed
Methods Approaches, second ed. Sage, Thousand Oaks, CA.
Dastmalchian, A., Blyton, P., Adamson, R., 1989. Industrial relations climate: testing
a construct. J. Occup. Psychol. 62, 21–32.
Dejoy, D.M., 2005. Behavior change versus culture change: divergent approaches to
managing workplace safety. Saf. Sci. 43, 105–129.
Deming, W.E., 2000. Out of the Crisis. MIT Press, Cambridge, MA.
Department of Defense, 1996. Handbook for Reliability Test Methods, Plans, and
Environments for Engineering, Development, Qualification, and Production.
MIL-HDBK-781A.
Egan, M., Bambra, C., Petticrew, M., Whitehead, M., 2009. Reviewing evidence on
complex social interventions: appraising implementation in systematic reviews
of the health effects of organisational-level workplace interventions. J.
Epidemiol. Community Health 63 (1), 4–11.
Federal Railroad Administration, 2001. Switching Operations Fatality Analysis.
Severe Injuries to Train and Engine Service Employees: Data Description and
Injury Characteristics. Switching Operations Fatality Analysis (SOFA) Working
Group. <www.fra.dot.gov/downloads/safety/sofa/SOFA_Injury.pdf> (05.01.11).
Federal Railroad Administration, 2006. Railroad Safety Statistics 2005 Annual
Report. FRA Office of Public Affairs. <http://safetydata.fra.dot.gov/officeofsafety/
ProcessFile.aspx?doc=bull2005-book.exe> (May, 2011).
Federal Railroad Administration, 2011. FRA guide for preparing accident/incident
reports. FRA Office of Safety, DOT/FRA/RRS-22. <http://safetydata.fra.dot.gov/
OfficeofSafety/ProcessFile.aspx?doc=
FRAGuideforPreparingAccIncReportspubMay2011.pdf> (July, 2013).
Federal Railroad Administration, 2008. The FRA Risk Reduction Program: A New
Approach for Managing Railroad Safety. White Paper. FRA Office of Research and
Development Division, Human Factors Program. <http://www.
fra.dot.gov/downloads/safety/ANewApproachforManagingRRSafety.pdf>
(05.01.11).
Frederick, J., Lessin, N., 2000. Blame the worker: the rise of behavioral-based safety
programs. Multinat. Monit. 21 (11), 10–17.
Funnel, S.C., Rogers, P.J., 2011. Purposeful Program Theory: Effective Use of Theories
of Change and Logic Models. Jossey-Bass, San Francisco.
73
Gamst, F.C., 1982. The development of operating rules. Proc. Railway Fuel Operat.
Officers Assoc. 46 (155), 162–171.
Geller, E.S., 2001. The Psychology of Safety Handbook. CRC Press, Boca Raton, LA.
Grimaila, R., 2007. Testimony Before U.S. House of Representatives Committee on
Transportation and Infrastructure, October 25, 2007.
Harrington, J.J., 1987. The Improvement Process: How America’s Leading Companies
Improve Quality. McGraw-Hill, New York, NY.
Hofmann, D.A., Stetzer, A., 1996. A cross-level investigation of factors influencing
unsafe behaviors and accidents. Pers. Psychol. 49 (2), 307–339.
Hosmer, D.W., Lemeshow, S., May, S., 2008. Applied Survival Analysis: Regression
Modeling of Time-to-event Data, secon ed. John Wiley & Sons, Hoboken, NJ.
Howe, J., 1999. Debunking behavior based safety. Occupational Health & Safety.
Newslett. UAW Health Saf. Department 1 (5).
Juran, J.M., 1964. Managerial Breakthrough: A New Concept for the Manager’s Job.
McGraw-Hill, New York, NY.
Krause, T.R., 1995. Employee-Driven Systems for Safe Behavior. Integrating
Behavioral and Statistical Methodologies. Van Nostrand Reinhold, New York, NY.
Krause, T.R., 1997. The Behavior-Based Safety Process. Managing Involvement for an
Injury-Free Culture, second ed. John Wiley & Sons, New York, NY.
Krause, T.R., Seymour, K.J., Sloat, K.C.M., 1999. Long-term evaluation of a behavior-
based method for improving safety performance: a meta-analysis of 73
interrupted time-series replications. Saf. Sci. 32, 1–18.
Johnson, K., Greenseid, L.O., Toal, S.A., King, J.A., Lawrenz, F., Volkov, B., 2009.
Research on evaluation use: a review of the empirical literature from 1986 to
2005. Am. J. Eval. 30 (3), 377–410.
Lee, E.T., Wang, J., 2003. Statistical Methods for Survival Data Analysis. John Wiley &
Sons, Hoboken, NJ.
Maguire, B.A., Pearson, E.S., Wynn, A.H.A., 1952. The time intervals between
industrial accidents. Biometrika 39 (1–2), 168–180.
McPhee, J., 2005. Coal Train—I: Disassembling the Planet for Powder River Coal.
New Yorker, October 3.
Miles, M.B., Huberman, A.M., 1994. Qualitative Data Analysis: An Expanded
Sourcebook, second ed. Sage, Thousand Oaks, CA.
Nelson, W.B., 1982. Applied Life Data Analysis. John Wiley & Sons, Hoboken, NJ.
Nelson, W.B., 2003. Recurrent Events Data Analysis for Product Repairs, Disease
Recurrences, and other Applications. Society for Industrial and Applied
Mathematics, Philadelphia, PA.
Patton, M.Q., 2002. Qualitative Research and Evaluation Methods, third ed. Sage,
Thousand Oaks, CA.
Pawson, R., 2002. Evidence-based policy: the promise of realist synthesis.
Evaluation 8 (3), 340–358.
Pawson, R., Greenhalgh, T., Harvey, G., Walshe, K., 2005. Realist review – a new
method of systematic review designed for complex policy interventions. J.
Health Serv. Res. Policy 10 (suppl. 1), 21–34.
Pawson, R., Tilly, N., 1997. Realistic Evaluation. Sage, Thousand Oaks, CA.
Pedersen, L.M., Nielsen, K.J., Kines, P., 2012. Realistic evaluation as a new way to
design and evaluate occupational safety interventions. Saf. Sci. 50, 48–54.
Phimister, J.R., Bier, V.M., Kunreuther, H.C., 2004. The accidents precursors project:
overview and recommendations. In: Phimister, J.R., Bier, V.M., Kunreuther, H.C.
(Eds.), Accident Precursor Analysis and Management: Reducing Technological
Risk through Diligence. National Academies Press, Washington, DC.
Railroad Safety Risk Reduction Program in the Rail Safety Improvement Act of 2008,
2008. 49 U.S.C. §20156.
Ranney, J., Nelson, C., 2003. Impacts of Participatory Safety Rules Revision in U.S.
Railroad Industry: An Exploratory Assessment. Federal Railroad Administration,
DOT-VNTSC-FRA-02-05.
Ranney, J., Zuschlag, M., Coplen, M., Nelson, C., 2010. Behavior-based Safety at
Amtrak-Chicago, Associated with Improved Safety Culture and Reduced Injuries
and Costs. Manuscript in Preparation (copy on file with author).
Reason, J., 1997. Managing the Risks of Organizational Accidents. Ashgate,
Aldershot, UK.
Reason, J., 2003. Managing Maintenance Error. Ashgate Publishing Company, UK.
Rosenthal, R., Rosnow, R.L., 1991. Essentials of Behavioral Research: Methods and
Data Analysis, second ed. McGraw-Hill, New York, NY.
Rossi, P.H., Freeman, H.E., Lipsey, M.W., 1999. Evaluation: A Systematic Approach,
seventh ed. Sage, Thousand Oaks, CA.
Shadish, W.R., Cook, T.D., Campbell, D.T., 2002. Experimental and Quasi-
Experimental Designs for Generalized Causal Inference. Houghton Mifflin,
Boston.
Simard, M., Marchand, A., 1994. The behaviour of first-line supervisors in accident
prevention and effectiveness in occupational safety. Saf. Sci. 17, 169–185.
Simard, M., Marchand, A., 1997. Workgroups’ propensity to comply with safety
rules: the influence of macro-micro organizational factors. Ergonomics 40, 172–
188.
Spigener, J.B., Hodson, S.J., 1997. Are labor unions in danger of losing their
leadership position in safety? Prof. Saf. (12), 37–39, 1997
U.S. Government Accountability Office (GAO), 2009. Program Evaluation: A Variety
of Rigorous Methods Can Help Identify Effective Interventions. Report to
Congressional Requesters. GAO-10-30, November 23, 2009.
Walton, M., 1986. The Deming Management Method. Berkley Publishing, New York,
NY.
Yin, R.K., 2009. Case Study Research, fourth ed. Sage, Thousand Oaks, CA.
Zuschlag, M., Ranney, J., Coplen, M., Harnar, M., 2012. Transformation of Safety
Culture on the San Antonio Service Unit of Union Pacific Railroad. Federal
Railroad Administration, DOT/FRA/ORD-12/16, DOT-VNTSC-FRA-10-07. <http://
www.fra.dot.gov/eLib/details/L04121> (17.12.12).