You are on page 1of 53

Discovery Bay Outage

Cause
Evaluation
Team

Discovery Bay, CA
Executive Summary

On the morning of December 27, 2015, the PG&E Gas Distribution


Control Center (GDCC) identified a pressure drop in the Discovery
Bay / Byron distribution system.
Operations personnel responding to the call from GDCC observed
that the regulating station was frosted over and that the pressure
was erratic. The primary and standby pressure regulating runs were
unable to maintain a steady downstream pressure.
To ensure the safety of downstream customers, the district
regulation station was isolated, resulting in the loss of gas supply to
customers in the Discovery Bay and Byron communities.
When withdrawal operations from Whisky Slough Station (WSS)
began on 12/26/15 at 0700, Valve 35 at Whisky Slough which
should have been closed was in the open position. This resulted in
sending un-dehydrated gas to the system through Line 57B.
As a result of the wet gas in Line 57B, the cold temperatures, and
the large pressure cut at the district regulation station within the
pressure limited station (PLS), hydrates were formed in
the pilots of the Mooney regulators and working monitors, causing
erratic pressure control at this station.
This report identifies the following 3 Root Causes focused on the
operations and processes at McDonald Island (MDI), and
specifically Whiskey Slough Station:
Root Cause 1:
PG&Es asset and risk management practices did not adequately
consider the risks associated with liquid or moisture intrusion into
the gas system
Root Cause 2:

Sumeet Singh, VP Gas


Asset & Risk Management,
Executive Sponsor
Terry White, Director
Facility Integrity
Management, RCE
Sponsor
Kimberley Corona,
Corrective Action
Program, RCE Lead
Mark Frauenheim,
Performance
Improvement Manager,
DCPP, RCE Process Expert
A complete list of team
members is listed in
Appendix A.

Discovery Bay Outage

An ineffective corrective maintenance system at MDI led WSS operators to distrust the controls
systems and to accept a normalization of deviation.
Root Cause 3:
Current control room practices at Whisky Slough Station were not sufficient for a facility which has this
level of complexity and risk.

Corrective actions:
Root Cause 1:
CA1 Complete assessment and revise liquid management program as necessary to support
site specific requirements. Include analysis of moisture analyzer locations, database of
moisture/liquids locations, actions to be taken when liquids/moisture are found, as well as
predictions of locations that might have moisture/liquids intrusion into the system.
CA2 Complete a design Failure Modes and Effect Analysis (FMEA) for Design Standard H-14;
per the results update H-14 accordingly.
CA3 Review and revise as needed the Regulation Station Design Manual ensuring it addresses
the risk of excessive moisture/liquid in the system.
CA4 Utilize process safety methodology to evaluate the level of protection and control at WSS
based on the risk posed by operating the facility; per the results, update as necessary to address
the findings. Include an evaluation of V-35 (i.e. the need for out of position alarms and indications)
and other key routing of valves at this station.
CA5 Make temporary ERX for DFM a permanent installation with communication to SCADA
CA6 Make temporary moisture analyzer at WSS a permanent installation with communication to
SCADA
Root Cause 2:
CA7 Identify, prioritize and complete all open high risk corrective work for control systems at
Whisky Slough Station. Identify and complete other high risk issues that have not been entered in
our corrective maintenance system.
CA8 Implement a prioritization system for transmission corrective work (i.e. a work and
compliance matrix).
Root Cause 3:
CA9 Review, and update on-going operator training as necessary for station operators at MDI.
Include system, alarm response training and refresher training.
CA10 Determine the scope of implementing select Control Room Practices (CRM) principles at
manned facilities. Implement based on leadership guidance.

Discovery Bay Outage

CA11 Revise and update operation procedures and develop addition tools (i.e. develop a
check list) for critical operations.
Additional information is included in the body of this report. This report was prepared by Kimberley
Corona and Katie Simone.

Discovery Bay Outage

to as the boot, which forms a seal against the throttling plate. No manual regulator bypass line was
installed at this station.
Shown in Figure 2 is a cut-away view of the two stage pressure regulator set. The upstream regulator
(working monitor) is fitted with two pressure pilots: one for pressure regulation and the second for overpressure protection (the pressure monitor pilot). Upstream line pressure enters the working monitor
housing through the loading line highlighted in red. Pressure is reduced to the intermediate pressure
value of 645 psig (highlighted in yellow) as gas flow through the pressure reducing pilot is reduced to
the set point value determined by the spring tension applied to the pressure sensing diaphragm. Gas
pressure is reduced from the upstream side of the working monitor to the downstream side through a
throttling valve in the pressure pilot. When the downstream pressure reaches the adjusted set point
value the gas pressure causes the pressure pilot throttling valve to close. If no demand exists, the
working monitor will remain closed. When demand returns, the working monitor will open and feed
more gas into the downstream system. The same functionality is demonstrated in the regulator, but with
a lower set point value of 380 psig. When the downstream system pressure drops to 375 psig the
secondary regulation run will actively flow to maintain the pressure set point.

Figure 2 Cut-away view of a two stage working monitor and regulator


For over-pressure protection, a pressure monitor pilot on the working monitor receives gas pressure
downstream of the regulator through a sensing line, highlighted in light blue in Figure 2. Under normal
operating conditions, the pressure monitor function is not active, and pressure is reduced in the two
stages as previously described. If for any reason, downstream pressure rises to the value of the
monitor set pressure (395 psig), intermediate gas pressure on the outlet of the working monitor will be
reduced to the monitor set point pressure value to prevent exceeding the downstream maximum
allowable operating pressure. Figure 3 illustrates how gas would flow through the regulator set when in
monitor control.

Discovery Bay Outage

decision was made by operations personnel to take the district regulation station supplying DFM 302201 at
out of service to avoid a potential unsafe over pressure condition.
Interruption of service was deemed necessary as there was no option available to manually regulate
the gas flow through a bypass line or to safely regulate pressure through one regulator run while
repairing the other run since both the primary and secondary runs were affected by the abnormal
operating condition.
The gas technicians responding to the event found during disassembly of the pressure pilot, that
hydrates had formed in the pilot throttling valve assembly. Figure 6 shows the location of the hydrates
within the pressure pilot. Technicians also observed hydrates in the openings of the regulator throttling
plate. No analysis of the hydrate was conducted to determine composition of the hydrate.

Figure 6 Cut-away details of pressure pilot and throttling valve (PRV)


No evidence was found during the root cause investigation, to establish the fact that free liquids went
through the regulator station. Considering the high moisture event that happened simultaneously with
the regulator problems and the cold temperatures that day, the team performed calculations to
determine the potential for hydrate formation (see Appendix). The calculation revealed that the
operating conditions were appropriate for formation of gas hydrates. Figure 7 presents images of the
hydrate as found within the pressure pilot throttling valve by the gas technicians.

Discovery Bay Outage

Figure 7 Photos showing hydrate plug removed from pressure pilot


Formation of a hydrate plug in the pilot throttling valve assembly could be expected to gradually begin
to restrict the movement of the throttling valve stem. Over time the formation of hydrates would be
expected to progress to the throttling valve seat area and eventually plug off the flow of gas through the
pressure pilot. Once flow is blocked in the pilot, the regulator would fail closed, assuming that the
diaphragm can form a complete seal. The district regulation station at
as installed had no
protection for the potential of liquid formation by condensation, slug flow, mist flow, or the formation of
hydrates. During the event, the natural gas going through the regulator station was saturated with water
vapor, as evidenced by the high moisture readings. There is no evidence that free liquids in two phase
flow regime passed through the regulators. Considering this, it is most likely that the formation of
hydrates began with moisture condensation in the top portion of the regulator and pressure pilot.
The overnight ambient temperatures on 12/26/15 and 12/27/1527 near
reached a low of
about 29F. The regulator body would be cooled by ambient temperature and the Joule-Thomson
cooling effect produced by pressure reduction across the regulator flow elements. The calculated
hydrate formation temperature for the first stage of regulation is 33.3F. The calculated temperature
reduction for the first stage of regulation yielded an outlet temperature of 35.6 F. The moist gas would
be further cooled by heat transfer through the pilot body and regulator body to the low temperature
necessary to form hydrates. The second stage of regulation produced an outlet temperature of about
17.5F, which is well within the hydrate formation temperature range.
There have been past instances of moist gas / liquid intrusion into the PG&E system. As noted in the
background section, Mooney Flowgrid valves are not designed for saturated gas or two phase flow and
would require additional layers of protection (safeguards) to prevent them from failing. Gas Design
Standard, H-14, provides general guidance on types of filters with which to equip pilot supply lines
when excessive moisture or other contaminants are known to occur at a specific station. Station design
engineers do not have adequate information regarding gas quality at specific stations during station
design. As a result, most stations have not been designed to protect against excessive moisture or
liquid.
In order to determine the source of moisture seen at
the team completed an analysis
utilizing available SCADA time stamped data, as shown in Figure 8.
can be supplied from
either L401 or L57B. The red line in the Figure represents the flow rate when
was supplied
from L57B, which comes from McDonald Island. The red line at zero indicates times
was
supplied from L401 or flow through
was stagnant. The yellow line represents the

10

Discovery Bay Outage

Based on the timeline above, it can be concluded that the moisture came from L57B, ultimately from
McDonald Island indicated by the dramatically increased moisture levels seen when
was
fed from L57B at increased flowrates.
McDonald Island Background Information
McDonald Island is PG&Es largest gas storage field. It is comprised of three stations, Whisky Slough
Station (WSS), Turner Cut Station (TCS), and McDonald Island Compressor Station (MCS). WSS and
TCS process and dehydrate gas. MCS includes two electric compressors (K-1 and K-2) and three
rental gas-fired reciprocating compressors (K-7, K-8, and K-9). The three stations work in conjunction to
inject and withdraw gas into and out of the wells and storage field as demand requires.
In 2011, PG&E started a project to rebuild WSS. The scope of the project focused on the replacement
of obsolete equipment, not enhancements. A process hazard analysis (PHA) was completed on this
project, however the PHA did not identify all potential hazardous scenarios. For example, the
misoperation/failure of WSS V-35 was not analyzed to determine the potential hazardous scenarios
associated with this valve.
Typical Withdrawal Operations
To set-up McDonald Island (MDI) for withdrawal, the TCS and WSS operators respond to the request
by the GTCC by setting valves at WSS and/or TCS for withdrawal, setting the max well flow rates,
ensuring the glycol system is circulating and hot, and aligning the MCS valve lot for withdrawal. One
key valve that needs to be aligned is V-35, which is the bypass valve around the processing and
dehydration equipment. During withdrawal, V-35 needs to be closed to ensure that the gas is sent
through the processing and dehydration equipment. The normal withdrawal operation of WSS is to
supply gas to Line 57B/C through WSS Emergency Block Valve, V-56. The gas is transported,
processed, dehydrated, metered through WSS Master Meter, M-1, and then pressure controlled on the
platform via WSS regulating valves, V-42 and V-37. Figure 10 shows a simple flow diagram highlighted
with the normal withdrawal flow, for additional details see the highlighted operating schematic located in
the Exhibit C.

13

Discovery Bay Outage

Figure 10 Normal Withdrawal Flow through WSS (V-35 Closed)

Typical Well Rework Operation


Well rework operation is performed routinely on gas storage wells to replace malfunctioning downhole
safety valves (DHSV), evaluate production casing conditions, and rework wells to maintain well
injection and withdraw performance to meet customer demands for pipeline system operations. As part
of well rework operations, Reservoir Engineering requests that the rework wells stay on continuous flow
for 33 days to complete the well rework clean-up process. The scheduling of the continuous flow is
planned for periods when the withdrawal of storage gas is needed to meet system needs. Depending
on system demand, GTCC normally transports the gas withdrawn from the reworked wells into the
pipeline. If system demand changes GTCC may then notify MDI Operations to set up the station(s) to
transport the gas withdrawn from re-work wells to where it is injected into other wells. During this reinjection set-up, V-35, the dehydration system bypass valve, needs to be open.
McDonald Island Event Description
On 12/21-24, 2015 two TCS and two WSS wells were undergoing rework; due to system demand, the
rework gas needed to be re-injected into the system. This was completed by sending the rework gas to
Injection Compressors (K-7/8/9) where it was then re-injected into the wells. Figure 11 shows a simple
flow diagram highlighted with the normal re-injection flow, for additional details of how MDI was aligned
on 12/23-24/15 see the highlighted operating schematic located in the Exhibit C.

Figure 11 Normal Re-Injection Flow through WSS (V-35 Open)


On the morning of 12/27/15, gas flow through
supplying DFM 3022-01 was interrupted. As
discussed earlier, this was caused by sending moist gas from MDI through L57B. The following
discussion represents the root cause teams understanding of the events that occurred at MDI that led
to this event:

14

Discovery Bay Outage

At 1200 on 12/26/15, another operator arrives on shift and takes over control of WSS, leaving the
other operator to maintain control of TCS. The WSS operator began rounds for the well reworks,
drained collected fluids from the rework separators, and then drained collected fluids from WSS
Filter Separators (C-11/12/13/14 and C-26). He did not drained the collected fluids from Separators
(C-34/35) at this time. Following his rounds, the WSS operator performed required inspections and
calculations required by the Spill Prevention, Control and Countermeasure regulations (SPCC)
which take about 2-3 hours to complete.
At 1700 on 12/26/15, shown on Figure 12, point 3, GTCC called to request additional withdrawal
capacity and wanted to identify the maximum capacity MDI could provide. Withdrawal rates were
increased to determine the maximum withdrawal rate. After conversations between GTCC and the
MDI operators, it was determined the higher rate was not required and withdrawal rates were
decreased.
At 1800 on 12/26/15, a shift change occurred, the TCS operator went off shift, the WSS operator
went to operate at TCS, and another operator began shift at WSS.
At 2345 on 12/26/15, GTCC called WSS and TCS to alert them of high moisture readings
downstream at
. The TCS and WSS operators checked the dehydration equipment,
checked valve positions at the station, dumped the separators, and took dew points samples. The
dew point samples results were 1 lbs/MMSCF at TCS and 2 lbs/MMSCF at WSS. The TCS and
WSS operators reported back to GTCC that everything was normal.
At 0300 on 12/27/15, shown on Figure 12, point 4, the Up Hole Safety Valves (UHSV) tripped on the
WSS wells. Around the same time, data indicates that V-35 closed. After reviewing data from the
processing equipment, the team concluded that V-35 was open when withdrawal started at 0700 on
12/26/15 and remained open until 0300 on 12/27/15. Figure 13 shows a simple flow diagram
highlighted with the withdrawal flow during this time frame, for additional details of how McDonald
Island was aligned on 12/26-27/15 see the highlighted operating schematic located in the Exhibit C.

Figure 13 Withdrawal Flow through WSS on 12/26/15 (V-35 Open)


Below is the list of the data points, provided by the MDI facility engineer, suggesting that the
processing/dehydration equipment was bypassed from 0700 on 12/26/15 through 0300 on 12/27/15,
due to V-35 being in the open position:

16

Discovery Bay Outage

No liquid accumulation in WSS Filter Separators C11 C13 (LIT-W42/W44/W46)


between 0700, 12/26/15 to approximately 0300, 12/27/15. Under normal withdrawal
circumstances, liquid accumulation is expected. This would normally increase the level
in the Filter Separators to a set point where the dump valve would automatically open
draining level to normal.
No differential pressure across WSS Filter Separators C11 C13 (PDIT-21/22/23) from
between 0700, 12/26/15 to approximately 0300, 12/27/15. When gas is flowing through
the Filter Separators it is expected that there would be a pressure drop from the inlet gas
to the outlet gas due to flow over the internals of the Filter Separator.
No flow through WSS contact tower (C-2/3/4) orifice meters M5/M6/M7 between 0700,
12/26/15 to approximately 0300, 12/27/15. Three data points from the orifice meter were
used to validate this conclusion:
o No flow. Under normal withdrawal circumstances, expect to see flow rates.
o No temperature increase when withdrawal was started. Under normal withdrawal
conditions, expect to see the temperature to increase to the normal operating
temperature of the contact tower.
o No differential pressure across the meter. Under normal flow conditions expect to
see some differential pressure across the orifice meter.
No temperature differential across WSS V-35 (TI-30 and TI-35) until after approximately
0300 on 12/27/15. Under normal withdrawal conditions, expect to see a temperature
differential across V-35. As gas is routed through the processing/dehydration equipment
it will be heated by the glycol in the contacting towers, resulting in a temperature
differential across V-35.
No differential pressure between the well header pressure (PIT-4) and the pressure
downstream of V-35 (PIT-13) until after 0300 on 12/27/15. Under normal withdrawal
operations, expect to see differential pressure as there is an expected pressure drop due
to flow through the processing/dehydration equipment.

The Controls Engineer confirmed that the indicators noted above are all independent and that a single
point of failure (i.e. PLC failure, power failure) would not result in the types of anomalies observed in the
data set.
When V-35 is open during withdrawal operations, moist gas from the wells would be sent through
L57B/C to
. The moisture levels observed at
coincides with the data supporting
the fact that V-35 was open during withdrawal.
Upon examination of the Operating Procedures for WSS, the Whisky Slough Station Operating
Procedure Section 3.0 Station Withdrawal (080661) does include a step to CHECK CLOSE Station
Block Valve, V-35. However, the Operating Procedure does not include further details on how to check
close the valve (i.e. check close using Cimplicity, the Plant Mimic Board, or field verification). There are
also no formal requirements to document the positions of key operating valves during operational
changes at WSS (i.e. a checklist). There is no formal shift turnover process. In addition, operators are
not required to perform a field walk down to verify key valve positions at the beginning of each shift.
Since the manned station at WSS is a remote facility from the GTCC, operators are not required to
practice Control Room Management (CRM) processes. This includes fatigue management, written shift

17

Discovery Bay Outage

turnovers, and thorough documentation of valve manipulations in operating logs best practices. WSS
operators do not get the same rigorous training provided to operators at GTCC/GDCC. GTCC/GDCC
training includes system and alarm response management as well as simulator training. WSS operators
receive primarily on the job training.

Unfortunately due to lack of visibility at WSS, the team could not determine why V-35 was left open
during withdrawal and then closed at approximately 0300 on 12/27. For instance, the position of V-35 is
not stored in Historian and interviews with operators on shift during this event were inconclusive. In
addition, the operator logs for this time do not include details regarding the position of V-35 at the time
of this event. The operator logs only stated the following items regarding starting withdrawal at WSS:

0820 on 12/24/15 the field was set up for WD.


1900 on 12/24/15 C34/35 were valved for W/D and V-56 was closed.
1930 on 12/24/15 the wells were set up for WD and the max well flows were set to the current field
pressure.
1815 on 12/25/15 V-51 (MCS) was opened for W/D availability at WSS.

In addition to the lack of visibility, the operators were required to operate the system with inadequate
controls and alarms. For example:

There is no out of position alarm or interlock to prevent V-35 from being open when withdrawal
operations are occurring.
At the time of the event there was no online moisture analyzer at MDI with visibility to GTCC.
Operators were required to take manual moisture readings once a shift, however on 12/26/15 no
moisture reading was taken from the time withdrawal started at 0700 on 12/26/15 until 2345 that
night after being notified of potential high moisture at
from GTCC.
There are no interlocks to prevent key valves from being operated in the local mode.

However it should be noted that during a site visit and interviews conducted by the root cause team on
1/20/16, MDI operators raised concerns with the controls and indication provided in the WSS control
room.For example, they provided the following information:

As of 1/20/16, 11 of 15 valve status indication lights on the WSS Plant Mimic Board were not
working. Specifically the status indication light for V-35 was not working.
The level transmitters seen in Cimplicity and the local level indicators show a different level for C34/35.
A week prior to this event, an operator had called for V-49 to open via Cimplicity, but upon field
verification, found this valve to still be closed and had to open the valve manually.
On 1/20/16, it was determined that the local/remote indication of V-35 on the Cimplicity HMI was
always indicating the valve to be in the local mode regardless of actual status in the field.
A corrective work order entered on 11/25/15 stated that C-14 level not reading correct, and that the
issue was fixed on 11/30/15. Approximately a month later the operator log identified another issue
with the C-14 level indication. However, no corrective action was entered on that day.

18

Discovery Bay Outage

In conjunction with the concerns of the control system mentioned above, operators also identified
reservation that any issues submitted via corrective work orders would be addressed I a timely manner.
During interviews, operators spoke to a large backlog of maintenance requests that have been
neglected due to the need to complete compliance maintenance work. There is currently no formal
process in place to prioritize maintenance requests at this facility. Due to this distrust, it appears that
the operators have stopped entering items into Pipeline Maintenances (PLM) for corrective work. Of the
6 issues noted above during the interviews, 3 had no corrective work request inputted in PLM and the
other two were added into PLM after the interviews were conducted.
Root Cause/Causal Factors
Root Cause 1:
PG&Es asset and risk management practices did not adequately consider the risks associated with
liquid or moisture intrusion into the system.

Liquid or moisture can cause both regulators and monitors to fail unpredictably and most
stations have been designed without sufficient protection.

Existing pressure regulation design standards assume that gas is typically pipeline quality and
doesnt adequately consider liquid or moisture potential.

There are multiple potential sources of liquid or moisture which are not incorporated in the
design and maintenance practices:
o

California gas production

PG&E and 3rd party gas storage

Pipeline interconnects

Hydrostatic testing and pigging activities

The control system at McDonald Island does not adequately address the potential for liquid or
moisture intrusion.

There is no on-line moisture analyzer at McDonald Island with visibility to GTCC.

Root Cause 2:
An ineffective corrective maintenance system at MDI led WSS operators to distrust the control systems
and an accept a normalization of deviation.

Control systems and indicators are not always accurate. Examples include:
o

11 of the 15 valve status indication lights on the WSS Status Board were not working (as
of 1/20/16).

Cimplicity (WSS HMI) was indicating V35 in local control regardless of actual remote or
local control in the field (as of 1/20/2016).
19

Discovery Bay Outage

Corrective Actions
Root Cause 1

CA1 Complete assessment and revise liquid management program as necessary to


support site specific requirements. Include analysis of moisture analyzer locations,
monitor feedback and actions to be taken when liquids/moisture are found, as well as,
predictions of locations where risk from moisture/liquids intrusion into the system is
high.
Assigned: Terry White
Due: March 1, 2017
CA2 Complete a design Failure Modes and Effect Analysis (FMEA) for Design
Standard H-14; per the results update H-14 accordingly.
Assigned: Austin Hastings
Due: December 10, 2016
CA3 Review and revise as needed the Regulation Station Design Manual ensuring
it addresses the risk of excessive moisture/liquid in the system.
Assigned: Raymond Stanford
Due: March 10, 2017
CA4 Utilize process safety methodology to evaluate the level of protection and
control at WSS based on the risk posed by operating the facility; per the results,
update as necessary to address the findings. Include an evaluation of V-35 (i.e. the
need for out of position alarms and indications) and other key routing of valves at this
station.
Assigned: Terry White
Due: April 1, 2017
CA5 Make temporary ERX for DFM a permanent installation with communication to
SCADA
Assigned: Terry White
Due: October 1, 2017

CA6 Make temporary moisture analyzer at WSS a permanent installation with


communication to SCADA

21

Discovery Bay Outage

Assigned: Terry White


Due: October 1, 2017
Root Cause 2

CA7 Identify, prioritize and complete all open high risk corrective work for control
systems at Whisky Slough Station. Identify and complete other high risk issues that
have not been entered in our corrective maintenance system.
Assigned: William Mojica
Due: June 30, 2016
CA8 Implement a prioritization system for transmission corrective work (i.e. a work
and compliance matrix).
Assigned: Terry White
Due: July 1, 2017

Root Cause 3

CA9 Determine the scope of implementing select Control Room Practices (CRM)
principles at manned facilities. Implement based on leadership guidance.
Assigned: Dan Menegus
Due: October 31, 2016

CA10 Review, and update on-going operator training as necessary for station
operators at MDI. Include system, alarm response training and refresher training.
Assigned: William Mojica
Due: August 30, 2016
CA11 Revise and update operation procedures and develop addition tools (i.e.
develop a check list) for critical operations.
Assigned: Terry White
Due: March 1, 2017

Extent Evaluation
Extent of Condition

22

Discovery Bay Outage

This problem could exist when excess moisture or liquid is introduced or left in the system upstream of
regulation facilities which do not have designs resistant to liquid/moisture in the gas stream. Several
actions were taken early in the event. See Appendix C for a complete report. The following additional
actions have been recommended to further mitigate this risk:
1. Implement a program for routine pigging downstream of high risk potential sources or known
intrusion of liquid in the system.
Assigned: George Karkazis
Due: November 30, 2016
2. Identify locations for immediate installation of heaters, based in increased likelihood of liquids or
moisture.
Assigned: Terry White
Due: April 1, 2017
3. Using the results of CA1, install moisture monitoring strategically throughout the system and
consider analyzer redundancy, and perform manual moisture readings as a short-term barrier.
Assigned: Terry White
Due: March 1, 2021
4. Confirm that other moisture analyzers within the system are scaled and calibrated properly.
Assigned: Dan Menegus
Due: August 15, 2016
5. Complete a design review of moisture analyzer installations to ensure adequacy.
Assigned: Terry White
Due: May 1, 2017

Extent of Cause
Root Cause 1 applies to other design standards that may be overlooking other key failure modes and
their effects on the system. Actions 1, 2, and 3 below have been recommended to address this
potential gap.
Root Cause 2 applies to other areas within Gas Operations that may have an excess backlog of open
corrective work for control systems and other high risk issues. Action 4 below has been recommended
to address this potential gap.

23

Discovery Bay Outage

Root Cause 3 applies to other manned stations where more rigorous control room management
practices will reduce risk. Corrective actions address this gap.
1. Consider completing a design FMEA for other design standards. (RC1)
Assigned: Austin Hastings
Due: June 12, 2017
2. Complete assessment of design standards to see if there are any other standards that provide
conflicting information. (RC1) Evaluate the effectiveness of the process to compare new/updated
procedures with existing procedures.
Assigned: Austin Hastings
Due: December 10, 2016
3. Evaluate existing stations based on the assessment results obtained from CA1. (RC1)
Assigned: Terry White
Due: March 1, 2018
4. Complete system-wide assessment for open corrective work for control systems at all other
stations. Identify and complete other high risk issues that have not been entered in our corrective
maintenance system for other manned stations. (RC2)
Assigned: William Mojica
Due: June 30, 2016
5. Complete assessment of level of concern at other manned facilities. Combine with CA4, use a
survey for both issues, undocumented issues, and distrust of controls.
Assigned: Terry White
Due: September 15, 2016

Analysis Tools and Methods


The team utilized a number of tools and methods to conduct this root cause analysis including event
and causal factor charting, factor trees, hazard barrier analysis, and engineering calculations which can
be found in the appendix.

24

Discovery Bay Outage

Effectiveness Review Plan


Root Cause 1: Upon completion of corrective actions for Root Cause 1, verify there have been no
reductions in system operability of the system due to liquid intrusion into the pipeline for a minimum of
12 months.
Root Cause 2: Upon completion of corrective actions for Root Cause 2, conduct a follow up survey to
validate the level of trust in the corrective maintenance system has an average rating of 4.5 out of 5.
Root Cause 3: Upon completion of corrective actions for Root Cause 3, verify there have been no
consequences to incorrectly aligned valves for a minimum of 6 months.
See Notification 7024114 for tracking of the effectiveness review plan
Appendices
Appendix A Root Cause Core Team Members
Appendix B Timeline
Appendix C Drawings & Pictures
Appendix D Preliminary Extent of Condition Report
Appendix E Interviews
Appendix F Event & Causal Factor Chart
Appendix G Fault Tree Analysis
Appendix I Engineering Calculations

25

Discovery Bay Outage

Exhibit B Timeline

2012-2013:

Whisky Slough Station (WSS) rebuild project

12/24/2015:

WSS set up for re-injection to support well cleanup activities.

12/26/2015:
0700:

WSS began withdrawal operations with V-35 left open.

0730:

Suction header of Compressors are involved in overpressure event at McDonald Island.

1030:
1430:

primary supply switches from L401 to L57B


Small spike in moisture readings at

1800:
1830:

primary supply switched from L57B to L401


Small decrease of pressure at Pt. of Timber, secondary reg run gains controls at

2300:
2345:

primary supply switched from L401 to L57B


GTCC receives high moisture alarm at

, calls out Tech to investigate

12/27/2015:
0050:

GDCC receives Low pressure alarm on Distribution system downstream of


call out Tech to investigate

0130:

Pressure at Pt. of Timber falls to 28 psig (normally 380 psig)

0300:

Process parameters indicated the V-35 was closed at WSS, gas sent from WSS is now
being dehydrated

0315:

Tech at

reports Mooney Regulators iced over and gas not flowing.

0500:

Primary supply switched from L57B to L401

0700:

primary supply switched from L401 to L57B

0800:

Decision to isolate Regulation Station feeding DFM 3022-01 at

0900:

Moisture readings at

top out.

Exhibit C Drawings and Pictures


1. System Overview

Discovery Bay Outage

Exhibit D Preliminary Extent of Condition Report

Discovery Bay Outage

Discovery Bay Outage

Dehydration System
bypassed (V-35
open)

OR
W/D from tubing
header and V-77
open

V-35 was open and


failed last

Valve open
(misoperation)

OR
Left open after
maintenance (valve
stroke)

Left open from


injection/reinjection

AND

Operator didnt
realized it was open

Operators not
required to
document valve
manipulations

AND

Operators do not
trust electronic
indicators

Corrective
Maintenance System
ineffective

Operator did not


perform a field walk
down to check close

Shift Turnovers are


oral

Control Room Practices


not rigorous

Discovery Bay Outage

Exhibit H Engineering Calculations

Date
12/26/2015
12/27/2015

M3
Temp
Time
M3 Flow Press (Assumed)
Zb
17:05
23.15
1027
50 0.9977
17:41
23.11
1015
50 0.9977
0:36

Zf
0.830883
0.830883

Uncorrected
Q
267,447
270,097

Cross
Section Velocity
Area (ft/sec)
2.29211 32.4115
2.29211 32.7327

Assuming an 8 minute sample lag time:


Time before dew point rise

Temp
50
17.8

1680 seconds
54,721.16 feet
10.36 miles

Water Saturation IGT8


psia
A
B
902.4
659.4

8553.3 4.3398609
2351.5 1.7063102

Wc
13.8
5.3

First Stage
Second Stage

J-T Cooling
Inlet
Outlet
Pressure Pressure
Outlet
(psia)
(psia) temp (F)
902.4
659.4
35.6
659.4
389.4
17.8

J-T Coefficient 1
0.05925
J-T Coefficient 2
0.06607
(assuming constant enthalpy)

Discovery Bay Outage

Gas Composition
methane
ethane
propane
isobutane
butane
isopentane
pentane
hexane
CO2
nitrogen

Btu
Relative Density

94.1
4.4
1.4
0.12
0.011
0.004
0.0026
0.006
0.82
0.5
101.36
1058.8
0.6014

Normalized
92.834
4.3408
1.3812
0.1184
0.0109
0.0039
0.0026
0.0059
0.809
0.4933
100.00

Discovery Bay Outage

The following equation [Kobayashi et al Correlation] was used to determine the hydrate formation temperature with the conditions at
.

Where:

T = Hydrate Formation Temperature (C)


P = line pressure (bar)
= Relative density
The constants are as follows:
A1
2.7707715E-03
A2
-2.7822380E-03
A3
-5.6492880E-04
A4
-1.2985930E-03
A5
1.4071190E-03
A6
1.7857440E-04
A7
1.1302840E-03
A8
5.9728235E-04
A9
-2.3279181E-04
A10 -2.6840758E-05
A11 4.6610555E-03
A12 5.5542412E-04
A13 -1.4727765E-05
A14 1.3938082E-05
A15 1.4885010E-06

Discovery Bay Outage

Kobayashi Calculation
Formation Temperature
Relative
Density
P (psia)
P (bar)
900
62.05
890
61.36
880
60.67
870
59.98
860
59.29
850
58.61
840
57.92
830
57.23
820
56.54
810
55.85
800
55.16
790
54.47
780
53.78
770
53.09
760
52.40
750
51.71
740
51.02
730
50.33
720
49.64
710
48.95
700
48.26
690
47.57
680
46.88
670
46.19
660
45.51
650
44.82
640
44.13
630
43.44
620
42.75
610
42.06
600
41.37

Method

for

Hydrate

0.6014
T(C)
0.73
0.74
0.74
0.75
0.76
0.77
0.78
0.79
0.80
0.81
0.82
0.83
0.84
0.85
0.86
0.87
0.88
0.90
0.91
0.92
0.94
0.95
0.97
0.98
1.00
1.01
1.03
1.05
1.07
1.08
1.10

T(F)
33.31
33.32
33.34
33.35
33.37
33.39
33.40
33.42
33.44
33.45
33.47
33.49
33.51
33.53
33.55
33.57
33.59
33.61
33.64
33.66
33.69
33.71
33.74
33.77
33.79
33.82
33.85
33.88
33.92
33.95
33.99

Discovery Bay Outage

590
580
570
560
550
540
530
520
510
500
490
480
470
460
450
440
430
420
410
400
390
380
370
360
350
340
330

40.68
39.99
39.30
38.61
37.92
37.23
36.54
35.85
35.16
34.47
33.78
33.09
32.41
31.72
31.03
30.34
29.65
28.96
28.27
27.58
26.89
26.20
25.51
24.82
24.13
23.44
22.75

1.12
1.14
1.17
1.19
1.21
1.24
1.26
1.29
1.32
1.35
1.38
1.42
1.45
1.49
1.53
1.57
1.61
1.66
1.71
1.76
1.81
1.87
1.94
2.00
2.08
2.15
2.24

34.02
34.06
34.10
34.14
34.18
34.23
34.28
34.33
34.38
34.43
34.49
34.55
34.61
34.68
34.75
34.82
34.90
34.98
35.07
35.16
35.26
35.37
35.48
35.61
35.74
35.88
36.03