Launch TapRooT Investigation

Gather data

Identify facts

Create SnapCharT

Brainstorm plausible scenarios

Fact-Hypothesis Matrix

Pattern Matching

Update SnapCharT

Re-test the same hypothesis

No

eXtreme Failure Analysis (XFA) ken.bloch@fhr.com

References: TapRooT: The System for Root Cause Analysis, Problem Investigation, and Proactive Improvement Machinery Failure Analysis and Troubleshooting 2nd ed. Guidelines for Investigating Chemical Process Incidents 2nd ed. Yes Does the hypothesis pass the Completeness Test?

Find the missing evidence

Dead End?

Go to the next hypothesis

Yes

No

Define the Logic solution

Reject hypothesis

Last hypothesis?

Accept hypothesis No No

Yes Yes

Does the hypothesis pass the Management System Causes Test?

Define the Fact problem

Seek outside support

No

Problem solved? Does the hypothesis pass the Overall Review Test? Yes

Yes Identify Latent Causes Using the TapRooT Root Cause Tree

No

Failure Analysis Data Sources Most Frequently Used Data Sources OEM manual (instructions, specifications, operating envelope, and procedures) PFDs P&IDs PI (process data archive) Work Order history Interviews Comprehensive? list People (possible people to interview) On shift operators Off shift operators Maintenance personnel checked into the unit Maintenance personnel assigned to the unit Process Engineers (Technical) Laboratory personnel Warehouse personnel Purchasing and accounting First responders Emergency response personnel Original Equipment Manufacturer (OEM) personnel Personnel previously involved in system maintenance and operation Personnel involved in previous failures on like equipment, process, or in the unit Janitorial, delivery, and other service personnel Relevant off-site personnel and visitors Project managers, design team, and installation contractors Security personnel Parts Tanks Valves Pressure boundary equipment such as gaskets and flanges Samples from all relevant vessels and piping Raw material samples Quality control samples Residuals or wastes generated (liquids, solids, gases) New chemicals Portable and temporary equipment Undamaged areas and equipment Relief system devices including rupture disks Metallurgical samples Conductivity measurements Obvious missing physical data (items, stains, and other items that should be present but are missing)

Page 1 of 3

Explosion fragments Data recorders Sensors Process controls Electrical switch gear Blast damage Pieces of process equipment Position As-found position of every valve related to the failure or process As-found position of controls and switches As-found position of hoses and temporary connections Relief device positions Tank levels Pointer needle positions from locally mounted temperature, pressure, and flow devices Location of flame and scorch marks Position and sequence of layers of materials and debris Direction of glass fragments Missile mapping Location of parts removed from the process as part of maintenance Location of personnel involved with maintenance and operation of the process Location of witnesses Location of equipment that should be present but is missing Smoke traces Location or position of chemicals in the process Melting patterns Impact marks Paper Supervisor Immediate Reports (SIR) Process data records strip and wheel charts Operating procedures, checklists, and manuals Shift logs Work permits Lockout-Tagout (LOTO) records Maintenance and inspection reports Repair records Run histories Batch sheets Raw material quality control records Retained sample documentation Quality control (QC) laboratory logs Emergency responder reports Process and Instrumentation Drawings (P&IDs) and detailed instrument and electrical drawings KPRP Reports

Page 2 of 3

OE (Process Engineer) Monthly Reports Cause and effect tables Equipment drawings and specification sheets Equipment data sheets Design calculations and design basis assumptions and stipulations Alarms and set points for trips Scenarios for relief, venting, and emergency equipment sizing Safe Limits of Operation (SLOs) Process Hazards Analysis and HAZOP records Material Safety Data Sheets (MSDS) Descriptions of expected and abnormal chemical reactions including incompatibilities Material balances Corrosion monitoring Site maps and plot plans Electrical classification drawings Instrument loop diagrams Interlock drawings Ladder logic diagrams Control system software logic Management of change (MOC) records Prior incident investigation reports Prior equipment failure analysis Training manuals and records Meteorological records Dispersion calculations Consequence analysis study results Phone logs Emergency response logs Process or product development data and reports Electronic Volatile control instrumentation records Work orders Logbook entries (electronic) Process incident tracking (unplanned event logs) Archived Process Information (PI) data Archived alarm sequence (ProcessGuard) PLC set points Security camera tapes Process simulations (Hysis, FlareNet, etc.)

Page 3 of 3

Failure Date TapRooT(R) SnapCharT Rev. Page

Tank 312 Explosion and Fire 8/14/2007 TRC-2008 1 of 2

CF=Causal Factor PIO=Potential Improvement Opportunity SF=Secondary Failure

New Refinery Construction

Sept 1955 Tank 312 Put into Operation

LPG Vapor Cloud in Refinery (Pre-1998)

November 2000 TK312 Inspection

2004 Poly Unit Hazop

5/24/04 Spent Caustic Degassing System Approved

GNOC Refinery startup

Fugitive vapor cloud at 31H2

Tank pressure relief device opens

New roof installed

Thickness increased by 0.125"

Vapor cloud scenario considered Objective: remove LPG prior in front of Tank 312 Project to install degassing vessel (42V43)

15' x 26' tank in the Poly Unit (42)

Vapor cloud dissipates without igniting

Near Miss

Tank overpressure

NFPA and API requirements satisfied

External grounding strap installed

Higher corrosion allowance

Tank 312 vapor cloud release possible Spent caustic may contain hydrocarbon (HC)

Project target complete date set 12/31/2005

Receives flow from LPG treatment process

Spent caustic storage system

LPG vaporization in tank

Corrosion prevention

Internal tank coating applied

Loss of LPG Contactor caustic interface level

Action item to mitigate LPG blowthrough risk

Work Order (#048497900) written Increase LPG Contactor Caustic (NaOH) strength Environmental compliance WWTP performance LPG vapor blowby incident Potential air/HC mixture in Tank CF

Internal fill line and nozzle replaced

HC entrained in spent caustic

Strategy to keep caustic strength above min limit

Replace spent caustic with fresh caustic

Spent caustic flow stabilization

LPG Contactor caustic interface level lost Potential human error Manual process

LPG vapor blowthrough scenario

Caustic loses strength during LPG treatment

LPG contaminants react with NaOH

Caustic drain valve open too long

4/24/04 12/26/05 43V43 Design & Construction

8/10/05 43V43 Operating Procedure Approved

Vortex breaker included in 42V43 design Mitigate the risk for a blowthrough scenario Procedure written to safely transfer caustic into tank 42V43 flare valve open and N2 valve closed

12/27/2005 42V43 Commissioned

Pyrophoric Iron Begins Forming in Tank 312

1/7/2006 42V43 Procedure Deviation Start

A
(Next Page)

Original scope to pump 42V43 spent caustic into Tank 312

42V43 receives LPG Contactor spent caustic flow

Flare gas (~4% H2S) contaminates Tank 312

CF

H2S contacts bare iron surfaces inside Tank 312

Spent caustic transfer into Tank 312 needed

Degassing vessel full

Caustic pump reliability assessment completed

Scope changed to pressurize caustic into Tank 312

Pump protection (reliability)

Maintain interface level above 42V43 bottom drain nozzle

Minimum hold time and valve lineup excluded CF

Design Team draft procedure input not considered

42V43 hand switch in the "Fill" position

Degassing vessel inlet valve closed

Spent caustic transfer into Tank 312 needed

42V43 headspace vapors entrained in spent caustic

New potential ignition source inside Tank 312

CF

Degassing vessel inlet valve left open

Spent caustic transfer from LPG contactor needed

Vortex breaker not included in 42V43 design

IFC Drawings released

FHR Pump Specs and Standards

Satisfy 2004 HAZOP action item objective

Procedure validated and posted

No technical review

42V43 hand switch moved to the "Dump" position

42V43 flare valve open and N2 valve closed

Degassing vessel full

Vortex forms in spent caustic draining into Tank 312

No vortex breaking device in 42V43

CF

Open dump valve between 42V43 and Tank 312

CF

Nitrogen hose disconnected

Nitrogen pressure not needed to start transfer

FHR Fixed Equipment Specs and Standards

CF

New utility (flare header) connection added

Degassing System (43V43) vents to flare header

No tech review trigger activated by the new 42V43 procedure

Nitrogen hose connected

Initiate spent caustic transfer into Tank 312

42V43 statisfies vortex level & velocity requirements

CF

Initiate spent caustic transfer into Tank 312

Dump spent caustic from LPG contactor into 42V43

CF

LPG used to initiate spent caustic transfer

No fixed equipment (industry) precedent PHA complete

Familiar Poly Unit process PMG requirement satisfied Procedure writers understand the Unit Process CF Control interface leve to avoid blowthrough

Plant nitrogen supply valves opened

42V43 pressurized to 30 psig

42V43 dumping procedure takes about 2 hours to complete

Procedure deviation takes about 30 minutes to complete (shortcut)

CF

Pressure spike in the degassing vessel

LPG vaporizes in 42V43 headspace

Design team input on safe system operation

7/25/05 Draft procedure e-mailed to Operations

PHA performed by the project team

Open dump valve between 42V43 and Tank 312

42V43 spent caustic siphons into Tank 312

43V43 elevation and Tank 312 internal fill line

LPG entrained in spent caustic entering 42V43

Specify min spent caustic holding time before transfer Specify valve lineup during dump - stop inlet flow

Spent caustic moves from 42V43 into Tank 312

CF

Close dump valve between 42V43 and Tank 312

Valve lineup does not allow simultaneous LPG Contactor and 42V43 dumping

Alternate valve lineup allows simultaneous LPG Contactor and 42V43 dumping (shortcut)

LPG and Flare Gas contaminate Tank 312

LPG vaporizes in 42V43 headspace

Close dump valve between 42V43 and Tank 312

Inert gas (nitrogen) not present in 42V43 headspace 42V43 headspace vapors entrained in spent caustic

42V43 hand switch returned to the "Fill" position

Spent caustic transfer procedure complete

Spent caustic transfer procedure complete

Degassing system left in "Fill" mode

Failure Date TapRooT(R) SnapCharT Rev. Page

Tank 312 Explosion and Fire 8/14/2007 TRC-2008 2 of 2

CF=Causal Factor PIO=Potential Improvement Opportunity SF=Secondary Failure

05/2007 TK312 Leak Detection and Repair (LDAR) Events

5/4/07 TK312 roof hatch fails LDAR test at 2213 ppm

08/07/07 Poly Unit Work Shut Down

8/14/2007 Penex Scrubber Dump Initiated

Dump Valve Between 42V43 and Tank 312 Opened

Penex Scrubber Dump Valve Opened

19:49:44 Pressure Spike in 42V43

Tank 312 pressure > 3.75 oz

Contractors (2 companies) leave Tank 312 area

"Propane" (mercaptan?) odor in Unit

Penex Scrubber NaOH recharge necessary

Low caustic concentration in Penex Scrubber

Procedure deviation started at 19:41

Shortcut to drain contactor and 42V43 together

B-side operator opens caustic valve to 42V43

Penex Scrubber NaOH recharge necessary Presure requirements in 43V43 satisfied

LPG in spent caustic enters 42V43

Roof hatch resealed (tightened)

Hydrocarbon in TK312

Contractors report Unit AVO condition to operators Level equalization between 42V43 and Tank 312

Night shift to dump Penex Scrubber into 42V43

Lab analysis report to Unit

A-side operator leaves TK312 to check 42V18 pH

42V43 level starts rising

Decreased Penex Scrubber caustic strength

Spent caustic siphons into Tank 312

5/17/07 TK312 roof hatch fails LDAR test at 2281 ppm

42V43 headspace vapor entrainment while dumping

Tank 312 level starts rising pH adjustment needed A-side operator walks away from 42V18 Acid (MSP) pumps installed behind totes

Operators investigate odor

42V43 dump into Tank 312 not started

Siphon into Tank 312 not initiated

LPG vaporizes in 42V43 headspace

42V43 dump into Tank 312 initiated

Roof hatch gasket replaced

Incident closed
CF

Odor source not identified

Pumps to adjust pH are located south of 42V18

A-side operator steps behind MSP totes

Process archive (PI) records 42V43 pressure increase

Transfer appears normal

5/18/07 TK312 roof hatch fails LDAR test at 981 ppm

Contractors re-enter Unit

Odor not present upon Unit re-entry

A-side operator starts MSP pump adjustment

A-side operator bends over behind MSP totes

42V43 pressure needed to dump spent caustic into Tank 312

5/22/07 TK312 roof hatch pass LDAR test at 0 ppm

Tank 312 pressure < 3.75 oz

Contractors complete work in Tank 312 area

Pressure spike in 42V43 needed Odor does not return that day. Alternate source for pressure (no nitrogen) LPG needs to be introduced into 42V43

LPG vaporization in 42V43 headspace

Compliance commitment achieved

Incident closed

CF

HC decrease in Tank 312 vapor space

LPG and Flare Gas Contaminate Tank 312

Explosive Mixture Contacts Iron Sulfide in Tank 312

Previous exposure to H2S while dumping 42V43

19:49:44 Explosion InsideTank 312

White Cloud Lifts Above Tank 312

Tank 312 Launches off Foundation

Fire Ignites in the Poly Unit

19:50 Emergency Response Coordination

8/15/07 RCFA Initiated

42V43 headspace vapor entrained in spent caustic

Vortex in spent caustic draining into Tank 312 No vortex breaking device in 42V43

Iron sulfide deposits in Tank 312

42V43 pressure & level and Tank 312 temp & level fail

Process transmissions (PI) interrupted

White smoke is generated in sulfide fires

Design relief capacity exceeded

(2) 20" roof hatches and (2) 3"X4" PSVs

Fireball lifts into the air before tank lands

EOF assembled

Tank 312 LPG concentration increase above 2%

CF

Air in Tank

Tank vacuum breakers open to avoid collapse

42V43 statisfies vortex level & velocity limits

Explosion heard throughout the refinery

SO2 generation

Pyrophoric iron sulfide combustion in Tank 312

Tank 312 separates from base (floor) weld

Frangible roof design not applicable to small (15' x 26') tanks

Fire spreads in the Poly Unit

All personnel accounted for

Tank inventory decreasing before 42V43 dump

A-Side Operator knocked down but okay

4FeS + 7O2 -> 2Fe2O3 + 4SO2

Tank 312 lands on its side, on its foundation

No secondary collisions with other assets

Fire dissipates quickly

Poly Unit Operators activate fire monitors

Iron fire in Tank 312

Tank 312 ignition source available

~40 ft away from valve between 42V43 and Tank 312

Residual ground fires remain

Firefighters In Poly Unit within three minutes

Incident preparedness

Iron sulfide deposit combustion in Tank 312

Chemical totes shield Operator from caustic and fire

Ground fires extinguished

Fire under control within 15 minutes

Drills being conducted in the refinery

EOF disbanded

Use Suction Energy To Assess Pump Reliability Risk The suction energy (SE) is determined by: SE = (De)(N)( Nss)(SG) Where: De = Impeller Eye Diameter N = Pump Speed SG = Specific Gravity of Liquid Nss = The suction specific speed = (N)(Q)0.5/(NPSHr)0.75 Where: Q = Flow at BEP NPSHr = Net Positive Suction Head Required Compare result to this guideline: If the suction energy is . . . . . . < 160 x 106 . . .* . . . < 120 x 106 . . .** . . . 160 x 106 240 x 106 . . .* . . . 120 x 106 180 x 106 . . .** . . . > 240 x 106 . . .* . . . > 180 x 106 . . .** * end suction pumps ** split case/radial inlet pumps Example (27P-106A or 27P-106B): Impeller Eye Diameter (De) = 7.2 Inches Pump Speed (N) = 3570 RPM Specific Gravity of Liquid (SG) = 0.64 (#2 FO @ 613 F) The suction specific speed (Nss) = (N)(Q)0.5/(NPSHr)0.75 = 16,200 Where: Flow at BEP (Q) = 1700 GPM NPSHr = 19 Feet The suction energy (SE) = (7.2)(3570)(16,200)(0.64) = 266 x 106 They are prone to noise, vibration, and cavitation problems if not operating close to BEP. Dont know (cant find) the Impeller Eye Diameter (De)? Its OK use the following estimation: ! End suction pump: De = Suction nozzle diameter x 0.9 ! Horizontal split case and / or radial inlet pump: De = Suction nozzle diameter x 0.75 What you need: 1. Pump data sheet 2. Head / flow curve . . .then we have . . . . . . low suction energy . . . . . . which means that . . . . . . these pumps are not prone to noise, vibration, or damage from cavitation or recirculation if not operating at or near Best Efficiency Point flow (BEP.) . . . these pumps may experience noise, vibration, and/or damage from cavitation or recirculation if not operating at or near BEP. . . . these pumps are prone to noise, vibration, and frequent repairs due to cavitation and recirculation if not operating at or near BEP.

. . . high suction energy . . .

. . . very high suction energy . . .

Failure 27P106B Thrust Bearing Failure and Unit Fire Date 15-Nov-04 TapRooT SnapCharT Rev. TRS-2008

CF=Causal Factor PIO=Potential Improvement Opportunity SF=Secondary Failure

27 Distillate Hydrotreater (DHT) Unit Construction

06/1993 27 DHT Unit Commissioned

~10/1994 Parallel 27P106 Pump Operation

11/22/94 Work Order Written for Cavitation Noise

4/27/98 Seal Leak Work Order Written

11/20/98 Thrust Bearing Failure

4/6/99 Seal Leak Work Order Written

3/30/2000 Seal Leak Work Order Written

9/3/2000 Seal Leak Work Order Written

2/20/02 Thrust Bearing Failure

10/21/02 Thrust Bearing Failure

3/26/03 Operators Request Vibration Check

9/9/03 27P106A Thrust Bearing Failure

No. 2 FO pumps

27P106 spare pump in standby mode

No. 2 FO product to storage constraint removal

Higher 27P106 discharge head needed

27P106A seal replaced

27P106B thrust bearings replaced

27P106A seal replaced

27P106B seal replaced

27P106A seal replaced

27P106B thrust bearings replaced

27P106B thrust bearings replaced

Noise at 27P106 Pumps

27P106A seal replaced

No. 2 FO productto-storage train

33 MBPD (design) rate

Rate increase to 50 MBPD

High No. 2 FO product-to-storage train dP

Bad noise at 27P106 Pumps

High vibration

Thrust and radial bearings replaced

Vibration problems

CF

Both pumps remain running

Vibration not high enough to trigger repair

1500 gpm actual flow (one) = 88% BEP

27P106 pumps operating above min (70%) BEP

dP increase at production rates above design

Unstable flow in parallel operation

Vibration monitoring completed

CF

27P106 pumps start making noise

27P106 pumps operating below min (70%) BEP

950 gpm actual flow (each) = 55% BEP

1/27/04 27P106 Vibration Check

3/00/04 Operators Request Vibration Check

9/24/04 27P106 Vibration Checked

10/15/04 DHT Unit Shut Down for TA

11/4/04 Unit Returned to Service

11/13/04 Operators Hear 27P106 "Howling Noises"

11/15/04 27P106B Thrust Bearing Failure

27P106B Seal Failure

SF

No. 2 Fuel Oil Auto-Ignition

05:38 Fire in the 27 DHT Unit

SF

05:41 Oil Mist Common Trouble Alarm

Emergency Response Initiated

Fire Extingushed RCFA Initiated

Routine vibration monitoring route

Noise at 27P106 Pumps

Routine vibration monitoring route

No work on 27P106 pumps

TA Complete

Weekend night shift

No. 2 Fuel Oil Leak 600F No. 2 FO Temp

27P106 Pump lube flow lost

Operator enters Unit

Operator sees fire in Unit

Noise at 27P106 Pumps

Both pumps remain running

Discharge line noise suspected

Elevated vibration detected

Vibration monitoring completed

Pump noise detected during normal rounds Uncontrolled shaft movement Vibration check not requested CF Loss of thrust bearing stability

Oxygen contact

Catastrophic pump failure 27TC2823 Stripper Reboiler Temperature Controller fails

Operator activates local fire monitors ti t d

Vibration not high enough to trigger repair

Monitoring frequency increased

Vibration monitoring completed

CF

Turnaround WO not written

False reading suspected

"Bad" noise considered "normal" noise

Conduit damage

SF

Vibration increase measured

Both pumps remain running

CF

CF

Both pumps remain running

Process flow changes

27P106 operating history

Both pumps remain running

CF

Console Operator calls Unit

Control switches to manual control

Operator Suffers Respiratory Distress

PPE not increased

CF

Vibration monitoring completed

Unit in transient operating condition

Auto control capability lost

Firefighters arrive in Unit