Detailed Incident Report-IR3

To be Sumitted by: Due Date: Report Completed

Incident ID Process Safety Incident at Urea/Utilities-1 PS-2014-52

Date of Incidient 12-03-2014 Time of Incidient 11:00

Titl Total site black out on DCS malfuntion

e
Unit Urea/Utilities-1
/
Are
a of
Inci
den
t
Eml EFERT Contractor Both None
poy
ee
Bel
ong
s to

Incident Category:
1 a) Actual or Potential Injury/Occupational Illness:
2 b) Process Safety incidents:
3
c) Miscellaneous incidents:

Process Release M / Equipment Failure

Human Error Instrument / Electrical Failure
Inoperable Safety Equipment Fire
Violation of PSRM Systems Teachnical Failure
Operations Failure Other
Des On 12th March 2014 at about 11:25 AM, PRC-2A and 2B closed causing GT-601 and 602 to ramp down. GT-604
crip
picked up load but its main breaker tripped due to under-frequency. GT-604 later tripped on high exhaust temperature
tion
of
spread. HRSG-611 FTO occurred. GT-601 and 602 failed to sustain power due to fuel valve closure resulting in total
Inci site black-out.
den
t:
Wh Before this incident, total site load was 38.4 MW with GT-601, 602 and 604 running at a load of 7.8 MW, 11.8 MW
at
and 18.8 MW respectively. As a result of this incident, Plant-1 and Urea-3 tripped immediately. On Amm-3, I-2 was
Res
ulte
actuated. After about 15 minutes, HRSG-611 tripped on malfunction of PIC-618AV positioner. This resulted in steam
d& failure and actuation of I-1 at Plant-2. After 50 seconds from fault initiation, Uty-1 DCS (PCU-3) normalized. Plant
Acti start-up was initiated.
on
Tak
en?
Ple Normal Operation Chemical Handling
ase
tick
Plant Satrtup Material Handling
the Plant Shutdown Cleaning
Bas Lifting (Mech) Maintenance
ic
Acti Lifting Manual Loading / Unloading
vity Commissioning Others
k Plant Shutdown Cleaning
the Lifting (Mech) Maintenance
Ba
sic Lifting Manual Loading / Unloading
Act Commissioning Others
ivit
y
in
Pro
gre
ss
at
the
tim
e
of
inc
ide
nt:
1. Goals & Objectives 6. Progressive Motivation
2. High Standards of Performance 7. Two-Way Communication
3. Supportive Safety Personnel 8. Effective Management Safety Audits
4. Safety as a Line Organization Responsibility 9. Transport & Driving Safety
5. Integrated Organization for Safety None
Se Before this incident, total site load was 38.4 MW with GT-601, 602 and 604 running at a load of 7.8 MW, 11.8 MW and
qu 18.8 MW respectively.
en
ce The event started with the shifting of various control loops on Uty-1 from AUTO to MANUAL mode with outputs held at
of -6.25%. This problem occurred on loops configured on PCU-3. It is noteworthy that none of these observations were
Ev recorded on PCU-1 (Urea-1), PCU-7 (HRSG-611) and PCUs at Urea-2.
ent
s:
At the same time, the upstream fuel gas supply pressure valves of GT601 and GT602; PRC-2A and PRC-2B
respectively (which are configured on PCU-3), started to close. As a result, GT-601 and GT-602 speed started to come
down causing these GTs to shed load. Consequently, GT-604 started to pick up this load upto 25.23 MW in 08
seconds. However, GT-604 Main Circuit Breaker tripped on under frequency. This was also observed by a dip in
frequency on the grid. GT-604 later tripped on High Exhaust Temperature Spread Alarm.

As GT-604 tripped, Flying Take-over of HRSG-611 occurred successfully from TEG to AA mode. However, HRSG-611
later tripped due to malfunction of fuel gas valve positioner (PIC-618AV)/Low air-pressure security (PIC-618AV
positioner was found faulty during subsequent start-up). Following GT-604 main circuit breaker trip, ILMS shed load as
per its logic. However, since PRC-2A and 2B were closing, GT-601 and 602 failed to normalize the system frequency.
GT-601 and 602 tripped on ‘Turbine Under-speed’ resulting in total site black-out.

Following is the time-stamped sequence of events:

T-0 = 11:25 AM

T= 0 s
1. Power supply failure in Bailey PCU-3
2. The generation of PFI (explained later) in Bailey causing the closure of PRC-2A and PRC-2B; fuel valves of
GT601 and Gt602 respectively.
T=15 s
3. Low fuel gas supply pressure alarms on GT601 and GT602.
4. Ramping down of GT601 and GT602.
5. GT604 tried to pick up the load lost by GT601/602
6. GT604 load reaches to a maximum of 25.2 MWs
 GT601 and Gt602 respectively.
T=15 s
3. Low fuel gas supply pressure alarms on GT601 and GT602.
4. Ramping down of GT601 and GT602.
5. GT604 tried to pick up the load lost by GT601/602
6. GT604 load reaches to a maximum of 25.2 MWs
T=23 s
7. Breaker of GT604 trips
8. The entire load now shifted on GT601/02
9. Since GT601 and GT602 are already ramping down, their breaker immediately trips
10. GT601/02 lose their minimum operating speed
T= 34 s
11. Turbines of GT601 and GT602 also trip
12. Turbine of GT604 trips on high exhaust spread.
Ne
w
Se
qu
en
ce
of
Ev
en
ts:
In The investigation was carried out based on Event logs, DCS trends, DCS board-man feedback and visual inspection of
ve
the hardware. The process is explained below:
sti
ga
tio A. UTY-1 DCS:
n/
Int Uty-1 DCS is a hybrid of Bailey INFI-90 DCS as the back-end and Honeywell Experion as the front end.
er Observations:
vie Following observations were recorded from hardware visual inspection, DCS trends, event logs and DCS board-man
ws feedback:
:
(If
ap DCS Trends:
pli 1. On 12th March, ’14 around 11:25 am, the fuel valves (PRC-2A, PRC-2B) closed causing low fuel flow to GTs and
abl eventually resulted in their tripping. These valves are configured on Bailey PCU-3.
e)
DCS board-man feedback:
1. The DCS operator confirmed that at the time of tripping, various DCS PID controller faceplates shifted from AUTO to
MAN mode with outputs held at -6.2%. This observation was recorded on both Bailey OIS and Experion HMIs.
2. This problem occurred for majority loops on PCU-3. It is noteworthy that none of these observations were recorded
on PCU-1 (Urea-1), PCU-7 (HRSG-611) and PCUs at Urea-2.

Visual Inspection of hardware:

1. Visual inspection of PCU-3 cabinet was performed 20 minutes after the incident. All NIS/NPM, MFP, Analog
Input/Output and Digital Output cards health LED was found Green. There is no health LED available on Digital Input
card. Both MFPs in PCU-3 (which are MFP302 and MFP304) were working fine with primary with 02 LEDs (7 & 8) lit
red and stand-by with 01 LED i.e. 8 lit red. The other cabinets i.e. PCU-1 and PCU-7 were also found ok.
2. A fuse of 3A rating installed on 01 Digital Input Termination board was found blown.
3. On PCU-3 Module Power Supply – II (MPS-II), voltage levels were recorded and found within acceptable range.
4. On the Power supply modules, the LEDs were found Green and healthy as well.
Visual Inspection of hardware:
1. Visual inspection of PCU-3 cabinet was performed 20 minutes after the incident. All NIS/NPM, MFP, Analog
Input/Output and Digital Output cards health LED was found Green. There is no health LED available on Digital Input
card. Both MFPs in PCU-3 (which are MFP302 and MFP304) were working fine with primary with 02 LEDs (7 & 8) lit red
and stand-by with 01 LED i.e. 8 lit red. The other cabinets i.e. PCU-1 and PCU-7 were also found ok.
2. A fuse of 3A rating installed on 01 Digital Input Termination board was found blown.
3. On PCU-3 Module Power Supply – II (MPS-II), voltage levels were recorded and found within acceptable range.
4. On the Power supply modules, the LEDs were found Green and healthy as well.

DCS Event Logs:

1. Event Log was retrieved from Bailey OIS. Please refer to Att # A for details. Following are the key-findings from DCS
log:
a. The log shows that all exception reporting blocks (DO/L, AO/L, STN, RMSC etc.) went into error state.
b. The Status tags e.g. STAT-1030426 etc. which represent health of the I/O module went to ZERO state.
c. MFP302 went to ‘Failed’ state at 11:26:05 am while MFP304 went to ‘Failed’ state at 11:26:17 am. Later, both of them
returned to ‘M’ state at the same time.

B. GT-601 TRIPPING:

All the time stamps mentioned in this section are with reference to the time of GT604 Mark-6 Control room HMI.

Before tripping of GT601, it was loaded with 7.8 MWs of load; its Fuel Stroke Reference (FSR) was at 50.92 % while
Gas Control Valve hydraulic oil pressure feedback was 121.2 psi conforming to the FSR value. As can be seen from the
trip log GT601, the first alarm to appear was of GAS FUEL PRESSURE LOW triggered by the signal L63FGL and
indicates the dropping of fuel gas pressure below the alarm set point.

This alarm is followed by FUEL GAS SUPPLY PRESSURE LOW which is triggered by the signal L63FG2L, which is the
trip signal to the gas turbine.
NOTE: Upon looking this signal name in the Mark-V IO assignment file, it is confirmed that this is a trip signal:
L63FG2L QD1-DTBA-033 Gas Fuel Supply Pressure Low (trip) [63FG-2]

Finally, GENERATOR BREAKER TRIPPED alarm appeared, clearly indicating opening of breaker.

C. GT-602 TRIPPING:

All the time stamps mentioned in this section are with reference to the time of GT604 Mark-6 Control room HMI.

Before tripping of GT602, it was loaded with 11.8 MWs of load; its Fuel Stroke Reference (FSR) was at 65.8 %. As
opposed to GT601, the value of Gas Control Valve hydraulic oil pressure feedback is not included in the trip history file
for GT602.

Similar to GT-601, it can be seen from the trip log GT-602 that the first alarm to appear was of GAS FUEL PRESSURE
LOW followed by FUEL GAS SUPPLY PRESSURE LOW LOW, thus tripping the GT602 at the same time as GT601.

Finally, GENERATOR BREAKER TRIPPED occurred (exactly at the same time the breaker of GT601 tripped).

D. GT604 Tripping

Before site blackout, GT604 was running at a load of 18.8 MW while its FSR value was at 34.6 %.

The Control System Toolbox of GT604 logs every event and marks it as a triggering point in its trip historian. However,
by observing its normal logging operation it is also observed that it skips logging of repetitive data after a certain
amount of time.

The same observation is made while reading the sequence of events of GT604, that there is no relevant data before
11:26:10. This shows that there occurred no significant or new event during that time period unless recorded.

The first three alarms in event log of GT604, namely FUEL GAS NOZZLE PRESS.SENS.HIGH DEV./TROUBLE ALM
triggered at 11:26:10, Gearbox proximity probes trip multiply funtion cmd triggered at 11:26:11 and LUBE OIL FILTER
DIFF.PRESS.HIGH ALARM triggered at 11:26:13 are recurrent in nature.
 D. GT604 Tripping

Before site blackout, GT604 was running at a load of 18.8 MW while its FSR value was at 34.6 %.

The Control System Toolbox of GT604 logs every event and marks it as a triggering point in its trip historian. However,
by observing its normal logging operation it is also observed that it skips logging of repetitive data after a certain amount
of time.

The same observation is made while reading the sequence of events of GT604, that there is no relevant data before
11:26:10. This shows that there occurred no significant or new event during that time period unless recorded.

The first three alarms in event log of GT604, namely FUEL GAS NOZZLE PRESS.SENS.HIGH DEV./TROUBLE ALM
triggered at 11:26:10, Gearbox proximity probes trip multiply funtion cmd triggered at 11:26:11 and LUBE OIL FILTER
DIFF.PRESS.HIGH ALARM triggered at 11:26:13 are recurrent in nature.

The next alarm in the sequence; GENERATOR BREAKER TRIPPED ALARM triggered at 11:26:16 shows the tripping of
the breaker. This alarm is signified by the signal L52GX changing its state from 1 to 0.
The turbine finally trips at 11:29:56; Master Protective Signal (L4_EVT=0).

By looking at the trends of GT604 it is seen that the turbine was running normally at a steady load of 18.8 MW until
11:26:07, when it starts gaining load. By 11:26:15 it reaches to a max of 25.23 MW when the circuit breaker trips. In
approximately 8 seconds GT604 picks up a load of 6.5 MW. At the time of tripping, its FSR had a value of 39.9 % while
the turbine speed (TNH) had ramped down to 96.7 %.

K
ey
i. Uty-1 DCS:
fi The observations mentioned earlier ascertain that the problem originated in PCU-3. Since upstream fuel
n valves of GT-601, 602 and 603 and associated HRSGs, SGs, steam system and other utilities are present on
di this PCU, it has become the life line for both plant-1 and 2.
n
g
s: What happened in PCU-3?
(L Based on the discussion with ABB Pakistan (OEM), Free-lance expert opinion by an ex-ABB Project
is Manager, system configurations and system manuals, it was ascertained that a Power Fail Interrupt (PFI)
t
in
was generated in PCU-3 which triggered the entire event. The entire event in PCU-3 is elaborated as follows:
lo
gi a) PFI alarm was generated
ca
l
b) A low voltage surge (below threshold) in system power supplies (+5/ +15/ -15 VDC) or I/O power supply
or
d (25 VDC) from MPS-II to system bus in PCU1, caused the PFI alarm.
er
s, c) PFI alarm caused to stop all the master modules (NPMs & MFPs) in PCU-3.
p
er
ti d) Low voltage serge in MPS-II was momentary and PFI alarm was not latched in IPMON01 module, so all
n master modules went in RESET (STOP >> RUN) mode.
e
nt e) When master modules (NPMs & MFPs) went in STOP mode, all associated Digital & Analog outputs went
fa
ct
in "Last State/ Value HOLD" state (including analog output to fuel PCVs for GTs), as configured in related
s function codes.
u
n f) But, when master modules (NPMs & MFPs) went in RUN mode (due to non-latching of PFI alarm in
c
MPS-II), it caused the MFPs to go in normal initialize then Execute mode, which resulted in initial zero value
o
ve to analog outputs (including analog output to PCVs for GTs). MFPs returned in Execute mode after 50
re seconds. After this, entire PCU-3 system restored and became fully operational.
d
in
th What is PFI?
e As per Bailey INFI-90 system manuals, PFI stands for Power Fail Interrupt. PFI is a signal that causes active
in
 controller or processor modules (MFCs or MFPs) to reset and the communication system to be bypassed,
ve
st when generated in the INFI 90 OPEN system by an out of tolerance bus voltage. These bus tolerance
ig voltages are mentioned below:
at
io
n)

The existing Modular Power Supply (MPSII) in Bailey PCU-1 & 3 (installed at URUT-1) and PCU-4 & 5
(installed at Urea-2) has a feature that if any of the system bus voltages (5V, +15V) or I/O system voltage
(25.5V) falls below pre-defined threshold voltage, it generates a Power Fail Interrupt (PFI) signal which puts
the MFP (Bailey controller) into HALT mode and cause Analog Outputs to go into default state (Configured as
Last Value Hold in our system).

Configuration options available for PFI:

For PFI generation, the standard options available in the INFI-90 system are:

a. PFI configuration in Latching/Non-latching:

If PFI is configured in Non-latching state, then whenever the PFI is generated, it causes MFP to Reset
causing Analog Output to go to ‘Zero’ value. The system then normalizes in around 1 minute. This is what
happened in PCU-3 in site black-out.

If PFI is configured in Latching state, then whenever the PFI is generated, it latches and causes MFP to
HALT and Analog Output values to go into Last Value Hold. When PFI bit is manually reset from IPMON01
monitor on MPSII, it will cause MFP to Reset causing Analog Output to go to ‘Zero’ value. The system then
normalizes in around 1 minute.

b. PFI generation configuration:

Options available for PFI generation include:

1. PFI generation on 25.5 VDC bus fault (Field I/O system voltages)
2. PFI generation on 5 VDC and 15 VDC (Module System voltages):

In Engro’s case, following PFI configurations were found:

· PFI configuration is NON-LATCHING
· PFI generation is configured on 5 VDC, 15 VDC and 25 VDC

Keeping in view the PFI configurations in Engro’s INFI-90 system, the entire can be narrated as follows:

What are the causes of Voltage Surge in PCU-3?

The major causes of voltage surge are:

1. Modular Power System – II (MPS II) Failure:

In PCU-1, 3, 4 and 5 Modular Power System – II (MPS-II), IPSYS01 modules are used for system voltage (5
VDC, +15 VDC) and I/O system voltage (25.5 VDC) generation. 04 IPSYS01 modules are used in each
MPS-II power system
Failure in power supplies can be due to:

· Ageing:
As per Bailey INFI-90 system manual, recommended replacement cycle of IPSYS01 power modules is 05
years.

From maintenance records, it was found that 02 IPSYS01 modules in PCU-3 MPS-II were replaced in 2011.
The other 02 IPSYS01 modules in PCU-3 were of old vintage. No other maintenance record could be found.
Hence, it appears that these 02 modules were in-use since system commissioning.

· Non-friendly Environment:
The standard environmental conditions for MPS-II power system are as follows:

Environm
Temperatu

Operating e

Storage an
 Relative hu

Air quality

The actual condition of the Power supply modules and PCU-3 is depicted in following pictures:
Overall, environment of PCU-3 cabinet was found deplorable. Although, Uty-1 DCS Room is a temperature
controlled environment, yet dust ingression is inevitable due to cable trenches and improper sealing of doors.

As evident from pictures above, a lot of dirt was observed on the IPSYS01 modules and IPMON01 of PCU-3
Power Supply. Main reason for this dirt is non-sealing of cabinet bottom.

It is highly probable that dirt caused reduced life of the capacitors and other electronics on IPSYS01
module which resulted in a voltage dip on power bus and subsequent generation of PFI.

2. Short-circuiting in I/O system power (25VDC) distribution:

Each TU is provided with a fuse which limits the current from 25VDC supply connected in daisy chain. If a
fuse blows on one of the TUs, power supply still remains available on other TUs.

A 3A fuse was found blown on one of the Digital Input Termination Units. This can be due to following
reasons:

· A short-circuit in field circuitry.

· Problem in Digital Output card.

Since, plant was soon started-up after the incident; the Digital Input card could not be inspected. Visual health
of field cables was verified and found to be ok.

Could this be detected in Preventive Maintenance?

There are 02 types of Preventive Maintenance done on Bailey INFI-90 DCS.

1. Online Maintenance – Limited to visual checks. Frequency: 06 months.

2. Shutdown Maintenance – Limited to redundancy switch over of controllers and communication modules.
Frequency: 02 years.

From verbal references, it was reported that cabinet de-dusting activity of PCU-3 was last conducted in 2011.

Maintenance history shows that last Online PM was done on 18-Nov-11. Last PM3 for Uty-1 DCS in SD was
done in 4-Aug-11. The last shutdown work-order was initiated in Oct ’13.
Since, it was a momentary dip in voltage, it couldn’t be detected in a PM. However, there is a need to
improve the on-line PM procedure for noting voltage levels on IPMON01 monitor and proper cleaning of
cabinet filters.

ii. Site-Black Out:

GT604 Under/Over Frequency Protection
When the prime mover speed of a generator decreases, there is a decrease in output frequency along with
an increase in the excitation current. This results in over-heating of the generator core: a phenomenon called
over-fluxing. To avoid this scenario, there are two protections available in Gas Turbine Generators.

1. Under-frequency protection at the Generator, which trips the Generator Breaker only
2. Under-speed protection at the Turbine, which trips both, the Generator and the Turbine as well

For GT601, GT602 and GT603, there is no frequency protection and the system is completely dependent on
the turbine under/over-speed protection. Generator & Turbine OEMs, Brush Electric & GE respectively, have
recommended over and under frequency protections for GT604. At the time of the subject incident, the
under-frequency settings of GT604 were as follows:

Frequency Delay
49.0Hz 5.0s
48.5Hz 0.5s

The over speed scenario is far less critical for the Generator, however, the same is not true for the Turbine.
To avoid this scenario, there are two protections available in Gas Turbine Generators.

1. Over-frequency protection at the Generator, which trips the Generator Breaker only
2. Over-speed protection at the Turbine, which trips both, the Generator and the Turbine as well

At the time of the subject incident, the Over-frequency settings of GT604 were as follows:

Frequency Delay
51.0Hz 5.0s
51.5Hz 0.5s

Revision of GT604 under/over-frequency settings

After discussion with our Network Consultant, Dr. Voelzke (Siemens Germany), we have evaluated that the
present over/under-frequency settings are quite stringent and need to be revised. The new settings are
determined as following:

New over-frequency Settings New under-frequency Settings

Frequency Delay Frequency Delay

52.0Hz 5.0s 48.0Hz 5.0s
52.5Hz 1.0s 47.5Hz 1.0s

Although these new relaxed settings will give GT604 more time to react to anomalies in the Electrical
Network, this would not have prevented a Black Out following this incident due to the extremely
detrimental effects of closing GT601 and GT602 fuel valves. Further relaxation in under-over frequency
protection settings is not possible since these settings are now close to the turbine over/under speed
protection settings.
GT604 Tripping
As is evident from G60 (Protection Relay for GT604) logs, GT604 Generator Main Breaker tripped on
under-frequency. The system frequency is recorded to have dropped as low as 48.4Hz, which is below the
trip settings. The reason for this drop is explained through the ILMS load trends. As can be seen in the MW
graphs, the load on GT601 and GT602 is gradually decreased due to closing of fuel valves of both turbines.
As a result, this load appears on GT604. The maximum load recorded on GT604 is approx. 25MW which is
beyond its load limit. The frequency of GT604 subsequently dropped and the Generator Breaker Tripped on
under-frequency. Since the fuel valves of GT601 and GT602 were already closing, both these turbines fail to
sustain the electrical load leading to Blackout.

As soon as GT604 is isolated from the Electrical Network, the turbine speed increases and the normal
Generator frequency of 50Hz is restored. However, after 3m40s the turbine trips as well. The cause of turbine
tripping is not visible in G60 or ILMS data.

Fig1. GT604 Protection Relay, G60, Event Log

Fig2. ILMS Trends - GT601, GT602 and GT604 MW Loads and GT604 Frequency
Note: Sampling rate for ILMS trends is 5 seconds. Although the graphs don’t explicitly show the chronological order of
events, they do depict the trends quite accurately.

Load Shedding Schemes

There are primarily two Load Shedding Scheme Design Philosophies:

1. Event Based Load Shedding Scheme

An Event based Load Shedding Scheme identifies the tripping of a Generating Unit through Status contacts
of the Main Generator Breaker. Whenever, one or more Generator Breaker trips, appropriate load is shed to
maintain the balance between available generation capacity and power demand.

2. Frequency Based Load Shedding Scheme

An Event Based Load Shedding monitors the network frequency constantly. Whenever frequency degrades
due to any reason (such as GT trip, fuel valve closing, load rejection by any GT, etc), appropriate load is
shed to stabilize the network. This scheme does not rely on the occurrence of an event, however, it is
reactive i.e. It waits for the network frequency to degrade before taking any actions.

ILMS Operation

ILMS utilizes an Event based Load Shedding Scheme since it is not possible to implement a frequency based
scheme in present Electrical Network due to its complexity causing power transfer limitation through
Reactor-1. Due to this restriction, each GT tripping scenario has to be individually studied and tweaked to
ensure a permissible power transfer through Reactor-1. As a result load shedding can’t be independent of
Events as long as this restriction is there.

During closing of GT601 and GT602 fuel valves, the Generator breakers for both GTs did not trip. Hence, as
per scheme, no action was taken by ILMS. The first event logged by ILMS is the tripping of GT604.
Appropriate loads were shed by ILMS accordingly, as is evident from the Event Logs. Soon after this load
shedding, the two other turbines, GT601 and GT602, also shutdown due to closing of fuel valves, resulting in
a Blackout. ILMS accordingly tripped/opened all MV breakers as per logic.

ILMS Modifications
Although a frequency based load shedding scheme would eliminate the need to rely on the occurrence of an
Event for load shedding, it is not possible to implement such a scheme in present Electrical Network due to
its complexity.

A project to replace existing Reactor-1 with new increased capacity reactor is in progress. Major scope of the
project has already been completed. The remaining scope requires EE620 Bus-3 outage. This is only
possible during complete Urea-3 outage and partial Urea-2 outage. Window for this modification has been
allocated during planed outage of Plant-2 from 4th-20th May.

Once the new reactor is taken in service, the next phase would be to implement a new ILMS scheme which
will couple frequency based load shedding with Event based load shedding. As per this new scheme, if
frequency of the Electrical Network degrades for any reason (e.g. GT tripping, fuel valve closing, load
rejection by any GT), ILMS will shed appropriate load to restore the frequency of the Network even if no Gas
Turbine trips. Although, if an event occurs, ILMS will not wait for the frequency to degrade and shed load
proactively Designing such a scheme involves quite a few complexities since it is tricky to determine which
scheme is to act in different scenarios. It is however, expected that current issues caused by Turbines will be
better handled with the new scheme.

GT-604 Low Low Fuel Pressure Security:

Low Low Fuel Pressure Alarm was observed in GT-601 and 602 Alarm summary. Approx 7 seconds after this
alarm, GTs tripped on Turbine under-speed which was taken as an event by ILMS. Moreover, by analyzing
trends of GT-604, it is observed that GT-604 approx same duration to pick-up load from 18.8 MW to 25 MW.

Upon looking in Mark-V logics of GT-601, 602 and 603, it was found that signals are not coming in the trip
logic. It is much possible that if the GT-601 and 602 had tripped on Low Low Fuel Gas Pressure, GT_604
tripping could have been saved.

ROOT CAUSE ANALYSIS:

C Based on above key-findings and analysis, it can be concluded that:
o
n
cl 1. The site black-out originated due to Uty-1 DCS (PCU-3) power-system failure due to voltage dip/surge in its
u power supply modules which caused MFPs to reset and Analog Outputs to ‘ZERO’ value. As a result, PRC-2A
si and PRC-2B closed. The causes of this failure are:
o
n: · Ageing of power-supplies
· Moisture/dirt on power-supply modules causing degradation of capacitors and other electronics on the printed
circuit board
· Short-circuiting in I/O system power distribution or Digital Input Module causing the voltage dip

2. The black-out occurred due to ‘Event based’ reaction of ILMS instead of a hybrid of “Event” and ‘Frequency’
based design. ILMS only responded when GT-604 breaker tripped on under-frequency. Even in this case,
black-out was inevitable due to tripping of GT-601 and 602 on under-speed owing to closure of PRC-2A and
PRC-2B.

Recommendations Job Type Responsibility Target Date

Evaluate & change PFI settings from Non-Latching to Latching in URUT-1 and Urea-2 TA - Turnaround Jehangir Alam Khan,Saad 12/31/2014
DCS Afzaal Shamsi

Remove PFI generation from 25.5 VDC. PFI to be kept for 5 V, +15 V due to requirement TA - Turnaround Jehangir Alam Khan,Saad 12/31/2014
in controller and communication cards Afzaal Shamsi

Ensure foaming is provided on bottom plinth for proper sealing of URUT-1 DCS Room SJ - Simple Job Jehangir Alam Khan,Saad 10/31/2014
cabinets Afzaal Shamsi

As a short term solution, addition of frequency based tripping of individual breakers TA - Turnaround Muhammad Imran 09/30/2015
on ILMS to be evaluated and implemented Khaliq,Muhammad Zaghum
Riaz

Ask GE about the slow response of FSR of GT604 upon addition of load EJ - Engineering Muhammad Hamza Khan 01/31/2015
Job

Process engineering to Evaluate providing Low Low Fuel Gas Pressure as a TRIP for EJ - Engineering Masab Javed 12/31/2014
GT-601/2/3/4 wherever applicable Job

On UREA-2, upgrade MPS-2 Power Supply to MPS-3 Power-supply. MPS-2 has phased TA - Turnaround Hira Shaukat 03/31/2016
out 6 to 8 years ago and has a history of failure in industry

On URUT-1 evaluate shifting from MPS-2 to MPS-3 power supply till the time Yokogawa TA - Turnaround Jehangir Alam Khan 06/30/2015
DCS comes on-line. Else, install re-furbished MPS-2 power-supply modules in URUT-1
DCS
Accountability
Measures
Taken:
Date of 12-03-2014 Rep 15-09-2014
occurrence of ort
incident due
on
# Extended Create Date Update By Old Date Extended
Date

Reasons for
sending the
report late (If
applicable)

Report prepared Date 13-03-2014

by:
R
e
p
or
t
to
b
e
fil
le
d
b
y:
In Completed
ci
de
nt
St
at
us
T Saad Afzaal Shamsi/EFERTDHK/ENGRO,
h
e
in
ci
d
e
nt
to
b
e
in
ve
st
ig
at
e
d
b
y:
C 10/17/2014 12:00:00 AM
o
m
pl
et
io
n
D
at
e:
Di
st
ri
b
ut
io
n
of
th
e
R
e
p
or
t:
At
ta
c
h
e
m
e
nt
/
R
ef
er
e
n
ce
R
e
p
or
ts
A
d
d
R
MDI
e
====================
m
It was discussed in last Manufacturing HSE to present the report in 1st week of May. Would request
a
extension till 15-Sep so that final report can be uploaded on MIS by then.
r
k
Best regards,
s
SASh
R
e
p
e
at
I
n
ci
d
e
n
t

3/13/2014 7:55:48 AM: Detailed Incident Report Routed to CN=Muhammad Imran Khaliq/OU=ECPLDHK/O=ENGRO and by
Mohammad Ismail
10/11/2014 7:32:21 PM: Detailed Incident Report Routed to CN=Mohammad Ismail/OU=ECPLDHK/O=ENGRO and CN=Arif
Jalil/OU=ECPLDHK/O=ENGRO by Saad Afzaal Shamsi
10/17/2014 3:17:23 PM: Incident Report Closed by Azmat Hayyat Bhatti
10/17/2014 3:18:09 PM: Incident Report Closed by Azmat Hayyat Bhatti