You are on page 1of 55

FINAL REPORT ON THE AUDIT OF THE

ETIWANDA POWER PLANT

CONDUCTED UNDER GENERAL ORDER 167


TO DETERMINE COMPLIANCE WITH
OPERATION, MAINTENANCE, AND LOGBOOK STANDARDS

ELECTRIC GENERATION PERFORMANCE BRANCH


CONSUMER PROTECTION AND SAFETY DIVISION
CALIFORNIA PUBLIC UTILITIES COMMISSION
505 VAN NESS AVENUE
SAN FRANCISCO, CA 94102

March 27, 2008

Richard W. Clark, Director


Consumer Protection and Safety Division
ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Executive Summary .......................................................................................................... 4


Introduction....................................................................................................................... 6
Plant Description............................................................................................................... 6
Plant Performance ............................................................................................................ 7
Audit Scope........................................................................................................................ 8
SECTION 1 FINDINGS ................................................................................................. 12
SECTION 2 FINDINGS ................................................................................................. 13
Finding 2.1. The Plant Fails to Establish and Follow Clear Test Procedures for
Turbine Overspeed........................................................................................................ 13
Finding 2.2. The Plant Delayed Evaluation of Needed Maintenance........................... 14
Finding 2.3. The Plant Fails to Maintain Proper Hydrazine Levels in the Condensate
and Feedwater. .............................................................................................................. 16
Finding 2.4. Work Management System Contains Obsolete Work Orders and Conflicts
with the Plant’s Maintenance Plan................................................................................ 18
Finding 2.5. The Plant Fails to Identify and Correct Hazards to Foot and Vehicle
Traffic. .......................................................................................................................... 19
Finding 2.6. The Plant Fails to Assure the Calibration of All Test Equipment........... 22
Finding 2.7. The Plant Lacks Checklists for Some Daily Inspections......................... 24
Finding 2.8. The Plant Fails to Maintain Calibration Logs for the Equipment that
Monitors Feedwater Chemistry..................................................................................... 26
Finding 2.9. The Plant Fails to Follow Chemistry Lab Housekeeping Procedures..... 29
Finding 2.10. The Plant Fails to Mark Evacuation Areas............................................ 31
Finding 2.11. Plant Security Entered an Incorrect List of Visitors on Site in Plant Gate
Log. ............................................................................................................................... 32
Finding 2.12. A Plant Report Describes Abandoned Fire Hose Stations as “Missing.”
....................................................................................................................................... 32
SECTION 3. OBSERVATIONS.................................................................................... 33
Observation 3.1. Reliant Regularly Audits its own Plants and Rates its own
Performance. ................................................................................................................. 33
Observation 3.2. The Plant Maintains Extensive Procedures and a List of Approved
Contractors.................................................................................................................... 35
Observation 3.3. The Plant Bases Employee Compensation on Performance
Evaluations.................................................................................................................... 36
Observation 3.4. The Plant Thoroughly Investigated Turbine Bearing Failures......... 37
Observation 3.5. The Plant Maintains and Uses a Detailed Procedure for Root Cause
Analysis......................................................................................................................... 38
Observation 3.6. According to a Detailed Procedure, the Plant Tracks Important
Inventory by Project...................................................................................................... 39
Observation 3.7. The Plant’s Document Control Room is Neat and Organized. ......... 39
Observation 3.8. General Housekeeping and Equipment Labeling are Good. ............. 40
Observation 3.9. The Plant Keeps Predictive Maintenance Reports. .......................... 42
Observation 3.10. The Plant Has Improved Its Procedures and Technical Support for
Contract Work............................................................................................................... 43
Observation 3.11. The Plant Utilizes a Training Program............................................ 44
Observation 3.12. The Plant Provides Security for the Plant and Personnel................ 45

Audit Number GO167-1006 2


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
Observation 3.13. The Plant Follows Tagout Procedures and is Implementing a
Lockout Procedure. ....................................................................................................... 45
Observation 3.14. The Plant Tests its Fire System Annually. ..................................... 46
Observation 3.15. The Plant Held an Evacuation Drill and Implemented Changes as a
Result. ........................................................................................................................... 46
Observation 3.16. The Plant Holds Daily Staff Meetings. ........................................... 47
Observation 3.17. The Plant’s Chemistry Lab Procedures are Comprehensive. ......... 48
Observation 3.18. Auditors Found no Discrepancies between Plant Logs and other
Reported Data. .............................................................................................................. 50

TABLE OF FIGURES
Figure 1. Water creates hazard in walkway. .................................................................... 20
Figure 2. The plant added non-slip surface to the walkways........................................... 20
Figure 3. The plant added calibration stickers on equipment. ......................................... 27
Figure 4. The plant left clutter in the chemistry lab.......................................................... 29
Figure 5. The plant used the fume hood for storage. ....................................................... 30
Figure 6. The plant cleaned and placed a warning sign on the fume hood...................... 30
Figure 7. The plant added signs to evacuation areas. ...................................................... 31
Figure 8. The plant uses secondary containment for hazardous waste. .......................... 40
Figure 9. The plant cleaned and placed barricade tape around Units 1 and 2. ................ 41

Audit Number GO167-1006 3


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Executive Summary
This is the Final Report from the audit of the Etiwanda Power Plant prepared by the
Consumer Protection and Safety Division (CPSD) of the California Public Utilities
Commission (CPUC). CPSD audited the plant for compliance with the Commission’s
General Order 167, which includes Operation, Maintenance, and Logbook Standards for
power plants.

On May 5, 2006, CPSD notified Etiwanda of the audit and requested pertinent documents
and data. CPSD staff visited the plant site from June 4, 2006 through June 8, 2006 in
order to observe plant operations, inspect equipment, examine documents, and interview
plant staff and managers. The CPSD Audit Team reviewed specific incidents and
problems in plant operation to evaluate whether the plant needed improvements in
operation and maintenance processes, training, or policies. The Audit Team also
examined whether the plant’s programs and procedures met various Operation,
Maintenance, and Logbook Standards.

CPSD sent Etiwanda a Preliminary Audit Report on July 13, 2007. Etiwanda responded
with a Corrective Action Plan (CAP) on August 21, 2007. CPSD and Etiwanda resolved
all outstanding issues in an October 23, 2007 conference call. The audit team leader
visited the plant on December 12, 2007 to verify completion of several corrective actions
and clarify some minor issues. Etiwanda cooperated fully throughout the entire audit
process. CPSD now issues this Final Report.

In the Preliminary Audit Report, CPSD identified 13 potential violations of various


standards. Auditors found no safety hazards requiring immediate action. Upon further
review, CPSD changed one potential finding to an observation. Reliant submitted a
Corrective Action Plan which agreed with many of staff’s recommendations. As
described below, CPSD and Reliant agreed on a CAP that will resolve all major issues.

Finding 2.1. CPSD found that the plant failed to establish and follow clear test
procedures for turbine overspeed. In response, the plant rewrote the procedures,
requiring annual full overspeed tests according to clear step-by-step instructions.

Finding 2.2. CPSD found that the plant deferred needed maintenance prescribed
by the plant’s own equipment condition assessment. In response, the plant hired a
new manager for the program, developed a new schedule and procedure, and
performed much of the high priority work from the original schedule. By
December 31, 2008, the plant agrees to provide CPSD with repair records for all
the original high priority items and all boiler and piping repairs listed on the
original schedule.

Audit Number GO167-1006 4


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
Finding 2.3. CPSD found that the plant fails to maintain proper hydrazine levels
in the condensate and feedwater during normal operations. In response, the plant
is refurbishing the entire chemistry lab by installing a system which continuously
monitors water chemistry. CPSD requests that the plant provide photos of the
refurbished lab and the plant’s report on the refurbishment project on completion.

Finding 2.4. CPSD found that the plant’s work management system contains
obsolete work orders, as well as cooling tower inspection orders that don’t match
the plant’s maintenance plan. In response, the plant updated the work
management system.

Finding 2.5. CPSD found that the plant failed to identify and correct hazards to
foot and vehicle traffic, including standing water on the turbine deck, loose
grating in a walkway, and a two-foot hole in a dirt road adjacent to the plant.
In response, the plant corrected these hazards.

Finding 2.6. CPSD found that the plant fails to calibrate all test equipment. In
response, the plant hired a contractor to implement a comprehensive test program.
The plant monitors the calibration through a website.

Finding 2.7. CPSD found that the plant fails to use written checklists for all daily
inspections. In response, as part of a Reliant corporate initiative to improve
policies and procedures, Etiwanda will begin using checklists for inspections,
including the daily inspection. CPSD requests that the plant submit quarterly
updates on the effort.

Finding 2.8. CPSD found that the plant fails to maintain chemistry equipment
calibration logs. In response, the plant created comprehensive calibration logs.

Finding 2.9. CPSD found that the plant fails to follow chemistry lab
housekeeping procedures. In response, the plant cleaned and organized the
chemistry lab and fume hood, and posted a sign prohibiting storage in the fume
hood.

Finding 2.10. CPSD found that the plant fails to mark evacuation areas. In
response, the plant added evacuation signs to both evacuation areas.

Finding 2.11. CPSD found that plant security entered an incorrect list of visitors
on site in the plant gate log. In response, the plant created a new business
manager position to oversee the work of the security personnel.

Finding 2.12. CPSD found that a plant report describes abandoned fire hose
stations as “Missing.” In response, the plant audited all fire station equipment and
updated fire system inspection lists. The plant removed all unused hose stations.

Audit Number GO167-1006 5


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Introduction
Beginning in May 2006, CPSD audited the Etiwanda Generating Station to determine
whether the plant was in compliance with General Order (GO) 167. GO 167 includes
Maintenance, Operation, and Logbook standards for power plants.1 The audit team
included Ben Brinkman, Winnie Ho, Chris Lee and Alan Shinkman.

CPSD conducted the audit through an examination of plant performance, data requests
and a visit to the plant site. First, the team examined outage reports by CPSD staff, as
well as databases maintained by California Independent System Operator (ISO). On
May 5, 2006, the team notified the plant of the audit. The team visited the plant site from
June 4, 2006 to June 8, 2006, examining documents, interviewing staff, inspecting
equipment, and observing operations. At the conclusion of the site visit, the team
presented the plant with a data request.

CPSD sent Etiwanda a Preliminary Audit Report on July 13, 2007. Etiwanda responded
with a Corrective Action Plan (CAP) on August 21, 2007. Etiwanda agreed with all but
four of the report’s recommendations. CPSD and Etiwanda held a conference call on
October 23, 2007 during which the four outstanding issues were resolved. The audit
team leader visited the plant on December 12, 2007 to verify completion of several
corrective actions. Etiwanda cooperated fully throughout the entire audit process.

Plant Description
Reliant’s Etiwanda Generating Station sits in the industrial outskirts of Rancho
Cucamonga, east of Los Angeles. At one time, the plant consisted of five units. In 2003,
the plant retired Units 1, 2, and 5, and mothballed Units 3 and 4. Reliant returned Units 3
and 4 to service in 2004 when CAISO awarded those units Reliability-Must-Run (RMR)
contracts.2 Units 3 and 4 remained on RMR contracts at the time of the site visit, but
CAISO released them from these contracts for 2007.

The plant now consists of Units 3 and 4, which generate 320 Megawatts (MWs) each, for
a total of 640 MW. Commissioned in 1962, Units 3 and 4 utilize General Electric (GE)
steam turbines and controlled-circulation boilers manufactured by Combustion
Engineering. While most boilers have the firebox below and water tubes above, this
boiler reverses that configuration, using pumps to drive water through the boiler. Unlike
most of California’s conventional steam plants, Etiwanda is located inland and utilizes a

1
Further information on the Commission’s Power Plant Performance program may be found at the
Commission’s Web Site at
http://www.cpuc.ca.gov/PowerPlantStandards.
2
The California Independent System Operator (CAISO) had declared Units 3 and 4 to be Reliability Must
Run (RMR) units to preserve the stability of the grid, but concluded this contract as of 2007. From the
CAISO website, RMR is “an annual process that identifies geographical areas with local reliability issues
in the ISO Control Area along with measures to remediate those issues. Reliability Must Run Contracts are
a means of ensuring power availability within identified areas.”

Audit Number GO167-1006 6


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
cooling tower. The plant discharges used water to the Inland Empire Utilities Agency
wastewater treatment plant, which reclaims the water for reuse.

Plant Performance

CPSD studied the performance of Etiwanda’s active units, numbers 3 and 4.3 Staff
examined detailed data for 2003-2006 (note that Reliant mothballed the plant briefly in
2004), and more aggregated data for 1998-2003. Like many conventional boiler units in
California, Etiwanda ran at low capacity factors; from 2004-2006 Units 3 and 4 ran at
12% and 15% respectively.

For 2004 to 2006, Etiwanda units performed as well or better than similar California units
in almost all categories. Although the units started slightly less reliably than other plants,
they outperformed other plants on all other measures. Unit 4’s EFORd (a measure of the
plant’s forced outages when the ISO needed it) was particularly low: 3.14 compared with
the California average of 9.95. Unit 3’s EFORD was roughly 8. Overall availability was
90%.

In the years before the plant was mothballed, the plant operated similarly, except that its
overall availability was somewhat lower; around 80%. The plant took relatively more
scheduled outages during this period. EFORd was around 8 for both units.

While overall performance has been good, a number of specific incidents reduced plant
capacity significantly. If the plant had prevented these problems, its performance would
have been much better.

• Unit 3’s turbine problems (see Observation 3.4) increased EFORd and reduced
equivalent availability in the 4th quarter of 2004.
• A cooling tower collapsed in July 2005, during a Stage 2 emergency (See
Observation 3.10). The collapse affected neither EAF nor EFORd significantly
during this period. However, during the 2nd and 3rd quarters of 2006, resulting
repairs to the cooling tower reduced overall availability significantly. The plant
completed repairs just before the heat storms of 2006, reducing EFORd during
this critical period.
• In the third quarter of 2003, prior to mothballing, brushes short-circuited on Unit
4’s low pressure generator, causing an extended outage in August and September
of 2003. The plant’s root cause analysis showed that a contractor’s technician had
misaligned the generator’s brushes. EFORd spiked to 70 for that quarter.
3
CPSD examined indices established by the North American Electric Reliability Council (NERC). The
study includes six performance indices: (1) Equivalent forced outage rate during demand (EFORd), (2)
Equivalent Availability Factor (EAF), (3) Net Capacity Factor (NCF), (4) Start Reliability (SR), (5)
Scheduled Outage Factor (SOF), and finally (6) Forced Outage Factor (FOF). A lower EFORd indicates
fewer equivalent forced outage hours when ISO needs the unit to generate, and is therefore generally better
than a higher EFORd.

Audit Number GO167-1006 7


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
Therefore, the audit team evaluated the plant’s root cause analysis practices
(Observation 3.5), plant oversight of contractor work (Observation 3.10), and
employee training (Observation 3.11).

Audit Scope
The team looked broadly at the plant’s compliance with standards, and also focused on
problems identified from the plant’s operating history. Major incidents and problems
include:

• Bearings that failed after Units 3 and 4 returned to service. In particular,


immediately after reactivation, Unit 3’s steam turbine bearings failed several
times (as described in Observation 3.4) The failures led to lengthy and expensive
outages during October 2004 to November 2004. As discussed in Observation
3.4, the plant conducted several stages of root cause analysis. Ultimately, the
plant concluded that the grease hardened on Unit 3’s turbine shaft during the year
the plant was mothballed, causing the bearing problems.
• Structural failure of the plant’s cooling tower. In November 2004, Etiwanda hired
a contractor to upgrade the cooling tower for Unit 4. Without notifying Etiwanda,
the contractor substituted a part not specified by plans. In July of 2005, that
cooling tower partially collapsed, shutting the plant down and contributing to a
Stage 2 Electrical Emergency. As described in Observation 3.10, Etiwanda
improved oversight of contractors.

The audit also examined the plant’s compliance with specific standards, including those
covering:

A. Logbooks, training, and human resources,


B. Equipment, parts, and tools,
C. Water chemistry,
D. Boiler maintenance and operation,
E. Regulatory compliance,
F. Engineering support,
G. Safety, including hazardous material handling, fire and spill prevention and
response, and
H. Maintenance and operations planning, performance, and documentation.

The team visited the plant site from June 4, 2006 to June 8, 2006. Team members toured
the plant, including the generating units, the control room, the warehouse, chemistry lab,
mechanical shop and electrical shop. Plant staff demonstrated the plant’s maintenance
management software. CPSD auditors attended a plant shift change (tailboard) meeting,
and a training session in which plant staff extinguished test fires. CPSD auditors
reviewed numerous documents and interviewed plant staff. CPSD auditors requested and
reviewed additional documents after the site visit.

Audit Number GO167-1006 8


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
The audit findings and observations are described below, along with relevant Standards
and Guidelines. Section 1 of CPUC audit reports contain findings of potential violations
that pose safety hazards that require immediate corrective action. Auditors found no
Section 1 violations at Etiwanda.

Section 2 includes findings of potential violations that require corrective action as soon as
reasonably possible.

Finding 2.1. In the Preliminary Audit Report, CPSD found that the plant failed to
establish and follow clear test procedures for turbine overspeed. In response, the
plant rewrote the procedures, requiring annual full overspeed tests and including
clear step-by-step instructions.

Finding 2.2. In the Preliminary Audit Report, CPSD found that the plant deferred
some needed maintenance activities, which were prescribed by the plant’s
condition-based maintenance assessment. In response, the plant hired a new
manager for the condition-based maintenance program, and developed a new
schedule and procedure. Additionally, the plant performed much of the high
priority work from the original schedule, including repairs to the cathodic
protection system. The plant provided an updated repair schedule. By December
31, 2008, the plant agrees to provide CPSD with repair records for all the original
high priority items and all boiler and piping repairs listed on the original schedule.

Finding 2.3. In the Preliminary Audit Report, CPSD found that the plant fails to
maintain proper hydrazine levels in the condensate and feedwater during normal
operations. In response, the plant is refurbishing the entire chemistry lab by
installing or replacing various analyzers (including the hydrazine analyzer),
sensors, and alarms that will feed continuous data into a centralized PI database.
Reliant headquarters will remotely monitor plant activities via PI data. The plant
plans to complete the $70,000 refurbishment project by the end of March 2008.
EGPB requests that the plant provide photos of the refurbished lab and the plant’s
report on the refurbishment project.

Finding 2.4. In the Preliminary Audit Report, CPSD found that the plant’s work
management system contains obsolete work orders, as well as cooling tower
inspection orders that don’t match the plant’s maintenance plan. In response, the
plant updated the work management system, removing the obsolete work orders,
the first quarter of 2007. The plant changed the frequency of preventive
maintenance (PM) for cooling tower inspections to match that in the maintenance
governance. The plant sent copies of these changes in PM to CPSD.

Finding 2.5. In the Preliminary Audit Report, CPSD found that the plant fails to
identify and correct hazards to foot and vehicle traffic. First, auditors found a
pool of standing water on the turbine deck. Second, an auditor nearly tripped on a
piece of grating in front of the West Boiler Circulating Water Pump. Third, in

Audit Number GO167-1006 9


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
2005, plant personnel received minor injuries when their truck fell into a two-foot
hole on one of the plant’s dirt roads.

In response, the plant took corrective steps and set procedures in place to address
the hazards to foot traffic. The plant placed non-skid surface all over the turbine
deck, including the area where water had accumulated at the time of the onsite
audit visit. In response to the loose grating piece found on a walkway, the plant
manager now conducts monthly housekeeping inspections. The plant also welded
down several areas of loose grating.

With regard to the erosion on a dirt road which caused an injury, the plant
reemphasized to the staff the importance of following plant vehicle procedures
and the incident reporting process. The plant called Southern California Edison
(SCE), which owns the road, to request repairs.

Finding 2.6. In the Preliminary Audit Report, CPSD found that the plant fails to
assure the calibration of all test equipment. In response, the plant contracted a
company called TRANSCAT to track and calibrate all test equipment, including
multimeters. TRANSCAT notifies the plant electronically when calibration is
due, and calibrates the equipment on the plant site. The plant has access to all
calibration data through a website.

EGPB requested the latest TRANSCAT calibration report for review.

Finding 2.7. In the Preliminary Audit Report, CPSD found that the plant fails to
use written checklists for all daily unit “walkdown” inspections. In response, as
part of a Reliant corporate initiative to improve policies and procedures, Etiwanda
will begin using checklists for most inspections, including the daily walkdowns.
The CPSD inspector reviewed progress on the checklists in December 2007. The
plant intends to implement the new checklists during 2008. CPSD requests that
the plant submit quarterly updates on the effort.

Finding 2.8. In the Preliminary Audit Report, CPSD found that the plant fails to
maintain calibration logs for equipment that monitors feedwater chemistry. In
response, the plant created equipment calibration logs. The logs list the
inspection cycle, calibration date, and calibration data for each piece of equipment
in the lab. In addition, during a December 12, 2007 plant visit, EGPB staff saw
stickers on equipment showing dates for the last calibration and next one due.

Finding 2.9. In the Preliminary Audit Report, CPSD found that the plant fails to
follow chemistry lab housekeeping procedures. In response, the plant removed
storage material from the fume hood area, and posted a sign indicating no storage
in the fume hood. During a December 12, 2007 plant visit, EGPB staff found the
chemistry lab clean. No further action is needed.

Audit Number GO167-1006 10


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
Finding 2.10. In the Preliminary Audit Report, CPSD found that the plant fails to
mark evacuation areas. In response, the plant added evacuation signs to both
evacuation areas. CPSD verified this corrective action on December 12, 2007.

Finding 2.11. In the Preliminary Audit Report, CPSD found that plant security
entered an incorrect list of visitors on site in the plant gate log. In response, the
plant created a new business manager position to oversee the work of the security
personnel. One of the responsibilities of the business manager is to periodically
check the gate logs for accuracy. The plant submitted to CPSD sample copies of
gate logs that have been checked and signed by the business manager.

Finding 2.12. In the Preliminary Audit Report, CPSD found that a plant report
describes abandoned fire hose stations as “Missing.” In response, the plant
audited all fire station equipment and updated fire system inspection lists. The
plant removed all unused hose stations.

Section 3 contains observations of plant activities where auditors found no apparent


violations.

Observation 3.1. Reliant Regularly Audits its own Plants and Rates its own
Performance.

Observation 3.2. The Plant Maintains Extensive Procedures and a List of


Approved Contractors.

Observation 3.3. The Plant Bases Employee Compensation on Performance


Evaluations.

Observation 3.4. The Plant Thoroughly Investigated Turbine Bearing Failures.

Observation 3.5. The Plant Maintains and Uses a Detailed Procedure for Root
Cause Analysis.

Observation 3.6. According to a Detailed Procedure, the Plant Tracks Important


Inventory by Project.

Observation 3.7. The Plant’s Document Control Room is Neat and Organized.

Observation 3.8. General Housekeeping and Equipment Labeling are Good.

Observation 3.9. The Plant Keeps Predictive Maintenance Reports.

Observation 3.10. The Plant Has Improved Its Procedures and Technical Support
for Contract Work.

Observation 3.11. The Plant Utilizes a Training Program.

Audit Number GO167-1006 11


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Observation 3.12. The Plant Provides Security for the Plant and Personnel.

Observation 3.13. The Plant Follows Tagout Procedures and is Implementing a


Lockout Procedure.

Observation 3.14. The Plant Tests its Fire System Annually.

Observation 3.15. The Plant Held an Evacuation Drill and Implemented Changes
as a Result.

Observation 3.16. The Plant Holds Daily Staff Meetings

Observation 3.17. The Plant’s Chemistry Lab Procedures are Comprehensive.

Observation 3.18 Auditors Found No Significant Difference between Plant Logs


and other Reported Information.

SECTION 1 FINDINGS
Staff found no safety hazards requiring immediate correction.

Audit Number GO167-1006 12


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

SECTION 2 FINDINGS
Operations Standard 7, “Operation Procedures and Documentation,” states:

“Operation procedures exist for critical systems and states of those systems
necessary for the operation of the unit including startup, shutdown, normal
operation, and reasonably anticipated abnormal and emergency conditions.
Operation procedures and documents are clear and technically accurate,
provide appropriate direction, and are used to support safe and reliable plant
operation. Procedures are current to the actual methods being employed to
accomplish the task and are comprehensive to ensure reliable energy delivery
to the transmission grid.”

Operation Standards 28, “Equipment and Systems,” states:

“GAO complies with these Operation Standards (1-27) considering the design
bases (as defined in the Appendix) of plant equipment and critical systems. The
GAO considers the design basis of power plant equipment when as required by
other standards it, among other things:

A. Establishes procedures for the operation of critical systems at each unit


(Ref. Standard No. 7).

B. For each system, identifies critical parameters that require monitoring (Ref.
Standard No. 8 and 13).

C. For each critical parameter, establishes values at which to increase


observation of the system or take actions to protect it (Ref. Standard No. 8 and
13).

D. Assures that systems are monitored and actions are taken (Ref. Standard 8
and 13).”

Finding 2.1. The Plant Fails to Establish and Follow Clear Test
Procedures for Turbine Overspeed.
The plant failed to establish and follow clear test procedures for turbine overspeed. First,
the plant’s test procedure is poorly written and unclear. Second, the plant failed to
conduct tests every year, as its procedures apparently require. Finally, reminders in the
work scheduling system appear ineffective.

The plant’s test procedure (ETS-00-TU-PO02) is poorly written and unclear. First, the
procedure fails to clearly explain when to use an overspeed test (where operators run the
turbine at 110% of synchronous speed) versus an oil trip test (where operators inject oil
into the turbine governor). Second, while the procedure requires annual testing, it is

Audit Number GO167-1006 13


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
unclear whether an oil trip test alone meets this requirement. Third, the procedure
appears to allow the plant to omit the annual test if the turbine has tripped at any time
during that year, failing to specify that only trips caused by overspeeding satisfy this
procedure. Fourth, procedures require testing during outages, when the plant customarily
carries no load; but the procedure also requires testing only after the unit carries 24-30
percent of load. The two requirements appear contradictory.

For Unit 4, the plant failed to conduct overspeed tests every year as the procedure appears
to require. Records show an overspeed test on Unit 4 in 2004 and on Unit 3 in 2006. The
plant manager stated that the plant conducted many oil trip tests in 2005, but no
overspeed test on either unit. In late June 2006, on three consecutive days, Unit 4’s
overspeed protection scheme tripped even though the turbine was operating normally.
Resulting outages totaled over 60 hours. In response, the plant tightened the governor
linkage, reset the trip speed, and replaced electrical monitoring equipment.4 CPSD
believes that proper tests might have detected these problems and prevented the outages.

Finally, reminders in the work scheduling system appear ineffective. Reminders for
overspeed tests (ETS04TUOPO01) appear every six months, even though procedures call
for tests only during outages. Apparently staff routinely ignores these reminders.

Final Outcome and Follow-up

In response to the Preliminary Audit Report, the plant rewrote its overspeed test
procedures to clearly require full overspeed tests at least every twelve months. The new
procedures detail required tests step-by-step, and are consistent with the reminders in the
work scheduling system. CPSD auditors verified the new procedures, and will check the
plant’s 2008 overspeed trip test records for compliance. The plant states that staff did not
ignore reminders; rather they performed “oil-trip” tests.

Maintenance Standard 10, “Work Management,” states:

Work is identified and selected based on value to maintaining reliable plant


operation. Work is planned, scheduled, coordinated, controlled, and supported
with resources for safe, timely, and effective completion.

Finding 2.2. The Plant Delayed Evaluation of Needed Maintenance.


The plant delayed evaluation of needed maintenance until after the summer peak,
potentially violating operating and maintenance standards. In particular, a week before
the audit visit, on May 31, 2006, the plant compiled the results of a survey of plant staff
on which 164 plant systems required maintenance.5 Work included reconnecting the

4
From an independent audit required under a settlement between Reliant and the CPUC, titled
“Independent Audit of Unit Outages Related to Joint Settlements of FERC Dockets.”
5
“Etiwanda Generating Station Critical Equipment Schedule.” This document is dated May 31, 2006.

Audit Number GO167-1006 14


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
plants cathodic protection system, and adding several work orders to the computerized
work management system. The plant manager stated that he would add work orders
soon, but delay major maintenance items until after the summer season. While CPSD
applauds the plant’s initiative in identifying needed work, and generally supports
scheduling maintenance in off-peak periods, CPSD is concerned that some of these latter
items could and should have been completed before the end of the peak season, including
possibly, repairs to the cathodic protection system.

Final Outcome and Follow-up

In response to the Preliminary Audit Report, the plant hired a new manager for the
condition-based maintenance program, and developed a new schedule and procedure.
Additionally, the plant performed much of the high priority work from the original
schedule, including repairs to the cathodic protection system. The plant provided an
updated repair schedule. By December 31, 2008, the plant agrees to provide CPSD with
repair records for all the original high priority items and all boiler and piping repairs
listed on the original schedule.

Audit Number GO167-1006 15


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Maintenance Standard 8, “Maintenance Procedures and Documentation”, states:

“Maintenance procedures and documents are clear and technically accurate,


provide appropriate direction, and are used to support safe and reliable plant
operation. Procedures must be current to the actual methods being employed
to accomplish the task and are comprehensive to ensure reliable energy
delivery to the transmission grid.”

Maintenance Standard 15, “Chemistry Control”, states:

“Chemistry controls optimize chemistry conditions during all phases of plant


operation and system non-operational periods.”

Operations Standard 12, “Operations Conduct”, states:

“To ensure safety, and optimize plant availability, the GAO conducts
operations systematically, professionally, and in accordance with approved
policies and procedures. The GAO takes responsibility for personnel actions,
assigns personnel to tasks for which they are trained, and requires personnel
to follow plant and operation procedures and instructions while taking
responsibility for safety. Among other things:

A. All personnel follow approved policies and procedures. Procedures are


current, and include a course of action to be employed when an adopted
procedure is found to be deficient.”

Finding 2.3. The Plant Fails to Maintain Proper Hydrazine Levels in


the Condensate and Feedwater.
The plant fails to maintain proper hydrazine levels in the condensate and feedwater
during normal operations, a potential violation of maintenance and operations standards.
First, during normal operation, hydrazine levels in condensate and feedwater often
exceed allowable limits. Second, even though plant procedures require continuous
monitoring during such excursions, the plant fails to do so, in part because monitors are
broken.

During normal operation, hydrazine levels in condensate and feedwater often exceed the
limit stated in plant procedures of 20 parts per billion (ppb), in part because the plant’s
chemistry technician believed that those higher concentrations are acceptable. The
technician stated that acceptable hydrazine levels fall between 20 to 120 ppb during
normal operations By contrast, the plant’s chemistry lab procedures require hydrazine
levels in condensate and feedwater to be 20 ppb or less. Above this level, the procedure

Audit Number GO167-1006 16


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
states that contaminants collect and corrode ferrous material in the feedwater system.6 .
Boiler sheets show daily test readings exceeding 20 ppb on the following days:

Date Unit 3 Unit 3 Unit 4 Unit 4


N2H4 -DI N2H4 -EI N2H4 -DI N2H4 -EI
2/14/06 37 62 25 21
2/16/06 20 26 12 10
2/17/06 56 48 14 10
2/21/06 18 33 13 7
2/23/06 36 26 49 45
3/2/06 98 90 62 54
3/3/06 26 20 62 52
3/7/06 221 224 86 87
3/8/06 181 191 79 86
3/9/06 79 74 70 81
3/10/06 24 14 32 31
3/14/06 133 119 49 43
3/15/06 52 47 60 51
3/16/06 46 38 15 22
3/17/06 69 63 38 33
3/21/06 25 20 42 33
3/22/06 21 18 29 20
3/24/06 24 20 47 40
3/27/06 71 61 58 54
3/28/06 36 30 18 12
3/30/06 42 37 26 20
3/31/06 45 41 20 14

Even though plant procedures require continuous monitoring during excursions,7 the
plant fails to do so, in part because monitors are broken. Instrumentation that
continuously monitors and controls hydrazine levels broke down in October 2004, and
the plant has not fixed it. As a result, the chemical technician must manually test and
control the hydrazine levels, which he cannot do continuously. Therefore the plant
cannot effectively control hydrazine levels during excursions.

6
Procedures titled “Etiwanda Generating Station Chemistry Lab.” Section 10.2, “Rationale for Action Levels” states
that “action levels for control parameters at critical sample points were developed to provide guidance on the relative
need for corrective action in the event of cycle chemistry excursions. Normal target values are defined as values
associated with long term stability. A safe margin has been provided to avoid concentration of contaminants on the
surfaces.”

Section 10.4, “Rationale for Target Values of Individual Parameters” states that hydrazine concentration at the plant
has “an upper limit of 20 ppb” since there is evidence linking higher hydrazine level with corrosion of ferrous material
in the feedwater train.
7
“Etiwanda Generating Station Chemistry Lab” procedures, more specifically, Section 4.2, titled
“Chemistry Control for the Deaerator Inlet,” requires continuous sampling of hydrazine levels during
excursions.

Audit Number GO167-1006 17


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Final Outcome and Follow-up


In response to the Preliminary Audit Report, the plant is refurbishing the entire chemistry
lab by installing or replacing various analyzers (including the hydrazine analyzer),
sensors, and alarms that will feed continuous data into a centralized PI database. Reliant
headquarters will remotely monitor plant activities via PI data. The plant plans to
complete the $70,000 refurbishment project by the end of March 2008.

EGPB requests that the plant provide photos of the refurbished lab and the plant’s report
on the refurbishment project.

Maintenance Standard 10, “Work Management,” states:

Work is identified and selected based on value to maintaining reliable plant


operation. Work is planned, scheduled, coordinated, controlled, and supported
with resources for safe, timely, and effective completion.

Finding 2.4. Work Management System Contains Obsolete Work


Orders and Conflicts with the Plant’s Maintenance Plan.
The plant’s work management system contains obsolete work orders, as well as
inspection orders that don’t match the plant’s maintenance plan, both potential violations
of maintenance and operations standards. First, the plant’s work management system
shows 38 open work orders dated prior to 2005, including number 33931894, dated May
7, 2002. According to the maintenance manager, the plant completed some of these work
orders. The plant cancelled the others in 2003, while mothballing the plant. The plant
failed to remove these work orders from the work management system. Second, the work
management system and the plant’s Maintenance Governance list different frequencies
for cooling tower inspections. That Governance states that operators, the maintenance
manager, and the plant engineer will conduct daily, quarterly and annual, and weekly
inspections, respectively.8 Entries in the work management system, however, schedule
such inspections every six months. Apparently, after specifying inspection intervals in
the work management system, the plant adopted the conflicting Maintenance
Governance.

Final Outcome and Follow-up


The plant updated the work management system, removing the obsolete work orders, the
first quarter of 2007. The plant changed the frequency of PMs for cooling tower
inspections to match that in the Maintenance Governance. The plant sent copies of these

8
Maintenance Governance (ETS-ZM-CW-PR01 Ver.1, Sections 2.1 – 2.3)

Audit Number GO167-1006 18


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
changes in PM to CPSD. Upon review, CPSD agrees the changes satisfactorily address
the conflict between the work management system and the maintenance governance.

Maintenance Standard 1 – Safety states:

The protection of life and limb for the work force is paramount. The company
behavior ensures that individuals at all levels of the organization consider
safety as the overriding priority. This is manifested in decisions and actions
based on this priority. The work environment, and the policies and procedures
foster such a safety culture, and the attitudes and behaviors of individuals are
consistent with the policies and procedures.

Finding 2.5. The Plant Fails to Identify and Correct Hazards to Foot
and Vehicle Traffic.
The plant fails to identify and correct hazards to foot and vehicle traffic, a potential
violation of maintenance and operations standards. First, auditors found a pool of
standing water on the turbine deck. Second, an auditor nearly tripped on a piece of
grating in front of the West Boiler Circulating Water Pump. Third, in 2005, plant
personnel received minor injuries when their truck fell into a two-foot hole on one of the
plant’s dirt roads.

Auditors found a pool of standing water on the Unit 3 and 4 turbine deck. Wind
sometimes blows the exhaust steam condensate to this area of the deck, causing water to
accumulate on the ground. The plant staff said the water does not harm the equipment,
which is designed for outdoor use. CPUC believes that the wet ground poses a potential
danger, as people may slip and fall. Therefore, the area should be barricaded or coned off
to alert people and to prevent unnecessary entry into the area. (See Figure 1 below)

Audit Number GO167-1006 19


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Figure 1. Water creates hazard in walkway.

Figure 2. The plant added non-slip surface to the walkways.


During the plant tour, the auditors saw a piece of grating lying on a platform in front of
the West Boiler Circulating Water Pump. The grating piece was lying partially in the

Audit Number GO167-1006 20


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
walkway. Since the grating is the same as the platform grating, it was not readily visible,
and posed a trip and fall hazard. The plant engineer moved it to the side so that it would
not be in the walkway.

Erosion of dirt roads (for example, the one leading to the center well pump) poses
potential danger to those driving through in a car or truck. On February 23, 2005, a
company truck on that road fell into a 2-foot hole created by erosion, causing very minor
injuries to the people in the truck.

Final Outcome and Follow-up

The plant took corrective steps and set procedures in place to address the hazards to foot
traffic. The plant placed non-skid surface all over the turbine deck (Figure 2 above),
including the area where water had accumulated at the time of the onsite audit visit. In
response to the loose grating piece found on a walkway, the plant manager now conducts
monthly housekeeping inspections. The plant also welded down several areas of loose
grating.

With regard to the erosion on a dirt road which caused an injury, the plant reemphasized
to the staff the importance of following plant vehicle procedures and the incident
reporting process. The plant called Southern California Edison (SCE), which owns the
road, to request repairs.

Audit Number GO167-1006 21


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Maintenance Standard 1, “Safety,” states:

The protection of life and limb for the work force is paramount. The company
`behavior ensures that individuals at all levels of the organization consider
safety as the overriding priority. This is manifested in decisions and actions
based on this priority. The work environment, and the policies and procedures
foster such a safety culture, and the attitudes and behaviors of individuals are
consistent with the policies and procedures.

Assessment Guideline 2.C states:

Work practice norms in the organization promote the safety culture through:
1. Appropriate defenses, such as technical accuracy,
precautions, cautions and notes, are explicitly embedded in
procedures, processes, and equipment configuration to
minimize the occurrences and consequences of inappropriate
actions.

Operations Standard 13, “Routine Inspections,” states:

Routine inspections by plant personnel ensure that all areas and critical
parameters of plant operations are continually monitored, equipment is
operating normally, and that routine maintenance is being performed. Results
of data collection and monitoring of parameters during routine inspections are
utilized to identify and resolve problems, to improve plant operations, and to
identify the need for maintenance. All personnel are trained in the routine
inspections procedures relevant to their responsibilities. Among other things,
the GAO creates, maintains, and implements routine inspections by:

D. Monitoring routine inspections.

Maintenance Standard 18, “Maintenance Facilities and Equipment,” states:

Facilities and equipment are adequate to effectively support maintenance


activities.

Finding 2.6. The Plant Fails to Assure the Calibration of All Test
Equipment.
The plant fails to assure the calibration of test equipment, a violation of the plant’s
procedures, and a potential violation of maintenance standards. First, the plant lacks
computerized, recurring work orders for calibration of safety equipment. Second, plant

Audit Number GO167-1006 22


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
procedures fail to specify which non-safety equipment requires regular, formal, and
traceable calibration, and which does not. 9

The plant lacks computerized, recurring work orders for calibration of safety equipment.
For example, electricians must remember when to calibrate the gas detection meters.
The CPSD auditor checked a gas detection meter and found a calibration certificate
attached. However CPSD is concerned that the plant’s informal procedure is inadequate,
because these meters are critical to safety and reliability, and because multiple
individuals use the meters.

Second, plant procedures fail to specify which non-safety equipment requires regular,
formal, and traceable calibration, and which does not. Instead, this decision is left to
individual staff judgment. The plant attaches calibration stickers to some pieces of
equipment (e.g. gloves and hotsticks) but not others. While the plant sends out some
equipment for regular calibration, the plant calibrates fluke meters and meggers as
needed. Electricians explained that they use fluke meters and meggers only for
"ballpark" measurements and therefore send these meters for calibration only when they
detect a problem. CPSD is concerned that over time, without clear written guidance, staff
practices could become inconsistent, introducing maintenance errors.

Final Outcome and Follow-up

The plant contracted a company called TRANSCAT to track and calibrate all test
equipment, including multimeters. TRANSCAT notifies the plant electronically when
calibration is due, and calibrates the equipment on the plant site. The plant has access to
all calibration data through a website.

EGPB requested the latest TRANSCAT calibration report for review.

9
Reliant’s high-level policy, Performance Objectives for Mini-FPPAT, Guideline M.5.E, requires that
“Measurement and test equipment is calibrated and controlled to ensure accuracy and traceability,” but the
plant lacks detailed procedures to implement this policy.

Audit Number GO167-1006 23


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Operations Standard 13, “Routine Inspections,” states:

Routine inspections by plant personnel ensure that all areas and critical
parameters of plant operations are continually monitored, equipment is
operating normally, and that routine maintenance is being performed. Results
of data collection and monitoring of parameters during routine inspections are
utilized to identify and resolve problems, to improve plant operations, and to
identify the need for maintenance. All personnel are trained in the routine
inspections procedures relevant to their responsibilities. Among other things,
the GAO creates, maintains, and implements routine inspections by:

A. Identifying systems and components critical to system operation (such as


those identified in the guidelines to Standard 28).

B. Establishing procedures for routine inspections that define critical


parameters of these systems, describe how those parameters are monitored,
and delineate what action is taken when parameters meet alert or action levels.

C. Training personnel to conduct routine inspections.

D. Monitoring routine inspections.

Finding 2.7. The Plant Lacks Checklists for Some Daily Inspections
According to the plant manager, the plant fails to use written checklists for all daily unit
“walkdown” inspections, in conflict with its own performance objectives,10 and
potentially violating maintenance and operations standards. Although the plant maintains
detailed inspection procedures, some procedures lack checklists. While plant staff
demonstrated knowledge of equipment and operations,11 the lack of checklists may lead
staff to overlook details, or to fail to report equipment conditions to following shifts.
Without documentation, the plant cannot confirm that staff made required inspections, or
track the history of equipment problems. The plant manager stated that the plant plans to
adopt a system that utilizes Personal Digital Assistants (PDAs), to record some of the
inspection activities, particularly the chemical levels in various storage areas and
reservoirs.

10
Reliant “Performance Objectives for Mini-FPPAT, Guideline O.2.A” audit guidelines states:
“Check sheets or other comparable means are used to ensure that proper conditions are established for each mode of
plant status and for mode shifts.”
11
The plant manager and training officer explained that the plant trains operators over a three-year apprenticeship.
Auditors interviewed a yard operator who correctly delineated the entire list of yard operator, or “number 3 operator”
duties listed in the “Operators Routine Duties and Inspections” documents. The operator quoted meter readings and
chemical levels. Therefore, CPSD has no evidence that there exists a problem with operator knowledge or training on
the daily walkdown.

Audit Number GO167-1006 24


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
Plant checklists include those for Resource Recovery and Conservation Act (RCRA)
inspections, an End of the Month Report, Unit 3 and 4 Start-up and Shutdown
Inspections, and the Boiler / Unit Walkdown Checklist (which the plant uses
occasionally). The plant uses additional checklists at critical times, such as unit startup
and shutdown.

Final Outcome and Follow-up

As part of a Reliant corporate initiative to improve policies and procedures, Etiwanda


will begin using checklists for most inspections, including the daily walkdowns. The
CPSD inspector reviewed progress on the checklists in December 2007. The plant intends
to implement the new checklists during 2008. CPSD requests that the plant submit
quarterly updates on the effort.

Audit Number GO167-1006 25


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Maintenance Standard 8, “Maintenance Procedures and Documentation”, states:

“Maintenance procedures and documents are clear and technically accurate,


provide appropriate direction, and are used to support safe and reliable plant
operation. Procedures must be current to the actual methods being employed
to accomplish the task and are comprehensive to ensure reliable energy
delivery to the transmission grid.”

Operating Standards 11, “Operations Facilities, Tools and Equipment”, states:

“Facilities and equipment are adequate to effectively support operations


activities.”

Operations Standard 12, “Operations Conduct”, states:

“To ensure safety, and optimize plant availability, the GAO conducts
operations systematically, professionally, and in accordance with approved
policies and procedures. The GAO takes responsibility for personnel actions,
assigns personnel to tasks for which they are trained, and requires personnel
to follow plant and operation procedures and instructions while taking
responsibility for safety. Among other things:

A. All personnel follow approved policies and procedures. Procedures are


current, and include a course of action to be employed when an adopted
procedure is found to be deficient.”

Operating Standards 17, “Records of Operation”, states:

“The GAO assures that data, reports and other records reasonably necessary for
ensuring proper operation and monitoring of the generating asset are collected by
trained personnel and retained for at least five years, and longer if appropriate.”

Finding 2.8. The Plant Fails to Maintain Calibration Logs for the
Equipment that Monitors Feedwater Chemistry.
The plant fails to maintain calibration logs for equipment that monitors feedwater
chemistry, a potential violation of plant procedures and maintenance and operation
standards. The plant’s procedures12 require the plant’s chemical technician to develop
and maintain a calibration log on site. Further, the plant must calibrate testing equipment
such as the conductivity meter, pH meter, and spectrophotometer annually to ensure

12
Etiwanda Generating Station Chemistry Lab, Section 2.3, Calibration and Performance Checks
states that the conductivity meter, pH meter, and spectrophotometer must be calibrated annually and
that “an instrument calibration log will be developed and maintained by the Chemical Technician on
site.”

Audit Number GO167-1006 26


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
accurate test readings. However, the chemical technician and the chemistry lab
supervisor stated that the plant calibrates these instruments as needed, without keeping a
log or using the plant’s computerized work management system. CPSD is concerned that
failure to document and track calibration could lead to problems in the plant’s water
chemistry, damaging feedwater equipment and piping.

Final Outcome and Follow-up

In response to the Preliminary Audit Report, the plant created equipment calibration logs.
The logs list the inspection cycle, calibration date, and calibration data for each piece of
equipment in the lab. In addition, during a December 12, 2007 plant visit, EGPB staff
saw stickers on equipment showing dates for the last calibration and next one due (see
Figure 3).

Figure 3. The plant added calibration stickers on equipment.

Audit Number GO167-1006 27


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Maintenance Standard 1, “Safety,” states:

“The protection of life and limb for the work force is paramount. The company
behavior ensures that individuals at all levels of the organization consider
safety as the overriding priority. This is manifested in decisions and actions
based on this priority. The work environment, and the policies and procedures
foster such a safety culture, and the attitudes and behaviors of individuals are
consistent with the policies and procedures.”

Maintenance Standard 8, “Maintenance Procedures and Documentation”, states:

“Maintenance procedures and documents are clear and technically accurate,


provide appropriate direction, and are used to support safe and reliable plant
operation. Procedures must be current to the actual methods being employed
to accomplish the task and are comprehensive to ensure reliable energy
delivery to the transmission grid.”

Operating Standards 1, “Safety”, states:

“The protection of life and limb for the work force is paramount. GAOs have
a comprehensive safety program in place at each site. The company behavior
ensures that personnel at all levels of the organization consider safety as the
overriding priority. This is manifested in decisions and actions based on this
priority. The work environment and the policies and procedures foster such a
safety culture, and the attitudes and behaviors of personnel are consistent with
the policies and procedures.”

Operations Standard 12, “Operations Conduct”, states:

“To ensure safety, and optimize plant availability, the GAO conducts
operations systematically, professionally, and in accordance with approved
policies and procedures. The GAO takes responsibility for personnel actions,
assigns personnel to tasks for which they are trained, and requires personnel
to follow plant and operation procedures and instructions while taking
responsibility for safety. Among other things:

A. All personnel follow approved policies and procedures. Procedures are


current, and include a course of action to be employed when an adopted
procedure is found to be deficient.”

Audit Number GO167-1006 28


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Finding 2.9. The Plant Fails to Follow Chemistry Lab Housekeeping


Procedures.
The plant fails to follow its housekeeping procedures in the chemistry lab, a potential
violation of maintenance and operations standards. The plant’s chemistry lab procedures
require aisles to be clear from obstructions, and fume hoods to be clean and clear for
work. The procedure states, “Hoods should never be used as storage areas.” However,
the auditor found several violations of these procedures.

During a tour of the lab on June 5, the auditor saw boxes, chairs, and other items
apparently stored in the aisle behind the instrumentation panel (see Figure 4). These
items obstructed pedestrian traffic and prevented one from safely walking the length of
the aisle. The auditor also saw chemical/reagent boxes and storage bins housed in the
fume hood (see Figure 5).

The plant cleared the aisles of debris by June 7, 2006 but continued to use the fume hood
as a storage area.

Final Outcome and Follow-up

During a December 12, 2007 plant visit, EGPB staff found the chemistry lab clean. The
plant removed storage material from the fume hood area, and posted a sign indicating no
storage in the fume hood (see Figure 6).

No further action is needed.

Figure 4. The plant left clutter in the chemistry lab

Audit Number GO167-1006 29


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Figure 5. The plant used the fume hood for storage.

Figure 6. The plant cleaned and placed a warning sign on the fume hood.

Audit Number GO167-1006 30


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Maintenance Standard 1 – Safety states:

The protection of life and limb for the work force is paramount. The company
behavior ensures that individuals at all levels of the organization consider
safety as the overriding priority. This is manifested in decisions and actions
based on this priority. The work environment, and the policies and procedures
foster such a safety culture, and the attitudes and behaviors of individuals are
consistent with the policies and procedures.

Finding 2.10. The Plant Fails to Mark Evacuation Areas


While the plant has established evacuation areas, it has failed to post signs at those areas,
a potential violation of maintenance and operations standards. During safety orientations
for visitors and contractors, the plant provides and discusses a handout designating the
evacuation assembly areas. The plant also conducts evacuation drills, during which the
Control Room Operator announces over the public address system which assembly area
to use. CPSD believes that in addition to these efforts, the plant should mark the
evacuation areas with clearly visible signs to prevent potential confusion or delays during
evacuations.

Final Outcome and Follow-up


The plant added evacuation signs to both evacuation areas. CPSD verified this corrective
action on December 12, 2007 (see Figure 7).

Figure 7. The plant added signs to evacuation areas.

Audit Number GO167-1006 31


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Maintenance Standard 1 – Safety states:

The protection of life and limb for the work force is paramount. The company
behavior ensures that individuals at all levels of the organization consider
safety as the overriding priority. This is manifested in decisions and actions
based on this priority. The work environment, and the policies and procedures
foster such a safety culture, and the attitudes and behaviors of individuals are
consistent with the policies and procedures.

Finding 2.11. Plant Security Entered an Incorrect List of Visitors on


Site in Plant Gate Log.
On at least one day, the plant’s gate log was inaccurate, a potential violation of
maintenance and operations standards. The gate log, an electronic spreadsheet, records
the times that visitors and contractors enter and exit the facility each day. The security
officer enters the information on the computer in the guard station. On June 7, the
security officer correctly entered the entry and exit times for the auditors. On June 8, the
auditors arrived early, but the security officer copied the log from the previous day. As a
result, the plant lacked an accurate record of visitors at the site.

Final Outcome and Follow-up


The plant created a new business manager position to oversee the work of the security
personnel. One of the responsibilities of the business manager is to periodically check
the gate logs for accuracy. The plant submitted to CPSD sample copies of gate logs that
have been checked and signed by the business manager.

Maintenance Standard 8 – Maintenance Procedures and Documentation states:

Maintenance procedures and documents are clear and technically accurate,


provide appropriate direction, and are used to support safe and reliable plant
operation. Procedures must be current to the actual methods being employed
to accomplish the task and are comprehensive to ensure reliable energy
delivery to the transmission grid.

Finding 2.12. A Plant Report Describes Abandoned Fire Hose Stations


as “Missing.”

Inspection reports of the plant’s fire control system refer to abandoned equipment as
“missing,” a potential violation of maintenance and operation standards. In particular,
the Hose Station report of May 4, 2006 states that stations #67 and #72 are “missing”.

Audit Number GO167-1006 32


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
The report offered no explanation on these “missing” hose stations, but the plant engineer
said the two stations have been abandoned. The report should show that the two hose
stations are abandoned. The stations themselves should be so labeled to avoid confusion
during emergency response.

Final Outcome and Follow-up


The plant audited all fire station equipment and updated fire system inspection lists. The
plant removed all unused hose stations.

SECTION 3. OBSERVATIONS

Operations Standard 3, “Operations Management and Leadership,” states

Operations management establishes high standards of performance and aligns


the operations organization to effectively implement and control operations
activities.

Operations Standard 4, “Problem Resolution and Continuing Improvement,” states:

The GAO values and fosters an environment of continuous improvement and


timely and effective problem resolution.

Maintenance Standard 3, “Maintenance Management and Leadership,” states:

Maintenance managers establish high standards of performance and align the


maintenance organization to effectively implement and control maintenance
activities.

Maintenance Standard 4, “Problem Resolution and Continuing Improvement,” states:

The company values and fosters an environment of continuous improvement


and timely and effective problem resolution.

Observation 3.1. Reliant Regularly Audits its own Plants and Rates its
own Performance.

Fossil Plant Performance Assessment Team (FPPAT) Audits

Reliant regularly selects one of its power plants for what Reliant calls a Fossil Plant
Performance Assessment Team (FPPAT) audit. Reliant sends staff from other Reliant

Audit Number GO167-1006 33


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
plants to perform the audit. The CPSD auditor reviewed Reliant audit procedures13 and
found that they cover administration, safety and environmental policies, and maintenance
and operations activities of the plants. Reliant scheduled such an audit of Etiwanda in
2006, but postponed it due to CPSD’s GO167 audit.

Performance Indices

Etiwanda calculates three plant performance indices, according to detailed Reliant


procedures, which determine team bonuses for the plant. The three indices are the Safety
Performance Index (SPI), the Environmental Performance Index (EPI), and the
Maintenance Performance Index (MPI). CPSD auditors reviewed checklists used to
calculate Etiwanda’s EPI and SPI for the first quarter of 2006. In these internal
evaluations, the plant assigns points to over 70 environmental and 60 safety performance
measures, and calculates what percentage of total points it attains. The plant reported
scores of 98% for environmental and 95% for safety performance, respectively.
Environmental criteria include air quality, solid waste, water quality, spill prevention, and
regulatory training. Safety criteria include training, procedures, incident prevention and
reporting.

The plant manager stated the plant corrected all discrepancies it found during the course
of the evaluation processes. For example, the plant scored zero out of five on
maintenance of oil storage tanks. The plant manager stated that the plant immediately
fixed leaking tanks. Auditors saw no evidence of tank leaks.

Maintenance Standard 8, “Maintenance Procedures and Documentation,” states:

Maintenance procedures and documents are clear and technically accurate, provide
appropriate direction, and are used to support safe and reliable plant operation.
Procedures must be current to the actual methods being employed to accomplish the
task and are comprehensive to ensure reliable energy delivery to the transmission
grid.

Operations Standard 7, “Operation Procedures and Documentation,” states:

Operation procedures exist for critical systems and states of those systems necessary
for the operation of the unit including startup, shutdown, normal operation, and
reasonably anticipated abnormal and emergency conditions. Operation procedures
and documents are clear and technically accurate, provide appropriate direction, and
are used to support safe and reliable plant operation. Procedures are current to the
actual methods being employed to accomplish the task and are comprehensive to
ensure reliable energy delivery to the transmission grid.

13
“Performance Objective Mini FPPAT”

Audit Number GO167-1006 34


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Observation 3.2. The Plant Maintains Extensive Procedures and a List


of Approved Contractors.
The plant utilizes an extensive collection of over 200 documented procedures and
processes. The plant manager and maintenance manager demonstrated Reliant’s
corporate intranet, where Reliant maintains many procedures. The plant also keeps hard
copies of these procedures in the control room, along with plant specific maintenance and
operations procedures. These procedures include detailed instructions for plant
operations and maintenance tasks, and are classified in the following groups:

ƒ System Operating Bulletins


ƒ Regional Maintenance Policies
ƒ Operations Policies
ƒ Operations Procedures
ƒ System Description
ƒ Maintenance Procedures
ƒ Administrative Policies

The plant also maintains a “maintenance call out book,” which contains the names and
contact information for all the approved vendors and contractors for the plant.

Auditors reviewed procedures that touched upon items examined during the audit.
Auditors found the test procedure for turbine overspeed to be confusing; see Finding 2.1.

Maintenance Manager’s Manual

The maintenance manager utilizes a “Maintenance Manager’s Manual” that is common to


all Reliant plants. The manual contains instructions and information on the conduct of
maintenance, handling of parts, and condition-based maintenance including instructions
on root cause analysis. The manual also described the Technical Service Team, Reliant
experts who are available to assist any Reliant plant in the resolution of maintenance and
operations problems. One section, entitled "Mobile Maintenance Team,” applied only to
plants outside of California, as confirmed by the plant manager.

Audit Number GO167-1006 35


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Maintenance Standard 2, “Organizational Structure and Responsibilities,” states:

The organization with responsibility and accountability for establishing and


implementing a maintenance strategy to support company objectives for reliable
station operation is clearly defined, communicated, understood and is effectively
implemented. Reporting relationships, control of resources, and individual authorities
support and are clearly defined and commensurate with responsibilities.

Maintenance Standard 4, “Problem Resolution and Continuing Improvement,” states:

The company values and fosters an environment of continuous improvement and


timely and effective problem resolution.

Operation Standard 3, “Operations Management and Leadership,” states:

Operations management establishes high standards of performance and aligns the


operations organization to effectively implement and control operations activities.

Observation 3.3. The Plant Bases Employee Compensation on


Performance Evaluations.
According to the plant manager, the plant appraises each employee annually, with
midyear informal discussions. The direct supervisor rates employee performance on a
five-point scale. Employees who receive the lowest rating, “needs improvement,” must
improve performance or risk termination. The plant bases raises on the performance
rating, within minimum and maximum levels for each position at the plant.

Additionally, Reliant awards team bonuses up to 5% of gross pay for meeting the
performance goals discussed in Observation 3.1. The plant calls this program the
“Annual Incentive Compensation Program”.

The CPSD auditor reviewed the plant organization chart. The chart contained three
repeat entries for the maintenance manager position: one entry listing the maintenance
manager’s name, and the other two entries omitting the name. The plant manager stated
that he was aware of this typographical error and would correct it.

Audit Number GO167-1006 36


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Maintenance Standard 14, “Engineering and Technical Support,” states:

Engineering activities are conducted such that equipment performance supports


reliable plant operation. Engineering provides the technical information necessary for
the plant to be operated and maintained within the operating parameters defined by
plant design.

Operations Standard 9, “Engineering and Technical Support,” states:


Engineering activities are conducted such that equipment performance supports
reliable plant operation. Engineering provides the technical information necessary for
the plant to be operated and maintained within the operating parameters defined by
plant design. Engineering provides support, when needed, to operations and
maintenance groups to resolve operations and maintenance problems

Observation 3.4. The Plant Thoroughly Investigated Turbine Bearing


Failures.
The plant followed its detailed procedure for root cause analysis (see Observation 3.5) to
solve a turbine bearing problem. On three separate occasions in September, October, and
November of 2004, thrust bearings on the Unit 3 steam turbine failed following high
vibration alarms.

Following the September 2004 and October 2004 problems, the plant performed in-house
root cause analysis and discovered that

ƒ Operators ignored what they thought were false or insignificant (“nuisance”)


alarms,
ƒ Operators wrote illegible entries in logs, and could not identify which alarms had
activated,
ƒ Plant contractors worked without written quality standards, and applied improper
torque to parts of the steam turbine.

In response, the plant analyzed the alarm system, eliminating nuisance alarms; retrained
operators; adopted typed logs in place of handwritten logs; and implemented a policy
requiring all contractors to use written quality standards.

In November 2004, the plant again detected excessive turbine vibration, along with high
temperature in bearing oil, which ignited. The plant asked General Electric (GE) to
thoroughly analyze the problem. CPSD auditors reviewed GE’s “Steam Turbine
Inspection Report”, which identified the main problem as grease on the turbine
midstandard (a supporting structure) which solidified when Reliant mothballed Units 3
and 4. The solidified grease impeded the turbine shaft, leading to high vibration and
damage to the thrust bearings. The GE report recommended two immediate corrective

Audit Number GO167-1006 37


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
actions, which the plant has implemented. First, the plant thoroughly cleaned turbine
surfaces and grease lines, used better grease, and added a preventive maintenance item
(#ETS-00-TU-PR02) for greasing. Second, at every startup, the plant monitors the
temperature of turbine thrust bearings, and compares them to normal operating
temperatures specified in GE’s turbine manuals.

Observation 3.5. The Plant Maintains and Uses a Detailed Procedure


for Root Cause Analysis.
The plant maintains and uses a detailed procedure for Root Cause Analysis. As part of the
Condition Based Maintenance (CBM) program, plant procedures include a documented
and detailed process flowchart for root cause analysis. The plant used the process to
perform a detailed root cause analysis on four occasions during the two years prior to the
audit. The Maintenance Manager Manual contains the flowchart, which describes how
the plant should determine if a root cause analysis is necessary, and how the plant should
conduct a root cause analysis. Additionally, the flowchart describes three levels of root
cause analysis - the incident report, the simplified root cause analysis, and the detailed
root cause analysis. The plant engineer showed the CPSD auditor a root cause notebook,
which contained examples of all three types of root cause analysis.

The plant engineer stated that Reliant is currently in the process of implementing
software called REASON to perform root cause analysis. REASON is a product
trademark and not an acronym for any process.

Maintenance Standard 12, “Spare Parts, Material and Services,” states:

Correct parts and materials in good condition, are available for maintenance
activities to support both forced and planned outages. Procurement of services and
materials for outages are performed in time to ensure materials will be available
without impact to the schedule. Storage of parts and materials support maintaining
quality and shelf life of parts and materials.

Operations Standard 11, “Operations Facilities, Tools, and Equipment,” states:


Facilities and equipment are adequate to effectively support operations activities.

Audit Number GO167-1006 38


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Observation 3.6. According to a Detailed Procedure, the Plant Tracks


Important Inventory by Project.
According to a documented and detailed procedure, the plant tracks important inventory
by project but distributes low-cost parts as needed. The plant maintains two distinct parts
of the warehouse. In one part, plant staff tracks inventory and assigns costs to specific
maintenance projects. In the other part, the plant distributes parts as needed, and restocks
when inventory runs low.

In the first section of the warehouse, devoted to relatively costly items, plant staff can
pick up parts only with requisitions, work orders, or planner approval on an emergency
basis. A dedicated worker runs the warehouse, and performs “receiving inspections” on
incoming parts. When workers take parts the dedicated warehouse worker records the
transaction on an “issue sheet,” which the plant later enters into the work management
system. The system bills parts and employee labor to the appropriate project
automatically. Generally, the system automatically reorders parts when inventory falls
below set points. If necessary, planners can order special parts.

The plant warehouse’s free stock area distributes small items such as fasteners and
electrical lugs and connectors. The plants contracts with Westco and Fastenol to
maintain electrical parts and mechanical parts, respectively.

Reliant employs a process known as “Partsco” for acquiring parts from warehouses that
the company shares between all power facilities. The CPSD auditor reviewed a
procedure for obtaining parts from the “Partsco” system.

Maintenance Standard 8, “Maintenance Procedures and Documentation,” states:

Maintenance procedures and documents are clear and technically accurate, provide
appropriate direction, and are used to support safe and reliable plant operation.
Procedures must be current to the actual methods being employed to accomplish the
task and are comprehensive to ensure reliable energy delivery to the transmission
grid.

Observation 3.7. The Plant’s Document Control Room is Neat and


Organized.
The CPSD auditor found the document control room very neat and organized. The CPSD
auditor checked a plant list of drawings against existing drawings and found no
discrepancies. One of the plant’s engineers took the initiative to update a list of available
schematics, drawings, and other documents. The plant stores the list in a notebook, and
in an Excel spreadsheet. The plant keeps these documents in a document control room.
The plant stores many of the documents in blueprint format on “sticks” stored in racks.

Audit Number GO167-1006 39


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
The CPSD auditor chose a stick at random (stick 13), and compared five documents on
the stick to the listing in the notebook. The CPSD inspector found no discrepancies.

Maintenance Standard 11, “Plant Status and Configuration,” states:

Station activities are effectively managed so plant status and configuration are
maintained to support reliable and efficient operation.

Operations Standard 8, “Plant Status and Configuration,” states:


Station activities are effectively managed so plant status and configuration are
maintained to support safe, reliable and efficient operation.

Observation 3.8. General Housekeeping and Equipment Labeling are


Good.
General plant housekeeping appeared to be very good, although a CPSD auditor observed
a loose piece of metal grating on one of the walkways on the boiler (see Finding 2.5).
The plant engineer immediately moved this metal out of the way. The equipment on the
units and major piping appears to be well labeled. The plant uses secondary containment
for hazardous wastes (Figure 8). Retired Units 1 and 2 were clean and marked with
barricade tape (Figure 9).

Figure 8. The plant uses secondary containment for hazardous waste.

Audit Number GO167-1006 40


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Figure 9. The plant cleaned and placed barricade tape around Units 1 and 2.

Audit Number GO167-1006 41


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Maintenance Standard 7, “Balance of Maintenance Approach,” states:

The maintenance program includes the proper balance of the various approaches to
maintenance, e.g., preventive, predictive, or corrective. The approach is adequately
documented with consideration of economics and reliability of equipment or
components, and their affect on reliable operation of the unit. Operating experience is
factored into the program. Maintenance procedures and documents should include the
generation equipment and all those components owned by the generation owner
directly connected to the plant that are an integral part of delivering power to the grid
including fuel supply systems, electrical

Maintenance Standard 14, “Engineering and Technical Support,” states:

Engineering activities are conducted such that equipment performance supports


reliable plant operation. Engineering provides the technical information necessary for
the plant to be operated and maintained within the operating parameters defined by
plant design.

Observation 3.9. The Plant Keeps Predictive Maintenance


Reports.
The plant keeps predictive maintenance reports. The plant stores those documents in a
dedicated storage area in the document control room. That area contained four shelves of
reports on predictive maintenance activities, including vibration, thermography, and oil
analysis testing. The CPSD auditor reviewed a September 2004 thermography report on
plant switchgear, which recommended no immediate repair.

Maintenance Standard 14 – Engineering and Technical Support, states:

Engineering activities are conducted such that equipment performance


supports reliable plant operation. Engineering provides the technical
information necessary for the plant to be operated and maintained within the
operating parameters defined by plant design.

Audit Number GO167-1006 42


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Observation 3.10. The Plant Has Improved Its Procedures and


Technical Support for Contract Work.

As a result of recent problems with the plant’s cooling towers, Etiwanda has improved its
management of contactor projects, through improved procedures and technical support.
In November 2004, Etiwanda hired a contractor to upgrade the cooling tower for Unit 4.
Without notifying Etiwanda, the contractor substituted a part not specified by plans. In
July of 2005, that cooling tower partially collapsed. Reliant adopted a corporate
procedure requiring the appointment of a project manager from Reliant’s Technical
Services office in Houston.

In 2004, Etiwanda contracted with a company specializing in cooling towers for


maintenance work. In particular, the company installed a distribution basin cover,
replaced the hot water deck, and installed new pipe saddle supports and discharge valves
in Etiwanda’s cooling tower. The contractor decided to replace existing valves with
smaller ones. Etiwanda failed to require the contractor to submit such design changes for
review. Whether or not the contractor notified Etiwanda of these changes, Etiwanda staff
failed to review them. Etiwanda’s outage planner was assigned primarily to monitor
safety, not to review design changes.

On July 21, 2005, the cooling tower partially collapsed. Because a drain was too small,
too much water collected in a header, whose supports were mounted in the wood of the
cooling tower. When workers removed the failed pipe header, they found extensive rot
and other structural damage to the wood. The combination of rot and excessive weight
caused the header and one of the decks of the cooling tower to fail.

Although Etiwanda does not believe that the contractor’s design change directly caused
the collapse, the plant recognizes that it needs to review the engineering and technical
calculations performed by its contractors. Etiwanda has assigned a liaison engineer to
oversee all contracted work. Further Etiwanda has asked the engineer to review all
contractor work since 2004.

Etiwanda is replacing much of the structure of the cooling tower, making extensive use of
engineering support from the plant, Reliant’s corporate office, and three independent
contractors. The plant’s engineer managed the first phase of the project, while an
engineer from Reliant’s corporate office is managing the second phase. The plant hired
three separate contractors to design the tower using computer modeling, build the tower,
and to inspect the quality of the construction.

Audit Number GO167-1006 43


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Maintenance Standard 5 - Maintenance Personnel Knowledge and Skills

Maintenance personnel are trained and qualified to possess and apply the
knowledge and skills needed to perform maintenance activities that support
safe and reliable plant operation.

Observation 3.11. The Plant Utilizes a Training Program.


Etiwanda provides training to employees to ensure that they have the knowledge and
skills to perform their jobs. Etiwanda offers training to its employees through computer-
based and instructor-led training.

Employees are automatically assigned to take computer-based training depending on their


responsibilities. Formerly, Etiwanda used “Plant Operation Online Learning” for its
computer-based training, but switched to General Physics (GP) about 3 months before the
audit. GP assigns each employee an account, which tracks training for that employee.

In addition to the computer-based training courses, the employees take instructor-led


classes. These classes include Boiler Stress Training and Turbine Starting and Loading
taught by Southern California Edison, and Confined Space Training by taught by David
Sullivan.

On the last day of plant visit, auditors observed lockout/tag-out training as well as hands-
on firefighting training conducted by an instructor from Inner-Flame (Oxnard, CA), a
company specializing in fire safety training. Standing about 8 feet away, each trainee
used a fire extinguisher to fight a fire in a metal structure about the size of a small trailer.
Observing from about 30 feet way, auditors felt the intensity of the heat.

Operation Standard 21 - Plant Security

To ensure safe and continued operations, each GAO provides a prudent level of
security for the plant, its personnel, operating information and communications,
stepping up security measures when necessary.

Audit Number GO167-1006 44


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Observation 3.12. The Plant Provides Security for the Plant and
Personnel.
Etiwanda provides security for the plant and personnel.
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________

________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
______________________

Operation Standard 14 – Clearances

Work is performed on equipment only when safe. When necessary, equipment is taken
out of service, de-energized, controlled, and tagged in accordance with a clearance
procedure. Personnel are trained in the clearance procedure and its use, and always
verify that equipment is safe before any work proceeds. Among other things:
The GAO prepares and maintains a clearance procedure. The clearance
procedure contains requirements for removing a component from service and/or
placing a component back into service.
The GAO ensures that personnel are trained in and follow the clearance
procedure.

Observation 3.13. The Plant Follows Tagout Procedures and is


Implementing a Lockout Procedure.
Etiwanda maintains a tagout clearance procedure and ensures that plant personnel follow
the procedure. An auditor watched plant staff confirm the procedure for tagout # 3442,
which isolated the East Heat Exchanger for Units 3 and 4 electrically and mechanically.
The operator printed out the list of tags for #3442, and then another staffer walked to
each physical location, checked each tag against the list, and verified that the heat
exchanger had been isolated.

Audit Number GO167-1006 45


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

About 10 months ago, Etiwanda began to physically lock equipment out-of-service,


rather than simply tagging it out. Some equipment, such as 4160 breakers, are not
designed for locks. The plant will modify such equipment to accommodate locks.

On the final day of the plant visit, the plant trained plant staff in lockout/tagout
procedures.

Maintenance Standard 1 – Safety states:

The protection of life and limb for the work force is paramount. The company
behavior ensures that individuals at all levels of the organization consider
safety as the overriding priority. This is manifested in decisions and actions
based on this priority. The work environment, and the policies and procedures
foster such a safety culture, and the attitudes and behaviors of individuals are
consistent with the policies and procedures.

Observation 3.14. The Plant Tests its Fire System Annually.


The plant maintains its fire system to ensure the safety of its property and personnel.
Each year, Etiwanda tests fire pump performance in October, and valve and alarm
performance in May. The auditor reviewed the Electric Fire Pump Performance Test
records from October 2005, which showed that five pumps passed the test. The auditor
also reviewed the 2005 Annual Deluge System Inspection Reports and Annual Fire
Alarm System Test and Inspection Reports for Units 3 and 4, and confirmed that the fire
system passed these tests.

Maintenance Standard 1 – Safety states:

The protection of life and limb for the work force is paramount. The company
behavior ensures that individuals at all levels of the organization consider
safety as the overriding priority. This is manifested in decisions and actions
based on this priority. The work environment, and the policies and procedures
foster such a safety culture, and the attitudes and behaviors of individuals are
consistent with the policies and procedures.

Observation 3.15. The Plant Held an Evacuation Drill and Implemented


Changes as a Result.

Audit Number GO167-1006 46


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
An evacuation drill on September 20, 2005 raised three issues. First, some personnel on
site could not hear the emergency siren or emergency instructions. Second, guards had
trouble printing lists of all people on site. Third, staff found inaccuracies in the plant’s
log of those entering and exiting the plant. The plant addresses all of these issues during
its review of the drill.

First, during the drill, the control operator started the emergency siren and ran it for one
minute. Then the operator used the public address system repeatedly to announce, “This
is an evacuation drill. Go to the #1 Evacuation Assembly Area in the Administration
Parking Lot.” Then the operator ran the siren again for one minute. People could not
hear the siren (or apparently, public announcements) from the storeroom, demineralized
water tanks, the guard shack, the makeup reservoir, the shop for Instrumentation &
Control (I & C) employees, or the electrician’s shop. The plant’s safety coordinator told
the auditor that the plant now uses radios to communicate with people in areas that the
siren and the public announcement cannot reach.

Secondly, the guard had trouble printing out a list of people on site in part because his
computer lacked an icon that could easily access the log file. Since then, the plant has
added such an icon. The guard did keep a handwritten log of contractors and guests.

Lastly, a review of the log showed that the guard had recorded incorrect information on
the day of the drill. The entry and exit time for a FedEx deliveryman was assigned to a
UPS deliveryman, and vice versa. Plant personnel pointed out the mistake. Plant
management counseled the security guard about the importance of accuracy.

Operation Standard 15 - Communications and Work Order Meetings states:

The availability of the generating asset and safety of personnel is ensured


during the execution of work orders by adequate communications and
meetings, which may be scheduled or as needed, to review work plans with all
affected personnel before work begins. Clear lines of communication exist
between personnel responsible for operations, maintenance and engineering
groups.

Observation 3.16. The Plant Holds Daily Staff Meetings.


On the third day of the audit, two auditors attended the 6:30 am meeting of the plant staff.
During that meeting, staff raised two issues. First, a valve on the cooling tower had been
accidentally cut off during refurbishing, and was replaced with a temporary plug to keep
water from leaking out. Staff briefly discussed a more permanent repair. Second, staff
discussed the plant gate, which had failed to open automatically and was operating in
manual mode. Management announced firefighting and lockout/tag-out training the next
day.

Audit Number GO167-1006 47


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

Maintenance Standard 8, “Maintenance Procedures and Documentation”, states:

“Maintenance procedures and documents are clear and technically accurate,


provide appropriate direction, and are used to support safe and reliable plant
operation. Procedures must be current to the actual methods being employed
to accomplish the task and are comprehensive to ensure reliable energy
delivery to the transmission grid.”

Operating Standards 1, “Safety”, states:

“The protection of life and limb for the work force is paramount. GAOs have
a comprehensive safety program in place at each site. The company behavior
ensures that personnel at all levels of the organization consider safety as the
overriding priority. This is manifested in decisions and actions based on this
priority. The work environment and the policies and procedures foster such a
safety culture, and the attitudes and behaviors of personnel are consistent with
the policies and procedures.”

Operations Standard 12, “Operations Conduct”, states:

“To ensure safety, and optimize plant availability, the GAO conducts
operations systematically, professionally, and in accordance with approved
policies and procedures. The GAO takes responsibility for personnel actions,
assigns personnel to tasks for which they are trained, and requires personnel
to follow plant and operation procedures and instructions while taking
responsibility for safety. Among other things:

A. All personnel follow approved policies and procedures. Procedures are


current, and include a course of action to be employed when an adopted
procedure is found to be deficient.”

Observation 3.17. The Plant’s Chemistry Lab Procedures are


Comprehensive.
The chemistry lab procedures appear comprehensive, covering:
o Safety
¾ Handling of equipment and chemical
¾ Housekeeping

Audit Number GO167-1006 48


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
¾ Work habits
¾ Protection for eyes, face, ears, respiratory
o Chemical Control
¾ Calibration of equipment
¾ Reagent management
¾ Standardization
¾ Records and reports
¾ Chemical control limits (boiler water, condensate, feedwater and
make-up water)
¾ Corrective actions on out-of-limit steam or water chemistry conditions
¾ Testing frequency
¾ Outage protection
¾ Startups
¾ Test procedures
o Chemical analysis (copper, hydrazine, iron, nitrate, ammonia,
benzatriazole, phosphorus, and silica)

The auditor interviewed the chemical technician and observed several tests performed
in the lab including pH, hydrazine, and O2 testing from the circulating and feed water
bench, plus a standardization test for pH level 7 and 10. Except as noted in Finding
2.3 and 2.9, the plant follows written procedures.

Audit Number GO167-1006 49


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT

GO 167, Appendix B. “Generator Logbook Standards (Thermal)”, Section II states:

Each generating facility shall maintain a Control Operator Log that contains
the chronological history of the facility including detailed entries regarding
the operations and maintenance of the facility. Where information is unit
specific, information for each unit must be recorded and so identified.

The Control Operator Log is a formal record of real time operating events as
well as the overall status of the generating units and auxiliary equipment
under the purview of the Control Room Operator. The log shall also contain
an accurate and concise record of important and/or unusual events involving
operations, maintenance, water chemistry, safety, accidents affecting
personnel, fires, contractor activities, environmental matters, and any other
pertinent information concerning the operation of the facility. The log shall
also record communications between the facility and outside entities including
but not limited to the Independent System Operator (ISO), scheduling
coordinators or headquarters facilities, regulators, environmental agencies,
CalOSHA or similar agencies. The log shall be maintained notwithstanding
and in addition to any other similar requirements that mandate that events be
recorded. The generator must collect and record all information specified in
these standards. All such information must be readily available to operators,
California Public Utilities Commission staff, and other authorized personnel at
all times.

GO 167, section 10.3.2 “Submission of Information to NERC” states:

Except for Generating Assets for which NERC does not accept data, each
Generating Asset Owner shall submit generator design, performance, and
event data to NERC for inclusion in GADS. Within the categories of data that
NERC accepts, CPSD may specify the data the Generating Asset Owner must
submit to NERC. If requested by CPSD, a Generating Asset Owner shall
concurrently provide CPSD with a copy of all data submitted to NERC for
inclusion in GADS.

Observation 3.18. Auditors Found no Discrepancies between Plant


Logs and other Reported Data.
The CPSD auditor reviewed GADS data, and compared that data to plant logbooks as
well as data from the SLIC data for selected random dates, and found no significant

Audit Number GO167-1006 50


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
conflicts in the information provided

The following table shows the dates that were chosen, the units involved, and the
corresponding GADS data, logbook entry data, and SLIC. Note that RS in GADS data is
an abbreviation for Reserve Shutdown.

Unit Date GADS data Logbook SLIC data Significant


Entry Discrepancies
3 7-01-05 NO GADS DATA Take unit 3 to Troubleshoot None
AGC0Low combustion
Range control
computer
problem
7-02-05 Start time –0:17:00, RS NO SLIC None
Type- RS, ENTRY
Cause Code – 0000,
Hours – 149.72,
MW loss – 0

7-03-05 NO GADS DATA RS NO SLIC None


ENTRY
7-04-05 NO GADS DATA RS NO SLIC None
ENTRY
7-05-05 NO GADS DATA RS NO SLIC None
ENTRY
7-06-05 NO GADS DATA RS NO SLIC None
ENTRY
7-07-05 NO GADS DATA RS NO SLIC None
ENTRY
7-10-05 NO GADS DATA RS NO SLIC None
ENTRY
7-11-05 NO GADS DATA RS NO SLIC None
ENTRY
7-12-05 NO GADS DATA 108 MW NO SLIC None
ENTRY
7-13-05 NO GADS DATA 316 MW NO SLIC None
ENTRY
7-14-05 NO GADS DATA 294 MW Total unit None
performance
testing
7-15-05 NO GADS DATA 70 MW NO SLIC None
ENTRY
7-16-05 NO GADS DATA 55 MW NO SLIC None
ENTRY

Audit Number GO167-1006 51


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
7-17-05 NO GADS DATA 25 MW NO SLIC None
ENTRY
7-18-05 NO GADS DATA Take #3 to low NO SLIC None
range AGC ENTRY
1-21-06 Start time 20:35:00, Down with NO SLIC None
Type- RS, boiler tube leak ENTRY
Cause Code – 0000, and
Hours – 197.79, miscellaneous
MW loss – 0 repairs – outage
#763079
1-22-06 NO GADS DATA RS NO SLIC None
ENTRY
1-23-06 NO GADS DATA RS NO SLIC None
ENTRY
1-24-06 NO GADS DATA RS NO SLIC None
ENTRY
1-25-06 NO GADS DATA RS NO SLIC None
ENTRY
1-26-06 NO GADS DATA RS NO SLIC None
ENTRY
1-27-06 NO GADS DATA RS NO SLIC None
ENTRY
1-28-06 NO GADS DATA RS Boiler None
inspection
1-29-06 NO GADS DATA In start up NO SLIC None
ENTRY
1-30-06 NO GADS DATA 70 MW – High NO SLIC None
range control ENTRY
1-31-06 Start time –0:20:00, RS NO SLIC None
Type- RS, ENTRY
Cause Code – 0000,
Hours – 23.67,
MW loss – 0

2-01-06 Start time –0:00:00, RS NO SLIC None


Type- RS, ENTRY
Cause Code – 0000,
Hours – 270.97,
MW loss – 0

2-02-06 NO GADS DATA RS NO SLIC None


ENTRY
2-03-06 NO GADS DATA RS NO SLIC None
ENTRY
2-04-06 NO GADS DATA RS NO SLIC None
ENTRY

Audit Number GO167-1006 52


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
2-05-06 NO GADS DATA RS NO SLIC None
ENTRY
4 7-01-05 Start time –0:00:00, RS NO SLIC None
Type- RS, ENTRY
Cause Code – 0000,
Hours – 93.08,
MW loss – 0

7-02-05 NO GADS DATA RS NO SLIC None


ENTRY
7-03-05 NO GADS DATA RS NO SLIC None
ENTRY
7-04-05 NO GADS DATA Start up NO SLIC None
ENTRY
7-05-05 NO GADS DATA 125 MW NO SLIC None
ENTRY
7-06-05 NO GADS DATA 280 MW NORMAL None
OUTAGE TO
COVER
TRANSITIO
N TO LOW
RANGE
7-07-05 NO GADS DATA 78 MW NO SLIC None
ENTRY
7-09-05 Start time –0:32:00, RS NO SLIC None
Type- RS, ENTRY
Cause Code – 0000,
Hours – 48.47,
MW loss – 0

7-10-05 NO GADS DATA RS NO SLIC None


ENTRY
7-11-05 NO GADS DATA 40 MW NO SLIC None
ENTRY
7-12-05 NO GADS DATA 248 MW NO SLIC None
ENTRY
7-13-05 NO GADS DATA 320 MW NO SLIC None
ENTRY
7-14-05 NO GADS DATA 196 MW NO SLIC None
ENTRY
7-15-05 NO GADS DATA 70 MW NO SLIC None
ENTRY
7-16-05 Start time –0:08:00, RS NO SLIC None
Type- RS, ENTRY
Cause Code – 0000,
Hrs–33.57,

Audit Number GO167-1006 53


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
MW loss – 0

7-17-05 NO GADS DATA Start up NO SLIC None


ENTRY
7-18-05 NO GADS DATA NO LOG NO SLIC None
ENTRY ENTRY
1-20-06 NO GADS DATA 120 MW on NO SLIC None
high range ENTRY
control
1-21-06 Start time 20:35:00, RS NO SLIC None
Type- RS, ENTRY
Cause Code – 0000,
Hours – 197.79,
MW loss – 0

1-22-06 NO GADS DATA RS NO SLIC None


ENTRY
1-23-06 NO GADS DATA RS NO SLIC None
ENTRY
1-24-06 NO GADS DATA RS NO SLIC None
ENTRY
1-25-06 NO GADS DATA RS NO SLIC None
ENTRY
1-26-06 NO GADS DATA RS NO SLIC None
ENTRY
1-27-06 NO GADS DATA RS NO SLIC None
ENTRY
1-28-06 NO GADS DATA RS NO SLIC None
ENTRY
1-29-06 NO GADS DATA In start up NO SLIC None
ENTRY
1-30-06 NO GADS DATA 70 MW – High NO SLIC None
range control ENTRY
1-31-06 Start time –0:20:00, RS NO SLIC None
Type- RS, ENTRY
Cause Code – 0000,
Hours – 23.67,
MW loss – 0

2-01-06 Start time –0:00:00, RS NO SLIC None


Type- RS, ENTRY
Cause Code – 0000,
Hours – 270.97,
MW loss – 0

Audit Number GO167-1006 54


ETIWANDA GENERATION STATION AUDIT PRELIMINARY REPORT
GENERAL ORDER 167 AUDIT
2-02-06 NO GADS DATA RS NO SLIC None
ENTRY
2-03-06 NO GADS DATA RS NO SLIC None
ENTRY
2-04-06 NO GADS DATA RS NO SLIC None
ENTRY
2-05-06 NO GADS DATA RS NO SLIC None
ENTRY

Audit Number GO167-1006 55

You might also like