Professional Documents
Culture Documents
TABLE OF FIGURES
Figure 1. Water creates hazard in walkway. .................................................................... 20
Figure 2. The plant added non-slip surface to the walkways........................................... 20
Figure 3. The plant added calibration stickers on equipment. ......................................... 27
Figure 4. The plant left clutter in the chemistry lab.......................................................... 29
Figure 5. The plant used the fume hood for storage. ....................................................... 30
Figure 6. The plant cleaned and placed a warning sign on the fume hood...................... 30
Figure 7. The plant added signs to evacuation areas. ...................................................... 31
Figure 8. The plant uses secondary containment for hazardous waste. .......................... 40
Figure 9. The plant cleaned and placed barricade tape around Units 1 and 2. ................ 41
Executive Summary
This is the Final Report from the audit of the Etiwanda Power Plant prepared by the
Consumer Protection and Safety Division (CPSD) of the California Public Utilities
Commission (CPUC). CPSD audited the plant for compliance with the Commission’s
General Order 167, which includes Operation, Maintenance, and Logbook Standards for
power plants.
On May 5, 2006, CPSD notified Etiwanda of the audit and requested pertinent documents
and data. CPSD staff visited the plant site from June 4, 2006 through June 8, 2006 in
order to observe plant operations, inspect equipment, examine documents, and interview
plant staff and managers. The CPSD Audit Team reviewed specific incidents and
problems in plant operation to evaluate whether the plant needed improvements in
operation and maintenance processes, training, or policies. The Audit Team also
examined whether the plant’s programs and procedures met various Operation,
Maintenance, and Logbook Standards.
CPSD sent Etiwanda a Preliminary Audit Report on July 13, 2007. Etiwanda responded
with a Corrective Action Plan (CAP) on August 21, 2007. CPSD and Etiwanda resolved
all outstanding issues in an October 23, 2007 conference call. The audit team leader
visited the plant on December 12, 2007 to verify completion of several corrective actions
and clarify some minor issues. Etiwanda cooperated fully throughout the entire audit
process. CPSD now issues this Final Report.
Finding 2.1. CPSD found that the plant failed to establish and follow clear test
procedures for turbine overspeed. In response, the plant rewrote the procedures,
requiring annual full overspeed tests according to clear step-by-step instructions.
Finding 2.2. CPSD found that the plant deferred needed maintenance prescribed
by the plant’s own equipment condition assessment. In response, the plant hired a
new manager for the program, developed a new schedule and procedure, and
performed much of the high priority work from the original schedule. By
December 31, 2008, the plant agrees to provide CPSD with repair records for all
the original high priority items and all boiler and piping repairs listed on the
original schedule.
Finding 2.4. CPSD found that the plant’s work management system contains
obsolete work orders, as well as cooling tower inspection orders that don’t match
the plant’s maintenance plan. In response, the plant updated the work
management system.
Finding 2.5. CPSD found that the plant failed to identify and correct hazards to
foot and vehicle traffic, including standing water on the turbine deck, loose
grating in a walkway, and a two-foot hole in a dirt road adjacent to the plant.
In response, the plant corrected these hazards.
Finding 2.6. CPSD found that the plant fails to calibrate all test equipment. In
response, the plant hired a contractor to implement a comprehensive test program.
The plant monitors the calibration through a website.
Finding 2.7. CPSD found that the plant fails to use written checklists for all daily
inspections. In response, as part of a Reliant corporate initiative to improve
policies and procedures, Etiwanda will begin using checklists for inspections,
including the daily inspection. CPSD requests that the plant submit quarterly
updates on the effort.
Finding 2.8. CPSD found that the plant fails to maintain chemistry equipment
calibration logs. In response, the plant created comprehensive calibration logs.
Finding 2.9. CPSD found that the plant fails to follow chemistry lab
housekeeping procedures. In response, the plant cleaned and organized the
chemistry lab and fume hood, and posted a sign prohibiting storage in the fume
hood.
Finding 2.10. CPSD found that the plant fails to mark evacuation areas. In
response, the plant added evacuation signs to both evacuation areas.
Finding 2.11. CPSD found that plant security entered an incorrect list of visitors
on site in the plant gate log. In response, the plant created a new business
manager position to oversee the work of the security personnel.
Finding 2.12. CPSD found that a plant report describes abandoned fire hose
stations as “Missing.” In response, the plant audited all fire station equipment and
updated fire system inspection lists. The plant removed all unused hose stations.
Introduction
Beginning in May 2006, CPSD audited the Etiwanda Generating Station to determine
whether the plant was in compliance with General Order (GO) 167. GO 167 includes
Maintenance, Operation, and Logbook standards for power plants.1 The audit team
included Ben Brinkman, Winnie Ho, Chris Lee and Alan Shinkman.
CPSD conducted the audit through an examination of plant performance, data requests
and a visit to the plant site. First, the team examined outage reports by CPSD staff, as
well as databases maintained by California Independent System Operator (ISO). On
May 5, 2006, the team notified the plant of the audit. The team visited the plant site from
June 4, 2006 to June 8, 2006, examining documents, interviewing staff, inspecting
equipment, and observing operations. At the conclusion of the site visit, the team
presented the plant with a data request.
CPSD sent Etiwanda a Preliminary Audit Report on July 13, 2007. Etiwanda responded
with a Corrective Action Plan (CAP) on August 21, 2007. Etiwanda agreed with all but
four of the report’s recommendations. CPSD and Etiwanda held a conference call on
October 23, 2007 during which the four outstanding issues were resolved. The audit
team leader visited the plant on December 12, 2007 to verify completion of several
corrective actions. Etiwanda cooperated fully throughout the entire audit process.
Plant Description
Reliant’s Etiwanda Generating Station sits in the industrial outskirts of Rancho
Cucamonga, east of Los Angeles. At one time, the plant consisted of five units. In 2003,
the plant retired Units 1, 2, and 5, and mothballed Units 3 and 4. Reliant returned Units 3
and 4 to service in 2004 when CAISO awarded those units Reliability-Must-Run (RMR)
contracts.2 Units 3 and 4 remained on RMR contracts at the time of the site visit, but
CAISO released them from these contracts for 2007.
The plant now consists of Units 3 and 4, which generate 320 Megawatts (MWs) each, for
a total of 640 MW. Commissioned in 1962, Units 3 and 4 utilize General Electric (GE)
steam turbines and controlled-circulation boilers manufactured by Combustion
Engineering. While most boilers have the firebox below and water tubes above, this
boiler reverses that configuration, using pumps to drive water through the boiler. Unlike
most of California’s conventional steam plants, Etiwanda is located inland and utilizes a
1
Further information on the Commission’s Power Plant Performance program may be found at the
Commission’s Web Site at
http://www.cpuc.ca.gov/PowerPlantStandards.
2
The California Independent System Operator (CAISO) had declared Units 3 and 4 to be Reliability Must
Run (RMR) units to preserve the stability of the grid, but concluded this contract as of 2007. From the
CAISO website, RMR is “an annual process that identifies geographical areas with local reliability issues
in the ISO Control Area along with measures to remediate those issues. Reliability Must Run Contracts are
a means of ensuring power availability within identified areas.”
Plant Performance
CPSD studied the performance of Etiwanda’s active units, numbers 3 and 4.3 Staff
examined detailed data for 2003-2006 (note that Reliant mothballed the plant briefly in
2004), and more aggregated data for 1998-2003. Like many conventional boiler units in
California, Etiwanda ran at low capacity factors; from 2004-2006 Units 3 and 4 ran at
12% and 15% respectively.
For 2004 to 2006, Etiwanda units performed as well or better than similar California units
in almost all categories. Although the units started slightly less reliably than other plants,
they outperformed other plants on all other measures. Unit 4’s EFORd (a measure of the
plant’s forced outages when the ISO needed it) was particularly low: 3.14 compared with
the California average of 9.95. Unit 3’s EFORD was roughly 8. Overall availability was
90%.
In the years before the plant was mothballed, the plant operated similarly, except that its
overall availability was somewhat lower; around 80%. The plant took relatively more
scheduled outages during this period. EFORd was around 8 for both units.
While overall performance has been good, a number of specific incidents reduced plant
capacity significantly. If the plant had prevented these problems, its performance would
have been much better.
• Unit 3’s turbine problems (see Observation 3.4) increased EFORd and reduced
equivalent availability in the 4th quarter of 2004.
• A cooling tower collapsed in July 2005, during a Stage 2 emergency (See
Observation 3.10). The collapse affected neither EAF nor EFORd significantly
during this period. However, during the 2nd and 3rd quarters of 2006, resulting
repairs to the cooling tower reduced overall availability significantly. The plant
completed repairs just before the heat storms of 2006, reducing EFORd during
this critical period.
• In the third quarter of 2003, prior to mothballing, brushes short-circuited on Unit
4’s low pressure generator, causing an extended outage in August and September
of 2003. The plant’s root cause analysis showed that a contractor’s technician had
misaligned the generator’s brushes. EFORd spiked to 70 for that quarter.
3
CPSD examined indices established by the North American Electric Reliability Council (NERC). The
study includes six performance indices: (1) Equivalent forced outage rate during demand (EFORd), (2)
Equivalent Availability Factor (EAF), (3) Net Capacity Factor (NCF), (4) Start Reliability (SR), (5)
Scheduled Outage Factor (SOF), and finally (6) Forced Outage Factor (FOF). A lower EFORd indicates
fewer equivalent forced outage hours when ISO needs the unit to generate, and is therefore generally better
than a higher EFORd.
Audit Scope
The team looked broadly at the plant’s compliance with standards, and also focused on
problems identified from the plant’s operating history. Major incidents and problems
include:
The audit also examined the plant’s compliance with specific standards, including those
covering:
The team visited the plant site from June 4, 2006 to June 8, 2006. Team members toured
the plant, including the generating units, the control room, the warehouse, chemistry lab,
mechanical shop and electrical shop. Plant staff demonstrated the plant’s maintenance
management software. CPSD auditors attended a plant shift change (tailboard) meeting,
and a training session in which plant staff extinguished test fires. CPSD auditors
reviewed numerous documents and interviewed plant staff. CPSD auditors requested and
reviewed additional documents after the site visit.
Section 2 includes findings of potential violations that require corrective action as soon as
reasonably possible.
Finding 2.1. In the Preliminary Audit Report, CPSD found that the plant failed to
establish and follow clear test procedures for turbine overspeed. In response, the
plant rewrote the procedures, requiring annual full overspeed tests and including
clear step-by-step instructions.
Finding 2.2. In the Preliminary Audit Report, CPSD found that the plant deferred
some needed maintenance activities, which were prescribed by the plant’s
condition-based maintenance assessment. In response, the plant hired a new
manager for the condition-based maintenance program, and developed a new
schedule and procedure. Additionally, the plant performed much of the high
priority work from the original schedule, including repairs to the cathodic
protection system. The plant provided an updated repair schedule. By December
31, 2008, the plant agrees to provide CPSD with repair records for all the original
high priority items and all boiler and piping repairs listed on the original schedule.
Finding 2.3. In the Preliminary Audit Report, CPSD found that the plant fails to
maintain proper hydrazine levels in the condensate and feedwater during normal
operations. In response, the plant is refurbishing the entire chemistry lab by
installing or replacing various analyzers (including the hydrazine analyzer),
sensors, and alarms that will feed continuous data into a centralized PI database.
Reliant headquarters will remotely monitor plant activities via PI data. The plant
plans to complete the $70,000 refurbishment project by the end of March 2008.
EGPB requests that the plant provide photos of the refurbished lab and the plant’s
report on the refurbishment project.
Finding 2.4. In the Preliminary Audit Report, CPSD found that the plant’s work
management system contains obsolete work orders, as well as cooling tower
inspection orders that don’t match the plant’s maintenance plan. In response, the
plant updated the work management system, removing the obsolete work orders,
the first quarter of 2007. The plant changed the frequency of preventive
maintenance (PM) for cooling tower inspections to match that in the maintenance
governance. The plant sent copies of these changes in PM to CPSD.
Finding 2.5. In the Preliminary Audit Report, CPSD found that the plant fails to
identify and correct hazards to foot and vehicle traffic. First, auditors found a
pool of standing water on the turbine deck. Second, an auditor nearly tripped on a
piece of grating in front of the West Boiler Circulating Water Pump. Third, in
In response, the plant took corrective steps and set procedures in place to address
the hazards to foot traffic. The plant placed non-skid surface all over the turbine
deck, including the area where water had accumulated at the time of the onsite
audit visit. In response to the loose grating piece found on a walkway, the plant
manager now conducts monthly housekeeping inspections. The plant also welded
down several areas of loose grating.
With regard to the erosion on a dirt road which caused an injury, the plant
reemphasized to the staff the importance of following plant vehicle procedures
and the incident reporting process. The plant called Southern California Edison
(SCE), which owns the road, to request repairs.
Finding 2.6. In the Preliminary Audit Report, CPSD found that the plant fails to
assure the calibration of all test equipment. In response, the plant contracted a
company called TRANSCAT to track and calibrate all test equipment, including
multimeters. TRANSCAT notifies the plant electronically when calibration is
due, and calibrates the equipment on the plant site. The plant has access to all
calibration data through a website.
Finding 2.7. In the Preliminary Audit Report, CPSD found that the plant fails to
use written checklists for all daily unit “walkdown” inspections. In response, as
part of a Reliant corporate initiative to improve policies and procedures, Etiwanda
will begin using checklists for most inspections, including the daily walkdowns.
The CPSD inspector reviewed progress on the checklists in December 2007. The
plant intends to implement the new checklists during 2008. CPSD requests that
the plant submit quarterly updates on the effort.
Finding 2.8. In the Preliminary Audit Report, CPSD found that the plant fails to
maintain calibration logs for equipment that monitors feedwater chemistry. In
response, the plant created equipment calibration logs. The logs list the
inspection cycle, calibration date, and calibration data for each piece of equipment
in the lab. In addition, during a December 12, 2007 plant visit, EGPB staff saw
stickers on equipment showing dates for the last calibration and next one due.
Finding 2.9. In the Preliminary Audit Report, CPSD found that the plant fails to
follow chemistry lab housekeeping procedures. In response, the plant removed
storage material from the fume hood area, and posted a sign indicating no storage
in the fume hood. During a December 12, 2007 plant visit, EGPB staff found the
chemistry lab clean. No further action is needed.
Finding 2.11. In the Preliminary Audit Report, CPSD found that plant security
entered an incorrect list of visitors on site in the plant gate log. In response, the
plant created a new business manager position to oversee the work of the security
personnel. One of the responsibilities of the business manager is to periodically
check the gate logs for accuracy. The plant submitted to CPSD sample copies of
gate logs that have been checked and signed by the business manager.
Finding 2.12. In the Preliminary Audit Report, CPSD found that a plant report
describes abandoned fire hose stations as “Missing.” In response, the plant
audited all fire station equipment and updated fire system inspection lists. The
plant removed all unused hose stations.
Observation 3.1. Reliant Regularly Audits its own Plants and Rates its own
Performance.
Observation 3.5. The Plant Maintains and Uses a Detailed Procedure for Root
Cause Analysis.
Observation 3.7. The Plant’s Document Control Room is Neat and Organized.
Observation 3.10. The Plant Has Improved Its Procedures and Technical Support
for Contract Work.
Observation 3.12. The Plant Provides Security for the Plant and Personnel.
Observation 3.15. The Plant Held an Evacuation Drill and Implemented Changes
as a Result.
SECTION 1 FINDINGS
Staff found no safety hazards requiring immediate correction.
SECTION 2 FINDINGS
Operations Standard 7, “Operation Procedures and Documentation,” states:
“Operation procedures exist for critical systems and states of those systems
necessary for the operation of the unit including startup, shutdown, normal
operation, and reasonably anticipated abnormal and emergency conditions.
Operation procedures and documents are clear and technically accurate,
provide appropriate direction, and are used to support safe and reliable plant
operation. Procedures are current to the actual methods being employed to
accomplish the task and are comprehensive to ensure reliable energy delivery
to the transmission grid.”
“GAO complies with these Operation Standards (1-27) considering the design
bases (as defined in the Appendix) of plant equipment and critical systems. The
GAO considers the design basis of power plant equipment when as required by
other standards it, among other things:
B. For each system, identifies critical parameters that require monitoring (Ref.
Standard No. 8 and 13).
D. Assures that systems are monitored and actions are taken (Ref. Standard 8
and 13).”
Finding 2.1. The Plant Fails to Establish and Follow Clear Test
Procedures for Turbine Overspeed.
The plant failed to establish and follow clear test procedures for turbine overspeed. First,
the plant’s test procedure is poorly written and unclear. Second, the plant failed to
conduct tests every year, as its procedures apparently require. Finally, reminders in the
work scheduling system appear ineffective.
The plant’s test procedure (ETS-00-TU-PO02) is poorly written and unclear. First, the
procedure fails to clearly explain when to use an overspeed test (where operators run the
turbine at 110% of synchronous speed) versus an oil trip test (where operators inject oil
into the turbine governor). Second, while the procedure requires annual testing, it is
For Unit 4, the plant failed to conduct overspeed tests every year as the procedure appears
to require. Records show an overspeed test on Unit 4 in 2004 and on Unit 3 in 2006. The
plant manager stated that the plant conducted many oil trip tests in 2005, but no
overspeed test on either unit. In late June 2006, on three consecutive days, Unit 4’s
overspeed protection scheme tripped even though the turbine was operating normally.
Resulting outages totaled over 60 hours. In response, the plant tightened the governor
linkage, reset the trip speed, and replaced electrical monitoring equipment.4 CPSD
believes that proper tests might have detected these problems and prevented the outages.
Finally, reminders in the work scheduling system appear ineffective. Reminders for
overspeed tests (ETS04TUOPO01) appear every six months, even though procedures call
for tests only during outages. Apparently staff routinely ignores these reminders.
In response to the Preliminary Audit Report, the plant rewrote its overspeed test
procedures to clearly require full overspeed tests at least every twelve months. The new
procedures detail required tests step-by-step, and are consistent with the reminders in the
work scheduling system. CPSD auditors verified the new procedures, and will check the
plant’s 2008 overspeed trip test records for compliance. The plant states that staff did not
ignore reminders; rather they performed “oil-trip” tests.
4
From an independent audit required under a settlement between Reliant and the CPUC, titled
“Independent Audit of Unit Outages Related to Joint Settlements of FERC Dockets.”
5
“Etiwanda Generating Station Critical Equipment Schedule.” This document is dated May 31, 2006.
In response to the Preliminary Audit Report, the plant hired a new manager for the
condition-based maintenance program, and developed a new schedule and procedure.
Additionally, the plant performed much of the high priority work from the original
schedule, including repairs to the cathodic protection system. The plant provided an
updated repair schedule. By December 31, 2008, the plant agrees to provide CPSD with
repair records for all the original high priority items and all boiler and piping repairs
listed on the original schedule.
“To ensure safety, and optimize plant availability, the GAO conducts
operations systematically, professionally, and in accordance with approved
policies and procedures. The GAO takes responsibility for personnel actions,
assigns personnel to tasks for which they are trained, and requires personnel
to follow plant and operation procedures and instructions while taking
responsibility for safety. Among other things:
During normal operation, hydrazine levels in condensate and feedwater often exceed the
limit stated in plant procedures of 20 parts per billion (ppb), in part because the plant’s
chemistry technician believed that those higher concentrations are acceptable. The
technician stated that acceptable hydrazine levels fall between 20 to 120 ppb during
normal operations By contrast, the plant’s chemistry lab procedures require hydrazine
levels in condensate and feedwater to be 20 ppb or less. Above this level, the procedure
Even though plant procedures require continuous monitoring during excursions,7 the
plant fails to do so, in part because monitors are broken. Instrumentation that
continuously monitors and controls hydrazine levels broke down in October 2004, and
the plant has not fixed it. As a result, the chemical technician must manually test and
control the hydrazine levels, which he cannot do continuously. Therefore the plant
cannot effectively control hydrazine levels during excursions.
6
Procedures titled “Etiwanda Generating Station Chemistry Lab.” Section 10.2, “Rationale for Action Levels” states
that “action levels for control parameters at critical sample points were developed to provide guidance on the relative
need for corrective action in the event of cycle chemistry excursions. Normal target values are defined as values
associated with long term stability. A safe margin has been provided to avoid concentration of contaminants on the
surfaces.”
Section 10.4, “Rationale for Target Values of Individual Parameters” states that hydrazine concentration at the plant
has “an upper limit of 20 ppb” since there is evidence linking higher hydrazine level with corrosion of ferrous material
in the feedwater train.
7
“Etiwanda Generating Station Chemistry Lab” procedures, more specifically, Section 4.2, titled
“Chemistry Control for the Deaerator Inlet,” requires continuous sampling of hydrazine levels during
excursions.
EGPB requests that the plant provide photos of the refurbished lab and the plant’s report
on the refurbishment project.
8
Maintenance Governance (ETS-ZM-CW-PR01 Ver.1, Sections 2.1 – 2.3)
The protection of life and limb for the work force is paramount. The company
behavior ensures that individuals at all levels of the organization consider
safety as the overriding priority. This is manifested in decisions and actions
based on this priority. The work environment, and the policies and procedures
foster such a safety culture, and the attitudes and behaviors of individuals are
consistent with the policies and procedures.
Finding 2.5. The Plant Fails to Identify and Correct Hazards to Foot
and Vehicle Traffic.
The plant fails to identify and correct hazards to foot and vehicle traffic, a potential
violation of maintenance and operations standards. First, auditors found a pool of
standing water on the turbine deck. Second, an auditor nearly tripped on a piece of
grating in front of the West Boiler Circulating Water Pump. Third, in 2005, plant
personnel received minor injuries when their truck fell into a two-foot hole on one of the
plant’s dirt roads.
Auditors found a pool of standing water on the Unit 3 and 4 turbine deck. Wind
sometimes blows the exhaust steam condensate to this area of the deck, causing water to
accumulate on the ground. The plant staff said the water does not harm the equipment,
which is designed for outdoor use. CPUC believes that the wet ground poses a potential
danger, as people may slip and fall. Therefore, the area should be barricaded or coned off
to alert people and to prevent unnecessary entry into the area. (See Figure 1 below)
Erosion of dirt roads (for example, the one leading to the center well pump) poses
potential danger to those driving through in a car or truck. On February 23, 2005, a
company truck on that road fell into a 2-foot hole created by erosion, causing very minor
injuries to the people in the truck.
The plant took corrective steps and set procedures in place to address the hazards to foot
traffic. The plant placed non-skid surface all over the turbine deck (Figure 2 above),
including the area where water had accumulated at the time of the onsite audit visit. In
response to the loose grating piece found on a walkway, the plant manager now conducts
monthly housekeeping inspections. The plant also welded down several areas of loose
grating.
With regard to the erosion on a dirt road which caused an injury, the plant reemphasized
to the staff the importance of following plant vehicle procedures and the incident
reporting process. The plant called Southern California Edison (SCE), which owns the
road, to request repairs.
The protection of life and limb for the work force is paramount. The company
`behavior ensures that individuals at all levels of the organization consider
safety as the overriding priority. This is manifested in decisions and actions
based on this priority. The work environment, and the policies and procedures
foster such a safety culture, and the attitudes and behaviors of individuals are
consistent with the policies and procedures.
Work practice norms in the organization promote the safety culture through:
1. Appropriate defenses, such as technical accuracy,
precautions, cautions and notes, are explicitly embedded in
procedures, processes, and equipment configuration to
minimize the occurrences and consequences of inappropriate
actions.
Routine inspections by plant personnel ensure that all areas and critical
parameters of plant operations are continually monitored, equipment is
operating normally, and that routine maintenance is being performed. Results
of data collection and monitoring of parameters during routine inspections are
utilized to identify and resolve problems, to improve plant operations, and to
identify the need for maintenance. All personnel are trained in the routine
inspections procedures relevant to their responsibilities. Among other things,
the GAO creates, maintains, and implements routine inspections by:
Finding 2.6. The Plant Fails to Assure the Calibration of All Test
Equipment.
The plant fails to assure the calibration of test equipment, a violation of the plant’s
procedures, and a potential violation of maintenance standards. First, the plant lacks
computerized, recurring work orders for calibration of safety equipment. Second, plant
The plant lacks computerized, recurring work orders for calibration of safety equipment.
For example, electricians must remember when to calibrate the gas detection meters.
The CPSD auditor checked a gas detection meter and found a calibration certificate
attached. However CPSD is concerned that the plant’s informal procedure is inadequate,
because these meters are critical to safety and reliability, and because multiple
individuals use the meters.
Second, plant procedures fail to specify which non-safety equipment requires regular,
formal, and traceable calibration, and which does not. Instead, this decision is left to
individual staff judgment. The plant attaches calibration stickers to some pieces of
equipment (e.g. gloves and hotsticks) but not others. While the plant sends out some
equipment for regular calibration, the plant calibrates fluke meters and meggers as
needed. Electricians explained that they use fluke meters and meggers only for
"ballpark" measurements and therefore send these meters for calibration only when they
detect a problem. CPSD is concerned that over time, without clear written guidance, staff
practices could become inconsistent, introducing maintenance errors.
The plant contracted a company called TRANSCAT to track and calibrate all test
equipment, including multimeters. TRANSCAT notifies the plant electronically when
calibration is due, and calibrates the equipment on the plant site. The plant has access to
all calibration data through a website.
9
Reliant’s high-level policy, Performance Objectives for Mini-FPPAT, Guideline M.5.E, requires that
“Measurement and test equipment is calibrated and controlled to ensure accuracy and traceability,” but the
plant lacks detailed procedures to implement this policy.
Routine inspections by plant personnel ensure that all areas and critical
parameters of plant operations are continually monitored, equipment is
operating normally, and that routine maintenance is being performed. Results
of data collection and monitoring of parameters during routine inspections are
utilized to identify and resolve problems, to improve plant operations, and to
identify the need for maintenance. All personnel are trained in the routine
inspections procedures relevant to their responsibilities. Among other things,
the GAO creates, maintains, and implements routine inspections by:
Finding 2.7. The Plant Lacks Checklists for Some Daily Inspections
According to the plant manager, the plant fails to use written checklists for all daily unit
“walkdown” inspections, in conflict with its own performance objectives,10 and
potentially violating maintenance and operations standards. Although the plant maintains
detailed inspection procedures, some procedures lack checklists. While plant staff
demonstrated knowledge of equipment and operations,11 the lack of checklists may lead
staff to overlook details, or to fail to report equipment conditions to following shifts.
Without documentation, the plant cannot confirm that staff made required inspections, or
track the history of equipment problems. The plant manager stated that the plant plans to
adopt a system that utilizes Personal Digital Assistants (PDAs), to record some of the
inspection activities, particularly the chemical levels in various storage areas and
reservoirs.
10
Reliant “Performance Objectives for Mini-FPPAT, Guideline O.2.A” audit guidelines states:
“Check sheets or other comparable means are used to ensure that proper conditions are established for each mode of
plant status and for mode shifts.”
11
The plant manager and training officer explained that the plant trains operators over a three-year apprenticeship.
Auditors interviewed a yard operator who correctly delineated the entire list of yard operator, or “number 3 operator”
duties listed in the “Operators Routine Duties and Inspections” documents. The operator quoted meter readings and
chemical levels. Therefore, CPSD has no evidence that there exists a problem with operator knowledge or training on
the daily walkdown.
“To ensure safety, and optimize plant availability, the GAO conducts
operations systematically, professionally, and in accordance with approved
policies and procedures. The GAO takes responsibility for personnel actions,
assigns personnel to tasks for which they are trained, and requires personnel
to follow plant and operation procedures and instructions while taking
responsibility for safety. Among other things:
“The GAO assures that data, reports and other records reasonably necessary for
ensuring proper operation and monitoring of the generating asset are collected by
trained personnel and retained for at least five years, and longer if appropriate.”
Finding 2.8. The Plant Fails to Maintain Calibration Logs for the
Equipment that Monitors Feedwater Chemistry.
The plant fails to maintain calibration logs for equipment that monitors feedwater
chemistry, a potential violation of plant procedures and maintenance and operation
standards. The plant’s procedures12 require the plant’s chemical technician to develop
and maintain a calibration log on site. Further, the plant must calibrate testing equipment
such as the conductivity meter, pH meter, and spectrophotometer annually to ensure
12
Etiwanda Generating Station Chemistry Lab, Section 2.3, Calibration and Performance Checks
states that the conductivity meter, pH meter, and spectrophotometer must be calibrated annually and
that “an instrument calibration log will be developed and maintained by the Chemical Technician on
site.”
In response to the Preliminary Audit Report, the plant created equipment calibration logs.
The logs list the inspection cycle, calibration date, and calibration data for each piece of
equipment in the lab. In addition, during a December 12, 2007 plant visit, EGPB staff
saw stickers on equipment showing dates for the last calibration and next one due (see
Figure 3).
“The protection of life and limb for the work force is paramount. The company
behavior ensures that individuals at all levels of the organization consider
safety as the overriding priority. This is manifested in decisions and actions
based on this priority. The work environment, and the policies and procedures
foster such a safety culture, and the attitudes and behaviors of individuals are
consistent with the policies and procedures.”
“The protection of life and limb for the work force is paramount. GAOs have
a comprehensive safety program in place at each site. The company behavior
ensures that personnel at all levels of the organization consider safety as the
overriding priority. This is manifested in decisions and actions based on this
priority. The work environment and the policies and procedures foster such a
safety culture, and the attitudes and behaviors of personnel are consistent with
the policies and procedures.”
“To ensure safety, and optimize plant availability, the GAO conducts
operations systematically, professionally, and in accordance with approved
policies and procedures. The GAO takes responsibility for personnel actions,
assigns personnel to tasks for which they are trained, and requires personnel
to follow plant and operation procedures and instructions while taking
responsibility for safety. Among other things:
During a tour of the lab on June 5, the auditor saw boxes, chairs, and other items
apparently stored in the aisle behind the instrumentation panel (see Figure 4). These
items obstructed pedestrian traffic and prevented one from safely walking the length of
the aisle. The auditor also saw chemical/reagent boxes and storage bins housed in the
fume hood (see Figure 5).
The plant cleared the aisles of debris by June 7, 2006 but continued to use the fume hood
as a storage area.
During a December 12, 2007 plant visit, EGPB staff found the chemistry lab clean. The
plant removed storage material from the fume hood area, and posted a sign indicating no
storage in the fume hood (see Figure 6).
Figure 6. The plant cleaned and placed a warning sign on the fume hood.
The protection of life and limb for the work force is paramount. The company
behavior ensures that individuals at all levels of the organization consider
safety as the overriding priority. This is manifested in decisions and actions
based on this priority. The work environment, and the policies and procedures
foster such a safety culture, and the attitudes and behaviors of individuals are
consistent with the policies and procedures.
The protection of life and limb for the work force is paramount. The company
behavior ensures that individuals at all levels of the organization consider
safety as the overriding priority. This is manifested in decisions and actions
based on this priority. The work environment, and the policies and procedures
foster such a safety culture, and the attitudes and behaviors of individuals are
consistent with the policies and procedures.
Inspection reports of the plant’s fire control system refer to abandoned equipment as
“missing,” a potential violation of maintenance and operation standards. In particular,
the Hose Station report of May 4, 2006 states that stations #67 and #72 are “missing”.
SECTION 3. OBSERVATIONS
Observation 3.1. Reliant Regularly Audits its own Plants and Rates its
own Performance.
Reliant regularly selects one of its power plants for what Reliant calls a Fossil Plant
Performance Assessment Team (FPPAT) audit. Reliant sends staff from other Reliant
Performance Indices
The plant manager stated the plant corrected all discrepancies it found during the course
of the evaluation processes. For example, the plant scored zero out of five on
maintenance of oil storage tanks. The plant manager stated that the plant immediately
fixed leaking tanks. Auditors saw no evidence of tank leaks.
Maintenance procedures and documents are clear and technically accurate, provide
appropriate direction, and are used to support safe and reliable plant operation.
Procedures must be current to the actual methods being employed to accomplish the
task and are comprehensive to ensure reliable energy delivery to the transmission
grid.
Operation procedures exist for critical systems and states of those systems necessary
for the operation of the unit including startup, shutdown, normal operation, and
reasonably anticipated abnormal and emergency conditions. Operation procedures
and documents are clear and technically accurate, provide appropriate direction, and
are used to support safe and reliable plant operation. Procedures are current to the
actual methods being employed to accomplish the task and are comprehensive to
ensure reliable energy delivery to the transmission grid.
13
“Performance Objective Mini FPPAT”
The plant also maintains a “maintenance call out book,” which contains the names and
contact information for all the approved vendors and contractors for the plant.
Auditors reviewed procedures that touched upon items examined during the audit.
Auditors found the test procedure for turbine overspeed to be confusing; see Finding 2.1.
Additionally, Reliant awards team bonuses up to 5% of gross pay for meeting the
performance goals discussed in Observation 3.1. The plant calls this program the
“Annual Incentive Compensation Program”.
The CPSD auditor reviewed the plant organization chart. The chart contained three
repeat entries for the maintenance manager position: one entry listing the maintenance
manager’s name, and the other two entries omitting the name. The plant manager stated
that he was aware of this typographical error and would correct it.
Following the September 2004 and October 2004 problems, the plant performed in-house
root cause analysis and discovered that
In response, the plant analyzed the alarm system, eliminating nuisance alarms; retrained
operators; adopted typed logs in place of handwritten logs; and implemented a policy
requiring all contractors to use written quality standards.
In November 2004, the plant again detected excessive turbine vibration, along with high
temperature in bearing oil, which ignited. The plant asked General Electric (GE) to
thoroughly analyze the problem. CPSD auditors reviewed GE’s “Steam Turbine
Inspection Report”, which identified the main problem as grease on the turbine
midstandard (a supporting structure) which solidified when Reliant mothballed Units 3
and 4. The solidified grease impeded the turbine shaft, leading to high vibration and
damage to the thrust bearings. The GE report recommended two immediate corrective
The plant engineer stated that Reliant is currently in the process of implementing
software called REASON to perform root cause analysis. REASON is a product
trademark and not an acronym for any process.
Correct parts and materials in good condition, are available for maintenance
activities to support both forced and planned outages. Procurement of services and
materials for outages are performed in time to ensure materials will be available
without impact to the schedule. Storage of parts and materials support maintaining
quality and shelf life of parts and materials.
In the first section of the warehouse, devoted to relatively costly items, plant staff can
pick up parts only with requisitions, work orders, or planner approval on an emergency
basis. A dedicated worker runs the warehouse, and performs “receiving inspections” on
incoming parts. When workers take parts the dedicated warehouse worker records the
transaction on an “issue sheet,” which the plant later enters into the work management
system. The system bills parts and employee labor to the appropriate project
automatically. Generally, the system automatically reorders parts when inventory falls
below set points. If necessary, planners can order special parts.
The plant warehouse’s free stock area distributes small items such as fasteners and
electrical lugs and connectors. The plants contracts with Westco and Fastenol to
maintain electrical parts and mechanical parts, respectively.
Reliant employs a process known as “Partsco” for acquiring parts from warehouses that
the company shares between all power facilities. The CPSD auditor reviewed a
procedure for obtaining parts from the “Partsco” system.
Maintenance procedures and documents are clear and technically accurate, provide
appropriate direction, and are used to support safe and reliable plant operation.
Procedures must be current to the actual methods being employed to accomplish the
task and are comprehensive to ensure reliable energy delivery to the transmission
grid.
Station activities are effectively managed so plant status and configuration are
maintained to support reliable and efficient operation.
Figure 9. The plant cleaned and placed barricade tape around Units 1 and 2.
The maintenance program includes the proper balance of the various approaches to
maintenance, e.g., preventive, predictive, or corrective. The approach is adequately
documented with consideration of economics and reliability of equipment or
components, and their affect on reliable operation of the unit. Operating experience is
factored into the program. Maintenance procedures and documents should include the
generation equipment and all those components owned by the generation owner
directly connected to the plant that are an integral part of delivering power to the grid
including fuel supply systems, electrical
As a result of recent problems with the plant’s cooling towers, Etiwanda has improved its
management of contactor projects, through improved procedures and technical support.
In November 2004, Etiwanda hired a contractor to upgrade the cooling tower for Unit 4.
Without notifying Etiwanda, the contractor substituted a part not specified by plans. In
July of 2005, that cooling tower partially collapsed. Reliant adopted a corporate
procedure requiring the appointment of a project manager from Reliant’s Technical
Services office in Houston.
On July 21, 2005, the cooling tower partially collapsed. Because a drain was too small,
too much water collected in a header, whose supports were mounted in the wood of the
cooling tower. When workers removed the failed pipe header, they found extensive rot
and other structural damage to the wood. The combination of rot and excessive weight
caused the header and one of the decks of the cooling tower to fail.
Although Etiwanda does not believe that the contractor’s design change directly caused
the collapse, the plant recognizes that it needs to review the engineering and technical
calculations performed by its contractors. Etiwanda has assigned a liaison engineer to
oversee all contracted work. Further Etiwanda has asked the engineer to review all
contractor work since 2004.
Etiwanda is replacing much of the structure of the cooling tower, making extensive use of
engineering support from the plant, Reliant’s corporate office, and three independent
contractors. The plant’s engineer managed the first phase of the project, while an
engineer from Reliant’s corporate office is managing the second phase. The plant hired
three separate contractors to design the tower using computer modeling, build the tower,
and to inspect the quality of the construction.
Maintenance personnel are trained and qualified to possess and apply the
knowledge and skills needed to perform maintenance activities that support
safe and reliable plant operation.
On the last day of plant visit, auditors observed lockout/tag-out training as well as hands-
on firefighting training conducted by an instructor from Inner-Flame (Oxnard, CA), a
company specializing in fire safety training. Standing about 8 feet away, each trainee
used a fire extinguisher to fight a fire in a metal structure about the size of a small trailer.
Observing from about 30 feet way, auditors felt the intensity of the heat.
To ensure safe and continued operations, each GAO provides a prudent level of
security for the plant, its personnel, operating information and communications,
stepping up security measures when necessary.
Observation 3.12. The Plant Provides Security for the Plant and
Personnel.
Etiwanda provides security for the plant and personnel.
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
______________________
Work is performed on equipment only when safe. When necessary, equipment is taken
out of service, de-energized, controlled, and tagged in accordance with a clearance
procedure. Personnel are trained in the clearance procedure and its use, and always
verify that equipment is safe before any work proceeds. Among other things:
The GAO prepares and maintains a clearance procedure. The clearance
procedure contains requirements for removing a component from service and/or
placing a component back into service.
The GAO ensures that personnel are trained in and follow the clearance
procedure.
On the final day of the plant visit, the plant trained plant staff in lockout/tagout
procedures.
The protection of life and limb for the work force is paramount. The company
behavior ensures that individuals at all levels of the organization consider
safety as the overriding priority. This is manifested in decisions and actions
based on this priority. The work environment, and the policies and procedures
foster such a safety culture, and the attitudes and behaviors of individuals are
consistent with the policies and procedures.
The protection of life and limb for the work force is paramount. The company
behavior ensures that individuals at all levels of the organization consider
safety as the overriding priority. This is manifested in decisions and actions
based on this priority. The work environment, and the policies and procedures
foster such a safety culture, and the attitudes and behaviors of individuals are
consistent with the policies and procedures.
First, during the drill, the control operator started the emergency siren and ran it for one
minute. Then the operator used the public address system repeatedly to announce, “This
is an evacuation drill. Go to the #1 Evacuation Assembly Area in the Administration
Parking Lot.” Then the operator ran the siren again for one minute. People could not
hear the siren (or apparently, public announcements) from the storeroom, demineralized
water tanks, the guard shack, the makeup reservoir, the shop for Instrumentation &
Control (I & C) employees, or the electrician’s shop. The plant’s safety coordinator told
the auditor that the plant now uses radios to communicate with people in areas that the
siren and the public announcement cannot reach.
Secondly, the guard had trouble printing out a list of people on site in part because his
computer lacked an icon that could easily access the log file. Since then, the plant has
added such an icon. The guard did keep a handwritten log of contractors and guests.
Lastly, a review of the log showed that the guard had recorded incorrect information on
the day of the drill. The entry and exit time for a FedEx deliveryman was assigned to a
UPS deliveryman, and vice versa. Plant personnel pointed out the mistake. Plant
management counseled the security guard about the importance of accuracy.
“The protection of life and limb for the work force is paramount. GAOs have
a comprehensive safety program in place at each site. The company behavior
ensures that personnel at all levels of the organization consider safety as the
overriding priority. This is manifested in decisions and actions based on this
priority. The work environment and the policies and procedures foster such a
safety culture, and the attitudes and behaviors of personnel are consistent with
the policies and procedures.”
“To ensure safety, and optimize plant availability, the GAO conducts
operations systematically, professionally, and in accordance with approved
policies and procedures. The GAO takes responsibility for personnel actions,
assigns personnel to tasks for which they are trained, and requires personnel
to follow plant and operation procedures and instructions while taking
responsibility for safety. Among other things:
The auditor interviewed the chemical technician and observed several tests performed
in the lab including pH, hydrazine, and O2 testing from the circulating and feed water
bench, plus a standardization test for pH level 7 and 10. Except as noted in Finding
2.3 and 2.9, the plant follows written procedures.
Each generating facility shall maintain a Control Operator Log that contains
the chronological history of the facility including detailed entries regarding
the operations and maintenance of the facility. Where information is unit
specific, information for each unit must be recorded and so identified.
The Control Operator Log is a formal record of real time operating events as
well as the overall status of the generating units and auxiliary equipment
under the purview of the Control Room Operator. The log shall also contain
an accurate and concise record of important and/or unusual events involving
operations, maintenance, water chemistry, safety, accidents affecting
personnel, fires, contractor activities, environmental matters, and any other
pertinent information concerning the operation of the facility. The log shall
also record communications between the facility and outside entities including
but not limited to the Independent System Operator (ISO), scheduling
coordinators or headquarters facilities, regulators, environmental agencies,
CalOSHA or similar agencies. The log shall be maintained notwithstanding
and in addition to any other similar requirements that mandate that events be
recorded. The generator must collect and record all information specified in
these standards. All such information must be readily available to operators,
California Public Utilities Commission staff, and other authorized personnel at
all times.
Except for Generating Assets for which NERC does not accept data, each
Generating Asset Owner shall submit generator design, performance, and
event data to NERC for inclusion in GADS. Within the categories of data that
NERC accepts, CPSD may specify the data the Generating Asset Owner must
submit to NERC. If requested by CPSD, a Generating Asset Owner shall
concurrently provide CPSD with a copy of all data submitted to NERC for
inclusion in GADS.
The following table shows the dates that were chosen, the units involved, and the
corresponding GADS data, logbook entry data, and SLIC. Note that RS in GADS data is
an abbreviation for Reserve Shutdown.