Boiler Reliabiliti Optimization 2001

EPRI Boiler Reliability Optimization Program
Case Studies from 1998-2001
M AT E
R I A L
N
LICE
SED
WARNING: Please read the License Agreement on the back cover before removing the Wrapping Material.
Technical Report
EPRI Boiler Reliability Optimization Program

Case Studies from 1998-2001
1006537
Final Report, December 2001
EPRI Project Manager P. Abbott
EPRI 3412 Hillview Avenue, Palo Alto, California 94304 PO Box 10412, Palo Alto, California 94303 USA 800.313.3774 650.855.2121 askepri@epri.com www.epri.com
DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIES

THIS DOCUMENT WAS PREPARED BY THE ORGANIZATION(S) NAMED BELOW AS AN ACCOUNT OF WORK SPONSORED OR COSPONSORED BY THE ELECTRIC POWER RESEARCH INSTITUTE, INC. (EPRI). NEITHER EPRI, ANY MEMBER OF EPRI, ANY COSPONSOR, THE ORGANIZATION(S) BELOW, NOR ANY PERSON ACTING ON BEHALF OF ANY OF THEM: (A) MAKES ANY WARRANTY OR REPRESENTATION WHATSOEVER, EXPRESS OR IMPLIED, (I) WITH RESPECT TO THE USE OF ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM DISCLOSED IN THIS DOCUMENT, INCLUDING MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, OR (II) THAT SUCH USE DOES NOT INFRINGE ON OR INTERFERE WITH PRIVATELY OWNED RIGHTS, INCLUDING ANY PARTY'S INTELLECTUAL PROPERTY, OR (III) THAT THIS DOCUMENT IS SUITABLE TO ANY PARTICULAR USER'S CIRCUMSTANCE; OR (B) ASSUMES RESPONSIBILITY FOR ANY DAMAGES OR OTHER LIABILITY WHATSOEVER (INCLUDING ANY CONSEQUENTIAL DAMAGES, EVEN IF EPRI OR ANY EPRI REPRESENTATIVE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES) RESULTING FROM YOUR SELECTION OR USE OF THIS DOCUMENT OR ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM DISCLOSED IN THIS DOCUMENT. ORGANIZATION(S) THAT PREPARED THIS DOCUMENT Eskom
ORDERING INFORMATION
Requests for copies of this report should be directed to EPRI Customer Fulfillment, 1355 Willow Way, Suite 278, Concord, CA 94520, (800) 313-3774, press 2. Electric Power Research Institute and EPRI are registered service marks of the Electric Power Research Institute, Inc. EPRI. ELECTRIFY THE WORLD is a service mark of the Electric Power Research Institute, Inc. Copyright 2001 Electric Power Research Institute, Inc. All rights reserved.
CITATIONS
This report was prepared by Eskom Megawatt Park P.O. Box 1091 Johannesburg, 2000 South Africa Principal Investigator D. McGhee This report describes research sponsored by EPRI. The report is a corporate document that should be cited in the literature in the following manner: Boiler Reliability Optimization Project Case Studies from 1998 2001, EPRI, Palo Alto, CA: 2001. 1006537.
iii
REPORT SUMMARY
The prospect of competing in a deregulated environment is driving utilities to explore more costeffective means of operating and maintaining their plants. Recognizing this, a number of companies including American Electric Power, Great River Energy, Detroit Edison, Arizona Public Services Company, and Hawaiian Electric Company have participated in EPRIs Boiler Reliability Optimization Program. This report provides seven case studies demonstrating key aspects of the program. Background EPRIs Boiler Reliability Optimization Program consists of several customized projects. Each project includes four phases that involve EPRI technical staff, with participation from the plants operating, maintenance, and engineering support staff, as follows: Phase 1technical and organizational assessments, Phase 2development of a boiler failure defense plan, Phase 3implementation of a failure defense plan, and Phase 4continuous improvement and process automation. Each phase, in turn, builds on the strengths of the preceding phase. EPRI compiled these case studies to demonstrate the scope of the program and its applicability to a wide range of power production issues. Objective To provide case studies from membership participation in EPRIs Boiler Reliability Optimization Program between 1998 and 2001. Approach The principal investigators on this report selected case studies that would demonstrate key facets of the program, including development of a maintenance strategy, implementation of Streamlined Reliability Centered Maintenance (RCM) procedures, outage task prioritization, Boiler Predictive Maintenance (BPdM), performance of root cause analysis, and application of EPRIs Boiler Maintenance Workstation (BMW) software. Each case study includes a scope of work description, findings and observations, conclusions, and recommendations. Results Two case studies from American Electric PowerBig Sandy Plant-Unit 2 demonstrate phases 1, 2, and 4 of the program, with specific application to water- and steam-touched tubing. This project focused on the long-term integrity of the water- and steam-touched tubing within the boiler envelope. It involved training on EPRIs Streamlined RCM software using the fuel system as an example, additional training on EPRIs tube failure reduction program, development of a
BPdM process, and the provision of EPRIs BMW software package for trending and tracking tube failure data and information. An outage task prioritization modelRisk Evaluation and Prioritization (REAP)was used to prioritize the tasks identified for the Unit 2 scheduled outage in April 2000. A case study from Great River Energys Coal Creek Station-Unit 2 discusses the decision to change the boiler maintenance overhaul frequency from two to three years. The project emphasizes development of a comprehensive and effective boiler inspection plan to take the place of the currently reactive plan, which focuses almost entirely on sootblower erosion and does not cover all boiler components in sufficient detail. A case study from Detroit Edisons St Clair Plant-Unit 7 focuses on technical and programmatic opportunities for improving boiler reliability. The project identifies the causes of boiler tube failures and highlights a number of opportunities for improvement with respect to managing the short- and long-term health of the boiler. Two case studies at Arizona Public Services Companys Four Corners and Cholla plants emphasize root cause analysis. A review of the 1999 Lost Generation Action Plans from both plants evaluates whether the true root causes of lost generation have been identified, and whether appropriate action plans are in place to address these causes. A case study from Hawaiian Electric Company (HECO) addresses EPRIs independent review of HECOs Remaining Useful Life and Generation Asset Management Program. The EPRI team identified a number of opportunities where current project management and root cause analysis efforts could be enhanced to provide additional assurance that availability and reliability goals would be achieved. EPRI Perspective A Boiler Reliability Optimization Program is only as effective as the people, management systems, component manufacturers, technology specialists, and predictive maintenance tools or equipment that comprise it. For a condition-based, highly planned, maintenance optimization program to be effective, it must integrate the work process, management, work culture, technologies, and people into the total plant operation. EPRI believes this documentation of case studies demonstrates innovative applications of EPRIs Boiler Reliability Optimization Program and methods for achieving such integration under a wide variety of circumstances. Utilities will find that implementation of the Boiler Reliability Optimization Program will help them compete cost-effectively in the deregulated power generation market. Keywords Boiler reliability Boiler Reliability Optimization Program Competitive power generation
vi
EPRI Licensed Material
CONTENTS
1 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1, 2 & 4 AT AMERICAN ELECTRIC POWERS BIG SANDY PLANT UNIT 2............................................................ 1-1 1.1 1.2 1.3 1.4 EXECUTIVE SUMMARY........................................................................................... 1-1 INTRODUCTION ...................................................................................................... 1-2 SCOPE OF WORK ................................................................................................... 1-3 FINDINGS AND OBSERVATIONS............................................................................ 1-3 1.4.1.1 Work Process................................................................................................ 1-5 Maintenance Strategy........................................................................................... 1-7 Findings:............................................................................................................... 1-8 Recommendations:............................................................................................... 1-8 Work Identification ................................................................................................ 1-8 Findings:............................................................................................................... 1-9 Recommendations................................................................................................ 1-9 Work Control .......................................................................................................1-10 Findings...............................................................................................................1-10 Recommendations...............................................................................................1-10 Work Execution ...................................................................................................1-11 Findings...............................................................................................................1-11 Recommendations...............................................................................................1-11 1.4.1.2 Technologies................................................................................................1-11 Findings...............................................................................................................1-12 Recommendation ................................................................................................1-12 Work Management System .................................................................................1-12 Findings...............................................................................................................1-12 Recommendations...............................................................................................1-12 Maintenance & Diagnostic Technologies .............................................................1-12 Findings...............................................................................................................1-12
1.4.1 Phase I and II - Technical and Organisational Assessment.................................. 1-3
vii
Recommendations...............................................................................................1-12 Information Integration.........................................................................................1-13 Findings...............................................................................................................1-13 Recommendations...............................................................................................1-13 1.4.1.3 Management and Work Culture....................................................................1-13 Recommendations...............................................................................................1-14 1.4.1.4 People..........................................................................................................1-15 Findings:..............................................................................................................1-15 Recommendations...............................................................................................1-15 1.4.2 Failure Defence Plan Water and Steam Touched Tubing..................................1-15 Findings ...................................................................................................................1-16 Economiser .........................................................................................................1-16 Waterwalls...........................................................................................................1-16 Platen Superheat.................................................................................................1-17 Finishing Superheat.............................................................................................1-17 First Reheat (High Pressure) ...............................................................................1-17 Second reheat (low pressure)..............................................................................1-18 Spray Attemperators............................................................................................1-18 Recommendations...............................................................................................1-18 1.4.3 Streamlined Reliability-Centered Maintenance.....................................................1-30 1.4.4 Unit 2 Outage Task Prioritisation..........................................................................1-30 1.4.5 Boiler Tube Failure Reduction and Cycle Chemistry Improvement Training .........1-33 1.4.6 Boiler Maintenance Workstation (BMW) installation and training..........................1-33 AEP Big Sandy Unit 2, Boiler Reliability Optimization - Phase 4, July 2001.......................1-34 Executive Summary ..........................................................................................................1-34 Introduction .......................................................................................................................1-34 Description of Phase 4 of a Typical Boiler Reliability Optimization Project.........................1-35 Findings ........................................................................................................................1-35 Conclusions.......................................................................................................................1-40 Recommendations ............................................................................................................1-41 Appendix 1: Interview List..................................................................................................1-44 Appendix 2: List of Typical headings that appear in a policy and or a procedure ...............1-45
viii
2 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 2 BOILER INSPECTION PLAN REVIEW AT GREAT RIVER ENERGYS COAL CREEK STATION UNIT 2 ................................................................................................................................. 2-1 2.1 Executive Summary..................................................................................................... 2-1 2.2 Introduction.................................................................................................................. 2-1 2.3 Findings and Observations .......................................................................................... 2-2 2.4 Recommendations....................................................................................................... 2-3 ANNEXURE I: Extract from a Boiler Inspection Report....................................................... 2-7 Inspection Areas ....................................................................................................... 2-7 1200 Waterwall Cleaning Devices, operational condition ............................................... 2-7 Inspection results........................................................................................................... 2-7 Recommendations for 2004: .......................................................................................... 2-8 1501 Superheat Division Panels, attachment problems ................................................ 2-8 Inspection results:.......................................................................................................... 2-8 Recommendations for 2004: .......................................................................................... 2-8 ANNEXURE II: Unit 2 Outage Inspection Plan as of August 21, 2000 ................................ 2-9 1. Burner Front Team Member...................................................................................... 2-9 1.1 Prior to the outage (within 2 weeks of the start of the outage) ............................. 2-9 1.2 At the start of the outage (During the installation of the boiler scaffolding)..........2-10 1.3 Burner Front inspection (Upon completion of the boiler scaffolding) ...................2-10 2. Waterwall Team Member .........................................................................................2-12 2.1 Prior to the outage (within 2 weeks of start of the outage) ..................................2-12 2.2 Prior to the installation of the boiler scaffolding (after the throat scaffolding is installed) ..................................................................................................................2-12 2.3 Upon completion of the boiler scaffold................................................................2-12 2.4 Upon completion of the bottom ash scaffolding ..................................................2-13 3. Superheat and Reheat Team Member .....................................................................2-14 3.1 Prior to the outage (within 2 weeks of start of the outage) ..................................2-14 3.2 During the outage...............................................................................................2-14 4. Back-pass Team Member ........................................................................................2-16 4.1 Prior to the outage (within 2 weeks of start of the outage) ..................................2-16 4.2 During the outage...............................................................................................2-16 5. External Boiler Team Member .................................................................................2-18 5.1 Prior to the outage (within 2 weeks of the start of the outage) ............................2-18 5.2 During the outage...............................................................................................2-18
ix
3 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1 AT DETROIT EDISONS ST CLAIR PLANT UNIT 7 AMERICAN ELECTRIC POWERS BIG SANDY PLANT UNIT 2 ....................................................................................................... 3-1 Executive Summary ........................................................................................................... 3-1 3.1 Introduction.................................................................................................................. 3-3 3.2 Findings and Observations .......................................................................................... 3-3 3.2.1 Programmatic and Data Sources Assessment ..................................................... 3-3 3.2.1.1 Maintenance Strategy................................................................................... 3-6 3.2.1.1.1 Findings ................................................................................................ 3-6 3.2.1.1.2 Recommendations ................................................................................ 3-7 3.2.1.2 Work Process............................................................................................... 3-7 3.2.1.2.1 Findings ................................................................................................ 3-8 3.2.1.2.2 Recommendations ................................................................................ 3-8 3.2.1.3 Predictive Maintenance Program and Technology.......................................3-10 3.2.1.3.1 Findings ...............................................................................................3-10 3.2.1.3.2 Recommendations ...............................................................................3-11 3.2.1.4 People.........................................................................................................3-11 3.2.1.4.1 Findings ...............................................................................................3-11 3.2.1.4.2 Recommendations ...............................................................................3-12 3.2.2 Failures and Root Causes...................................................................................3-12 3.2.2.1 Boiler Tubes ................................................................................................3-12 3.2.2.1.1 Failure Mechanisms, Actions taken and Recommendations ................3-14 3.2.2.2 Boiler Headers.............................................................................................3-18 3.2.2.2.1 Boiler Test and Inspection Plan Guideline............................................3-19 Purpose...............................................................................................................3-19 Definitions............................................................................................................3-19 Scope ..................................................................................................................3-19 Inspection and Test Plan Development ...............................................................3-20 3.3 Conclusions................................................................................................................3-21 3.4 Recommendations......................................................................................................3-21 Appendix 1: List of interviewees ........................................................................................3-23 Appendix 2: Example of a Typical Equipment Condition and Technology Matrix ...............3-24 Appendix 3: Remnant Life Assessment Flow Chart for Thick Section................................3-25
4 BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1- ROOT CAUSE ANALYSIS AT ARIZONA PUBLIC SERVICES COMPANYS FOUR CORNERS AND CHOLLA PLANTS.................................................................................................................. 4-1 FOUR CORNERS PLANT .................................................................................................. 4-1 4.1 Executive Summary...................................................................................................... 4-1 4.2 Introduction................................................................................................................... 4-2 4.3 Plant Description .......................................................................................................... 4-3 4.4 Incidents investigated ................................................................................................... 4-3 4.4.1 Unit 5 #4 Turbine Control Valve chatter - 197,471 Mwh lost................................ 4-4 Incident Description................................................................................................... 4-4 Root Cause Analysis Chart ....................................................................................... 4-4 4.4.2 Unit 3 Scrubber unit failure - 33,446 Mwh lost...................................................... 4-5 Incident Description................................................................................................... 4-5 Root Cause Analysis Charts...................................................................................... 4-6 4.4.3 Unit 4 Loss of Booster Fans 38,413 Mwh.......................................................... 4-7 Incident Description................................................................................................... 4-7 Root Cause Analysis Chart ....................................................................................... 4-7 4.4.4 Unit 3 Pulverizer Pinion & Bull Gear Failure - 34,881 MW Hrs. ............................ 4-7 Incident Description................................................................................................... 4-7 Root Cause Analysis Chart ....................................................................................... 4-8 4.4.5 Unit 3 Back-pass Flyash Erosion 39,378 Mwh ................................................... 4-8 Problem Description.................................................................................................. 4-8 Root Cause Analysis Chart ....................................................................................... 4-9 4.4.6 Unit 4 Air Pre-heater Failure 166,744 Mwh. ....................................................... 4-9 Problem Description.................................................................................................. 4-9 Root Cause Analysis Chart ....................................................................................... 4-9 4.4.7 Units 4 Flyash Erosion 39,288 MW/Hrs.............................................................. 4-9 Problem Description.................................................................................................. 4-9 Root Cause Analysis Chart ......................................................................................4-10 4.4.8 Unit 3 Turbine H.P. Nozzle Pluggage 100,628 Mwh..........................................4-10 Problem Description.................................................................................................4-10 Root Cause Analysis Chart ......................................................................................4-10 4.4.9 Unit 4 LP Exciter Field Failure 45,229 Mwh........................................................4-11 Incident Description..................................................................................................4-11 Root Cause Analysis................................................................................................4-11
xi
4.4.10 Unit 4 HP Exciter Field Failure 33,500 Mwh .....................................................4-11 Problem Description.................................................................................................4-11 Root Cause Analysis................................................................................................4-11 4.5 Findings ...................................................................................................................4-12 4.6 Conclusions.................................................................................................................4-12 4.7 Recommendations.......................................................................................................4-13 4.8 Appendix 1: Root Cause Analysis Charts ...................................................................4-13 CHOLLA PLANT ...............................................................................................................4-26 4.9 Executive Summary.....................................................................................................4-26 4.10 Introduction................................................................................................................4-27 4.11 Plant Description .......................................................................................................4-28 4.12 Incidents Investigated................................................................................................4-28 4.12.1 Mechanical Dust Collector Retrofit 41,146 Mwh. ..............................................4-29 Problem Description.................................................................................................4-29 Root Cause Analysis Chart ......................................................................................4-29 4.12.2 Unit One ID Booster Fan Bearing Failures 7,544 Mwh. ....................................4-30 Incident Description..................................................................................................4-30 Root cause Analysis Chart .......................................................................................4-30 4.12.3 Unit one Turbine Thrust Trips 9,971 Mwh. .......................................................4-31 Incident Description..................................................................................................4-31 Root Cause Analysis Chart ......................................................................................4-31 4.12.4 Unit Two A Booster ID Fan Motor Failure 30,464 Mwh. ................................4-32 Incident Description..................................................................................................4-32 Root Cause Analysis Chart ......................................................................................4-32 4.12.5 Unit Four Air Pre-heater Pluggage 48,740 Mwh. ..............................................4-33 Problem Description.................................................................................................4-33 Root Cause Analysis Chart ......................................................................................4-34 4.12.6 Unit Four Scrubber Problems 76,917 Mwh.......................................................4-34 Problem Description.................................................................................................4-34 Root Cause Analysis Chart ......................................................................................4-34 4.12.7 Unit Four High Opacity Shutdown 42,731 ........................................................4-35 Problem Description.................................................................................................4-35 Root Cause Analysis Chart ......................................................................................4-35 4.12.8 Unit Two Waterwall Tubes Hydrogen Damage 188,944 Mwh...........................4-35
xii
Problem Description.................................................................................................4-36 Root Cause Analysis Chart ......................................................................................4-36 4.12.9 Unit Two Cable Tray Fire 28,899 Mwh. .............................................................4-36 Problem Description.................................................................................................4-36 Root Cause Analysis Chart ......................................................................................4-37 4.12.10 Unit Four Air Pre-heater Guide Bearing Fire 6,099 Mwh. ...............................4-38 Problem Description.................................................................................................4-38 Root cause Analysis Chart .......................................................................................4-38 4.13 4.14 4.15 Findings...............................................................................................................4-39 Conclusions.........................................................................................................4-40 Recommendations...............................................................................................4-40
4.16 Appendix 2 ...............................................................................................................4-40 Root Cause Analysis Charts (Excel file)........................................................................4-40 5 BOILER RELIABILITY OPTIMIZATION PROJECT PROGRAM AUDIT AT HAWAIIAN ELECTRIC COMPANY........................................................................................ 5-1 5.1 EXECUTIVE SUMMARY ............................................................................................. 5-1 5.2 Introduction.................................................................................................................. 5-1 5.3 Findings....................................................................................................................... 5-2 5.3.1 Boiler Reliability Optimization Project Plan........................................................... 5-2 5.3.2 Project Activities................................................................................................... 5-2 5.3.2.1 BTFR/CCI Program ...................................................................................... 5-2 5.3.2.2 Streamlined Reliability Centered Maintenance ............................................. 5-3 5.3.2.3 Root Cause Analysis .................................................................................... 5-3 5.3.3 Condition based data collection ........................................................................... 5-4 5.4 Conclusions................................................................................................................. 5-4 5.5 RECOMMENDATIONS ............................................................................................... 5-4 Appendix 1: Boiler Reliability Optimization Storage Plan .................................................... 5-7 Appendix 2 ........................................................................................................................5-10 Appendix 3 ........................................................................................................................5-11 List of Typical Headings that Appear in a Policy or a Procedure ...................................5-11
xiii
LIST OF FIGURES
Figure 1-1 Maintenance Optimization Program ....................................................................... 1-4 Figure 1-2 Big Sandy benchmarked against Industry Best Practice ........................................ 1-5 Figure 1-3 Work Process ........................................................................................................ 1-6 Figure 1-4 Work Process Model.............................................................................................. 1-7 Figure 1-5 Task Scatter Diagram - Value to Cost ...................................................................1-31 Figure 1-6 Accumulated Value verse Cost .............................................................................1-31 Figure 3-1 Spider Chart........................................................................................................... 3-4 Figure 3-2 ..............................................................................................................................3-13 Figure 4-1 Unit 5, #4 Control Valve 197,471 Mwh ..................................................................4-14 Figure 4-2 Unit 3 Scrubber Problems 33,446MW/Hrs.............................................................4-15 Figure 4-3 Unit 3, Scrubber Problems 33,446MW/Hrs............................................................4-16 Figure 4-4 Unit 3 Scrubber Recycle Pump Problems 33,446MW/Hrs.....................................4-17 Figure 4-5 Unit 4, Loss of Booster Fans, 38, 413 MW/Hrs .....................................................4-18 Figure 4-6 Unit 3 Pulverizer Pinion and Bull Gear Failures, 38,881 MW/Hrs. .........................4-19 Figure 4-7 Unit 3 Backpass Flyash Erosion 39,378 MW/Hrs. .................................................4-20 Figure 4-8 Unit 4 Air Preheater Failure 166,744 MW/Hrs. ......................................................4-21 Figure 4-9 Units 4 and 5 Backpass Flyash Erosion ................................................................4-22 Figure 4-10 Unit 3 HP Turbine Nozzle Plugging, 100,628 MW/Hrs.........................................4-23 Figure 4-11 Unit One, Mechanical Dust Collection Retrofit, 41,146 Mwhs..............................4-41 Figure 4-12 Unit One, ID Booster Fan Bearing Failures, 7,544 Mwhs ....................................4-42 Figure 4-13 Unit One Turbine Thrust Trips, 9,971 Mwhs........................................................4-43 Figure 4-14 Unit Two, A Booster ID Fan Motor Failure, 30,464 Mwhs....................................4-44 Figure 4-15 Unit Four, Air Preheater Pluggage, 48,740 Mwh .................................................4-45 Figure 4-16 Unit Four Scrubber Outage, 76,917 Mwh ............................................................4-46 Figure 4-17 Unit Four, High Capacity Shutdown, 42,731 Mwh ...............................................4-47 Figure 4-18 Unit Two Waterwall Tubes Hydrogen Damage, 188,944 Mwh.............................4-48 Figure 4-19 Unit Two Cable Tray Fire, 28,899 Mwh ...............................................................4-49 Figure 4-20 Unit Two Cable Tray Fire March 1999, 28,899 Mwh............................................4-50 Figure 4-21 Unit Four Air Preheater Guide Bearing Fire, 6,099 Mwh .....................................4-51
xv
LIST OF TABLES
Table 1-1 Component Description and Engineering Identification Number.............................1-18 Table 1-2 Phase 2 Recommendations and Actions Taken .....................................................1-36 Table 3-1 Boiler tube Failures, Causes and Recommended Action........................................3-15 Table 3-2 Typical Header and Steam Drum Problem Areas ...................................................3-18 Table 4-1 Cholla - Megawatt-hour loss summary report-fourth quarter 1999..........................4-28
xvii
1
BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1, 2 & 4 AT AMERICAN ELECTRIC POWERS BIG SANDY PLANT UNIT 2
1.1
EXECUTIVE SUMMARY
Preparing for competition American Electric Power (AEP) requested EPRI to undertake a Boiler Reliability Optimisation project at their Big Sandy Plant. An availability of 95 % during peak periods May through August and 90 % for the remainder of the year was set as a corporate target. Meeting this target would give Senior Management the assurance that they could compete successfully in the power generation market. The project focussed on the long-term integrity of the water and steam touched tubing within the boiler envelope, training on EPRIs tube failure reduction program and the provision of a EPRIs Boiler Maintenance Workstation (BMW) software package for trending and tracking tube failure data and information. The project scope also included training on EPRIs Streamlined Reliability Centered Maintenance software using the fuel system as an example. An outage task prioritisation model Risk Evaluation and Prioritisation (REAP) was used to prioritise the tasks identified for the Unit 2 scheduled outage in April 2000. This project, Phase II of EPRIs Boiler Reliability Optimisation project, commenced in March 2000 and was finalised with the installation of the BMW software and user training in July 2000. Phase I was completed with the issuing of the final report in January 1999. Between March and July 2000 a number of visits to the plant were made to interview and collect data for analysis. This report focuses on the development of a failure defense plan for the water and steam touched boiler tubing and contains a number of recommendations. The main recommendation are listed below: 1. In the Technical and Organisational Assessment section there are a number of specific recommendation. Collectively these are related to improving planning and scheduling of boiler maintenance work. Thus, a high level recommendation would be to align the boiler maintenance strategy i.e. frequency and duration of scheduled outages and inspection scopes of work to the overall station performance goals of 95% availability during peak periods. Specific boiler availability targets should be developed relating to the number of tube leaks per year. One tube leak per unit per year, no repeat failures, no failures because of sootblower erosion are considered reasonable targets. The strategy and performance targets
1-1
EPRI Licensed Material Boiler Reliability Optimization Project Phase 1, 2 & 4 at American Electric Powers Big Sandy Plant Unit 2
should be compiled into separate plant directives, which are reviewed annually for effectiveness and appropriateness. 2. To repair and calibrate all boiler instrumentation critical to the efficient and effective operation of the boiler. This includes air, gas, water and steam temperatures and pressures and reheater and superheater metal temperatures. 3. The installation of on-line water and steam chemistry instrumentation and the establishment chemical performance index. This index can the be used in a proactive way to ensure compliance to EPRI water and steam chemistry guidelines. 4. To prepare detailed inspection plans based on the tasks identified in the Boiler Failure Defence Plans presented in this report. These inspection plans should be loaded into the CMMS. This would facilitate the development of a scheduled and or forced outage plan at short notice. 5. Once the installation and the user training on the Boiler Maintenance Workstation (BMW) have been completed, the BMW database needs to be updated with all the history of previous tube failures. This will enable tube leaks and repairs to be tracked and trends to be established. This will also assist in formulating focussed boiler inspections plans. The plant has knowledgeable and skilful people. Given time, management support and coaching to implement the recommendations highlighted in this report the plant team will achieve the goals identified by Senior Management.
1.2
INTRODUCTION
During the period 1997 to 1999 AEP Big Sandy Plants availability loss was 7.9 % of which 4.2 % can be attributed to boiler tube failures. The causes of these failures varied however the dominate repeat failures were due to fly ash erosion, oxygen pitting, long term over heating and firing side corrosion. During 1999, in excess of $2.0 million dollars of lost revenue can be attributed to outages on Unit 2. Concerned about this loss and a possible future increase in the Equivalent Forced Outage Rate (EFOR) lead senior management at Big Sandy Plant to request assistance from EPRI. Participation in EPRIs Boiler Reliability Optimization program was proposed. The program consists of several customised Boiler Reliability Optimisation Projects. Each project consists of four phases that involve EPRI technical staff, with participation from the plants operating, maintenance and engineering support staff. Each phase builds on the strengths of the preceding phase. These phases are: Phase I - Technical and Organisational Assessments Phase II - Development of a Boiler failure defence plan Phase III - Implementation of a failure defence plan Phase IV - Continuous Improvement and Process automation
1-2
AEP Big Sandy Plants management chose to limit their participation to elements of Phase I. This was completed in January 1999. Satisfied with the results, AEP Big Sandy management requested EPRI to continue with the program, i.e. complete Phase I Technical and Organisational Assessment, and implement a tailored Phase II. Phase II was limited to only the water- and steam- touched boiler tubing. This report discuss the findings of this limited scope project.
1.3
SCOPE OF WORK
This consisted of the following activities: 1.1 Complete Phase I Technical and Organisational Assessment. 1.2 Develop a Failure Defence Plan for water and steam touched tubing. 1.3 Provide Streamlined Reliability Centered Maintenance (SRCM) software and training, and analyse two systems. 1.4 Evaluate and prioritise all outage tasks on an upcoming outage, using EPRIs Risk Evaluation and Prioritise Program. 1.5 Provide Boiler Tube Failure Reduction and Cycle Chemistry Improvement training. 1.6 Supply, install and provide training on EPRIs Boiler Maintenance Workstation (BMW) software for tracking boiler tube failure information.
1.4
FINDINGS AND OBSERVATIONS
1.4.1 Phase I and II - Technical and Organisational Assessment A maintenance process assessment was conducted as a part of the work performed in the Boiler Reliability Optimization Project. The purpose is to understand the current processes, technologies, organizational strengths and weaknesses, and plant people to design a manageable change within the implementation plan. Data was collected and evaluated and benchmarked against best practices identified and tailored by EPRI. The assessment team spent one week on site interviewing a cross section of plant staff to determine how boiler maintenance work is identified and performed. The team focused on four main topics namely Work Process, Work Culture/Management, Technology and People. The data and information obtained was compared to industry best practice as identified in the EPRI Best Practice, to identify opportunities for improvement. This section of the report presents the findings and recommendations to align AEP - Big Sandy Plants work process with Best Practice i.e. to move boiler maintenance from a reactive mode to one that is planned. The benefits of moving towards a more planned approach will be seen in reduced O&M cost and improving boiler reliability. 1-3
With this information and available resources AEP - Big Sandy Plant can become the Best Practice Plant within the AEP organisation for which it has been striving to become. Given time, management commitment and coaching, the plant team has the vision and can successfully lead this plant into full implementation of the boiler reliability optimization processes. From the data and interviews, strengths and weaknesses were plotted on a spider chart and recommendations formulated For a condition-based, highly planned, maintenance optimization program to be effective, it must integrate the Work Process, Management and Work Culture, Technologies, and the People into the total operation of the pant. Each category is equally important to the success of the program, as illustrated in Figure 1-1.
P LE
O R
P E O
E C S S LO G Y N O
Work culture/Management Setting Goals Organisation Communications Global Metrics Leadership Accountability People/Skills Training Utilization Qualifications Communication
PDM
TE C H E G A E M T N
Figure 1-1 Maintenance Optimization Program
To derive maximum value, all four categories and their sub-categories must be integrated to form one cohesive program. The key sub-categories are listed below:
Work Process Work Identification Work Planning Work Execution Work Close-out
M N A
Technology Work Management System Diagnostic Technologies Integration Tools/Techniques
1-4
Figure 1-2 is the benchmarked Spider chart utilizing the Power Industry as the source database. The outer circle at 8.0 represents best practice elements found in industry. The inner circle at 5.8 is the industry norm. Big Sandy Plant was evaluated and found to be at or near the industry norm. Big Sandy Plant has second quartile performance when compared to industry benchmarks. Information Integration, Work Execution, Metrics, and Utilization elements are slightly below industry norms. It should be noted that overall, Big Sandy has relatively equal strengths throughout most elements. Information Integration can be improved in accordance with some of the recommendations coming from this project. Metrics are also contained in this project. Utilization will improve as more effective planning and a scheduling processes are put into place. Big Sandy staff given the direction has the strengths that are necessary to succeed.
P M B as is Q ualific ation Com m unic ation Intra Utiliz ation

8.00 7.00 6.00 5.00 4.00 3.00 2.00 1.00 0.00
W ork Identific ation W ork Control W ork E x ec ution
Training
W ork M anagem ent S y s tem s
B enc hm ark ing
CB M Tec hnologies
S etting G oals Com m unic ation Inter M etric s Leaders hip
Inform ation Integration Tools Continuous Im provem ent A c c ountability O rganiz ation
Figure 1-2 Big Sandy benchmarked against Industry Best Practice
1.4.1.1 Work Process For a maintenance program to function cost effectively, all equipment maintenance work should proceed through four distinct steps Maintenance Strategy, Work Identification, Work Control (Planning and Scheduling) and Work Execution (including feedback) as shown in Figure 1-3.
1-5
Work Flow Process

PM Basis Maint. Strategy CBM Work ID Planning Scheduling Work Control Work Activities Work Execution
Feedback CBM
Figure 1-3 Work Process
Using the Work Process Model - Figure 1-3 the assessment team produced a detailed model that highlights the areas of responsibilities. These are shown in Figure 1-4.
1-6
PM Basis Change Control
RCA
PM Basis
Boiler Specialist
PM's
Planners
Work Identification
CBM Data Collection & Analysis
CBM Coordinator Technology Owners
Information Integration & Decision CD's, PR's
Boiler Specialist CAP Technology Owners, CBM Coordinator, RCA Process Owner Boiler Specialist
Work Mgmt System

Operators Initiator
Work Order Generation
CM's Planning
Planners Boiler Specialist
Work Control
Scheduling
Planners Boiler Specialist
Work Execution
Work Execution
Boiler Specialist Craft Boiler Specialist Boiler Specialist, CBM Coordinator
Post Maintenance Testing
Close Out
Boiler Specialist Planners, Craft
Figure 1-4 Work Process Model
Maintenance Strategy
The starting point for developing a maintenance strategy is to first determine a maintenance objective. An objective should be quantified in terms of availability and performance. These requirements should be determined jointly between the Maintenance and Operating and or the 1-7
Improvement
Production Departments. Maintenance Management is then in a position to determine the best mix of maintenance techniques to used, resources requirements and the costs thereof. A maintenance strategy is then developed from these agreed objectives. It defines the process for identifying and achieving sound operational objectives, selecting the maintenance approaches, monitoring performance and providing effective management control. The strategy therefore consists of a management and technical strategies. The Management strategy defines how the business management skills are used to integrate people, policies, equipment and practises to identify improvement opportunities. The technical strategy states how the technical knowledge and experience are used to identify and implement the best proactive maintenance, repair, service and replacement of all equipment in line with AEPs business performance objectives. The maintenance strategy is built primarily from the preventive maintenance (PM) basis that exists at the plant. The aim is to develop an optimized set of PM tasks that will ensure Big Sandy Plant - Unit #2 Boiler will achieve its reliability targets.
Findings:
The Boiler Specialist owns the Boiler System PM Basis. The Boiler System PM contains most of the inspection requirements, physical improvement PMs, etc for work to be performed at various intervals on the various boiler components. Boiler Specialist ensures that the technology owners complete their inspection PMs in accordance with the PM Basis for the Boiler. The Boiler Specialist actively participates in the Root Cause Analysis performed on the boiler events. Operating procedures have not been reviewed for some time and are out of date with current practice.
Recommendations:
1. The PM Basis must be reviewed and updated to take into account the failure mechanisms identified during the past outage inspection. 2. The Boiler Specialist should have an Operating Department counterpart who would be responsible to review and update and align operating procedures to current practice. 3. Boiler tube failures/Root Cause Analysis should be tracked and trended. 4. The boiler inspection plans need to be updated to take cognise of the change in the frequency of boiler periodic outages.
Work Identification
Work on equipment is identified from one of the following three sources of information:
1-8
Corrective Maintenance Repair tasks (CM) - tasks as a result of loss-of-performance, component breakdown, or catastrophic equipment failure that must be attended to immediately. These tasks are generally not scheduled. Preventive Maintenance Tasks (PM) - tasks that are time based recurring work that is deemed necessary to keep equipment in optimum running condition. Implementation of this work should be both planned and scheduled. Predictive Maintenance tasks (CD) - tasks that are condition directed based on information derived from a variety of condition information sources. These sources are identified and controlled on an E&CI matrix. All condition directed efforts to maintain equipment should be both planned and scheduled.
Findings:
Prioritization seems to be fix it now or fix it later. When dealing with boiler issues, which are mainly outage tasks, value of work concepts should replace prioritization. The work process is split into two separate planning windows. One for emergency work i.e. works requiring immediate attention and the other for normal work, which passes through a daily scheduling activity. Emergency maintenance work is discussed between operations and maintenance leaders, while all other work orders (CM, PM and CD) are reviewed, selected and assigned to be executed by the PSLs. The various Crafts persons walk the job down and plan job requirements. There is no formal root cause analysis process in place to ensure failures are thoroughly investigated to prevent recurrence.
Recommendations
1. A formal RCA program will result in additional information valuable in managing the Boiler. 2. The roles and responsibilities of the PdM Co-ordinator should be clearly defined and should include the following aspects: Ensure data collection routes and analyses are performed on schedule. Conduct pre-outage inspections. Post-maintenance testing. Working with system owners to integrate condition based data/information Develop and maintain a condition status report. Document case histories and cost-benefit calculations. Ensure PDM results are used to moderate the PM program extend periodicity, PM activity modifications, removing redundant PM etc. Thus taking full advantage of the CBM program. 1-9
3. Infrared thermography should be used on the boiler to identify casing leaks, valve passing etc. 4. Develop an Equipment and Condition Indicator matrix (E&CI) for all critical boiler equipment/components such as in Figure 1-5. This will assist the equipment owner to assessing the condition of equipment under his control.
Work Control
The work control function shown in Figure 1-3 is an important function when focussing on reducing O&M costs and simultaneously increasing reliability One of the activities when optimizing work control is to review and purge if necessary the corrective maintenance (CM) component of the workload. CM negatively impacts the ability to meet performance goals of reducing O&M cost performance and Equivalent Forced Outage Rate (EFOR). Decreasing the amount of CM does not mean that equipment failures will no longer occur. Rather, it means that management will be actively involved in deciding whether the best economic decision for the station is to allow a specific component, with known problem conditions to run-to-failure; or, should plans be established to effect the repair at a convenient time.
Findings
The frequency of major outages has been changed to every 3 years with an inspection/minor repair outage of short duration at approximately 18 months. Outage work scopes are agreed to prior to the outage. However, a considerable amount of outage work is uncovered during the outage with the inspections. Review of the data available prior to the outage indicates some of the inspection results could have been predicted and repairs planned. Planners prepare work packages for critical work orders. Boiler Specialist serves the role of QA/QC for all boiler activities whether performed by AEP forces or contractors. Forced outages are planned, reviewed, and published once per month. Not all items on the forced outage list are fully planned and trigger ready. Forced outage plans are coded in the PIMS and maintained by planners.
Recommendations
1. In order for the 3 yearly boiler outages to be successful a detailed inspection plan needs to be developed by the boiler specialist to all known and potential problem areas are adequately inspected and repaired to ensure a high availability between periodic outages. 2. Develop a four-week rolling scheduling process for all maintenance to assure high utilization of resources. 1-10
3. Include post maintenance testing requirements into the planning process. This is a must for most CD work orders.
Work Execution
This is the final activity and includes work close-out.

Findings
AEP support personnel and contractors do a reasonable job judged by the small number of QA/QC issues raised and re-work done. Pad welding and temporary repairs are tracked and noted to enable final repairs to be done during a suitable planned outage. Limited post maintenance testing is done. Work close out is performed by the PSLs and at times without the knowledge of the Boiler Specialist.
Recommendations
1. System owners and the Boiler Specialist should review close-out information for
effectiveness.
2. Compile a management directive identifying the minimum acceptable standards to be

complied with when performing boiler maintenance or repair work. This would include such statements as every tube leak/failure should be investigated, pad welding should be discouraged etc. 1.4.1.2 Technologies An Optimisation program is made up of people, management systems, component owners, technology specialists and predictive maintenance tools or equipment. The key predictive maintenance tools used in a Boiler Reliability Optimisation project are infrared thermography, vibration, oil analysis, ultra-Sonics and plant process data and information such as air, gas water and steam temperatures and pressures, metal temperatures and water and steam chemistry data. Equipment owners and technology specialists collect and collated data on various boiler components. This data is organised and presented in an Equipment and Condition Indicator Matrix. The matrix specifies the predictive maintenance technology to be used, the equipment to be monitored and when the data should be collected. The how, what, and when of condition data analysis is based on exceeding preestablished threshold levels and this information is presented in an Event Report and in an Asset Condition Status Report. Analysis of the data and diagnosis of the cause incorporates all available information including diagnostic data from all related technologies, operating conditions from process data, operating log information, maintenance history, design information 1-11
and knowledge of the experienced personnel. Condition Assessment Reports should list, by asset, any marginal or critical determinations by each technology, and include recommendations for action based on the integrated analysis of the data. The what, how and when of action is based on analysis of the data, and this establishes what action is to be taken to correct the anomaly, how the work should be accomplished, and when the work needs to be done to prevent an unplanned failure. Determining the threshold levels to declare an event is based on technology standards and deviations from baselines for the monitored equipment. Therefore, it is necessary to establish baselines for all equipment in the PDM Program, and re-baseline the equipment following any related repairs by performing post-repair acceptance testing.
Findings
Draught gauges and metal temperature thermocouples were found to be defective. No on line sodium analysers for continue monitoring of steam and water quality was available.
Recommendation
1. Repair and calibrate all essential/critical boiler instrumentation.

Work Management System Findings
PIMS meets the basic needs of the organization and has the capability to capture boiler inspection data.
Recommendations
1. PIMS should be expanded to capture all boiler inspection data

Maintenance & Diagnostic Technologies Findings
NDE Inspections are consistent with industry practice Limited boiler performance testing and trending of results is done.
Recommendations
1. Establish technology owners with responsibilities to include condition analysis. 1-12
2. Need to have a more comprehensive PDM reports as health report not exception reports.
Information Integration
The key to utilizing the condition-based information in an effective manner is the ability to quickly access the information and to trend the data over time to enable a prediction of when work should be performed. EPRIs Boiler Maintenance Workstation (BMW) is one such system, which can be used to track and trend boiler tube failure data and information. Another useful tool is the WEB based Condition Status Report software. The software has the capability to store equipment data and information, capture recommended actions, and report on the overall condition of the critical plant equipment in detail. From this information, management can determine whether the overall condition of the plant is deteriorating or improving by tracking the history and costs of the equipment. Decisions made and or action taken and work order numbers are also recorded. Post maintenance test results are recorded to determine whether the diagnosis and maintenance was appropriate.
Findings
There is no integrated condition status reports produced on the boiler or in fact any other part of the plant.
Recommendations
1. Formalise monthly Boiler Condition Based Maintenance (CBM) meetings. Attendance by Equipment/Component and Technology owners, PdM Co-ordinator, Operating and Maintenance staff and the Boiler Specialist to make maintenance decisions based on the condition of the equipment/component. Once a year, perhaps prior to any budgeting cycle, the Boiler Specialist should present a Boiler Failure Defence Plan indicating the current condition, short and long term activities/tasks that need to be undertaken to ensure boiler reliability. 2. Case histories and cost benefits analysis of using PDM technologies should be document either in the BMW or PIMS databases. 1.4.1.3 Management and Work Culture The objective is to develop a work environment in which an effective combination of people, work processes and technologies are used to achieve the overall business goals. To achieve this management must exhibit leadership i.e. the ability to delegate responsibility and authority, and to hold people accountable for their performance. Leadership also includes defining organisational direction, setting realistic measures and targets and communicating these through out the organisation. Measures (metric) can be lagging measure the output of the process or leading a measure of the effectiveness of the elements of an organisation.
1-13
Recommendations
1. Roles and responsibilities must be clearly defined and understood by the individuals in the plant. 2. Measures (metrics) and targets need to be developed for the boiler. These metrics could be the following: # of tube leaks per year suggest one tube per boiler per year, no tube leaks caused by sootblower erosion, no repeat boiler tube leak failures, boiler availability, boiler forced outage rate as a contribution to the overall forced outage rate and long term plant health measures like a thermal index which is a measure of the equivalent number of operating hours or extra life consumption experienced by boiler headers as a result of operation at metal temperatures in excess of design. These measures and targets should be specific, measurable, achievable within a specified period of time and reasonable and should be communicated through out the organisation. 3. An annual review of the maintenance program should be undertaken to determine its effectiveness i.e. the right technologies are being applied to the right assets at the right frequency to meet the overall business objectives and maintenance strategy. These assessments should validate the following questions: Were there unexpected or premature failures that were not detected by at least one technology? This could result in applying PDM to an asset that was not previously being monitored, or increasing the frequency of monitoring on a previously established route, or indicate the need for a new PDM technology, or readjustment of a CM Task. Were failures occurring that were detected but without sufficient time to take action? This could result in adjustment of alert threshold levels. Were there long periods of operation without any developing problems that contained numerous condition data sets? This could result in decreasing the frequency of monitoring. Was a technology ineffective in detecting an emerging problem? This could result in removal of an ineffective technology from the program. Did the costs, measured in dollars, exceed the benefits of the program? This could signal the need to redesign the program. Was all of the data collected analyzed? Was action taken on detected events? This could result in a refocus of the program leaders and a re-aligning of sponsors. Did the program fail to meet the established goals? This could result in a redesign of the program. 1-14
4. Consideration should be given to having an annual Boiler Specialist Meeting and forming a Boiler Specialists Users Group. This will aid in the transferring of information and knowledge amongst the Boiler Specialists within the AEP organisation. 1.4.1.4 People Human beings are an important asset. Their reliability making and keeping commitments has a huge impact on plant reliability. All equipment can in some way be traced back to human beings. If human beings are reliable, your equipment will be reliable.
Findings:
Training meets the basic needs of the plant. New equipment training for the craft is lacking. Communications among PSLs is good. Good team work. Utilization is lower than expected. This is as a result of inadequate planning and staff has to wait for other disciplines.
Recommendations
1. Consider providing Level of Awareness training (LOA) training for the entire staff on Plant Maintenance Optimization. 2. Re-train all PIMS users on the details of PIMS and how best to it. 1.4.2 Failure Defence Plan Water and Steam Touched Tubing To have an effective defence plan, it is important to uniquely identify Equipment Identification Number (EID) all the various sections and/or components to be included in the Defence Plan. This number is then used in the Computerised Maintenance Management System (CMMS) to track inspection details and trends. Table 1-1 shows all the various tubing sections of the boiler. Table 1-2 represents the Failure Defence Plans for the various sections of tubing listed in Table 1-1. These plans were developed from a detailed boiler inspection and discussions with the Boiler Specialist. They identify the failure mechanisms, direct causes, short and long-term actions to be taken to prevent or at least minimise the failure and or damage. These action plans also give the various NDE and NDT techniques to be used to identify and quantify damage by the various failure mechanisms and the criteria to be used when using performing boiler maintenance and or repair work. The following failure mechanisms were identified as being active in the various tubing sections: Sootblower erosion Fly ash erosion 1-15
Fire side corrosion Corrosion fatigue Dissimilar metal welds Oxygen pitting Short term overheating Long term overheating
Findings
Economiser
No significant tube failures have been experienced in the economizer. The economizer is of the ringed tube type and suffers from pluggage especially on the North side. Baffles have been installed to equalize gas flow and minimize ash pluggage. Clean air velocity tests have not been done to verify the correct position of the baffles. Plans are in place to open up gas passes to minimize pluggage. One leak was encountered last year due to flyash erosion. Tube shields were installed 2 years ago to minimize flyash erosion. No header cracking has been noted.
Waterwalls
The ash hopper slope has channelling between the membrane and the tube in the tube material. About 82 tubes have been found with some amount of material loss due to slag erosion. No leaks have been attributed to this problem. Slagging of these tubes has not been a problem since changing to low NOx burners in 1994. The sidewalls have experienced severe wall loss since conversion to low NOx burners. Portions of the sidewalls were overlaid in the fall of 1999 with Type 312 SS to mitigate this problem. No further testing has been performed since this inspection. The unit has only experienced one trip since the outage and did not stay off line long enough to allow for testing. This overlay will be inspected in March and an additional 500 sq. ft. will be overlaid. Since the overlay outage, testing has been performed using 3/8 tubing inserted through the wall membrane during the outage on the left side. Initial measurements showed no oxygen during operation. Burner modifications and adjustment have raised the oxygen content to a positive level. Plans are to install several more taps into the firebox during the next outage to help facilitate future burner adjustments. Currently measurements are performed manually on the sample positions. Automatic sampling should be developed and tied into the database with feedback to controls or maintenance. The upper furnace has experienced no problems.
1-16
The center division wall has experienced no problems. One failure has been reported in the roof due to erosion from a casing leak. The attic has since been pressurized and no more leaks have been observed. Several areas of casing/membrane leaks were noted in the upper firebox. These should be repaired at the next outage to prevent erosion from ash or casing damage from heat. Screen tubes have had some erosion from flyash and sootblowers. Tube shields have been tried with no success. The screen tubes are carbon steel with a wall thickness of 0.300. Some pad repair or weld overlay work is expected next outage. More information is needed on this problem including UT thickness readings. This wear is probably from water in the sootblowing system but could be aggravated by large coal size, late combustion and LOI. Plant staff indicated coal mill fineness tests were performed twice a year. No severe problems were noted with the waterwalls other than the wastage due to low NOx burners. This is being addressed with the weld overlay and burner adjustments.
Platen Superheat
Approximately 300 DMW are located in the platen superheat section and are scheduled for replacement using EPRI developed technology.
Finishing Superheat
Some sootblower erosion has occurred in this section. Some tubing has been pad welded to restore minimum wall. No leaks have occurred due to erosion. Alignment handcuffs in this area are in need of repair. Current plans are to install heavier handcuffs next outage to keep tubing in alignment and avoid flyash erosion. No plugging problems have been noted in this area except when sootblowers are out of service. The tubing in this area was too long when installed and hit the slope when hot. This tubing has been shortened. No operational problems are noted at this time.
First Reheat (High Pressure)
This area probably contains the most damage of any part of the boiler. Past lay-up procedures allowing moisture to condense in this area has led to oxygen pitting in the horizontal sections. Current lay-up procedures of using nitrogen blanketing, drying out with dried air, or heating up to dry after coming off line are believed to have stopped the damage. The second or third tube in the bundle has suffered from long term overheat fish-mouth type failures. Other AEP 800 MW plants have replaced this tubing with P91 material. A life assessment using EPRI TUBELIFE and other NDE should be performed on this tubing. If tubing requires replacement, T91 or other new upgraded material such as T23 should be used. Damage is also occurring in this section from vibration fretting of the cane tube spacers. Egg shaped spacers are placed vertically between tubing bundles and turned until tight. Then 1-17
straps are welded between adjacent spacers to hold in place. No attachment is made to the boiler tubing. During operation these spacers become loose and allow the tubing to vibrate, fretting the tubing and causing leaks. AEP is currently not replacing spacers when they fall out.
Second reheat (low pressure)
The outlet header and top two sections of tubing have been replaced with SA 213 T-91 material in the highest temperature areas. No other leaks or failures have occurred in this area.
Spray Attemperators
The nozzles have been inspected and replaced. The liner has been replaced on the Main Steam Attemperator. This equipment is currently on a seven-year inspection cycle. This inspection frequency should be monitored and adjusted as required. Liners should be inspected with boroscope. Nozzles should be removed and inspected. Water stop valves should be inspected to prevent thermal shocks to the superheat and reheat piping.
Recommendations
1. Develop a series of inspection tasks using the data and information identified in the Failure Defence Plans in Table 1-2. These Defence Plans need to be reviewed and converted into specific action tasks with allocated responsibilities for completion. The inspection type tasks need to be formatted and standardised and put into the Computerise Maintenance Management System database for reference and future boiler inspections.
Table 1-1 Component Description and Engineering Identification Number # 1 2 3 4 5 6 7 8 9 EID 5512230200 ECON 1 5512230200 ECON 2 5512240200 DIVWL 5512240200 LFFWL 5512240200 LFSWL 5512240200 LFRWL 5512240200 UFRWL 5512240200 UFFWL 5512240200 UFSWL DESCRIPTION Economiser Tubes U2 1 Reheater side Economiser Tubes U2 2 Reheater side Waterwall Tubes U2 Division wall Waterwall Tubes U2 Lower furnace front wall Waterwall Tubes U2 Lower furnace side walls (incl. 40 tubes front and rear) Waterwall Tubes U2 Lower furnace rear wall Waterwall Tubes U2 Upper furnace rear wall Waterwall Tubes U2 Upper furnace front wall Waterwall Tubes U2 Upper furnace side walls
nd st
1-18
EPRI Licensed Material Boiler Reliability Optimization Project Phase 1, 2 & 4 at American Electric Powers Big Sandy Plant Unit 2 10 11 12 13 14 15 16 17 5512243200 5512241200 HRAT 5512241200 5512240200 FROOF 5512210200 5512212200 5512220200 5512222200 Screen Tubes U2 Furnace Heat Recovery Area (HRA) U2 Incl. Side, front and rear walls, partition wall and roof Aperture Tubes U2 Incl. Side walls, floor and screen Waterwall Tubes U2 Furnace roof tubes Platen Superheater Finishing Superheater 1 Reheater 2 Reheater
nd st
1-19
Supercritical Waterwall Cracking
Acid Phosphate Corrosion
Possible Causes
# Malfunction of a sootblower or 1 the sootblowing system because of incorrect operation and/or inadequate maintenance (control logic sequence, operating temperature and pressure, mechanical misalignment, drainage pipe slopes etc)
Fatigue
Pitting
Preventive Measures Short/Long Term
NDE Detection Technique
Acid Dewpoint Corrosion
Low Temperature Creep
Short Term Overheating
Coal Particle Erosion
Corrosion Corrosion
Falling Slag Erosion
Sootblower Erosion
Fire Side Corrosion
Hydrogen Damage
Corrosion Fatigue
Caustic Gouging
Thermal Fatigue
Fly Ash Erosion
CRITERIA
A B C D E F G H
L M N O P Q #
Confirm mechanism by VT inspecting tubes around and in /EMAT the immediate vicinity of the blower. Determine the extent of the UT erosion by mapping the tube thickness in the affected ares. Compare with previous readings and calculate an erosion rate and trend results if applicable.
Check for tube wastage - flat and/or 'shiny' surfaces or if the boiler has been washed check for a fresh layer of rust and gouges from steam cutting. Use AEP's Standard for repairing or replacing tubes. Typically 70% of wall thickness is used. Do not pad weld to restore thickness for < 70% because of the potential to introduce other detrimental failure mechanisms - copper embrittlement and hydrogen damage. If root cause of erosion has been determined and adequately addressed no additional erosion should place.
Components 1 through 16 (all components within sootblower range)
Check, correct and verify the VT operation of moisture traps and remove remaining orifices. Check, correct and verify VT blower or system operation to ensure correct sequence, frequency, angel of rotation, travel, misalignment, operating steam temperature and pressure. Check, correct and verify VT steam supply (pressure and temperature), drainage slopes and piping insulation. Use the OEM's manual or site specific procedure.
1-20
Possible Causes
#
Fatigue
Pitting
NDE Detection Technique VT
Corrosion Corrosion
Sootblower Erosion
Fire Side Corrosion
Hydrogen Damage
Corrosion Fatigue
Caustic Gouging
Thermal Fatigue
Fly Ash Erosion
CRITERIA
A B C D E F G H
L M N O P Q # Emergency repairs such as pad welding should be replaced at the next opportunity. Develop a map identifying the exact location of each pad weld to ensure all locations are recorded. Optimize blowing sequence to ensure an effective boiler sootblowing program. Establish and implement a routine maintenance schedule to inspect and test blower operation.
1-21
Possible Causes
# Primary causes are excessive 2 local velocities and/or dust burden. Excessive local velocity design/configuration distorted/misalignment tubes - Gas flows above design rating Dust burden - Increase in erosive particles ie increase % ash - Sootblower operation Other - Incorrectly install shields and/or baffles Inappropriate and/or incorrectly applied coatings - Broken or misaligned "canes" and "handcuffs.
Fatigue
Pitting
Corrosion Corrosion
Sootblower Erosion
Fire Side Corrosion
Hydrogen Damage
Corrosion Fatigue
Caustic Gouging
Thermal Fatigue
Fly Ash Erosion
CRITERIA
A B C D E F G H
L M N O P Q # Confirm mechanism and UT VT determine the extent by completing a dirty and a clean inspection. The dirty inspection will highlight areas where excessive local velocities occur. These can be comfirmed by a detailed clean inspection. The areas mostly likely to show signs are at the base of the fins of the first economiser bank, the element ends between tube and wall and adjacent to any flow staighteners and/or baffles. Perform a CAVT to confirm high velocity areas. If high velocities are detected - design and install baffles or flow straighteners. Perform a second CAVT to confirm actual velocity profile. If baffles and/or flow straighteners have been installed perform an annual inspection to determine the effectiveness of the modification and adjust if necessary. Use AEP's standard for tube replacement (<70% wall for waterwall tubing and <85% for steam tubing) in conjunction with EPRI's remaining life assessment calculations results from Level I or II as judgement criteria - TR111559. If trend data available, replace tubing that will drop below wall thickness criteria before next outage. If erosion correction action taken and confirmed to eliminate future wall loss, can leave tubing in service with >70 % but <100%.
1-22
Component # 1, 2 and 12 through 17
Possible Causes
#
Fatigue
Pitting
Corrosion Corrosion
Sootblower Erosion
Fire Side Corrosion
Hydrogen Damage
Corrosion Fatigue
Caustic Gouging
Thermal Fatigue
Fly Ash Erosion
CRITERIA
A B C D E F G H
L M N O P Q # Repair, replace and/or align damaged tubes and/or elements. For major damage, plan for element/bank replacement. If damage is at the base of the fins on the first bank and replacement is necessary then consider designing out the problem by installing straight tubes. Repair, replace and/or align damaged "canes" and "handcuffs". Check for and remove any ash build up/plugage in the economizer tube banks. 70 % through 200 mesh See EPRI document for economizers GS-5949 Use AEP's Standard for repairing or replacing Tubes
1-23
Possible Causes
Components 3, 4, 5 and 6 Primary cause is a substoichiometric (reducing) enviroment as a result of poor combustion - burner air and fuel distribution and incorrectly set burner registers. Secondary causes are related to fuel composition in particlar the amount of chlorine and aggressive ash and also flame impingement. # 3 A B C D E F G H I
Corrosion Corrosion
Sootblower Erosion
Fire Side Corrosion
Hydrogen Damage
Corrosion Fatigue
Caustic Gouging
Thermal Fatigue
Fly Ash Erosion
Fatigue
Pitting
CRITERIA
L M N O P Q #
Confirm mechanism and UT Use AEP's standard for replacement ie extent of wall thinning by /EMAT < 70% wall. inspecting the area around the CO level of < 1% and/or Oxygen levels burners (approx 2 diameters) of < 0.1% and on the side walls at Wastage rate of 12 mills/yr is considered approx. midway between front excessive. and rear walls. these surveys should be conducted in Use AEP's standard for repairing or conjunction with remain ing life replacing tubes assessment techniques to determine wastage rates. Also monitor/measure the levels of Oxygen and CO near the tubes on the side walls. For damage, repair by the use of high chromium weld overlay material in affected areas to resist corrosion attack. Optimize combustion to minimize the affect of substoichiometric conditions and institute long-term monitoring - inspections and oxygen and CO at or near the tube walls. Optimize combustion to minimize the affect of substoichiometric conditions and institute long-term monitoring of oxygen and CO at or near the tube walls.
1-24
Possible Causes
# A B C D E F G H I

Inspect, repair or replace damaged burner quarls. Develop a spreadsheet to be use to collect and trend thickness measurement data.
Corrosion Corrosion
Sootblower Erosion
Fire Side Corrosion
Hydrogen Damage
Corrosion Fatigue
Caustic Gouging
Thermal Fatigue
Fly Ash Erosion
Fatigue
Pitting
CRITERIA
L M N O P Q #
Components 4 and 6 (hopper slope tubes and areas of high heat flux)
Cyclic stresses brought about 4 by improper start-up and shut down practices and Poor water chemistry control especially the control of ph during shut down and start-up.
Confirm mechanism and VT UT Using specialised UT/EMAT test for ID determine the extent by initiated cracks. If cracks are > 50% inspection and sampling. through wall then replace tubing. Remove sample attachment tubes and areas of high stress. Replace damaged tubes. Modify tube attachments to relieve obvious stress raisers. Ensure compliance to AEP's water chemisrty practices. Ensure compliance to startup and shut down procedures. Ensure compliance to AEP's chemical cleaning procedures. Correct tube spacing Set inspection levels to determine the effectiveness of any modification and monitor damage accumulation. Use AEP's Standard for repairing or replacing Tubes
1-25
Chemical cleaning Damage
Possible Causes
Component # 14 Higher than expected metal temperatures (brought on by overfiring conditions) and stresses - both thermal and differential expansion # 5 A B C D E F G H I J K L M N O P Q #
Stress Corrosion Cracking
Short Term Overheat
Long Term Overheat
Sootblower Erosion
Fireside Corrosion
Rubbing/Fretting
Dissimilar Metal
Fly Ash Erosion
Graphitization
Fatigue
Pitting
Creep
CRITERIA
Use AEP's stanard for tube insert- 18 inches preferably 36 inches. Use EPRI's standard for weld geometry.
Confirm mechanism and VT UT determine the extent. Map location and inspect all DMWs. Replace DMW with a "dutchman"of upgraded material. Depending on the specific circumstamces re-positioning the DMW may be more cost effective. Perform a remaining life assessment calculation (Level I, II or III). Inspect and correct tube supports, tube bindings etc leading to axial stresses at welds. Ensure functionality of metal and steam temperature thermocouples.
DMW life code from SIA (based on EPRI PODIS code)
1-26
Possible Causes
Component # 17 Oxygen-saturated stagnant condensate as a result of a unit shutdown - forced cooling and/or imprper draining and venting and aggrevated by the Carryover of sodium sulfate from the reheat steam (Sodium Sulfate tends to form at typical reheat pressures) # 6 A B C D E F G H I J K L M N O P Q #
Short Term Overheat
Long Term Overheat
Sootblower Erosion
Fireside Corrosion
Rubbing/Fretting
Dissimilar Metal
Fly Ash Erosion
Graphitization
Fatigue
Pitting
Creep
CRITERIA
Use the remaining life assessment calculations results from Level I or II as judgement criteria.
Confirm mechanism and VT UT determine the extent by inspection and sampling in particular areas where the tubing has sagged allowing condensate to accumulate. If root cause has been identified and rectified, minor pitting damage can be left alone ie there is no need replace tubing. Continue to monitor and take samples for analysis to check for deterioration. For major damage, plan to replace, modify operating procedure and institute long-term monitoring. Review and correct if necessary the effectiveness of the shut down and lay up procedures. Enforce compliance to shut down and lay up - nitrogen blanketing procedures. If the lay up procedure nitrogen blanketing cannot be used because of boiler work, some other means of drying must be used to ensure a moisture free enviroment in the reheater.
Use AEP's Standard for repairing or replacing Tubes
1-27
Check drainage slopes and tube bundle sagging. Correct if possible, if not possible then to check for any deterioration by inspection and sampling. Chemical cleaning Damage NDE Detection Technique Stress Corrosion Cracking
Short Term Overheat
Long Term Overheat
Sootblower Erosion
Fireside Corrosion
Rubbing/Fretting
Dissimilar Metal
Fly Ash Erosion
Graphitization
Fatigue
Pitting
Creep
Possible Causes
Component # 3 to 17 (water and steam circuits) Exfoiliation induced: Caused by the natural process of growth and exfoiliation of oxides. These oxide flakes tend to collect at sharp bends. Mainenance induced: Caused by poor quality control during welding, improper/incorrect chemical cleaning or poor combustion resulting in flame impingment on water walls. Operatging induced: Caused by improper start-up and shut down practice (not boiling out condensate collected in the loops/bends) and overfiring to compensate for low feedwater temperature. # 7 A B C D E F G H I J K L M N O P Q #
CRITERIA
Confirm cause by either an VT MT analysis of a sample of the debris or start-up and/or shut down records. Confirm extent through inspection of tube in the immediate vicinity. Check for swelling and cracks. Check to ensure blockage has been removed by using compressed air or water. Repair by inserting a 36 inch Use AEP's Standard for chemical cleaning "dutchman". Ensure compliance to start-up and shut down practices. Ensure compliance to start-up and shut down practices. Check combustion and or burners are optimized to ensure no flame impingment is taking place. Use AEP's Standard for repairing or replacing tubes
1-28
Possible Causes
Component # 17 Prolonged operation at high temperatures (design operating temperature and above) # 8 A B C D E F G H I J K L M N O P Q #
Short Term Overheat
Long Term Overheat
Sootblower Erosion
Fireside Corrosion
Rubbing/Fretting
Dissimilar Metal
Fly Ash Erosion
Graphitization
Fatigue
Pitting
Creep
CRITERIA
Use AEP's Standard for repairing or replacing Tubes
Confirm mechanism and VT UT determine the extent by inspection. Perform UT metal and ID oxide thickness measurements. Remove several samples both across and within the bank for analysis and replace withn a 36 inch "Dutchman" Perform a remaining life assessment calculation (Level I, II or III).
For major damage, plan to replace, modify operating procedure and or design and institute long-term monitoring. Perform hydraulic test as an integrity check.
Use the remaining life assessment calculations results from Level I or II as judgement criteria- EPRI's TUBELIFE program. Replacement should be T91 or equivalent material such as T23.
Compliance to ASME code requirements fo r test pressures and temperatures. Increasing pressures above code to force failure is a poor practice and fruitless as crack > 50% through wall can tolerate design pressures without failure.
Calibrate and ensure functionality of all installed metal temperature thermocouples. Set inspection levels to determine the effectiveness of any modification and monitor damage accumulation.
1-29
1.4.3 Streamlined Reliability-Centered Maintenance Streamlined Reliability Centered Maintenance is a logical, systematic, functionally based methodology used in the evaluation of a facilitys system or unit. The evaluation includes a stepby-step consideration of the integral functions related to the operation of the system or unit. The failure mechanisms of each of these functions, the effects of the failure and the selection of appropriate and effective maintenance tasks to mitigate these failures are identified. The goal is to develop a cost-effective maintenance program, based on system functionality, which will enhance system reliability. This makes optimum use of available maintenance resources and provides a documented base for future additions/revisions to the maintenance program. Two systems were reviewed steam- and water-touched tubing and the fuel system. Appendix 1 gives the results of this analysis. These results still need to be compared with those in Computerised Maintenance Management System. The purpose of this comparison is to identify changes to the existing program, to optimise the Fuel system preventive maintenance program. The comparison also provides another check of the analysis to ensure completeness and validity of assumptions. Once the comparison is complete and has been approved the new tasks can be incorporated in the CMMS. 1.4.4 Unit 2 Outage Task Prioritisation An EPRI Risk Evaluation and Prioritisation (REAP) model was used to demonstrate how outage tasks can be evaluated and prioritised. Thus providing a documented basis for decision making. The model is based on determining the risk of performing or not performing a preventive or corrective maintenance task. This is achieved by scoring a number of answers to questions about the task at hand. Labour hours and cost are also used to determine the value and cost of the task. The following is the results of a REAP analysis performed using the Unit 2 outage task list. A total of 119 maintenance outage tasks were reviewed. These tasks had been determined from a much longer list, which had been subjected to a gut feel prioritisation method. Unfortunately the total list was not available to compare the gut feel method verse the approach taken in the EPRI model. Figure 1-5 represents the resultant scatter diagram of all 119 tasks analysed.
1-30
T a s k C o s t V a lue S c a t t e r
1.E+09 1.E+08 1.E+07 1.E+06 1.E+05 1.E+04 1.E+03 1.E+02 1.E+01 1.E+00 $100 $1,000
C os t
V alue/C o s t O p timiz atio n Line s
$10,000
Figure 1-5 Task Scatter Diagram - Value to Cost
The angled lines in figure 1-5 are generated by recognising that outage work scopes are built by starting with the most valuable least cost work and continuing to add more expensive valuable work until all to work has been included. Integrating all this work results in the Accumulative Value and Cost curve Figure 1-6.
N o. of Tasks & V alue vs C ost
5.E + 09 4.E + 09 4.E + 09 3.E + 09 3.E + 09 2.E + 09 2.E + 09 1.E + 09 5.E + 08 0.E + 00 0 20000 40000 60000 Costs 140 120 No. of Ta sks 100 80 60 Task s V alue 40 20 0 80000 100000
Figure 1-6 Accumulated Value verse Cost
V a lue
1-31
Figure 1-6 represents the accumulated value versus cost versus number of tasks and shows that at the 32,700 cost level (cost is number of man-hours), 3.74 E09 value is captured with 95 of the 119 outage tasks. In other words with 40% of the outage task hours, 95% of the outage task value can be captured with 80% of the tasks being performed. The vertical lines show where there is a significant slope change. In this case they show perhaps where one can draw the line on an outage work scope i.e. where 80 % of the value is captured with 20 % of the cost. At that threshold, the following tasks would not be performed:
AIR HEATER SEALS AIR HEATER SEALS AIR HEATER SEALS AIR HEATER SEALS 4KV ROOM WALL INSPECT GAS OUTLET DUCTS CIRC WATER PUMP DISCHARGE EXP. JOINTS DEAERATOR INSPECTION OVERLAY SIDEWALLS NDE ON FACTORY WELDS 1ST RH PIPING PA & FD FANS - CLEAN & INSPECT BOILER DIVISION VALVE-OPEN, INSPECT & REPAIR COAL CHUTE RENEW WEAR PLATES COAL CHUTE RENEW WEAR PLATES REPLACE COAL CHUTES REPAIR PENTHOUSE CASING LEAKS REPLACE 80 TUBES REAR ASH HOPPER SLOPE REPAIR APERATURE MEMBRANE 801 ASH GATE BURNER REGISTERS - INSPECT & REPAIR BURNER REGISTERS - INSPECT & REPAIR GENERATOR INSPECTION MAIN TURBINE OIL COOLERS INSPECT #1 BEARING ON TURBINE 3356430 3423125 3381022 3423136 3374302 3422425 3356636 3422484 3357443 3374254 3422941 3420561 3422915 3422904 3415274 3421165 0577695 3365526 3358014 3420966 3420970 3374350 3423195 3424116
Although the outage tasks have all been approved and scheduled, this REAP analysis shows how outage scope of work in terms of cost and value can be determined, evaluated and prioritised.
1-32
1.4.5 Boiler Tube Failure Reduction and Cycle Chemistry Improvement Training From 810 February 2000, an EPRI team presented the above training programs to a number of plant and corporate staff. Appendix 1 lists the names of all attendees. A vertical and horizontal cross section of staff attended, representing all disciplines operating, maintenance engineering and management. The two-and-half day course was well received by all attendees, and they were eager to proceed with the development of a Boiler Tube Failure Reduction program. A plant directive Boiler Reliability Optimisation Project has been signed by the Plant Manager in support of the program. The directive (see Appendix 2) outlines the minimum requirements to manage a successful Boiler Reliability Optimisation program. 1.4.6 Boiler Maintenance Workstation (BMW) installation and training During July 2000 the BMW software was successfully installed. Software user training was given to a number of site staff. BMW is a windows based software program used to track and trend boiler tube failure data and information tube failures, correction and prevention, and control actions. It consists of two modules - Boiler Works and Tube Condition. The boiler works module is a database and graphics program designed to track maintenance failure data/information failure location and mechanism, repair costs and methods, etc. This collated information is used to focus attention on specific activities, for both planned and forced outages, to prevent future repeat failures. The Tube Condition program stores and analyses data collected from various inspections, e.g. tube wall thickness, oxide thickness data. It uses two or more data sets to determine wastage rates. These wastage rates are used to determine remaining tube life and predict tube wall thickness. This helps plant personnel plan future boiler tube inspections, maintenance activities and tube replacements.
1-33
AEP Big Sandy Unit 2, Boiler Reliability Optimization - Phase 4, July 2001 Executive Summary
Utilities throughout North America are facing the challenges of deregulation and competition. The prospect of competing in a non-regulated environment is driving utilities to find more cost effective means to operate and maintain their plants. Recognizing this, Big Sandy Plant participated in EPRIs Boiler Reliability Optimization Program. The project started during the fourth quarter of 1999. The project aim was to identify improvement opportunities to attain and sustain an availability of 95% during peak periods May through August and at least 90% during the remainder of the year. During 2000 phase 1,2 and a portion of phase 3 were successfully implemented resulting in sustained plant performance. As a result Big Sandy Staff were keen to continue with the Boiler Reliability Optimization Project Phase 4, albeit with limited funding. Phase 4 is the continuous improvement phase the proactive maintenance aspects of the program. The emphasis is on the operations aspect that is what the operator can do to minimize maintenance. During July 2001 an EPRI team spent a week on site reviewing actions taken as a result of the Phase 1 to 3, interviewing operating and maintenance staff and published a draft report. This report documents all the findings, conclusions and recommendations accepted by Big Sandy Plant staff.
Introduction
Utilities throughout North America are facing the challenges of deregulation and competition. The prospect of competing in a non-regulated environment is driving utilities to find more cost effective means to operate and maintain their plants. Recognizing this, Big Sandy Plant participated in EPRIs Boiler Reliability Optimization Program. The project started during the fourth quarter of 1999. During 2000 phase 2 and a portion of phase 3 was completed with the issuing of a final report. The staff at Big Sandy were keen to continue with the Boiler Reliability Optimization Project Phase 4, albeit with limited funding. Phase 4 is the continuous improvement phase the living aspects of the program. The scope of work of this project covered the implementation of a portion of Phases 4 of EPRIs Boiler Reliability Optimization Project and was limited to the boiler steam and water system tubes and headers. The project consisted of the following tasks: 1. Review the progress made in implementing recommendations from previous Boiler Reliability Optimization phases. 2. Develop a framework for specific boiler long-term plant health indicators. 3. Review and optimize/enhance the daily operation of Unit 2 to result in proactive maintenance. 1-34
Description of Phase 4 of a Typical Boiler Reliability Optimization Project

Phase 4 is the continuous improvement - the living aspects of a Boiler Reliability Optimization Project. It builds on the successful implementation of the previous phases and provides proactive action and feedback on the overall boiler performance reliability and maintainability aspects. Phase 4 focuses on the human reliability aspects as there is a direct relationship between plant reliability and the people who operator and maintain the boiler. If the people are reliable then the plant will be reliable. In this context human reliability is making and keeping commitments. Phase 4 also introduces the concepts of proactive maintenance into the world of the operator. Proactive maintenance is an activity performed to detect and correct root cause abnormalities of failure. It is a first line of defense against material degradation and subsequent performance degradation, failures that ultimately lead to precipitous and catastrophic forms of failure and plant breakdown. The operator can correct a conditional failure mode that is the cause of an unstable condition and ensure that degradation type failures never occur. Thus proactive maintenance is not an activity that reacts to material and/or performance type failure conditions of a plant. Rather, it prevents such plant degradation from occurring. In reality, proactive maintenance is a preemptive first strike against failure a true avoidance activity. Thus, the focus of Phase 4 is on the: role operations personnel play in the effective and efficient operation of the boiler, tools, both hardware and software at their disposal instrumentation to control and monitor critical component parameters and Standard Operating instructions and procedures.
The ultimate success of a Phase 4 Boiler Reliability Optimization project is largely dependent on the following: Operator skills and knowledge of the boiler and the steam raising process, Operator understanding of the effects operating practices have on both the short and longterm plant health and The extent to which operating is involved in both the long and short-term maintenance decision making process.
Findings 1. All of the recommendations made in the Phase 2 report have been accepted. More than 80% have been implemented while the remaining are in the budgetary and approval processes. Table 1-1 below summarizes the status of the actions taken and for each recommendation given in Phase 2. A recommendation that has still to be implemented is a formalized root cause analysis process. This is key to the successful implementation of a boiler reliability optimization project.
1-35
EPRI Licensed Material Boiler Reliability Optimization Project Phase 1, 2 & 4 at American Electric Powers Big Sandy Plant Unit 2 Table 1-2 Phase 2 Recommendations and Actions Taken # 1 Recommendations Tabled Specific boiler availability targets should be developed relating to the number of tube leaks per year. The strategy and performance targets should be compiled into separate plant directives, which are reviewed annually for effectiveness and appropriateness. Action taken The following the best in class (BIC) goals for unit 2 have been established: 01 FOF MOF POF 5.38 0.41 0.0 02 1.69 0.41 26.85 03 0.82 0.41 1.92
To meet these goals a Strategic Plan for the boiler has been compiled 2 To repair and calibrate all boiler instrumentation critical to the efficient and effective operation of the boiler. This includes air, gas, water and steam temperatures and pressures and reheater and superheater metal temperatures. The installation of on-line water and steam chemistry instrumentation and the establishment chemical performance index. This index can be used in a proactive way to ensure compliance to EPRI water and steam chemistry guidelines. To prepare detailed inspection plans based on the tasks identified in the Boiler Failure Defense Plans presented in this report. These inspection plans should be loaded into the CMMS. This would facilitate the development of a scheduled and or forced outage plan at short notice. Once the installation and the user training on the Boiler Maintenance Workstation (BMW) have been completed, the BMW database needs to be updated with all the history of previous tube failures. This will enable tube leaks and repairs to be tracked and trends to be established. This will also assist in formulating focussed boiler inspections plans. Fireside corrosion Confirm mechanism and extent of wall thinning by inspecting the area around the burners and side wall. Also monitor O2 and CO near the tubes on the side walls During the planned outage in 2000 the boiler tube temperature recorders and draft gauges were repaired. However since then some are out of service awaiting spare parts
Improved cycle chemistry monitoring with the installation of sodium analyzer in sample room and ChemExpert software in the control room. Chemical indices still need to be developed. Prepared detailed inspection plans on noted problems and drew up inspection PMs. Developed a short notice forced outage plan. Tasks are discussed at a weekly meeting every Thursday morning. BMW was installed and the history data file has been updated with data from 1994 to 2000. Currently working on updating the history file with data from 1986 to 1994. Training classes for welders is scheduled for August 2001
Monitoring NOx strategy has taken top priority to measuring O2 on the side walls however inspections and NDE is done during forced outage.
1-36
EPRI Licensed Material Boiler Reliability Optimization Project Phase 1, 2 & 4 at American Electric Powers Big Sandy Plant Unit 2 7 Fly ash erosion Perform a CAVT to confirm high velocity areas. If high velocity areas are detected design and install baffles or flow strengtheners. Perform a second CAVT to confirm results and velocity profile DMW failures Replace all DMW with Dutchmen of upgraded material. Depending on the specific circumstances re-positioning of the DMW may be more cost effective Internal pitting Revise and update the boiler lay-up procedure Only CAVT completed on the economizer and repositioned baffles and re-tested. Results not as expected. Re-testing will continue as outage work permits
All DMW replaced.
1 Reheater lay-up procedure revised and updated. st Performed remaining life estimation on 1 reheater, replaced tubes below minimum wall thickness. Plan to replace reheater in 2002. NDE results taken in 2000 show that there was no need to do a chemical clean. Completed during 2000
st
10
Perform a level II assessment of all final outlet headers in accordance with EPRI guidelines. The PM Basis must be reviewed and updated to take into account the failure mechanisms identified during the past outage inspection. The Boiler Specialist should have an Operating Department counterpart who would be responsible to review and update and align operating procedures to current practice. Boiler tube failures/Root Cause Analysis should be tracked and trended The boiler inspection plans need to be updated to take cognize of the change in the frequency of boiler periodic outages A formal RCA program will result in additional information valuable in managing the Boiler. The roles and responsibilities of the PdM Coordinator should be clearly defined Infrared thermography should be used on the boiler to identify casing leaks, valve passing etc.
11
Developed new inspection PM to inspect repeat mechanisms
12
As a result of the adopting the Process Centering model, this task is now the responsibility of the Sub-process owner.
13 14
BMW software will assist in tracking and trending tube failures and their associated causes The weekly outage-planning meeting discusses and develops a forced outage plan. Inspections plans (PM) are included in these forced outage plans RCA still not formalized. Tube failures are analyzed and tracked through BMW The PdM Coordinator position has been made obsolete by the re-organization Process Centering. However, the PdM activities still remain with the previous incumbent. Infrared thermography survey are performed quarterly as described in EPRI Boiler reliability Optimization IR TAD document
15
16
17
1-37
18
Develop an Equipment and Condition Indicator matrix (E&CI) for all critical boiler components. This will assist the equipment owner to assessing the condition of equipment under his control. To ensure a high availability between periodic outages, 3 yearly, a detailed a inspection, test and repair plan of all known and potential problem areas Develop a four-week rolling scheduling process for all maintenance to assure high utilization of resources. Include post maintenance testing requirements into the planning process. This is a must for most CM work orders. System owners and the Boiler Specialist should review closeout information for effectiveness. Compile a management directive identifying the minimum acceptable standards to be complied with when performing boiler maintenance or repair work. This would include such statements as every tube leak/failure should be investigated, pad welding should be discouraged etc.
Trending boiler process data, using the E&CI matrix as a base, is done using the limited available instrumentation. Addition instrumentation will be installed, if, and when a new DCS is approved. It is intended to install FEGT indication during the next outage.
19
20
It is intended to incorporate a four week rolling schedule in the new CMMS Indus-passport system Post maintenance testing will be incorporated into the new CMMS
21
22
The newly appointed Process Centering Subowners review closeout information.
23
A Boiler Reliability Optimization Directive has been signed by the Plant Manager and is in effect.
24
Establish technology owners with responsibilities to include condition analysis. Need to have a more comprehensive PDM reports as health report not exception reports Formalize monthly Boiler Condition Based Maintenance (CBM) meetings. Once a year, perhaps prior to any budgeting cycle, the Boiler Specialist should present a Boiler Failure Defense Plan indicating the current condition, short and long term activities/tasks that need to be undertaken to ensure boiler reliability.
Process Centering sub-owners have been nominated in place of technology owners.
25
An existing PdM health report is considered adequate
26
Plan to include monthly Boiler CBM meetings into the Boiler Reliability Optimization Directive
1-38
EPRI Licensed Material Boiler Reliability Optimization Project Phase 1, 2 & 4 at American Electric Powers Big Sandy Plant Unit 2 27 Case histories and cost benefits analysis of using PDM technologies should be document either in the BMW or PIMS databases. An annual review of the maintenance program should be undertaken to determine its effectiveness i.e. the right technologies are being applied to the right assets at the right frequency to meet the overall business objectives and maintenance strategy. Consideration should be given to having an annual Boiler Specialist Meeting and forming a Boiler Specialists Users Group. This will aid in the transferring of information and knowledge amongst the Boiler Specialists within the AEP organization. Consider providing Level of Awareness training (LOA) training for the entire staff on Plant Maintenance Optimization. Case histories are being established in the BMW database. Metric are being developed through the centering process
28
Included in the business strategy feedback loop.
29
AEP has formed a Boiler Reliability Optimization Team whose responsibility is to support the implementation of the program
30
LOA training has been done previously. Not considered necessary at this stage.
31
Re-train all PIMS users on the details of PIMS and how best to it
PIMS is to be replaced by Indus passport by January 2002. Training on the new system starts in the last quarter of 2001 Inspection tasks for 2000 have been completed.
32
Develop a series of inspection tasks using the data and information identified and update the CMMS.
2. Five operators with a wide range of experience were interviewed (four Unit operators and one Equipment Operator). The interviews focussed on the knowledge of the boiler and the steam and water cycle. Responses to questions from some of the operators regarding boiler operation and operating parameters on critical equipment were incorrect. Critical components within the boiler were not known. Under base or transient loads the focus is on steam temperature only, metal temperatures are not considered. These same operators have been given latitude to develop their personal operating practices and to experiment with set points on critical equipment. The combined effect of the above can lead to an increased risk with the consequent reduction in unit performance, both efficiency and reliability. 3. Each unit operator interviewed uses his or her own set of procedures - start-up procedure, marked up with his or her own notes. The start-up procedure reviewed in the control room was dated 1970, where as, the set given at the beginning of this project was dated October 1998. Not all operators were aware there was a new/revised start-up procedure. Those who had a revised copy preferred to use their own marked up copy. This results in inconsistent start-up times, and possible inappropriate actions during unit start-ups. 1-39
Operating instructions in the form of unsigned, undated, hand written notes were found taped to the control panel. The operator bases identification of the author of these instructions on trying to recognize the handwriting. Operators have no way of knowing what new operating instructions have been issued unless they see a new piece of paper taped on the board or page through the log book assuming it was written. There is no way of ensuring that all operating personnel are being made aware of changes in critical operating parameters. 4. There is no documentation standard or control of Operating Procedures and Instructions. Procedures reviewed did not follow a common format, approval signatures, date issued, revision number etc. The new Unit Lay-up procedure given for review was in the form of an email. 5. Operators are not always informed of equipment, including chart recorders, being taken out of service for maintenance work. Unanticipated requests to have equipment taken out of service and clearances issued results in inefficiencies and delays for both maintenance and operators. The introduction of the Forced Outage Maintenance Work matrix posted in the control room has eased this situation. The matrix is an excellent communication tool. It keeps operating personnel informed of what work will be done under several forced outage scenarios. This puts the operators in a proactive mode regarding clearances and equipment shut down priorities. 6. Experienced and qualified Laboratory personnel control unit chemistry within EPRI guidelines. Operating personnel have limited knowledge of unit chemistry or the effects of operating outside EPRI guidelines on unit reliability. Operators are in a reactionary mode regarding chemistry excursions, depending on laboratory personnel for direction and corrective action. There are no feedwater or boiler pH indications in the control room. On a once-through, super critical unit using an oxygenated treatment (OT), maintaining a proper pH level is of the highest importance, as there is only a short time window of opportunity to react to low pH levels. Lack of pH readings in the control, coupled with the limited knowledge of operating, presents an increased risk to the reliability of the unit. 7. Operators spend approximately 25% of their time taking manual readings for staff personnel. A number of these reside in the computers in the control. The need for and the information from these readings has not been explained and or justified. This leads to resentment and frustration between operators and other staff members.
Conclusions
1. Big Sandy staff needs to be congratulated on the effort they have expended in successfully implementing the recommendations from Phase 2 Boiler Reliability Optimization Report. This effort shows the commitment to improve and sustain high performance. 2. There is a lack of management control in the operations area. Operators are not held accountable for their individual actions, nor the results of their actions on Unit efficiency or reliability.
1-40
3. Operators are not being given the tools or resources they need to operate the Unit in a consistently safe, reliable, and efficient manner. Operating procedures and instructions are written and distributed in a uncontrolled manner. Training of operators is inadequate and inconsistent. As a result, there are no primary standards for operation or control of critical Unit processes. 4. The loss of many older, more experienced operators and supervisors has diluted the knowledge base of the operations group. This has resulted in a wide range of operating philosophies among the Unit Operators. 5. Chemistry control is good due primarily to the efforts of the laboratory group personnel. Operations contribution to the success of this program is minimal due to their lack of knowledge of the chemical processes involved. 6. Control room instrumentation is an extension of the Unit Operators senses. The sense of urgency related to instrument repair and/or replacement needs to be raised to a new level. This should be accomplished in the near future if the practice of assigning only one operator to the control room is to be successful. 7. If Big Sandy is to successfully implement a Pro-Active Maintenance (PAM) program, it is essential that the Operations group plays an active role with its implementation. Without their full commitment, success will be difficult. Serious consideration should be given to the preceding recommendations before plant management attempts to implement such a program.
Recommendations
1. Continue with the efforts to implement the outstanding recommendations from Phase 1 (above table) especially with installation of a FEGT monitoring system. This will allow the operator to load the unit to its maximum without placing excessive thermal stresses on the super heater. 2. Develop a series of long term plant health indices for the boiler: thermal excursion, chemical and trip index. Thermal excursion index is the number of additional equivalent operating hours experienced by the most sensitive header in the boiler as a result of metal temperature excursions in excess of design temperature with a boiler pressure > 80% of normal o operating pressure. In addition, all excursions in excess of design + 100 F shall be included irrespective of boiler pressure. The Additional Operating hours = Constant (k) x ( T x t) Where: T = difference between the peak temperature reached in an excursion and the design operating metal temperature
t = total duration of the excursion in hours
1-41
Constant (k) = approximation, assuming a straight-line relationship between elevated temperatures and equivalent operating hours.
Thermal index = Additional Operating hours x Total hours in month divided of unit operating hours for the month. Chemical Excursion Index is a set of indices in which measurements taken on the plant of chemical conditions of steam, feedwater, etc are normalized to a base of one. The actual values are manipulated mathematically to give a result that is of the order of one. A result of 0.5 is excellent and greater than 0.9 is considered unacceptable. Results of between 0.6 and 0.9 indicate chemical condition under control and no risk to long term health. Results between 0.9 and I are acceptable for short duration only. Results greater than 1 require immediate attention. These results can be represented in a bar chart and displayed to the operator Chemical index = of all incidents of unit parameters > 0.9 for current month x Total hours in month divided by of operating hours for the current month Trip index is the accumulative total number of automatic and manual trips of the unit. This will give a feel for the number of stress cycles the plant has been subjected to no matter what was the cause Trip Index = of all trips for current month x Total hours in the month divided by sum of unit operating hours for the month
These indices can be developed for all units and used as a means to compare unit performance Note the numbers quoted above are guidelines, they should be adjusted to suite the particular unit circumstances and conditions
3. Analyze the effects of the wide range of conditions under which the Unit is operated. Select the Best Practice and develop a set of procedures and hold all Unit Operators to those standards. The start-up procedure should have target times for mile stone activities and check boxes for the operator to insert the actual time and reasons for any deviation from target. After the start-up these times can be analyzed for improvement opportunities. 4. Operations group should be involved in planning and scheduling of maintenance on any equipment that needs to be taken out of service. Operators should be informed on the status of any instrumentation that is out of service. They should know why, and the expected return to service date. 5. Develop a document control system and conduct periodic compliance audits to ensure the correct document/procedures are being used. This system should include all operating procedures and instructions. As a minimum, these should include signed approval levels, revision date, distribution list, standardized format, and collection and disposal of outdated documents. A list of typical headings found in a procedure can be found in Appendix 2. One complete set of operating procedures and a temporary operating instruction book should reside in the control room. All entries in this book should be signed and dated. Operators at the beginning of their shift should review this book for updated changes. 1-42
6. Provide training to all operating personnel on basic unit water and steam chemistry and the alarm and action level guidelines. 7. Provide pH indication for condensate, economizer inlet, and main steam in the control room. 8. Review the need and frequency for the manually recorded readings. Values, which reside in the computer, should be collected automatically by creating a specific computer log. The use of and value obtained from the readings collected manually should be feed back to the operators. 9. Develop and formalize an improvement program - root cause analysis process. Use the following as a guide. Obtain Management support such a program financial and human resources Select and nominate a champion owner of the process Review the industry and select a root cause analyses system using the following seven factors: Easy to use in the field by non-experts Effective in consistently identifying root causes Well documented Accompanied by effective user training Credible with the workforce Helpful in presenting results to management Designed to allow collection, comparison, and measurement of root cause trends. Train investigators/ Coaches representation from Operating and Maintenance Give awareness training to operators and craftsperson Compile an administrative procedure purpose, objective, scope responsibilities, investigation and reporting matrix etc Develop a database for causes and corrective actions 10. Provide training to ALL operating personnel on Unit chemistry guidelines and emergency responses. Of special importance is the relationship between specific conductivity, ammonia levels, and pH values in the condensate/feedwater systems. The ammonia feed-rate is controlled using the specific conductivity of the feedwater, which maintains the target pH values. The relationship between conductivity values and ammonia is a straight-line relationship, whereas the pH is logarithmic. Changes in conductivity are the operators first indication of approaching problems. Operators should also understand the difference between cation and specific conductivity, as well as the importance of sodium levels. Provide Ph reporting instrumentation to the control room. This should pH values on the condensate, feedwater, boiler, and main steam.
1-43
Appendix 1: Interview List

Name 1 2 3 4 5 6 7 8 9 10 11 12 D. Mell J. Burton E. Dillow A. Robinson S. Jenks J. Skaggs D. Pinson R. Lewis J. Adkins R. Peck M. Keene M. R. Mckenzie Designation Production Support Manager Region 3 Engineer Region 3 Engineer Production Services Leader Production Services Leader Lab Technician Unit Operator Unit Operator Unit Operator Unit Operator Equipment Operator Production Services Leader
1-44
Appendix 2: List of Typical headings that appear in a policy and or a procedure

1. Group name 2. Document type: policy, standard, guideline, procedure etc 3. Reference number 4. Revision number 5. Description of revisions made 6. Review date 7. Date authorized 8. Date when document came into effect 9. Compiled by 10. List of designations who have approved the document 11. Authorized by 12. Title 13. Background/preamble/introduction 14. Purpose 15. Scope of document 16. References 17. Definitions and or abbreviations 18. Responsibilities 19. Responsibility matrix 20. Requirements: step by step of what is required to be done 21. Work process flow and responsibility matrix 22. Distribution list 23. Appendices
1-45
2
BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 2 BOILER INSPECTION PLAN REVIEW AT GREAT RIVER ENERGYS COAL CREEK STATION UNIT 2
2.1 Executive Summary

Senior Management made the decision to change the boiler maintenance overhaul frequency from two to three years, and wanted assurance prior to the Spring 2001 outage that the existing boiler inspection plans were comprehensive and complete. Confidence in the newly adopted maintenance strategy is essential, as the next general overall outage after the Spring 2001 outage is in 2004. EPRI was requested to review the current Unit 2 outage inspection plan and recommend changes and/or additions, to provide this assurance. During the week of August 21, 2000, two EPRI representatives interviewed a number of Coal Creek Power Plant staff members from the operations, maintenance, engineering and management groups. The unit history, maintenance, operating practices, and the boiler inspection plan were reviewed and analyzed. The current boiler inspection plan is reactive, i.e. it focuses almost entirely on known problem area sootblower erosion - and does not cover all the components within the boiler in sufficient detail. There is a limited predictive element to the plan. It does not address any future or potential failure mechanisms, and is therefore not comprehensive and will not give the necessary assurance that all risk areas have been adequately covered. This report highlights a number of recommendations that would assist in developing a comprehensive and effective boiler inspection plan. These recommendations, together with comments/suggestions added to the Unit 2 Outage Inspection Plan given to the EPRI team on 21 August 2000, will provide a quality inspection plan. Annexure II An extract from another utilitys boiler inspection plan and is given in Annexure I to illustrate the detail that is required in a typical Boiler Inspection Plan.
2.2 Introduction
Faced with de-regulation and competition, Senior Management at Great River Energy Coal Creek Station, challenged the existing plant maintenance strategy and sought opportunities to improve plant availability and reliability. One of the changes made was to the overall unit 2-1
EPRI Licensed Material Boiler Reliability Optimization Project Phase 2 Boiler Inspection Plan Review at Great River Energys Coal Creek Station Unit 2
maintenance strategy, i.e. the frequency and duration of outages was changed from 2 to 3 year intervals and a shortened duration. This change would improve the overall unit availability. To improve unit reliability the focus shifted to understanding and eliminating the causes of boiler forced outages; the predominant cause being boiler tube leaks. To ensure no surprises a comprehensive boiler inspection plan needs to be developed and implemented during the next outage Spring 2001 . A boiler internal inspection is one of the most important inspections to be carried out during an outage. It must be done timeously, systematically, accurately and by experienced inspectors. The quality of this inspection will ensure that a reliable boiler will be put back into service. The only approach is do it right the first time all the time The selection and number of inspectors should not be based on quantity but rather on quality. The staff chosen must be committed to the assignment and have sufficient experience and background to identify any abnormalities. The staff should have a good knowledge of the gas flow patterns, temperature profiles, material composition and movement/expansion of the boiler. EPRI was requested to review the existing plans and suggest changes and or additions that will give this assurance, to provide assurance that an adequate and comprehensive plan was in place. During the week August 21, 2000 two EPRI representatives visited the plant, reviewed and discussed the current boiler inspection plan and tube leak history data with the boiler engineer, and interviewed a number of staff from the operating, maintenance and engineering groups.
2.3 Findings and Observations

The findings below are based on the Unit 2 outage inspection plan as presented during the review (see Annexure II) and on the personal interviews. 2.3.1 The inspection plan is divided into five sections: burner front, waterwall, Superheater and reheater, back -pass and boiler external. Although the boiler as a whole is covered, the specific inspection details of the areas within each section are not covered sufficiently e.g. superheater, DMW etc. 2.3.2 Each section has been assigned to a responsible person who will manage the inspection and data collection. The boiler engineer has overall responsibility to ensure adequacy of inspections, data collection and coordination between the various sections. 2.3.3 The plan is reactive and focuses almost entirely on inspecting known problem areas e.g. identifying and correcting soot blower erosion damage.
2.3.4 Areas within each of the five sections are not prioritized and no go no go criteria given e.g. minimum tube wall thickness and crack/groove size/length and shape.
2-2
2.3.5 Documentation of the pre-outage external inspection and walk-down of the boiler is comprehensive 2.3.6 Infrared survey of the boiler casing and valves is not included in the pre-outage inspection. 2.3.7 No provision is made to complete a dirty boiler inspection. 2.3.8 The present sootblower maintenance strategy is not in line with current or future operating mode of the unit. Sootblower preventive maintenance has not been adequate, as shown by the number of corrective maintenance activities that have needed to be performed. The top five SB problem areas are: Feed tube wear, Sprocket and chain wear, Bearing failures, Packing failure and Carriage wear 2.3.9 Thermal drains are being replaced with steam traps without a thorough root causes analysis being done to ensure that this is the correct action to take. There is a limited history regarding failures and repairs. 2.3.10 Result trending and predictions cannot be adequately completed, as there is no central database to record and keep boiler test and inspection data and information. Boiler inspection reports that identify actions taken, future actions needed, or lessons learned, are inadequate. All history data on the boilers tube leaks is owned and maintained by the Boiler Engineer and is in hard copy format. 2.3.11 Root Cause Analysis is not formally done. The Boiler Engineer maintains tube leak reports, using CE Availability data sheets. Not all tube leaks are investigated. History summary is available from 1989. The summary includes date, suspected failure cause, corrective action taken, and recommended outage follow up actions. 2.3.12 Startup and shutdown procedures are combined. They are too general and give little guidance as far as criteria to be met metal temperatures have been exceeded in the past because of this. No precautionary measures are given to the operator, e.g. when firing the boiler at over pressure.
2.4 Recommendations
2.4.1 A more detailed proactive inspection plan is needed to support a three-year outage cycle plan, while meeting plant goals. The boiler components are time and temperature dependent. Over-stressing these components will accelerate aging. By adopting a proactive stance, knowing the remaining life of these components, replacement plans can be developed and implemented before aging/failure affects unit reliability. 2.4.2 A schedule should be developed for all inspection areas in the boiler. Prioritization should be based on potential findings that would affect outage duration, safety, budget restraints, and unit reliability. All areas that have a high priority should be completed by the end of the first week. This requires a pre-outage study of the duration and cost of inspection techniques to be used. It should include component disassembly, scaffolding, 2-3
insulation removal, surface preparation, ventilation, lighting, and environmental compliance. 2.4.3 Criteria need to be developed for the inspection of each area, based on historical data, possible damage mechanisms, typical locations, inspection techniques, required preparation and support, and estimated inspection times. It is important to establish a definition of what are permissible flaws, and what action should be taken regarding a repair/replace decision when an unacceptable flaw is found. Non-critical flaws should be duly noted, entered into the database, and trended over at least two outage cycles. 2.4.4 All data gathered during the inspection, as well as all repair/replace decisions, should be entered into a database maintained by the Boiler Engineer. This database should include as found and as left conditions, repairs made, and recommendations for future outages. Examples of both inspection and repair record formats are included in the annexure I 2.4.5 Expand inspections of the superheater and reheater sections beyond soot blower damage, broken clips, rubbing damage, and ash erosion to include remaining life assessments. UT measurements should be taken to determine the oxide thickness on the superheater. These values can be used to determine the high temperature areas in the SH and will validate the temperature profile as measured by the installed tube metal temperature instrumentation. Tube samples should be taken from hot areas of the SH for metallurgical analysis to determine remaining life. These analyses may have an impact on operating practices. 2.4.6 Horizontal and pendant SH sections are difficult to drain completely, if at all. As a result, they often suffer internal damage due to pitting from condensation. Tube samples should be taken and analyzed for this. 2.4.7 Header base line data, internal and external, needs to be obtained, as it would indicate the effect of past operating modes, and provide information to determine future operating modes. Superheat and reheat headers have never been inspected. Fiber optics can be used to determine if any erosion or cracks have developed. 2.4.8 The current inspection plan calls for inspection of the remaining wall blower refractory tabs. Some of these tabs have been removed, as they have been a source of tube leaks. The remaining tabs should be removed, as they are neither used nor needed and are potential areas for tube leaks. 2.4.9 High stress raisers are potential sources for tube leaks. Inspect and itemize all internal boiler tube attachments for cracks, erosion or damage. 2.4.10 An infrared survey of the boiler casing, valves, and other potential sources of heat loss, should be performed as part of a pre-outage inspection. 2.4.11 A dirty boiler inspection should be included in the boiler inspection plan. Many will argue that this inspection is uneconomical. However, there is much to be gained by 2-4
observation and correct interpretation of what is observed. An experienced boiler engineer can benefit enormously and gather insight of what is going on inside the furnace from this inspection. Visually examining ash deposits color, size, build up and profiles - and tube surface conditions, will highlight areas of erosion and high or low gas flow regions. 2.4.12 Although no Dissimilar Metal Weld (DMW) failures have been reported, welds should be inspected visually at every outage using UT. DMW tube samples should be taken and analyzed for remaining life. 2.4.13 The sootblowing systems design should be reviewed. This should include an evaluation of reliability, availability, maintenance and operating issues. This review should also include an evaluation of the effectiveness of steam traps in eliminating water in the system. From this design review an optimized sootblower operation and maintenance strategy can be developed, implemented, and issued regularly via the Maximo system. 2.4.14 Perform a functional check on all sootblowers. Each sootblower should be operated in turn and checked visually from the inside of the furnace for correct nozzle position, blower insertion length and rotation.
2.4.15 Full implementation of the Maximo system should include the following. All identified boiler inspection tasks should be kept in the Maximo system as preventive maintenance (PM). These PM should include, but not be limited to, duration, location, man-hours, equipment needed, and collaboration needed with other groups. 2.4.16 Implement EPRIs Boiler Maintenance Workstation (BMW). BMW is a Windows-based software program used to track and trend boiler tube failure data and information tube failures, correction and prevention, and control actions. It consists of two modules Boiler Works and Tube Condition. The boiler works module is a database and graphics program designed to track maintenance failure data/information failure location and mechanism, repair costs and methods, etc. This collated information is used to focus attention on specific activities, for both planned and forced outages, to prevent future repeat failures. The Tube Condition module stores and analyzes data collected from various inspections, e.g. tube wall thickness, oxide thickness data. It uses two or more data sets to determine wastage rates. These wastage rates are used to determine remaining tube life and predict tube wall thickness. This helps plant personnel plan future boiler tube inspections, maintenance activities and tube replacements. 2.4.17 Following every planned outage, a detailed report should be issued. This report should identify inspection results, repairs and modifications, lessons learned, and activities that need to be planned for the next outage. 2.4.18 All NDE data on boiler pressure parts should be trended and used for failure prediction and long term planning. 2-5
2.4.19 Senior Management should support a formalized Boiler Tube Failure Reduction team. The team, headed by the Boiler Engineer, should have members from the operations, maintenance and engineering groups. The team should follow a recognized root cause analysis process when analyzing all boiler tube leaks. Records should be kept in a central database and failure statistics and tube condition minimum wall thickness, oxide thickness and remaining life results, etc., should be tracked and trended. 2.4.20 Separate Unit Start-up and Shut Down procedures should be written, as these are two distinctly different activities each with specific criteria and operator requirements. Each procedure should identify the precautions to be taken before or during each step. This should be followed by a step by step sequence of actions with criteria to be met, e.g. metal temperature rates of change, pressure ramp rate, differential temperatures, etc. Each action should have a check box and a remark column where the operator can check off the action and explain/give reasons for any deviation from the specified criteria.
2-6
ANNEXURE I: Extract from a Boiler Inspection Report

This report covers all boiler internal inspections, including tubes, burner tips, and dead air spaces performed by Plant personnel. Inspections of the mud drums, the main steam drum, and penthouse are included.
1st digit (location):
1 2 3 4 5 1 2 3 4 5
Firebox Upper Arch Down draft Dead-air Spaces, Drums Penthouse Economizer Tubes Waterwall Circuit Tubes Steam Cooled Tubes Reheater Circuit Tubes Superheater Circuit Tubes
2nd digit (tube type):
Inspection Areas Most of these are known areas where failures have occurred or repairs have been made during past Periodic Outages and Forced Outages. DM designates D-meter inspections for tube thickness.
1200 Waterwall Cleaning Devices, operational condition

This inspection involves both pre-outage and outage responsibilities. Operations must check blower and lance operating conditions prior to Unit shutdown. This requires that all water-lances be tested for proper speed settings and visually verified for operating effectiveness, noting condition of the booster pumps and blowing pressures. All "partial-arc" wall-blowers must be verified for proper operation. Note any existing conditions that may effect tube life and closely inspect theses areas (see item Nos. 1209 and 1210). A furnace side inspection is to be performed by the Boiler Inspection Team from a furnace scaffold, including nozzle head conditions and insertion depth settings on all wall blowers. Inspection results All wall blowers were inserted for inspection and checked for appropriate depth setting, yielding 20 required adjustments (vs. 43 total in 1996). Lists of all damage and needed adjustments were given to the Unit Planner. Depth setting adjustments were made during the outage, while the lance changes were scheduled for after the outage due to parts availability. Over half of the original water-lances (WL) were found with damaged lance tips (cracking or holes).. Lists of all damage and needed adjustments were given to the No. 7 Unit Planner 2-7
Recommendations for 2004: Several of the lances were binding on start-up, and at least three of the lance tubes will need replacement. Air cooling to the control boxes will be added to keep the box temperatures below 0 122 F. Access to WL nozzles inside the windbox will need to be made during a furnace cleaning outage. For 2004, wall blower and WL repairs and adjustments are expected to be minimal. 1501 Superheat Division Panels, attachment problems Look for misalign tubes and possible resulting damage. Look for failed "wrapper tube" ties. Look for abrasion damage from the attachment ears rubbing into adjacent tubes. Recommend repairs.
division tub
attachment rub damage Attachment ear area(s)
Figure No.
Inspection results:
There were 10 locations found and repaired (vs. 31 found and 23 completed in 1996). Of these, two were failed attachments, and 8 were from attachment rubbing damage. Some of the rub damage locations were severe, causing abrasion damage in excess of 1/2 wall thickness (see Fig. 6). The recommended repair for the ear rubbing was to remove that portion of the attachments. Recommendations for 2004: Continue inspections for failed attachments and attachment ear rubbing. Expect nearly the same amount of work as in 2001.
2-8
ANNEXURE II: Unit 2 Outage Inspection Plan as of August 21, 2000

This plan contains numerous visual checks and does not give guidance on what to look for, how to look for it, and what to do with the data/information obtained? Some guidance needs to be documented for each team on what to do if cracks or dents are identified, e.g. measure wall thickness, cut out and insert a Dutchman, weld over lay, etc. Comments/additions have been made in each section below. For ease of identification they have been underlined. 1. Burner Front Team Member Will perform burner front inspection and coordinate all repairs of the burner from inside and outside the boiler. The following inspections will be performed: 1.1 Prior to the outage (within 2 weeks of the start of the outage) Conduct and external walk down of the burner fronts from 5th floor to 10th. The items to check shall include: Look for any coal or ash leaks around the coal piping and nozzles. Check the condition of all coal piping spring hangers. A. Record the hot position of the hangers. B. Check if any hangers are bottomed out. C. Visually inspect the hanger rods and attachment pins. D. Inspect the hanger lugs on the support steel and on the coal piping. Note if any hagen drives are disconnected. Inspect the air supply and signal lines to the hagen drives. Check for any broken or disconnected lines and to see if any lines have broken loose from their supports. Check the condition of the multi tube lines from the burner front to the LIE cabinets. Check with Operations and the E&I techs to see if any of the tilt drives are short stroked. Visually inspect the external tilt linkage for any members that are bent, broken or disconnected. Note from the Honeywell any flame scanner with low intensity readings. These scanners will be inspected in detail after the unit is off line. Check with the E&I techs to find out if there are any flame scanners that they have had trouble inserting or removing. Also check to see if there are any work orders on the flame scanners. Check for any oil leaks around the burner front oil guns and supply piping. 2-9
Inspect all of the oil guns and spark igniters and oil gun air lines. Visually inspect the new SAS air ducts on 9th and 10th floor. A. Visually inspect the SAS duct hanger rods and attachment pins. B. Check for any interference's between the SAS registers and the structural steel. C. Inspect the windbox hanger rods on 9th floor.
Visually inspect the fabric expansion joints. Inspect all external pipe work of the various systems for steam and water leaks, sagging, abnormal internal noise, signs of shock, pipe guides, hangar supports free movement of constant and spring supports, missing bolts and nuts and dust covers, signs of corrosion and integrity of the supports. Inspect boiler skin casing cladding for damage, missing or corroded sheets. Inspect all ducts for air and gas leaks, abnormal movement or vibration, defective damper linkages or fouling with moving parts. Inspect the boiler main and sub structure for deformation, missing bolts and nuts, flame cutting damage, severe corrosion, signs of unauthorized welding on main load bearing beams, beams/buckstays for sagging and or distortion.
1.2 At the start of the outage (During the installation of the boiler scaffolding) Inspect the windbox and windbox expansion joints. Stroke all of the windbox dampers manually, checking for any that are binding and that the connector link between the damper blades is intact. Remove all the flame scanners and insert an old one in each guide tube noting any problems with inserting the flame scanner. Manually stroke the tilts on the over and underfire air compartments and the underfire air compartment dampers. Stroke all aux air and SAS yaws. Inspect the inside of the SAS ducts. Check condition of the fabric expansion joints and air dampers.
1.3 Burner Front inspection (Upon completion of the boiler scaffolding) Insert all oil guns and check the dimension from the face of the diffuser to the oil gun tip. See the CE drawing for details. Insert all of the ignitors and check the insertion depth and position of the ignitor tip. Inspect the oil nozzles looking for the following items.
2-10
A. Condition of the tip. Look for distortion and burning of the tip. (Oil guns should be tested in a test rig to ensure proper oil spray angles. This help determine the performance of each gun) B. Check condition of the oil gun and the flame scanner guide tubes. C. Look for broken or bent tilt linkage. Note if any lever arms are working off of the stationary pivot pin. D. Note if any of the diffuser discs are missing on the elevation 5/6 and 7/8 oil nozzles. Inspect the coal nozzles looking for the following items. A. Condition of the two piece tip. Determine which tips will be repaired or replaced. B. Check for thinning of the removable transition piece on the front of the coal nozzle body. C. Check for missing or loose ceramic tile in the coal nozzle body and gate body. D. Look for broken or bent tilt linkage. Note if any lever arms are working off of the stationary pivot pin. Inspect the aux air and SAS nozzles looking for the following items. A. Condition of the tip. Look for distortion and burning of the tip B. Check condition of the yaw linkage, looking for any bent or broken pieces. C. Check condition of the scanner guide tubes in the aux air nozzles. D. Look for broken or bent tilt linkage. Note if any lever arms are working off of the stationary pivot pin. Use the form shown as Attachment 1 to record all of the repairs that the contractor shall perform. All repairs such as nozzle replacement, linkage repairs and nozzle ceramic repairs should be detailed on the form.
2-11
2. Waterwall Team Member Will perform inspections and coordinate repairs of the waterwalls, center wall and slopes. The following inspections will be performed. 2.1 Prior to the outage (within 2 weeks of start of the outage) Review the tube leak records for the last three years and identify any areas that need further repairs. This should include any areas where window were installed or any areas where tubes were just welded shut.
2.2 Prior to the installation of the boiler scaffolding (after the throat scaffolding is installed) Perform a general visual inspection of the slope before the scaffold is installed. Check for large dents in the slope, dented tubes and worn areas. The wear areas should include the following. A. The slope along the side walls under corners 2 and 7. B. Both the front and rear slope along both sides of the center wall. C. The side walls at the throat opening. D. The center wall where it penetrates the slope. 2.3 Upon completion of the boiler scaffold Clean the waterwalls and perform an UT inspection of the waterwalls between the upper burner front and the SAS registers. See the attached drawing, Attachment 2 for details of the area to be inspected. Inspect all of the wall blower openings. The following items should be checked. A. Visually inspect for any worn or eroded tubes. Mark any tubes that may need a follow up UT inspection. Mark any tubes that require replacement. (Criteria need to be developed 70 % wall thickness is commonly used) B. Check the condition of the refractory and mark any openings that require refractory repairs. A list of the openings requiring refractory repairs should be forwarded to the External Boiler Team Member. C. Visually inspect for any cracking occurring at the tabs welded around the opening. (Remove unnecessary tabs by carefully grinding) Inspect the crotch tubes at the top and bottom of the burner front and SAS register openings.
2-12
Perform an UT inspection of the boiler nose. (Go and no go criteria need to be provided). See the attached drawing, Attachment 2 for details of the area to be inspected. Inspect the bends in the waterwall tubing below the nose. Perform a random UT inspection of the tube bends. (record the measurement position and tube location as a reference) Inspect the bends at the bottom of the radiant reheat section. Perform a random UT inspection of the bends. Also check for any tubes out of alignment or bent tubes. Visually inspect the joints between the center wall panels. Check for rubbing damage on the tubes at the panel joints. Perform a visual and UT inspection of the slope looking at the following areas. A. The front and rear slope along both sides of the center wall. The area to be inspected should be the full length of the slope for a 20 tube wide panel. B. The slope along the side walls under corners 2 and 7. The area to be inspected should be the full length of the slope for a 20 tube wide panel. C. The slope just below the upper bends directly under burner fronts 2, 4, 5 and 7.
Inspect the front wall between the burner front and radiant reheat for any tubes that are smashed or dented. Replace the lower radiant reheat bend and section of tubing in furnace A that previously failed. Remove a two tube samples from the waterwalls. The sample should be removed from the front or rear wall on ninth floor between the burner fronts. (consider taking more samples to get a representative set)
2.4 Upon completion of the bottom ash scaffolding Visually inspect for any smashed or dented tubes at the throat opening. Check the side wall tubes at the throat opening. Check the horizontal tubes at the center joint of the side wall. Inspect the lower side of the throat tubes.
2-13
3. Superheat and Reheat Team Member Will perform inspections and coordinate repairs of the superheat and reheat pendants. This work will be completed on night shift. The following inspections will be performed. 3.1 Prior to the outage (within 2 weeks of start of the outage) Review the tube leak records for the last three years and identify any areas that need further repairs. This should include any areas where windows were installed or any areas where tubes were just welded shut.
3.2 During the outage Visually inspect all of the sootblower lanes on the leading edge of the superheat pendants. This work will be performed using the boiler scaffolding. Visually inspect the fluid cooled spacer tube that runs through the front of the superheat pendant. Check for broken clips or rubbing damage. Scaffold the sootblower lanes in the center of the superheat pendants and visually inspect all sootblower lanes. A. Special attention should be given to inspecting the double offset bends in the crossover area. B. Visually inspect the fluid cooled spacer tube that runs along the center wall. Check for rubbing damage. C. Check for any broken or missing flex connectors in the pendants D. Inspect the bent tubes located on the A side of the furnace. Determine what repairs will be required to straighten the tubes. Scaffold the sootblower lanes between the rear waterwall hanger tubes and the reheat pendants. The following inspections should be performed. A. Perform a UT inspection of the trailing edge of the rear waterwall hanger tubes at each sootblower lane. B. Inspect the rear waterwall hanger tubes where they penetrate the slope. Check for ash erosion on the trailing edge of the tube from ash sliding down the slope and check the leading edge for flyash erosion. C. Visually inspect the leading edge of the reheat pendants and the trailing edge of the superheat pendants at each sootblower lane, checking for sootblower erosion. D. Visually inspect the fluid cooled spacer tubes located in the center of the boiler on the back side of the superheat pendants. Check for rubbing damage between the spacer tube and the rear waterwall hanger tubes and superheat pendants. 2-14
E. Inspect the fluid cooled spacer tube that runs through the front of the reheat pendants. Check for broken clips or rubbing damage. F. Inspect the front lower bends of the reheat pendants. Check for sootblower erosion on the inside and outside of the bends and for any missing or damaged tube shields. Perform oxide thickness measurement on the hottest across the superheater to determine the temperature profile. Remove samples for metallurgical analysis creep damage and remaining life. Inspect DMW for cracks. If cracks are found cut out a section and replace with a Dutchman. If no cracks found cut out a section from the hottest area for metallurgical analysis. Scaffold the sootblower lane on the back side of the reheat pendants. The followings inspections should be performed. A. Visually inspect the sootblower lane looking for sootblower erosion. Special attention should be directed to the double offset bends in the crossover and to the bottom side of the horizontal crossover tubes. B. Visually inspect the spacer strap in the reheat finishing section, looking for broken or missing U straps. C. Check for any broken or missing flex connectors D. Inspect the tube shields at the bottom sootblower on the trailing edge of the reheat pendants. Check for missing or damaged shields. Inspect all sootblower openings along the side wall. Check for tube erosion and for cracking along the tabs. Also visually check the condition of the lance nozzles. Inspect the roof and slope tubes, checking for bent or warped tubes. When performing pendant inspections special attention should be given to the first five pendants from each side wall. Historically the first five pendants have suffered more severe sootblower erosion.
2-15
4. Back-pass Team Member Will perform inspections and coordinate repairs of the back-pass from the furnace screen tubes to the economizer. The following inspections will be performed. 4.1 Prior to the outage (within 2 weeks of start of the outage) Review the tube leak records for the last three years and identify any areas that need further repairs. This should include any areas where windows were installed or any areas where tubes were just welded shut.
4.2 During the outage Perform an UT inspection of the inner radius of the lower bend of the furnace screen tubes just above the refractory dam. Visually inspect the spacer straps in the front of the superheat finishing section looking for missing or broken U clips. Also inspect the spacer straps that wrap around each pendant. Check for any broken or cracked welds on the front of the wrap around straps. 30 new straps have been ordered based on a previous inspection. Inspect for sootblower erosion along the lower sootblower lane on the front of the superheat finishing section and the rear of the furnace screen tubes. Scaffold the sootblower lanes on the front side of the superheat finishing section. Visually inspect the condition of the tube shields on the trailing edge of the furnace screen tubes. Visually inspect the condition of the superheat pendants checking for sootblower erosion and broken or missing flex connectors. Visually inspect the rear extended backpass floor tube located directly in front of the superheat backpass front tubes. Check for sootblower erosion on the floor tube that occurs in the spaces between the backpass front tubes. Scaffold the upper sootblower lane on the back side of the superheat finishing section. Check for sootblower erosion on the trailing edge of the superheat pendants and on the leading edge of the backpass front wall tubes. Visually inspect the spacer straps located on the backpass front wall tubes. Check for any U clips that are broken or have burned off. Also check for any damaged tube shields in this area. The sootblower lane for retracts 19 and 48 which is located at the rear of the front wall tubes will require several inspections. missing or superheat
A. The trailing edge of the superheat front wall tubes and the top side of the horizontal superheat tubes should have a UT inspection performed on them along this sootblower lane. B. The front side of the superheat low temperature pendants should be visually inspected for sootblower erosion and for missing or damaged tube shields. Special attention should be given to the lower bends on the low temp pendant section. 2-16
C. There was a previous tube failure on the West Side on the top of the superheat horizontal section. Tubes number 2 and 3 from the West Side wall should be replaced. Visually inspect the top of the horizontal superheat section. Check for any tubes that have broken loose from the hanger tubes and check the bends on each end for any bend that may be displaced and rubbing on an adjacent tube row. Also check the condition of the refractory and flyash screens located at the rear bends of the horizontal superheat. Inspect the rear bends for any indication of flyash erosion. Inspect the area between the two horizontal superheat sections. Check for any tubes that have broken loose from the hanger tubes. Visually inspect the spacer straps located on the front hanger tubes. Check for any broken or missing U clips and to ensure that the straps are still raised up off of the horizontal tubes in the center half of the boiler. The spacer straps were raised because they were deflecting the sootblower steam down against the horizontal tubes. Special attention should be given to inspecting the sootblower lanes for retracts 10 and 39. This area has been a very high wear area. A. Check the condition of the tube shields located on the lower side of the upper horizontal section. The condition of these tubes on the lower side of the upper horizontal section should be checked at least 5 tubes deep up into the bundle. B. Check the condition of the deflectors installed on the hanger tubes. The horizontal tubes should be checked for sootblower erosion where they are attached to the hanger tube. C. Visually inspect the condition of the pad welding on the top side of the lower horizontal section. A UT inspection of these tubes should be performed on any tubes that are not covered with tube shields. Visually inspect the area on the topside of the economizer. Check for any broken economizer hangers or bent tubes. Perform an UT inspection on the topside of the bends located at the superheat horizontal inlet header. These tubes have shown signs of flyash erosion on the outer radius of the bends. Inspect the area located between the economizer sections. Check for any broken hangers or bent tubes. Check the condition of the offset bends located along the superheat backpass front inlet header located along the front wall. These offset bends have been subjected to some flyash erosion. Check the condition of the refractory dam at the furnace screen tubes. Check for any broken or missing pieces. When performing pendant inspections special attention should be given to the first five pendants from each side wall. Historically the first five pendants have suffered more severe sootblower erosion.
2-17
5. External Boiler Team Member Will perform inspections and coordinate the repairs external to the boiler. The following inspections will be performed. 5.1 Prior to the outage (within 2 weeks of the start of the outage) Conduct an external walk down of the boiler from 21st to 2 1/2 floor. The items to check shall include: Condition of the buckstays. Checking for any binding or bent buckstays. Look for any missing or loose pins in the connector links for the boiler trusses. Check for any external steam leaks at the sootblowers, sootblower piping and boiler vent and drain valves. Note any lagging or flashing that may require repair. Check the Maximo system for workorders for any boiler vent and drain valves that may require a code repair. Review all boiler valves that are going to be replaced to determine if the repairs should be code repairs. 5.2 During the outage A visual inspection should be performed in the front and rear lower dead air spaces. The following items should be checked. A. Inspect all of the hanger rods checking for broken or missing pins. Also check for cracking on the hanger rod lugs. B. Check for any missing or cracked tube clips on the horizontal buckstays. C. Perform a dye penetrant inspection where the side wall capture angle irons are welded to the back side of the slope tubes. D. Visually inspect the scallop bars located at the bottom of the side walls. Check for any cracking where the scallop bar was repaired and for any cracking were the scallop bar is welded to the tube. E. Check for any cracking in the tube membrane located towards the inside of the side wall scallop bars. F. In the rear dead air space, check the condition of the centerwall tube nipple that was cut off and plugged. G. Check for any bent or cracked buckstays and structural steel. H. Visually inspect for any tube membrane cracking at the throat bends and where the slope is welded to the side walls. 2-18
Visually inspect both upper dead air spaces. Check for any membrane or skin casing leaks and check the condition of the structural steel. After the boiler has cooled, walk down the entire external boiler checking for any bent or warped buckstays. Inspect the upper boiler drum, check the following items. A. The turbo separators, checking for alignment, loose hold down clamps and the condition of the chevrons. B. Missing or loose screens at the downcommer nozzles. C. Loose or missing feed pipe hold down clamps. D. The condition of the secondary separators. E. Plugged lines for the boiler level devices.
Inspect the lower boiler drums, check the following items. A. Check for any missing orifices or orifice clamps. B. Check the condition of the screens. Look for any loose or missing screen bolts. C. Visually inspect the drum crossover links.
Remove the hand hole covers on the superheat desuperheaters and use the borescope to check the condition of the liner, set screws and the nozzle. Visually inspect all of the boiler access and inspection doors recording any that require refractory or other repairs. Perform an internal inspection of the penthouse checking for the following items. A. Check the condition of all the hanger rods. Check for broken rods or missing pins. B. Look for any tubes that exhibit signs of overheating. The overheated tubes will have a dark discoloration and may have some scale exfoliation. C. Look for any cracks and flyash leaks in the skin casing. Also check for any flyash leaks at the crown seals. Special attention should be given the skin casing at the center of the boiler. The boiler has a history of skin casing cracks along the center of the boiler D. Check the condition of the insulation at the superheater outlet header and links. E. Visually inspect all of the trapeze hangers located on the connector links.
2-19
F. Visually examine all of the headers. Look for any signs of cracking at the nipples or ligaments.(Suggest you use some NDE method than just looking dye penetrant) Perform and inspection on top of the penthouse. Check the condition of the upper hanger rod attachment looking for any missing pins or cracked lugs.
2-20
3
BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1 AT DETROIT EDISONS ST CLAIR PLANT UNIT 7 AMERICAN ELECTRIC POWERS BIG SANDY PLANT UNIT 2
Executive Summary
In an effort to improve boiler reliability at the Detroit Edison St. Clair Plant, EPRI was contracted to identify technical and programmatic opportunities for improving Unit 7 boiler reliability. The scope of work was limited to a portion of Phase 1 of EPRIs Boiler Reliability Optimization project, namely boiler tubes and headers. The study was requested by the then Production Manager Phil Thigpen from St Clair Plant. A team from EPRI spent two weeks on site reviewing plant record, boiler inspection plans, procedures etc and conducted interviews with key operations, maintenance and engineering staff. A list of those interviewed can be found in appendix 1. The report focuses on two areas - the programmatic and data collection and tube failures and their causes and highlights a number of opportunities for improvement with respect to managing the short and long term health of the boiler. From the review of the data/information obtained and the interviews revealed that the current maintenance strategy, of a periodic every three years, is in line with other utilities however the duration is out of step with best practice in the industry. Utilities with comparable units have 4 to 5 week outages with a Forced Outage Rate of between 4 and 6 percent. . Unit # 7 has a comprehensive boiler tube inspection plan. Tube leaks, their causes and action plans to prevent recurrence are recorded and well managed as evidenced by the reduction in the number of tube leaks over the past five years. The inspection plan is reactive as it only includes known boiler tube problem areas. Inspection of the headers is limited to the superheater and reheater outlet header. This is of concern as there is little to no information is available about the condition of the remaining headers within the boiler. The recent down sizing and re-organization has resulted in some uncertainty regarding the roles and responsibilities of the newly formed service groups. Key staff have been moved to perform other project activities, thus losing the advantage gained from past inspections. Organizational measures and measurements were also not well understood. To improve and sustain a high boiler reliability/availability and to minimize the future risk of down time and subsequent production loss it is recommended that Detroit Edison St Clair Plant 3-1
EPRI Licensed Material Boiler Reliability Optimization Project Phase 1 at Detroit Edisons St Clair Plant Unit 7 American Electric Powers Big Sandy Plant Unit 2
Invest into a comprehensive Boiler Reliability Optimization program to develop a detailed Boiler Failure Defense Plan. This plan would assist in the development of a specific Boiler Test and Inspection Plan that would cover all the major components within the boiler envelop. This plan can then be integrated into an expanded 15 years plus Long term Reliability Plan. This can be accomplished through EPRIs Boiler Reliability Optimization Program Review the basis of their current outage management philosophy and approach. The current nine week outage on unit # 7 is out of step with other USA and international utilities even with the current turbine problems being experienced. Huge operating and maintenance cost saving and increased revenue can be made by reducing outage times by bring them inline with best practice. Implement the recommendations identified in the various section of this report.
Participating in Phase II of EPRIs Boiler Reliability Optimization Program would not only enhance the efforts being put into the program but will give Senior Management the assurance that the goals being set will be met. Membership details and costs related to a Phase II can be obtained from the author. Armed with the information in this report and available resources, St Clair plant can become the Best Practice plant within the Detriot Edison organization for which it has been striving to become. Given time, management commitment and coaching in EPRIs Boiler Reliability Optimization Program, the plant team has the vision and capability, as evidence in the Boiler Tube Inspection Reports, to successfully lead the plant into full implementation of a Boiler Reliability Optimization Project.
3-2
3.1 Introduction
Detriot Edison St Clair Unit 7 had a history of tube failures causing considerable down time with corresponding loss of production. Phil Thigpen, the St. Clair Production Manager, requested that EPRI be contracted to review their boiler technical plans and identify opportunities for improvement. The scope of work was limited to the high pressure, high temperature boiler components tubes and headers. Four EPRI team members - Dave McGhee, Pat Abbott, Jack MacElroy, and Jim Cossey spent varying amounts of time on site studying documentation and conducting interviews. Interviews were conducted with both Plant and Support staff responsible for activities on the boiler. These interviews were conducted during the weeks of May 15 and August 29. 2000. The names of those interviewed are listed in the appendix 1. All relevant and available data and information on Boiler 7 was reviewed. In addition, data and information relating to the overall management of the boiler, from an operating, maintenance and engineering point of view was reviewed. This enable the team to gain an insight into the overall approaches taken to manage the technical aspects of the boiler.
3.2 Findings and Observations

Based on the scope of work, the following represents the findings and observations in the two areas, viz. programmatic/data sources, and failures and root causes. 3.2.1 Programmatic and Data Sources Assessment A maintenance process assessment was conducted as part of the work performed in this project. The purpose is to understand how the management processes and technologies used in the plant affect or influence boiler operation and maintenance and therefore the long-term health. Data was collected and evaluated and benchmarked against known best practices. The assessment team spent a week on site reviewing this data and interviewing key site and support staff. From the data analyzed, together with the information obtained from the interviews a spider chart was drawn.
3-3
Score
Score
Maintenance Strategy 8 Utilization Work Identification Skills/Experience Profile Technical Resource Development Metrics Communication Roles & Responsibilities Performance Monitoring 6 4 2 0 Work Control Work Closeout Documentation Project Management Change Control Operating Practices Boiler Inspection Plan Technical & Action Plans
W orld Cla ss
Figure 3-1 Spider Chart
3-4
EPRI Licensed Material Boiler Reliability Optimization Project Phase 1 at Detroit Edisons St Clair Plant Unit 7 American Electric Powers Big Sandy Plant Unit 2 Attribute Boiler Maintenance Strategy Key Findings Conscience change from 2 to 3 years Inspection and repair plan is aligned with outage frequency Work Process 50% of the backlog are preventive maintenance activities. as found and as left conditions were not being documented Project/Outage Management Capital projects are detailed and well planned Performance/Reliability of a unit returning from a periodic outage does not meet expectations i.e. not optimal Unreliability and unavailability occurrences are not investigated to determine their cause and root cause. Operating Practices Critical boiler metal temperatures are not monitored or trended Metal temperature excursions are not analyzed or recorded No objective evidence to indicate that boiler start-up or shut down procedures are followed Sootblowing procedure and criteria are not being adhered to. Boiler Test and Inspection Plan Well documented boiler tube inspection and repair plan No detailed condition assessment of all boiler headers only superheater and reheater outlet are inspected Metrics and targets Overall plant performance targets exists Boiler EFOR is not measured Boiler tube leak target is place Roles and responsibilities Many of the staff interviewed were unsure of their roles and responsibilities and that of the support group. This was illustrated by the performance of unit 4 after its periodic outage
Performance Monitoring
Superheater metal temperature excursions are not investigated or recorded No objective evidence that excess oxygen is controlled or monitored
Skills/experience profile
Experienced people are being lost due down sizing and retirements. This affects the transfer of technical know how to new incumbents - operators
3-5
3.2.1.1 Maintenance Strategy The starting point for developing a maintenance strategy is to first determine a maintenance objective. An objective should be quantified in terms of availability and performance. These requirements should be determined jointly between the Maintenance and Operating and or the Production Departments. Maintenance Management is then in a position to determine the best mix of maintenance techniques to use, resources requirements and the costs thereof. A Maintenance strategy is then developed from these agreed objectives. It defines the process for identifying and achieving sound operational objectives, selecting the maintenance approaches, monitoring performance and providing effective management control. The strategy therefore consists of a management and technical strategies. The Management strategy defines how the business management skills are used to integrate people, policies, equipment and practices to identify improvement opportunities. The Technical strategy states the what, who and how the technical knowledge and experience are used to identify and implement the best proactive maintenance, repair, service and replacement of all equipment in line with the Plants business performance objectives. The technical strategy therefore consists of a combination of corrective, preventive, predictive and proactive maintenance activities where: Corrective Fix it when it breaks Preventive Fix it before it breaks Predictive Fix it when it starts to break Proactive Fix it before it starts to fail The maintenance strategy is built primarily from the preventive and predictive maintenance basis that exists at the plant.
3.2.1.1.1 Findings
The Preventive Maintenance (PM) base has not been updated for some years. Preventive maintenance is not being done judged on the age and extent of PM in the backlog. Random outage rate (ROR) defined as the sum of all forced outages and de-rates and all unforeseen outages and de-rates divided by the available generation excluding periodic outages and de-rates and expressed as a percentage is the availability/reliability measure used. Nearly all the staff interviewed could not define ROR nor did they know what the target was for St Clair. The planned outage frequency has been increased from two years to three years. The next periodic is scheduled for Spring 2001
3-6
During the 1998 periodic an extensive inspection and repairs were carried out on the boiler tubes. Since the return to service to August 2000, only two boiler tube leaks outages have occurred. Thus meeting managements expectations/target, of one tube leak per unit per year. This achievement illustrates the effort that has been put in by the boiler engineer to reduce the risk of tube leaks. Boiler tube inspections are detailed and thoroughly done, the data and information obtained from these inspections and subsequent repair work is stored on the Boiler Engineers computer. Ownership of the boiler preventive and predictive maintenance basis is unclear. EPRIs Boiler Maintenance Workstation (BMW) has been purchased however implementation has been restricted because of the lack of allocated financial and human resources.
3.2.1.1.2 Recommendations
Clearly define and allocate the roles and responsibilities of the various plant system engineers. Clearly define plant performance measures and targets to all staff. These should be tracked, trended and displayed where everyone can see them. Review and update the PM basis on the boiler and its auxiliaries. This can be achieved through the use of a reliability centered maintenance package. The aim is to develop an optimized set of PM tasks that will ensure the plant will achieve its reliability targets. Continue with the detailed boiler tube inspection program and develop a common database, accessible to other key people, that enables trending of the data and information and predictions to be made. Allocate resources to complete the implementation of the BMW throughout all Detroit Edison plants
3.2.1.2 Work Process A typical work process consists of four separate, but equally important steps. Work Identification - Work is identified as corrective, preventive, predictive, or pro-active. Work Control - Establishing a balance between the four types of identified maintenance which maximizes the use of manpower while ensuring that the proper maintenance is performed on equipment at the appropriate time. Work Execution - Work is planned and scheduled. Parts and tools are staged and the proper level of manpower is dedicated to the job. Work Closeout Recording of as found and as left information on all work orders. This information is maintained in an electronic database and used to plan future maintenance activities.
3-7
3.2.1.2.1 Findings
Periodic outage duration is between 7 and 9 weeks. Boiler inspection is an on going activity that can take up to three weeks to complete while boiler cleaning and scaffolding is accomplished during the first week of the outage.. Boiler tube repairs made during planned periodic outages are well documented albeit in an independent system to the CMMS. The inspection report and repair records are well written, detailed, and complete. The repair record identifies the location of all repairs and the type of repair made on the tubes, as well as the suspected failure mechanism. All boiler tube repairs during a periodic are prioritized using three levels, #1 being the highest priority. Numerous priority # 2 and # 3 are not completed because of lack of time and resources. Quality control plans for boiler tube repairs are used and checked by the boiler engineer. Recommendations made by the Boiler Engineer as a result of a periodic outage inspection are not acted on until a few months before the next periodic outage. With the current outage frequency, this could be a three-year gap. The responsibility for acting on these recommendations is not clearly understood or defined. There is no backlog of planned boiler work preventive, predictive etc to be worked during forced outages, nor is there a team dedicated to boiler inspections during forced outages. Boiler performance data does not play a role in planning boiler outage work. The large amount of data generated by the DCS system or stored in the PI system to assist in directing or planning boiler work is not being used. Parameters such as tube metal temperatures, temperature changes across reheat and superheat sections, furnace gas temperatures, heat rate, and changes in gas pressure and flows can be used to direct inspection and repair activities. A long-term Reliability Management plan has been developed. The plan establishes five year ROR, LOP, and generation targets for the St. Clair Power Plant. The plan details work to be done at the next periodic outage, and there are no specific plans for the high pressure sections which are approaching or exceeded the expected 100,000 hours life expectancy. Although net heat rate is mentioned in the long-term Reliability Plan, there are no goals established for heat rate improvement, or heat rate improvement projects listed
Conduct a technical review of outage management. This review should be and based on sound project management principles and include a technical review of the scope of work from each discipline. Many USA and International utilities are achieving 4 to 5 week outages every three years. These plants are similar to St Clair Unit # 7 in size and capacity burning Powder River Basin Coal. Shortening a 9 week outage by two weeks would have huge impacts on outage costs and revenue. The boiler inspection plan should be integrated into an all-inclusive Predictive Maintenance (PdM) plan. A PdM program consists of a series of activities that are performed prior to breakdown in order to forestall the immediate occurrence of impending failures. The present
3-8
boiler inspection program focuses on identifying and correcting existing problems. Application of PdM methods for boiler inspections and inclusion of findings into an integrated plant PdM database would enhance the scheduling and planning of boiler maintenance work and contribute to reduced ROR goals. The inspection report and repair records prepared by the boiler engineer are key indicators of the boilers condition. These indicators should lay the groundwork of both long- and short-term planning for the boiler maintenance program and improvement projects. Responsibility for compiling work orders, and following through on recommendations made as a result of the boiler inspections, should be assigned and acted on in a timely manner. The reports should be made available in electronic form to all Detroit Edison employees involved with the boiler maintenance program. Advantage should be taken of forced outages to inspect available areas of the boiler. Trigger ready forced outage plans - work orders/work packages need to be developed for inspection of identified problem areas in the boiler immediately following both periodic and forced outages. Thus, in the event of a forced outage advantage can be taken to perform boiler inspection and or repair work. The present long-term plan should be expanded to a minimum of ten years and a maximum of twenty years. Mile stone years should be based on the outage frequency i.e. the plan should have significant detail for years one, three, six etc. The steps to consider in formulating this long-term plan are: History 1. Design review 2. Construction review 3. Operating and maintenance history review 4. Inspection reports 5. Industry experiences Evaluation of Present Condition 1. Metallurgical evaluations 2. Visual inspections 3. Non-destructive examination (NDE) prioritization 4. Destructive examination prioritization 5. Operating conditions and practices 6. Maintenance strategy 7. Database compilation Plan Formulation 1. Remaining life assessment 2. Long-term inspection plan 3. Risk assessment 3-9
4. Short and long-term recommendations Plan Re-evaluations 1. Review and updating of inspection plan 2. Reviews of new NDE technology 3. Inspection planning and scheduling 4. Risk assessment re-evaluation 5. Confirmation of remaining life assessment 6. Updating of plan database Boiler performance and process data (metal temperatures, excess oxygen, steam and water temperatures and pressures and air and gas temperatures) is available via the DCS and PI system and should be tracked and trended. Deviations from specified should be investigated analyzed and action taken to prevent recurrence. Enforce the use of the CMMS historian. No Work Order should be closed out until the craftsman who has done the work fills out this section of the Work Order. Every Work Order should be reviewed for repeat failures, and a proper action plan developed. This is a key factor in the continuous improvement cycle.
3.2.1.3 Predictive Maintenance Program and Technology An optimization boiler maintenance program is made up of people, management systems, component owners, technology specialists and predictive maintenance tools or equipment. The key predictive maintenance tools used in a Boiler Reliability Optimization project are infrared thermographs, vibration, oil analysis, ultra-sonic etc. Equipment owners and technology specialists collect and collate data on various boiler components. This data is organized and presented in an Equipment and Condition Indicator (E&CI) Matrix. The matrix specifies the predictive maintenance technology to be used, the equipment to be monitored and when the data should be collected, threshold/action limits/levels and actions to be taken. An example of such E&CI matrix can be found in Appendix 2 The how, the what, and the when of condition data analysis is based on exceeding preestablished threshold levels and this information is presented in an Event Report and in an Asset Condition Status Report. Direct causes of deviations from threshold are determined and corrective actions are communicated to the relevant people. Threshold levels are based on industry, design criteria and baseline data.
3.2.1.3.1 Findings
During periodic outage, visual inspections and ultra sonic measurement methods are used extensively by the boiler engineer. The focus is on the boiler tubes. Infrared thermography is used on electrical equipment, the boiler casing, and identification of leaking valves. Metallurgical analysis of failed boiler tubes is performed on a limited basis.
3-10
An oil analysis program on the fans and pulverizes is in place. The unit water chemistry program is well controlled and the units operated within EPRI guidelines. The unit has not been chemically cleaned for nine years, and there have been no boiler tube failures due to waterside problems. Report 97E08-81, dated 10/20/97, gives the results of deposit analysis on three Unit 7 waterwall tubes. The north wall sample had the highest grams/ft2 and 19.15% copper, as Cu. 2 The two other tube samples showed significantly less grams/ft and copper values Boiler efficiency/performance tests are no longer performed.
The boiler inspection report, tube leak reports, DCS data, and PdM data need to be integrated into a single database. The key to utilizing the condition-based information in an effective manner is the ability to quickly access the information and to trend the data over time to enable a prediction of when work should be performed. EPRIs Boiler Maintenance Workstation (BMW) is one such system, which can be used to track and trend boiler tube failure data and information. Another useful tool is the WEB based Plant view software. Remove additional tube samples from areas where high copper had been detected (see Report 97E08-81 dated 10/20/97) and check and confirm the presence of copper. Use the data collected by the DCS and PI proactively eg trend and track metal temperatures, excess oxygen etc, in order to identify incipient failures.
3.2.1.4 People Human beings are an important asset. Their reliability making and keeping commitments has a huge impact on plant reliability. The design, manufacture, installation, operation and maintenance of all equipment is in some way or other related to the performance of human beings. Therefore, if human beings are reliable your equipment will be reliable.
3.2.1.4.1 Findings
The staff at St. Clair has recently undergone reorganization, with a reduction in staff numbers. The present organization is highly matrixed, and the roles and responsibilities of individuals are not clearly understood. Communication within the matrix is poor, with many of the plant staff reporting they are out of the loop regarding decisions on the boiler. As an example, there is no clear understanding, of who is responsible for following up on recommendations made by the boiler engineer in the boiler inspection report. Is it the boiler engineer or maintenance supervisor? At the time of writing this report, the current boiler engineer had been assigned other duties which would take him away from doing thorough inspections. There is no succession planning for plant staff. Technical expertise and experience is concentrated among a small number of staff within AMO and ESO. Plant staff are therefore dependent on these organizations for support and boiler expertise. 3-11
An extensive on-job-training program for new operators has recently been put in place.
A Boiler Tube Failure Reduction team consisting of plant, ESO, and AMO staff should be formed. Every tube failure should be investigated and an in-depth report issued. This report should include a Root Cause Analysis, short and long-term recommendations, and a copy of Work Orders written by the team. Semi-annual progress reviews should be issued giving the status of action being taken on recommendations. Management needs to clearly define and communicate roles and responsibilities of the Plant, AMO, and ESO staff, and the relationship of individuals within these groups. With the current age profile of the plant work force, it is essential to develop and put in place a succession plan that will provide qualified people.
3.2.2 Failures and Root Causes The section deals with the causes of failures, actions taken and the actions to be taken to mitigate future failure. 3.2.2.1 Boiler Tubes From 1995 to April 2000, on Unit 7 there have been 21 forced outages due to boiler tube failures. Analysis of these reports shows 54 tube failures, involving eight failure mechanisms. See Figure 3-2 below. The documenting of tube leaks is done via the Short Form report. The report also includes details of the other repairs due to the consequence of the first failure.
3-12
16
14
12
A tta c h m e n ts C li n k e r D a m a g e F a ti g u e C ra c k i n g O xi d a ti o n o f R H T u b e s Q u a li ty C o n tro l/W e ld s S la g E ro s i o n S o o t B lo w e r E ro s i o n
10
W a te r L a n c e Q u e n c h i n g
0 A tta c h m e n ts C li n k e r D a m a g e F a ti g u e C ra c k i n g O xi d a ti o n o f R H T ub e s Q u a li ty C o n tro l/W e ld s S la g E ro s i o n S o o t B lo w e r E ro s i o n W a te r L a n c e Q ue nc hing
1995
1996
1997 2
1998 1 2
1999 1
2000
4 1 1 3
15 3 2
2 5 4 4 1 1 1
Figure 3-2 Tube Leak Failures 1995 - 2000
Analysis of Figure 3-2 reveals the following trends: Failures due to water lance quenching peaked in 1995 and 1997, with four tube failures in each year. Since replacement of the water lances, there has only been one incident, which was due to damage prior to replacement. This pro-active approach has been successful in eliminating this problem. Failures due to soot blower erosion peaked in 1995 with four failures, and only one reported in 1998 and 1999. This is due to extensive pad welding done during the 1998 outage. This
3-13
approach seems to have solved this problem for the short-term, but the large number of pad welds threatens the units future reliability. Failures due to clinker damage were the highest in 1997 with four failures, followed by two reports in 1998. Installation of reinforced wear bars during the 1998 outage to date has eliminated this mechanism. Failures caused by the lack of adequate welding quality control has increased. Of the eight tube failures reported in 1999 and 2000, five, or 63%, were due to poor workmanship or using the wrong material. There were also three incidents in 1997. These failures peak following periodic outages. Analysis of a failed reheat tubes in 1997, showed the cause of the failure to be excessive external oxidation. Inspections performed during the 1998 periodic outage revealed additional fifteen tubes with excessive wall thinning due to excessive oxidation. Although there was only one forced outage due to this mechanism, it emphasizes the importance of monitoring, trending, and controlling tube metal temperatures.
3.2.2.1.1 Failure Mechanisms, Actions taken and Recommendations
The 1998 Boiler Inspection and Action Plan was reviewed and the following table represents the Findings and Recommendations. The bold text represents the additional recommendations as a result of the review process.
3-14
EPRI Licensed Material Boiler Reliability Optimization Project Phase 1 at Detroit Edisons St Clair Plant Unit 7 American Electric Powers Big Sandy Plant Unit 2 Table 3-1 Boiler tube Failures, Causes and Recommended Action Attachment Failures Findings
In 1997 two seal trough failures occurred In February 2001 an inspection revealed several rigging/support lugs welded onto the waterwall and back pass/economizer. Burned or eroded superheater alignment clips are rlaced and or repaired.
Cause/effect
The cause of the stress cracks was not determined. In many instances these lugs if not carefully removed can cause future tube leaks
Action taken/Recommendations
During 1998 the boiler seal system was replaced with an upgraded design to minimize the design. Welding lugs on pressure parts waterwall, economizer etc is not a good practice and should be discouraged.
Continual welding as in repair/replace can result in damage to the external and internal tube surface of the tube, resulting in a tube leak The cause, as recorded in the Short Form Report is erosion which lead to thinning of the material and increased mechanical stress.
Hancuffs should be used after carefully removing broken or damaged clips.
In September 1998 and April 1999 two failures occurred in the horizontal portion of the steam cooled spacer tube attachments. This failure mechanism was first noticed in 1994 and 1996 periodic outages
During the 2001 periodic outage replace the water-cooled spacers utilizing the new PIB-74 attachment system as designed by ABB-CE In future, when an attachment failure occurs - tube samples should be removed to check for possible corrosion fatigue damage.
Clinker Damage/Slag erosion Findings

Six failures have occurred, one in 1996, four in 1997 and two in 1998.
Cause/effect
Localized slagging three to four feet from each end has been the major cause.
Investigate the cause of this localized slagging. During 1998 periodic outage, all tubes on the lower slope have been replaced with a wear barring tube extra material on top of the tube. Continue to inspect these tube as and when opportunities arise - during forced and planned outages, to determine a wear rate from which predictions can be made
3-15
EPRI Licensed Material Boiler Reliability Optimization Project Phase 1 at Detroit Edisons St Clair Plant Unit 7 American Electric Powers Big Sandy Plant Unit 2 Fatigue/Stress Cracking Findings
Nine fatigue/stress failures have occurred. Four in both 1995 and 1997 and one in 1999. In February 1997 a superheater rear wall hanger tube
Cause/effect
These failures were all caused water quenching from defective/deficient water lances.
These water lances have been replaced with an upgraded design
Loose/broken hanger resulting in the tube vibrating loose and crack caused this failure.
Compile a hangar inspection PM with all the necessary data/ information to be recorded so it could easily be issued during a forced outage. This same PM would also be issued during a periodic outage.
Long Term Overheating/Creep Findings

In 1997 and 1998 there were one and fifteen failure on the reheater respectively. Analysis of tube samples by ESO (Report # 99V0080001) indicated tube failures were due to thinning by oxidation. The protective oxide coating had spalled off because of mechanical stress causing thinning, The ESO report failed to identify or mention if any short or long term overheating had taken place. There was no mention of the metallurgical aspects/condition of the tube. The report states thermocouple data does higher temperature in the center however no conclusions or recommendations are made regarding the high temperatures.
Cause/effect
Two causes were given high temperature and mechanical stress because of hangar support failures. A hanger support pin was missing.
The action taken during the 1998 periodic outage consisted of repairing the hangers and insert 60 Dutchman Regarding recommendation for hanger inspection see recommendation under fatigue/stress cracking. Use UT to measure oxide thickness across the superheater, correlate with temperature profile thermocouples and remove tube samples metallurgical analysis and conformation of oxide thickness measurement
The ESO report therefore did not provide adequate information/guidance on long term heath/life aspects of the reheater tubing.
All analysis reports of tube samples should give recommendation regarding future inspection strategies and or examination requirements.
Remove during the Spring 2001 a representative sample of tubes from across the reheater for remaining life assessment.
3-16
During start-up and under normal operation little attention is given to adhering to metal temperature limits and or recording the extent and duration of excursions. Creep damage is dependent on stress level and tube metal temperature. Thus conditions. Therefore conditions that deviate from design which lead to increase stress and temperature will result in reduced tube life. The importance of operating within design parameters must be impressed upon operating staff. Metal temperatures should be continuously recorded and monitored by the operator. Rates of change and maximum limits should be determined and adhered to. Deviations from expected should be recorded and investigated. Actions to prevent recurrence should be reviewed and followed up to ensure compliance.
Quality Control Findings

Since 1997 a number of pad welds and or welding have failed
Cause/effect
An inspection and or investigation revealed that the welders failed to comply with the welding norms and standards. Quality control of these welds was also inadequate.
Pad welding should be discouraged not only because it destroys the evidence but it also can create the potential for other mechanisms to occur hydrogen damage. However if it is used, criteria should given to when and when not to use pad welding. Welding procedures should be compile with and all welding should be inspected quality control
Soot Blower Erosion Findings

Since 1995 several tube failures have occurred because of soot-blower erosion. Forty nine tube were repaired during the 1998 periodic outage
Cause/effect
Soot-blower erosion is controllable. The causes are related to excessive use operational and improper maintenance.
Continue with the soot blower replacement project new helix design. Optimize the soot blowing sequence and frequency by using thermal condition in the boiler to determine when and how long to blow.
Waterwall Fireside Corrosion Findings

During the 2001 periodic outage low Nox burners will be fitted
Cause/effect
Apart from the environmental benefits low Nox burners can cause substoichiometric conditions on the side and rear wall near the burners. These conditions nearly almost result localized corrosion taking place.
Obtain a baseline by measuring tube thickness local to the burners - near the bottom burner level to about 10 feet above the top burners and the side walls midway between the front and rear walls at the same height of the burners. Ensure that the newly installed burners are correctly optimized measure CO and Oxygen near tube walls if possible. High levels of CO (>1%) and low levels of oxygen (<0.1%) near the walls would be of concern.
3-17
3.2.2.2 Boiler Headers Detroit Edison has a high-pressure piping and header inspection program in place for St Clair Unit # 7. The plan reviewed did not give any details beyond the next outage. The program includes the final superheater and reheater outlet headers and excludes all other boiler headers. The condition of all remaining headers is unknown even though the unit has operated for well over 100,000 hours. It is recommended that a Level II Remnant Life Assessment be undertaken. See Appendix 3 for a typical flow chart. The findings from Level II may lead to further examination Level III. From these assessments a Boiler Test and Inspection Plan should be developed. The plan should cover at least 15 years. The various planning windows would be dependant amongst other things on the particular maintenance strategy adopted outage frequency, duration, extent of inspections etc. Table 2-2 shows the typical failure modes and failure mechanisms found in steam drums and headers that the inspections plan should cover.
Table 3-2 Typical Header and Steam Drum Problem Areas Header or Drum Final Superheater and Reheater outllet Failure mode Internal Cracking Swelling Local bulging Reheater inlet Final Superheater inlet All headers Sagging Internal Cracking External laminations External Cracking Failure Mechanism Creep-Fatigue High Temperature Creep Creep or Thermal expansion Support problem and or over temperature Thermal Fatigue Fabrication Defects Thermal Shock or expansion Thermal Expansion Fatigue Tube nipple Cracking Drums Internal surface pitting Ligament and bore hole cracking External nozzle weld cracking Economizer Inlet Internal Cracking Thermal and Corrosion Fatigue Oxygen control in feedwater Thermal and corrosion Fatigue Thermal expansion Fatigue
Below is a brief guideline of a typical Boiler Test and Inspection Plan
3-18
3.2.2.2.1 Boiler Test and Inspection Plan Guideline
An effective inspection and test plan based on design review, historical operational data and selective physical examination allows accurate assessment of plant condition and future life cycle management. Inspection of critical items of plant on design reviews together with component reviews of results and historical data will enable effective planning of repairs/ replacement programs to be formulated with accurate budgets. His will enhance plant availability and safe operation.
Purpose
To provide guidance in the preparation of inspection programs aimed at providing a comprehensive plant condition database, allowing accurate assessment of current plant condition and predicting future replacement and refurbishment requirements.
Definitions
Major Component - This is a component with an internal diameter exceeding 4 inches. It includes headers, attemperators, internal boiler pipework etc. Heat exchanger tubing is excluded. Component A pressure containment operating in the creep range or subject to fatigue together with its weldments comprising the weld metal and the heat affected zones. Sub Components Are those components when welded together form a major component. Component Inspection Plan This is a detailed listing for each generating unit giving a specific schedule of components (major and sub components) to be inspected, the type and extent of the inspection. Inspection Report Inspection reports are generated following the physical inspection based on the component inspection plan. Plant Condition Review This is a rigorous review of relevant inputs that will indicate the requirements for inspection. These include operating (including temperature excursions and major events) and maintenance history and previous condition assessment reports. Critical Component A component that has been identified by calculation or inspection as having a limited life and is expected to suffer the greatest degree of life exhaustion. Calculated values should be based on actual dimensions where possible.
Scope
The scope of the plan should cover all components (major components, components and sub components) within the boiler envelope. The intent of the plan should be that by the time the plant has operated for 50% of the nominal design life, at least 50% of the components have been physically inspected and their condition known and documented.
3-19
Inspection and Test Plan Development
Compile a list of uniquely identifiable Test and Damage code.

Inspection and Test Plan Codes Test Codes 1. Visual examination 2. MT 3. UT 4. ----Damage Codes A. No Damage/ defects B. Defects removed < 0.1 deep C. Defects removed < 0.2 deep D. -------
Using the manufacturer flow diagrams and isometric sketches compile a list and uniquely identify each major component, components, sub components (tee pieces, spool pieces, end caps etc) and welds (girth, nozzle, seam etc). Using the information from 1 and 2 above, compile a master inspection plan spreadsheet. Use the vertical axis (rows) to list the components (major, component and sub components) on the horizontal axis (columns) list the following information. 1. Operating parameters 2. Date of last inspection 3. Frequency of test 4. What damage/indication to identify 5. Responsible person/ system owner 6. Reference drawings 7. Dimensions 8. Material specifications 9. Test procedure 10. Database reference 11. Comment/remarks
Conduct a plant condition review and identify the critical components. Using the master plan compile a component specific inspection plan. This plan should be linked to a CMMS work-scheduling package. This will allow the test and inspection plan to be included into a short or periodic outage. Conduct the inspection and record all the relevant data. Assess the component specific data to determine the action to be taken operate at design operating temperature or below, refurbish, repair or replace. Document and record actions and decisions.
3-20
3.3 Conclusions
The down sizing and re-organization has had an impact on the effectiveness of various management processes on site as staff are getting to grips with their new roles and responsibilities. The work flow process from work order initiation to work close out has been affected. Planning and scheduling of short and long term work has been most affected. The current outage on unit # 7 of 9 weeks is long in comparison to other utilities. Many USA and International utilities have outages of between 4 and 5 weeks every 3 years. It is understood that the outage scope of work may well be totally different however these differences need to be analyzed and improvement opportunities determined. The periodic outage boiler tube inspection plan and outage reports are well thought out, well organized, and well executed. An important key to achieving a three- year outage cycle is the continued excellence of this program. This information is stored on the boiler engineers computer even though a Boiler Maintenance Workstation exists. The inspection and repair efforts during the last periodic have paid dividends a reduction in the number of boiler tube failures due extensive repairs made during the last periodic however the number of forced outages due to tube leaks has remained constant. The current boiler inspection plan although detailed and thorough is reactive in that it focuses on repairing the immediate problems identified during the inspection. Areas that are not problematic now are not inspected. These areas may well result in forced outages in the future. . There is single central boiler tube failure database that integrates inspection results, root causes, recommendations, failure history and metallurgical analysis Extensive pad welding is done in lieu of resolving the sootblower replacement and or modification/maintenance issues. Pad welding not only destroys evidence of a failure mechanism but can be the source of a potential failure. The Fossil Generation Long Term Reliability Plan for St Clair only includes work to be undertaken up to the next outage. This is considered to be too short time frame to obtain the real benefits of long term planning.
3.4 Recommendations
Consideration should be given to having a Plant Maintenance Optimization Workshop. The workshop focuses on the broader issue surrounding maintenance. It helps identify opportunities for improvement in all facets of maintenance, where as this report really focussed on the maintenance issues affecting boiler maintenance. The two-day workshop is available to members of the Plant Maintenance Optimization Target. Compile an action plan, with allocated responsibilities and completion date, to implement the recommendations made in the above sections of this report. 3-21
Review and benchmark the current outage management philosophy and approach with the intent to reduce the overall outage time frame in line with best practice. Management needs to clearly communicate the roles and responsibilities of the newly formed service groups to all personnel involved. Expectations need to be clearly defined and individuals held accountable for their successes and failures. Compile a root cause analysis process guideline to be complied with by all Detroit Edison plants. Train all staff in the basic concepts of root cause analysis and a number of engineers at each site on the details of applying root cause analysis to incident and problem resolution. Assemble a Boiler Tube Failure Reduction team to investigate ALL boiler tube failures. Their responsibilities would be, determine root causes, recommend solutions, follow up on recommendations, issue annual reports on the boilers long term health, maintain an integrated database, and, put together and continuously update the long term plan for the boiler. Complete the soot blower upgrade project, assign a higher priority to soot blower and water lance PMs, and analyze unit operating data to optimize the combustion process and decrease soot blowing frequency. These actions would greatly decrease the number of pad welds needed to counteract soot blower damage.
3-22
Appendix 1: List of interviewees

Melanie McCoy Phil Thigpen Bob Sausser Dan Kuhlman Dan Lavere Tom Seaton Mike Mallon John Shinske John Renehan Harry Mueler Bill Fielhauer Mike Bennet Bill Harvey Willie Myers Fred Cutter Dave Hurt Jack Miketich Bill Hornby Barry Wagner Jeff Hat Charles Burmann Jim Jordan Ken Brinker Dave Stephan Plant Manager Reliability Engineer Supervisor Plant Operations Supervisor Plant Operations Predictive Maintenance Coordinator General Foreman Boiler Reliability Engineer Unit #7 Foreman Unit # 7 Planner Unit # 7 Engineer Unit # 7 Engineer Water Chemistry Engineer Instrument Shop Foreman Water Laboratory Technician Operations Coordinator Shift Supervisor Shift Supervisor Shift Supervisor Supervising Operator Supervising Operator Supervising Operator Supervising Operator Supervising Operator Training
3-23
Appendix 2: Example of a Typical Equipment Condition and Technology Matrix
3-24
Appendix 3: Remnant Life Assessment Flow Chart for Thick Section
Assemble design and service information Design: * Dimensions * Pressure * Materials * Temperature * Minimum * Stresses creep-rupture Service: * Boiler running hours * Past repairs and replacements * Dimension and composition checks
Level II Assessment: Thick Section Parts
Level III Assessment: Thick Section Parts
Conduct careful visual inspection. Are there cracks at ligaments or at penetration welds? No Yes Level III Assessment
Conduct detailed inspection by best NDE methods available. Measure dimensions of all crack-like defects.
Answer key questions
Has unit significantly exceeded design P and/or T?
Yes Attach thermocouples; determine temperature distribution. Monitor temperature at hot spots to obtain representative sample. Level II Assessment Using actual temperature and pressure, calculate life fraction expended and remaining life
Perform transient and steadystate thermal analyses by finite element or finite difference methods.
Will future service involve P and/or T above original design?
Yes
Perform detailed stress analysis by finite element or finite difference methods.
Yes Has failure history of boiler been excessive? Yes Steam temperature records not available? No
Perform fatigue, creep and fracture mechanics analyses. Calculate remaining life to crack initiation and to fracture after crack growth. Is remaining life greater than the projected life extension? Yes No Make repair or replace decision Set inspecti on interval; maintain accurate operatin g records Yes
Is remaining life greater than the projected life extension? Level I Assessment No
Using design and basic operating data, calculate hoop stress, life fraction expended and remaining life Level III Assessment Calculate remaining life RL = (1 LFE)tR
Set inspection interval; maintain accurate operating records
Estimate costs and schedule
Is remaining life greater than the projected life extension? Yes No Set inspection interval; maintain accurate operating records Level II Assessment
Input to plant and corporate plans
3-25
4
BOILER RELIABILITY OPTIMIZATION PROJECT PHASE 1- ROOT CAUSE ANALYSIS AT ARIZONA PUBLIC SERVICES COMPANYS FOUR CORNERS AND CHOLLA PLANTS
FOUR CORNERS PLANT 1999 Lost Generation Incident Review 4.1 Executive Summary
In an increasingly competitive environment, the ability to reliably produce the lowest cost power is essential for the long-term survival of a utility. With this vision in mind Arizona Public Services Company (APSC) requested EPRI to review the 1999 Lost Generation Action Plans from the Four Corners Plant. The intent of this review was to provide assurance or not that the true root causes of lost generation have been identified, and appropriate action plans are in place to address these causes. The review took place between 30 May to 9 June 2000 on site at the Four Corners Plant. Ten incidents, resulting in a loss of 728,978 Mwh were reviewed. Of the ten incidents reviewed only one was adequately documented - backing up the plants root cause analysis efforts. The review team, two from EPRI, one from Cholla Plant and one from Four Corners Plant, tried, through a series of interviews, to re-construct the incident. More than twenty staff members were interviewed. Two methods, Behavior Justification and Event and Causal Factor Charting were used to reconstruct the incidents. Diagrams/charts were constructed from information that depended, almost entirely, on the interviewees' long-term recollection of incidents. Verification of information was difficult, as there was a lack of formal incident-specific documentation and because of the time that has elapsed between the incidents and the analyses. In many incidences, perceptions and memory of what happened and in what sequence varied, emphasizing the importance of performing Root Cause Analysis in a timely manner, and then documenting the results. Based on the information obtained during the two weeks at the plant, the EPRI team uncovered many opportunities for improvements in operations, equipment maintenance, inter-departmental communications at all levels of management, human performance, root cause analysis documentation and engineering response. The Lost Generation Action Plans as developed by the 4-1
EPRI Licensed Material Boiler Reliability Optimization Project Phase 1- Root Cause Analysis at Arizona Public Services Companys Four Corners and Cholla Plants
plant need these improvements to ensure that they have a sound base. Documentation of incident investigations, no matter how minor, provides an audit trail to ensure a historical record of what has occurred at the plant, how it was analyzed, what the cause of the problem was and how it was solved. Lack of this audit trail often leads to repeat incidents because staff are not aware of what has happened previously/how problems were solved, often leading to short-term, inadequate solutions, rather than structured, long-term and cost-effective solutions. No formal structured root cause analysis process exists. Some plant employees have been trained in a root cause analysis technique, but do not use it as it was intended to be used. The staff members were cooperative and mostly forthcoming and interested in using a Root Cause Analysis tool to solve long-term problems at the plant. Root Cause Analysis training/re-training needs to be implemented plant to ensure that staff members are confident in their abilities to solve problems that arise. Each incident reviewed has a number of recommendations. These recommendations need to be evaluated and implemented. The main recommendation is that a formal, structured approach to incident investigation needs be developed and implemented at each plant. A corporate directive should be developed to give guidance on the requirements. As a start, all unit trips/forced outages, load loss, and failures to meet expected outputs, should be subjected to thorough root cause analysis. Incidents where plant damage, exceeding a predetermined amount, has occurred should also be investigated and any significant capital expenditure needs to be backed up with a justification that is based on a thorough root cause analysis.
4.2 Introduction
In an ongoing effort to improve reliability and reduce maintenance costs in the APS system, EPRI was contracted to review the 1999 Loss Generation Action Plans at Four Corners Plant. The main objectives of this review were: 4.2.1 To provide assurances that the true root causes of lost generation have been identified, and appropriate action plans are in place to address these causes: 4.2.1.1 determine and verify the true root causes of technical and/or production problems, and 4.2.1.2 identify preventive actions to effectively manage the identified problems and minimize the affect on availability. The team consisted of, two from EPRI and two from APSC, one from Cholla Jack Stant and one from Four Corners Plant Mike Rosner. The reviews took place between May 30 through June 9 on site at Four Corners Plant. More than twenty staff members were interviewed regarding the 10 incidents reviewed. These incidents represented a loss of 728,978-Mwh at the Four Corners Plant. The incidents selected represented the major Mwh losses and were prioritized by the plant management team. The selection was based their significance, available documentation, and availability of staff having knowledge of the incidents.
4-2
The Behavioural Justification method of analysis was used to analyze some of the incidents that occurred some APS personnel have been trained in this method. Events and Casual Factors (E&CF) Charting was used as an alternative method. In this method the sequence of events is charted chronologically, from left to right, and information/conditions relating to events in the sequence are charted directly underneath the events. Events are shown as text within rectangles, joined by arrows, ending in a circled unwanted event (reason for the analysis). The information/conditions are shown in ovals, joined by lines that show their association. Dotted ovals or rectangles indicate information that still needs to be verified. Causal factors are identified by asking the question Which of these factors, if eliminated, would have either prevented the incident from occurring or at least minimizing its impact? From these causal factors Root Cause are determined. Root Cause is the cause/s which, if eliminated, would prevent this event recurring or minimize its impact.
4.3 Plant Description

Four Corners Power Plant is owned by Arizona Public Service and is situated in the Four Corners region of New Mexico. Units 1 & 2 are Riley Stoker front fired, natural circulation boilers, rated at 170 Mw each. They were placed in service in 1962 and 1963 respectively. Unit 3 is a Foster Wheeler front fired, natural circulation boiler rated at 220 Mw and went into service in 1964. Units 4& 5 are supercritical B&W units rated at 755 Mw each. They went into service in 1969 and 1970 respectively. All units burn high ash Western coal.
4.4 Incidents investigated

The incidents selected and the Mwh lost are listed in the following tables: Four Corners - Megawatt-hour loss summary report-fourth quarter 1999
Mw Hour Lost Area Unit 5 #4 Control Valve Unit 4 Air Pre-heater Failure Unit 3 Turbine Copper Fouling Unit 4 LP Exciter Field Ground Unit 3 Boiler Tube Leaks Unit 4 Flyash Erosion Unit 4 Battery Failure Units3 Pulverizers Unit 4 HP Exciter Field Ground Unit 3 Scrubber, and Recycle Pumps Total Mwh. Lost 197,471 166,744 100,628 45,229 39,378 39,288 38,413 34,881 33,500 33,446 728,978
4-3
4.4.1 Unit 5 #4 Turbine Control Valve chatter - 197,471 Mwh lost Limited documentation was made available on this incident. To construct a sequence of events the following people were interviewed - Jack Rogers, Willard Billey, Dave Saliba, Joe Rogers, John Bates, and Randy Wilde Incident Description From June 1 - 20, 1999, and from July 1 - Dec. 27, 1999, Unit 5 was de-rated by 6 % due to # 4 turbine control valve problems. On June 17, 1999 during a tube leak outage the valve linkages were repair and the valve operated satisfactorily for 11 days. On July 1 an operator reported that the valve was once again chattering and later slammed shut, and could not be re-opened. The unit was operated with the load limiter set at 94% until a boiler tube leak outage on Nov. 26. During this outage the control valve was checked and found to be operational and the governor pilot valve was rebuilt using re-conditioned parts and other parts, which were not from the original equipment manufacturer (OEM). The unit was brought back on line, and continued to run with the load limiter set at 94%. On Dec.25 the unit trip because of a switchyard failure, the unit was returned to service the same day with no load limitations due to the # 4 control valve. Root Cause Analysis Chart Figure 4-1 was drawn based on interviews and operator logs. Knowledge of the problem and actions taken, or not taken, differed between individuals. This made it difficult to accurately record the sequence of events. However, the information as reflected in the chart was accepted by all those interviewed as a fair reflection of what happened. An analysis of the chart identifies a number of opportunities, that if investigated, at the time, would have eliminated the event or at least minimized the impact of the event. These are: There was no attempt to inspect or repair the valve during the June 11 boiler tube leak outage. On July 1 it was reported that the valve was chattering and then had slammed shut and could not be reopened. This was accepted without an investigation and the unit ran de-rated until November 26. Reason for not investigating at time could not be established. During the November 26 outage the value was checked and verified as operational. However the unit ran de-rated until the December 25 outage. The reason for this continued de-rating could not be established No valve work was performed during the December 25 outage, however after the outage Operations were informed that the valve could be operated. The unit was returned to service with no de-rating attributed to the control valve.
To determine the root cause of why so many Mwh were lost, the above points need to be investigated further. This could not be accurately done because of the uncertainty surrounding the sequence of the events from June though December 1999. However, given that the facts in the chart are a fair reflection of what happened, two causes come to mind that if corrected would 4-4
have minimize the impact of operational failure of the valve. These relate to problem ownership no person took responsibility to solve the problem and communication between individuals and between departments. An operational test of the value undertaken at the time of the review showed that the valve continues to chatter. The cause of the chattering is still unknown and waiting for suitable outage to determine the cause. 4.4.2 Unit 3 Scrubber unit failure - 33,446 Mwh lost No documentation was made available on the two incidents that made up this Mwh loss, however Phil Valencia provided some background information. Incident Description These two incidents involved the following deficiencies. Increasing rate of failure of scrubber recycle pumps. Scrubber debris was being caught in the pump impeller, causing pump imbalance. This imbalance resulted in pump bearing failures. Repeated blockage of the overflow and emergency overflow pipes with rubber lining and debris. Rubber-lined piping is used throughout the scrubber system to protect the piping from the highly erosive fluids, which it handles. Portions of this lining break off and block the overflow pipes. Repeated unit outages because of high levels in the scrubber unit. Slurry flowing out of the emergency and or overflow pipe is used as an indication of level. These overflow pipes can get blocked from excessive amounts of debris is in the slurry system. Structural failure the Dentist Bowl. The bowl is attached to a telescoping assembly used to maintain bowl position. Debris collects in the assembly and prevents free movement. This caused high stresses on the bowl when operated which result in eventual failure.
The above resulted in two trips of Unit 3. The first incident was the result of a high/high level caused by blocked overflow pipe. The emergency overflow was also plugged with debris. The scrubber flooded preventing flow. The delta pressure across the scrubber increased resulting in a unit trip - high furnace pressure The second incident involved a recycle pump failure. Refractory had fallen off a metal skirt in the scrubber and jammed in the recycle pump impeller causing an imbalance. This imbalance caused the pump bearings to fail resulting in a pump tripped. This caused the unit to trip.
4-5
Root Cause Analysis Charts Figure 4-2 shows the correctable opportunities identified for all four deficiencies highlighted the Behavioral Justification methodology. Clearances had been opened on the dentist bowl reducing the risk of any jamming of the telescopic section. Figure 4-3 represents the E&CF chart was used to analyze the lining failure and blockage problems. Since the piping failures and scrubber level problems were interrelated, they were analyzed together. At low temperatures the gum rubber separates from the metal surface cold wall effect forming blisters. These blisters eroded by the flowing slurry, break open and tear resulting in pieces of the lining coming off the pipe. These pieces of rubber block both the normal and emergency overflow pipes. Operators use the flow through the normal overflow pipe as an indication of correct level in the tank. Figure 43 highlights the correctable opportunities. Using a lining, which is more resistant to cold wall effect. Experience at Cholla Plant has proved that the use of a different rubber - chlorobutyl, which has a higher resistance to the cold wall effect, significantly reduces blistering. A more sophisticate method to determine the level in the tank should be installed rather than use the normal overflow pipe.
Figure 4-4 is an analysis of scrubber recycle pump failures and highlights some correctable opportunities. The pumps have urethane-coated impellers in which debris can easily become lodged, causing rotor imbalance. This imbalance causes the seals fail resulting in slurry entering the bearings, resulting in bearing failure. The major source of debris is refractory, which had been applied to a metal skirt in the upper portion of the scrubber in an attempt to protect the skirt. Moisture gets in between the skirt and the refractory. The resulting corrosion causes large pieces of the refractory to break off and fall into the sump. These pieces get sucked into the pump suction and jam in the pump impeller. Further investigation is required design review, to determine whether the skirt is required for the correct operation of the system. There was some doubt by members of the review team of its purpose. If, in deed it is required, then it should be replaced with a corrosive resistant material rather than a material with refractory. As and when opportunities present themselves, the urethane impellers should be replaced with white iron impellers. These impellers are more robust and fit for purpose in the scrubber environment. This has proved to be successfully at Cholla In addition suitably sized suction screens need to be installed. This is an inexpensive activity if undertaken during a suitable outage. These screens would be the first lines of defense to prevent over sized material entering the pump. These items are not specifically addressed in the 2000 Action Plan but can be accommodated in the $500k is budgeted for scrubber work.
4-6
4.4.3 Unit 4 Loss of Booster Fans
38,413 Mwh.
Mel Begayle and Richard Bagienski provided some background information, as there was no documentation available. Incident Description Regular preventive maintenance tasks performed on the twelve years old Booster fan control backup batteries had shown that the batteries needed to be replaced. For some unknown reason, AC power to the DC converter was interrupted; this required the controls to be supplied by the batteries. The DC voltage dipped below 108 volts. This dip in voltage caused the BETA alarm panel to reset, since the batteries were unable to maintain the required voltage. The booster fans tripped, tripping Unit 4. Batteries purchased for another purpose were used for replacement and another set of batteries ordered. The alarm panel was also changed. Root Cause Analysis Chart The E&CF chart - Fig. 4.3 shows the sequence of events. Lack of any documentation and relying on the memory of the individuals involved made it difficult to re-construct the sequence of events and hence determine the root causes. The direct cause of this incident was the poor condition of the batteries. However the following was determined from the interviews: electrical maintenance staff did not know how to correctly check the batteries Incorrect meter was used to perform the battery check. No action had been taken of the battery deterioration as noted on the preventive maintenance (PM) schedule. The crystal growth in the batteries had been visually verified and noted on the PM.
Additionally, a crossover switch between the Unit 4 batteries and Unit 5 batteries has been installed. The proper meter is now being used and the PM has been rewritten. All the actions discussed are included in the Lost Generation Action Plan. 4.4.4 Unit 3 Pulverizer Pinion & Bull Gear Failure - 34,881 MW Hrs. Bill Burnett and Calvin Charlie provided the information to re-construct this incident. Incident Description This incident involved the following: The foundation of 3B pulverizer had failed due to excessive vibration caused by incorrect alignment of the bull and pinion gears. It failed at the same place where a similar repair was made ten years previously. 4-7
The pinion gears, which had been purchased from different suppliers, did not meet the original specification and were installed by maintenance staff. A bull gear known to be out of specification was accepted and installed. Failure of the pulverizer lube oil pump.
Root Cause Analysis Chart The Behavioral Justification methodology was used to analyze this incident. This is shown graphically in Fig.4.4. Four correctable opportunities were identified. A procedure needs to be developed with criteria on the acceptance/rejection of equipment/components that do not meet specification/requirements. Plant management had accepted a bull gear, which had been improperly manufactured and had installed it. Reason for this decision could not be established. A Quality Assurance/Quality Control (QA/QC) process needs to be developed and followed to ensure that equipment maintained or manufactured by a supplier conforms to minimum requirements and is fit for purpose. This is especially essential with components like pinions and gears. Pinion gears are purchased from several suppliers and no QA or QC is performed. Operator rounds need to be enforced. The failed lubrication oil pump went unnoticed for an unknown length of time. The lack of lubrication caused the gears to overheat and contributed to the failure.
The Lost Megawatt Action Plan does not address this issue. Replacement of the pulverizers is being considered. However, the problems discussed above will not be solved with new pulverizers. New equipment will not resolve procedural, management, or communication problems. 4.4.5 Unit 3 Back-pass Flyash Erosion 39,378 Mwh David Hughes, Drexel Pruitt and Duane Pilcher shared with the review team their insight and knowledge of the erosion in the back-pass of Unit 3 Problem Description Flyash erosion and the resulting tube failures have been the major cause of forced outages on Unit 3. Changes in coal quality higher ash content combined with a furnace size, which was marginal for the originally specified coal, and continued operation at loads equal to or greater than design output have all contributed to fly ash erosion. Extensive shielding and numerous modifications have further increased the gas/particle velocities, which has compounded the fly erosion problem.
4-8
Root Cause Analysis Chart Fig.4.5 identifies a number of correctable opportunities. These all need to be evaluated separately and collectively. The purpose is to reduce gas velocities. Some of them are related to poor pulverizer maintenance. However, a significant contributor is the number of unqualified modifications. In the past, a number of modifications have been done to solve an immediate problem without considering the effects of such a modification on other areas of the boiler. Many of these modifications have indeed met the original intent and created an erosion problem elsewhere in the boiler. This is compounded by the lack of documentation on the number and rational behind the modifications. A Backpass study scheduled for completion in Sept. 2000 is in the Lost Generation Action Plan. It is recommended that an E&CF chart be drawn/used to analyze all past modifications made to the unit. This would be a daunting task, but extremely useful in resolving and understanding all the erosion problems. 4.4.6 Unit 4 Air Pre-heater Failure 166,744 Mwh. Ron Bertram, Jim Justice and Randy Wilde provided some detail about this incident, as there was no documentation regarding this failure. Problem Description Two modifications to the air pre-heater had been done to facilitate the early operation of the bag filter during unit start-up. These were to slow down the rotational speed of the pre-heater and delay its in-service operation during unit start-up. The net effect of these modifications on the air pre-heater was an increase in the number and extent of thermal cycles. This resulted in the thermal fatigue failure of a cold end basket retaining frame/clip. This allowed the basket to shift and bind the air pre-heater causing extensive damage. Root Cause Analysis Chart Figure 4-8 diagrammatically shows the causes and correctable opportunities. The two operational changes significantly contributed the accelerated thermal fatigue failure of the cold end basket-retaining frame. Both of these conditions have been corrected and an extensive inspection and repair effort is included in the Lost Megawatt Action Plan. 4.4.7 Units 4 Flyash Erosion 39,288 MW/Hrs. Mike Rosner provided the background to this problem. Problem Description Units 4 & 5 have a history of boiler tube failures due to flyash erosion in the back passes. This is because of a marginal boiler design and over firing conditions. When over firing for extra megawatts, operations increases excess air in order to prevent clinker formation. This results in 4-9
higher gas volumes and increased velocities. The short-term effect is less clinker formation. In the longer term this mode of operation increases flyash erosion resulting in unwanted tube leaks. This significantly impacts the reliability of the unit Root Cause Analysis Chart Fig. 4.7 represents diagrammatically the information provided. The flyash erosion problem is the result of a marginal boiler design as in furnace size BTU/square foot, choice of the most economical fuel and the operational philosophy. Erosion rate is dependent on a number of factors velocity being a major factor. The relationship between erosion rate and velocity, is exponential - velocity to some power between 2 and 3. So, velocity hence gas flow should be kept at a minimum and under control. A flue gas velocity distribution test is scheduled for 2003. Extensive efforts are planned in the Lost Megawatt action Plan to repair and mitigate the effects of flyash erosion. 4.4.8 Unit 3 Turbine H.P. Nozzle Pluggage 100,628 Mwh. The following provided some documentation and described the turbine pluggage problem - John Huff, Joe Baker and Dave Viser Problem Description Shortly after switching from hydrazine to Eliminox for dissolved oxygen control in 1993, the Unit 3 turbine/generator MW output started to decrease. In 1996 an investigation was launched a fault tree analysis was drawn. This analysis identified that the reduced output was due to a restriction in the high-pressure (HP) turbine inlet nozzle. Results of a chemical cleaning the nozzle showed the deposits to be copper. Analysis of the economizer inlet water indicated that the copper was coming from the feedwater system. It was assumed the source of copper was the condenser. The HP turbine was cleaned several times, and each time nine to twelve pounds of copper were removed Root Cause Analysis Chart The Behavioral Justification methodology was used to analyze the problem. Figure 4-10 shows four correctable opportunities. The replacement of hydrazine with Eliminox in 1993 followed an industry wide trend to remove hydrazine, a known carcinogen, from power plant sites. Substitutes were not as effective and many units observed an increase in copper corrosion and subsequent copper carryover in units operated in excess of 2300 PSI-drum pressure. Early in 2000 the unit has returned to hydrazine treatment. Hydrazine levels are adjusting to maintaining a negative oxidizing/reducing potential at the economizer inlet. The condenser is to be replaced with 29-4-C, a high performance stainless steel. It is also planned to replace copper metallurgy feedwater heaters with stainless steel as the need arises.
4-10
These actions will prevent further migration of copper corrosion products into the boiler systems. However, there is copper in the water wall and superheater tubes which can be a major contributor to tube failures. A chemical cleaning of the waterwall tubes was done in Feb. of 2000.
4.4.9 Unit 4 LP Exciter Field Failure 45,229 Mwh. Richard Bagienski was interviewed. After the interview details of the incident were provided. Incident Description Unit 4-exciter failed resulting in a unit trip. Inspection of the brushes and brush holders revealed that some misalignment between brush and brush holder tension arm. The brush arm arced and overheated the collector ring, which, in turn, caused other brushes to arc. The increase in resistance tripped the unit on low excitation. Root Cause Analysis Documentation provided was clear, concise, and complete, and is included in the appendix. The problem was quickly identified and the appropriate actions taken. Access to check brush alignment is difficult. The importance of ensuring correct alignment was not identified on the PM. This can be considered as the root cause. The PM has now been rewritten to include an inspection of the tension arms. The Lost Megawatt Action Plan has extensive Exciter work planned for both Units 4&5 in 2000. 4.4.10 Unit 4 HP Exciter Field Failure 33,500 Mwh Richard Bagienski was interviewed. Problem Description Unit 4 was taken off line after analysis of an exciter field ground alarm indicated that there was a ground in the field about at 50% through the winding. After the unit was taken off line, the exciter field was isolated and the ground was verified. A copy of APS documentation covering this event is included in the appendix. Root Cause Analysis As in the LP exciter event (see 4.9.1), the problem analysis was well done. The exciter was shipped to General Electric for inspection and repair. The inspection revealed that a previous rewind was substandard. In 1992 an alternate vendor was used to do the rewind. This substandard rewind was considered to be the direct cause of this incident, Lack of proper QA/QC is the root cause of the problem.
4-11
4.5
Findings
At each of the stations, the review team consisted of the two EPRI employees and a staff member from the other station involved in this exercise. This was a very positive aspect of the review as experiences and problem solutions at both stations were shared. The opinions/ideas of peers were welcomed. Root Cause Analysis is done by exception, i.e. as and when requested by management. Some staff members were trained in the Root Cause Analysis technique taught by Bobby Jones and Associates Behavioural Justification, but they dont regularly use any formal analysis technique. Root Cause analysis, commonly, consists of group brainstorming sessions, where solutions (often short-term) are formulated and actions allocated. These solutions are not documented and limited follow up is done on allocated actions. There was very little documentation available from which the review team could gather information. Therefore, in an attempt to reconstruct the incidents key information had to be obtained from staff members who had knowledge thereof. E&CF charts or Behavioral Justification diagrams were constructed from this information and causal factor determined. Of the ten lost generation incidents reviewed, only one had some usable documentation Unit 3 Turbine Nozzle plugage. This investigation was started in 1996. In more than 70 % of the reviewed incidents, it was found that staff chose to rather replace/fix/modify equipment or plant, instead of assessing root causes. No documents were provided to support the1999 Lost Generation Action Plans and the Action Plans, therefore, could not be validated as required in the Scope of Work for this project. In almost all the incidents reviewed /reconstructed, the fix repair/replace or modification did align with those recorded in the 1999 lost Generation Action Plan. Preventive/Predictive Maintenance schedules/activities are not implemented, as staff are focused on corrective maintenance, as in crisis management. There is a lack of problem ownership, as responsibility for resolution of problems is not specifically allocated, e.g. the turbine control valve chattering. The long-term health of the plant is neglected. The plant is run in excess of the design maximum continuous rating, e.g. boilers are over-fired to a point where clinkers start to form; continuous sootblowing is then used to remove this build up and this causes erosion which results in long-term damage and/or a tube leak. respectively. Considerably less management staff attended the exit meeting.
Project kick-off and exit meetings were held to confirm scope of work and give feedback
4-12
4.6 Conclusions
The findings of the EPRI team led to the following conclusions being drawn: 4.6.1 Formally-documented Root Cause Analysis is not being done at the Four Corners. The most common reason for not doing a root cause was, we are too busy putting out fires. This made verification of the actions identified in the 1999 Lost Generation Action Plan difficult, however through interviews actions could be verified. 4.6.2 Although no formal analysis is being done, the actions (solutions) identified in the 1999 Lost Generation Action Plans are considered short-term quick fix solutions. 4.6.3 Knowledge of a Root Cause Analysis process was limited even though staff had been trained in an analysis technique.
4.7 Recommendations
4.7.1 A formal structured Root Cause Analysis process needs to be implemented. This process should be developed based on a corporate directive requiring those major incidents (greater than a $X, 000 value to be determined by Corporate), unit trips and a significant unexpected maintenance expenditure be investigated to ensure repeat failures/incidents/trips do not occur. 4.7.2 Staff should be trained/retrained in a Root Cause Analysis technique that includes Human Performance aspects. 4.7.3 The various recommendations identified in Section 4.4 Root Cause Analysis should be evaluated and implemented.
4.8
Appendix 1: Root Cause Analysis Charts
(Excel file) Note: 1. A number of the charts are incomplete as there was limited documentation and the allocated time per incident was constrained. However they do reflect the information given or presented by the various interviewees. 2. The charts need to be read in conjunction with the text in the main body of the report
4-13
6\1\99
6\1\99
6\11- 6\13
6\17- 6\19
6\20
6\21
S .S . Reports #4 control valve chattering
Unit curtailed to 710 M W 's
Unit shutdown for tube leak No attem pt m ade to repair or inspect valve
Unit shut down for tube leak #4 c ontrol valve link age repaired
S .S . reports that #4 valve is not chattering Unit operates with no load c urtailm ent
Turbine planner & Operations M anager verify that #4 c ontrol valve is operational.
Turbine planner notified
V alve clos ed to stop chatter
Load lim iter set to 93%
Lost Opportunity B roken bolts replaced
B rack et m ac hined
No chattering detected
E videnc e of previous failure
72 hour outage required for inspec tion
No failure analys is of bolts
7\1
7\???
7\11- 11\26
11\26-12\2
12/2-12/25
12\25
S .S . reports #4 valve is c hattering
S .S . is notified that #4 valve is s tuc k in clos ed pos ition
Unit operates with load c urtailed
Unit shut down for tube leak repairs
Unit continues to operate with load lim iter at 94%
Unit shutdown due to switchyard failure
197,471 M wh lost due to #4 control valve problem s .
Operating with load lim iter at 94%
Control operator attem pts to open valve.
Load lim iter set to 94%
#4 c ontrol valve chec ked and found to be operational
Operators unaware valve c hec ked and found to be operational
Unit is run at overpressure to m eet load requirem ents . No written verification No investigation initiated
Governor pilot valve rebuilt with non-OE M rec onditioned parts
Curtailm ent lifted after startup - Unit at full load
Figure 4-1 Unit 5, #4 Control Valve 197,471 Mwh
4-14
Lost Mwh due to Scrubber O utages
Recycle pump failures
Rubber-lined piping failure
High level of slurry in scrubbers
S truc tural failure of Dentis t B owl
Foreign objec ts c ause im peller imbalanc e
P um p c avitation
I.D.
Cold wall effec t pipe blis tering

C.O . Material selection
Overflow plugged with debris
No level indic ation

C.O .
B lowdown not s uffic ent
Teles c ope jam m ed
Refac tory falling off of s k irt

C.O .

C.O .
Tight c learanc es
C.O .
Debris between pipe clearanc es

N.C.
C.O . 1. Replace skirt w/o refactory , Screens, W hite iron pumps
Figure 4-2 Unit 3 Scrubber Problems 33,446MW/Hrs.
4-15
B listering in rubber-lined piping.
Rubber separates from piping
Seperated rubber plugs normal overflow
S crubber debris plugs em ergency overflow
*
A
Gum rubber very sus ceptible to c old wall effect Gum Rubber vers us Chloro B utyl
Rubber piece 20" diam eter
Pipe diam eter is 14"
*
Norm al overflow is used for level c ontrol
P ipe diam eter is 14" Overflow pipe is long and alm os t horiz ontal
Overflow pipe is long and alm os t horiz ontal
High water level block s gas flow
dP across scrubber increased suddenly
Fans were fully loaded - 100%
Unit 3 trip due to high furnace pressure
No level ins trum entation
*
*
Figure 4-3 Unit 3, Scrubber Problems 33,446MW/Hrs.
4-16
Refac tory applied to m etal s kirt
Refac tory falls off m etal s kirt
Refac tory lodges in im peller
Rotor bec om es im balanced
Purpos e of refac tory is not k nown
No s tuds or m etal hangers d Bas e m etal c orrodes
No pum p s uction s c reens
Im peller des ign Urethane coated Free diam eter pas s age is 5"
Pum p fails
Unit derated
Loss of M W due to rec yc le pum p failures
Slinger rings wear out
E nvirom ental regs require pum p to be in s ervic e
P ac k ing fails
Figure 4-4 Unit 3 Scrubber Recycle Pump Problems 33,446MW/Hrs.
4-17
B attery PM perform ed
E lec tricians vis ually inspect batteries
DC inverter output voltage drops .
DC voltage dropped to les s than 108 V
B E TA alarm panel res et
Incorrect voltage m eter used to test batteries
*
Cry stal growth in batteries A C power to inverter was interrupted B atteries were bad
*
B E TA panel s ensitive to input voltage
Indication of end of life Data not c hec k ed or analy z ed
*
Replac em ent batteries were not ordered
B ooster fan trips
Unit 4 trips due to los s of boos ter fans
New batteries ordered
Figure 4-5 Unit 4, Loss of Booster Fans, 38, 413 MW/Hrs
4-18
UNIT 3, 3B P ULV E RIZE R E XCE S S IV E P INION GE A R FA ILURE S
Foundation was brok en
Im proper alignm ent between gears
P inion/B ull gears got too hot
Im properly m anufac tured gears
E x c ess ive vibrations & stres s es induc ed to foundations

C.O. A s s ure f oundation integrity at eac h pinion replac ement .
Im proper ins tallation of gears P roc edure was inc orrec t

C.O. Rev is e proc edure to ac c ommodate dif f erenc es betw een pulv eriz ers
M ec hanic al failure of m ounting bolts
Lube oil sy s tem failed Lube oil pum p failed I.D..
Tim ely c heck of pinion/bull gears not perform ed C.O.
Im properly m anufac tured gears were k nowingly ac c epted C.O.
Operarions w as not perf orming c hec ks . Maint. began c hec king f or s urv iv al. Now us ing better tec hnology . Res tored lubeman pos ition. Improv ed ins trumentation.
Figure 4-6 Unit 3 Pulverizer Pinion and Bull Gear Failures, 38,881 MW/Hrs.
4-19
U3 RE HE A TE R B A CK P A S S FLY A S H E ROS ION
Unquantified effec ts of unit m odific ations in the boiler
High as h c ontent in c oal
High flue gas veloc ities
P oor as h dis tribution
B oiler dam per operation
Furnac e s m all for heat releas e required
Clos e tube s pac ing in bac k pas s
Reheat & s uper-heat outlet tem perature c ontrol
A s h m om entum c onc entrates partic les in reheat bac k pas s and S .H. boundary wall.
B oiler dam per operation
B oiler des ign
Reheat & s uper-heat outlet tem perature c ontrol
P oor m ill perform anc e, finenes s & dis tribution
Fuel level c ontrol C.O.
P oor m ill m aintenanc e C.O.
B all c harge im proper C.O.
Figure 4-7 Unit 3 Backpass Flyash Erosion 39,378 MW/Hrs.
4-20
UNIT 4 AIR P REHEA TE R FA ILURE
Cold side bask ets fell out
B ask et fas tners failed
Fram e around AP H bas ket failed
Acc elerated therm al fatigue of m etal
Therm al fatigue of m etal.
A PH rotation speed was set too low. C.O.
AP H was being rem oved from service on cold s tartups to place baghouse in s ervic e earlier. C.O.
Figure 4-8 Unit 4 Air Preheater Failure 166,744 MW/Hrs.
4-21
Fly as h E ros ion in B ac kpas s of Units 4&5
High fly as h levels in B HP c oal
Loc aliz ed high velocities in flue gas
B HP c oal is low c os t option N.C.
High flue gas volum es exis t for boiler des ign N.C.
Flue gas has poor dis tribution I.D. P erform flue gas veloc ity distribution tes ts
Unit overfired for ex tra M W s
Ex c es s air us ed to prevent pluggage
Figure 4-9 Units 4 and 5 Backpass Flyash Erosion
4-22
UNIT 4 A IR P RE HEA TE R FA ILURE
Cold s ide bas k ets fell out
B as k et fas tners failed
Fram e around A PH bas k et failed
Ac c elerated therm al fatigue of m etal
Therm al fatigue of m etal.
A P H rotation s peed was s et too low. C.O.
A P H was being rem oved from s ervic e on c old s tartups to plac e baghous e in s ervic e earlier. C.O.
Figure 4-10 Unit 3 HP Turbine Nozzle Plugging, 100,628 MW/Hrs.
4-23
MW Hour Lost Area Unit 5 #4 Control Valve Unit 4 Air Preheater Failure Unit 3 Turbine Copper Fouling Unit 4 LP Exciter Field Ground Unit 3 Boiler Tube Leaks Unit 4 Flyash Erosion Unit 4 Battery Failure Units3 Pulverizers Unit 4 HP Exciter Field Ground Unit 3 Scrubber, and Recycle Pumps Total
MW/Hrs. Lost 197,471 166,744 100,628 45,229 39,378 39,288 38,413 34,881 33,500 33,446 728,978
4-24
UNIT # Type Furnace 1natural circ/front wall fired 2natural circ/front wall fired 3natural circ/front wall fired 4super critical/front & rear fired 5super critical/front & rear fired
Fuel MFG. Start up Date Description Riley Stoker 1962sub-bituminous Riley Stoker 196320-25 % ash Foster Wheeler 19649200 btu/ft3 B&W 1969 " B&W 1970 "
Oper Pressure MW Rating Pollution Control 1800 170wet scrubbers 1800 170wet scrubbers 2000 220wet scrubbers 3500 755baghouses/wet scrubbers 3500 755baghouses/wet scrubbers net
4-25
CHOLLA PLANT 1999 Lost Generation Incident Review 4.9 Executive Summary
In an increasingly competitive environment, the ability to reliably produce the lowest cost power is essential for the long-term survival of a utility. With this vision in mind Arizona Public Services Company (APSC) requested EPRI to review the 1999 Lost Generation Action Plans of Cholla Plant. The intent of this review was to provide assurance or not that the true root causes of lost generation have been identified, and appropriate action plans are in place to address these causes. The reviews took place between 19 June to 30 June 2000 at the Cholla Plant. Eleven incidents were reviewed, of which three had some documentation backing up the plants root cause analyses. The review team, two from EPRI and one from Cholla and one from Four Corners used a series of interviews with staff to re-construct the incidents. A number of staff involved with the incidents were interviewed. Two methods, Behavior Justification and Event and Causal Factor Charting were used to re-construct the incidents. Diagrams/charts were constructed from information that depended, almost entirely, on the interviewees long-term recollection of the incidents. Verification of information was difficult, as there was a lack of formal, incidentspecific documentation and because of the time that had elapsed between the incidents and the analyses. In many incidences, perceptions and memory of what happened, and in what sequence, varied, emphasizing the importance of performing root cause analysis in a timely manner, and documenting the results. Based on the information obtained during the two weeks at the plant, the EPRI team uncovered many opportunities for improvements in operations, equipment maintenance, inter-departmental communications at all levels of management, human performance, root cause documentation and engineering response. The Lost Generation Action Plans as developed by the plant need these improvements to ensure that they have a sound base. Documentation of incident investigations, no matter how minor, provide an audit trail to ensure a historical record of what has occurred at the plant, how it was analyzed, what the causes of the problem were and how they were solved. Lack of this audit trail often leads to repeat incidents because staff are not aware of what has happened previously/how problems were solved. This often leads to short-term, inadequate solutions, rather than structured, long-term and cost-effective solutions. No formal structured root cause analysis process exists. Some plant employees have been trained in a root cause analysis technique, but its use is limited and the results are not documented. The staff members interviewed were all cooperative, forthcoming and interested in using a root cause analysis tool to solve long-term problems at the plant. Each incident reviewed has a number of recommendations. These recommendations need to be evaluated and implemented. The main recommendation is that a formal structured approach to incident investigation needs be developed and implemented. A corporate directive should be developed to give guidance on the requirements. As a start all unit trips/forced outages, load loss 4-26
and failures to meet expected output should be subjected to thorough root cause analysis. Incidents where plant damage has occurred should also be investigated and any significant capital expenditure needs to be backed up with a justification that is based on a thorough root cause analysis. Training a Root Cause Analysis Technique needs to be implemented.
4.10 Introduction
In an ongoing effort to improve reliability and reduce maintenance costs in the APS system, EPRI was contracted to review the 1999 Loss Generation Action Plans at both the Cholla and Four Corners plants. The main objectives of this review were: 4.10.1 To provide assurances that the true root causes of lost generation have been identified, and appropriate action plans are in place to address these causes: 4.10.1.1 determine and verify the true root causes of technical and/or production problems, and 4.10.1.2 identify preventive actions to effectively manage the identified problems and minimize the affect on availability. The team consisted of, two from EPRI and two from APSC, one from Cholla Jack Stant and one from Four Corners Dan Wilson. Jack Stant Cholla participated in both reviews. The review took place between June 19 though June 30 at the Cholla Plant. Over fifteen staff members were interviewed regarding the ten incidents reviewed. These incidents represented a loss of 481,455 Mwh. The incidents selected represented the major Mwh losses and were prioritized by the respective plant management teams. The selection was based on their significance, available documentation, and availability of staff having knowledge of the incidents. The Behavioral Justification method of analysis was used to analyze a number of the incidents APS personnel have been trained in this method. Three of the ten incidents had adequate documentation. Events and Casual Factors (E&CF) Charting was used as an alternative method. In this method the sequence of events is charted chronologically, from left to right, and information/conditions relating to events in the sequence are charted directly underneath the events. Events are shown as text within rectangles, joined by arrows, ending in a circled unwanted event (reason for the analysis). The information/conditions are shown in ovals, joined by lines that show their association. Dotted ovals or rectangles indicate information that still needs to be verified. Causal factors are identified by asking the question Which of these factors, if eliminated, would have either prevented the incident from occurring or at least minimizing its impact? From these causal factors Root Cause are determined. Root Cause is the cause/s which, if eliminated, would prevent this event recurring or minimize its impact.
4-27
4.11 Plant Description

4.11.1 The Cholla power plant is situated in NE Arizona and consists of four corner fired units, manufactured by Combustion Engineering and went into service between 1962 and 1981. They are all drum units operating at a pressure of 2000-PSI. Unit 1, rated at 110 Mw, has SO2 scrubbers. Unit 2 rated at 245 Mw is equipped with SO2 scrubbers and an absorber. Unit 3, with a precipitators and is rated at 260 Mw. Unit 4, rated at 380 Mw, is fitted with a precipitator and an absorber. Units 1, 2, and 3 have Westinghouse turbine/ generator and Unit 4 has a GE turbine/generator train.
4.12 Incidents Investigated

The incidents selected and the Mwh lost are listed in the following tables:
Table 4-1 Cholla - Megawatt-hour loss summary report-fourth quarter 1999 Megawatt hour loss area Unit 2 Waterwall Hydrogen Damage Unit 4 Scrubber Outage Unit 4 Air Pre-heater Blockage Unit 4 High Opacity Shutdown Unit 1 Mechanical Flyash Collector Unit 2 A ID Booster Fan Motor Failure Unit 2 Cable Tray Fire Unit 1 Turbine Thrust Trips Unit 1 ID Booster Fan Bearing Failure Unit 4 Air Pre-heater Guide Bearing Fire Total Mwh lost 188,944 76,917 48,740 42,731 41,146 30,464 28,899 9,971 7,544 6,099 481,455
4-28
4.12.1 Mechanical Dust Collector Retrofit 41,146 Mwh. Those interviewed were: El Pahi, Dennis Stewart and Mike Machusak Problem Description The original Dust Collector on Unit 1 was becoming too expensive to maintain because of the increasing number of failures also because of its size, number, and location of tubes it was also difficult to maintain. The design as such made it difficult to address equipment wear problems. In 1997, an Engineering Service Request (ESR) was written in order to find a permanent solution to replace the dust collectors. In order to replace the collector during the March 1998 the project was placed on a fast track. This was the first opportunity as the unit outage frequency had been changed from two to three years. This allowed three-months between Request for Proposals and installation of the equipment. Because of this time constraint, no modeling studies were required or performed prior to installation. Performance tests showed that the new collector met opacity requirements, but failed to meet the pressure drop (dP) specification. The dP was 7 inches versus the 3.5 inches specified. In May of 1999, two modifications based on modeling studies made by the manufacturer were installed on the dust collector at the vendors expense. Subsequent tests showed slight improvement to the dP, but it still did not meet guarantee. The fans could not accommodate the additional pressure drop. As a consequence the unit was de-rated by 5 Mw. Root Cause Analysis Chart The E&CF chart in Figure 4-11 identifies a number of causal factors. These are: Lack of long term planning. The poor performance of the dust collector was known some time before it was decided to proceed with a retrofit. The Engineering Services Request to resolve the collector problems was written in late 1997. Shortly there after, the scheduled outage interval was increased from two years to three years. This increase in frequency changed the priority to perform the modification during the 1998 Spring outage. Failure to do so would result in a three year delay of the project, and engineering studies predicted that the collectors remaining life was less than three years A bid package was put together using inlet design and pressure drop specifications, and sent out in January of 1998 to three vendors, with a deadline of three weeks for responses. Due to time constraints, no modeling studies were required. ABB, the OEM for the collector, did not respond. The bids were not thoroughly analyzed and or evaluated. No design review was performed to validate the suppliers technical guarantees. Contract management was questionable. After installation, the collector failed acceptance testing due to a high-pressure drop of seven inches. In May 1999 the contractor modified the plant based on a modeling study. Test showed a decrease of 1.6 inches in dP to approximately 5.4 inches against a guarantee of 3.5 inches. This additional dP was greater 4-29
than the capability of the induced fans at full load. This resulted in a 5 Mw de-rate on the unit. No additional requirements were put onto the contractor. A feasibility study is being undertaken to either replace the fans, or changing the blade design to overcome the collector pressure drop and return the unit to full load capability
The above represent the casual factors that if removed would have either prevented this incident from happening or at least would have minimized its impact. The root causes would need to be established from these causal factors by answering the question What management processes should have been in place and enforced to ensure timely and cost effective project management? 4.12.2 Unit One ID Booster Fan Bearing Failures 7,544 Mwh. No documentation was provided however El Pahi and Jim Fogel gave some background to this bearing failure. Incident Description Repeat booster fan bearing failures have occurred. The bearing are water-cooled sleeve bearings. The failure mode has generally been overheating caused by lack of adequate lubrication. The ash-laden gas that is conveyed by the fan erodes the fan blades causing an imbalance in the fan rotor. This imbalance results in increased vibration. The vibration is transmitted through the bearing housing to the cooling water supply housing, cracking the inlet nipple. This allows water to displace the oil in the bearing sleeve, causing the bearing over heat and fail. Root cause Analysis Chart Figure 4-12 shows the E&CF chart drawn from the information obtain from the interviews. Plant personnel assumed that the increased fan vibration due to flyash erosion resulted in accelerated metal fatigue of the cooling water supply nipple. The following actions have been taken to resolve the problem: Cooling water has been replaced with an air- cooled system. In 1998 the worn/damaged bearing housing on A fan was replaced and the fan shaft repaired. A newly installed vibration monitoring system now trips the fan at seven mils. Installation of an external lube oil-cooling system.
The above actions have so far halted the failures. However, these actions only address the symptoms and/or give additional protection. None of the above address the causes, which are vibration, caused by fan imbalances, caused by erosion caused by excessive ash-laden gas.
4-30
4.12.3 Unit one Turbine Thrust Trips 9,971 Mwh. Limited documentation was made available. George Ross and Tim Vachon provided some background to this incident. Incident Description One thrust trip setting was desired for all units at the plant. However, Unit 1 Turbine always ran with a higher thrust of plus or minus 10 mils compared to an expected value of plus or minus 6 mils. This was aggravated by the fact that the Bentley system and manual methods did not agree on thrust position. This resulted in numerous spurious trips of the Unit 1 turbine. During the spring 1998 outage, the abnormally worn thrust bearing was replaced. The turbine was thoroughly inspected, and all possible contributors to the abnormal thrust were examined. New thrust probes were also installed and the thrust was statically calibrated. 2 mils of thrust were available for adjustment. On returned to service, the unit tripped on high thrust. For some unknown reason the thrust zero point had moved to -15 mils from the cold set point. While off load the probes were adjusted and the unit returned to service. The unit again tripped on high thrust. While off load the probes were again adjusted. Several trips occurred and adjustments made to the trip limit until a trip setting was obtained where the unit did not trip on high thrust. The probes could not be adjusted online. The above actions have eliminated the trips however the turbine now runs with a thrust value of between 12 to 15 mils which is a concern to plant staff. Root Cause Analysis Chart The E&CF chart in Figure 4-13 identifies the causal factors for the repeated trips after the new probes were fitted. The reason for the high thrust in the first place still needs to be established. There are three causal factors identified. Incorrect calibration of the newly installed probes. Inexperience/lack of know how and insufficient data are the main reason for the incorrect calibration. The decision to limit the meter range in order to maintain plant standards. Since the cause of the excessive thrust had not been identified, a conservative approach was taken to protect the turbine from damage. The inability to adjust the probes while the turbine is online. Conservative adjustments were made to the thrust trip set points each time the turbine tripped since a large adjustment could have had serious consequences.
Although the trips have been eliminated, there is concern over the turbines excessive thrust readings. The investigation is ongoing and listed in the Lost Megawatt Action Plan. Those interviewed revealed an in-depth knowledge of the problem and are committed to its elimination.
4-31
4.12.4 Unit Two A Booster ID Fan Motor Failure 30,464 Mwh. Those interviewed were: Andre Norwood, Lee Holly, Jim Wilber, Allen Bushman, Jack Stant and Joe Lacey Incident Description In June of 1999 two attempts were made to start Unit 2 A Booster ID fan following a planned unit outage. Both attempts resulted in an instantaneous ground fault. An investigation revealed that a power supply cable that had been disconnected for testing purposes during the outage had not been reconnected. The fan motor insulation was tested acceptable after reconnection of the cable and placed in service. While in service the vibration level increased. On July 11, the unit was returning to service after a tube leak repair and the fan was started six times for washing and balancing. On the sixth start the motor failed and burned-out. The badly damaged motor sent off site for repair. Root Cause Analysis Chart The E&CF chart in Figure 4-14 shows the sequence of events. The analysis of this incident was difficult and required extensive interviews and review of available documentation. The following needs to be read in conjunction with the events mapped in Figure 4-14 During the 1999 spring outage, APS Generation Engineering personnel performed extensive Doble testing on the Units large electrical equipment. Test results on the A booster fan motor showed some deterioration of the motor cables. In order to quantify the problem, the testing procedure was altered. This consisted of disconnecting the cables at the breaker panel, which is not normally done. No one in the plant was informed of this change in the scope of work. Manpower scheduling problems necessitated using four different contract electricians on the job. The breaker cabinet was returned to service with the middle phase disconnected. The Doble test showed that the cable had deteriorated and needed to be replaced during the next planned outage. On June 18, operations attempted to start the fan for balancing. The fan tripped on instantaneous ground fault. No evidence of damage could be found. The protection flags were reset and a second attempt was made to start the fan. This time it tripped on overload. The production supervisor contacted electrical maintenance personnel to investigate the fan failure. A motor insulation test indicated that the motor had an open phase. At this point, the Outage contract electrical foreman, who was still on site, remembered that the cables had been disconnected during the outage. It was then discovered that the cable was still disconnected. After returning the cables to their proper configuration, the motor was tested and found acceptable, and the fan returned to service. No Doble testing was performed on the motor or the cable. Fan balance and cleanliness was verified. The fan operated for seventeen days with no problems. On July 6, the fan inboard bearing vibration increased to 3.5 mils horizontal and 2.5 mils vertical. The outboard bearing remained constant at 1.9. Plant personnel assumed that fan vibration was the result of deposition of material on the fan blades as a result of scrubber de-mister failure. However,
4-32
the de-mister problem had been addressed during the spring outage and appeared to be functioning properly. There was no detailed analysis done on the increasing vibration. On July 10, the unit was shut down for tube leak repairs, and the fans removed from service once the boiler was cooled. On July 11, based on an increase in vibration levels, the fan was washed. The fan was started six times between 19:20 and 00:06, a period of four hours and fourteen minutes. Analysis of the start and stop times table in figure 4-14 shows that the plant was in compliance with the industry standard of not starting a motor more than three times in any one hour period. From previous washing experience a decrease in the vibration levels is noticed after the first washing cycle. This was not the case with this fan - washing of the blades had negligible effect on vibration levels. On the sixth start, the motor failed immediately.
The following represents the causal factors i.e. those causal factors that if were not present would have either prevented this incident from occurring or at least minimized its impact. The original Doble testing work scope was expanded without notifying the Plant Clearance Authority. Because of this, no one on the plant staff was aware that the cables had been disconnected for testing. The two ground faults may have significantly contributed to the failure. However, this cannot be accurately determined as the motor was repaired without any failure analysis being done on it. Doble testing was not performed on the fan motor following the two unsuccessful fan starts on June 18. Doble testing is a more rigorous test and may have revealed deeper problems. No analysis of the increased vibration levels. It was assumed that ash was lodged on the fan blades. This assumption was made in spite of the fact that the de-mister problem had been corrected, and the vibration increase was on the inboard bearing only. Excessive fan starts. A careful look at the fan start timetable shows that the production supervisor was following the guidelines regarding the time limit on fan starts. This guideline is part of the plants oral history and could not be found in any written procedures.
4.12.5 Unit Four Air Pre-heater Pluggage 48,740 Mwh. The following were interviewed - Layne Miller, Jack Stant, Pete Beltran and Mark Soloman Problem Description In September of 1999, Unit 4 was removed from service due to a blocked air pre-heater. The first indications of blockage were observed in late 1998 Plant instrumentation and alarms indicating blockage were either ignored or misdiagnosed by both operations and maintenance staff. The blockage continued unabated until the unit was force to be taken out of service. During this period two opportunities to inspect the air pre-heater and or check instrumentation were missed. An inspection of the air pre-heater revealed that the sootblower nozzles had broken off, rendering the sootblower system ineffective.
4-33
Root Cause Analysis Chart The E&CF chart in Figure 4-15 identifies a number of causal factors resulting in this incident. In mid-1998 no investigation was undertaken or initiated when plant personnel noticed that no opacity spikes occurred during air pre-heater sootblowing. This is a normal occurrence when sootblowing the air pre-heater in operation. An investigation would have lead to the realization that the air pre-heater was blocking up. In October of 1998, operations wrote a Work Order on high dP on Unit 4 air pre-heater B wheel. Maintenance diagnosed it as a transmitter problem. In February of 1999, operations wrote another Work Order on high dP on both A and B wheels of Unit Four air pre-heater, specifying potential blockage of the air pre-heater. Again, the diagnosis was faulty transmitters. Plant engineers do not trend or review data on the pre-heater. Trending the dP would have highlighted the blockage earlier.
The air pre-heater soot blowing nozzles have been added to the units inspection routine. It is also planned to install pre heater airside soot blowers. This will solve the mechanical problems identified in this root cause analysis. However, the habit of ignoring or not repairing instrumentation with known problems (see Figures 4-17 and 4-19) is a serious trend that needs to be addressed. 4.12.6 Unit Four Scrubber Problems 76,917 Mwh. Those interviewed were Jack Stant and Pat Reynolds Problem Description Unit 4 stack SO2 levels were increasing to unacceptable levels. Examination of operating parameters showed low flue gas flow through the scrubbers. The high dP across the de-misters confirmed that some blockage was taking place. During the spring 1999 planned outage, Unit 4 scrubber was inspected to gather data to be used for analysis. In May of 1999 the unit was removed from service and extensive repairs and modifications made to the scrubber. In the spring of 2000, proper nozzles were installed on Unit 4. Post outage SO2 and dP are within limits. Root Cause Analysis Chart Figure 4-16 represents the analysis completed by plant staff using the Behavior Justification Root Cause Analysis methodology. All three identified correctable opportunities were addressed during the outages. The de-misters were replaced with a type less prone to plugging. Correctly sized nozzles were installed.
4-34
The control scheme for spraying was improved in order to give Operations control over spraying frequency and duration.
This analysis was simple, but effective in identifying the problems and solutions before shutting down the unit for repairs. Advantage was taken of opportunities to inspect the equipment for further information. The approach was well planned and coordinated. This is an example of how plant problems/incidents should be managed. 4.12.7 Unit Four High Opacity Shutdown 42,731 Those interviewed were: Jack Stant, Elbert Watts, Junior Ward, Layne Miller, Mark McKenzie, Pete Beltran and Mark Soloman Problem Description Due to a strike at the McKinley mine, coal was purchased from an alternate supplier. On 5 June 2000, 5,500 tons of this coal was bunkered to Unit 4. Unknown to plant staff, the coal was a 30% ash coal instead of the expected 22%. The fly ash removal system was unable to cope with the increased ash loading, and Transformer Rectifier (RTs) sets in the precipitator began to short out because of high ash hopper levels. Unit load was decreased to control opacity, but was ineffective and opacity reading continued to increase. On June 11 the unit was taken off load to prevent further violations of opacity limits. Root Cause Analysis Chart Although this event was not in the original scope of work, it was included at the request of Cholla plant management. An E&CF chart was drawn. Figure 4-17 shows the charted sequence of events. Four causal factors were identified The coal quality directive as written did not address the ash content of the coal received. APS purchased coal from an alternate mine that had previously supplied coal. Ash content is not addressed in the specifications, however the mine informed APS that coal in storage had an ash content no greater than 22% in fact the coal contained 30 % ash. The practice of verifying ash hopper empty was no longer done. It was customary for operators to check hopper level prior to starting the emptying sequence. This practice had not been passed onto the new operators and consequently did not get done on a routine basis. Hopper level indication was inoperative. The level indicator were considered unreliable and therefore were neglected. Preventive maintenance on the Nuva Feeders had been suspended and as a result numerous failure occurred simultaneously, which could not be kept up with.
4.12.8 Unit Two Waterwall Tubes Hydrogen Damage 188,944 Mwh. Those interviewed were: Dan Nass and Kim Belknap 4-35
Problem Description In the early nineties several poor quality water excursions, caused by condenser tube leaks, occurred on Unit 2. This, coupled with a lack of chemical cleaning, resulted in internal tube scaling - hydrogen damage. In 1999 fifteen tube failures occurred, of which, eleven were identified as Hydrogen Damage. Root Cause Analysis Chart Figure 4-18 represents the root cause analysis completed using the Behavior Justification methodology. The unit was operated with high chlorides, which entered the boiler via condenser leaks. Poor control of the demineralizer regeneration process contaminated the makeup water with acid, reducing boiler pH. The boiler was operated for hours with a pH of seven. The practice of draining or chemically cleaning the boiler following severe pH excursions was not followed. The unit operated with high dissolved oxygen, producing high levels of corrosion products in the feedwater system, which eventually deposited on waterwall tubes. The chemical cleaning in 1997 was the first in eleven years.
The effects of the poor chemical control in the early nineties was felt in 1996 when the unit began experiencing tube failures due to hydrogen damage. Appropriate action has and is being taken by the plant staff. The condenser has been replaced. Makeup water supply has been contracted out and is of a good quality. EPRI guidelines are followed and the unit chemistry has been improved. Waterwall panels have been replaced. Management has placed a high priority on maintaining an effective boiler water quality control program. 4.12.9 Unit Two Cable Tray Fire 28,899 Mwh. Those interviewed were: Jack Stant, Pete Beltran and Walt McMillan Problem Description In March of 1999, scaffolding boards lying on top of a cable tray located above a pulverizer hot air duct ignited. The fire melted the leads to the Unit 2 service water pumps, which supply cooling water to Units 2 and 3 soot blower air compressors. The air compressors had not tripped in spite of high temperatures and were manually removed from service by the Shift Supervisor to prevent further damage. The Shift Supervisor ordered Unit 2 to be shut down and load on Units 3 and 4 to be reduced. While lowering load on Unit 4, it tripped off line due high furnace pressure resulting from damper-control low air pressure. Extensive damage was done to Unit 2 soot blower air compressor because the wires on the high temperature trip were shorted out and not repaired since replacement was being considered.
4-36
Root Cause Analysis Chart The Behavior Justification analysis figure 4-19 performed by plant staff in 1999 was reviewed. The E&CF charting method figure 4-20 was also used as a comparison. Both analyses are lengthy and contain a number of causal factors. The following discussion will be based on the E&CF chart analysis. Scaffolding boards had been left on top of the pulverizer hot air duct for an unknown period of time. The duct conveys combustion air at a temperature of 500 degrees F to the pulverizers. The location is near the coal transfer point and coal dust settles onto the ductwork. The area is enclosed. The fire melted the leads to both service water pumps, which led to loss of cooling water to the soot blowing air compressors (SBAC). Unit 2 service water pumps provide cooling water to both Units 2 and 3, as the Unit 3 pumps had been abandoned for ten years. The Unit 4 Shift Supervisor discovered the loss of service water cooling pumps, as the supervisor for Units 1,2, &3 had assumed the role of Fire Commander. He immediately ordered the shut down of Unit 2. The Supervisor then manually took the Unit 2 and 3 SBACs off line. They had not tripped on high temperature since the high temperature trip wires had shorted and not been repaired. An alternate source of cooling water, using the fire water system, could not be located because the operators were not familiar with the plant layout. Unit 1s soot blowing air system was isolated and load reduced on Units 3 and 4 to compensate for loss of soot blowing air. Unit 4 SBAC could not be used as it was down for repairs. The Assistant Operator (AO) attempted to isolate the Unit 4 air systems but was unable to do so, as there was no procedure for him to follow, and he was inexperienced. The AO was manually closing 4B ID Fan damper, but 4A damper had insufficient instrument air pressure to follow. The resulting high furnace pressure tripped the unit.
An analysis of this incident reveals that if the scaffold plank had not been left on the hot duct, this incident would not have occurred. Therefore the primary root cause of this incident is poor housekeeping and safety culture. Eighteen months prior to this incident a similar fire, involving scaffold boards, had occurred on Unit 4. The other seven causal factors identified contribute to the loss of Unit 4 and the extensive damage incurred by the air compressors. As a result of the BJ analysis performed in 1999, an action plan was put together and is reviewed below. The shorted wires on the air compressors were immediately repaired. The trip system was replaced. The Unit 3 service water pump was returned to service. However, the pump supply is from the Unit 3 cooling tower and is too high a temperature for use as a cooling water source for the air compressors. The project is now with engineering. Perform a monthly combustibles inspection of the units. This was done monthly in the second half of 1999, but has only been done once in 2000.
4-37
Write a procedure for isolating Unit 4 from Units 1,2, & 3. A rough draft was written in March of 1999. As of June 2000, no further action has been taken. The Unit 4 ID fan damper drives have been replaced with Jordan drives, and no longer depends on instrument air. In order to ensure a 30-day turn-around on sootblower air compressor overhauls, rebuild kits have been purchased.
4.12.10 Unit Four Air Pre-heater Guide Bearing Fire 6,099 Mwh. Those interviewed were: Jack Stant, Dan Hyde, Pete Beltran, Walt McMillan and Bryant Peterson Problem Description Unit 4 had been on line 27 hours following a unit shutdown to perform an air heater wash when a high temperature alarm on the 4A Air Pre-heater guide bearing in the control room was received. The AO who investigated the alarm reported a fire and the fire alarm sounded. The Shift supervisor and AO attempted, five times, to extinguish the flames. At this point, the SS ordered that the shut down of the unit. The Joseph City fire department arrived on site but could not make radio contact with the Supervisor. Members of the station fire team arrived and put the fire out. The bearing was visually inspected and found fit for purpose. The unit was returned to service. Root cause Analysis Chart Figure 4-21 shows the E&CF chart used to analyze this incident. The high bearing temperature alarms, which prompted the AO to inspect the guide bearing, was the result of fire in the vicinity to the thermocouple. The area around both A and B guide bearings is packed with about ten inches of flyash, which on its own, is not considered to be a fire hazard. Over a period of time, the flyash had become soaked with bearing oil from oil spillage and leaks.
Three conditions are necessary for a fire. Fuel -The oil, which had soaked the flyash, had a flash point of 440 degrees F. Temperature- the air passing through is 560-580 degrees F. Oxygen- The SS reported that the flames were coming from up under the guide bearing, which was covered with flyash. Only when the nozzle of the fire extinguisher was inserted all the way into the flyash was the fire put out.
A plant walk down by the review team revealed the exact same conditions, which had led to the fire, still existed. In an attempt to determine the temperature of the flyash, it was discovered that
4-38
the bearing seal had failed and hot gas was being blown out, and was providing the heat as well as the oxygen source. Approximately fifteen feet from the bearing was a drum of oil, two buckets containing oil, oil soaked rag and gloves. As discussed in incident 4.9, poor housekeeping is the root cause of this incident. The fact that the exact same conditions still existed, a year later, shows a lack of follow through/follow-up by supervisory staff and the Fire Chief as they were given that responsibility.
4.13 Findings
At each of the stations, the review team consisted of the two EPRI employees and a staff member from the other station involved in this exercise. This was a very positive aspect of the review as experiences and problem solutions at both stations were shared. The opinions/ideas of peers were welcomed. Root Cause Analysis is done by exception, i.e. as and when requested by management. Based on the information provided, Cholla management requested more analyses than Four Corners. These are undertaken by the Operation Technical Advisor Jack Stant Many staff members were trained in the Root Cause Analyses technique taught by Bobby Jones and Associates Behavioral Justification, but they dont regularly use any formal analysis technique. Root Cause analysis, commonly, consists of group brainstorming sessions, where solutions (often short-term) are formulated and actions allocated. Problem solutions are not well documented and in some case not documented at all. Allocated actions/solutions to brain storming analysis are rarely followed up There was very little documentation available from which the review team could gather information. Therefore, in an attempt to reconstruct the incidents key information had to be obtained from staff members who had knowledge thereof. E&CF charts were constructed from this information and causal factor determined. Of the ten lost generation incidents reviewed, three were adequately documented using the Behavioral Justification technique. In more than 50 % of the reviewed incidents, it was found that staff chose to rather replace/fix/modify equipment or plant, instead of assessing root causes. In almost all the incidents reviewed /reconstructed, the fix repair/replace or modification did align with those recorded in the 1999 Lost Generation Action Plan. Preventive/Predictive Maintenance schedules/activities are not implemented, as staff are focused on corrective maintenance, as in crisis management. There is a lack of problem ownership, as responsibility for resolution of problems is not specifically allocated, e.g. Cable tray/duct fires. Project kick-off and exit meetings were held at both plants. At Cholla the management team showed a keen interest and participated.
4-39
4.14 Conclusions
The following conclusions have been derived from the Findings above: 4.14.1 Formal documented Root Cause Analysis is not being done on all major incidents. The most common reason for not doing a root cause was, we are too busy putting out fires. This made it difficult to verify the actions identified in the 1999 lost Generation Action Plan. 4.14.2 Although no formal analysis is being done, the actions (solutions) identified in the 1999 Lost Generation Action Plans are considered adequate short-term solutions. Long term solutions, come from an in-depth root cause analysis of the incident and/or problem. 4.14.3 Knowledge of a Root Cause Analysis process was limited even though staff had been trained in an analysis technique.
4.15 Recommendations
4.15.1 A formal structured Root Cause Analysis process needs to be implemented. This process should be developed based on a corporate directive requiring those major incidents (greater than a $X, 000 value to be determined by Corporate), unit trips and a significant unexpected maintenance expenditure be investigated to ensure repeat failures/incidents/trips do not occur. 4.15.2 Staff should be trained/retrained in a Root Cause Analysis technique that includes Human Performance aspects. 4.15.3 The various recommendations identified in Section 4.4 Root Cause Analysis should be reviewed, evaluated and implemented.
4.16 Appendix 2
Root Cause Analysis Charts (Excel file)
Note: 1. A number of the charts are incomplete as there was limited documentation and the allocated time per incident was constrained. However they do reflect the information given or presented by the various interviewees. 2. The charts need to be read in conjunction with the text in the main body of the report
4-40
Unit 3 HP Turbine Nozz le Copper Deposition
Copper in m ain steam
Copper in c ondens ate
Copper in feedwater
Switched from hy draz ine to Elim inox in 93 C.O. Returned to hy draz ine in early 2000 Copper levels in E con. Inlet reduced
Drum pres sure 2250-2300 P S I N.C.
Copper m etallurgy in c ondenser C.O. Condenser being replac ed with S .S. 29-4-C
Copper m etallurgy in LP feedwater heaters C.O.
Copper m etallurgy in HP feedwater heaters C.O.
Figure 4-11 Unit One, Mechanical Dust Collection Retrofit, 41,146 Mwhs
4-41
High Fan Vibration
Cooling water s upply nipple c rac ks
Cooling water enters lube oil res ervoir
Cooling water dis places lube oil
B earing over heats .
Repeated Boos ter ID Fan B earing Failures
Fan inbalanc e due to flyash erosion
Fatigue
No hi-hi bearing tem perature trip
No analys is of failed nipple
Rigid design
Hous ing worn
Figure 4-12 Unit One, ID Booster Fan Bearing Failures, 7,544 Mwhs
4-42
1998
Repeated trips due to high thrus t
Turbine overhaul c om ponent replac em ent
New thrus t probes ins talled
Unit c am e up on load
Thrus t ex c eeded trip s et point
Turbine has m ore thrus t range than other Cholla units
W orn thrus t bearing replac ed
P robes c alibrated bas ed on pas t ex perienc e
Unit 1 Turbine Thrus t Trips .
Turbine thrus t goes inc reas ingly negative with inc reas ed load.
P robes replac ed Different thrus t loc ating m ec hanis m
Lac k of ex perienc e
Turbine ex pands with inc reas ed tem perature
Turbine thrus t m eas ured by c old s tatic m ethods
Lac k of data
No c apability to adjus t probes on line
M eter range lim ited to m aintain plant s tandards
Trip s etpoint determ ined by c old s tatic m eas urem ents
Figure 4-13 Unit One Turbine Thrust Trips, 9,971 Mwhs
4-43
6/18
6/18
Unit down for s c heduled outage
E lec . E ng. Doble tested 2A B oos ter fan m otor M otor c hec k ed out OK
Cable dis c onnec ted at break er c abinet for Dolby tes ting
Operator attem pts to s tart fans for balanc e c hec k
Operator again attem pted to s tart the fan
E lec tric ian m eggered the m otor & c able
Fan s tarted for balanc e c hec k s
P lanner not notified of c hange in work s c ope
Fan tripped on ins tantaneous ground
Fan tripped on overload
M otor c hec k ed out OK
B alanc ing c hec k ed out OK
Center phase c able Dolby tes ted m arginal
Us ed four different elec ts . for the tes ts
Flags were down and res et
P roduc tion S upv. inves tigates
M egger tes t indic ates open phas e
Fan verified c lean after June outage
Contrac t elec trician fails to rec onnec t c ables
*
No inves tigation to determ ine c aus e
E lec tric al forem an rem em bers that c able had been dis c onnec ted
Fan plac ed in s ervic e, ran for 17 day s
6/21
7/6-7/8
7/10
Fan inboard bearing vibration inc reas es
Unit 2 s hut down for boiler tube repair
Dec is ion m ade to was h fans
7/11 Fan has repeated s tarts for was hing and balanc ing
Unit 2 A B oos ter ID Fan M otor S horted Out
V ibration levels ac c eptable
Inboard Horiz ontal 3.5 V ertic al 2.5
Dec is ion bas ed on as s um ed fouling of fan blades
Fan s tarted 6 tim es , no proc edures or guidelines
*
DA TE S TA RT S TOP E LA P S E D
Outboard rem ained c ons tant at 1.9 Firs t was hing had negligible effec t on vibration levels
No detail readings /analy s is done
TIM E --------------------------------------------------------------------7/10 19:20 19:24 19:54 20:05 34 M in. 20:24 21:11 30 21:22 21:31 58 23:07 23:08 105 7/11 00:06 00:06 59 Tripped
Figure 4-14 Unit Two, A Booster ID Fan Motor Failure, 30,464 Mwhs
4-44
Oc t-98
Feb-99
Mar-99
A pr-99
Opac ity spikes c aus ed by s oot blowing s top Obs erved by plant pers onnel
Operator observes high dP on "B " wheel. (A larm?) W .O. W ritten C429475
Operator obs erves high dP on "A " & "B " wheels . W .O. W ritten C432684
Cable tray fire outage 40 hours
Operator observes high dP on A & B wheels .
M iss ed opportunity
W .O. W ritten C434285 Fowarded to E ngineering
*
No W ork Order written
*
M aintenance diagnos ed inc orrec tly W .O. s pec ified pluggage "Not ins t. problem , c hange alarm s etpoint s "
M aintenanc e diagnosed incorrectly
A P H dP data not reviewed or trended
M aintenance diagnos ed inc orrec tly
May-99
Unit Outage for s crubber repairs 192 hrs .
Unit load curtailed- ran out of fan
Ins trument shop verifies trans mitter ac c urac y
Unit Four A ir P reheater plugged 9/99
M is s ed opportunity
Dis c us s ed at m orning m eeting Unit Outage to wash wheels Operations trouble shooting problem
Found A P H S oot blowing nozz le fallen off
Figure 4-15 Unit Four, Air Preheater Pluggage, 48,740 Mwh
4-45
Unit 4 stack SO2 is too high
Low flue gas flow through scrubber
P lugged dem is ters reduc ing flow "A"
Upper dem is ters are plugged
Lower dem is ters are plugged
Too m uch carryover from lower dem is ters C.O. S ee "A" S pray frequenc y is incorrect C.O.
S praying is ineffec tive
S praying is ineffec tive
Incorrec t nozzles
Inc orrect nozzles C.O. I& S
S pray duration is too short
C.O. I&S
Operator can adjus t to as low as one m inute C.O. S
Figure 4-16 Unit Four Scrubber Outage, 76,917 Mwh
4-46
5/31 Coal purchas ed from alternate supplier Unit on line after tube leak
6/3 Opacity trending upward. 7% Un-noticed by operators
6/5 5,500 tons of 30% as h c oal bunkered to Unit 4 Operations uninform ed of high ash
6/6 Opac ity base line rise continues
6/7 High hopper indications on "B" side
Mc Kinley mine on strike P rec ip not inspected during outage
Not spiking
Coal quality directive did not addres s as h c ontent
"A" side alarms not working
Coal was 30% as h
27% opac ity rule us ed
P .M.'s not performed
Internal direc tive being rewritten
Analys is not received until after loading.
Practice of hopper em pty verification no longer done.
Nuvafeeders not adequately m aintained
Did not m eet specs
Do
5,000 tons /day typical burn
Training Repair
Hopper level indication is unreliable
PM's
6/8 TR's tripping on A &B side
6/8 Elect.'s working on "A " s ide high hopper indications Obsolete, parts not available Operators unaware level alarm inoperative
6/8 A/O's begin to manually pull hoppers
6/11 Unit has opacity ex ceedences Dum ped A 18, 17, &14, and B18 hoppers to ground
6/11 Decis ion m ade to shut unit down Unit outage due to high opacity
High level in fly ash hoppers
Hopper inspec tions
Operators did not realize the s ituation or cons equences they were fac ing
Operators verify ing A side hoppers
Load is dropped
"A " s ide had m ore dam age
No load reduction cons idered 6/16 6/3 6/5 6/6 6/7
Figure 4-17 Unit Four, High Capacity Shutdown, 42,731 Mwh
4-47
Chloride c ontam ination in boiler water
Heavy s cale depos its on boiler waterwalls
Operating with c ondenser leaks S
Contam inates in boiler m akeup water
Lack of proper c hem ic al c leaning Nonc om plianc e to proc edure No procedure in place
Lac k of c hem ical cleaning to rem ove s c ale
Im purities in feedwater s ys tem M S Operating with c ondens er tube leaks Corros ion in feedwater s y stem Im purities in m ak eup water
Dem ineraliz er regeneration m alfunc tion
Did not boros cope water wall tubes
Sam ples tak en from nonrepresentative area
P oor pH c ontrol
Operating with high dis s olved ox y gen
Flow ass is ted c orrosion
Figure 4-18 Unit Two Waterwall Tubes Hydrogen Damage, 188,944 Mwh
4-48
S im ilar inc ident on U4 earlier (18 M onths ?)
Coal dus t ac c um ulates on hot air duc t

Poor dust c ontrol on transf er points
Unk nown maintenanc e ac tivity performed
AO reports fire
Fire brigade res ponds
S .S . orders Unit 2 s hutdown

SW pump disc harge pres sure at z ero
S .S . s huts down U-2 SBAC

Hot enough to smell
Inves tigated?
*
Plank lef t on duc t
Fire in U2 bottom as h enc l.
Cooled by Pow er f eeds to
s erv ic e Fire loop c ooling H2O s upply not identif ied High temp trip w ires shorted w ater
500 degree gas temp
SW pumps grounded
*
High temp trip w ires not repaired
S .S . orders U-1 S B A is olated & 1A S B A C s tarted
S .S . s huts down U-3 SBAC

Hot enough to s mell
S .S . orders U-3 to drop load

No soot blow ing air
S .S . orders U-4 to drop load

No s oot blow ing air
A O attem pts to is olate U-4 air s y s tem s

Not suc ces sf u l
A O c los es 4B ID Fan damper
Unit 4 trips
Los s of 28,899 MW hours
3 to 5 perc ent
High f urnac e pres sure
Cooled by s erv ic e w ater U-4 SBA C out of s erv ic e High temp trip w ires s horted
U-4 SBA C out of servic e
No procedures
Inst. A ir pres sure too low to open 4 A ID Fan damper
*
U-3 SW pumps abandoned
*
A O new to unit
Fire loop cooling H2O supply not identif ied
Figure 4-19 Unit Two Cable Tray Fire, 28,899 Mwh
4-49
Tw o units of f and one c urtailed
U-3 c urtailed
U-2 s hut dow n due to s af ety
Unit 4 tripped of f
Not able to blow s oot
Unknow n Los s of adequate s ootblow ing air to plant Units 2,3,&4 SBA C's not running damage due to c able tray f ire ID
High f urnac e pres s ure
Operator low ered B ID damper 3-5% i h d
A damper w ould not rais e in A uto
A ir pres s ure w as at U-4 SBA C not running SBA C s hut dow n f or maintenanc e U-3 SBCA not running U-2 SBA C not running Operator w as low ering load Units 2,3,&4 SS s hut of f SS ins truc ted operator to low er load See prev ious Bobby Jones High temperature trips did not w ork Bearing temperature ov er the high temperature trip poimt Units 2,3, &4 Los t c ooling w ater to SBA C SBA C's not running Not able to blow s oot SBA C's not running Tie v alv e betw een 2/3 and 4 w as open or below 50 lbs .
Same as U-2 SBA C
* Cont. in upper left

Operator did not c los e tie v alv e C.O. M1,2,3,4
* Cont. in upper left

Wires s horted ID Did not repair CO M1,2,3,4 U-3 s erv ic e w ater pumps not running No pow er f eeds to pumps U-2 s erv ic e w ater pumps not running
Pumps s hut dow n f or 10 y ears CO M1,2,3,4
Both motors pow er leads in tray melted to ground. Cable tray f ire ID
Figure 4-20 Unit Two Cable Tray Fire March 1999, 28,899 Mwh
4-50
0:18 9/24 U-4 s hut down for A H was h
14:46
27-S ep Hi Tem p alarm on 4A guide bearing
U-4 on line for 27 Hrs
17:48 28-S ep A O confirm ed reports fire at guide bearing
17:56 SS & AO could not put out fire
28-S ep
S S takes unit offline
High dP
Norm al s tart up
Tem p spik ed
Oil soak ed fly ash pack ed around housing appox . 10" deep
S S tries five tim es to put out fire
No work done on guide bearing
Fire dam aged therm oc ouple
*
A ir out tem p 560580 F
Tem p. m eas. @ 420 plus F above s eals on A & B
Flam es em anating from bearing hous ing
Oil Flas h point 440 F A ir seal leak ing on A side
1:18 A
9/25 15:46 S S c alls Joe c ity fire dept
28-S ep 6099 MW Hours lost SS & AO put out fire
18:48
29-S ep M aintenance chec ks guide bearing Unit bac k on line
18:56
29-S ep
Joe City fire brigade too late
* $700 labor c harged to WO

Fire dam aged therm oc ouple
4A guide bearing tem p in alarm
Could not m ak e radio content
A ir s eal leak ing on A guide bearing
Guide bearing ins pected and found to be OK
Figure 4-21 Unit Four Air Preheater Guide Bearing Fire, 6,099 Mwh
4-51
EPRI Licensed Material Boiler Reliability Optimization Project Phase 1- Root Cause Analysis at Arizona Public Services Companys Four Corners and Cholla Plants Problem Description Unit 1 Mechanical Flyash Collector Unit 1 ID Booster Fan Bearing Failure Unit 1 Turbine Thrust Trips Unit 2 A ID Booster Fan Motor Failure Unit 4 Air Preheater Pluggage Unit 4 Scrubber Outage Unit 4 High Opacity Shutdown Unit 2 Waterwall Hydrogen Damage Unit 2 Cable Tray Fire Unit 4 Air Preheater Guide Bearing Fire Root Cause MW/Hrs. Lost 4.1 4.2 4.3 9971 4.4 30,464 4.5 4.6 76,917 4.7 4.8 188,944 4.9 28,899 4.10 6,099
4-52
5
BOILER RELIABILITY OPTIMIZATION PROJECT PROGRAM AUDIT AT HAWAIIAN ELECTRIC COMPANY
5.1 EXECUTIVE SUMMARY

As a result of Hawaiian Electric Corporation embarking on a Remaining Useful Life and Generation Asset Management program in 1999, and including in the project the Boiler Reliability Optimization Program, EPRI was requested to provide an independent review of this area of the project. During week 18 June 2001, a series of interviews and meetings were held with a number of HECO personnel and a number of documents were reviewed. The EPRI team found that progress had been made in successfully implementing a number of planned activities. The team also identified a number of opportunities where the current efforts could be enhanced and which, when implemented, would give additional assurance that the goals set for availability and reliability would be achieved. These opportunities are highlighted in the main portion of the report.
5.2 Introduction
In 1999 Hawaiian Electric Company Inc. (HECO) embarked on a Remaining Useful Life and Generation Asset Management program, to enable all the HECO generating plant to continue to operate well beyond the 20-year horizon. One of the projects in this program was the Boiler Reliability Optimization Program. The objectives of the Boiler Reliability Optimization Program are: To reduce short- and long-term boiler maintenance costs, while maintaining or improving overall boiler reliability. To develop and implement boiler maintenance strategies using EPRIs Streamlined Reliability Centered Maintenance process.
The project was started in the last quarter of 1999 with a Boiler Tube Reduction and Cycle Chemistry Improvement training program, then continued in 2000 with a pilot Streamlined Reliability Centered Maintenance project, which is currently being implemented. Progress has been made in implementing many of the actions, along with many other initiatives as part of the overall RUL/GAM program. The program had reached a point where it was expeditious to have an independent review of the current project status and plan to give assurance that the goals set would be reached. EPRI was 5-1
EPRI Licensed Material Boiler Reliability Optimization Project Program Audit at Hawaiian Electric Company
requested to undertake such a review. The review would give management the assurance that the overall RUL/GAM program and Boiler Reliability Optimization project goals will be met. The specific objectives of this review are: To review the current Boiler Reliability project action plan To identify cost effective improvements; and To recommend changes in their action plan to address the identified gaps.
5.3 Findings
5.3.1 Boiler Reliability Optimization Project Plan There are basic plans, outlined in the PDM Team Strategic Plan 2001 and the Boiler Reliability Optimization Program Envisioned Role of BRO Team Members, but they differ. The plans are fairly loosely structured and therefore lack focus, giving little direction for the development of specific action plans. There is no overall strategy with key deliverables and dates. Roles and responsibilities of the key players are not clearly defined. This results in misunderstandings and confusion about who should be doing what and when. This impacts the cost effectiveness of this program. An engineer was specifically employed to assist with the implementation of the Boiler Reliability Optimization Program. His energies have been redirected into solving a variety of other technical problems, and he has, therefore, been unable to devote the necessary time and attention to his primary role. This has occurred because of the above-mentioned lack of direction regarding roles and responsibilities. Boiler performance goals tube leaks are outlined in the Boiler Tube Failure Reduction/Cycle Chemistry Improvement Program Management Mandate. Short-term goals, as outlined, have only been partially achieved. Long term goals may be achieved but are aggressive. The Boiler Reliability Optimization Strategic Plan that is being followed (shown in Appendix 1), shows that a number of deadlines have not been met, or have been extended. The plan has been filled out with the relevant information regarding the current progress and updated deadlines.
5.3.2 Project Activities 5.3.2.1 BTFR/CCI Program A BTFR/CCI Improvement Program Management Mandate was issued in December 1999, after the BTFR/CCI training was completed. The mandate outlines the ground rules and
5-2
guidelines for the implementation of a Boiler Tube Failure Reduction Program. This gives direction and focus to the efforts of the core Boiler Tube Failure Reduction team. This Management Mandate has an annual review cycle. To date this document has not been reviewed and/or revised. Some of the short- and long-term goals outlined in the Mandate have been achieved. As a result of the Mandate being issued, a comprehensive, standardized Boiler Tube Failure report format was developed, and is being followed. The Boiler Maintenance Workstation (BMW) program, which is a Windows-based software program used to track and trend boiler tube failure data and information, was purchased some time ago. Software compatibility problems have now been solved and the database is being populated with drawings and tube leak data. Initial training on this program was given to a selected group some years ago, and no other employees have been trained since, so knowledge of its true benefits and uses is restricted. In excess of $3million has been budgeted for the procurement and installation of core water and steam chemistry instrumentation. Phase 1, consisting of 3 instruments per boiler is to be installed by the end of August 2001 for Kahe units. Installations of phase 1 instruments are scheduled for completion by 12/31/01 and 3/31/02 for Waiau and Honolulu respectively. This will enhance the control of water chemistry, which will minimize boiler tube leaks that result from poor water chemistry, improving reliability.
5.3.2.2 Streamlined Reliability Centered Maintenance A pilot project on three systems Boiler Air and Gas, condensate and Feedwater at two sites, has been successfully completed. The resultant maintenance tasks have been accepted and are being integrated into the work order management system for Kahe 3. This provides a basis for cost effective maintenance. There is no overall preventive/predictive maintenance basis for the boiler pressure parts, as in tubes. Repairs are carried out as and when failure occurs. This strategy has an impact on the overall reliability and maintenance costs. 5.3.2.3 Root Cause Analysis Training in basic Root Cause Analysis was done some time ago, but has not been formally used in practice besides investigation involving boiler tube failures. Problem solving is done on an ad hoc basis and repeat failures are occurring, causing problems with reliability and impacting costs. The importance of Root Cause Analysis being integrated in the work management process is not being recognized. It is the most effective means of ensuring improvements in reliability and saving costs, by preventing/minimizing failures from occurring or recurring.
5-3
5.3.3 Condition based data collection A hardware breakdown structure does not exist for the boiler i.e. uniquely identifying the system, sub-systems and components within the boiler. Therefore tracking the condition and or performance of each system, sub-system is difficult. Condition based data is collected on some sub-systems and components. However the data collection-analysis-action process is no consistent across all boiler components. This makes it difficult to determine the true condition of each sub-system or component and prioritize the inspection and repair work on the boiler.
5.4 Conclusions
In an effort to provide assurance that HECOs current initiatives in the Boiler Reliability Optimization project are adequate and cost effective, reviewers from EPRI studied the given findings and concluded that: 5.4.1 The Boiler Reliability Optimization Team and others associated with reliability improvements have made a significant effort to implement the boiler tasks associated the RUL/GAM project. All involved need to be congratulated on their efforts and having the courage to request that a third party review their program. 5.4.2 Basic project management principals of time, cost and quality have not been followed in the development of an adequate program and schedule. 5.4.3 Some project planning has been done, but it is ineffective. The effectiveness of the teams efforts is impaired because of the lack of clarity in the plan on roles and responsibilities, direction, objectives and deadlines. 5.4.4 There is a lot of emphasis and effort being focused on the Boiler Tube Failure Reduction program and the program is achieving its objectives. 5.4.5 The current efforts do not provide for an adequate maintenance strategy regarding boiler tubes. 5.4.6 As root cause analysis has not been rigorously practiced since the training sometime ago, it is doubtful that a significant proportion of that knowledge and skill gained is available. 5.4.7 Significant financial resources have been allocated to install state of the art water and steam chemistry instrumentation however little resources have been set aside for the on going maintenance of these newly installed instruments, some of which will be installed shortly
5.5 RECOMMENDATIONS
The following recommendations are given to enhance the current Boiler Reliability Optimization initiatives. 5-4
5.5.1 A project leader should be designated in writing, whose prime responsibility it would be to oversee the implementation of this project. 5.5.2 Develop a comprehensive plan/schedule for the implementation of all activities. This plan should identify: boiler-related
The people responsible for the various activities identified in the program, and should clearly define their roles. What these activities are as in scope The resource loading required to implement these activities The duration of each activity.
A typical Microsoft Project plan would provide this framework. 5.5.3 The Management Mandate and various other programs mandates/directives (i.e. Root cause Failure, Streamlined RCM etc) should be reviewed and updated, according to the given schedule, as they serve as the bases for Boiler Reliability Optimization. A two yearly review cycle is considered to be the norm. 5.5.4 In order to compliment the current BTFR initiatives, it is recommended that a Failure Modes and Effects Analysis (FMEA) be done on boiler pressure parts, as a means of identifying potential failure mechanisms, and the financial effects of their occurrence. This analysis will also focus the organization on what the most significant failures are, so that a quantum leap in reliability can be made with fewer organizational resources being utilized. FMEA is an engineering technique used to define, identify and eliminate known and/or potential failures, problems, etc. on a system and/or components within that system. Appendix 2 gives an example of an FMEA analysis. 5.5.5 Continue with the current SRCM initiatives on the remaining systems, i.e. to develop a cost effective maintenance program, based on system functionality that will enhance system reliability. This will make optimum use of available maintenance resources and provide a documented base for future additions/revisions to the maintenance program. Develop a hardware breakdown structure for each boiler. The structure will allow for each system, sub-system and component to be uniquely identified. This numbering system can be used in MIMS, Plant View and BMW thus enabling the condition of each system, sub-system and component to be tracked and enabling cost effective run-repairreplace decisions to be made. Clearly define and document the roles and responsibilities of the individuals whose responsibility is to continuously monitor the overall condition of each boiler system, subsystem and component and to develop a short and long term inspection plan for each boiler and that of managing the Boiler Reliability Optimization Project. Training in a suitable Root Cause Analysis technique/process be given to a small number of individuals. It is suggested that a number of key individuals be trained to become trainers of a suitable process that meets the needs of HECO. These individuals can 5-5
5.5.6
5.5.7
5.5.8
then develop a Root Cause Analysis Awareness course that can then be given to engineers, technicians, operators and craftsperson. 5.5.9 An administrative mandate/policy/procedure to be drawn up to: standardize what gets investigated; outlines the investigation standards and processes to be followed; the roles and responsibilities of the various team members and the reporting requirements. See appendix 3 for a typical list of headings that appear in a procedure. 5.5.10 A maintenance strategy be developed and documented in the CMMS system for the new cycle chemistry instrumentation being installed. The information required to develop the maintenance requirements can be sort from in house expertise, the OEM and or EPRI.
5-6
Appendix 1: Boiler Reliability Optimization Strorage Plan

HECO BRO IMPLEMTATION PLAN AS PER STRATEGIC PLAN
No. Activity By Whom
Page 1
Revised completion date
Goal
Issue BTFR Procedure Manual Issue CCI Procedures Manual Fully Implement BTFR and RCFA Process
By When
Progress to date
Formalize and Organize Unified BRO Team and Procedures
Bill and Bob Isler
1Q01 3Q01 2Q01
Draft 90% complete
Final Draft 08/31/01 Finalize and issue 12/31/01
Establish Formal Use of BTFR and RCFA Process for Boiler Tube Failures Formalize Data Collection, Management and Distribution of Boiler Information Assist with Development of Boiler Start Up Procedures and Combustion and Flame Management Guidelines
Bill, supported by BRO Team
Bill, supported by PDM specialist, Ed Chang Jose, Lane, Ed, Bill, Barney and Russell
Develop Process Map on Data Collection and Storage Completed Procedures in place
3Q01
3Q01
Develop Air Pre-Heater Maintenance Program
Bill and Larry
Develop Feed-water Heat Exchanger Maintenance Program
Bill McCraw and PDM Specialist
Develop Condition Baseline for all Air Pre-Heaters. Plan in place. Develop Condition Baseline for All Heat Exchangers. Select Mapping software.
4Q01
Completed and established 08/16/01 90% established Not yet formalized No progress to date. Still waiting for schedule guidelines from Operations Approximately 40% complete on baseline history. Progress is subject to unit outages
Predict completion by 12/31/01
Predict completion by 12/31/01. Predict selection by 09/31/01.
2Q01
5-7

Page 2
Progress to date Revised completion date
Goal
Refine and Implement existing Procedures Issue Procedures for the Cycling of Units Started 10/00 Complete
By When
Develop and Implement Lay-up Procedures for Feedwater Headers Develop and Implement Lay-up Procedures for Boilers Install and Implement Nitrogen Blanketing Systems and all Feedwater Heaters Optimize Boiler Chemical Cleaning Frequency, using Condition Based Techniques. Oversee the Chemical Cleaning Process Develop Strategy and Install all Core Cycle Chemistry Instrumentation
Bill, Barney and Russell Bill, Ed, Barney and Russell Bill and Barney
2Q01
Approximately 60% complete. Approximately 40% complete. Waiau by 03/31/02. Practice in place.
4Q01
4Q02
Predict Waiau Honolulu by 12/31/01. Predict Waiau Honolulu by 12/31/01. .
10
Bill, Barney, Russell and Ed
Develop and have in place the programs and Procedures Install the 2000 increment Develop Strategy for implementation for remaining core instrumentation Plan to be in place
4Q01
11
Bill, supported by Barney, Russell and P&E
2Q01 2Q01
Strategy is completed and included in 5year plan.
Formalize program and procedures by 12/31/01. Phase I will be 85% complete by 12/31/01.
12
Develop Plan to Maintain Cycle Chemistry Instrumentation
Larry and Bill
2Q01
Dormant waiting for completion of Phase I CCI.
5-8

Page 3
Progress to date Revised completion date
Goal
Select system. ChemExpert will probably be chosen
By When
13
Evaluate Cycle Chemistry Procedures and Software
Bill, Bob Isler, Barney and Russell
3Q01
Dormant waiting for EPRI Solutions to convert current software to Windows NT. (expected by 08/31/01). No progress to date. Doubtful will meet schedule completion. Practice in place. Formal program not yet written..
Predict system selection by 12/31/01.
14
Train Operators in Cycle Chemistry following installation of instruments
Bill, Barney and Russell
Develop Training Program Take the lead in Implementing Training
4Q01 1Q02
15
Develop and Implement Air Inleakage Improvement Program
PDM Group, Barney and Russell
Implement Formal Program
2Q01
Predict 2Q02
5-9
Appendix 2
FMEA Worksheet
System: Heat recovery Sub-system: Economizer Component: Economizer inlet header
FMEA Process
Detection (1- 10) Severity (1 10) Failure Modes/ Failure mechanism Cracks on the internal diameter, parallel to the stub tube axis Thermal fatigue If detected during a periodic inspection, repairs would necessitate extending the outage. If, however the cracks went undetected failure would result in a forced outage with the consequent loss of revenue. Effects of failure Causes of Failure Frequency (1 10) Detection Method RPN Recommended Action
1. Start up or shut down procedure not being followed
128
Conduct periodic reviews/audits of compliance to the start-up and or shut down procedures
2. No start-up or shut down procedure available or
256
3. Start-up and or shut down procedures is inadequate and highlights no precautions.
128
Compile separately a start-up and shut procedure detailing the steps and precautions to be taken by the operator Conduct periodic reviews/audits of compliance to the start-up and or shut down procedures
5-10
Appendix 3
List of Typical headings that appear in a policy and or a procedure 1. Group name 2. Document type: policy, standard, guideline, procedure etc 3. Reference number 4. Revision number 5. Description of revisions made 6. Review date 7. Date authorized 8. Date when document came into effect 9. Compiled by 10. List of designations who have approved the document 11. Authorized by 12. Title 13. Background/preamble/introduction 14. Purpose 15. Scope of document 16. References 17. Definitions and or abbreviations 18. Responsibilities 19. Responsibility matrix 20. Requirements: step by step of what is required to be done 21. Work process flow and responsibility matrix 22. Distribution list 23. Appendices
5-11
Target: Predictive Maintenance Program Development and Diagnostic Tools
SINGLE USER LICENSE AGREEMENT THIS IS A LEGALLY BINDING AGREEMENT BETWEEN YOU AND THE ELECTRIC POWER RESEARCH INSTITUTE, INC. (EPRI). PLEASE READ IT CAREFULLY BEFORE REMOVING THE WRAPPING MATERIAL. BY OPENING THIS SEALED PACKAGE YOU ARE AGREEING TO THE TERMS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO THE TERMS OF THIS AGREEMENT,PROMPTLY RETURN THE UNOPENED PACKAGE TO EPRI AND THE PURCHASE PRICE WILL BE REFUNDED. 1. GRANT OF LICENSE EPRI grants you the nonexclusive and nontransferable right during the term of this agreement to use this package only for your own benefit and the benefit of your organization.This means that the following may use this package: (I) your company (at any site owned or operated by your company); (II) its subsidiaries or other related entities; and (III) a consultant to your company or related entities, if the consultant has entered into a contract agreeing not to disclose the package outside of its organization or to use the package for its own benefit or the benefit of any party other than your company. This shrink-wrap license agreement is subordinate to the terms of the Master Utility License Agreement between most U.S. EPRI member utilities and EPRI.Any EPRI member utility that does not have a Master Utility License Agreement may get one on request.
About EPRI EPRI creates science and technology solutions for the global energy and energy services industry. U.S. electric utilities established the Electric Power Research Institute in 1973 as a nonprofit research consortium for the benefit of utility members, their customers, and society. Now known simply as EPRI, the company provides a wide range of innovative products and services to more than 1000 energyrelated organizations in 40 countries. EPRIs multidisciplinary team of scientists and engineers draws on a worldwide network of technical and business expertise to help solve todays toughest energy and environmental problems. EPRI. Electrify the World
2. COPYRIGHT This package, including the information contained in it, is either licensed to EPRI or owned by EPRI and is protected by United States and international copyright laws. You may not, without the prior written permission of EPRI, reproduce, translate or modify this package, in any form, in whole or in part, or prepare any derivative work based on this package. 3. RESTRICTIONS You may not rent, lease, license, disclose or give this package to any person or organization, or use the information contained in this package, for the benefit of any third party or for any purpose other than as specified above unless such use is with the prior written permission of EPRI.You agree to take all reasonable steps to prevent unauthorized disclosure or use of this package.Except as specified above, this agreement does not grant you any right to patents, copyrights, trade secrets, trade names, trademarks or any other intellectual property, rights or licenses in respect of this package. 4.TERM AND TERMINATION This license and this agreement are effective until terminated.You may terminate them at any time by destroying this package. EPRI has the right to terminate the license and this agreement immediately if you fail to comply with any term or condition of this agreement. Upon any termination you may destroy this package, but all obligations of nondisclosure will remain in effect. 5. DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIES NEITHER EPRI,ANY MEMBER OF EPRI,ANY COSPONSOR, NOR ANY PERSON OR ORGANIZATION ACTING ON BEHALF OF ANY OF THEM: (A) MAKES ANY WARRANTY OR REPRESENTATION WHATSOEVER, EXPRESS OR IMPLIED, (I) WITH RESPECT TO THE USE OF ANY INFORMATION,APPARATUS, METHOD, PROCESS OR SIMILAR ITEM DISCLOSED IN THIS PACKAGE, INCLUDING MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, OR (II) THAT SUCH USE DOES NOT INFRINGE ON OR INTERFERE WITH PRIVATELY OWNED RIGHTS, INCLUDING ANY PARTYS INTELLECTUAL PROPERTY, OR (III) THAT THIS PACKAGE IS SUITABLE TO ANY PARTICULAR USERS CIRCUMSTANCE; OR (B) ASSUMES RESPONSIBILITY FOR ANY DAMAGES OR OTHER LIABILITY WHATSOEVER (INCLUDING ANY CONSEQUENTIAL DAMAGES, EVEN IF EPRI OR ANY EPRI REPRESENTATIVE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES) RESULTING FROM YOUR SELECTION OR USE OF THIS PACKAGE OR ANY INFORMATION,APPARATUS, METHOD, PROCESS OR SIMILAR ITEM DISCLOSED IN THIS PACKAGE. 6. EXPORT The laws and regulations of the United States restrict the export and re-export of any portion of this package, and you agree not to export or re-export this package or any related technical data in any form without the appropriate United States and foreign government approvals. 7. CHOICE OF LAW This agreement will be governed by the laws of the State of California as applied to transactions taking place entirely in California between California residents. 8. INTEGRATION You have read and understand this agreement, and acknowledge that it is the final, complete and exclusive agreement between you and EPRI concerning its subject matter, superseding any prior related understanding or agreement. No waiver, variation or different terms of this agreement will be enforceable against EPRI unless EPRI gives its prior written consent, signed by an officer of EPRI.
2001 Electric Power Research Institute (EPRI), Inc. All rights reserved. Electric Power Research Institute and EPRI are registered service marks of the Electric Power Research Institute, Inc. EPRI. ELECTRIFY THE WORLD is a service mark of the Electric Power Research Institute, Inc. Printed on recycled paper in the United States of America 1006537
EPRI 3412 Hillview Avenue, Palo Alto, California 94304 PO Box 10412, Palo Alto, California 94303 USA 800.313.3774 650.855.2121 askepri@epri.com www.epri.com

Boiler Reliabiliti Optimization 2001

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Boiler Reliabiliti Optimization 2001

Uploaded by

Copyright:

Available Formats

EPRI Boiler Reliability Optimization Program

Case Studies from 1998-2001

EPRI Boiler Reliability Optimization Program

Final Report, December 2001

EPRI Project Manager P. Abbott

DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIES

EPRI Licensed Material

1.4.1 Phase I and II - Technical and Organisational Assessment.................................. 1-3

EPRI Licensed Material

EPRI Licensed Material

EPRI Licensed Material

EPRI Licensed Material

EPRI Licensed Material

EPRI Licensed Material

EPRI Licensed Material

EPRI Licensed Material

EPRI Licensed Material

FINDINGS AND OBSERVATIONS

Figure 1-1 Maintenance Optimization Program

P M B as is Q ualific ation Com m unic ation Intra Utiliz ation

W ork Identific ation W ork Control W ork E x ec ution

W ork M anagem ent S y s tem s

B enc hm ark ing

S etting G oals Com m unic ation Inter M etric s Leaders hip

Figure 1-2 Big Sandy benchmarked against Industry Best Practice

Work Flow Process

Figure 1-3 Work Process

PM Basis Change Control

CBM Data Collection & Analysis

CBM Coordinator Technology Owners

Information Integration & Decision CD's, PR's

Work Mgmt System

Work Order Generation

Planners Boiler Specialist

Boiler Specialist Craft Boiler Specialist Boiler Specialist, CBM Coordinator

Post Maintenance Testing

Boiler Specialist Planners, Craft

Figure 1-4 Work Process Model

This is the final activity and includes work close-out.

2. Compile a management directive identifying the minimum acceptable standards to be

1. Repair and calibrate all essential/critical boiler instrumentation.

1. PIMS should be expanded to capture all boiler inspection data

1. Establish technology owners with responsibilities to include condition analysis. 1-12

First Reheat (High Pressure)

Acid Phosphate Corrosion

Preventive Measures Short/Long Term

NDE Detection Technique

Acid Dewpoint Corrosion

Low Temperature Creep

Short Term Overheating

Coal Particle Erosion

Falling Slag Erosion

Fire Side Corrosion

Fly Ash Erosion

Components 1 through 16 (all components within sootblower range)

Acid Phosphate Corrosion

Preventive Measures Short/Long Term

NDE Detection Technique VT

Acid Dewpoint Corrosion

Low Temperature Creep

Short Term Overheating

Coal Particle Erosion

Falling Slag Erosion

Fire Side Corrosion