SUCCESSFUL COMMISSIONING: LESSONS LEARNED FROM THE START-UP OF WEST TRAVIS COUNTY REGIONAL WTP EXPANSION By Chris Chen,

PE, AECOM Kevin Koeller, PE, AECOM 400 West 15th Street, Suite 500 Austin, Texas 78701 John S. McLeod, PhD, PE, Lower Colorado River Authority 3700 Lake Austin Blvd., Austin, Texas 78703 ABSTRACT Plant commissioning is the final phase of any major facility improvement or expansion project and it is one of the most critical stage of a successful project since it will confirm and verify if the newly installed facility would be able to meet the performance criteria and established project goals. However, this phase sometimes receives little attention with short up-front planning from the contractor due to other competing constructing priorities until it is time to start the plant up. In some cases the contractors forces have left the site resulting in repeated trips from suppliers and vendors to resolve outstanding issues. Because of an extremely tight schedule, budget, and treatment capacity constraints, the Lower Colorado River Authority (LCRA) could not afford an unsuccessful startup or delayed commissioning of the expansion and improvements to the West Travis County Regional Water Treatment Plant (WTCR WTP). The WTCR WTP project expanded a conventional water treatment plant from 9.8 mgd to 20 mgd and increased the raw water pumping capacity from 10 mgd to 22 mgd. The construction scope covered nine major plant components or systems from the Raw Water Intake to the High Service Pump Station including the plant wide electrical system and instrumentation and control system upgrades. Using a detailed and system based approach, a carefully coordinated sequence of construction, acceptance testing, and turnover plans were implemented to successfully commission the new facilities to avoid any failures or breakdowns in plant processes, or interruptions in water service. These goals were achieved by applying best management practices for plant startup including early planning, tight project control, correct construction sequence, coordinated implementation, and clearly identified acceptance metrics. Key areas which led to this smooth commissioning included (1) systematic planning, (2) phased implementation, (3) identification of critical tie in points and carefully planned sequence of construction, (4) customized testing and acceptance methodologies and (5) proactive transition of project roles and responsibilities from construction phase to startup phase. This paper summarizes the challenges, solutions, and lessons learned in testing and commissioning of this complicated plant expansion project. BACKGROUND The Lower Colorado River Authority (LCRA) owns and operates the West Travis County Regional Water Treatment Plant (WTCR WTP), located west of the City of Austin in the City of Bee Cave, Texas. Originally constructed in 1984 as a 1.8 MGD conventional water treatment plant, the plant has been expanded to 9.8 MGD through numerous improvement projects over the past 20 years prior to this project. Continued rapid growth in the area required further expansion

of the WTCR WTP. In December 2005, the LCRA selected AECOM as the lead engineering consultant on the plant expansion to 20 mgd project. In order to appreciate the emphasis placed on plant start-up and commissioning, it is important to understand the importance of the project schedule in this project. This schedule was critical and tied to significant growth in the City of Bee Cave. The project team found that the growth of the water supply in the service area of the WTCR WTP was one to two mgd per year from 2003 to 2006. At this rate, the projected demands would exceed the 9.8 MGD treatment capacity by summer 2007, requiring immediate action to increase plant capacity. There were limited alternatives and after review, the plant needed to increase capacity at least to 10 mgd by summer 2007, 11 mgd by summer 2008, and 13 mgd by summer 2009. Accordingly, there was a significant amount of scrutiny and interest placed in getting the expansion project to produce water on time and within budget. The operation and reliability of the WTCR WTP was critical to the West Travis County regional water supply and the LCRA could not afford any schedule delay or unsuccessful execution. To address the schedule challenge, the project team implemented an innovative two-phase approach for the proposed upgrade and expansion. Phase 1 Expansion contract addressed the immediate short-term peak water demands anticipated in the summers of 2007 by up-rating the pumping and treatment capacity though modification of the original plant facilities. Once this was achieved, the long-term expansion requirements could be addressed in a subsequent Phase 2 Expansion contract focused on installing and integrating additional facilities and improvements to provide ultimate plant built-out capacity. Figure 1 shows the key dates of the phased implementation of the expansion work. This paper is mainly focused on the start-up and commissioning for the Phase 2 expansion. Figure 1 Phased Implementation and System Demands

MAJOR EXPANSION IMPROVEMENTS AECOMs engineering team identified major areas of improvement needed to increase WTCR WTP to the ultimate treatment build-out capacity of 20 mgd under the Phase 2 Expansion. Expansion improvements included nine major systems/components: 1) Raw Water Supply System including a new Raw Water Intake, a new Raw Water Pump Station, and improvements along the Raw Water Transmission main; 2) Water Treatment Process Units including adding new rapid mix, flocculation, sedimentation, and filtration process units; 3) Transfer Pumping Station and Backwash Water Supply System; 4) Disinfection and Treated Water Storage System; 5) High Service Pumping System; 6) Chemical Storage and Dosing Facilities; 7) Backwash Water Lagoons and Return Pumping System; 8) Plant Wide Electrical System Upgrade; and 9) Plant Wide Instrumentation and Control Systems Improvements. MAJOR SYSTEMS AND COMPONENTS Raw Water Supply System: The original raw water pumping station is equipped with three 6 mgd vertical turbine pumps that convey raw water from Lake Austin to the WTCR WTP and to off-site irrigation customers. Phase 2 Expansion of this system includes construction of a new raw water pump station and its own intake to provide 10 mgd additional pumping capacity; and modifications to the raw water transmission main. Water Treatment System: The original water treatment systems at the West Travis Water Treatment Plant prior to the expansions in 2007 included a 1.8 mgd conventional treatment system and four 2 mgd high-rate packaged (Trident) treatment processes. The conventional treatment facilities (known as the Old Plant) has been up-rated to 3 mgd under the Phase 1 Expansion by adding inclined plate settlers in the solids-contact clarifier and automating conventional gravity filter controls. Proposed improvements under the Phase 2 Expansion to the water treatment systems includes construction of additional conventional water treatment process units to provide an additional 10 mgd treatment capacity including two 5-mgd solids-contact clarifiers and four automatic-backwashing granular media filters. Transfer Pumping and Backwash Water Supply System: The original treated water transfer pump station and backwash water supply system consists of two horizontal split case backwash pumps located in a transfer pump station. These transfer pumps and backwash pumps were found in poor operating condition and is too small for the proposed expansion. Proposed improvements under Phase 2 expansion includes construction of a new transfer pumping station with three 10mgd vertical turbine pumps equipped with variable-frequency drives (VFDs) controlled to maintain a near-constant water level in Clearwell No. 1 by pumping the filtered water to a new Disinfection Tank and Clearwell No.2. Disinfection Tank System: Prior to the 2007 improvements, water was pumped from Clearwell No. 1 directly to Clearwell No. 2. The Phase 2 Expansion included construction of a Disinfection Contact Tank with baffles preceding Clearwell No. 2 to provide the required disinfection contact time for the increased treatment capacity. Since there is insufficient time for completing on all the proposed work under Phase 2 Expansion to meet the summer 2008 demand, AECOM prioritize the completing of the disinfection tank ahead of others and have it ready by May 2008

and use it in conjunction with the Clearwell No.2 to provide sufficient disinfection contacts and ground storage capacity allowing the plant meeting the peak system demands in 2008. In addition, the disinfection tank would supply water by gravity for backwashing filters in all water treatment facilities to replace the old backwash pumping at the end of the Phase 2 Expansion. High Service Pumping System: The original High Service Pump Station (HSPS) was equipped with three 4 mgd horizontal split case pumps and one 2 mgd jockey pump. Phase 2 Expansion included replacement of one 4 mgd and one 2 mgd pumps with two new 6 mgd pumps; and installation of a new 6 mgd pump in a currently unoccupied pump bay; and installation of two new chlorine ejector water booster pumps. Chemical Storage and Feed Systems: The chemical storage and feed system expansion is required to match with the increased treatment capacity and to various treatment process units. The expansion work involves improvements and expansion related to chlorine gas injection; liquid ammonium sulfate (LAS) for chloramines disinfection; aluminium chlorohydrate (ACH) for primary coagulant; polymer for coagulant aid; fluoride; and a proprietary taste and odor control oxidant (Bayox 446). In addition, a new gas/vacuum ejector was required to improve chlorine dosing capacity and its associated injector booster water system; and a 12-inch irrigation water supply line relocation to move it upstream from the proposed chlorine dosing point to minimize chemical usage. Washwater Lagoons and Return Pumping System: Phase 2 Expansion includes improvements on the backwash water recycle and enlarging the lagoons and flow patterns for better mixing. Plant Wide Electrical System Upgrade: The upgrade work includes converting multiple Austin Energy power supply sources at the water treatment plant to one 12,470V source for the water treatment plant, resulting in the water treatment facility being metered only at one point. An outdoor 12,470V Load Break Switchgear becomes the main distribution bus for each process area in the water treatment facility which lowers the unit rate of the power cost and makes the quality of the power more reliable and easier to manage. Other improvements include new power distribution transformers, conduit, conductors, and terminations for the expanded plant and for the raw water pump station. Plant Wide Instrumentation, and Control System Improvements: The upgrade work includes a new plant wide fiber optic network, and a new SCADA system utilizing a Distributed Control System (DCS) with a controller per process area/function and creating interdependencies between process areas and to have as much reasonable automation of the plant as possible to achieve maximum efficiency and be sensitive to energy usage and cost without causing complexity to operations or maintenance. MAJOR STAGES OF THE COMMISSIONING PROCESS To ensure a smooth plant start-up and to make sure all major systems for the plant will work according to the design at the commissioning, the project team lead by the LCRA project manager and supported by the AECOM project and construction managers started the preparation and planning at early stage of the project development during the design phase and worked with the Contractor on setting the schedule and expectations at the beginning of the

construction phase. At six months prior to the planned substantial completion date, the project team started detailed planning and execution of the proposed process. The details of the key stages of the commissioning process are explained below and as shown in Figure 2. Figure 2 Major Stages of the Plant Commissioning Process

SYSTEMATIC COMMISSIONING PLANNING Systematic and detailed planning began well in advance of start-up. This planning began in the design stage phase of the WTCR WTP project. Although it is clear, but it is often not stated enough, first, you need to understand what plant operations needed to accomplish from the end product. The operators will take over the plant after it is constructed and they need to have input on what the end product will look like and how it will function. Similar to AECOMs approach on other projects, the design team met with plant operations and held workshops to determine the goals of the WTCR WTP project. Once these goals were determined, they were put into a set of documents, initially with a preliminary engineering report, followed by plans and specifications. With this planning, the Contract Documents were shaped to add requirements on when work could be completed and under what conditions to meet overall project objectives.

One of the major components AECOM used to insure that the construction contractor followed the requirements for planning and completing the work, was a sequence of construction. This document developed out of the AECOMs design teams previous experience working in operational water treatment plants required a systematic approach to upgrading the plant while maintaining plant operations. The coordination necessary to upgrade each of the major systems and components required a process to organize and schedule the contractors work to facilitate effective communication and avoid adverse impacts to plant operations. Additionally, the sequence of construction provided an overview of proposed construction, anticipated problems, and emphasized key steps of the construction sequence as proposed by the Engineer. The sequence of construction met the intent of the design team and served two key purposes: First, it provided an overview of background information, detailed operational constraints of the plant, and identified current and potential conditions which could impact construction activities. Second, because the sequence of construction was included in the construction documents, it required the construction contractor prepare and submit a written plan of their proposed work. The sequence of construction served as a format for the construction contractor to prepare their written work sequence and submit this sequence for review. During development of this sequence of construction and the submittal process during the construction phase, details on start times, finish times, water levels, opening / closing of valves and switches were identified and defined, thus avoiding any surprises during construction. The completed sequence of construction detailed how and when the contractors work would impact plant operations. In addition, Contract specifications detailed the start-up, testing, and commissioning requirements that were referenced in the sequence of construction, but were in separate sections that were associated with the requirements for Operational Readiness Testing and Performance Acceptance Testing. These items are discussed in greater detail in the customized testing and acceptance methodologies section of this paper. IDENTIFICATION OF CRITICAL TIE IN POINTS AND PLANT SHUTDOWNS Working in an operating plant many things can impact operations. The project goals were to minimize impacts on plant operations, thus, it was essential to identify all tie in points that would require a change in plant operations. Once these were identified, these items were put into three categories 1) minor change in operations, 2) partial shutdown or 3) full plant shutdown. On the WTCR WTP Phase 2 Expansion to 20 mgd, a total of 31 shutdowns were identified, approximately 21 of these partial plant shutdowns and approximately 10 major plant shutdowns. One of the most complicated and most critical plant shutdowns was associated with a new tee connection from the new raw water pump station discharge to the existing raw water transmission main at the raw water intake and pump station site under more than 150 psi pressure. Another critical shutdown was the installation of a new cross on the finished water piping at the plant site. All plant shutdowns were identified ahead of time that allows the plant operator to store as much as the treated water in the system and to allow the time for the construction. The contractor is required to have all the materials and equipment ready on site with the approved sequence of construction prior to the start of the shutdown construction

activities. Since this is a water plant project, disinfection on the treatment process units, clearwells and treated water pipelines are also required. Therefore, sufficient time for the disinfection and verification process needs to be included in the allowed shutdown time. This means that the Contractors actual construction duration is less than the allowed time. During this period, the Owners lab was standby the work to collect the bacteria samples when it was ready. Upon the confirmation of positive disinfection, the system was then put back in service. Besides mechanical pipe connection related shutdowns, there are other plant shutdowns associated with the new electrical service and SCADA system to the plant. To minimize the surprises and risks associated with the electrical and SCADA system conversions, all related parties including the Owner, the Owners operator, the Engineer, the Programmer and System integrator, and the Contractor were having multiple meetings to discuss the sequence of the shutdown and converting work as well as to establish contingency plans if there are any surprises. For the SCADA conversion, the programmer has performed simulation tests in the office and approve it works prior to the field work. During the conversion periods, sufficient resources were allocated to standby the site to cover any needed field correction work. With detailed planning and well prepared the resources, the proposed major shutdowns and system conversions were very smooth and match with the plan without any surprises. CUSTOMIZED TESTING AND ACCEPTANCE METHODOLOGIES The testing and acceptance procedures are the most critical and important component of any plant commissioning process. How items will be tested and when these tests are considered completed determines a successful transition, tests and acceptance methods, define what is checked and when it is acceptable. Equally important, the plant operations group develops a level of confidence with the work and new systems during the test and acceptance stage. On the project, the testing program was broken down into two major stages as detailed below. Operational Readiness Test or ORT: Testing by the Contractor of the installed system to verify operational integrity of the system under operational conditions without being placed in actual operation. Steps completed during this test would include aligning and leveling equipment, wiring and loop checks, calibrating all instruments, etc. During this function test, representatives of all major components for the equipment were on-site for timely identification and resolution of equipment issues. Performance Acceptance Test or PAT: Testing where the Contractor with the Engineer demonstrated to the LCRA the operational integrity of the installed work. PAT is broken down into two phases, PAT Phase 1 and PAT Phase 2. During PAT Phase 1, all systems were tested individually. During PAT Phase 2, all systems were tested simultaneously with the plant facilities to verify operational integrity of the whole of the WTCR WTP under operational condition prior to issuance of substantial completion certificate. The performance acceptance test verified the operation of the system under day-to-day usage, after installation and startup was completed and the system was fully functional. Any system deficiencies were noted, and system downtime was assessed. The PAT Phase 2 test was determined successful after 30 consecutive days elapsed with system availability within the specified limits. This test was performed by the Contractor, with support provided by the Engineer and plant operations group.

The planning phase of the project was broken down into steps, and for each of the systems, the steps were defined and a form completed. This form included the following details. 1) Basic System Information and Reference Drawings 2) Checklist and Associated Forms 3) Responsibilities of Key Personnel, broken down into lead, advisory, and support. 4) Equipment Readiness List, identifying the equipment and accessories which need to be in place for testing. 5) Operational Readiness Testing Requirements 6) Plant Conditions for Performance Acceptance Testing including: a. Plant Setup for Testing b. Duration of Testing c. Sequence and Requirements for Dry Test d. Sequence and Requirements for Wet Test e. Volume of Water Required f. Placement of Water 7) Test Reporting Requirements 8) Training Requirements This information was assembled and complied for each of the nine major systems. This Contract requirement was a team effort with the Contractor, Owners Management Team, Owners Inspection and Operations Group, and the Engineer. This process went through several iterations, starting with the easiest systems first, and followed by more complicated systems. In addition to the above planning documents, there were checklists and forms associated with the instrumentation and controls as part of the work. For all of the wiring on the project, loop status reports were completed. These loop checks confirmed that integrity of the wiring, but also checked wire tagging and terminations. In addition, instrumentation calibration sheets were completed for all instruments prior to the ORT. These instrument calibration sheets included records on setting up the instrument, identifying the range and value of the instrument, type of control, scale, input and output for each of the major components and associated loop checks. The Project Team started the planning and checking process at least 6 months ahead of the targeted substantial completion date to make sure there are sufficient time for the preparation and testing and to minimize surprises and unplanned plant interruptions. TRANSITION OF ROLES The LCRAs plant operations group is the end user and therefore, the project needed to be transitioned to the plant operation staff. This required several steps. First, the work needs to be substantially completed. In other words, the work needs to be complete to meet the design intent. Agreement on substantial completion requires input and oversight by plant operations so that they have a level of comfort to accept the items. Combined with the substantial completion is operator training on new process and new equipment. Third, there needed to be a transition period, when the equipment would be operated, but still have the Contractor around to address

any defect, deficiency, or adjustment items that may arose. And finally, there needed to be a formal handover and sign off of the equipment to be the responsibility of plant operations. At West Travis, each of the major systems and components were separate components which the associated testing and training could be done one at a time to the operations group. Although they were separate systems but there was a fair amount of overlap and connection between the systems. Thus, the transition of the newly constructed facilities to the Owners operation group needs to be a plant wide transition rather than by systems. This was accomplished with the 30 day PAT2 testing. During this 30 day PAT2 test, the plant operated all of the new systems together trying different scenarios and operational tests. This 30 day test was broken down into the following four major sub-categories: 1) Week one was for testing the plant in the primary automatic mode based on set treatment outputs. 2) Week two was to test the plant in secondary automatic mode based on system pressures and clearwell levels. 3) Week three was to run in primary automatic mode and take key equipment and elements out of service to check and verify key alarms and set points. 4) Finally, in week four of the PAT2, the equipment was operated in manual mode. During this 30 day test, records were taken and the goal was to give all of the new major equipment at least 200 hours operating time. In addition, during this 30 day test, all new major equipment was automatically started and stopped 10 times, manually started and stopped 5 times, and had the major alarms tested at least 5 times. Any defects were noted and repaired by the contractor. Our specifications allowed the test to be restarted, if a major defect was found, but this did not happen. At the end of the 30 day test, the plant was handed over to operations. LESSONS LEARNED Key lessons learned on the project are the following: Lessons learned from Phase 1 project guided the subsequent Phase 2 expansion, especially on selection of Contractors and I&C programming implementation. Proper planning is critical to success. In a complex plant expansion project like the WTCR WTP, testing and start-up can take up a good amount (recommend about 6 months) of the construction time from planning to complete testing for a successful project. The project team has planned ahead of time and had everything ready and in place several months before we were ready to use the newly installed facility to produce water. In the case of this project, there was no margin for error. Water was needed in the early summer of 2009 and the plant expansion had to be ready and operational to produce this water in less than 18 months. Experience of the project team is critical. Having an experienced commissioning team led to a smoother, more successful startup. We had a seasoned engineering team, with experience inspectors. LCRAs construction inspector, Levi Cash, was a former operator, and his work was extremely valuable during the testing and startup phase of the project. He spoke in terms that operators understood and the operations team knew that he would address their comments or concerns, if any. Teamwork - The commissioning team consisted of members from the Owner (LCRA), the Engineer (AECOM), the Contractor (Pepper Lawson and T.Morales), the Inspector (LCRALevi Cash), the Operator (LCRA), and I&C programmer (HEI). All involved parties were

working closely to meet the specification and schedule requirements. If there were any issues, the team was able to have open discussions to find a common solution. Contractor Buy in on the proposed testing plan is also a key. The LCRA and AECOM were putting a lot of thoughts on the sequence of construction and plant acceptance and performance testing. The Contractor is willing to work with the Project Team on preparing and setting up the testing work around the site constraints and operational limitations to minimize plant shutdowns and prevent unplanned interruptions. With the Contractors cooperation, the plan could be implemented successfully without surprises. There is a time associated with setting up and implementing detailed startup and testing procedures, but it is our opinion that the time to prepare and take the tests saves time in the long run because a rigid performance testing ensures a good quality product that prevent or minimize the follow up repairs and trouble shootings.

Overall, the customized testing and acceptance methods established for the WTCR WTP project worked well as planned. CONCLUSION LCRA has recently completed a major expansion and upgrade at its WTCR WTP. The completed plant is shown in Figure 3. The expanded plant was commissioned successfully in summer 2009 according to a detailed start-up and acceptance testing plan. The WTCR WTP facility has more than doubled the capacity from 9.8 to 20 million gallon per day with plant wide electrical and instrumentation system upgrades to meet significant growth and demands for increased water supply, occurred under extremely tight schedule and budget constraints and was a high profile project due to environmental and neighbourhood requirements. The success was accomplished through a partnership of the Owner (LCRA), the Engineer (AECOM), and the Contractor (Pepper Lawson) that focused on best project management practice, a detailed and systematic implementation and commissioning approach, a carefully coordinated sequence of construction, acceptance testing, and turnover plans that successfully commissioned the new facilities with no failures or breakdowns in plant processes or interruptions in water services. Figure 3 Aerial Photo of the Completed WTCR WTP