MTA/OIG Report #2012-01

April 2012

THE LIGHTNING STRIKE AND LONG ISLAND RAIL ROAD SERVICE DISRUPTION—SEPTEMBER 29, 2011
Barry L. Kluger
MTA Inspector General State of New York

INTRODUCTION At approximately 4:30 p.m. on September 29, 2011, the beginning of the evening rush, lightning struck near Long Island Rail Road (LIRR) tracks, creating a power surge that disabled the signal system controlling the train interlocking just west of Jamaica Station (called the “Jay” signal hut).1 Limited train service was operating after 5:30 p.m. on manually set routes west of Jamaica while technicians diagnosed the situation. Approximately three and a half hours after the strike, in an attempt to repair a computer server believed to have been damaged by the power surge, a LIRR employee erroneously disabled the separate signaling system controlling the train interlocking just east of Jamaica Station (called the “Hall” signal hut). At that point, all service was suspended. It took some time for the LIRR technical crew to identify what occurred and then fix the problem at Hall, which was not fully functional until approximately 10:30 p.m. As a result of these two events, passengers onboard nine trains standing at platforms and 17 stranded between stations were in limbo — sometimes moving eastward, sometimes moving back west to Penn Station. Other commuters were stuck in Penn Station and Jamaica Station for hours seeking alternative means to reach their destinations or waiting until service resumed. Limited service was not restored until 12 midnight, with full service back at 4 a.m., almost 12 hours after the power surge. Given that LIRR provides over 280,000 rides per weekday for its customers, most of whom use the LIRR for commutation in the morning and evening, this outage caused a serious disruption for tens of thousands of commuters. The affected signaling equipment was relatively new, having been designed by Ansaldo STS (ASTS) under a contract awarded in 2003 to a predecessor firm and in use only since 2010. ASTS had been required under the contract to provide lightning protection suitable for a state of the art facility. In the days following the September 29th lightning strike, LIRR reached agreement with ASTS and a second engineering firm, Systra Engineering (Systra), to conduct an investigation of the incident, to determine how the signals were disabled. LIRR asked the engineering firms to determine what actions were needed to provide more lightning protection
1

In railway signaling, an interlocking is an arrangement of signal apparatus that prevents train collisions and other conflicting movements through an arrangement of tracks such as junctions or crossings.

MTA Office of the Inspector General

1

MTA/OIG Report #2012-01

April 2012

and to make recommendations to ensure that such an event cannot happen again. As a result of that inquiry, LIRR and ASTS reached an agreement for ASTS, substantially at its own expense, to provide a series of improvements to the railroad's signal system in Jamaica in order to greatly reduce the chances of another outage like the one that followed the September lightning strike. The Office of the MTA Inspector General (OIG) reviewed the circumstances of that lightning strike and its aftermath, including LIRR’s investigation, to be sure that the railroad and its consultants identified and carefully analyzed all of the critical factors contributing to the outage and produced an effective action plan. As part of its review, the OIG examined the 2002 Request for Proposals (Signals Design RFP) and resulting contract for the design and construction phase services work as well as reports on the system failure produced by ASTS and Systra. Further, OIG conducted numerous interviews with LIRR operational, supervisory, and management personnel. OIG also interviewed representatives from both Systra and ASTS. Our review found that actions by LIRR and ASTS jointly weakened the Jamaica Interlocking signal system, leading to failure. Specifically, OIG found that: ● In accordance with its contract, ASTS designed the new signaling system for the Jamaica Interlocking but LIRR employees installed it. During the installation, LIRR added a piece of computer equipment called a “serial server,” which was not part of the ASTS design. This server allows LIRR to remotely monitor various pieces of the equipment. In the course of attaching the server to the new signaling equipment, a LIRR employee used one incorrect connector. ASTS, LIRR, and Systra all agree that this connector created the pathway by which the power surge generated by the lightning damaged the signal system and brought it down. ● The surge protection designed by ASTS included main and redundant systems, but both failed once the power surge entered the main system, because the twin systems were not isolated from each other. ● LIRR personnel performed deficient Quality Assurance/Quality Control both during and after installation of the new system. Specifically, LIRR failed to detect both the installation of the wrong serial server connector as well as the non-installation of certain components shown on the original ASTS design. ● The diagnostic tools pre-programmed by ASTS into the new signaling system failed to pinpoint which critical components were not functioning. This complicated LIRR’s identification of the failure’s cause, thereby extending the duration of the incident. ● ASTS did not provide LIRR with operating manuals for the system as a whole, nor did ASTS provide LIRR with adequate troubleshooting procedures. Additional training of LIRR personnel by ASTS on troubleshooting could have mitigated the duration of the

MTA Office of the Inspector General

2

MTA/OIG Report #2012-01

April 2012

outage and prevented the human error that brought down the signals at the second signal hut. ● LIRR employees did not have adequate replacement parts to diagnose and correct system problems. For its part, ASTS did not provide LIRR with a list of critical spare parts after its design was completed. ● The LIRR Signals Department was unaware of a separate contract modification with ASTS to provide emergency response services in situations just like the lightning strike. Further, ASTS failed to provide LIRR with the contract-required phone number and email address to obtain immediate emergency assistance. LIRR did not attempt to contact ASTS using existing known contact information during the first five hours of the disruption. ● Believing that it had contracted for and installed a system providing appropriate redundancy and protection, LIRR was not adequately prepared for this emergency. During the course of its investigation, OIG discussed its concerns with LIRR regarding the agency’s lack of preparedness and ASTS's design limitations, as well as the railroad’s installation deficiencies. LIRR and its consultants have finished their investigation and released an action plan that addresses the weaknesses that were identified by LIRR, the engineers, and the OIG. The plan proposes the addition of more surge protection; better isolation of redundant systems in the huts; modification of pre-programmed diagnostics in the equipment; implementation of enhanced Quality Assurance and Quality Control inspection procedures; and addressing training and operator manual deficiencies. Our involvement in this process also provided an opportunity for us to revisit a review we performed with LIRR several years ago. Specifically, in 2007, following five serious service interruptions caused by downed power lines and right-of-way fatalities that demonstrated weaknesses in LIRR’s emergency response procedures, OIG worked with the railroad to evaluate these procedures and recommend ways to strengthen them where necessary. Thereafter, with substantial input from LIRR, OIG issued a report entitled “Response to LIRR Service Disruptions, Winter 2007; MTA/OIG 2008-03.” In that report LIRR committed to changes in staffing, technology, and operations intended to better equip the agency to handle large-scale service interruptions. As part of our current review regarding LIRR’s response to the lightning strike, OIG staff examined whether the LIRR implemented and utilized the improvements promised in 2007 during its response on September 29, 2011. Based on our review, we note that in the intervening years LIRR took significant action to make its operational response more effective and efficient. These included the creation of liaisons for Penn and Jamaica stations to monitor conditions and equipment needs; the distribution of cell phones to train conductors to facilitate the exchange of information; and the relocation of reserve engines to reduce response time for stalled/malfunctioning trains. However, we believe that two of the promised improvements — increased staffing levels during emergencies and more

MTA Office of the Inspector General

3

MTA/OIG Report #2012-01

April 2012

straightforward communications with passengers — are not yet fully realized. In particular, we found that: ● As with its disruption in 2007, LIRR is still unable to secure the necessary level of customer communication staff required to adequately disseminate information to passengers onboard trains and customers at stations. ● Despite the development of a communication strategy that aims to provide detailed and informative content, the substance of onboard messages still does not adequately and consistently explain travel conditions and offer useful information that allows customers to evaluate alternate travel options. In our next section (Background), we briefly discuss LIRR’s approach to lightning protection. The remainder of this report is then divided into two parts. The first sets out our findings and recommendations regarding the signal system failure and LIRR’s lack of preparation for this emergency. The second part reflects our findings and recommendations regarding the continued need for increased staffing levels and improved communication with passengers during emergencies. While all of these recommendations for improvement going forward are the product of our own review and analysis, they were surely furthered by the frank and cooperative discussions we had with Systra and the operating, supervisory, and management personnel at LIRR. In the same spirit, we have recommended that MTA request that its own Independent Engineering Consultant (IEC) review the upgrades planned under the agreement between LIRR and ASTS to confirm for the benefit of LIRR, the MTA Board, and the public that all necessary steps are being taken to provide the appropriate level of lightning protection. The LIRR, with whom we shared our findings and recommendations throughout our review, has confirmed acceptance of all of our recommendations and agreed to provide us with interim reports regarding their implementation.2 For its part, the MTA agreed to ask its IEC to review the upgrades planned by the LIRR; indeed, the IEC has already commenced its review. We are encouraged by the prompt and cooperative responses by the LIRR and MTA. Certainly, we will continue to monitor the state of the lightning protection utilized by LIRR and will take any further action made necessary or appropriate by the IEC’s review.

2

This confirmation is contained in a letter dated March 30, 2012 from LIRR President Helena Williams responding to each recommendation individually. We have reflected that response in words or substance in italics below the corresponding recommendation.

MTA Office of the Inspector General

4

MTA/OIG Report #2012-01

April 2012

BACKGROUND LIRR management has long recognized that the decades-old electromechanical switching systems guiding trains from one track to another were antiquated and required replacement with newer electronic technology. Accordingly, in 2003, LIRR signed a contract with a firm (US&S) to design new electronic switchgear for the three signal huts (Jay, Hall, and Dunton) that control the Jamaica Interlocking. US&S was purchased in 2008 by ASTS. While US&S/ASTS designed the new system, LIRR employees had to install it given their collective bargaining agreement with LIRR. Because electronic signaling systems are naturally susceptible to lightning strikes, LIRR had in the years leading up to the Signals Design RFP and contract taken the following actions:    In 1989 LIRR established a Lightning Protection Committee, which issued a technical report in 1994 with findings and recommendations. All signal projects approved in the 2000 – 2004 Capital Plan (including the Jamaica Interlocking upgrade) underwent a review for lightning protection. In 2002, LIRR hired a consulting firm, Electro Magnetic Applications, to examine best practices for the implementation of lightning protection in LIRR’s territory.

LIRR incorporated the lessons learned from these and other actions into its Signals Design RFP and contract for the Jay Interlocking upgrade. Two of these lessons had particular significance for LIRR in its overall lightning protection plan. First, the signal switching system had to be “redundant;” meaning that the main switching system had to be backed up by a separate but equal (redundant) system. Second, but at least as important, the main and redundant systems had to be configured to prevent common mode (i.e., single act) failures. Interestingly, while correspondence among LIRR and bidders on the Jay Interlocking upgrade evidences LIRR’s commitment to a redundant system design that could not be disabled by a single lightning strike, that correspondence also foreshadows the risk to the integrity of that design from the quality of LIRR’s installation of the system. Specifically, LIRR required written confirmation that no common mode failure would shut down both the main and redundant systems. ASTS replied that it did not foresee any common mode failures that would shut down both the main and redundant systems in the proposed design “provided diverse routing and sources are used for power and recommended wiring, grounding, and surge protection practices are adhered to by construction forces.” (Emphasis added by OIG). Following the award of the signal switching system contract to ASTS’s predecessor in 2003, LIRR employees began installation in 2003, and the system went into operation in 2010. ASTS was retained to provide Construction Phase Services. These services included participation in system cutover (the transition between the old and new system), as well as the provision of training and other post-cutover support to LIRR employees.

MTA Office of the Inspector General

5

MTA/OIG Report #2012-01

April 2012

PART ONE: SEPTEMBER 29, 2011 THE SYSTEM FAILURE As made clear above, a specific requirement of the contract with ASTS was a design incorporating redundant signal systems. To be effective, the redundant design called for two independent and isolated switching systems, each capable of operating the system should the other become disabled for any reason. This meant that should one system fail, the second system would “take over” and continue to operate the switches with no interruption of service. ASTS provided a design for each signal hut that included power supplies, two separate systems of computer processors or “boards” that controlled and monitored the switches, and two separate terminal servers that connected and communicated with the boards and other necessary equipment. The number of boards in each signal hut within the Jamaica Interlocking differs depending on the number of switches controlled by each respective location. All switches controlled within each hut are grouped together into zones, with each zone requiring two boards, one for the primary system and the second for the redundant system. One additional board per hut is required by the Federal Railroad Association (FRA) for monitoring purposes. Thus, the Jay signal hut includes 13 boards (six boards each on the primary and redundant systems plus the FRA board). Similarly, Hall signal hut includes 11 boards and Dunton signal hut includes five boards. Nevertheless, on September 29, 2011, both the main and the redundant system in the Jay hut were incapacitated by the single lightning strike. Preliminary findings of Systra, ASTS, and LIRR indicate that the failure of both the main and redundant systems was caused, at least in part, by a wrong connector used by LIRR personnel during the installation of the serial server, which facilitates remote monitoring of the diagnostics by connecting all of the boards in each hut to a maintainer’s computer. What made the connector wrong for the installation, and disastrous for the systems, was that it contained a built-in path to ground. This allowed the surge to enter the system and flow through both the main and redundant systems. By installing the serial server, LIRR created an association among the boards that was not anticipated in the original protection plan for the system design. LIRR employees installed the server without apparently seeking input from ASTS — most basically, without having ASTS incorporate the component within its design so that compatibility and any potential consequences of the added equipment could be assessed in timely fashion. Clearly, future installations must avoid similar failures of communication. As we said earlier, this first recommendation and others throughout this report for improvements going forward are products of our own review furthered by frank discussions with LIRR and its consultant.

MTA Office of the Inspector General

6

MTA/OIG Report #2012-01

April 2012

Recommendation 1 Prior to installing any modifications affecting any signal design, LIRR should confirm design compatibility with the designer. In December 2011, LIRR reported that it directed development of a plan to ensure that “[a]ny pre, concurrent or post inter-discipline work activities must be coordinated with the designer of record to ensure design liability and reliability.” Further, LIRR reports that it is contracting with consultants to improve its “Configuration Management” processes, meaning the way it introduces changes in equipment and/or technology into existing systems. LIRR Response: The LIRR accepted the recommendation and confirmed that it is moving forward with outside expertise to revise its “Configuration Management” processes. LIRR Needs to Improve Quality Assurance (QA) and Quality Control (QC) When LIRR installed the serial server, connectors were required to enable two dissimilar connections to be joined. The correct connectors were designed to be fastened securely using two screws each and contained no built-in path to ground. LIRR procured "kits" enabling the installer to assemble the required connectors. During installation the installer was one kit short and instead used a look-alike connector on hand in the hut. To the contrary, though, the substitute connector contained a grounding wire and, because it was not specifically designed for the required purpose, the screws failed to line up to secure the connection. LIRR staff reported that this enabled the connection to “wiggle” and become “intermittent.” According to LIRR, because of the reported intermittent nature of the connection, the ground was not detected when testing of the system was performed. Interviews with LIRR personnel involved in the installation and testing of the system both before and after cutover, indicate that they tested the circuits several times a day during installation, and at least weekly since cutover in October 2010, but never detected the grounding effect. LIRR has been unable to explain why the extensive testing procedures described by its personnel failed to detect the flaw in the system. Because adequate Quality Assurance and Quality Control procedures should have detected the different physical appearance of the connector through visual inspection, we remain concerned about the adequacy of those procedures. We are also concerned because the QA/QC function is not currently independent of the department(s) charged with performing the construction or installation in question. The failure analysis conducted by LIRR and its consultant after the September 29th incident also identified other concerns with the installation and testing procedures for the signals project. LIRR management, as well as Systra in its report, noted that while three surge protective devices were shown on the contractor’s Jay circuit plans to be installed on the primary power circuits, they were not actually installed by LIRR. While these devices were not installed on the primary side as designed, other protection devices were installed on the load side as required. The omission should have been timely detected by Quality Assurance and Quality Control. Except

MTA Office of the Inspector General

7

MTA/OIG Report #2012-01

April 2012

for damage to some relays, no other damage to the power circuits occurred in September. Nevertheless, their omission further evidences a pattern of inadequate oversight.3 Recommendation 2 LIRR should make the Quality Control/Quality Assurance function independent of the department(s) charged with performing the construction or installation in question. The oversight encompassed within this function must ensure compliance and compatibility with design standards at all stages of the process. In its December 2011 directive, LIRR consolidated the QA/QC functions, formerly in the Engineering Department and the Strategic Investments Department respectively, into the Department of Program Management. LIRR informed us that this newly formed department will initially focus on signal projects. At LIRR’s request, MTA Audit Services is scheduled to assess the unit’s activities late in 2012. The OIG certainly supports LIRR’s QA/QC initiative. However, until the unit is fully operational, interim measures, including the employment of outside quality control/assurance experts, must be utilized to ensure that existing installations have been properly performed. LIRR Response: The LIRR accepted the recommendation, confirmed the above progress, and added that along with reviewing protocols used to disseminate standards and specifications, LIRR intends to strengthen written lightning protection protocols including means and methods. Near-term field expertise will be provided by a third-party contractor while existing staff will provide the core effort for both process and procedural quality control oversight. Expertise from independent consultants will be used until permanent staffing can be provided. Avoiding Future Failures LIRR and OIG agree that the railroad and its riders must have confidence in the system’s protection from future lightning strikes. Toward that end, the failure analysis conducted by LIRR and its consultant Systra determined that additional surge protection should now be incorporated into ASTS’s design. For example, Systra concluded that to reduce the redundant system’s exposure to lightning, each system should have an independent battery source instead of one shared source. Systra also recommended adding “isolation modules,” to protect communication ports between adjacent boards, further increasing system protection. LIRR informed the OIG that it has already initiated a number of corrective actions. For example, LIRR reported that it replaced the incorrect connector in the Jay hut and confirmed that the other connectors in that hut were properly installed. Additionally, because ASTS also supplied the
3

LIRR reported that it installed these devices following the incident.

MTA Office of the Inspector General

8

MTA/OIG Report #2012-01

April 2012

signal design for other LIRR interlockings, Systra sensibly advised a current inspection of these other signal locations to determine if these locations raise similar concerns and, if so, that these concerns are addressed. LIRR reports that its Engineering department performed this inspection. LIRR is also reportedly working with ASTS and Systra to evaluate improvements for isolating the main and redundant systems. LIRR further informed OIG that it is retaining consultants with special expertise to assist in developing the new QA/QC unit’s responsibilities including independent testing and inspection of signal construction activities. In addition, LIRR reported that it is using this lightning event to redouble its efforts regarding lightning protection in general. A Peer Review is planned with MTA New York City Transit and MTA Metro-North Railroad to discuss lightning and surge protection best practices. Going forward, Systra is reportedly researching the most effective lightning and surge protection technologies to ensure that LIRR truly has state of the art protection.

MTA Office of the Inspector General

9

MTA/OIG Report #2012-01

April 2012

LIRR WAS NOT PREPARED FOR THIS EMERGENCY In addition to the physical causes of the system’s failure, several other factors, including LIRR’s belief that it had contracted for and installed a system providing appropriate redundancy and protection, impeded LIRR’s ability to diagnose and correct the problem. These issues, while not directly causing the failure, hampered efforts to bring the system back online, and extended the impact and duration of the incident for LIRR’s ridership. Diagnostic Program Failure; Missing Replacement Parts When LIRR technical personnel responded to the Jay hut, they could not determine where the damage occurred because there was no visible evidence. While the technical crew could determine that there was no communication occurring between the boards and the server (meaning no exchange of information), the diagnostic program for the boards did not reveal to them that this breakdown in communications resulted from damaged communication ports because the ASTS pre-programmed diagnostics did not cover these ports. Without this information, the crew assumed that the boards were operating and initiated a troubleshooting methodology that involved replacing other components in the system in an attempt to diagnose the problem. Having now incorrectly ruled out the boards as the most likely cause, the technical crew then assumed that the terminal servers were the likely source of failure. Finding no spare terminal server in the Jay hut, the crew borrowed a server from the redundant system in the Hall signal hut. Significantly, though, each terminal server is programmed with its originating hut’s unique location on the system. Because the “address” for Hall is different than that for Jay, the borrowed terminal server first had to be reprogrammed with the Jay address. In reprogramming the location, the LIRR programmer made an error that shut down the signaling system at the Hall Interlocking. After Hall went down, some members of the technical crew were diverted to the Hall signal hut to troubleshoot that problem. Other signal maintainers began manually setting the tracks (a process known as “blocking and spiking”) so that limited service could operate through the interlocking. Although the cause of the problem at Hall was identified, Hall was brought back on line, and no physical damage resulted to any Hall equipment from the system crash, the Hall Interlocking was inoperable for almost two hours, significantly complicating LIRR’s ability to return all trains to service. Once the communication ports of all the individual boards were identified as having failed, it was necessary to replace the six boards and the one server of the main system in the Jay hut to bring Jay back on line. However LIRR staff did not have enough spares to replace all six boards. Because the three huts at Jamaica (Jay, Hall and Dunton) all used the same signal equipment, LIRR was able to take boards from the redundant systems at Hall and Dunton to temporarily repair Jay.

MTA Office of the Inspector General

10

MTA/OIG Report #2012-01

April 2012

As part of its proposal, ASTS was required to submit a minimal “recommended spare parts list” that LIRR would need to have on hand after cutover to the new system. ASTS provided a list that included only two boards and no terminal servers. More significantly, perhaps, given industry practice, ASTS did not provide LIRR with a list of critical spare parts after its design was completed. LIRR told OIG that it independently opted to procure one spare terminal as backup for all three huts, but that it had been removed to test new software updates. As for spare boards, although two were procured none were available during the incident. The presence of replacement terminals and boards would surely have reduced the amount of time spent resolving the emergency. On the other hand, “borrowing” the terminal server and the five boards from Hall, as well as a board from Dunton, increased the vulnerability of the latter two huts and actually brought one of them down. The time required to travel to the different locations and bring parts back to the Jay hut also extended the incident’s duration. Most significantly, if replacement boards had been on hand in Jay, LIRR could have used them as part of its troubleshooting, leading to a much faster diagnosis of all causes of the outage. Recommendation 3 Each signal hut must have a full complement of essential replacement parts, including servers and boards, to enable installation of a complete system, if necessary. LIRR has advised the OIG that it has purchased, programmed and installed all necessary components damaged or relocated during the incident. This effectively restored each hut to its pre-incident status with fully operational main and backup systems. LIRR also purchased sufficient spare parts to replace an entire main or backup system in each hut if necessary, and has pre-programmed all spares and stored them directly in their respective huts to facilitate repair. LIRR Response: LIRR accepted the recommendation and has completed this activity. Operating Manuals and Training Were Deficient The OIG interviewed several members of the LIRR technical crew who either directly responded to the Jay hut following the incident or were otherwise involved in restoring the Jay Interlocking to operating status. None of the personnel interviewed believed that the operating manuals for the system would have been useful in the problem diagnosis. In that regard, the department head commented that it was fortunate that certain employees with experience were available that night given that turnover in the department has been significant. Additionally, Systra found deficiencies in the troubleshooting sections of various operating manuals. Specifically, in eight separate recommendations concerning the Jay Maintainer’s Manual, Systra’s report noted that the manual requires updating to provide “comprehensive, integrated, real world step by step instructions to address system-wide (global) failures, such as occurred on 9-29-11.” Systra noted further that the short section in the manual on network

MTA Office of the Inspector General

11

MTA/OIG Report #2012-01

April 2012

management software offers little to no guidance during the type of catastrophic multiple equipment failures experienced during the lightning strike and needs to be supplemented with more reference material and proper training. Further limiting its usefulness, “the manual” is actually a collection of multiple stand-alone manuals supplied with each component of the equipment by the major suppliers. Therefore, it has no cross references, indexes, or even a table of contents. LIRR installed the new electronic switchgear technology specifically because the old LIRR electromechanical interlocking equipment was no longer supportable and replacement parts for it were no longer available. While LIRR forces have extensive experience on the replaced electromechanical equipment, the new systems are more reliant on computer hardware, software, and other electronics. Consequently, designers and suppliers must provide proper training, documentation, and suitable follow-up services to ensure that LIRR personnel are sufficiently prepared to appropriately install, operate, maintain and troubleshoot these upgraded systems. Recommendation 4 LIRR and ASTS must properly prepare LIRR personnel who install, operate, maintain, and/or troubleshoot signal equipment, with complete and well organized documentation, enhanced by sufficient training and follow-up services to prevent emergency conditions to the extent possible, and resolve emergencies that do occur in effective and timely fashion. LIRR has advised OIG that it is currently working with ASTS to improve the built-in software diagnostics in the computer boards. LIRR also reports that specific procedures for reprogramming the signal system software have already been developed and are in place to remedy the server address issues that crashed the Hall signal hut and prolonged the outage. LIRR Response: LIRR accepted the recommendation and, as noted above, has completed activities associated with software programming procedures. LIRR will continue to work with ASTS to improve software diagnostic tools. ASTS did not Provide Contract-Required Emergency Contact Information The OIG found that ASTS, as part of its contract, was required to provide to LIRR a phone number and e-mail address for emergency Office System Technical Support for one year ending on November 30, 2011 – a time period that included the incident on September 29th. This phone and e-mail support was to be available 24 hours a day, seven days a week, and 365 days a year. Indeed, the contract specified that an emergency involving a “safety issue,” defined as the “system is inoperative, it has a critical impact on LIRR’s operations, and/or safety is a concern,” requires that ASTS respond within 15 minutes. The entire failure of both redundant systems in the Jay hut surely qualified for this support, which certainly should have lessened the impact of the incident on LIRR and its customers alike.

MTA Office of the Inspector General

12

MTA/OIG Report #2012-01

April 2012

LIRR technical personnel involved in the incident have stated that they never received the contact information, nor were they aware of the 15-minute-response requirement prior to the OIG showing them the documentation. ASTS confirmed to OIG that it had no record of providing the contact information to LIRR. However, the LIRR technicians did have a contact list of ASTS personnel involved in the design and post cutover support tasks that ASTS was performing for the railroad. In any event, no LIRR personnel used the contact list until approximately 10 p.m. – five and a half hours after the lightning strike. LIRR technical managers acknowledged that following this contact, an ASTS manager had an ASTS technician call LIRR promptly, though at this point, according to the LIRR managers, ASTS did not have a role in the recovery. LIRR should have, and utilize, every means available to promptly restore service. Here again, ASTS shares responsibility in that regard. Clearly, ASTS should have provided contracted-for support information, and LIRR personnel should have been aware of and pursued the support. At the least, LIRR personnel should have promptly utilized the contact list to facilitate recovery from the catastrophic failure. The railroad informed us that effective February 2011, it put in place what it calls a “Commissioning, Acceptance and Maintenance Plan” (CAMP). Although the rollout of this Plan preceded the September outage by nine months, it post-dated cutover of this project, which occurred in November 2010. Therefore, according to LIRR, CAMP did not apply to this project, meaning that there was no verification of prompt distribution of systems-related information, and of compliance with other procedures involving “commissioning, acceptance, maintenance, and warranty requirements” now required by CAMP. Recommendation 5 LIRR management must maintain, manage, and utilize support services provided to the railroad under warranty and service agreements. This and other systemsrelated information must be promptly distributed to appropriate personnel, be posted in accessible locations, and remain readily available in the event of a disruption or otherwise. Further, LIRR should revisit all pre-CAMP projects to determine which completed projects would benefit from CAMP, to ensure the appropriate distribution of systems-related information to such projects. LIRR agreed that this incident illustrated a need to enhance existing procedures to insure that systems-related information is promptly distributed, posted, and readily available. LIRR Response: LIRR confirmed acceptance of this recommendation and clarified that “the emergency service contract involved provides only for software services which are associated with the office side of the system. That contract does not provide for emergency response with regard to hardware located wayside.”

MTA Office of the Inspector General

13

MTA/OIG Report #2012-01

April 2012

ACTIONS GOING FORWARD In the wake of the disruption caused by the lightning strike near Jamaica, LIRR took immediate action to shore up what the agency found were technological weaknesses in the lightning protection design of its new electronic signal system at that location. Toward that end, as noted, LIRR and ASTS, the system’s designer, reached an agreement pursuant to which ASTS will provide, substantially at its own expense, an upgrade designed to greatly reduce the chances of another catastrophic outage. This upgrade will include additional surge protection, diagnostics, troubleshooting, and enhanced isolation of critical components. These design changes should provide additional layers of protection against single act failures, such as another lightning strike, by either eliminating the risk or significantly reducing the impact. As we also noted earlier, LIRR and OIG agree that the railroad and its riders must have confidence in the system’s protection from future lightning strikes. In this regard, certainly, LIRR acted prudently by involving its signals consultant in this review. And we commend the railroad and its contractor ASTS for promptly reaching an agreement that promises additional protections for the railroad and its riders. In our view, though, confidence in the system requires further, independent assurances that the protections are sufficient. Our review found that the Independent Engineering Consultant (IEC) under contract to advise the MTA Board had not been asked to assess these design upgrades. The IEC advises the Board on issues affecting capital projects on an ongoing basis and is available for special projects. OIG believes that this additional layer of oversight of the signaling upgrade is warranted given the significant importance of avoiding future catastrophic disruptions. Recommendation 6 MTA should request that its IEC review the upgrades planned under the agreement between LIRR and ASTS to confirm for the benefit of LIRR, the Board, and the public, that all necessary steps are being taken to provide the appropriate level of lightning protection. MTA Response: The MTA agreed to independently review the upgrades planned under the LIRR/ASTS agreement and associated reports to determine whether all concerns have been adequately addressed. Through its Independent Engineering Consultant the MTA has begun its review and will report its results to the Capital Program Oversight Committee of the MTA Board, the LIRR, and the Office of the MTA Inspector General. The OIG will continue to monitor the state of the lightning protection provided by LIRR, and will take any further action made necessary or appropriate by the IEC’s review.

MTA Office of the Inspector General

14

MTA/OIG Report #2012-01

April 2012

PART TWO: ONGOING PASSENGER COMMUNICATION DEFICIENCIES During the worst of the 2007 disruptions — as well as on September 29th — LIRR was unable to provide adequate staffing levels in two passenger communication areas for the evening rush hour. Throughout many of its departments, staff typically begin work early and complete their shift at or about 4 p.m. each weekday. Once employees leave the property at the end of the day, they are not readily available to respond to emergencies during the evening rush, and it is difficult to call them back; in part, because they may be dependent for return transportation on the same LIRR train affected by the emergency. Staffing at the LIRR Public Information Office A major improvement born of the previous joint report was the creation of a new Public Information Office (PIO) regularly staffed, on a 24-hour basis, with one Public Affairs (PA) representative and one Assistant Station Master (ASM). In the event of a major service disruption, PIO staff doubles to two Assistant Station Masters and two PA representatives. As an added enhancement, a fifth staff member, outside the PIO, is assigned the role of Customer Advocate, which was created at the recommendation of the OIG and included in the 2007 joint report. 4 The responsibilities of the Assistant Station Masters include dissemination of service information to train crews and the Public Address Announcer, retrieval of crew cell phone numbers for contact, completion of required logs, and the update of internal records to reflect crew communications. PA representatives distribute service information to media outlets and customers via web postings, text messages, email alerts and message boards, while coordinating with ASMs to ensure message uniformity. The Customer Advocate obtains information from the operational command center (Command Center), and disseminates it to PIO staff; ensures that protocols such as those regarding crew communications and record keeping are followed; and monitors standing and/or stranded trains until they reach their destination. 5 The Customer Advocate ensures that phone contact is established with each affected train in order to assess passenger and other conditions until those trains reach their destinations. To facilitate real-time communication among the Assistant Station Masters and the PA representatives, the Customer Advocate moves between the PIO and the Command Center. On September 29th, the additional PA representative was onsite. However despite several attempts to bring in more staff, only one Assistant Station Master was present. Senior staff in the Movement Bureau reported that calls were placed to numerous off duty Assistant Station Masters that evening to fill the second position but were unsuccessful. As a result, the Lead Transportation Manager assigned to the Customer Advocate role instead had to cover responsibilities that otherwise would have been handled by the second ASM.
4

The Customer Advocate is typically a seasoned employee within the Transportation Department, such as a lead transportation manager or road foreman. 5 A “standing” train refers to one kept at the platform of a station where passengers are able to disembark, while a “stranded” train is one stopped in-between stations where passengers are unable to leave.

MTA Office of the Inspector General

15

MTA/OIG Report #2012-01

April 2012

The staffing vacancy exacerbated the problems stemming from the outage. By 8:05 p.m., when Hall Interlocking became disabled, some trains were cancelled, others were stranded between stations, while still others remained at station platforms, unable to proceed. Emergency crew reassignments made it hard for the PIO to determine which crew was on each affected train. To expedite communications, crews were instructed by text message to contact the PIO. The Customer Advocate explained to OIG that having the PIO receive crew calls regarding onboard conditions seemed preferable to having the PIO initiate contact, because receiving the calls eliminated the need for the ASM to hunt for crew phone numbers. Under the circumstances, though, the Manager said that he found it challenging to handle phone calls from all affected trains with only two employees and believed that the PIO’s duties would have been more easily executed if a second ASM had been secured. According to OIG analysis of available records and interviews of personnel in the PIO, the staffing shortage predictably made it harder for them to compile and disseminate progress reports and other information. Records indicate that although train crews were instructed by text message at 6:48 p.m. to contact the PIO, there was no contact between the PIO and 12 of 23 trains that were standing or stranded between stations. During the period when several crews phoned in to report their conditions, the understaffed PIO tried to contact two stranded crews who had not called in, but those efforts were unsuccessful. To strengthen staffing in the PIO in the event of an emergency during the evening rush hours when fewer employees are available, LIRR has now permanently assigned a second Assistant Station Master to the PIO during the applicable period. By doing so, LIRR avoids the uncertainty associated with requesting that staff return to the property for an event occurring after personnel have left for the day. Customer Assistance in Stations According to the Jamaica Emergency Action Plan, which governs agency response to major service interruptions occurring in the Jamaica Station vicinity, 53 management assistants are needed to provide adequate on-site customer support for a major afternoon/evening peak service disruption. An Assistant General Manager assigned to Station Operations confirmed that the number 53 is for planned or expected service outages, e.g., a forecasted snow storm. For sudden disruptions he assumed that the same number of employees — 53 — would be a reasonable target. To meet the designated number of management assistants needed at Jamaica and other LIRR stations in the event of an emergency, LIRR employs what it calls the Customer Assistance Program (CAP). CAP consists of 400 to 415 non-union employees who are trained to provide onsite customer information during service disruptions and summoned on short notice. On September 29th, LIRR management sent a “mass email” to CAP participants requesting assistance, but only seven were ultimately present and providing support at Jamaica through midnight of that day. Another sixteen non-CAP employees (e.g., station cleaners) were also

MTA Office of the Inspector General

16

MTA/OIG Report #2012-01

April 2012

pressed into varying types of service that evening, some providing direct customer assistance, even though these employees did not have CAP training. The on-site responders, even those participating in CAP, were handicapped in their efforts to inform passengers about service options because there was no effective mechanism in place to provide the responders themselves with real-time operational information. For example, at Penn Station, customers received information from email alerts and message boards that service was suspended west of Jamaica and that they should use the E subway line to travel to Jamaica Center. However, CAP participants at Penn Station could not confirm that if a passenger got to Jamaica, the LIRR was actually running east of there. As part of efforts to improve onsite response since September, LIRR reorganized the program so that the more than 400 CAP participants are rotated out of their respective work locations in groups of 20, and allocated between Penn Station and Jamaica Station for two to three weeks each year to familiarize those participants with these stations and have them available during the evening rush. These BlackBerry-equipped CAP participants are on duty through 7 p.m. every weekday during this period and receive intensive training on the emergency plan, applicable procedures, and customer interaction. Should an evening disruption occur at Jamaica, the 20 CAP participants are already onsite at those stations to provide customer support. To help achieve the total number of management assistants needed for the disruption — currently established in the Jamaica Action Plan at 53 — a mass email will go out to recruit additional CAP participants. However, given that the number of additional participants that are likely to respond is unknown, it is not clear to us how LIRR can be confident of meeting its current stated goal of 53 individuals. To resolve this dilemma, we believe that LIRR must first establish to a reasonable degree of certainty the total number of individuals that it actually needs in the event of a disruption to provide appropriate customer support. To begin, consideration should be given to multiple factors, such as the extent to which the presence of newly installed electronic reader boards and/or the 20 additional well-trained managers would reduce the need for some portion of the 53 “management assistants.” On the other hand, passenger counts may have increased since that number (53) was originally adopted, suggesting that even more qualified employees need to be on hand to provide assistance. Further, mindful of the delicate balance between the importance of the function and the limited resources available to accomplish it, the plan must include a cost-effective means for assuring that the target number of people needed can reasonably be met. One option for filling the additional demand for staff during emergencies could be to change the regular work schedule of other personnel already onsite, so that more of them remain there after 4 p.m. when, as noted above, daytime staff typically complete their shift. In any event, LIRR must have a plan that is feasible to execute during the evening rush on a moment’s notice.

MTA Office of the Inspector General

17

MTA/OIG Report #2012-01

April 2012

Recommendation 7 To best meet customer needs in future emergencies, LIRR must:  Determine analytically how many additional staff are needed at its major hubs to supplement existing customer assistance personnel on duty during major disruptions on short notice. This number should be specified in the agency Emergency Action Plans. Devise a cost-effective plan for producing the requisite staff at the major hubs in the event of a disruption.

LIRR Response: The LIRR accepted this recommendation and intends to undertake a review of the Emergency Action Plan (EAP) to determine the appropriate level of staffing required. LIRR stated that upon completion of that review it will evaluate the current Customer Assistance Program plan “in order to make any necessary adjustments in a cost-effective and efficient manner.” Content of Passenger Communications LIRR introduced the PlainSpeak initiative in 2007 to “provide customers with accurate, timely and useful information that is easy to understand, particularly to those already aboard trains. The communication style uses brief, concise and conversational language, without hard-tounderstand technical details or railroad industry jargon, to describe the impact and length of service disruptions.”6 Nevertheless, passengers interviewed by OIG about their experience on trains during the outage reported that a number of announcements confused and/or frustrated them. 7 For example, they expressed a desire to be better informed about specific operational issues, rather than merely being told that LIRR was experiencing “massive delays” or that there were “signal problems.” In terms of the quantity and quality of the communication, their concerns tracked those of passengers interviewed by OIG in 2007. A number of passengers reported an onboard announcement indicating in words or substance that there was “congestion up ahead in Jamaica and [they were] waiting for paperwork to begin moving [the train forward].”8 Aside from the frustrating ambiguity of the term “paperwork,” it inadvertently masked and unfortunately trivialized the critical purpose for holding up the train: to ensure safe travel through areas with dozens of employees along the track bed. The announcement would certainly have been more meaningful and reassuring if “paperwork” was replaced with “authorization” and included a reference to safety protocols.

6 7

See “Response to LIRR Service Disruptions, Winter 2007 MTA/OIG 2008-03” (at pp. 3-4). OIG reviewed 180 emails and letters submitted by affected customers and conducted in depth interviews with 13 passengers. Additionally, OIG sought to independently determine the verbatim content of information actually transmitted to passengers onboard stranded trains but was unable to verify it. 8 Though not explained, the paperwork involved was the Clearance Card (C-card) process by which trains obtain authorization to move past each inoperable signal while a crew member records relevant information on the card.

MTA Office of the Inspector General

18

MTA/OIG Report #2012-01

April 2012

In another instance, passengers aboard a separate train could not reconcile their travel experience following a declaration that they were “a priority train.” Understandably, their expectation was that their train would proceed east, ahead of all other affected trains, once service resumed. To the contrary, though, after being stranded west of Jamaica for approximately one hour they were returned to Penn Station. Operations personnel told OIG that the announcement was unauthorized and they could not theorize why a crew would use the term “priority train” in that context. We also noted that many passengers affected on September 29th were unaware of the constructive steps the LIRR was taking to restore service and believed that the LIRR had no plan. In particular, passengers on stranded trains did not know that efforts were underway to open up routes so that they could progress eastward once tracks were manually fixed (i.e., blocked and spiked) to allow for limited service. LIRR surely recognizes that its customers want clearer and more specific information about disruptions in service. And it has made adjustments to improve and control the content of onboard announcements. Toward that end, train conductors have been reinstructed to refer to the revised section of their procedural manuals. This section contains approved text for clear and concise messages intended for authorized announcements. However more detailed information delivered to onboard passengers about the nature of a major delay and recovery efforts are needed to further PlainSpeak goals. For example, on September 29th, riders could have been informed to the effect that “the signaling system is currently down but the LIRR is working out a safe way to allow for train movement at reduced speeds. We will keep you informed.” A subsequent progress report could have explained that “we have secured two of five routes and are continuing recovery efforts. We will keep you informed of our progress.” Instead, the first PIO text sent to crews for announcement onboard stopped trains was sent 25 minutes after the strike, once the recovery plan had been decided, and stated that “service is temporarily suspended in both directions between Penn and Jamaica and between Atlantic terminal and Jamaica due to weather-related signal trouble at Jay interlocking.” That limited statement, which raised more questions than it answered, did not provide passengers with any sense of the specific and continuing efforts being made on their behalf. That said, OIG recognizes that the levels of information provided and of expectations raised or lowered are subjects of legitimate debate. During this recent emergency, LIRR management consciously refrained from disseminating information about specific recovery steps to train crews, expressing concerns that such information could have erroneously been perceived as operational instructions, which are solely within the purview of the Command Center. Also to be considered, whether the approach is full or restrained communication, are customer anxiety and dangerous efforts at self-evacuation. Whether and to what extent information can be safely communicated are certainly issues the railroad should explore consistent with its goal of keeping passengers appropriately informed. In any event, though, we found that the substance of onboard messages still does not consistently achieve the primary PlainSpeak objectives: namely

MTA Office of the Inspector General

19

MTA/OIG Report #2012-01

April 2012

to adequately explain travel conditions and offer useful information that allows customers to evaluate alternate travel options. To help passengers better understand their situation and the expected length of the delay, LIRR has revised procedures regarding onboard announcements and devised a matrix consisting of average resolution times for 18 known conditions. The railroad should build on these efforts. Recommendation 8 LIRR should further develop and refine its protocols to facilitate the dissemination of appropriate information to passengers on stranded or standing trains regarding why they are stopped, and the plans being pursued and progress being made to get them going again. LIRR Response: The LIRR accepted this recommendation and will continue to develop and refine current protocols to ensure effective delivery of appropriate customer information.

MTA Office of the Inspector General

20

MTA/OIG Report #2012-01

April 2012

CONCLUSION The Office of the MTA Inspector General finds that the LIRR and ASTS, its lightning protection designer, share responsibility for the crippling effects of the power surge and its aftermath following a lightning strike near Jamaica Station at the beginning of the evening rush hour last September 29th. Following that strike, LIRR reached agreement with ASTS to investigate the incident, along with Systra, the LIRR consultant, to determine its cause and to make recommendations to ensure that such an event cannot happen again. As a result of that inquiry, LIRR and ASTS reached an agreement for ASTS, substantially at its own expense, to provide a series of improvements to the railroad's signal system in Jamaica in order to greatly reduce the chances of another outage like the one that followed the September lightning strike. OIG reviewed the circumstances of that strike and its aftermath, including LIRR’s investigation, to be sure that the railroad and its consultants identified and carefully analyzed all of the critical factors contributing to the outage and produced an effective action plan. Furthermore, our current review regarding the lightning strike provided an opportunity for us to revisit a review that we performed with LIRR in 2007 involving downed power lines; our purpose now being to determine whether the LIRR implemented and utilized during its response on September 29th, the improvements it promised several years ago. In short, we find that the power outage and subsequent delay resulted from ASTS's design limitations and the railroad’s installation deficiencies, the critical deficiency being the use of a single wrong connector to add a remote monitor to the system. In addition to the physical causes of the system’s failure, several other factors, including LIRR’s belief that it had contracted for and installed a system providing appropriate redundancy and protection, coupled with insufficient preparation for this emergency by ASTS of the railroad and its staff, impeded LIRR’s ability to diagnose and correct the problem. These issues, while not directly causing the failure, hampered efforts to bring the system back online, and extended the impact and duration of the incident for LIRR’s ridership. We also found that since 2007, the LIRR took significant action to make its operational response more effective and efficient. However, we believe that two of the promised improvements — increased staffing levels during emergencies and more straightforward communications with passengers — are not yet fully realized. The LIRR, with whom we shared our findings and recommendations throughout our review, accepted all of our recommendations and agreed to provide us with interim reports regarding their implementation. For its part, the MTA not only agreed to ask its Independent Engineering Consultant to review the upgrades planned by the LIRR, as we recommended that it do, the IEC has already commenced its review. We are encouraged by the prompt and cooperative responses by the LIRR and MTA. Certainly, we will continue to monitor the state of the lightning protection utilized by LIRR and will take any further action made necessary or appropriate by the IEC’s review.

MTA Office of the Inspector General

21