Accuracy in Quantitative Risk Assessment?
13th International Symposium on Loss Prevention
J.R.Taylor Dome Oillfield Services, Abu Dhabi JRT@ITSA.DK Abstract This paper describes the results of a study stretching over 25 years, to determine how accurate the risk assessment process can be made for process plants. During that time, risk assessment has become a standard working tool for safety engineering, not just in land use planning, but in design of safety shutdown and interlock systems, gas detection systems, building blast resistance, and fire protection systems. It becomes highly desirable that QRA’s should be just as accurate as calculations in other areas of engineering. The study used several methods of approach including QRA followed by a 20 year serial study of a number of plants; comparison of results from several methodologies and a large number of models against each other; comparison with actual accident reports and statistics; and third party reviews of QRA’s. The main results are that while models have been developed which agree very well with experiment, they are not always relevant to real life; that several very important accident types are generally ignored; that the accuracy within the plant and especially at short distances is much poorer than the accuracy at a distance; that the calculation methodologies most commonly used introduce unnecessary additional errors; and most importantly, that most of the deficiencies in current practice can be fairly easily corrected. Acknowledgements The authoer gratefully acknowledges the help in completing this study from Herlinde Beerens, Y Weber, Abid Sayyed, Tamir Said, Shastri Ranjiputri Accuracy in QRA accident frequencies The most important property of any practical application of scientific theory is that the results should agree with observation. Secondarily, the results should be repeatable. Until a few years ago, even repeatability in QRA was not achievable to better than two orders of magnitude (ref. 1). This was a saddening result, considering that QRA methodology has been under development for over 30 years. The cooperative study organised by RIVM and published in 2004 (ref. 2) showed that experienced analysts, using the same data, and a carefully made guideline, could produce result in agreement to within a factor of 2, even when using different software packages. This was encouraging in that it showed that consistency, at least, can be achieved. Even at the best, though, such calculations can only be “accurate on average”, that is, they are accurate for an average plant. If you have an average plant, then you are in luck. The data that are used in practice today for frequency determination are largely drawn from just a few sources, and to a considerable extent, from the North Sea oil and gas production
and this error is independent of any statistical uncertainty.036
Ratio Predicted/observed 1. and this is already generating a lot of plant specific failure rate data. Note that the issue here is not uncertainty in failure rate values. 6 Crude unit 4. with accident frequencies taken from ref.033
0.systems (e.1 0. and also derived as part of this project.70 0. excluding the 3 worst ammonia release plant 0.04 0. per year Statistics from Fyman. ref. 6). Others are often based on the original engineering judgements in the Rijnmond study (ref.62
Plant specific data may be undesirable in regulatory circumstances.053 Unit type
Predicted 5.12. and process fluid. for those plant types for which sufficient data are available. Given that actual failure rates for some items of equipment have vary between different plant types by up to three orders of magnitude (ref.5). Plant specific frequencies of major accidents derived using this approach were compared with actual major hazard accident frequencies.9*10-4 Alkylation unit 5.56E-04 0. The curious case of human error It is generally agreed that human error is one of the most important causes of major accidents. 4. feasible causes.500 plants of different types. whereas generic vessel frequencies in tables are of 5*10-5 to 5*10-7 per year i. 6) this is a very extensive collection of data from in all about 2.057 large Explosions 0.31 0. at least the large variations arising from different technologies and different process fluids should be taken into account. with dependency on technology. The generic values used are simply wrong for many cases. 7. This was confirmed from the accident reviews made as part of the present study. because of the complexity of deriving fairly precise values. 0. To get over this problem. this means that “accuracy on average” may be very inaccurate. However. The historical frequency of vessel rupture is about 2. such as in land use planning. Yet curiously.07 excluding 3 worst plant and ammonia truck loading Ammonia releases. It does not. 3). with updates through the years. It allows plant specific release frequency values to be used. flanges and valves. and generally none at all in counting opportunities for
. the RELBASE release frequency data base was developed (ref. many companies are already using risk based inspection as a tool.6*10-3 per year (based on US experience). An example of this is the urea reactor on older urea plants.1*10-4 Ammonia 0. a factor of 1000 inaccuracy. so far.45E-04 1. ref.e. 0. g. It simply does not make sense to use the same values for failure of piping for the case of simple carbon steel and for 360 SS steel for example. risk analysts working in the process industries spend large amounts of time counting pipes. Comparisons of prediction and observed frequencies are shown in Table 1. Looking to the future. allow process safety management variations to be taken into account. Frequency of UVCE. all sources synthesis. ref 7 and from ref.21.
given the number of accident reports available. 8). We did not find any good way out of this deadlock so far. using as examples the cases from the Dutch Yellow Book (ref. Examination of the sources of data used in commercial risk analyses show that this is not actually the case. gas dispersion and explosion. By far the biggest human error contributions to major hazards accidents are design error and management error (ref. It is possible to determine the frequency on a statistical basis.. 3. and the sources of these can then be investigated. but at least it allows anomalies to be identified. not waste time calculate their frequency. At distances over 100 to 200 m. The impact of model differences was assessed by completing QRA’s for six reference plants. Consequence modelling In order to assess the importance of modelling on QRA results. The results from this exercise were encouraging. but agreement with experiment to better than a factor 2 in 90% of cases seemed to be the usual case. This approach does not guarantee accuracy. Several commercial consequence modelling packages were obtained. 9. many of the models agreed with experiment to within a factor of 2 or better. several approaches were used: 1. A large proportion of the. It has been argued that release frequency data used in QRA’s already include an element of human error. 10).human error. The US RMP data which was used as part of the input to the RELBASE database includes a significant percentage of operator and maintenance related accidents. and results compared. 5. were less good. The model results were compared with published experimental data. Agreement at short distances. By contrast. i. Accounting for these presents a philosophical problem – if you can predict these errors it is best to remove them. Examples were:
. operator and maintenance errors often give consequence which are quite different from leaks and pipe ruptures recorded in QRA reference databases. 9). There were largely due to the common use of oversimplified or inappropriate models. The least consistent of the models were those for which turbulence is important. Some possibilities for investigation are described in ref. The model results were compared with accident results. The kinds of operator and maintenance errors which appear as causes of major hazards accidents are a fairly small set (ref.e. 4. Alternatively human reliability analysis methods can be used for analysis of the typical tasks which can give releases. 2.11) and the results compared. published models were implemented in one QRA package (ref. as needed for in-plant risk assessment for hazards to employees and for domino effects. investigating the difference. The impact of including human error into petroleum and chemical plant QRA have not so far been investigated as part of the project. The task risk analyses (TRA’s or JSA’s) carried out by many companies provide a useful basis for such analyses. In any case. HRA is regarded as a standard part of nuclear power plant risk assessments.
. for the UK HSE proved more accurate at short ranges (refs.12. Humber CAI. (Perhaps not surprising since the work is relatively recent and could draw on extensive earlier work). Jamestown Motiva Enterprises. Use of a single surface radiation intensity for pool and jet fires. spraying vent.Omission of these effects was found to be important in many cases. Reduction in pool size is the prime purpose of the slope. for example).)
Table 2 shows examples of accidents with their coverage in standard QRA methodologies. ref. Rouseville. splashing flow. hammer. UVCE HMI on valving. 1994 Giant Industries.• • • • • •
Modelling jet fires as simple straight flames (in fact. which has a very large effect on pool size. UVCE Y N N N Models available ? Y Y Y Y Y Y Y Y Y Y Y
Texas City 2005 Tosco Avon 1997 Valero Sunray 2008 Formosa Plastics 2005 Totale. rater than varying intensity Neglect of gas jet impingement Neglect of cross wind dispersion Assumption that all liquid releases occur on flat. HSCE Dead leg. jet fire Acid tank explosion Vapour release near hot work. KO drum overflow. non absorbent ground (in fact most chemical plants have a slope of about 2˚. Rew et al. and their treatment in standard methodologies.
Accident Mechanisms Included in standard methodologies N N Y Y N Y N Hammer N. RBI. UVCE Overflow. UVCE Confined VCE HMI. Delaware Pennzoil. HSCE Hydrocracker reactor runaway. examples of accidents. Most of the simplifications of the models result in underestimate of risk. jet fire Fork lift truck crash. HSCE. Especially the model reviews performed by Deaves. horizontally and obliquely directed flames always bend upwards) Neglect of flame drag and flame dip in tank fire and pool fire models. UVCE Injection point corrosion. 1995
KO drum overflow. Buncefied 2005 Conoco Phillips. so the cumulative effects are important. tank explosion
Table 2. 10 and 19 Models for all of these effects were found in the literature. Danvers 2006 Texaco. Milford Haven.
only a small percentage would be calculated using standard methodologies and tools. Lacunae in consequence modelling Many of the phenomena which occur in practice in process plant are not included at all in standard methodologies and model sets. For the full range of major hazard accidents. and chemical waste plants. Examples are: • • Splashing releases of volatile liquids. less than 20% would be calculated by standard model sets such as the Yellow and Green Books. rather than exceptions. in ref. For petrochemical plant. and using superposition of acoustic waves for calculation of explosion pressures. as at Buncefield Spraying releases of volatile liquids. evaporation. and Buncefield (ref.One perhaps surprising feature of modelling which became obvious from the comparisons with actual accidents was that hazard zones for all pool fires. coatings. or by commercial QRA calculation tools. Examples of accidents with extended linear geometries are the fire at Bellingham (ref. only one of which has well researched full scale experimental support. flash fires and explosions are calculated as circles in commercial QRA packages. which could be calculated separately and incorporated into QRA’s. However. It proved relatively straightforward to implement this kind of extended geometry. using terrain sensitive gas dispersion models for gases. and San Rafael de la Laya. which had to be developed. 6 . fires and explosions) as needed for many fine chemicals. 17) Reactor runaway explosions. 16) for the liquid flows. It could be argued that the omitted scenarios are exceptional cases. 16) (Note that spraying releases of liquefied gases are normally included in QRA’s) Boilover and slopover Fire induced tank explosion. depending on the case. it appears especially that spraying and splashing releases are the norm for liquids. Flame acceleration is the key problem. see table 2. A review was made of major hazard accidents investigated by the US Chemical Safety Board and the UK HSE. Cubatao (ref. That is the prediction of unconfined vapour cloud explosion pressures. (as occur fairly frequently.15). The error induced by these lacunae varied from a significant percentage to a very large factor. and as illustrated dramatically at Port Edouard Heriot. and as illustrated dramatically in the St. for example in fired heaters Dust explosions Indoor process accidents (releases.
• • • • • • • •
Models were available for all of these except fire induced tank explosion. Not all accidents are circular. as in most liquid releases from small and medium size holes in pipes. Herblain accident (ref. Lyon. One problem that requires considerable further research was identified from the comparisons of models with accident data. see ref. 13). Vessel rupture explosions (without fire) including boiler explosions Fired heater explosions Confined explosions. using models developed for environmental protection spills assessments (ref. 14). and many different mechanisms for this have been identified.
. both for continuous and batch reactions.
two. and worse. the Dutch purple Book (ref. mitigation calculations are essential. and often also many of the parameters which should be used. Whether to include domino effects at all. To support this study. confined explosion etc. The location of holes along pipes (one representative location. 18). to include just calculation of the frequency of domino effects. Use of four hole sizes is sufficient to give reasonable accuracy in the risk result. 21). 5) and the CCPS guideline (ref. The result depends on the use of usual pipe sizing and pump sizing rules. three or four) The selection of vessels to be analysed. increasing risk by only about 50%. Proper ALARP analysis including prevention measures requires that the accident scenarios in the QRA are related to hazop or high quality hazid studies. Domino effect calculations have moderate effect on risk to the public. three. Domino effects though are critical for determining safe evacuation distances and fire fighting procedures. Typical methodologies in use today are the World Bank guideline(ref. Applying vessel size cut offs.Choice of methodology Methodologies for QRA specify which scenarios should be calculated. so that only the larger vessels are calculated. four or five. an analysis of sizes for several hundred cases was made). For this study. The biggest source of inaccuracy in analyses is the accident types which are overlooked or neglected. or analysing only for isolatable sections. and whether each vessel should be analysed.
. domino effects dominate the risks. with pipe runs typically 10 to 50 m. or the demanding calculation of both the sizes and frequency of domino effects. or groupings of vessels in an isolatable section. The parameters investigated were: • Choice of scenarios. For long inter unit pipes with large pressure drops. characteristics of methodologies were investigated in order to determine the effect of methodology choice on results (ref. makes accident escalation calculations impossible. just leaks and spontaneous ruptures. Use of three locations for holes along pipes is sufficient to achieve reasonable accuracy for in plant analyses.19). or a full range of scenarios which could be identified by hazop studies such as overpressuring. More locations may be needed to allow accurate assessment of escalation. as provided for in different methodologies. The range of hole sizes considered (two. overflow. Whether to include mitigation calculations in the methodology. Since the ALARP criterion is increasingly being used. the lacunae scenarios described in the previous section were not included. makes it very difficult to make rational recommendations on risk reduction. This compares well in comparison with the 100 largest incidents published by Marsh (ref. For petrochemical plant asset risk.
• • •
Some conclusions from these comparisons are: • Focussing on a standard tables of release frequencies given in the literature as the basis of the methodology results in overlooking a large fraction of the total. To keep the analyses comparable. pipeline release profiles may be needed. 21) showed that only about 1/3 of these involved escalation.
were developed during the 1990’s for land use planning. The most important improvement is to make sure that the accident scenarios calculated are the ones which occur in practice. 27 Nov . The price for improvement in accuracy is that more thorough assessments are required. but not “black box”. to the same level as other engineering disciplines? It appears difficult to improve models for gas dispersion much until better experimental results become available. and particularly. or possibly. K. Is accuracy really necessary? Could it be just accepted that repeatability and consistency is sufficient. Similar problems arise for explosion calculations. The main problem with this approach is that it does not provide a good basis for risk reduction and ALARP analysis. “factor of two accuracy” seems the best to be achievable for most consequence calculations. Vol. improvements in CFD techniques will allow a priori calculation of dispersion. Hazardous Materials. Earlier guidelines such as refs. Amendola.
Conclusions Quantitative risk assessment has become an important tool in plant safety engineering.O. It also appears possible to improve frequency calculations.1 Dec 2000. ASSURANCE: Assessment of uncertainties in risk analysis of chemical establishments. However even the most demanding of the analyses with all vessels and major pipes in a refinery (450 vessels and associated piping. Amendola. All results for each stage of the calculations are tabulated). The additional cost of calculations themselves is not a significant factor. which means that it will lead to hazard types being neglected unless the model set is complete... though manageable effort. which will need better instrumentation that used until now. Kozine. References 1.
.. Markert. however. 200 inter-unit pipes). Ziomas. using more thorough methodologies. These methodologies. though factor of two accuracy seems difficult to guarantee except under the best circumstances. 8 and 18 have provided an open and well researched basis for the use of QRA. 5. increasing the scope of accident scenarios covered. Factor of five accuracy seems achievable in a wide range of cases. I. Contini.. F. the criteria for risk acceptance can always be adjusted to take into account an estimated factor of error. Uncertainty bands in results have also narrowed considerably as more data have become available. The present study indicates that accuracy can be much improved by using more recent models. 1. The collection of data as a basis for the calculations. M. however. Can this make the analyses truly accurate. Lauridsen. Uncertainties in Chemical Risk Assessment. In: Proceedings. For now though. A. including process parameters for all vessels containing hazardous materials in a plant.Most of these conclusions imply a considerable increase in the amount of analysis work. Also. represent a significant. International conference on probabilistic safety assessment and management (PSAM 5). the regulatory model approach is very inflexible. three hole locations for each vessel and pipe required only a few hours of calculation on a modern laptop computer. as is the case for regulatory models? After all..: Results of a European bench mark exercise J. with four hole size classes. 1992 and Christou. (The calculations are automated. Osaka (JP).
BP International Ltd. Technical Reference Manual 7 edition. 2009 9. 100 Largest Accidents. Loss Prevention in the Process Industries. Furuta. 1993. 1996 11.E. and Mouilleau Y. Report No.R. The Blaze at Shell Storage at Port Edouard Heriot Harbour. 13. HID Statistics Report HSR 2001 002. The Buncefield Incident 11 December 2005. NTSB/PAR-02/02 14.. 221031. Safety Science (2006). Lees.). HMSO 2008 16.P. Kayes. 2000) (Frontiers Science Series) p. (Universal Academy press. F. RIVM report 6100660015/2001 3. Butterworth/Heinemann 15. TNO (1996).2006. 17. 2005 to 2008. 1996 8. 1985 and 2002 editions 21.F.R. 2000 20. J. 2007 12. Washington June 10. B. Guidelines for quantitative risk assessment.Taylor Hazardous Materials Release and Accident Frequencies for Process Plant. Rijnmond public Authority Risk Analysis of Six potentially hazardous Objects in The Rijnmond Area. London. Understanding and combating design error in process plant design.M Ale et al. World Bank. “Yellow Book”. Current Status and Advances in Flash Fire Modelling”. Taylor Associates. 2004 7.Kondo.Taylor. J. The QRA Quality Project Vol 1 to 7. CCPS. Health & Safety Executive 4.R.J. 1985: Manual of industrial hazard assessment techniques. Human Error in Process Plant Operations and maintenance. P. Director General for Social Affairs and Employment 1999 6. October the 7th 1991.J. J. Deaves D and Maddison T (1995). Pipeline Rupture and Subsequent Fire in Bellingham.014 10. cited in Guidelines for Evaluating Process Plant Buildings for External Explosions and Fires. Available on request
. Antwerp. A Screening Methodology for Assessing Adequacy of Blast Protection for Control Rooms and Other Onsite Buildings. Benchmark risk analysis models. Methods for the Calculation of the Effects of the Escape of Dangerous Materials.ssci. J.Taylor.. France. Tokyo. CCPS. S.Belgium. J.1016/j. 1999. Reidel. Loss Prevention and Safety Promotion in the Process Industries 18. C. (ed.Taylor. England 19. 1978.A case study : Saint Herblain. 369-374 2. 1982 5. QRA Quality Project. Chemical Process Risk Assessment.. 1993. Rew P J. B Ale et al.Taylor QRA Pro. 2 and3 June 1987. Marsh McLennan.. Assessment of an accidental vapour cloud explosion .08. pp 333-348. (eds. CCPS.). Sunbury. Symp.R. 6th Int. Fryman. Lechaudel J. K. Purple Book. Mansot. Dutch Ministry of Labour. New Orleans. “Offshore Hydrocarbon Release Statistics 2001”. Proceedings of International conference on modelling and mitigating consequences of accidental releases of hazardous materials. HSE (2002). Vol 8. Proceedings of the 8 th International Loss Prevention Symposium. Buncefield Major Incident Investigation Board. J. doi:10.