Abbott DIAGNOSTICS Six Sigma Metric Analysis for Analytical Testing Processes Sten Westgard, MS, Westgard QC Lah ikers tesco Laboratorias sock objective assesemant and comparison of analytical methods and instrumentation performance. Unfortunately, there are few ways to compare systems ona level playing fild to mako an “apples lo applos" comparison ‘Current mathods of assessment can be arbitrary relying on unclear ‘site of the ar assessments, or focusing moreon easily tangible efficiency metrics, such as speed, cost, or ease of use. Analytical goals and requirements for the: quality doivered by a tost are offon overlooked during ‘ho decision-making process loading to the purchase of instrumentation. Rapidly changing regulatory schemes increase the confusion over accaptable standards for instumen! and mathod quality. ‘A technique to objectively and quantitatively assess the perlormance of methods, instruments, and laboratories is laid outin this paper. Tha technique consists of three ‘components: (1) the Six Sigma matric, a widaly-accoptad measure of quality management, process improvement, _and universal benchmarking; (2) quality requiremants in ‘the form of specific quantitative goals for analytical tosts; _and (2) performance data from method validation andi ‘verification studies or routine laboratory data, One way to understand how Sigma metric analysis combines ‘these three components is to picture a target with an arrow (Figuro 1). The shape of tho target is determined by Six Sigma matrics. The size ofthe targotis determined by the size of tho quality requirement. Whore the arrow hits that ‘target is determined by the method performance data. Figure t ‘Sigma Metric Analysts provides not only an objective assessment of analytical methods and instrumentation, ‘but it also provides the critical design information needed {or operational implementation. The Sigma Matric analysis process leads naturally to a Quaitty Control (QC) (Design scheme using quantitative and graphic tools to ‘determine the necessary quality contral procedures for routine monitoring of methods and instruments. CHOOSE TRANSFORMATIONCc) Abbott DIAGNOSTICS rr eee ean ‘Sx Sigma isa widel-accepted aualty menagerent system, ‘perhaps best known outside of heathcare as the product Glinnovaion at General Elecinc ard Motowia' Sx Sgr isaiso wel known cr the colertl ites of is practioner geen batt (partie Sis Sigma worker), back bet (flim Sx Sigma worker), maser black bet (consultant to baack belts), and champion (executive proponent of Six Sigma ‘eflorts). Sx Sigma has been acopted by both manufactring ‘and corvce industries, as well as heathoare inetittions form hospitals reference leboratores. ‘Si Sgma isa metre that suantfios the perlarmanse of [processes as arate of Delects-Pe-Nilion Opportunites, (OPM, oF DEMO) Sx Sigma programs aiso encompass ‘busi techniques such as Deline- Measure Anelyzo- ImproveGonttel (OMAIO) an Foot Cause anya to ind ‘and slmnais dslacts and variation within a process The goalot Six Sigma, in is smplest distillation, isto ‘lminate or reduce all varaticn in a process. Veriaion ina process leads io wasted effort and rescurces on retesing and workarounds for example. Reducng detects reduces costs, and improves performance ‘and profitability. A process that achieves the goal of ‘Six Sgma dalivers noth quality and eficianey “The quartitaive goal of Six Signa is to create a process that sinivizee vavition unt standard deviasone can ft wihin the tolerance it (Figure 2) Althe level of Six Sigma performance (weld class qualty performance], approximately {tree defects will occur per milion cpportunites. _— many laboretories use them fo alltesing prozesses The msuse of 2s mis in aboraiony testing frequently results inerroneously epeatad coniols, excessive trouble-shooting, cr worse ail, workarounds tha! atficialy wen conta iets to the poirt that Idborstovias can no longer detect eral anaiyical errs. Pat ofthe power of tho Sx Sigma soa it abily to provide @ universal Benchmark. Sigma metrics alow Comparison of diferent processes wih each cher, even Comparing processes across dierent nstutions and diferent industries, For example, afine safety fs known be beter han Sis Sigma wih a rate ofonly 15 crashes por milion departures, wale arine baggage handing inthe US. ison 41 Sgma since aproumately 1% of uggace is misplaced r ket and US. alive departures perform st only 2 3 Sigma sina neatly S03 of lights are delayed ‘which helps fo explain chronic customer complaints. In hoaltheara, the Sigma performance af commen processes are less well nown. When the institute of Medicine sued is landmark repod, To Er is Human? ‘tfamcusty reveated that between 48,000 and 90,000 Lnsecessary deaths occurred in US. hospitals every year. Examining the cleath rates at tha hospitals that formed the basis ofthe study reveals that healthcare is perforring at only 38 Sgma. It healthcare were acheving Six Sigma, the death rate would be only 1€ 10 84 deaths per yes Nevalsiners® greundoreaking work in Sigma assessmert inthe clincal lab analyzed the perforrance of comrron laboratory procesecs and found that many were woo!ully inadequate: 3 SS ‘%Eror DPM Signs 3| ‘Order accuracy sax | 18000 | 360 3| Dipacae oe a [m0 |S F tard eros robo ees [ose [eso Tiedt Sug Morar TOW imigenos [asa [avcno [=o fe Se te to ta 41 Os to 20 ee Be Ge eratingyspesnen sep) a CE Front Rania epee 0 mea Sale (Charsiny pecinen sap as [ame fame fd alowabitotal err (Tea) predctng dsc ‘Supapataloy pore cessonna [se | saaoo_ [350 “The Six Sigma ceale typically une from zero tozix, but | ony sseenenefeaoy 7a | raroa [295 process can actually exceed Six Sgma, ifvariablity is [Lapomiry patingy sing cs [eae [sas ‘sufficient low as to decrease the defect rate. h industries TSrgza pana cen soa outside of healthcare, 3 Sigma is considered the minimal | aagnosteascouance or |r fame ‘acceptable periormance for e process. When performance | per smaresceampssrmames [oe [mame [ae falls below 3 Sigma. the process 's considered to be —=—_ ——— cessentialy unstable and unacceptable. sora Incontrast to other industia, heabheare and clinical lcboratores appear to be cperating ina 210 3 Sigma, ‘envionment. The routine use of "2s" (ie, 2 standard {deviations or 2 SD) conto! imis is indicative ofa complacent tradition h quality control practices. Despite the wellknown problems of 2s mits — they can generate alse rejection rates ‘fupto 10 to 20%, depending onthe numberof controls run Figure 3: Sa matics cl conmanatoraiy proossess,Nealahen data Revising the arrow end target mode, Si Sigma provides the shape of the target. The shane defines the goa! of SixSigma performance as the bulls eye, as well a the inner rings of ‘and 5 Sigma. Outside the 3 Sigma ring, perfermance is Considered iohave missed the target and the process is not ‘considered tobe "fit or purpose.” CHOOSE TRANSFORMATIONCc) Abbott DIAGNOSTICS Defining Quality Requirements Knowing the shape ofthe target is not enough. The size must also be described. In Six Sigma terminology, the tolerance limits must be defined. In the clinical laboratory, the quality required by an anaiyiical testing process must be defined. Tolerance limits, in the laboratory, are best expressed a3 a total allowable error (TEa) specification. Ta is a well-sccepted concept in healthcare laboratories, ‘as @ model that combines both the imprecision and the ‘inaccuracy (bias) of a method to calculate the total impact ona test result“ An allowable total error isthe expression ‘of how much combined imprecision and inaccuracy can bbe tolerated in the test result without negatively impacting ppatient care based on interpretation ofthat result Determining the quality required by a laboratory testis not 2s simple as it sounds. Most laboratories do not know the: ‘analytical quality required by their tests. Indeed, many laboratories assume that itis not even necessary to know. ‘As long as there are no direct complaints about testing ‘qualiy, many laboratories assume that the analytical qualiy they are providing is adequate. Ths isnot the only crippling assumption that laboraicries make. Sometimes, laboratories assume that the quality of any testis sufcient simply because a manufacturer buit the instrument and ‘made the reagents. While its common to assume that no ‘manufacturer would produce instruments and reagents that perform pooty itis not good laboratory practice. Finally, laboratories frequently assume that simply folowing the manufacturers directions is enough to assure the quality ofthe tests they provide. Again, the fact that a manufacturer provides directions does not guarantee that the directions ‘are adequate. Professional standards as well as regulatory requirements place the burden of selecting appropriate ‘quality control procedures on the laboratory, spectically the laboratory director. Part of the dificully for laboratories is defining quality specifications. Decades ago, only a few sources existed Fortunately, a wealth of quality requirements and targets have become available and sre easily obtainable. Fist ‘and foremost, US. laboratories are governed by CLIA proficiency testing guidelines. For nearly 80 analytes, CLIA provides specific quality requirements. Other naltical benchmarks are provided by proficiency testing programs, external quality assurance programs, cr peer groups. Ouiside the US., some quality specifications are availabe from the Royal Collage of ‘Australasian Pathologists (RCPA), as well as the Guidelines of the German Medical Associaton (RIB). Clinical benchmarks can also be used to generate quality requirements. Dr. Carmen Ricas and her colleagues have provided a continuously updated database of biologic variation since 2000. For over 300 different analytes, they have tabulated desirable specifications for imprecision, inaccuracy, and total allowable error* ISO 1169, the ‘new international ab accreditation standard for quality in laboratories, also provides guidance on anaiyical testing. Finally, the growing body of research on Evidence-Based Laboratory Medicine (EBLM) Guidelines can be used to develop Clinical Decision intervals. These intervals can, in turn, be used to determine quality requirements for individual tests. At the very least, a laboratory can consu the clinicians who use its test results and, by documenting haw test results are interpreted, determine the quality required by their testing processes. Establishing quality requirements, returning to the arrow and target model once more, determines the size of the target. Since the use and performance of diferent tests varies, 50 too does the size of the target that the artow/process must hit. Together, Six Sigma and Quality Requirements provide the shape and size ofthe target. Now all that remains is determining where (and i) the arrow hits the target. For that, we need data on the actual performance of the process. CHOOSE TRANSFORMATIONUsually, Sigma performance is assessed by counting defects, then by converting that count into a Defects Per \ilion Opportunities (DPM, or DPMO) rate. Once the DPM is known, a Six Sigma table, available in standard text- ‘books, can be consulted to obtain the Six Sigma metric. Counting defects relies on two capabilities. First, it must be possible to define what a process defect means” Second, ‘Rtmust be possible to detect a process defect when i occurs. For most processes, these are simple tasks. Mest processes analyzed in Si Sigma projects use the counting defect approach. In the laboratory, counting defects is also the usual Six ‘Sigma metric technique. For example, turn-around time (TAT is very easy to define for laboratories. A laboratory might st a target of returning test results within 60 minutes cf specimen receipt. Thus, when a test result is returned after 61 minutes, its simple to detect the defect (ie., TAT {is > 60 minutes). Counting the number of defective test results (> 60 minutes) aver a period of time is an easy way to determine the Sigma performance of the laboratory's TAT. For laboratory test results, however, determining and detecting defects is more difficult. When a single test result is generated, ifs not possible to Know what the true value ofthat test result should be, even ifthe sample is tested multiple times. For example, if a cholesterol test result is 212ma/dl, the “rue value" ofthat testis not known, unless the Specimen was also analyzed by an accepted reference method. Thus, ifs unknown if the result falls, within the tolerance limits or quality requirements. If the true value is 190 mg/dL, the observed test result is probably a defect. I the true value is 206 mg/dL, the observed test results probably acceptable. But without knowing the true value, there is no way of courting how many defects are being generated by a testing process. Fortunately, there is another method of determining the ‘Sigma metric of a process: by measuring variation, Converiently or laboratories, measuring variation through the use of controls is part ofthe daily routine. Controls are a known value, so variation of an observed test result can be measured. With multiple contol results, information on the standard deviation of testing processes can be collected {and the imprecision (coeficient of variation, % CV) can be calculated, Information about the inaccuracy (bias) of an analytical testing process can readily be calculated by results between the testing method and a reference method, or by analyzing the resuts of the testing ‘method in proficiency testing, peer group, or some other form of external quality assurance program. DIAGNOSTICS Menace ee Ideally, the data on imprecision and inaccuracy is collected during the same time frame and at the same critical level (medical decision level) of test interpretation. In other words, the data on performance should be an accurate snapshot cof method performance at a specific point in ime and at a specific concentration of analyte. Thus the resulting Sigma metric best reflects actual test performance. For example, if the critical levels in the lower end of the dynamic range, & bias estimate should also be obtained from the same concentration range, or the regression equation from @ ‘comparison of methods study can be used to estimate bias at the critical level. For tests with muitiple critical levels, it may be desirable to make Sigma metric estimates at each level The relationship of imprecision and inaccuracy to Sigma rmetics can be graphically depicted (Figure 2). Given @ normal distribution of test resuits, as well as a known standard deviation (the imprecision) and a known bias, the acceptable performance range can easily be calculated and, conversely, the concentration ranges in which results are unacceptable can also be defined (.e., the concentration ranges above and below the tolerance mits that define TEa). The relationship between imprecision and inaccuracy to Sigma metrics can be summarized mathematically by the following equation: Sigma metric = (TEa — bias obsened)/CVobserved Asimple example with the Sigma-metric equation reveals that the “stale ofthe art” in healthcare is not Six Sigma. For cholesterol, CLIA defines an allowable total eror of 10%. That is, a cholesterol test resuit must be within 10% ofits true value. The National Cholesterol Education Program (NCEP) established separate goals for imprecision and inaccuracy of 3% each. A method that performs with 33% CV and 3% bias is considered acceptable by the NCEP. The Sigma metric calculations tell another story: (10-3)/3 = 2.38 Sigma This is star proof that laboratories currently operate in an environment in which world class performance is nat the goal Returning once again to the arrow and target model, the Sigma approach gives us a target, the quality requirement gives us the size ofthat target, and the performance data Of the method give us the arrow, which should land as. close to the bulls eye (Six Sigma) as possible. CHOOSE TRANSFORMATIONCo) DIAGNOSTICS Abbott Measuring Six Sigma Performance in the Laboratory (continued! Even when data on method bies is missing, a modified Sigma Normalized Method Decision Chart metric can be calculated. The resuiting metric documents the capability of the method to achieve world class performance under ideal concitions when no bias is present. Sin the real laboratory always operates with some amount of bias, the performance observed will ahvays be lower than the Sigma capabily. The benefit of such an assessments that it allows the laboratory to estimate how much “room for eror” is left after accounting for imprecision. For some instruments, even Sigma capabiliy metric will allow a laboratory to make judgments on the suitability of methods. eve naecursy, Bias eee esa eee Sigma metric capability = (TEs)/CVobsenea ” The performance of methods can be graphically illustrated using a Method Evaluation Decision chart (MEDx, Figure 4) with Six Sigma metic lines imposed upon them. The Figure §: Normatzed Method Decision chart Method Decision chart displays inaccuracy on the y-axis, imprecision on the x-axis. Typically the chart is drawn for Normalized Method Decision Chart each specific quality requirement (ie., a 10% quality requirement would use @ Method Decision chart drawn for 10%), but muitiple methods with different quality requirements can be displayed on a Normalized Operating Specifications (OPSpecs) chart. In a Normalized Method Decision char, the axes are each set to 100% and the ws x and y values are determined for the test by calculating 3 its percentage of the quality requirement (Figure 5). fo. For example, if test had a quality requirement of 10% and a 2° CV of 1% anda bias of 2%, the coordinates on a Normalized ae a Method Decision chart would be (10, 20) (Figure 6). te ay ON Method Decision Chart TEa=10% °e 7 = = = 0 ‘Observed imprectson, CV Figure 6: Normazed Method Decision chart with sample cata pot While Normalized Method Decision charts with Six Sigma limits incorporate many complex features and calculations into a single display, the result ofthe chart sil is within the arrow and target model. The chart can be visualized as the upper right quadrant of the target. The area around the origin (0,0) of the chart (and below all ofthe lines) is the bull's eye. The Sigma lines drawn on the chart are similar to the rings ofthe target, with 3 Sigma representing the edge ofthe target (anything below 3 Sigma is considered off the target, i.e. unacceptable). The x-and y- coordinates of a plotted test represent the performance: of the test and the spot where the arrow “landed.” Observed naccuraey,% Bias BEBE SE CHOOSE TRANSFORMATIONCc) Abbott DIAGNOSTICS Meee Given the simple parameters of the Sigma-metric equation, laboratories can easily determine the current performance’ of altheir current methods. The data acquired during the standard method validation protocols of a new instrument ccan also be used to determine performance metrics. In ‘addon, formal method validation studies are typically conducted on all new metheds ard are often presented 4s posters. scientific conferences, or as more scholarly reports published in scientiic journals, or simply as data rca SE eee ea rrovided by the manufacturer upon request. With that ata, laboratories can calculate Sigma metrics, compare them to the Sigma metrics of competing instrurvents, and use ths toal as part of their dession-making precess. This application, to objectively assess and compare instrument rrerformance before purchasing a new instrument, is of considerable value. It gives the laberatory the power fo fredicl which methods will peform to thet clinical needs | ‘and wiich will nat. Pxeeneea OPSpecs Chart TEa=10% with 90% Error Detection $8825 88 és Allowable Inaccuracy s . 20] ew at, | Sy - 7 z tas2oeFgcltis. 25s, a ba Scart faaas fag 355 1 20. 30 40 Allowable Imprecision (s, %) ‘Sigma metic analysis is not confined tothe role of ‘assessment and methcd validation, Sigma metics can also be used torefine and ereamine the operational routines of, ‘amathod, Cambning Sigma metres with OC Design ‘ook, ‘suchas the Operating Specifications chart (OPSpecs), allows the lsboratory lo customize and optimize the OC ‘procedures conducted by the laboratory. A rational QC Design can eliminate much ifnot al of the wasteful 22 QC practices, replacng them instead with aporopriale contol line and numbers of control measurements ‘An OPSpecs chart provides a graphic description of the imprecision and inaccuracy that are allowable and the contol rules and number of control measurements that are necessary for & OC procedure to achieve an appropriie| level of analytical qualty assurance for a defined quality requirement.’ The diagonal inae in this chart present the error detection performance of actual QC procedures (conirol rules and numbers of measurements). These lines are arranged from topto botom according to their error detection capability; the highest line providesthe Iighes' err detection thus there is more room” beneath thal ine forthe mehod te hit). Other detais abou the OC Frocedure are noted inthe key on the right side ofthe hart, such as the iloe rejection (Pr) and numberof control measurements (N)and the number cf runs(R). The imprecision end inaccuracy ofa method are used as the xcoordineieand y coordinate, respectively Ithis “aperaing pein los bel one of th ines ofthe OPSpcos Chart, at indicates thatthe OC procedure represented by that line wil provide the eppropriate performance (Figize 7) prising the arrow and target model one final ime, the (OPSpecs chart can be viewed inthe same way asthe ‘Sigma metic analysis and the Method Decision chart The OPSpees chat is ke the upper right quacrant ofthe target, wih the orgin asthe bull's eye. Method perfor. mance (the erow) should be ac clote ac possible to be bull's eye. Ths time, however the diferent rings onthe target represent deren’ GC procedures for use inthe laboratory. The closer tothe bul’s eye, the more OC Frocedures available for qualiy maragerrent. CHOOSE TRANSFORMATIONCo) DIAGNOSTICS Abbott Senda een ae ke kena nent (OPSpees charts, as with Method Decision charts, are typically generated for specific qualty requirements, But OPSpecs charis can also be normalized so that multiple tests with diferent quality requirements can be displayed on the same chart (Figure 8). NORMALIZED OPSpecs Chart TE, 00.00% with 90% AQA(SE) 00: 8 100 200 300 40.0 50.0 Allowable Imprecision (s, Figure 8 Sample Normalasd OPSpecs chat Cees ‘Sigma metric analysis, Method Decision charts, and OPSpecs charts provide easy tools for laboratories to determine the performance of their current methods and QC design, and to compare competing instrurnents on the markets. Both {quantitative calculations and visual assessment can be made with this approach. These techniques give the laboratory «practical way to select the right method and then select the right QC for thet method. The results an optimized testing process that fulils the quality required for appropriate test interpretation. meas”) CHOOSE TRANSFORMATIONCo) DIAGNOSTICS Abbott 1 Wisi 0, Six Some OC Designs Coto, don, Maton WE West C2005 ins ofMciche. ris Huma: ain Sr Hah System Wishigton-Nio Aeadany Pas, 201 ‘Soebsinen 0 Bete. Kea Leigh £ Mon T Fiske ingbinby enhoaccs on fy indésrs wih sit sigma scale Ach Patol Mad B24 16519 -¢ Wisp 0 sc Ping fr ty, Meson tO, 27 ‘Sif C Abser\ Cae Gacie tab Harada dinens CW Milind Pei, Sion I Debi spectator Tl fc, inrasion and is died Blog Vision ita/hamewssoardcombidases Lim Acaseed canbe t2 24 (5 Weipa 00. A Mata Sautn Decision Cin (ELK cin) jlog hod prbrmane, lial stata Sci 196. ra 80 §, we. 27-2 -FWesgar600, Osteo! paris pas sce (DPSpas cater aacing fs pci, uray ae gwihyonta maths ti pacing sing poten contra ined Otani 98-425 120) WP (SRR OO ya CHOOSE TRANSFORMATION