Does Six Sigma Improve Performance?

S. THOMAS FOSTER JR. BRIGHAM YOUNG UNIVERSITY
© 2007, ASQ

This article presents the long-term financial and operational results coming from the implementation of Six Sigma. The results from Six Sigma programs were somewhat mixed. The author found a significant effect on free cash flow (FCF), earnings before interest, taxes, depreciation, and amortization (EBITDA), and asset turnover. Six Sigma did not seem to affect sales return on assets, return on investment, or firm growth. As a result, if firms want to improve cash, earnings, or productivity in using assets, Six Sigma might be of use. In 1998, companies with high cash flows and no quality management program (QMP) had lower FCF in 2002 than companies that had adopted Six Sigma. Companies with low cash flow and no QMP did better than companies using Six Sigma, suggesting that for cash-poor firms, Six Sigma may be a drain on resources. Also, these companies may not have the cash necessary to sustain effective Six Sigma results over four years. Among companies with low and medium asset turnover, Six Sigma led to higher asset turnover. It could be that companies with low asset turnover could benefit more from process improvement implicit in Six Sigma than firms with high asset turnover. Key words: financial and operational outcomes, quality management, Six Sigma

INTRODUCTION
In recent years, many firms have adopted Six Sigma in an effort to improve quality and reduce costs (Pyzdek 2003). Six Sigma has been attractive to business executives, as it is thought to overcome some of the pitfalls of historical quality management implementations (Linderman et al. 2003). Quality management programs have been criticized for relying on improvement without mechanisms for ensuring that positive results will ensue (Howard, Foster, and Shannon 2005). That is, if employees are trained and empowered in quality improvement approaches, quality improvements and benefits will “percolate to the top,” thereby improving company performance. On the other hand, the Six Sigma approach requires more direction and leadership from top management than traditional quality management. This is termed “leadership for Six Sigma” (Foster 2007; Treichler et al. 2002). Along with this higher degree of leadership is a more structured process for improving performance. One approach to Six Sigma is the five-phase define, measure, analyze, improve, and control (DMAIC) process. Included with DMAIC is a method for leaders to prioritize potential improvement projects based on the probability that such projects will result in financial benefit to the organization. Similarly, another defining aspect of Six Sigma is its greater emphasis on cost reduction through quality improvement. This aspect includes a target of at least $200,000 in cost reduction for each Six Sigma project taken on with two to three projects per Black Belt over a one- to two-year period (Bisgaard and DeMast 2006). Such returns are the results expected from the high cost of investing in Black Belt employees.

www.asq.org 7

Does Six Sigma Improve Performance?
The convergence of these dimensions of leadership, structured process for improvement, and focus on financial results present in Six Sigma was designed to address the perceived weaknesses of traditional quality management. It is expected that this increased focus on financial outcomes should result in improvements in financial and operational results. The purpose of this research is to determine if this is indeed the case. The most common process for Six Sigma is DMAIC. This is similar to the plan, do, check, act (PDCA) cycle proposed by Walter Shewhart and W. Edwards Deming (Deming 1986). Since results from Six Sigma are closely related to the DMAIC process, the author discusses it in detail. It should be noted that as recently as the 1990s, many companies used DMAIC to guide Six Sigma projects. In the define phase, projects are identified and selected. Project selection is performed under the direction and with the participation of a Six Sigma champion. Also involved are Master Black Belts and Black Belts or Green Belts. Project selection is performed in four steps: 1) developing the business case; 2) project evaluation; 3) Pareto analysis; and 4) project definition. Business case development involves identifying a group of possible projects, writing the business case, and stratifying the business case into problem and objective statements. Project evaluation often involves risk and return assessment. The key individuals in performing this analysis are project champions. Project champions are usually top management executives who have legitimate and financial authority to support Six Sigma projects (Treichler et al. 2002). The process for defining Six Sigma projects helps in prioritizing which projects will provide the greatest financial and operational returns. The measure phase involves two major steps: 1) selecting process outcomes; and 2) verifying measurements. To select process outcomes, process mapping is used to help understand and define the process itself. A process map is a flowchart showing responsibilities (Gourishankar 2003). The goal of a process map is to identify nonvalue-added activities. Two important measures that are monitored are defects per unit (DPU) and defects per million opportunities (DPMO). Measurement systems analysis (MSA) is used to determine if measurements are consistent (Conklin 2006). The analyze phase involves gathering and analyzing data relative to a particular Black Belt project (Pyzdek 2003). The analyze-phase steps are as follows: 1) define the performance objectives; 2) identify independent variables (Xs); and 3) analyze sources of variability. Defining objectives involves determining

LITERATURE REVIEW AND HYPOTHESIS DEVELOPMENT
Several studies have examined the impact of quality practices on financial and operational results (Kaynak 2003). Most of these studies compare the results of firms that adopt quality practices versus control groups to determine if quality practices significantly influence financial and operational results. The purpose of this literature review is threefold. The author defines Six Sigma. Then he identifies similar studies that have examined the financial and operational impacts of different quality improvement approaches such as total quality management (TQM). Finally, he uses this review to provide a basis for selecting variables to be studied in this research.

Six Sigma
The author’s literature review revealed only one article relating to Six Sigma in an A-level journal (Linderman et al. 2003). Linderman et al. studied Six Sigma from a goal-theoretic perspective. They examined the relationship between goals and Six Sigma success and developed a series of propositions suggesting that high, yet attainable, goals were important to the success of Six Sigma programs. They also examined the intervening effects of effort, persistence, and direction resulting from explicit Six Sigma goals. Given the wide adoption of Six Sigma in many organizations around the world, there is a need for more research elucidating the benefits and costs of Six Sigma implementation. Currently, no studies investigate the relationship between Six Sigma implementation and financial and operational results.

8 QMJ VOL. 14, NO. 4/© 2007, ASQ

Does Six Sigma Improve Performance?
what characteristics of the process need to be changed to achieve improvement. Next, independent variables are identified for gathering data. These are variables that significantly contribute to process or product variation. The goal of analyzing sources of variability is to use visual and statistical tools to better understand the relationships between dependent (X) and independent (Y) variables for use in future experimentation. A variety of tools are used in this analysis, including histograms, box plots, scatter plots, regression analysis, and hypothesis tests. The improve phase of the DMAIC process involves off-line experimentation (Antony and Esamilla 2003; Montgomery 2004). Off-line experimentation involves studying the identified variables and using design of experiments (DOE) to determine whether the independent variables significantly affect variation in dependent variables. The control phase involves putting into place process checks to ensure that improvements are long lasting. The DMAIC process provides a process for improving operational results that is cost-reduction oriented. This results in the removal of waste from processes. These improvements are expected to result in operational and financial improvements. In this research, the author examines whether this is the case. (1998) reviewed the financial results of 108 firms that started TQM programs between 1981 and 1991. They found that these firms’ stocks outperformed a control group by finding excess cumulative returns. Hendricks and Singhal (1996; 1997; 2001a; 2001b) found similar results by analyzing the market returns of quality award winners. Hendricks and Singhal used winners of various awards to determine whether implementing effective TQM resulted in improved operating performance in firms. They used the winning of a quality award as a proxy for effective implementation of TQM. In their research, they found strong evidence that winners of quality awards outperformed control firms on operating-income-based measures. Subsequent to their studies, other researchers have continued to study receipt of awards such as the Malcolm Baldrige National Quality Award as indicative of mature quality implementation (Rajan and Tamimi 1999; Wilson and Collier 2000; Dean and Tomovic 2004). The literature, however, is not unanimous. York and Miree (2004) studied the links between TQM and financial performance. Conducting a comparison of the financial performance of quality award winners against their control firms by SIC groups, they studied performance both before and after the winning of the Baldrige Award. York and Miree found that TQM firms had better financial performance before and after they won the awards—some for even 20 years prior to winning the awards. The author suggested that winning the award was a covariate for financial success. Adams, McQueen, and Seawright (1999) studied the stock performance of Baldrige Award winners from the day their award was announced. They found only limited evidence to support the hypothesis that stockholders are rewarded with abnormal returns on the day of the quality award announcement. They suggested that stock analysts may have been forewarned that the company was winning or that they were at least aware of prior TQM efforts. They also suggested that there may be little impact on stock price, as stock impact is not the purpose of the Baldrige Award. They do propose that stock impacts relating to announcements are not as important as improvements relating to quality improvement efforts. While many studies have focused on the

Financial Results
Previous studies on the financial and operational impacts of quality efforts have provided mixed results. The Jacobsen and Aaker (1987) study using the Profit Impact of Market Strategy (PIMS) database examined the relationship between quality and market share, return on investment (ROI), relative price, relative cost, and relative quality. Jacobsen and Aaker (1987) found the impacts of quality improvement practices to be positive and significant. Depending on how businesses were grouped, however, they found variance in results that might be reflective of business strategies. Phillips, Chang, and Buzzell (1983) found similar relationships and added that product quality appeared to be negatively related to cost. Several researchers have investigated the impact of quality practices on stock prices. Easton and Jarrell

www.asq.org 9

Does Six Sigma Improve Performance?
financial and operational results of TQM and the Baldrige Award (Pannirselvam, Siferd, and Ruch 1998), the author’s literature review revealed no such studies relative to Six Sigma. It is expected that Six Sigma should result in improved financial performance. Since Six Sigma emphasizes reductions in cost and increases in measures such as rolled-throughput, one should find that cash flows, earnings, and other financial measures will show improvement. General Electric (GE) has reported more than $12 billion in savings due to Six Sigma. Similarly, Motorola reported $15 billion in savings over 11 years from Six Sigma implementation. As processes and products are improved, it is also expected that sales will improve. This research was performed to see if the expected financial results occurred for companies that implemented Six Sigma overall. As the author has discussed, when compared with traditional quality management, Six Sigma is much more cost-reduction and financial-results oriented. A comprehensive review of the quality management literature reveals that financial and operational performance could generally be categorized as measuring profitability, cost, efficiency, and growth (Phillips, Chang, and Buzzell 1983; Jacobsen and Aaker 1987; Ward, Leong, and Boyer 1994; Flynn, Schroeder, and Sakakibara 1995; Mohrman et al. 1995; Hendricks and Singhal 1996; 1997; Easton and Jarrell 1998; Samson and Terziovski 1999; Curkovic, Vickery, and Droge 2000; Lapre, Mukherjee, and VanWassenhove 2000; Wilson and Collier 2000; Cua, McKone, and Schroeder 2001; Devaraj, Matta, and Conlon 2001; Douglas and Judge 2001; Fynes and Voss 2001; Hendricks and Singhal 2001a; 2001b; Park, Hartley, and Wilson 2001; Sousa and Voss 2001; Eriksson and Hansson 2003; Fullerton, McWatters, and Fawson 2003; Kaynak 2003). For this study, the author used a modified version of Hendricks and Singhal’s (1997) performance measures. Financial measures used in this study include free cash flow per share; cost per dollar share; earnings before interest, taxes, depreciation, and amortization (EBITDA); sales; and sales per employee. The author starts with the following proposition: Proposition 1: Six Sigma adoption will positively affect financial results. Next, this proposition is translated into five hypotheses for this study. The author posits that investment in Six Sigma training and implementation signals a focus on aggressive cost reduction coupled with process and organizational improvement. Given that costs are reduced, cash is freed up for other uses. This results in hypotheses H1a and H1b: • H1a: Six Sigma adoption is positively associated with free cash flow per share. • H1b: Six Sigma adoption is positively associated with cost per dollar sales. As a result of this signaling of a focus on process and organizational improvement, operating margins should improve. This argument is similar to Deming’s value chain—that focusing on quality will result in lower costs and improved performance. Since operating margin can be expressed as EBITDA/sales, the author proposes the following three hypotheses: • H1c: Six Sigma adoption is positively associated with EBITDA. • H1d: Six Sigma adoption is positively associated with sales. • H1e: Six Sigma adoption is positively associated with sales per employee.

Operational Results
In addition to looking at financial results, many studies have examined the relationship between quality implementation and operational results. Again, the author reviews these studies to understand impacts found by traditional quality improvement approaches. Dow, Samson, and Ford (1999) studied the effect of quality practices on quality outcomes. They categorized quality practices into nine dimensions. Not all of their dimensions, however, contributed to superior quality performance. Employee commitment, shared vision, and customer focus yielded positive correlations with quality outcomes. Conversely, other hard quality practices, such as benchmarking, work teams, advanced manufacturing technologies, and close supplier relations, were not related to superior quality operational results.

10 QMJ VOL. 14, NO. 4/© 2007, ASQ

Does Six Sigma Improve Performance?
Focusing on the competitive aspects of quality performance, Douglas and Judge (2001) found strong empirical support for a positive relationship between the degree of TQM implementation and organizational performance. They also found some empirical evidence that this relationship between TQM implementation and organizational performance was moderated by organizational structure. Das et al. (2000) used structural equation modeling to study the impacts of international competition on quality. They found that competitive intensity provided an explanation for the absence of returns from investments in quality capital. Obtaining customer satisfaction performance from quality practices was shown to be contingent on the degree of international competition present in the business environment. Curkovic, Vickery, and Droge (2000) studied the direct effects of 10 quality action programs on six firm performance outcomes as well as their indirect effects through eight quality performance dimensions. Two paths from action programs through quality performance to firm performance in the automotive supply industry were identified. The first path was product quality, whose hallmarks were superior performance on conformance and design quality. The second path involved relationship quality. This path included superior customer responsiveness and service. Both the product-quality and the relationship-quality paths led to superior operational ROI. Product quality led to enhanced return on assets (ROA), and relationship quality led to enhanced market share performance. Flynn, Schroeder, and Sakakibara (1995) constructed a framework that focuses on both core quality management practices and the infrastructure used to engender an environment supportive of their use. They incorporated two measures of quality performance and their roles in establishing and sustaining competitive advantage. The author used path analysis to test a proposed model that explained that perceived quality market outcomes were primarily related to statistical control/feedback and the product design process, while the internal measure of percent that passed final inspection without requiring rework was strongly related to process flow management and statistical control/feedback. The focus of most Six Sigma efforts is on operational improvements. By focusing on the processes and variables associated with operational outcomes (for example, y=f(x)), it is expected that Six Sigma programs will result in more efficient uses of assets and higher ROI. The general proposition is stated as: Proposition 2: Six Sigma adoption results in better operational results. Using COMPUSTAT, the author computed several operational measures using productivity ratios. Part of the process of Six Sigma is to improve the use of assets. During the improve phase of the DMAIC cycle, DOE involves tolerance design, systems design, and parameter design (Foster 2006). Systems design involves choices, tradeoffs, and improvements to existing assets. These improvements may include the optimization of plant, equipment, or technology. Implicit in this improved usage is a more productive usage of assets. This results in hypotheses H2a through H2d. • H2a: Six Sigma adoption is positively associated with asset turnover. • H2b: Six Sigma adoption is positively associated with return on assets. • H2c: Six Sigma adoption is positively associated with return on investment. • H2d: Six Sigma adoption is positively associated with total assets. Improvement efforts such as Six Sigma could help a company to grow as profitability improves. Often these efforts result in downsizing and the more productive use of employees. Therefore, Six Sigma could result in either growth in the number of employees or a reduction in the number of employees. The literature is not clear on this issue. Therefore, hypothesis H2e is stated in null form. • H2e: Six Sigma adoption is not related to number of employees.

METHODS
This study examines the long-term financial and operational impacts of implementing Six Sigma. In order to create a population of Six Sigma-adopting firms, the author performed a keyword Lexis/Nexis (LexisONE)

www.asq.org 11

Does Six Sigma Improve Performance?
online search of corporate financial reports for the terms Six Sigma, ISO 9000, TQM, and Baldrige through the LexisONE subscription database. Since annual reports are used to transmit information to shareholders, the mention of a quality management initiative such as Six Sigma is interpreted as signaling an organizational commitment to that effort. Since annual reports are used to elucidate strategic emphases, it can be said that these quality improvement efforts are strategic in nature. Using this rationale, the author performed a content analysis to determine which firms had announced the implementation of Six Sigma. Initially, the year 1998 was searched. Since only 12 Six Sigma firms were returned, the author also searched 1996 and 1997. This content analysis of annual reports resulted in a list of 30 companies that had adopted Six Sigma as a means of improving performance. In a further step, they assembled a panel of six judges, including academics and quality professionals, to perform a content analysis of the annual reports found in LexisONE to determine which firms showed strategic commitment to Six Sigma. Based on this content analysis, six firms were removed from the analysis, resulting in a final set of 24 Six Sigma firms. They also found 24 Baldrige firms, 26 TQM firms, and 23 ISO 9000 firms. In addition, a randomly selected list of 50 companies from the 1998 Fortune 500 list was used as a control group. Company ticker symbols were obtained using Hoover’s Online, and the database of firms the author created was searched for duplicates and firms that were returned in more than one search. If a firm in the control group was also returned in the list of Six Sigma companies, it was deleted from the control group and another firm in the Fortune 500 was randomly selected. In addition, firms no longer in business or that had been acquired since 1998 were also stricken. In the end, the database of control firms was reduced to 41 companies. As stated, a modified version of Hendricks and Singhal’s (1997) performance measures was used. Profitability measures used were earnings (EBITDA), ROA, operating margin (EBITDA/sales), and net sales. Cost structure was analyzed by calculating cost per dollar of sales and FCF per share. Relative firm growth measures were obtained using total assets and number of employees. Firm data were collected using Standard and Poor’s COMPUSTAT database. Data were collected for the firms drawn from the Lexis/Nexis database for fiscal years 1998 through 2004. This provided data for the author’s longitudinal analysis. In this article, the author compares results from the time Six Sigma was mentioned in the annual report to four years later. This four-year interval is needed, as it takes time for quality improvement programs to yield significant results. In a study of best quality practices, Ozan (1992) found that quality improvement programs should be implemented gradually. A study by the United States General Accounting Office (1991) stated that, on average, 3.5 years were required to see significant results for TQM programs. In a study of the U.S. auto industry, Narasimhan, Ghosh, and Mendez (1993) found a 2.26-year lag between quality improvement efforts and sales improvements. Foster (1996) studied the speed of quality improvement in five different production facilities and found that plants that improved more slowly actually had better financial results than plants that attempted to improve rapidly. Given these prior research results, it was thought that a four-year interval would provide necessary time to see significant improvement in results if it was present.

RESULTS
The results are organized according to the outcome variables studied. For each dependent variable the author discusses data transformation (when needed). Next, analysis of covariance (ANCOVA) results for main effects are presented. ANCOVA was used to test where significant differences exist between outcomes for each of the treatments. ANCOVA is a general linear model with one explanatory variable and multiple factors. It was applied where a potentially strong correlation exists between independent and dependent variables. Then, the author explains interactions between various quality programs (Six Sigma, Baldrige, TQM, and ISO 9000) and no quality program relative to each financial or performance outcome. The interaction tests used ANCOVA to determine if there are significant interaction effects between quality management programs (QMP) and outcomes. Model-based estimates are presented for

12 QMJ VOL. 14, NO. 4/© 2007, ASQ

Does Six Sigma Improve Performance?
Table 1 Summary statistics for FCF in 1998 and 2002.
1998
Number of companies Minimum 25th percentile Median 75th percentile Maximum Mean Standard deviation 138 -14.05 -0.54 0.25 0.85 17.58 0.19 2.82

Table 2 ANCOVA results main effects.
Term
Free cash flow Log cost per dollar sales EBITDA Log of sales Sales per employee Asset turnover
© 2007, ASQ

F value
.94 36.07 9.57 579.24 306.97 3.01 1.12 1.81 1.66 860.94

Pr>F
0.3300 <.0001 0.0027 <.0001 <.0001 0.0800 0.2910 0.1806 0.2006 <.0001
© 2007, ASQ © 2007, ASQ

2002
138 -22.18 0.33 1.05 2.38 12.40 1.19 3.63

ROA ROI Total assets Log number of employees

companies in the 10th percentile, median, and 90th percentile. The percentiles were determined according to firm size, as prior research has suggested that firm size has a moderating effect on QMP effects (Hendricks and Singhal 2001a; York and Miree 2004). Finally, hypothesis tests were performed to study the author’s research propositions. The results are presented according to financial or performance outcome in order to enhance clarity of presentation.

Table 3 ANCOVA results interaction terms.
Term
Free cash flow by QMP Log cost per dollar sales by QMP EBITDA by QMP Log sales by QMP

F Value
6.08 16.99 8.56 .83 0.46 14.94 2.45 2.42 17.4 1.14

Pr>f
0.0002 <.0001 <.0001 0.5101 0.7600 <.0001 0.0490 0.0517 <.0001 0.3400

Free Cash Flow Per Share
Summary statistics for free cash flow per share (FCF) in 1998 and 2002 are presented in Table 1. This summary shows that 138 companies were evaluated for this research. These companies are rank ordered, with the 34th company in the 25th percentile, company 69 at the median, and the 104th company at the 75th percentile. These statistical tests were performed using SAS. The model for FCF in 2002 was fit with all data present and no transformations. Table 2 shows the ANCOVA results for FCF (as well as the other measures). The author has combined tables here for efficient presentation. Table 2 shows that the main effect for FCF was not significant. However, as is shown in Table 3, the interaction term for FCF by QMP was significant (p < 0.0002). With a low R2 of 0.22, this suggests that other variables are having an effect on an otherwise null model. The first five rows in Table 4 present

Sales per employee by QMP Asset turnover by QMP ROA by QMP ROI by QMP Total assets by QMP Log employees by QMP

model-based estimates of FCF in 2002 at key levels of FCF in 1998. For reference, Table 5 shows the counts of companies in various percentiles of FCF in 1998 to provide an indication of the range of data behind the estimates shown in previous tables and figures. Companies without a QMP appear to have a different relationship between FCF in 2002 and 1998 than all other companies. While FCF in 2002 appears to be more or less positively related to FCF in 1998, these

www.asq.org 13

Does Six Sigma Improve Performance?
companies have a negative relationship between FCF in 2002 and 1998. Figure 1 plots this interaction. The author evaluates H1a in Table 6. Among companies with high FCF in 1998, those with no QMP have an estimated FCF for 2002 that is substantially lower than all other companies, whereas companies using Six Sigma have an estimated 2002 FCF that is substantially higher than all other companies. Thus, they see Six Sigma companies and the average of all companies with a QMP as having significantly higher FCF in 2002 than those companies without a QMP. At low FCF in 1998, however, those companies without a QMP had significantly higher FCF in 2002, leading to the patterns of significance displayed in Table 6.
Table 4 Model-based mean estimates (std. error of estimates) of effects in year 5, based on levels of year 1.
Free Cash Flow
Six Sigma Baldrige TQM ISO No QMP

10th
1.59 (0.75) 1.27 (0.76) -0.29 (1.35) 0.79 (1.03) 2.59 (0.67)

Median
1.64 (0.68) 1.94 (0.73) 1.61 (0.69) 1.21 (0.77) 1.13 (0.55)

90th
1.7 (0.83) 2.65 (1.04) 3.61 (1.38) 1.65 (1.43) -0.4 (0.56)

Cost Per Dollar Sales
Six Sigma Baldrige TQM ISO No QMP -0.18 (0.06) -0.34 (0.04) -0.1 (0.02) -0.16 (0.03) -0.35 (0.03) -0.13 (0.02) -0.13 (0.02) -0.1 (0.02) -0.07 (0.02) -0.16 (0.02) -0.11 (0.04) -0.02 (0.03) -0.098 (0.02) -0.02 (0.03) -0.07 (0.02)

EBITDA
Six Sigma Baldrige TQM ISO No QMP 2.76 (3.56) 15.11 (4.96) 2.63 (4.8) 10.98 (4.27) 5.69 (3.17) 11.88 (2.97) 8.73 (3.49) 7.58 (3.07) 13.68 (3.29) 13.8 (2.19) 23.99 (3.14) 0.263 (3.79) 14.15 (7.04) 17.26 (3.79) 24.57 (3.22)

Cost Per Dollar Sales
It was evident from the review of the 1998 data that cost per dollar sales value was extremely skewed, and to better accommodate model fitting, the author fitted the model to the natural log of cost per dollar sales. The 2002 values, however, were evenly distributed. Table 2 shows the ANCOVA results for the model fit to logged cost per dollar sales. As shown in Table 3, because of the significant interaction between 1998 cost per dollar sales and QMP, the effect of QMP is different depending on the level of cost per dollar sales in 1998. Table 4 gives the model-based estimates for mean (logged) cost per dollar sales in 2002 by QMP and level of 1998 logged cost per dollar sales. As can be seen in the plots of means in Figure 1, the interaction appears to result from different characteristics with respect to cost per

Asset Turnover
Six Sigma Baldrige TQM ISO No QMP 15,272 (2,550) -2,352 (3,392) 11,426 (5,604) 9,435 (8,878) 6,266 (1,957) 10,116 (2,083) -627 (2,685) 4,765 (2,519) 4,937 (2,614) 6,011 (1,579) 812 (2,725) 2,487 (2,826) -7,255 (6,004) -3,182 (12,084) 5,551 (2,252)

ROA
Six Sigma Baldrige TQM ISO No QMP 2.18 (2.25) -1.17 (2.77) 3.74 (3.76) -1.52 (1.96) 1.98 (1.71) 2.28 (1.75) -1.12 (1.77) 3.08 (1.84) 2.65 (1.88) 2.87 (1.29) 2.37 (2.34) -1.13 (2.42) 2.55 (2.54) 5.96 (2.08) 3.58 (1.8)

ROI
Six Sigma Baldrige TQM ISO None 3.91 (3.8) -6.05 (4.76) 7.26 (7.96) -2.08 (3.77) 8.2 (3.3) 4.25 (2.98) -0.82 (3.02) 6.88 (3.06) 6.81 (3.45) 6.46 (2.19) 4.47 (3.67) 2.63 (3.68) 6.63 (5.32) 12.67 (4.26) 5.3 (2.93)

Total Assets
Six Sigma Baldrige TQM ISO None -705.3 (1659.4) 7682.9 (1937.1) -28.3 (3916.3) 1502.8 (1846.6) 4135.2 (1193.8) 1894 (1 j610.9) 7523.7 (1804.5) 1784.6 (1638.3) 1950.3 (1718.7) 4545.1 (1167.6) 6140.6 (1562.7) 7263.5 (1741.2) 4746.2 (6746.5) 2681.4 (2220.2) 5214.9 (1396.8)

14 QMJ VOL. 14, NO. 4/© 2007, ASQ

© 2007, ASQ

Does Six Sigma Improve Performance?
dollar sales among compaTable 5 Count of companies in each percentile category and QMP. nies using TQM and those Six Sigma Baldrige TQM ISO None Total using either no QMP or one of the other three under 0–10th percentile 3 6 1 4 14 study. All companies other 10th–50th percentile 7 10 14 16 7 54 than those with TQM appear 50th–90th percentile 12 5 11 6 22 56 to have a direct relationship between cost per dollar sales 90th percentile and higher 2 3 1 8 14 in 2002 and 1998, while Total 24 24 26 23 41 138 TQM companies appear to have no such relationship. Evaluation of the hypotheses of interest is given in was fairly symmetric, but it was peaked. Therefore, the model for EBITDA in 2002 was fit with all data present Table 6. Null hypotheses regarding Six Sigma compaand no transformations. nies cannot be rejected. Among companies with low cost Table 2 shows significant ANCOVA results (p < .0027). per dollar sales in 1998, companies using the Baldrige Table 3 shows the model-based estimates for EBITDA in Award criteria have a significantly lower predicted cost 2002. Because of the significant interaction between per dollar sales than companies using any of the other EBITDA 1998 and QMP, the effect of QMP is different three QMPs. Therefore, H1b is not supported. depending on the level of EBITDA in 1998. Table 4 shows model-based estimates of average EBITDA in 2002 at levels of 1998 EBITDA. Inspection of Figure 1 suggests the following as an A review of the data showed that the distribution of interpretation of the interaction: Among companies with EBITDA in 1998 was skewed because of the presence of the highest EBITDA in 1998, those using the Baldrige outlying companies. In 2002 the distribution of EBITDA

EBITDA

Figure 1 Interaction Plots by Effect.
FCF 1998 by QMP interaction 4 3.5 3 2.5 FCF 2002 2 1.5 1 0.5 -2 -1 0 -0.5 -1 FCF 1998 0 1 2 3 Six Sigma Baldrige TQM ISO None 2002 Return on Assets Return on Assets by QMP interaction 7 6 5 4 3 2 1 -4 -2 0 -1 -2 1998 Return on Assets 0 2 4 6 8 10 Six Sigma Baldrige TQM ISO None

Asset Turnover 20,000 15,000 2002 Asset Turnover 10,000 5,000 Six Sigma Baldrige TQM ISO -3000 -2000 -1000 (5,000) (10,000) 1998 Asset Turnover level 0 1000 2000 3000 4000 5000 6000 None -0.35 Logged cost/dollar sales, 2002

Logged Cost per dollar sales by QMP interaction -0.3 -0.25 -0.2 -0.15 -0.1 0 -0.05 0 -0.05 -0.1 ROI, 2002 -0.15 -0.2 -0.25 -0.3 -0.35 -0.4 Logged Cost/dollar sales, 1998 -10 Six Sigma Baldrige TQM ISO None -5 5 0 -4.1 15 10

ROI 1998 by QMP interaction

Six Sigma Baldrige TQM 8.4 16.6 ISO None

ROI, 1998

Mean EBITDA in 2002, by QMP and EBITDA in 1998 30.00 25.00 20.00 15.00 10.00 5.00 0.00 Total assets 2002 Six Sigma Baldrige TQM ISO None 9000 8000 7000 6000 5000 4000 3000 2000 1000 0 -1000 0 -1000 0 5 10 15 EBITDA 1998 20 25 30 -2000

Total assets by QMP interaction

EBITDA 2002

Six Sigma Baldrige TQM ISO

1000

2000

3000

4000

5000

6000

7000

Total Assets 1998

www.asq.org 15

© 2007, ASQ

None

© 2007, ASQ

Does Six Sigma Improve Performance?
Award criteria had the lowest EBITDA in 2002, and those using Six Sigma or no QMP had the highest. That result is somewhat reversed among companies with the lowest EBITDA in 1998. Evaluation of the hypotheses of interest is given in Table 6. The EBITDA in 2002 is statistically equivalent between Six Sigma companies and companies with no QMP, regardless of EBITDA in 1998. Among companies with a QMP, Six Sigma companies have a higher EBITDA in 2002. However, this only holds among companies with a higher EBITDA in 1998; for all others, performance of EBITDA in 2002 is equivalent. This provides partial support for H1c.
Table 6 Hypothesis results.
Free cash flow
Six Sigma vs. No QMP Six Sigma vs. Other QMP Any QMP vs. No QMP

10th1
-1 (0.99) 1 (0.99) 1.75 (0.85)*

Median2
0.5 (0.87) 0.05 (0.81) -0.46 (0.67)

90th3
2.1 (1.01)* -0.94 (1.12)* -2.8 (0.82)*

Cost per dollar sales
Six Sigma vs. No QMP Six Sigma vs. Other QMP Any QMP vs. No QMP 0.17 (0.08) 0.01 (0.07)
-0.16 (0.04)*

0.03 (0.03) -0.03 (0.03)
-0.06 (0.02)*

-0.04 (0.05) -0.06 (0.05) -0.01 (0.03)

EBITDA
Six Sigma vs. No QMP Six Sigma vs. Other QMP Any QMP vs. No QMP -2.93 (4.71) -6.81 (4.44) -2.17 (3.84) -1.92 (3.66) 1.88 (3.52) 3.33 (2.76) -0.57 (4.54)
13.43 (4.3)* 10.65 (4.13)*

Asset turnover
Six Sigma vs. No QMP Six Sigma vs. Other QMP Any QMP vs. No QMP 9005.74 (2813.7)* 9101.92 (4355.24)* -2179.08 (3458.55) 4105.52 (2592.55) 7091.12 (2571.04)* 1213.03 (2110.05) -4738.12 (2696.38) 3462.13 (5065.76) 7334.95 (3709.69)

ROA
Six Sigma vs. No QMP Six Sigma vs. Other QMP 0.20 (2.82) 1.81 (2.85) 1.16 (2.22) -0.58 (2.17) 0.76 (2.06) 1.15 (1.63) -1.21 (2.97) -.09 (2.70) 1.15 (2.21)

Sales

Any QMP vs. No QMP

Despite a large range of Total assets sales data, the raw data Six Sigma vs. No QMP were left in their original Six Sigma vs. Other QMP scales. The author examined the summary of sales Any QMP vs. No QMP data and found they were * p < .0001 extremely skewed. To better accommodate model fitting, he modeled the natural log of sales. Table 2 gives the ANCOVA table for the model for logged sales. While the overall R2 for the model is 0.93, no terms other than sales in 1998 have any explanatory power for sales in 2002 (see Table 3). For this reason, the author concludes that the null hypotheses of interest cannot be rejected and that Six Sigma has no influence on sales. Therefore, H1d is not supported.

-4,840.5 (2,029.3) -3,757.9 (2,293.2) 2,022.1 (1,754)

-2,651.1 (1,975.6) -1,858.9 (1,894.2) 1,257 (1,476.2)

925.7 (2,087.3) 1,243.5 (2,895.8) 7 (2,362.3)
© 2007, ASQ

Sales Per Employee
The raw data were divided by 1,000 to drop the scale to a more useable level. It was evident from the summary data that this value was extremely skewed, and to better accommodate model fitting, the author used the natural log of sales per employee (in thousands). Table 2 gives the ANCOVA table for the model fit. While the overall

16 QMJ VOL. 14, NO. 4/© 2007, ASQ

Does Six Sigma Improve Performance?
R2 for the model is a high 0.77, no terms other than sales per employee in 1998 have any explanatory power for sales per employee in 2002 (see Table 3). For this reason, they concluded that the null hypotheses H1e cannot be rejected and that QMP has no influence on sales per employee. means and standard errors of the return on asset data used in the model by QMP, company size, and year. Therefore, H2b is not supported.

Return on Investment
ROI for either year looks bell-like in distribution except for the presence of very long tails that show outlying observations in both the positive and negative direction. Attempts at transformation did little to pull in these strong outliers. Thus, the ANCOVA model was fit on the raw data. Table 2 shows the ANCOVA table. As one can see in Table 3, at p < .05, the interaction of QMP by ROI in 1998 is technically not statistically significant. However, it is of borderline significance. Table 5 gives the model-based estimates for ROI in 2002 by percentiles of ROI in 1998 and QMP. Figure 1 plots these means for easier interpretation. From Figure 1 it appears that Baldrige and ISO 9000 companies may be responding differently than the companies with the other three QMPs. This is descriptive only, however, owing to the lack of significance of any of the terms in the ANCOVA model. Therefore, H2c is not supported.

Asset Turnover
Once again, Table 2 shows the ANCOVA table for the model fit, and Table 4 shows the model-based estimates for asset turnover in 2002. These estimates indicate that companies with high asset turnover in 1998 had a lower 2002 asset turnover than companies with a low asset turnover in 1998. This is generally true for companies using Six Sigma, TQM, and ISO 9000. The 2002 asset turnover for Baldrige companies shows the opposite pattern (companies with high asset turnover in 1998 had higher asset turnover in 2002 than those with lower 1998 asset turnover). Those companies with no QMP showed no real difference in 2002 asset turnover, regardless of their 1998 asset turnover. Figure 1 plots these estimates from Table 4. Evaluation of the hypotheses of interest is given in Table 6. Among companies with low and median 1998 asset turnover, companies using Six Sigma QMP had significantly higher asset turnover in 2002 than companies with no QMP; the asset turnover of these Six Sigma companies was also higher than the average of companies with the other three programs. Among companies with high asset turnover in 1998, there was no significant difference among any of the hypotheses of interest. Therefore, H2a is partially supported.

Total Assets
To bring the data to a more manageable scale, the original data were divided by 1,000. The model was fit with all available data for total assets in 2002. The nonsignificant ANCOVA result is presented in Table 2. As is shown in Table 3, because of the significant interaction between total assets 1998 and QMP, the effect of QMP is different, depending on the level of total assets in 1998. Table 4 shows the model-based estimates of the average total assets (in thousands) for 2002 for each QMP by levels of total assets in 1998. Figure 1 shows a plot of the estimates to facilitate general interpretation of the interaction. Generally, it appears that the relationship between 1998 total assets and 2002 total assets does not hold for companies using Baldrige, ISO 9000, or no QMP. Table 6 contains the tests of the hypotheses of interest. Among companies at the highest level of total assets in 1998, there is no difference in 2002 total assets. For all others, however, those using the

Return on Assets
Table 2 gives the ANCOVA table for the model fit, and Table 3 shows the model-based estimates for ROA in 2002. Table 3 shows a significant interaction between ROA and sales (p < .05). Table 4 provides the modelbased estimates of average 2002 return on investments at levels of 1998 ROA. Evaluations of the hypotheses of interest are given in Table 6. While the interaction is significant, none of the hypotheses of interest are. Table 6 provides

www.asq.org 17

Does Six Sigma Improve Performance?
Baldrige Award criteria have higher total assets in 2002 than those using any other QMP. Therefore, H2d is not supported. Among companies with low and medium asset turnover, Six Sigma led to higher asset turnover. Therefore, H2a was supported for companies with low and medium asset turnover in 1998. It could be that these companies with low asset turnover could benefit more from process improvement implicit in Six Sigma than firms with high asset turnover. The adoption of Six Sigma seemed to not affect ROA and ROI significantly. This may not be surprising because many factors affect ROA and ROI, as evidenced by the DuPont model. Further investigation showed that large firms using Six Sigma performed better. That is, the improvement rates were in the correct direction. The differences, however, were not significant. Reflecting on the original research question regarding financial and operational outcomes, it appears that at a macro level, the effects, while in some cases significant, are somewhat modest. While the author did see benefits in both financial and operational areas, quality professionals should be careful to not oversell the benefits of Six Sigma or any other QMP. That Six Sigma did not affect firm growth was not surprising. There were many exogenous macroeconomic factors, including 9/11, that could have influenced growth during this period. Powell (1995) and others have stated that there are many factors that affect firm growth other than quality management approaches. Indeed, many quality programs have been associated with reductions in number of employees. On the other hand, that one is able to find any significant results for these small numbers of firms demonstrates that quality programs such as Six Sigma can potentially provide important financial and operational returns. This research is very high level with much room for future study. Future studies should examine specific effects on specific projects in companies. It is expected that such studies could verify and clarify these findings.

Number of Employees
Because these data are skewed, the author used the natural logarithm as the outcome in modeling efforts. Table 2 gives the results of the ANCOVA model. For this model, adding the indicator for company size was not appropriate, so it was not included here. Because no term other than the number of employees in 1998 is statistically significant, H2e is not supported.

DISCUSSION AND CONCLUSIONS
As has been seen, the results from Six Sigma programs are at best mixed. The author found a significant effect on FCF, EBITDA, and asset turnovers; however, Six Sigma did not seem to affect sales, ROA, ROI, or firm growth. As a result, if firms want to improve cash, earnings, or productivity in using assets, Six Sigma might be of use. An intervening variable in this analysis was firm size, which confirmed Hendricks and Singhal’s finding on the importance of firm size. It appears that firm size mitigates the effects of programs like Six Sigma. Certainly, large firms such as GE and Motorola have resources and assets to invest in Six Sigma programs. These firms also showed great capacity for improvement. Smaller firms may not have the same resources to apply to such programs; however, smaller firms that invest in efforts such as Six Sigma may be impacted more significantly as a total proportion of the firm’s business results. Companies with high cash flows in 1998 and no QMP had lower FCF in 2002 than companies that had adopted Six Sigma. Companies with low cash flow and no QMP did better than companies using Six Sigma, suggesting that for cash-poor firms, Six Sigma may in fact be a drain on resources. Also, these companies may not have the cash necessary to sustain effective Six Sigma results over four years. Therefore, H1a was supported for firms with high cash flows, but may have a negative effect for companies with poor cash flows.

Limitations
There were several limitations to this research. As was discussed previously, financial results during this period were influenced by 9/11 and other events. In addition, these firms’ use of Six Sigma or other quality programs in 1998 does not guarantee that they implemented these

18 QMJ VOL. 14, NO. 4/© 2007, ASQ

Does Six Sigma Improve Performance?
programs effectively or maintained their strategic support through 2002 and beyond. Of course, the normal provisos concerning COMPUSTAT hold, as these data were firm-level data, and many times Six Sigma programs are implemented at the business unit level. While the methodology the author used is adapted from prior QMP studies, it is difficult to isolate the effects of Six Sigma at a macro level. Future studies at a more micro level may help the understanding of these phenomena. This study included firms from a variety of industries. For future research, the author suggests industryspecific studies of the effects of Six Sigma adoption. It has been suggested that Six Sigma is best suited for process industries. It is expected that in some ways such industries would have been leading-edge adopters.
Cua, K. O., K. E. McKone, and R. G. Schroeder. 2001. Relationships between implementation of TQM, JIT, and TPM and manufacturing performance. Journal of Operations Management 19: 675-694. Curkovic, S., S. Vickery, and C. Droge. 2000. An empirical analysis of the competitive dimensions of quality performance in the automotive supply industr y. International Journal of Operations & Production Management 20, no. 3: 386-403. Das, A., R. Handfield, R. Calantone, and S. Ghosh. 2000. A contingent view of quality management: The impact of international competition on quality. Decision Sciences Journal 31, no. 3: 649-690. Dean, M., and C. Tomovic. 2004. Does Baldrige make a business case for quality? Quality Progress 37, no. 4: 40-46. Deming, W. 1986. Out of the crisis. Cambridge, Mass.: MIT Press. Devaraj, S., K. Matta, and E. Conlon. 2001. Product and service quality: The antecedents of customer loyalty in the automotive industry. Production and Operations Management 10, no. 4: 424-439. Douglas, T. J., and W. Q. Judge Jr. 2001. Total quality management implementation and competitive advantage: The role of structural control and exploration. Academy of Management Journal 441: 158-169. Dow, D., D. Samson, and S. Ford. 1999. Exploding the myth: Do all quality management practices contribute to superior quality performance? Production and Operations Management 8, no. 1: 1-27. Easton, G. S., and S. L. Jarrell. 1998. The effects of total quality management on corporate performance: An empirical investigation. Journal of Business 71, no. 2: 253-307. Eriksson, H., and J. Hansson. 2003. The impact of TQM on financial performance. Measuring Business Excellence 7, no. 1: 36-50. Flynn, B. B., R. G. Schroeder, and S. Sakakibara. 1995. The impact of quality management practices on performance and competitive advantage. Decision Sciences Journal 26, no. 5: 659-684. Foster, S. T. Jr. 1996. Examining the impact of speed of quality improvement on quality-related costs. Decisions Sciences Journal 24, no. 4: 623-646. Foster, S. T. Jr. 2006. Managing quality: Integrating the supply chain. Upper Saddle River, N.J.: Prentice Hall. Foster, S. T. Jr. 2007. Quality survival guide: Leadership. Quality Progress 40, no. 7: 25-35. Fullerton, R. R., C. S. McWatters, and C. Fawson. 2003. An examination of the relationship between JIT and financial performance. Journal of Operations Management 21: 383-404. Fynes, B., and C. Voss. 2001. A path analytic model of quality practices, quality performance, and business performance. Production and Operations Management 10, no. 4: 494-513. Gourishankar, T. 2003. Back to basics: A simple process map. Quality Progress 39, no. 1: 112-120.

Implications for Managers
This research has implications for managers. The author found negative cash flows for small firms, but positive cash flows for larger firms. It is apparent that managers in smaller firms should carefully consider the cash flow impacts of quality improvement investments. Managers should also carefully monitor the effects of Six Sigma efforts and focus on “hard dollar” benefits. If not skillfully implemented, the benefits of Six Sigma may be marginal. The “fad” element of Six Sigma should also be noted. It is more likely that companies that understand the benefits of Six Sigma are able to effectively manage to achieve those benefits. It could be that companies that implement Six Sigma to merely parrot industry practice will likely find modest competitive benefits from Six Sigma or any other quality improvement effort.
REFERENCES

Adams, G., G. McQueen, and K. Seawright. 1999. Revisiting the stock price impact of quality award. Omega 27: 595-604. Antony, J., and J. Esamilla. 2003. Lean sigma. Manufacturing Engineering (April): 40-42. Bisgaard, S., and J. DeMaat. 2006. After Six Sigma, what’s next? Quality Progress 39, no. 1: 30-36. Conklin, R. 2006. Measurement system analysis for attribute measuring processes. Quality Progress 39, no. 3.

www.asq.org 19

Does Six Sigma Improve Performance?
Hendricks, K. B., and V. R. Singhal. 1996. Quality awards and the market value of the firm: An empirical investigation. Management Science 42, no. 3: 415-436. Hendricks, K. B., and V. R. Singhal. 1997. Does implementing an effective TQM program actually improve operating performance? Empirical evidence from firms that have won quality awards. Management Science 43, no. 9: 1258-1274. Hendricks, K. B., and V. R. Singhal. 2001a. Firm characteristics, total quality management, and financial performance. Journal of Operations Management 19: 269-285. Hendricks, K. B., and V. R. Singhal. 2001b. The long-run stock price performance of firms with effective TQM programs. Management Science 47, no. 3: 359-368. Howard, L., S. T. Foster, and P. Shannon. 2005. Team climate and teamwork in government: The power of embedded leadership. International Journal of Quality and Reliability Management 22, no. 8: 769-795. Jacobsen, R., and D. Aaker. 1987. The strategic role of product quality. Journal of Marketing 51: 31-44. Kaynak, H. 2003. The relationship between total quality management practices and their effects on firm performance. Journal of Operations Management 21: 405-435. Lapre, M., A. Mukherjee, and L. VanWassenhove. 2000. Behind the learning curve: Linking learning activities to waste reduction. Management Science 46, no. 5: 597-611. Linderman, K., R. Schroeder, S. Zaheer, and A. Choe. 2003. Six Sigma: A goal theoretic perspective. Journal of Operations Management 21, no. 2: 193-204. Mohrman, S. A., R. V. Tenkasi, E. E. Lawler III, and G. E. Ledford Jr. 1995. Total quality management: Practice and outcomes in the largest U.S. firms. Employee Relations 17, no. 3: 26-41. Montgomery, D. 2004. Design and analysis of experiments. Hoboken, N.J.: John Wiley and Sons. Narasimhan, R., S. Ghosh, and D. Mendez. 1993. A dynamic model of product quality and pricing decisions on sales response. Decision Sciences Journal 20, no. 4: 893-908. Ozan, T. 1992. International quality study: Best practices report. New York: Ernst and Young. Pannirselvam, G., S. Siferd, and W. Ruch. 1998. Validation of the Arizona governor’s quality award: A test of the Baldrige criteria. Journal of Operations Management 16, no. 6: 529-550. Park, S., J. Hartley, and D. Wilson. 2001. Quality management practices and their relationship to buyer’s supplier rating: A study in the Korean automotive industry. Journal of Operations Management 19, no. 3: 695-712. Phillips, L. W., D. R. Chang, and R. D. Buzzell. 1983. Product quality, cost position, and business performance: A test of some key hypothesis. Journal of Marketing 47 (Spring) 26-43. Powell, T. C. 1995. Total quality management as competitive advantage: A review and empirical study. Strategic Management Journal 16, no. 1: 15-37. Pyzdek, T. 2003. The Six Sigma handbook: The complete guide for greenbelts, blackbelts, and managers at all levels. New York: McGraw-Hill. Rajan, M., and N. Tamimi. 1999. Baldrige award winners: The payoff to quality. Journal of Investing 8, no. 4: 39-43. Samson, D., and M. Terziovski. 1999. The relationship between total quality management practices and operational performance. Journal of Operations Management 17: 393-409. Sousa, R., and C. Voss. 2001. Quality management: Universal or context dependent? Production and Operations Management 10, no. 4: 383-404. Treichler, D., R. Carmichael, A. Kusmanoff, J. Lewis, and G. Berthiez. 2002. Design for Six Sigma: 15 lessons learned. Quality Progress 35, no. 1: 33-43. United States General Accounting Office. 1991. Management practices: U.S. companies improve performance through quality efforts. Washington, D.C.: GAO Report to the Honorable Don Ritter, House of Representatives. Ward, P., K. Leong, and K. Boyer. 1994. Manufacturing proactiveness and performance. Decision Sciences Journal 25, no. 3: 337-355. Wilson, D. D., and D. C. Collier. 2000. An empirical investigation of the Malcolm Baldrige National Quality Award causal model. Decision Sciences Journal 31, no. 2: 361-383. York, K., and C. Miree. 2004. Causation and covariation: An empirical re-examination of the link between TQM and financial performance. Journal of Operations Management 22, no. 3: 291-311.
BIOGRAPHY S. Thomas Foster is area leader of global supply chain at Brigham Young University. He has published more than 50 qualityrelated ar ticles in journals such as Decision Sciences, International Journal of Production Research, the Journal of Operations Management, the International Journal of Quality and Reliability Management, the Quality Management Journal, and Quality Progress magazine. His book is titled, Quality Management: Integrating the Supply Chain (Prentice Hall). Foster is founder of http://www.freequality.org, was awarded the ASBSU 2000 Outstanding Faculty Award, and received the prestigious 2002 Decision Sciences Institute Instructional Innovation Award. He is currently serving as guest editor for a special issue on supply chain quality for the Journal of Operations Management. He served twice as an examiner for the Malcolm Baldrige National Quality Award and is a member of ASQ. He can be reached by e-mail at tom_foster@byu.edu.

20 QMJ VOL. 14, NO. 4/© 2007, ASQ

Sign up to vote on this title
UsefulNot useful