Journal of Applied Psychology 1990, Vol. 75, No.

3, 227-234

Copyright 1990 by the American Psychological Association, Inc. OO21-9OIO/9O/$OO.75

Operationalization of Goal Difficulty as a Moderator of the Goal Difficulty-Performance Relationship

Patrick M. Wright
Department of Management University of Notre Dame
Examined the research studies cumulated in recent quantitative reviews of the relationship between goal difficulty and performance to determine how goal difficulty has been operationalized. 4 categories (assigned goal level, self-set goal level, performance improvement, and difficulty perceptions) of operationalization were discovered, and the operationalization of goal difficulty was tested as a moderator of the relationship between goal difficulty and performance. Strong support for this moderating role was found; the different operationalizations accounted for 26% of the variance in effect sizes. Implications for operationalizing goal difficulty in future goal setting research are discussed.

The effectiveness of goal setting as a motivational tool is seldom questioned. Substantial empirical evidence has supported the proposition that specific, difficult goals lead to higher performance than do instructions to "do your best" or easy goals (Locke, Shaw, Saari, & Latham, 1981). On the basis of their recent meta-analyses, both Mento, Steel, and Karren (1987) and Tubbs (1986) concluded that the relationship between goal difficulty and performance has been well established. In fact, Mento et al. (1987) stated, "If there is ever to be a viable candidate from the organizational sciences for elevation to the lofty status of a scientific law of nature, then the relationships between goal difficulty, difficulty/specificity and task performance are most worthy of serious consideration" (p. 74). Conclusions such as this may need to be tempered because quite different results were obtained in these two meta-analyses, despite the fact that the two had a large percentage of studies in common. Tubbs (1986) found an effect size of .816 for goal difficulty, whereas Mento et al. (1987) reported an effect size of .581. In addition, Tubbs and Mento observed different results regarding moderators of the relationship between goal difficulty and performance. Tubbs (1986) found that both the setting (laboratory vs.field)and the manner in which goal difficulty was operationalized (direct measurement vs. subjective) moderated the goal difficulty-performance relationship, whereas Mento et al. (1987) found no consistent moderators of this relationship. Wood, Mento, and Locke (1987) added three studies to the Mento et al. (1987) meta-analysis, however, and found that task complexity explained 5.7% of the variance in effect sizes.
This research was partially supported by a grant from the Center for Research in Business, University of Notre Dame. I would like to extend special thanks to John Hollenbeck, Robert Vecchio, Howard Klein, the late Stephen Premack, and three anonymous reviewers for their valuable comments on an earlier version of the manuscript. 1 also thank Robert Wood, Anthony Mento, and Edwin Locke for providing their task complexity ratings. Correspondence concerning this article should be addressed to Patrick M. Wright, who is now at the Department of Management, Texas A&M University, College of Business, College Station, Texas 77843-4221. 227

In this study I examined the possible moderating role of the operationalization of goal difficulty in the relationship between goal difficulty and performance. This examination seems warranted for three reasons. First, Latham, Erez, and Locke (1988) found that the large variance in the relationship between participation and performance was attributable to minor differences in how the participation manipulation was operationalized. Second, both Mento et al. (1987) and Tubbs (1986) found significant unexplained variance in effect sizes across the goal difficulty-performance literature. Finally, Tubbs (1986) found some support for the moderating role of the operationalization of goal difficulty in the goal difficultyperformance relationship. Thus, this study tested the hypothesis that the operationalization of goal difficulty moderates the relationship between goal difficulty and performance. It goes beyond Tubbs's (1986) analysis of operationalization of goal difficulty as a moderator of the difficulty-performance relationship. Tubbs (1986) made a distinction only between objective and subjective measures of difficulty Although the difficulty perceptions category used here overlaps with Tubbs's (1986) subjective category, this analysis further differentiates the objective goal measures. Method
To test the hypothesis that the operationalization of goal difficulty moderates the relationship between goal difficulty and performance, I subjected the 70 studies analyzed by Mento et al. (1987) to the Hunter, Schmidt, and Jackson (1982) meta-analysis procedure. I coded these studies according to the categories described in the next paragraph and subjected them to the normal meta-analysis and then to moderator tests. The studies and the codings can be found in Table 1. My examination of these 70 studies revealed four coherent categories of different operationalizations.' Two raters rated the 70 studies. ' Two additional operationalizations of goal difficulty were observed from the Mento et al. (1987) studies. Afinalcoherent operationalization of goal difficulty is labelled "percentile goals," which consisted of assigning goals to perform in a certain percentile of the experimental group. Only two studies (three effect sizes) used this operationalization (Erez, 1977; Motowildo, Loehr, & Dunnette, 1978). In addition, two studies (Organ, 1977; Terborg, 1976) fell into the

228

PATRICK M. WRIGHT

Table 1 Studies of the Relations Between Goal Difficulty and Performance, Broken Down by Operationalization Investigators N Assigned goal level Locke (1966) Study 1 Study 2 Study 3 Locke, Bryan, & Kendall (1968) Study 1 Study 2 Sales (1970) Rothkopf& Billington (1975) Campbell & I lgen (1976) Laporte&Nath(1976) Masters, Furman, & Barden(1977) Bavelas& Lee (1978) Study 1 Study 2 Study 2a Bavelas& Lee (1978) Study 3 Study 4 Dossett, Latham, & Mitchell (1979) Bassett(1979) Latham & Saari (1979) Mowen, Middlemist, & Luther (1981) Mowen, Middlemist, & Luther (1981) Garland (1982) Latham & Marshall (1982) Locke (1982) Peters, Chassie, Lindholm, O'Connor, & Kline (1982) Wofford (1982), Study 1 Garland (1983) Campbell (1984) Locke, Frederick, Lee, & Bobko(1984) Garland (1984) Taylor, Locke, Lee, & Gist (1984) 49 56 23 70 30 73 92 82 96 32 48 128 54 48 30 40 116 60 62 62 60 125 247 120 92 58 56 181 71 169 0.847 1.658 1.81 0.966 0.990 0.919 0.636 0.467 0.654 1.116 0.10 1.76 1.066 1.128 1.273 0.913 0.344 1.099 0.585 -0.335 1.145 0.163 0.871 0.793 0.464 1.032 0:346 0.931 0.738 0.406 .396 .645 .679 .44 .456 .422 .307 .230 .314 .494 .05 .664 .476 .500 .551 .424 .171 .488 .284 -.165 .504 .082 .384 .373 .228 .458 .187 .424 .351 .200 d r Investigators N Self-set goal level (continued) Dachler& Mobley (1973) Dachler& Mobley (1973) Hamner & Harnett (1974) Latham, Mitchell, & Dossett (1978) Yukl& Latham (1978) Rakestraw & Weiss (1981) Latham & Marshall (1982) Matsui, Okada, & Kakuyama(1982) Latham ASteele (1983) Wood & Locke (1984) Study 2 Study 3 Study 4 Study 5 173 366 80 76 41 174 57 91 48 216 329 142 37 0.787 0.257 0.982 0.967 0.924 0.827 1.280 0.684 0.501 0.273 0.467 0.671 0.953 .368 .128 .445 .44 .428 .384 .546 .327 .248 .136 .228 .320 .440 d r

Performance improvement Locke & Bryan (1967) Locke (1968) Locke & Bryan (1969) Pritchard& Curtis (1973) London & Oldham (1976) Becker (1978) Strang, Lawrence, & Fowler (1978) Strang, Lawrence, & Fowler (1978) Mento, Cartledge, & Locke (1980) Study 1 Study 2 Locke & Shaw (1984) 69 20 40 81 180 40 50 50 195 406 212 0.436 0.549 0.356 0.507 0.129 0.356 0.364 -0.273 0.524 0.513 0.289 .216 .278 .158 .248 .065 .179 .183 -.135 .256 .248 .144

Difficulty perceptions Andrews & Farris (1972) Steers (1975) Hall & Hall (1976) Oldham (1976) Hall & Foster (1977) Ivancevich & McMahon (1977a) Ivancevich & McMahon (1977b) Ivancevich & McMahon (1977c) Ivancevich & McMahon (1977c) 78 133 283 42 61 190 141 90 38 0.336 0.035 0.241 0.205 0.000 0.424 0.111 0.668 0.204 .168 .016 .120 .103 .00 .232 .056 .32 .104

Self-set goal level Locke & Bryan (1968) Locke, Cartledge, & Knerr (1970), Study 5 323 54 0.581 0.189 .280 .096

Note. N = number of subjects; d = effect size in standard deviation units; and r = the correlation between goal difficulty and performance.

and agreement in codings was observed for 97% of the cases. The raters discussed the remaining studies until they reached consensus regarding the category of operationalization used. One rather general category of goal difficulty operationalization consisted of studies in which goal difficulty was operationalized in the

"other" category because they really could not be construed to measure goal difficuhy at all. However, because of the small number of studies in these categories, the categories were not included in the moderator analyses.

design phase as an absolute level of performance without a measured reference to ability This general category was further differentiated into two categories: "assigned goal level" and "self-set goal level." Thirty studies fell in the assigned goal level category (N= 2,555). Most often, these designs consisted of assigning two or three goals to experimental groups. The self-set goal level category consisted of 15 studies (N= 2,207). These studies allowed subjects to self-set goal levels, but did not instruct subjects to do so relative to their past performance (although one might assume that the subjects did use past performance as a reference). It is important to note that these two categories are by far the most common operationalizations of goal difficulty, and are most

OPERATIONALIZATION OF GOAL DIFFICULTY likely the categories that come to mind when one thinks of the relationship between goal difficulty and performance. I labelled the third category of goal difficulty operationalizations "performance improvement." Studies using this operationalization of goal difficulty normally set performance goals as a percentage of a past trial or block of trials. In all cases these studies used at least two different goal levels (e.g., a 10% or 20% increase), and the analysis compared the performance of the different goal groups. Eleven studies (N= 1,343) fell into this category. My fourth category of goal difficulty operationalizations, "difficulty perceptions," is similar to what Tubbs (1986) classified as "subjective" measures of goal difficulty These operationalizations were selfreports of intentions to perform well or perceptions of the difficulty of the goal. Nine studies (A^= 1,056) used this operationalization of difficulty. Because these operationalizations varied so much, they are covered in detail to illustrate the different ways recent meta-analyses (and consequently the goal setting literature) have defined goal difficulty. Andrews and Farris (1972) measured scientists' "experienced time pressure." Hall and Hall (1976) and Hall and Foster (1977) used 3-item scales measuring subjects' "intentions to do well." Ivancevich and McMahon (1977a, 1977b, 1977c) used a 5-item measure of goal "challenge" with items such as "It takes a lot of effort on my part to achieve the results expected for my job." Although no sample items were given. Steers (1975) also used a multiple-item measure that he termed "challenge." Finally, Oldham (1976) asked subordinates of a focal manager to indicate how often "My supervisor sets specific performance goals or quotas for me to achieve." This measure was correlated with that focal supervisors' manager's evaluation of those subordinates' productivity as a group. It should be noted that this category of operationalization really does not measure the specific quantitative goals discussed by Locke (1968). In fact, in two cases (Hall & Hall, 1976; Hall & Foster, 1977) the operationalization of goal difficulty clearly is more of the "Do your best" goal condition maligned by goal theorists. Andrews and Farris's (1972) measure of experienced time pressure would have to be considered more role overload than goal difficulty Level of analysis issues plague the Oldham (1976) study. Thus, from a theoretical perspective one could question whether or not this operationalization truly tests goal theory. If the operationalization of goal difficulty moderates the observed relationship between goal difficulty and performance, then these different categories of operationalization should exhibit different mean effect sizes. A number of tests for moderators are available and there is little consensus as to which test is best. Mento et al. (1987) used two moderator tests. The first consists of subgrouping the studies on the basis of the suspected moderator variable and then examining the unexplained variance within each subgroup. Hunter et al. (1982) stated that if the unexplained variance of each of the subgroups is less than the unexplained variance overall, then support is shown for the moderating relationship. The second test, the regression approach (Mabe & West, 1982; Steele & Ovalle, 1984) entails regressing the effect size on the values of the subgroups. Support is shown if the moderator variable explainsa significant amount of variance in effect sizes. This test is more powerful than the former test. In fact, in the Woodetal. (1987)reanalysisofthe Mento data, only the regression approach was used to test for task complexity as a moderator of the goal difficulty-performance relationship. To maintain consistency with the original data, I used both tests in this study.

229

Results In the preliminary analysis using all of the studies in the Mento et al. (1987) meta-analysis, these authors' results were exactly replicated with an effect size of .5451, an effect

corrected for unreliability to be .5831, an observed variance of .1495, and an unexplained variance of .1029. However, because of the deletion of five of the studies, the overall results were slightly different. Using the 65 studies to be tested in the moderator analysis resulted in a mean effect size of .5538, an effect corrected for unreliability of .5906, an observed variance of. 1447, and an unexplained variance of. 1070. These results are displayed in Table 2. The first moderator test was performed by subgrouping the studies as mentioned above and comparing the mean effect sizes and variances within each subgroup with the mean effect size and variance of the sample as a whole. It should be noted that at this point no corrections were made for measurement error. The rationale for this will be discussed later. This analysis revealed substantial differences in mean effect size between the assigned goal level and all other operationalizations. Examination of Table 2 reveals that the assigned goal level operationalization had the largest effect size (.1411) followed by self-set goal level (.5729), performance improvement (.3798), and difficulty perceptions (.2663). The unexplained variance was substantially reduced within all but the assigned goal level operationalization. Wood et al. (1987) noted that differences in criterion reliabilities might have accounted for the different effect sizes observed because of task complexity. A similar argument might be made here regarding the predictor reliabilities. Mento et al. (1987) used a mean predictor reliability of .72 for goal difficulty, but this reliability was based almost solely on the reliabilities reported in difficulty perception studies. Other studies either manipulated goal difficulty or used 1-item self-reports of a self-set goal level. In these cases, the reliability of the predictor seems unimportant. For these reasons, I made the correction for predictor unreliability only for the difficulty perception studies, not for the other studies (thus assuming a reliability of 1). The result of this strategy was to increase the estimated effect size of difficulty perceptions relative to the other operationalizations, but this increase did not appreciably change the results. As was previously mentioned, the subgrouping analysis is not extremely powerful for detecting moderators. Thus, I used the regression approach to determine if this categorization scheme explained a significant amount of variance in effect sizes and if the mean effect sizes within each operationalization differed significantly from one another. When more than two subgroups exist, this method can be performed by dummy coding the g subgroups as ^ - 1 variables and applying a multiple regression procedure (Cohen & Cohen, 1983). To do this, I created three dummy coded variables to reflect the four categories of operationalizations (the assigned goal level, self-set goal level, performance improvement, and difficulty perceptions categories, dummy coded 1, 0, 0; 0, 1, 0; 0, 0, 1; and 0, 0, 0 respectively). The coding provided for the observed B weights to reflect the contrasts of each of the first three categories against the difficulty perceptions category. I regressed the effect size on the three dummy coded variables, which resulted in a multiple R of .509 (R^ = .259, adjusted R^ = .224, p < .001). The correlations and B weights are displayed in Table 3.1 conducted tests for significant differences in B weights according to the method recommended by Cohen and Cohen (1983). This examination indicated that the assigned goal level operationalization was significantly different

230

PATRICK M. WRIGHT

Table 2 Meta-Analysis Results of Relationship Between Goal Difficulty and Performance, Broken Down by Operationalization of Goal Difficulty
Number of studies 65 30 15 11 9 Mean observed d .5538 .7477 .5729 .3798 .2663 Mean corrected d .5906 .7797 .5974 .3960 .3414 Variance corrected for sampling error .0377 .0732 .0283 .0334 .0344 Percent variance unexplained 73.9 63.0 59.3 6.2 4.7

Variable Overall Assigned level Self-set level Improvement Perception

Sample srze 7161 2555 2207 1343 1056

Observed variance .1447 .1981 .0696 .0356 .0361

Unexplained variance .1070 .1249 .0413 .0022 .0017

Note, d stands for effect size in standard deviation units.

from the performance improvement and difficulty perception categories. In addition, the self-set goal level operationalization was significantly different from the difficulty perception category No other categories were different from one another. I performed an additional analysis to examine the joint role of operationalization of goal difficulty and task complexity as explanations of variance in effect sizes. I obtained the task complexity codings used in the Wood et al. (1987) study from the authors and computed two regressions. The first equation regressed the effect sizes on task complexity The second regression equation entered the dummy coded variables in the first step and task complexity in the second step. In the first regression equation task complexity explained 13% of the variance in effect sizes.^ When I controlled for the operationalization of goal difficulty in the first step of the regression, the amount of incremental variance explained by task complexity was reduced, yet task complexity still accounted for a significant 7% (p < .01) of the variance (total R^ = .33, p < .01).

Discussion
Over 20 years of research on goal setting has shown substantial support for goal theory This article argues, however, that recent cumulations of this research have ignored the construct validity issues relative to the most basic construct of the theory: goal difficulty The results of this moderator analysis show that 26% of the variance in effect sizes observed in goal difficultyperformance relationships can be explained by the manner in which goal difficulty was operationalized. In contrast. Wood et al. (1987) demonstrated that task complexity moderates this relationship based on its ability to explain 5.7% of the variance in these effect sizes. If task complexity is viewed as a moderator, then serious consideration must be given to the operationalization of goal difficulty as a moderator of the relationship between goal difficulty and performance. In this study the operationalization explained twice as much variance as did task complexity, and, even after controlling for task complexity, the operationalization of goal difficulty explained an incremental 20% of the variance in effect sizes. The purpose of this article is not to imply that goal setting is not an effective motivational tool, nor to deny that a positive relationship exists between goal difficulty and performance; my purpose is to show that the way in which goal difficulty is operationalized may have profound implications for the effect size observed. The concept of goal difficulty seems so com-

mon-sensical that one might assume that all manipulations are the same. This article has demonstrated, however, that this simply is not the case. The results showed that although all the operationalizations were positively related to performance, these relationships differed significantly in their strength. These results indicate that perhaps operationalizations that have been cumulated in past meta-analyses of goal difficulty have either measured different constructs or measured the same construct with different levels of validity Examination of these operationalizations reveals that three of the four operationalizations (assigned goal level, self-set goal level, and performance improvement) truly represent the specific difficult goals from which goal theory makes predictions. However, these operationalizations exhibited significantly different effect sizes. Examining the reason for the differences between these operationalizations reveals some extremely important implications for the interpretations of past, as well as the design of future, goal-setting research. The assigned goal level operationalization may display higher effect sizes than performance improvement because the range in difficulty was greater than for the other operationalizations. If easy, moderate, and difficult goals (according to group performance norms) are assigned to individuals randomly, some high-ability subjects will have extremely easy goals and some low-ability subjects will have extremely difficult goals. This situation would lead to high-ability subjects lowering their level of performance to be in line with the goal. For this reason Locke and Latham (1990) stated that do-best goals would be more effective than specific easy goals (Locke, Mento, &Katcher, 1978). Under performance improvement goals, however, all subjects' goals would be much closer to their ability levels. Under self-set goal level operationalizations, in spite of the spurious relationship between goal level and performance attributable to ability, subjects would still be expected to set goals that would be neither extremely difficult nor extremely easy Thus, these operationalizations would be expected to have smaller ranges in goal difficulty, and thus lower observed effect sizes.
^ In the Wood et al. (1987) meta-analysis, task complexity explained 5.7% of the variance in effect sizes. That study, however, added three unpublished studies to the original Mento et al. (1987) meta-analysis, and these studies were not included in this analysis. The exclusion of these studies, as well as the studies mentioned in Footnote 1, is the reason for the differences in the amount of variance explained by task complexity (5.7% vs. 13%).

OPERATIONALIZATION OF GOAL DIFFICULTY

231

Table 3 Regression Results of Moderator Analysis

Operationalization Level (assigned) Level (self-set) Improvement Perceptions .391 .076 .295 B weight'
.557

Significance contrasts Significantly different from perceptions category at .01 and significantly different from improvement category at .05. Significantly different from perceptions category at .01.

.437 .088 .253"

Note. N= 65. The B weights do not match the mean effect sizes in Table 2 because Table 2 uses sample-weighted effect sizes, whereas Table 3 uses unit-weighted effect sizes. ^ Because of dummy coding, the B weight for the perceptions category is the residual term of the full regression equation.

Although these three operationalizations may be equally valid, these results have implications for research on the goal difficulty-performance relationship. First, the greater range in goal difficulty inherent in a goal level operationalization is useful in that it should increase the effect size and thus increase the power for detecting differences between goal levels. Thus, research directed at examining variables that might moderate or mediate the relationship between goal level and performance can most efficiently be conducted using the assigned goal level operationalization. However, a weakness of the operationalization is that one must question the accuracy of including such studies in the computation of a percentage increase attributable to goal difficulty as Mento et al. (1987) did. For example, Mento et al. (1987) referred to goal setting as "a motivational approach for enhancing productivity" (p. 52), and expressly stated that one purpose of their study was ". . . to know, for example, what percentage increase in productivity might be expected when specific hard goals are used as an organizational intervention" (p. 56). On the basis of their analysis they stated that "for goal difficulty the effect size ^= .5813 (across all studies) is equal to a productivity increase of 11.63%" (p. 76). Such a statement is misleading because the effect size they computed does not represent a level of performance increase, but only a linear relationship between goal level and performance. In other words, a large effect size will be observed in an assigned goal level operationalization, but the way that goal setting causes this effect may be by causing a number of subjects to decrease their performance (Locke etal., 1978). With regard to the true effect size of goal difficulty for increasing performance, a more accurate estimate may be much closer to the .3798 observed in the performance improvement operationalization. This may support Dunnette's (1973) contention that ability is a much more potent determinant of performance than motivation. Although this does not preclude the value of motivational interventions, it may call for a more conservative estimate of their effectiveness for increasing performance. This study demonstrated that vast differences in effect sizes have been observed in past research because of differences in the way goal difficulty was operationalized. Future research should pay more attention to operationalizing goal difficulty in a way that accurately depicts the theoretical construct being examined. This is becoming increasingly important because a variety of motivational theories, such as expectancy theory (Campbell & Pritchard, 1976; Dachler & Mobley, 1973; Matsui, Okada, & Mizuguchi, 1981), NPI theory (Naylor & Ilgen,

1984), control theory (Campion & Lord, 1982; Klein, 1989), cognitive mediation theory (Garland, 1985), and goal theory (Locke et al. 1981), have attempted to explain the relationship between goal difficulty and performance. These explanations will require more theoretically specific definitions of the construct of goal difficulty, and these definitions may vary according to the specific theory being tested. In conclusion, my analysis supports Schwab's (1980) discussion of the positive value of examining and exploring construct validity previous to or concurrently with substantive validity Until now, however, the goal-setting literature has been virtually devoid of construct validation of one of the most basic constructs of goal-setting theorygoal difficulty This study suggests that the construct of goal difficulty needs to be theoretically analyzed before it is operationally defined. Such construct validity consideration in the design phase of goal-setting research may make future meta-analyses of the goal difficultyperformance relationship better able to explore potential situational moderators such as monetary incentives or individual differences. If so, then our understanding of substantive relationships may increase as a result of the increased concern for construct validity References Andrews, F M., & Farris, G. G. (1972). Time pressure and performance of scientists and engineers: Afive-yearpanel study. Organizational
Behavior and Human Performance, 8,185-200. Campbell, J. P, & Pritchard, R. D. (1976). Motivation theory in industrial and organizational psychology In M. D. Dunnette (Ed.), Handbook ofindustrial and organizational psychology (pp. 63-130). Chicago, IL: Rand McNally Campion, M. A., & Lord, R. G. (1982). A control systems conceptualization of the goal setting and changing process. Organizational Behavior and Human Performance, 30, 265-287. Cohen, J., & Cohen, P (1983). Applied multiple regression/correlation analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum. Dachler, H. P, & Mobley, M. F (1973). Construct validation of an instrumentality-expectancy-task-goal model of work motivation: Some theoretical boundary conditions [Monograph]. Journal ofApplied Psychology 58, 397-418. Dunnette, M. D. (1973). Performance equals ability and what? (Jtch. Rep. No. 4009, Contract No. N0014-68-A-0141-003). San Diego: Office of Naval Research, Personnel Training and Research Programs, Psychological Sciences Division. Erez, M. (1977). Feedback: A necessary condition for the goal settingperformance relationship. Journal of Applied Psychology, 62, 624-627. Garland, H. (1985). A cognitive mediation theory of task goals and human performance. Motivation and Emotion, 9, 345-367.

232

PATRICK M. WRIGHT

Hall, D. T, & Foster, L. W (1977). A psychological success cycle and Matsui, T., Okada, A., & Mizuguchi, R. (1981). Expectancy theory goal setting: Goals, performance, and attitudes. Academy of Manageprediction of the goal theory postulate: The harder the goals, the ment Journal, 20, 282-290. higher the performance. Journal ofApplied Psychology, 66, 54-58. Hall, D. T., & Hall, FS. (1976). The relationship between goals, perforMento, A. J., Steel, R. P, & Karren, R. J. (1987). A meta-analytic study mance, success, self-image, and involvement under different organiof the effects of goal setting on task performance: 1966-1984. Orgazational climates. Journal of Vocational Behavior, 9, 267-278. nizational Behavior and Human Decision Processes, 39, 52-83.^ Hunter, J. E., Schmidt, F L., & Jackson, G. B. (1982). Meta-analysis: Motowidlo, S., Loehr, V, & Dunnette, M. D. (1978). A laboratory study Cumulating research findings across studies. Beverly Hills, CA: Sage. of the effects of goal specificity on the relationship between probabilIvancevich, J. M., & McMahon, J. T. (1977a). Black-White differences ity of success and performance. Journal of Applied Psychology, 23, in a goal-setting program. Organizational Behavior and Human Per172-179. formance, 20, 287-300. Naylor, J. C, & Ilgen, D. R. (1984). Goal-setting: A theoretical analysis Ivancevich, J. M., & McMahon, J. T. (1977b). Education as a moderator of a motivational technology In B. Staw & L. L. Cummings (Eds.), of goal-setting effectiveness. Journal of Vocational Behavior, 11, Research in organizational behavior (Wo\. 6, pp. 95-140). Greenwich, 83-94. CT: JAI Press. Ivancevich, J. M., & McMahon, J. T. (1977c). A study of task-goal attri- Oldham, G. R. (1976). The motivational strategies used by supervisors: butes, higher order need strength, and performance. Academy of Relationships to effectiveness indicators. Organizational Behavior Management Journal, 20, 552-563. and Human Performance, 15, 66-86. Klein, H. J. (1989). An integrated control theory model of work motiOrgan, D. W (1977). Intentional versus arousal effects of goal setting. vation. Academy ofManagement Review, 14,150-172. Organizational Behavior and Human Performance, 18, 377-389. Latham, G. P, Erez, M., & Locke, E. A. (1988). Resolving scientific Schwab, D. P (1980). Construct validity in organizational behavior. In disputes by the joint design of crucial experiments by the antagoB. Staw & L. L. Cummings (Eds.), Research in organizational behavnists: Application to the Erez-Latham dispute regarding participaior (Vol. 2, pp. 3-43). Greenwich, CT: JAI Press. tion in goal setting. Journal of Applied Psychology, 73, 753-772. Steele, R. P, & Ovalle, N. K. (1984). A review and meta-analysis of Locke, E. A. (1968). Toward a theory of task motivation and incentives. research on the relationship between behavioral intentions and Organizational Behavior and Human Performance, 3,157-189. employee turnover. Journal of Applied Psychology, 69, 673-686. Locke, E. A., & Latham, G. P (1990). A theory ofgoal setting and task Steers, R. M. (1975). Task-goal attributes, achievement, and superviperformance. Englewood Cliffs, NJ: Prentice-Hall. sory performance. Organizational Behavior and Human PerforLocke, E. A., Mento, A. J., & Katcher, B. L. (1978). The interaction of mance, 13, 392-403. ability and motivation in performance: An exploration of the meanTerborg, J. R. (1976). The motivational components of goal setting. ing of moderators. Personnel Psychology, 31, 269-280. Journal ofApplied Psychology, 61, 613-621. Locke, E. A., Shaw, K. N., Saari, L. M., & Latham, G. P (1981). Goal setting and task performance: 1969-1980. Psychological Bulletin, Tubbs, M. (1986). Goal setting: A meta-analytic examination of the 90,125-152. empirical evidence. Journal of Applied Psychology, 71, 474-483. Mabe, P A., & West, S. G. (1982). Validity of self-evaluation of ability: Wood, R. L., Mento, A., & Locke, E. (1987). Task complexity as a A review and meta-analysis. Journal of Applied Psychology, 67, moderator of the goal difficulty-performance relationship. Journal 280-296. of Applied Psychology 73, 416-425.

OPERATIONALIZATION OF GOAL DIFFICULTY

233

Appendix Studies Used for Categorization

Andrews, F M., & Farris, G. F (1972). Time pressure and performance of scientists and engineers: Afiveyear panel study. Organizational Behavior and Human Performance, 8,185-200. Bassett, G. A. (1979). A study of the effects of task goal and schedule choice on work performance. Organizational Behavior and Human Performance, 24, 202-227. Bavelas, J. B., & Lee, E. S. (1978). Effects of goal level on performance: A tradeoff of quantity and quality Canadian Journal of Psychology 52,219-240. Becker, L. J. (1978). Joint effect of feedback and goal setting on performance: Afieldstudy of residential energy conservation. Journal of Applied Psychology 63, 428-433. Campbell, D. J. (1984). The effect of goal contingent payment on the performance of a complex task. Personnel Psychology. 37, 23-40. Campbell, D. J., & Ilgen, D. R. (1976). Additive effects of task difficulty and goal setting on subsequent task performance. Journal ofApplied Psychology, 61, 3\9-i24. Dachler, H. P, & Mobley, W H. (1973). Construct validation of an instrumentality-expectancy-task-goal model of work motivation: Some theoretical boundary conditions [Monograph]. Joumalof Applied Psychology, 58, 397-418. Dossett, D. L., Latham, G. P, & Mitchell, T. R. (1979). The effects of assigned versus participatively set goals, KR, and individual differences when goal difficulty is held constant. Journal of Applied Psychology 64,29 \-29S. Erez, M. (1977). Feedback: A necessary condition for the goal settingperformance relationship. Journal of Applied Psychology, 62, 624-627. Garland, H. (1982). Goal levels and task performance: A compelling replication of some compelling results. Journal of Applied Psychology 67, 245-248. Garland, H. (1983). Influence of ability, assigned goals, and normative information on personal goals and performance: A challenge to the goal attainability assumption. Journal of Applied Psychology 68, 20-30. Garland, H. (1984, August). A cognitive mediation theory oftask goals and human performance. Paper presented at the 44th annual convention of the Academy of Management, Boston. Hall, D. T, & Foster, L. W (1977). A psychological success cycle and goal setting: Goals, performance, and attitudes. Academy of Management Journal, 20, 282-290. Hall, D. T, & Hall, F S. (1976). The relationship between goals, performance, success, self-image, and involvement under different organizational climates. Journal of Vocational Behavior, 9, 267-278. Hamner, W C, & Harnett, D. L. (1974). Goal-setting, performance and satisfaction in an interdependent task. Organizational Behavior and Human Performance, 12, 217-230. Ivancevich, J. M., & McMahon, J. T. (1977a). Black-White differences in a goal-setting program. Organizational Behavior and Human Performance, 20, 287-300. Ivancevich, J. M., & McMahon, J. T. (1977b). Education as a moderator of goal-setting effectiveness. Journal of Vocational Behavior, 11, 83-94. Ivancevich, J. M., & McMahon, J. T. (1977c). A study of task-goal attributes, higher order need strength, and performance. Academy ofManagement Journal, 20, 552-563. Jackson, S. E., & Zedeck, S. (1982). Explaining performance variability: Contributions of goal setting, task characteristics, and evaluative contexts. Journal of Applied Psychology, 67, 759-768. LaPorte, R. E., & Nath, R. (1976). Role of performance goals in prose learning. Journal of Educational Psychology 68, 260-264. Latham, G. P, & Marshall, H. A. (1982). The effects of self-test, participatively set, and assigned goals on the performance of government employees. Personnel Psychology 35, 399-404. Latham, G. P, Mitchell, T. R., & Dossett, D. L. (1978). The importance of participative goal setting and anticipated rewards on goal difficulty and job performance. Journal of Applied Psychology, 63, 163-171. Latham, G. P, & Saari, L. M. (1979). The importance of supportive relationships in goal setting. Journal of Applied Psychology, 64, 151-156. Latham, G. P, & Steele, T. P (1983). The motivational effects of participative versus assigned goal setting on performance. Academy of Management Journal, 26, 406-417. Locke, E. A. (1966). The relationship of intentions to level of performance. Journal of Applied Psychology 50, 60-66. Locke, E. A. (1968). The effects of knowledge of results, feedback in relation to standards, and goals on reaction time performance. American Journal of Psychology 81, 566-574. Locke, E. A. (1982). Relation of goal level to performance with a short work period and multiple goal levels. Journal ofApplied Psychology 67,512-514. Locke, E. A., & Bryan, J. F (1967). Performance goals as determinants of level of performance and boredom. Journal ofApplied Psychology 57,120-130. Locke, E. A., & Bryan, J. F (1968). Grade goals as determinants of academic achievement. Journal ofGeneral Psychology, 79, 217-228. Locke, E. A., & Bryan, J. F (1969). The directing function of goals in task performance. Organizational Behavior and Human Performance, 4, 35-42. Locke, E. A., Bryan, J. F, & Kendall, L. M. (1968). Goals and intentions as mediators of the effects of monetary incentives on behavior. Journal of Applied Psychology 52,104-121. Locke, E. A., Cartledge, N., & Knerr, C. (1970). Studies of the relationship between satisfaction, goal setting and performance. Organizational Behavior and Human Performance, 5,135-158. Locke, E. A., Frederick, E., Lee, C, & Bobko, P (1984). Effects of selfefficacy, goals, and task strategies on task performance. Journal of Applied Psychology 69, 241 -251. Locke, E. A., & Shaw, K. N. (1984). Atkinson's inverse-U curve and the missing cognitive variables. Psychological Reports, 55, 403-412. London, M., & Oldham, G. R. (1976). Effects of varying goal types and incentive systems on performance and satisfaction. Academy of Management Journal, 19, 537-546. Masters, J. C, Furman, W, & Barden, R. C. (1977). Effects of achievement standards, tangible rewards and self-dispensed achievement evaluations on children's task master. Child Development, 48, 217-224. Matsui, T, Okada, A., & Kakuyama, T. (1982). Influence of achievement need on goal setting, performance, and feedback effectiveness. Journal of Applied Psychology, 67, 645-648. Mento, A. J., Cartledge, N. D, & Locke, E. A. (1980). Maryland vs. Michigan vs. Minnesota: Another look at the relationship of expectancy and goal difficulty to task performance. Organizational Behavior and Human Performance, 25, 419-440. Motowidlo, S., Loehr, V, & Dunnette, M. D. (1978). A laboratory study of the effects of goal specificity on the relationship between probability of success and performance. Journal of Applied Psychology, 23, 172-179. Mowen, J. C, Middlemist, R. D, & Luther, D. (1981). Joint effects of assigned goal level and incentive structure on task performance: A laboratory study. Journal of Applied Psychology, 66, 598-603.

234

PATRICK M. WRIGHT Strang, H. R., Lawrence, E. C, & Fowler, P C. (1978). Effects of assigned goal level and knowledge of results on arithmetic computation: A laboratory study. Journal ofApplied Psychology, 63, 29-39. Taylor, M. S., Locke, E. A., Lee, C, & Gist, M. (1984). Type A behavior and faculty research productivity: What are the mechanisms? Organizational Behavior and Human Performance, 34, 402-418. Terborg, J. R. (1976). The motivational components of goal setting. Journal of Applied Psychology, 61, dX 3-621. Wofford, J. C. (1982). Experimental tests of the goal-energy-effort requirement theory of work motivation. Psychological Reports 50 1259-1273. Wood, R. E., & Locke, E. A. (1984). The effects of self-efficacy on academic performance. Unpublished manuscript. University of New South Wales, Australia. Yukl, G. A., & Latham, G. P (1978). Interrelationships among employee participation, individual differences, goal difficulty, goal acceptance, goal instrumentality and performance. Personnel Psychology, 31, 305-323.

Oldham, G. R. (1976). The motivational strategies used by supervisors: Relationships to effectiveness indicators. Organizationat Behavior and Human Performance, IS, 66-86. Organ, D. W (1977). Intentional vs. arousal effects of goal setting. Organizational Behavior and Human Performance, 18, 377-389. Peters, L. H., Chassie, M. B., Lindholm, H. R., O'Connor, E. J., & Kline, C. R. (1982). The joint influence of situational constraints and goal setting on performance and affective outcomes. Journal of Management, 8, 7-20. Pritchard, R. D., & Curtis, M. I. (1973). The influence of goal setting and financial incentiveson task performance. Organizationat Behavior and Human Performance, 10, 175-183. Rakestraw, T. L. Jr., & Weiss, H. M. (1981). The interaction of social influence and task experience on goals, performance, and performance satisfaction. Organizationat Behavior and Human Performance, 27, 326-344. Rothkopf, E. Z., & Billington, M. J. (1975). A two-factor model of the effect of goal-descriptive directions on learning from text. Journal of Educational Psychotogy, 67,192-204. Sales, S. M. (1970). Some effects of role overload and role underload. Organizationat Behavior and Human Performance, 5, 592-608. Steers, R. M. (1975). Task-goal attributes, achievement, and supervisory performance. Organizationat Behavior and Human Performance, 13, 392-403.

Received April 26,1989 Revision received October 23,1989 Accepted November 6,1989

Correction to Paese and Switzer

A replication of Paul W Paese's and Fred S. Switzer's study, "Validity Generalization and Hypothetical Reliability Distributions: A Test of the Schmidt-Hunter Procedure" (Journal of Applied Psychology, 1988, Vol. 73, No. 2, pp. 267-274) found artifactual variance estimates considerably smaller than those in the original study. The discrepancies were traced to an error in the original Monte Carlo computer program. (We thank Frank Schmidt for calling to our attention the possibility of such an error.) As in the original study, true reliability distributions and sample sizes were systematically varied to observe their effects on variance estimates produced by the Schmidt and Hunter noninteractive and interactive validity generalization (VG) equations. Consistent with the original study, the results of the replication indicated that artifactual variance was overestimated by the noninteractive equation. Contrary to the original study, the interactive equation provided estimates of artifactual variance that were unbiased. Moreover, the replication supported our original conclusion that the use of hypothetical reliability distributions can lead to inaccurate conclusions in VG research. (Interested readers can obtain revised tables andfiguresfrom Paul W Paese, Department of Psychology, University of Missouri-St. Louis, 8001 Natural Bridge Road, St. Louis, Missouri 63121.)