©00O0 i JPROD INDY MANAG 109 Tear :too nt Forecasting New Product Market Potential: Combining Multiple Methods Robert J. Thomas Multiple measurement approaches are ofien rec- ommended to forecast market potential for new products; however, methodologies to accomplish this have not been fully developed. In this article, Robert Thomas describes an implementation of a ‘multiple methods approach for combining inde- pendent forecasts of new product market poten- tial 10 improve new product planning decisions. The approach involves (1) defining a concept of market potential for the new product, (2) obtain- ing independent operationalizations and mea- sures of the concept with multiple methods, and (3) using the concept as a basis for reconciling and combining forecasts obtained from the dif- ferent methods. In the last step, a systematic pro- cedure linked to the conceptual definition of mar- ket potential leads to the estimation of judgmental weights for combining the indepen- dent forecasts. A case study for a new type of electronic mail service illustrates the approach. Adress correspondence o Robert J. Thomas, Georgetown Uni versity, School of Business Administration, Washington, D.C 20087, 52 Vanderbit ve. New York, NY 10017 Introduction Developing new products and services requires reasonably good forecasts of market potential Newness of products or markets to both con- sumers and manufacturers and lack of historic data complicate the estimation of market poten- tial and, subsequently, new product planning de- cisions. Strategic directions, go/no-go decisions, and resource allocations for the new product may depend on the forecasted size of the potential market. For example, consider a potential market with a forecasted volume much larger than its true size. A small firm following a niche market- ing strategy may decide nor to enter this appar- ently large market since it might attract larger competitors, thereby losing a valuable niche op- portunity. Likewise, another firm deciding to pur- sue this ‘apparently large market may fail to achieve expected return on investment since true market size is much smaller than forecast. Guidelines to improve forecasts of new prod- uct market potential generally include the use of multiple methods. Wind [16), for example, re ommends at least two forecasting methods at each stage of the new product development pro- cess. As he notes, “this multiple method ap- proach would offer cross-validation of the out- comes and would increase management's confidence in the forecasting results" (16, p. 460]. Urban and Hauser (15, p. 291] ‘*. . . feel that itis sound practice to use these multiple techniques to get the best estimates of purchase potential.”” In the forecasting literature, Armstrong [1, p. 298] recommends: “The best way to evaluate the fore~ orm ereanns3 0110s PROD INNOV MANAG Ioart0o-119 BIOGRAPHICAL SKETCH Rober J. Thomas isan anita professor inthe School of Business ‘Admiaistation af Georgetown University. He received bis Ph.D. ia Business and Applied Economics from the Wharton Scho! at the Universiy of Pennylvans, His primary interests ae isthe ares of Amand estination for sew products and organizational buying be “The stor woul! ike to thank the Provost of Georgstwn University forthe nancial swsivance ma valle trowzh the Sommer Gran Program in support of his paper The thor woul also ks thank Professor Otkmar Winkle fr his comments on an eae ral ofthis casting ability of a model is to compare it with other models.” The motivation to use multiple methods recog- nizes the inherent difficulty of measuring a con- cept—market potential—which is largely an ab- straction. Unlike “sales,” which can be tallied at year end, market potential represents what could have been sold by all competitors under certain market conditions and cannot easily be counted at year end—if at all, Changing marketing mix decisions, consumer behavior, competitor actions, and environmental factors (technology, economy, etc.) can have substantial effect on the magnitude of market potential during a planning period. Market potential is an especially elusive concept to measure for new products where con- sumers, competitors, and other market factors are yet to be clearly defined. While multiple methods are often recom- mended to estimate new product demand con- cepts, managers receive few guidelines to imple: ment such a methodology. This article describes an approach to cope with this problem: (1) a con- cept of market potential must be defined, (2) mul- tiple operationalizations and measures of the con- cept must be obtained, and (3) these multiple measures can be combined (if desired) to produce a single forecast. Since the approach relies on the use of expert judgement to combine forecasts, systematic procedures are prescribed to impart discipline to the process. The next section in- cludes a description of the proposed approach. Subsequently, a case study forecasting market potential for a new type of electronic mail service illustrates the approach. RJ. THOMAS: A Multiple Methods Approach Using two or more methods to study the same phenomenon derives from the multiple operation- ism of the behavioral sciences [4.5.14]. Like be- havioral phenomena such as power or attitudes, market potential for a new product is a concept for which exact values may never be known Multiple methods help to ensure that what is mea- sured is due to the phenomenon under study and not the method. The approach assumes that inde- pendent multiple methods used to measure the same concept lead to greater accuracy than a sin- gle method. The effectiveness of the approach rests on the premise that weaknesses in each sin- gle method will be compensated by the counter- balancing strengths of another method. Implic- ly, multiple and independent methods do not share the same weaknesses or potential for bias Therefore, the approach purports to exploit the assets of each method and neutralize the liabil ties [71 The first step in implementing a multiple meth- ods approach requires defining a concept of mar- ket potential to provide structure to the phenome- non under study. The second step requires at least two operationalizations of the defined con- cept of market potential which should produce two forecasts. The third step involves using the conceptual definition and various operationaliza- tions to reconcile differences or similarities in the resulting forecasts of market potential. In the third step, alternative procedures for combining independent forecasts into a single forecast can be considered to facilitate planning decisions The sections below review the three steps. Defining a Concept of Market Potential Kotler [8] defines market potential as “the limit approached by market demand as industry mar- keting effort goes to infinity, for a given environ- ment,” [8, p. 230]. Included in this definition is the concept of market demand: “Market demand for a product is the total volume that would be bought by a defined customer group geographical area in a defined time period in a defined marketing environment under a defined marketing program,” (8, p. 228]. d terms in this definition of demand represent eight key factors of the concept of market potential asindustry marketing effort goes to infinity. Nota- bly, each factor is also a concept with gener- ally accepted definitions in the marketing litera- ture The network of eight related concepts gives meaning to the concept of market potential and provides a basis for measurement. By operationa- lizing each factor with specific measures, the value ultimately assigned to market potential can be qualified for analytical and strategic purposes In effect, the eight factors defining the concept of market potential provide the basis for a disci- plined way to investigate and explain differences and similarities in forecasts obtained from the various methods used. Obtaining Forecasts Through Multiple Methods To implement a multiple methods approach, two ‘or more estimates of market potential based on independent methods must be obtained. A vari- ety of methods based on surveys, experimenta- tion, observation, secondary information, and expert judgement can generate independent estimates of market potential. For example, a survey of buyer intentions, a secondary analysis of relevant SIC codes, and expert estimates rep- resent three different methods for deriving three different forecasts of market potential The assumptions used to operationalize each method can be systematically defined and com- pared according to the eight dimensions of the market potential concept. For example, did each method use the same or different definitions of the customer group to operationalize this dimen- sion of market potential? Different operationali- zations of customer groups through sampling may have been used; however, this could be a source of explaining differences (if any) in the forecasts. When feasible, the methods and their conceptual operationalizations can be designed as needed into the market research program for the new product development process. This imparts some control over the differences in methods. How- ever, in practice, one or more secondary studies may be available to purchase. In these cases, un- less access to the methodologies (e.g., sampling, data collection, analyses) are available, interpre- tations of the operationalization must be made to reconcile different forecasts. J PROD INNOV MANAG Mm Reconciling and Combining Forecasts If two different methods generate similar fore- casts, a manager might be comfortable with the results, If, however, two very different forecasts emerged, the manager could (1) reject the esti mates, (2) accept the two estimates as the high and low ends of a range, (3) select only one of the estimates, or (4) attempt to reconcile or combine the estimates in some way to arrive at a single forecast for planning purposes. The first two op- tions require no further consideration, however, the latter two do. The third option requires a careful evaluation of each method employed to select one over the other. The fourth option re- quires a careful evaluation plus a way to combine the two forecasts into one. Reconciling and combining the forecasts from different measures can be accomplished with the analyst's judgement and/or a quantitative model. Jick [7] recommends a “‘holistic"® approach that relies on the analyst’s interpretations of the results with respect to the different measures, methods, and underlying assumptions. This might include directly adjusting the quantitative forecasts from a survey upward or downward ac- cording to more detailed interpretations of the product and potential buyer reaction. This holis- tic approach relies largely on analysts to interpret the results of various studies. It can be used to select one forecast over the other, or to attempt to combine them. At the other extreme, a purely quantitative model for combining requires a mathematical ag- gregation rule. The simplest example is the arith- metic average of the estimates produced from ich of the methods: M = m/o. a where: M = market potential forecast for new product; and m; = market potential forecast from method i, for i = 1 to n methods. Makridakis et al. [10] found that such simple av- eraging of forecasts was more accurate than weighted average methods. In a later study, Ma- kridakis and Winkler [11] found that using simple averages of forecasts improved forecast accuracy and decreased variability of accuracy. They als112. PkOD INNOV MANAG oar 109-119 found that the average mean absolute percentage error (MAPE) dropped substantially as the num- ber of methods increased beyond a single method. They concluded that unless an adequate theory exists concerning the data generating pro- cess or strong evidence is available indicating that a particular forecasting method is better than other methods for a given situation, then combin- ing forecasts by simply averaging them is recom- mended. A third approach assumes an adequate theory exists to measure market potential (i.e., the con- ceptual definition presented above) and uses judgement and quantitative procedures to recon- cile and combine the results of different fore- casts. It involves the systematic development of a judgemental parameter for equation (1) above weighting the various estimates. More specifi- cally: M Sa + pitm)/(n), @ where: M = market potential forecast for new product; m, = market potential forecast from method i, for i= 1 to n methods; and p; = judge- mental parameter weighting market potential forecast i, for i = 1 to n methods. The reference value for the judgemental parame- ter (p) is zero; i.e., the full weight of the market potential estimate from a particular method should be counted in the combined forecast. The value of p can, however, be greater than or less than zero if a review of the method suggests an under- or overestimation of potential volume for the new product. The judgemental parameter (p) can be ex- pressed as a percent increase or decrease in mar- ket potential. Thus, p = .20 indicates a 20% posi- tive adjustment in the original estimate. While numerous procedures may be used to judgemen- tally obtain p, Kotler’s [8] eight factors defining the concept of market potential provide the nec- essary discipline to assure a systematic estimate of p. Each forecast should be systematically eval- uated along the conceptual criteria in order to obtain a p, value for each method. More specifi- cally: RJ. THOMAS: P= dG 3) where: p; judgemental parameter weighting mar- ket potential forecast i, for i = 1 to n methods; and cy = percent increase or decrease in market potential based on an evaluation of the extent to which the method used to derive each forecast affects market potential on each of j = 1 to k iteria. As in the case of p,, cj can be referenced at a value of zero; i.c., zero means that the criterion has no appreciable effect on increasing or de- creasing market potential. If c; is assigned a value of say —.20, then it would indicate the effect on market potential of this particular criterion to be a 20% decrease. To the extent possible, the same conceptual criteria should be considered in evalu- ating p, for each method. A case study of elec- tronic mail services illustrates this approach. Case Study ‘The emerging technology of electronic mail pro- vides an opportunity to illustrate a multiple meth- ‘ods approach to estimate new product market po- tential. The United States Office of Technology Assessment (OTA) [6] defines three general types of electronic mail and message systems (EMS). Generation | EMS involves hard copy input, elec- tronic transmission (telephone, microwave, and/ or satellite), and hard copy output. Generation IT EMS involves electronic input, electronic trans- mission, and hard copy output. Generation ILL EMS involves electronic input, electronic trans- mission, and electronic output. In this case, the market potential for a type of Generation Il EMS (EMS-II) service is considered. Decision Situation The new product concept involves clectronic ter- minal input of a message up to 2 pages in length, electronic processing (merging, etc.) of the mes- sage, and hard copy output (stationery and enve- lope) of at least 200 units, delivered through the U.S. Postal Service. The concern with forecast- ing market potential was part of a larger concern for estimating year-to-year demand for the newFORECASTING NEW PRODUCT MARKET POTENTIAL service launched in 1982. Expected demand would be input for decisions to change product features, price, and promotion. While managers in the organization supplying the service were the final decision makers, the type of forecasts pre- sented below were prepared by outside analysts. For purposes of illustration, this analysis focuses on the problem of forecasting market potential during 1985 for the new service. Three Forecasts of Market Potential Three independent forecasts of market potential from three methods were available for analysis Method 1, a study of EMS-II published by OTA in August 1982 [6], involved secondary informa tion sources, expert judgement input, and curve- fitting. Method 2, a primary study (produced in early 1983) of an original formulation of the prod- uct concept, used a survey methodology. Method 3 also used a survey methodology, however, it Table 1. Three Methods of Forecasting El Conceptual Product (Concept) Definition Electronic input/ Method 2 (1983) SPRODINNOV MANAG 113 waren) involved a reformulated product concept; i.e., with different product features. The third study was conducted in late 1983. Table | includes a description of the summary operationalization of each method with respect to Kotler’s [8] eight conceptual factors defining mar- ket potential. Each method employed a some- what different operationalization of the concept of market potential for the new product. As indi cated in Table 1, the three methods generated forecasts of 3.6, 1.4, and 3.0 billion messages, respectively, that could potentially be sent in 1985 via EMS-IL. In this case, the difference of some 2.2 billion messages between the first two methods is substantial. Acceptance of one or the other of these estimates of potential could result in very different 1985 sales forecasts and subse- quently, marketing decisions. Had the three methods produced very similar forecasts, one would feel more comfortable about the true state of market potential. However, the differences Method 3 (1983) Electronic input’ Electronie input hard copy output ro minimum input ‘or output required rho, messages no. messages sent hard copy output ‘minimum input and output required no. messages point purchase intention scale hard copy output ‘minimum input no minimum output hho, messages point purchase intention scale Sales volume measure Purchase measure Customer group definition all potential organizations with organizations (not ‘customers 20+ employees limited by size) Geographic area measure U.S, US. us. Time period of measure 1985 1985 1985 ‘Marketing Environment Economic no assumptions ‘no assumptions ‘no assumptions Technological several assumptions no assumptions. ‘no assumptions Demographic zo assumptions no assumptions no assumptions Political no assumptions zo assumptions no assumptions Cultural 1g assumptions no assumptions rho assumptions Marketing Program Product features see product above see product above _see product above Price no assumptions set price/message set price/message Promotion buyer knowledge buyer knowledge buyer knowledge Distribution ‘maximum availability maximum availability maximum availability Estimate of market potential® 3.6 1a 3.0 As billions of messages.114s PROD INNOV MANAG. Tear 10-119 compel consideration of possible methodological explanations to reconcile the forecasts. Reconciling Differences in the Forecasts To reconcile the differences in forecasts, a com- prehensive review of the methodologies underly- ing each of the three estimates was conducted ‘This revealed that the different methods may have accounted for some of the differences in the estimates. An initial review of the three methods in Table 1 suggests that major differences occur on the product concept definition, the purchase measures, the customer group definition, the technological environment, and assumptions about price. Table 2 summarizes the judgemental effects of these differences on the three market potential forecasts. The values in Table 2 repre- sent the “most likely” estimates of cy percent increase or decrease in forecast potential as de- fined in equation (3) above. While “best” and “worst”? case estimates and various sensitivity analyses were performed for each estimate, Table 2 and the discussion below includes only the most likely values.' The sections below provide a more detailed comparison of the assumptions on the " Winklers [17] approach For deriving subjective probability dis tributions fr the c, estimates could also have been used fo quantify the analysts" judgements Table 2. Market Potential Adjustments Based on Conceptual Factors R. 4, THOMAS. five conceptual factors and illustrate how the forecast of market potential for each study might be adjusted to account for these method differ- ences. Product concept differences. The OTA study in Method 1 did not test potential buyer reaction to a specific product concept of EMS-II. Based on historical U.S. mailstream data and other fac- tors, OTA analysts estimated all possible mes sages that could be sent by any t cation service and selectively that were inappropriate for EMS-II. For exam- ple, estimates of electronic funds transfer and ter- minal-to-terminal (Generation II) EMS were subtracted from the total number of potential messages that could be sent. After removing those types of messages inappropriate for EMS- IL, the residual was assumed to be its market po- tential. The other two methods included specific concept descriptions of EMS-II. The EMS-IL concept in Method 2 (as launched in 1982) in- cluded a2 page maximum per message and a min- imum of 200 messages per transaction. The EMS- I concept tested in Method 3 assumed a 2 page maximum per message, but no minimum number of messages. With respect to product concept differences, it was therefore assumed that Method 3 would pro- duce a larger forecast than Method 2, since elimi- nation of the 200 message minimum would broaden the number of potential senders of the Judgmental estimates of cj Conceptual factors Method | Method 2 Method 3 Product (concept) definition 00 10" 05 Purchase measure 00 0 00 Customer group definition 0 -+.80 4.05 Technological environment 20 00 00 Price =.10 200 00 [Net change in potential estimate (p) based on Equation (2) ~.30 0 +10 Original estimate of potential (a1)® 36 14 3.0 Revised estimate of potential based on Equation (3)" 25 27 33 "Interpretation: due te product concept differences, i was judged from analysis of study dat See tent for further interpretation of these e, estimates. messages should be adjusted upward by 10% As billions of messagesFORECASTING NEW PRODUCT MARKET POTENTIAL message. It was further assumed that Method 1 would produce a larger forecast than either of the other methods, since no restrictions were placed on the input or output of the service. The concept in Method 1 was deemed closest to the ideal, and therefore required no adjustment in its forecast (ie., the value for c on the product concept factor under Method I was assumed zero, see Table 2) Relative to Method 1, the forecasts of Methods 2 and 3 required upward adjustment to compen- sate for the input/output restrictions, To accom: plish this, the analysts reviewed responses to questions in the Method 2 and Method 3 surveys and derived estimates of the effects of relaxing the output restrictions. Questions asked of re- spondents on why they would not use the service were used to estimate proportions for adjusting volume differences without the input or output restriction. The Method 2 forecast was adjusted upward by 10% (input and output restrictions) and the Method 3 forecast was adjusted upward by 5% (input restriction only). Thus the ¢ values in Table 2 were estimated to be +.10 and +.05, respectively, for Method 2 and Method 3 on the product concept adjustment Purchase measure differences. The purchase measures used in the Method 2 and Method 3 studies were four-point purchase intention scales Each was converted to a market demand factor using a purchase intent translation procedure de- scribed in Urban and Hauser [15, p. 282]. The purchase measure used in Method | involved es- timates of the number of EMS-II messages sent based on historical U.S. mailstream data. The measures used within each of the three studies were assumed to be reliable and valid. The pu chase measures between Method I and the other two methods differed; however, there was no ba- sis for assuming that the measures alone would result in different forecasts. It was difficult to ar- gue that survey-based purchase intention mea- sures may be biased upward or downward any ‘more than the measures based on historical mail- stream data. The result was assigning zeroes to the c values in Table 2 for all three methods; i.e., despite differences between measures, they were assumed to have no appreciable effects on the forecasts. Customer group differences. The methodol- ogy employed in the OTA study of Method I fo- cused on all potential customers of EMS-II mes- 41PKOD INNOV MANAG: us Iowrsttos-119 sages. On this factor it was assumed to be close to the ideal view of market potential; the c value for it in Table 2 was set to zero, The survey method- ology of Method 2 and Method 3 involved ran- dom samples of organizations. Method 2 sampled organizations with 20 or more employees and Method 3 sampled all organizations. To compen- sate for the narrow sample in Method 2, the fore- st was adjusted according to an assumed pur- chase potential factor for smaller organizations not represented in the sample. This purchase po- tential factor for smaller organizations (less than 20 employees, but some 87% of all organizations) was estimated from the Method 3 survey that in- cluded organizations with less than 20 employ- ces. The result was an upward adjustment of the Method 2 forecast by +.75, or the effect of con- sidering all U.S. organizations was estimated to be an increase of 75% in the forecast potential. The Method 2 and Method 3 forecasts repre- sented all organizations assigned an SIC code in the United States, not individuals who might use the service (e.g., private consultants). Lack of accurate data reflecting the size and purchase po- tential of this group made it difficult to estimate the effect on the forecast of individual uses. How- ever, based on judgement and rough estimates of the number of home offices in the United States the Method 2 and Method 3 forecasts were ad- justed upward by +.05; i.e., the increase in mat ket potential due to individual users was esti- mated to be 5%. Thus, in Table 2, the effects on market potential due to differences in customer group definition were estimated to be zero for Method 1, +.80 (+.75, +.05) for Method 2, and +.05 for Method 3. Technological differences. For the near term 1985 estimate of potential, it was assumed that survey responses in Methods 2 and 3 reflected the effects of technology on purchase potential. In fact, all environmental factors were assumed cap- tured in the purchase potential responses of Methods 2 and 3, and no explicit marketing envi- ronment assumptions were made. The c values in Table 2 for each of these methods were therefore set at zero. The OTA estimate of Method 1 included a se- ries of explicit “very high" assumptions about the growth of key technologies. For example, e timates on home computers, business computer data communications, and printer resolution116 J PROD INNOV MANAG: were considered. Over time, these may be the appropriate baseline assumptions for estimating potential; however, to estimate annual potential for 1985, the “very high’ technology-based as- sumptions were felt to be inconsistent with sumptions of the other two methods. Therefore, the Method I forecast was adjusted downward by 20% to reflect more plausible estimates of near term technology. Numerous sensitivity analyses based on data in the OTA study facilitated the development of this estimate. Consequently, the c value in Table 2 for technological differences on ‘Method I was set at ~.20, and zero for Methods 2 and 3. Price differences. Methods 2 and 3 contained identical assumptions about price. The OTA study in Method | included no explicit assump- tion about price. At best, price would be reflected in the historic mailstream growth patterns. Since the new product would be marketed at a set price, a downward adjustment of the Method 1 forecast was needed. This was estimated from the re- sponses to the Method 3 survey by comparing the interest in using the new product without a price associated with it to expected purchase with a price. This comparison suggested that the Method 1 forecast would be about 10% lower with a specific price attached to it. It should be noted that the launch price of the service (used in Methods 2 and 3) was thought to be a relatively good value for customers and close to the notion of “optimal” market conditions for purposes of estimating market potential. The c values in Ta- ble 2 due to the effects of price differences were therefore estimated to be —.10 for Method 1 and zero for the other methods. Combining the Three Forecasts At this point, analysts have variety of options for selecting a forecast of market potential, de- pending on their decision situation. For example, if the concern is to develop some ballpark es! mates of market potential they could retain the original independent estimates of market poten- tial as a range, i.e., 1.4 to 3.6 billion messages. Alternatively, a single best estimate based on the reconciliation analyses above could be selected. In this case it was desired to combine the fore- casts into a single figure as input to a model de- signed to forecast 1985 sales; consequently, some RJ. THOMAS: form of combining the independent forecasts was necessary. Following Makridakis and Winkler [11] and equation (1) above, a simple average of the three forecasts provided a point estimate of 2.7 billion messages market potential for 1985. If the com- parative analysis above indicated little or no basis for distinguishing among the three methods, the simple mean of the three estimates might provide a satisfactory measure of market potential. How- ever, the comparison and reconciliation of meth- ods in the preceding section suggested some meaningful explanations for differences in the es- timates. Using the judgementally quantified esti- mates of the effects of these differences summa- rized in Table 2, a weighted model of market potential as proposed in equations (2) and (3) was used to combine the three forecasts.’ As noted, Table 2 summarizes the cy values estimated to have an effect on market potential for the various criteria, Using equation (3), the p, values were computed for each method and presented in Ta- ble 2 as the net change in market potential. Using equation (2) and the obtained values of p,, the combined weighted forecast of market potential would be some 2.8 billion messages for 1985. Discussion Multiple methods to forecast new product market potential raise some key issues in using the ap- proach. These include the difficulty in assessing “accuracy” of forecasts, the kind and number of methods represented in the analysis, the model used to combine forecasts, the use of judgement in combining forecasts, and the cost-benefit as- pects of implementing such an approach at differ- ent stages of new product development. Accuracy of Market Potential Forecasts For any kind of forecasting, accuracy is an impor- tant criterion in evaluating the method used to generate the forecast. For example, is the com- * IC should be noted thatthe three forecasts covered a time pe- riod of almost 2 years, and that events during this time period could explain changes in the estimates. However, it was believed that clecironic mail services during 1982—I9R3 were sufficiently new to Support the assumption of no differences due to changes in buyer intentions or other market factors,FORECASTING NEW PRODUCT MARKET POTENTIAL bined estimate of market potential of EMS-II in 1985 of 2.8 billion messages more or less accurate than any of the individual estimates, or the mean of all estimates (2.7 billion messages)? The an- swer, and the key difficulty in assessing accuracy for new product market potential, is that for im- mediate planning purposes true market potential for EMS-II in 1985 won't be known. ‘The concept of accuracy in forecasting relies on knowing the true state of market potential af- ter the forecast period. This would allow a com: parison of the estimate against actual to measure accuracy. While this is possible in sales forecas ing efforts, it may take several years of market development and historical analysis before any indication of true market potential in a given year can be known. Thus, accuracy may be inappro- priate as a criterion for evaluating the quality of new product market potential forecasts and fore- casting models. Consequently, the important cri- terion may be the quality of the conceptual de- finition of market potential and multiple operationalizations of it; i.e., a carefully defined concept and the operationalization of multiple methods producing multiple measures may be the best guide for developing forecasts of market po- tential. Number and Kinds of Methods To Use The second issue of concern is the number and kind of methods to include in a multiple methods approach to estimate market potential. While the minimum number is two methods, there is some evidence from Makridakis and Winkler [11] in a sales forecasting context with historical data which suggests that gains in MAPE accuracy af- ter five to six methods begins to decline rapidly. Ashton and Ashton [2] found satisfactory accu- racy levels in combining two and three judgemen- tal forecasts. These studies suggest two to six forecasts would be sufficient to produce satisfac- tory results; however, guidelines are still needed from future research to determine situations when two are better than six, and vice versa. The number of methods also depends on the kinds of methods. In an analysis of three forecast models, Bopp [3] found that a two-model combi- nation which excluded the best single model was better than any three-model combined forecast. Sewall [13] found that expert judgement and a PROD INNOV MANAG 17 survey of buyer intentions each provided signifi- cant incremental information for purposes of pre- dicting new mail order catalog product sales. The different methods provided about equal predic tive validity. Both studies suggest that benefits will be derived from incorporating different kinds of methodologies to produce combined forec: The Model Used 10 Combine Forecasts ‘The third issue involves the model to combine forecasts. After several forecasts are obtained, a natural tendency is to limit the range to the low and high forecasts from the available studies. The obvious problem with this is that “true” market potential may be higher than the highest estimate and lower than the lowest estimate. This problem is implicit in the simple averaging approach of combining forecasts of market potential. Simple averaging in new product situations does not en- courage the consideration of forecasts outside the range of available measures. ‘The more system- atic approach of weighting the forecasts by the conceptual factors defining market potential could lead to forecasts outside the range of the individual methods used. In the above case study, the difference between the simple mean of 2.7 billion and the weighted estimate of 2.8 billion messages was not substantial; however, with the weighted approach, the forecast could have been outside the 1.4-3.6 billion message range. By foc- using on the conceptual factors defining market potential, a broader range than generated by the available forecasts is possible. Use of Concept Definition to Guide Judgement in Combining Forecasts The value of subjective estimation for quantifying judgements is well supported in the literature on. model building [9, 12]. The general approach for eliciting managerial judgements is to use a deci- sion maker's experience to systematically quan- tify estimates of point values, sales response functions, and probabilities. Unfortunately, ex- perience in specific new product situations may not be available and the resulting direct estimates of market potential may contain considerable variability To improve on direct estimates, Armstrong [1] suggests a decompositional approach that breaksTIS pRoD INNOV MANAG: Ioxrsst-t19 down a complex problem into simpler prediction problems that are more relevant to the decision maker. Judgements quantified from these simpler problems can be recombined to predict an overall outcome. The approach described in this paper breaks down market potential into eight major factors. Quantified judgements about the effects of these factors on market potential are assumed to be more systematic and tractable than direct judgements about market potential. While re search and development on other procedures to elicit decompositional judgements and recombine them is required, the approach presented above illustrates guidelines for a decompositional meth odology for which judgement can be employed to combine forecasts. Cost-Benefit of Multiple Methods at Different Stages of Development Unfortunately, market potential forecasts early in a product’s development are often viewed as low cost efforts based on some paper and pencil com- putations or taken from a published study pur- chased from a commercial research firm. As the product concept becomes more carefully de- signed at later stages of the development process, more expensive customized studies are often commissioned to assess the new product's mar- ket and sales potential. Taken together, research expenditures on a new product clearly become expensive. Pressures to reduce or allocate overall research costs may result in the use of inexpen sive market potential estimation approaches dur- ing the early stages of product development—just when reasonably good estimates are needed to make good go/no-go decisions. For market potential forecasts and go/no-go decisions, it is therefore argued that greater effort should be expended earlier in the product's de- velopment. Market potential provides an impor- tant indication of the overall market opportunity, and therefore reasonably good forecasts should be available to make early go/no-go decisions. If'a firm decides to allocate relatively little on market research during early development, then a multi- ple methods approach may prove to be cost effec- tive. For example, low cost secondary studies, expert judgement, and purchased commercial studies can be incorporated into the proposed ap- proach. Reliance on any one study may be risky; RJ THOMAS. however, the rigorous use of the conceptual fac- tors to define market potential will help to care- fully qualify each method's estimate and develop a systematically produced combined forecast ‘This may avoid the pitfall of implicitly assuming a “go”” decision or making other erroneous deci- sions about the new product idea, While more specific cost-benefit analyses are nei each situation, the effort to implement a multiple methods approach as described would seem mer- ited early in a product’s development to reduce the chance of over- or underestimating the size of a market opportunity. Conclusions Forecasting market potential for a new product can be a complex task due to the many factors that might drive market demand. To manage this task it is recommended that a multiple methods approach be implemented. This approach can be summarized in the following three steps: 1. Define a concept of market potential for the particular product/market in question using conceptual criteria identified in Table 1 2. Obtain two to six forecasts of market potential utilizing as many different and independent methods as possible that attempt to measure the concept of market potential in question. If only two methods are used, they should be different (e.g., expert judgement and survey of buyer intentions) 3. Evaluate the different methods with respect to the criteria defined and decide whether to use one method's results, a range of results, or results from combining the methods. If the lat- ter, operationalize the variables in equations (2) and (3) as described above to obtain the weighted forecast. If, in undertaking a multiple methods approach, the opportunity arises to conduct a primary study to estimate new product market potential, then it would be desirable to evaluate the other available methods with respect to the eight conceptual fa tors prior to the study design, Not only would this improve the study design, but questions could be incorporated that might provide better data to support the judgemental adjustments to demand. For example, commercial studies often provide market size estimates without consideration ofFORECASTING NEW PRODUCT MARKET POTENTIAL market price. Including a question that measures “interest” in the new concept without regard to price would provide guidelines to improve judge- ment in reconciling and combining the forecasts. Otherwise, analysts’ experience and interpreta- ns of methods and data available from pub- hed studies remain the predominant basis for quantifying judgement. While the proposed approach to market poten- tial forecasting stems from a behavioral science tradition and has logical benefits, additional re- search is necessary to better understand and trust, the approach. For example, alternative forms of the composite model in equations (2) and (3) may need to be developed and evaluated. A series of multiplicative zero to one index variables may provide an alternative formulation. Rudimentary decision support models of industry effects of key marketing variables on market potential may pro- vide valuable input, particularly in evaluating the different methods. Additional research on the kind and number of forecasts of market potential to be combined is also needed. Further, will findings on the kind and number of methods from historical sales fore- casting studies be applicable to new product fore- casting? Finally, who should provide the judge- ment to reconcile and combine forecasts—inde- pendent analysts who are expert on methods or involved product managers who may be expert on the product class, or both? How many ana lysts or managers should be involved, and how should their judgements be aggregated or weighted? Is a Delphi approach appropriate for this part of the analysis? Despite the numerous issues that can be stud- ied further, the proposed approach provides an implementation of the often cited recommenda- tion of using multiple methods. It represents a JPROD INNOV MANAG 119 Tow t 19 disciplined and systematic way of coping with the difficult task of forecasting market potential for new products. References 1. Armstrong, J. 8. Long-Range For 1985 2, Ashton, A. H. and Ashton, R. H. Aggregating subjective fore fasts: some empirical resulls. Management Science 31:1499- 1308 (December 1985), 3. Bopp, A. E. On combining forecasts: some extensions and reals, Management Science 31:1492~1498 (December 198). 4. Campbell, D.T, and Fiske, D. W. Convengent and diseriminant validation by the multitrai-multimethod matrix, Psychological Bulletin 56:81-105 (1989) 5. Denzin, N. K. The Research Act, 2nd ed. New York: MeGrave Hill, 1978 asting. New York: Wiley 6. Implications of Electronic Mail and Message Systems for the U.S. Posial Service. Washington, D.C.: Ofhee of Technology Assessment, 1982, 7, Jick, T. D, Mixing quantitative and qualitative methods: aa lulation in Action. Adiinisiraixe Science Quarterly 4:60? {611 (December 1979), 8. Kotler, P. Marketing Management, Sth ed, Englewood Cis, Ni: Prentice-Hall, 1984 9, Lilien, G. L. and Kotler, B. Marketing Decision Moking. New York: Harper & Row, 1983 10, Makridakis,S. etl. The accuracy of extrapolation (time serie methxls: results of a forecasting competition, Journal af Fore: fasting 1:11 153 (1982) 1, Makridakis, S. and Winkler, R. L. Averages of forecasts: some empirical resuits, Management Science 29:987-996 (September 183), 12, Naert, P. and Lesflang, P. Building Implementable Marketing Models. Boston: Martinus Nijholf Social Sciences. Division, 1978, 13, Sewall, M. A. Relative information contributions of consumer purchase intentions and management judgement as explanators (Of sales. Journal of Marketing Research 18:249-253 (May 1981). 14, Smith, 1. W. Sratesies of Social Research: The Methodologt eal Imagination. Englewood Clif, Ni: Prentice Hall, 197. 15, Urban, G. L. and Hauser. J. R. Design and Marketing of New Products, Englewood Cliffs, Ni: Prentice-Hall, 1980, 16, Wind, ¥. Product Policy. Reading, MA: Addison-Wesley. 1982, 17. Winkler, RL. The consensus of subjective probability distrib tions. Management Science 1S:R61-B75 (168).