19487169a, 2000, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/j.1948-7169.2000.tb00152.x by Chalmers University Of Technology, Wiley Online Library on [23/10/2022].

See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
T H E D E S I G N
M A N A G E M E N T
I N S T I T U T E

D ESIGN
M ANAGEMENT
J OURNAL
Article Reprint

Performance Metrics to
Measure the Value of Design
Joseph J. Paul, Research Associate, Eastman Kodak Co.

This article was first published in Design Management Journal Vol. 11, No. 4
Merging Design and Business Strategies
Copyright © Fall 2000 by the Design Management Institute. All rights reserved.
No part of this publication may be reproduced in any form without written permission.
To place an order or receive photocopy permission –
(617) 338-6380 x223 Tel • (617) 338-6570 FAX • E-mail: dmistaff@dmi.org

Reprint #00114PAU71
 19487169a, 2000, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/j.1948-7169.2000.tb00152.x by Chalmers University Of Technology, Wiley Online Library on [23/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
D ESIGN
M ANAGEMENT
JOURNAL
VOL. 11, NO. 4 FALL 2000

EDITOR 'S NOTES

Defining the Connections Between Design and Business 00114WAL06
Thomas Walton, Editor; Professor, School of Architecture and Planning, The Catholic University

KEYNOTE ARTICLE
Managing Design for Competitive Advantage: A Process Approach 00114OLS10
Eric M. Olson, Associate Professor, University of Colorado at Colorado Springs; Stanley Slater,
Professor of Business Administration, University of Washington, Bothell; Rachel Cooper, Professor of
Design Management, University of Salford, UK

MARKETING
Strategic Branding: Bringing the Customer Closer 00114BOU18
Marie-Claude Peyrache, Group Vice President of Corporate Communications, France Télécom; Jean-
Léon Bouchenoire, Director of Brand Equity, Compaq Computer Corporation

CASE STUDY
The E-commerce Blueprint: Creating Online Brand Experiences 00114NOR25
Dave Norton, Vice President of Research, Yamamoto Moss; Lise Hansen, Director of Information
Architecture, Yamamoto Moss

STRATEGY
Corporate Strategy: Bringing Design Management into the Fold 00114JOZ36
Frans Joziasse, Founding Partner, PARK Design Strategists

Design and Business: Who Calls the Shots? 00114TUR42

Raymond Turner, Group Design Director, BAA

Barriers between Design and Business Strategy 00114FIL48

Anna Filson, Research Officer, National Centre for Product Design and Development Research,
University of Wales Institute, Cardiff (UWIC); Alan Lewis, Director of Research Enterprise,
National Centre for Product Design and Development, University of Wales Institute, Cardiff
(UWIC)

DEVELOPMENT
Creating and Managing Brand Experience on the Internet 00114SCH53
Bernd Schmitt, Professor of Business, Director, Center for Global Brand Leadership, Columbia
Business School

PRODUCTION
Creative Product Analysis to Foster Innovation 00114BES59
Susan P. Besemer, Principal and Founder, Ideafusion

MARKETING
Establishing Strategic Objectives: Measurement and Testing in Product Quality 00114NOE65
and Design
Noel Mark Noël, Associate Professor of Marketing and Business Policy, University of South Florida

RESEARCH
Performance Metrics to Measure the Value of Design 00114PAU71
Joseph J. Paul, Research Associate, Eastman Kodak Co.
 19487169a, 2000, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/j.1948-7169.2000.tb00152.x by Chalmers University Of Technology, Wiley Online Library on [23/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
SUPPORT

Performance Metrics
to Measure the
Value of Design
A METHODOLOGY that would rank
consumer responses to both product
attributes and alternative designs would help rationalize the design
process and make it a more reliable business resource. Based on two
studies—one involving the design of single-use cameras and the other
investigating the design of Advanced Photo System cameras—Joseph Paul
summarizes an approach he is refining to predict the link between design
and purchase behavior.

By Joseph J. Paul
In May 1997, Eastman Kodak began a formal
effort to measure the value of design with
the creation of a new position within the
Corporate Design Center. The mission was
to provide the industrial and graphic design
groups with information about market
JOSEPH J. PAUL IS A structure (size, distribution, share, and so on)
RESEARCH ASSOCIATE and market function (target group behavior,
AT THE EASTMAN intentions, perceptions, motivations, and so
KODAK CO. on). The primary objective was to identify the
performance metrics—the design factors that
attract buyers to a product and that buyers
employ in appraising their experience with
the product. My new job was the result of a
career-long interest in increasing the breadth
and depth of knowledge about the markets
in which manufacturers compete and my
experience in applying newly developed
research techniques to marketing problems.
Kodak’s aim was to more fully engage
design as a competitive advantage and
bring it into the earliest stages of product
development. The vision was elegant in its
simplicity—bring the required skill set of
one discipline (research—specifically,
psychometrics) to address the needs of
another discipline (design).
My investigation into performance
metrics has resulted in a body of knowledge
synthesized from the literature; conversations
with accomplished design managers in busi-
ness, academe, and professional associations;
and what I learned from the two research
studies I describe in this article. From this
body of knowledge, I have constructed a
conceptual model, or paradigm, the founda-
tion theory for a research system to measure
the value of design in a valid, reliable, and
predictive manner. Employing the Perfor-
mance Metrics Research System, a company
can build the required information base from
which to measure the value of design for a
single product or a series of products.
The Pilot Study
After a year of literature search, benchmarking
with manufacturing firms, and discussions
with academics and practitioners in industrial

DESIGN MANAGEMENT JOURNAL FALL 2000 71


MERGING DESIGN AND BUSINESS STRATEGIES
and graphics design, marketing, manage-
ment, R&D, and product development
groups, our team devised a pilot study to
determine whether “functional utility” and
“emotional involvement” were engaged in
attracting buyers to particular products and in
their appraisal of their experience with those
products, and if so, to what degree they have
an impact. However, for the product we were
targeting—the Kodak Single-Use Camera—
traditional research methods (that is, direct
questions and assorted rating scales) did not
hold out a great deal of promise for searching
out the criteria buyers were likely to use in
appraising their experience with the product.
Direct questions and rating scales are based
on the frame of reference of the person(s)
asking the questions. I wanted to understand
and make explicit the frame of reference of
the respondents.
We decided to conduct an experiment
with a relatively new research methodology,
referred to in the research trade as language
analysis and in academe as semiotics—the sci-
ence of the meaning of (particularly) visual
and verbal symbols. We were helped in the
research effort by Communication Develop-
ment Co., of West Des Moines, Iowa. That
company conducted multiple computer
analyses as we continued to learn about the
user/camera interaction and the relationship
between performance metrics and the
The Value of Design
What is the best way to discover the value of design in a product or service? And
why has this question never been answered to the satisfaction of the design commu-
nity? After three years of collecting information, I have made a series of observations
that may account for this circumstance.
• Arrogance—“I know what good design is.” (“I don’t need your stinking
metrics!”)
• Buyer Theory (or lack thereof)—There is no cohesive view of how buyers/users
relate to products, which is really what forms the foundation of the purchase
decision.
• Company Organization—The position of design within the organizational
structure tempers the perception of the need for having and using performance
metrics.
• Definitions—The fact that beauty and art are involved in the aesthetics of design
is advanced as the reason the value of design cannot be defined or measured.
• Expectations that may be characterized as the quick fix mental mode: that is,
“Take a pill, cure your ill.” This applies to those who expect that some small
number of metrics can be universally applied across product categories. They
express irritation toward and reject out of hand the idea that they might have to
work at the problem.
• Formidable Complexity—Resulting from widely diverse views of what design is
and/or should be; how buyers understand and relate to products; what research
can provide; what management is capable of understanding and accepting.
• Gross Knowledge Void—“Measurement,” as a discipline, falls outside the
training and experience of design practitioners.
72 DESIGN MANAGEMENT JOURNAL FALL 2000
19487169a, 2000, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/j.1948-7169.2000.tb00152.x by Chalmers University Of Technology, Wiley Online Library on [23/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
conceptual model we were building.
LANGUAGE ANALYSIS
An essential component of our language
analysis were the “VOCAL” interviews.1
An acronym for Voice of the Consumer
Analyzed Linguistically, these are adminis-
tered in a face-to-face situation and can be
recorded to generate input for computer
analyses of buyers’ appraisals and experiences
with the product. They are personal,
individual interviews based on a nondirective
interviewing technique that investigates
generalizations, deletions, distortions, and
(apparent) contradictions on the part of the
respondent. Interviewers are trained in a
standard psychiatric method2 that teaches
them to go beyond the surface structure of
response by eliciting the deep structure and
meaning from an individual’s language usage.
For example: A husband comes home
from work and greets his wife:
Husband: “How did your day go?”
Wife: “I’ve been slaving all day over a hot stove
preparing your dinner.”
Husband: “Yeah, sure.”
In this example, the wife’s statement is an
exaggeration with the possible underlying
meaning of discontent. The husband’s
“Yeah, sure” is a double positive that is an
outwardly contradictory statement with an
underlying meaning of nonbelief.
By actual measurement, about 80 percent
of everyday language is expressed in gener-
alizations. But generalizations are not helpful
when one is attempting to understand
motivations underlying behavior. VOCAL
interviews and the computer analysis used to
study them can tease out their true meaning.
The tape-recorded interviews are then
transcribed into computer-readable form.
This “contextual analysis” is done through
multiple computer runs, which are
ultimately synthesized into a final analysis
resulting in study findings and conclusions.
The team used three types of computer runs:
1. Semantics analysis. This form of analysis
is used to determine how respondents
construct combinations of words to
express ideas about the stimulus.
Computer retrieval determines which
1. VOCAL® is a CDC trademark.
2. I.e., Richard Bandler and John Grinder, The
Structure of Magic: A Book About Language and Therapy
(New York: Science and Behavior Books, 1975).

words and ideas co-occur in response to
that stimulus.
2. Syntactic analysis. This analysis provides the
order and grammar of the ideas that come
from respondents thinking about the
stimulus. It identifies the degree to which
respondents are intellectually interested
in, involved in, or behaviorally motivated
by the stimulus. This is done by algo-
rithms that analyze or parse the subject,
verb, and direct object of responses using
probabilistic motivation models.
3. Pragmatic analysis. This analysis is defined
as a study of the context of matrix (that is,
the language response as a whole) in
which the stimulus event occurs. The
computer does a contextual analysis to
determine the kind of situation within
which the respondent places the product
service or idea.
INTERVIEW OBJECTIVES
The goal of the interviews was to obtain a
comprehensive linguistic description of the
respondents’ experience with the product
under “normal” conditions. Recruiting took
place at a Wal-Mart store. Shoppers who put
the designated camera into their shopping
cart were intercepted and asked to participate
in an interview. That first interview took
place in an interviewing facility directly out-
side the store. The second interview took
place when respondents brought their
single-use cameras back to have the pictures
processed. Duration between first and
second interview was typically three to
four weeks.
Here’s what was gained from the
interviews:
• Twenty-one precise metrics, obtained
from the language analysis process,
for measuring the usability design
component of single-use cameras
• The relative importance of specific
product attributes
• The benefits that emanated from specific
product attributes
• The motivating factor in this purchase
MODES OF INVOLVEMENT
The interviews also gave us insight into the
language (words and ideas) associated with
the respondents’ modes of involvement. These
included:
• Practical involvement, or functional utility: the
degree to which the product fulfills its
19487169a, 2000, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/j.1948-7169.2000.tb00152.x by Chalmers University Of Technology, Wiley Online Library on [23/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
PERFORMANCE METRICS TO MEASURE THE VALUE OF DESIGN
intended purpose
• Intellectual involvement: the degree to which
the product stimulates curiosity, or holds
potential for developing knowledge or skill
• Emotional involvement: the degree to which
the product enhances the ego, self-
concept, or self-ideal
• Social involvement: the degree to which the
product represents affinity to a political,
social, economic, religious, or intellectual
group
The team then developed “perceptual maps”
(figures 1-4) that displayed the buyers’ level
of involvement with the product for each of
the four modes.
Next, our respondents’ language and
thought processes were identified by the
methods described
above. VOCAL
interviews and con-
Kodak’s aim was to more
textual analysis were
also used to identify
fully engage design as a
the camera’s value
proposition, expressed
competitive advantage and
as the sum total of
the tangible and
bring it into the earliest stages
intangible values of
the product over
of product development
its cost.
RESULTS
Overall, we felt, low-cost, low-involvement
products like single-use cameras were less
than optimal stimuli for demonstrating and
testing the principles of the conceptual
model we were developing. They were
relatively cheap at $12, and they were only
used for about a month before being turned
in for picture processing. Hence, there was
no long-term user/product relationship or
commitment. Indeed, the perceptual maps
clearly showed low levels across all four
Modes of Involvement. In retrospect, this
finding is consistent with what now seems
should have been an obvious expectation for
this product category: Why and how could
anyone have a high level of involvement
with any product that is low-cost and must
be disposed of to fulfill its use application?
We did, however, discover that purchasers
of single-use cameras tend to repurchase the
product, and therein lies a long-term rela-
tionship. Our interviews allowed us to probe
respondents’ reasons for repurchasing the
product, and this helped us obtain a new,
deeper, and more meaningful understanding
DESIGN MANAGEMENT JOURNAL FALL 2000 73
 19487169a, 2000, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/j.1948-7169.2000.tb00152.x by Chalmers University Of Technology, Wiley Online Library on [23/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
MERGING DESIGN AND BUSINESS STRATEGIES

Modes of Involvement of the target user group. As for uncovering

These maps portray buyers’ levels of involvement with single-
use cameras for each of the four Modes of Involvement. The
the “value of design” for this product, we felt
probability of occurrence relative to the use of those terms in this would have been more successful had
the popular or common parlance are shown by the numbers there been similar products competing
on the x and y axes. The range of involvement and the ideas against it.
associated with them as indicated by a “Low,” “High,” and The reality of the quantified documenta-
“Centroid,” or center, of the distribution of response.
tion now brings an appropriate set of
expectations for business management and
Levels of Practical and Intellectual Involvement
(Practical Involvement Held Constant) places them in perspective: Single-use
cameras elicit low levels of involvement and
Practical Involvement—
Functional Utility are used for the most part by people who have
0.4 a low level of involvement in photography.
These findings prompted a return to
0.3
Communication Development Co.’s
0.2 language database to identify the language
Low High
0.1
"Acceptance "Suggestions used to express the psychological processes
of Camera" Centroid for Involvement"

0.0
-0.4 -0.3
• -0.2
•
-0.1 0.0 0.1
• 0.2 0.3 0.4
associated with emotional involvement. It
was anticipated that this language could
Intellectual Involvement form the foundation for the measures of
emotional involvement described in table 2.
Levels of Practical and Emotional Involvement
(Practical Involvement Held Constant) Study Number Two: Pre-Test of
Practical Involvement— Appearance Design
Functional Utility In my previous experience, studies of
0.4
product appearance have essentially been
0.3 little more than tactical beauty contests,
sometimes referred to as “disaster checks.”
0.2
Management has been content to “go with
Low
0.1
High
the product” as long as no serious problems
•
"Evaluative" Centroid "Involvement"

0.0
• • were indicated. This approach may give faint
-0.4 -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3 0.4 assurance in the short term, but it doesn’t
Emotional Involvement
lend itself to product leadership, marketing
excellence, or world-class operations.
Levels of Practical and Emotional Involvement A commonly applied method for
(Emotional Involvement Held Constant)

Practical Involvement—
conducting research on appearance design has
Functional Utility been to ask (typically within the context of a
0.4
group interview) which design alternatives
0.3 respondents “like,” “prefer,” or “would buy,”
0.2 • High
"Price Issues" and then ask why. Rigorous examination of
these responses tends to find any cogent or
0.1

0.0
-0.4 -0.3 -0.2 -0.1 0.0
••0.1
Centroid
Low "Concentration on Convenience"
0.2 0.3 0.4
consistent explanation lacking, especially
across studies. Responses tend to be little
Emotional Involvement more than surface playback of product fea-
tures. As the language has evolved, the word
Levels of Practical and Social Involvement “like” is particularly problematic as a criterion
(Practical Involvement Held Constant) for appearance design. In US English, “like”
Practical Involvement— is the 51st most commonly used word, and it
Functional Utility is now devoid of a behavioral referent—that
0.4
is, a specific behavior consistently associated
0.3 with a verbal expression.
0.2 OBJECTIVE
Low High
"Lack of "Fulfillment
0.1 The thrust of this effort was to determine
•
Understanding" Centroid of Expectations"

0.0
-0.4 -0.3
• -0.2 -0.1 0.0
•
0.1 0.2 0.3 0.4
whether a quantitative research approach
worked well for early testing of design issues
Social Involvement—Product Expectations related to the “appearance” component of

74 DESIGN MANAGEMENT JOURNAL FALL 2000


Table 2
Measures of Emotional Involvement
Psychological Measure (for Application to Scales)
Process
Internal/Core Self This product “is like me/is not at all like me.”
Personal Identity This product gives/does not give meaning to what I
want to become.
External Self I would like to be seen using/would be embar-
rassed to be seen using this product.
Joy of Self This product is great fun to use/no fun to use. This
product makes my life better/worse.
Identification This product does/does not give meaning to what
I do.
Mood I am at ease/not at ease with this product.
appearance and usability design. The plan
called for applying the measures of emo-
tional involvement derived from our pilot
study. The sample recruited for the study
consisted of active picture-takers intending
to buy a camera in the $50 to $150 range
over the next six months. Again, we used
personal, individual interviews. Respondents
viewed and handled, in a systematically ro-
tated procedure, a set of seven competitive
Advanced Photo System cameras. These
cameras all fell into what the Kodak team
judged a “narrow” design range (that is, the
cameras were similar in appearance), which
judgment was confirmed by the study.
Criterion measures (applied as 7-point
scales, in which positive statements were
assigned a value of 7 and negative statements
were assigned a value of 1), included:
• Appearance—very appealing/not at all
appealing (aesthetics)
• Great fun to use/no fun to use
• Fit and finish—very good/not very good
• Product—is me/is not me
• Would like to be seen using/Would be
embarrassed to be seen using
• Innovative/ordinary
RESULTS
The principal discoveries obtained from this
study were:
• The criterion measures generated a 0.75
correlation with interest in making the
purchase.
• Clear distinction was obtained in
the “narrow” design range—that is,
respondents were more consistent in
their judgments than the designers.
• The explanations advanced by respon-
dents for their choices were consistent
across the design alternatives—that is,
19487169a, 2000, 4, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/j.1948-7169.2000.tb00152.x by Chalmers University Of Technology, Wiley Online Library on [23/10/2022]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
PERFORMANCE METRICS TO MEASURE THE VALUE OF DESIGN
similar appealing design features were
cited across camera models.
• The “best” design received the highest
ratings across the criterion measures.
This was determined by a funnel
analysis (figure below), which counts
the number/percent of people giving a
6 or 7 rating across all criteria.
IMPLICATIONS
In the year following this second study, sales
data revealed that cameras scoring high on
our measures of emotional involvement had
unit sales in excess of 36 to 1 over the camera
with the lowest “emotional” score.
Net Summary
The findings from the Pilot Study
demonstrated the viability of semiotics as a
technique to measure buyers’ involvement
in a product both at the time of purchase
and following use experience. The findings
from the second study demonstrated that
measures of Emotional Involvement
administered in a quantitative study work
as effective (and efficient) predictors of
product performance in the marketplace.
Both efforts supported the theory upon
which the conceptual model is based.
Next Steps
A continuing effort will be made among
Design Management Institute participants to
employ a research system—the Performance
Metrics Research System—to measure the
value of design within their respective orga-
nizations. For more information, please
contact Earl Powell, president of DMI, at
epowell@dmi.org. ◆
(Reprint #00114PAU75)
Funnel Score-Criterion Measures: APS Cameras
Camera
A 54
B 25
C 24
D 20
E 20
F 20
G 14
%
0 20 40 60 80
% of respondents giving a 6 or 7 rating (7 point scale) on all Criterion Measures
DESIGN MANAGEMENT JOURNAL FALL 2000 75