Embed Doc
Copy Link
Readcast
Collections

CommentGo Back

Download

What Constitutes Good Design? A Reviewof Empirical Studies of Design Processes*

MATTHEW M. MEHALIK and CHRISTIAN SCHUNN

Learning Research and Development Center, University of Pittsburgh, 3939 O'Hara Street, Pittsburgh,PA 15260 USA. Email: mmehalik@pitt.edu and schunn@pitt.edu

The authors conducted a meta-analysis in which they sampled over 40 journal articles of empirical studies of the design process. The articles were selected from several journals that are indexed inPsychInfo. They were reviewed and indexed according to whether or not those articles mentioned and/or focused one or several of 15 common design process elements. These elements consist of abootstrapped framework that the authors constructed from past experience in studying the design process and from validation from literature. The coding was conducted by an additional expert indesign, and the results were shown to have reliability greater than 80%. The results show whichelements of the design process have been studied most frequently, and of those studied, which weredocumented to contribute to an effective design outcome. Both the framework and the coding outcomes contribute to a stronger understanding of the overall design process that relies uponexisting, more fine-grained empirical studies of design.

Keywords:

design processes; meta-analysis; design outcomes.

BACKGROUND

WHAT CONSTITUTES good design? For thisarticle we surveyed and reviewed over 40 journal-published, empirical studies and case studies of design in order to derive a common set of reporteddesign process elements. Further, we explored thissame literature from the perspective of whichelements are reported to be associated with good(effective) design practice. The intentions of thiseffort are to offer a common set of effective designprocess steps and to focus attention onto areas of design that require additional exploration or inves-tigation. The point of identifying the designprocess elements in this paper is to take a meta-perspective on empirical approaches that havelooked at design and have documented observable,describable design activities.What are some reasons for taking such a meta-perspective to understanding good design? First,there are a number of published empirical studieson various parts of the design process. Suchempirical approaches typically focus in-depth ona particular aspect of design and attempt to situatethat aspect in a larger design context. Studies of this type lend themselves to providing quality dataabout these particular aspects. The value of thesestudies can be extended if their findings can beassembled into a broad, integrated perspective.The end result of this exercise ought to be acommon set of effective design stages that canbe useful for practitioners of design and fororganizations whose practices rely upon effectivedesign processes. Such is the case especially fororganizations that specialize in innovative design,for which new contexts, markets, and productsemerge and shape worldwide practice.Second, an integrated, meta-perspective ondesign based on previous empirical studies canfocus attention onto understudied areas in thedesign process. Not all aspects of the designprocess have been studied or documented empiri-cally at equal degrees of depth or breadth. As itmay turn out, the landscape of empirical studies of design can point to areas in which further explora-tion can generate more understanding of effectivedesign practice.Third, a more integrated, meta-perspective ondesign can point to ways for which design can beused as an effective vehicle for learning. Design-based learning has been explored as a way toengage students to enhance their abilities to solvereal-life problems and to reflect on their learningprocesses. This style of active learning is an exten-sion of project-based learning, which is argued toenable students to relate problems to scienceconcepts [1, 2]. Design-based learning differsfrom project-based learning in that, in additionto constructing and building, students engage in adesign and planning process that follows engineer-ing design. The ability for learning scienceresearchers to anchor their evaluations of design-based projects can be informed by having a vali-dated framework for design that considers multi-ple, integrated aspects of the overall designprocess. Such a study can also permit learningscience evaluations of design to consider howdifferent learning styles [3] can be engaged atvarious stages of the design process from anoverall design process perspective.

* Accepted 22 November 2005.

519

Int. J. Engng Ed.

Vol. 22, No. 3, pp. 519±532, 2006 0949-149X/91 $3.00+0.00Printed in Great Britain.

2006 TEMPUS Publications.

Finally, in terms of engineering education,design has become explicitly recognized as animportant outcome of what undergraduate engi-neers need to experience and to achieve [4]. Speci-fically, Criterion 3 of accreditation standardsstates that their program must demonstrate thattheir graduates must have:b) an ability to design and conduct experiments, aswell as to analyze and interpret data;c) an ability to design a system, component, orprocess to meet desired needs [4].Inaddition,Criterion4specifiesthatanengineeringstudent's professional development must include amajor design experience that integrates much of their earlier skills, coursework, and standards:b) One and one-half years of engineering topics,consisting of engineering sciences and engineer-ing design appropriate to the student's field of study. . . . Engineering design is the process of devising a system, component, or process tomeet desired needs. It is a decision-makingprocess (often iterative), in which the basicsciences, mathematics, and the engineeringsciences are applied to convert resources opti-mally to meet these stated needs [4].An examination of previous empirical designaspects in order to identify which contribute togood design can serve as a way for engineeringprofessions to re-examine design education inorder to emphasize the aspects of design thatlead to effective design.A meta-analysis of a design process can alsohighlight in greater specificity what has beendocumented to show the uniqueness of the typeof thinking that is associated with design. HerbSimon emphasized that design mainly involvesthinking about the artificial in a process involvingmostly synthesis, whereas other types of thinking,such as scientific inquiry, focus more on analyticalthinking. Design is also a highly goal-directedenterprise in the pursuit of a normative outcome.What designers produce is the result of makingclear an outcome that embodies some attributesthat can be specified but also have not yet beenachieved. Design places more emphasis on thefunctional nature of what is being produced. Italso involves the functional decomposition of anoverall design outcome, so that these functionalaspects can be examined separately and laterintegrated into a functional whole. Finally,design involves a large degree of symbolic manip-ulation and representation in order to explorecomplex relationships among various aspects of aparticular design [5].

FRAMEWORKS ON DESIGN

One place to start a process of constructingevaluating a set of design elements involves look-ing at some other frameworks that discuss thedesign process. Simon [5] proposed what know-ledge needs to be taught so that a true science of design can emerge:1. Frameworks for rational choice among givenalternatives (e.g. Utility theory, decision analy-sis, statistical modeling)2. Procedures for optimizing among alternatives(e.g. linear programming)3. Effective methods for searching space of solu-tions for `satisficing' goals and constraints4. Effective, or satisfactory, allocation of resources for searching among alternatives,including partial paths of searches [5].In addition, Simon advocated exploring alternativerepresentations of design problems. The authorshave considered these elements of design whenthey began compiling a list of common designelements.Techniques and frameworks from systemsanalysis and design also provide some past insightsinto key stages in designing large-scale systems.For example, Gibson [6] wrote that the process of systems analysis and design consists of seven stagesthat rely upon interaction and iteration amongthese stages:1. Determine the goals of the system2. Construct meaningful indices of performance3. Consider alternate candidates for solutions4. Rank alternatives5. Validate solutions6. Iterate7. Take action [6].These two frameworks reflect some of the disci-plinary origins of the authors, who have pooledtheir expertise in the areas of studying the cognitivescience of design and systems analysis and engin-eering in order to begin a bootstrapping processfor identifying and validating a common set of design elements.The order in which the design process elementsidentified in this paper are presented should not betaken as a descriptive or normative representationof the way in which the design process proceeds inpractice or as observed in an educational tasksetting. As Mawson [7] argues, sequential modelsof the design process tend to be misrepresentationsof the way practitioners actually do design, andthey do not offer a useful structure for the way thatpupils learn design; instead, such models, whichtend to be oversimplified, linear representations of design, serve as a useful `administrative' functionfor teachers who may lack expertise in designactivities. Portraying and teaching design asfollowing a linear, or algorithmic structure there-fore serves the wrong constituency in education.The point of identifying the design processelements in this paper is to take a meta-perspectiveon empirical approaches that have looked atdesign and have documented observable, describ-able design activities. Although they are theresult of clustering similarities, the categories are

M. Mehalik and C. Schunn

520

nevertheless representative of design activities.Identifying these elements serves to highlightwhich activities get the attention of researchersreporting on case studies and empirical studies of design.

METHODOLOGY

The authors began the process of identifying andclustering design process elements/stages fromtheir past experiences in cognitive studies of design and systems engineering design. The discus-sion of earlier frameworks by Simon [5] andGibson [6] reflect some of the disciplinary originsof their initial thinking. These past experiences ledthem to consider sampling articles from several journals, mostly in

Design Studies

and

CognitiveScience

, to check the feasibility of identifyingempirical studies of design and being able tocharacterize some of the design stages discussedin such articles. The authors next recorded a seriesof elements that they believed characterized thedesign process, an initial framework of designactivity descriptors. This framework was used tocode an initial set of five empirical journal articlesin

Design Studies

and

Cognitive Science

. The frame-work was adjusted and re-clustered after extensivediscussions on whether the identified elementswere adequately descriptive, comprehensive, anddistinct.After this initial validation, additional articleswere identified through a database search of PsychInfo using these keywords: design; engineer-ing design; design process; design methodology;and cognitive analysis of design. The searchesusing these keywords located articles between theyears 1993 and 2003.From this initial set, the articles were screenedbriefly according to some basic criteria for inclu-sion. Articles that were included must be from apeer-reviewed journal and must involve an empiri-cal investigation of design. The article needed tocome from a journal in which a review of a designprocess represents a major interest in the commu-nity that would read such an article. The empiricalanalysis needed to emphasize the cognitive aspectsof design, not personality issues.

COMMON DESIGN PROCESSELEMENTS/STAGES

The bootstrapping procedure discussed in theprevious section resulted in a set of 15 elements/stages commonly found in the design process. Eachstep represents a distinct aspect of a design activity.Each is mutually exclusive in terms of document-ing a specific describable aspect of the designprocess. However, because a designer can beengaged in activities that have multiple goals andmeanings, it is possible for more than one criterionto be applicable at a time; for example, if adesigner is working on framing what problemneeds to be solved (Explore Problem Representa-tion) while using a sketch (Use Graphical Repre-sentation), two design aspects are applicable. Thedesign aspects are thus mutually exclusive in termsof defining specific aspects of design, but they arenot necessarily mutually exclusive with respect to aparticular event during the design process.

Explore problem representation

This design criterion refers to how designers goabout framing the design task or problem. Fram-ing the design task can involve defining or explor-ing a problem, issue, or artifact that needs to beanalyzed, synthesized, investigated, or constructed.The way in which designers construe their taskcan have an impact on what aspects of a design adesigner emphasizes, on what solutions pathsdesigners choose, and on which goals andconstraints designers meet. This aspect of designcan occur early in the design process when adesigner or design team deliberates what theirtask really is or is about. However, designers canalso decide in the midst of exploring a particulardesign that what he or she actually wants to makeor solve is different from the way they previouslyconstrued it.For example, Guenther

et al

. [8], reported thatone of the differences between experienceddesigners, who had no university educationaltraining or training in design methodology, anddesigners who did have university training indesign methodology was that the former groupgenerally pursued shorter design processes withnarrower objectives and singular solutions. Thelatter group tended to spend more time exploring,clarifying, and changing the problem.Another characteristic associated with exploringthe representation of the problem involves devisingor discovering when to stop pursuing the design orthe problem; in other words, when the solution ordesign fits how the problem or design has beenframed. Simon [5] has characterized the designprocess as an open-ended search process with nouniversal stopping rule at optimality; instead,design involves a process of `satisficing', or decid-ing when a configuration is good enough to becalled complete. Therefore, evidence that adesigner is working on the problem representationinvolves instances in which the designer decideswhat the end product will be.

Explore graphical representation/visualization

Designers often use a strategy of representingsome aspect of a design or design problem usinggraphics or visual media. The main distinction of this category involves the use of visual means toconstruct a representation, contrasting with averbal representation (such as a list of require-ments), a quantitative representation (such as alist of measurements), or other alternative forms of representation (such as kinesthetic, as in the case of a prototype or mock-up). Designers often sketch

What Constitutes Good Design? A Review of Empirical Studies of Design Processes

521