See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/222558225

Understanding the knowledge needs of novice designers in the aerospace

industry

Article  in  Design Studies · March 2004

DOI: 10.1016/j.destud.2003.10.006

CITATIONS READS

77 227

2 authors, including:

Saeema Ahmed-Kristensen
University of Exeter
140 PUBLICATIONS   2,328 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

RCA posdoctoral Project: Biological Materials in Everyday Products View project

Robustness, reliability and safety in conceptual design View project

All content following this page was uploaded by Saeema Ahmed-Kristensen on 20 December 2018.

The user has requested enhancement of the downloaded file.

Understanding the Knowledge Needs of Novice

Designers in the Aerospace Industry

DR. S. AHMED & PROF. K. M. WALLACE

Engineering Design Centre, Keynes House

Engineering Department, Cambridge University, Trumpington Street,

Cambridge, UK, CB2 1PZ.

Email: sa233@eng.cam.ac.uk

TEL: 01223-332742 FAX: 01223-332662

1
Understanding the Knowledge Needs of Novice Designers in the

Aerospace Industry

S. AHMED & K. M. WALLACE

Engineering Design Centre, Engineering Department, Cambridge University,

Trumpington Street, Cambridge, UK, CB2 1PZ. sa233@eng.cam.ac.uk tel: 01223-

332742 fax: 01223-332662

Abstract. The long-term aim of this research is to understand how to support

designers through the provision of appropriate knowledge. This paper describes an

empirical study that analysed the interactions between novice and experienced

engineering designers in the aerospace industry. In total 633 queries between

novice and experienced designers were analysed. The analysis of the queries found

that novice designers were aware of their knowledge needs in only 35% of their

queries. The findings have implications for the type of support required by novice

designers.

Keywords: Design process; design knowledge; design strategy; design

behaviour; and communication.

The aerospace industry, along with other industries, has recognised the need

to capture, store and reuse more knowledge. This knowledge includes that

traditionally retained in the heads of experienced designers. Methods to

support novice designers have tended to focus primarily on Expert Systems

(refer to Studer et al. 1 for an overview of Experts Systems). Cao states that

2
Experts Systems are more successful in solving problems when there is a

definable goal, a clear set of initial conditions, and an outline of a procedure

or a set of rules by which the goal can be reached 2.

One of the issues surrounding the management of such knowledge is the

identification of what knowledge should be captured. Attempts to identify

the knowledge needs of designers have focused largely upon characterising

the type of knowledge designers require. Examples include, Marsh who

characterised knowledge in terms of product, process, and resources; and

3,4
Cantamessa who uses cost, environment, time, and quality . Baya

describes the information requests of designers in terms of alternatives,

construction, location, operation, performance, rationale, and relation and

requirement, and describes the subject-class of the questions in terms of

various levels of the physical product, requirements or the design-concept 5.

Kuffner’s analyses of three protocols identified question and conjectures, the

topic of these queries were classified based upon the construction, location,

operation or purpose of various levels of the physical product 6. These

studies largely focused on questions asked explicitly by the participants, i.e.

the studies assumed that the participants were deficient solely in the areas in

which they asked questions. This implies that the designers are aware of

their knowledge needs and know what they need to know.

In order to understand how to support designers through the provision of

knowledge, it was first necessary to understand better what knowledge

should be captured and to understand the designers’ awareness of their

3
knowledge needs. These long-term research aims have been addressed in

two separate studies: 1) an observational study to identify the differences

between how novice and experienced designers approach design tasks 7; and

2) discourse analysis of interactions of novice and experienced designers to

understand if they are aware of experienced designer behaviour. The latter

study is the focus of this paper.

The overall research approach was data driven and the outcome of the

previous observational study determined the need for the second study of

discourse analyses. The first observational study identified experienced

designer behaviour characterised by design strategies, and therefore gained a

better understanding of what knowledge should be captured. However, the

second part of the research aims was not addressed, i.e. to gain an

understanding of the awareness of novice designers of their knowledge

needs, hence the need for the second study, described in this paper. In

addition, one of the observations, from the observational study of novice and

experienced designers indicated that novice designers were reluctant to get

up and ask for advice 7. They preferred to note any questions down. This

may be due to a preference to ask experienced designers several questions at

once or may have been a consequence of being observed. However, one of

the novice designers involved in the study explained this behavior as related

to the level of knowledge possessed by the novice designers and their level

of confidence. These observations are inline with Marsh’s study of designers

in the aerospace industry, which showed that designers needed to have

4
confidence in their queries 3. During the observations, the number of

interactions observed between novice and experienced designers was

insufficient to draw any conclusions about the types of questions being

asked. Therefore, a second study, using discourse analysis of novice and

experienced designers was carried out, without the presence of a researcher.

A series of discourses provided novice designers the opportunity to ask

experienced designers questions. The aim of this study was to understand the

awareness of novice designers of their knowledge needs. The use of

discourse analysis and the research approach are described in the following

sections.

1. Discourse analysis

A discourse means a stretch of language longer than a sentence, e.g. a

sequence of sentences. Discourses are identified through the examination of

texts. All spoken and written material can be conceptualised as text and

subjected to discourse analysis 8. The term discourse analysis is often used

to mean the study of discourses, and was derived from the work of Foucault
9
. The analysis of discourses needs to take into account sentence sequences

as well as sentence structures 10. Discourse analysis may place emphasis at a

micro level focusing on textual detail, although discourse analysis is

generally more concerned with the social organisation of talk than linguistic

organisation 11. Discourse analysis can be applied in a wide range of research

settings. Potter and Wetherell view the term discourse in its most open

5
sense, to cover all forms of spoken interaction, formal and informal, and
11
written texts of all kinds . The discourses described in this study are

interactions between pairs of novice designers and experienced designers.

The term discourses has been used in its most open sense, and differentiates

the research method from interviews. Unlike interviews, no researcher was

present during the interactions, and all of the questions were determined

solely by the novice designers. Any analysis of interviews tends to focus on

the interpretation of answers, and needs to take into account their

retrospective nature and the bias of the interviewees. The analysis of the

discourses focused upon identifying patterns of behaviour in the interactions

of the designers. Therefore, the retrospective nature of the responses of the

experienced designers was not an issue. The use of discourse analyses for

this research study is described in the following sections.

1.1 RESEARCH APPROACH AND HYPOTHESIS

In total 11 discourses were analysed. These discourses formed part of a

Knowledge Acquisition Project (KAP) set up within a large aerospace

company. The twelve-week KAP was a collaboration between the aerospace

company and the Psychology Department of Nottingham University. The

aims of the KAP were to serve as a method of training by increasing the

understanding of a particular design process for the participating designers;

and also to produce web-pages for the company Intranet describing that

design process.

6
The discourses consisted of three pairs of trainee designers, i.e. three teams,

each interviewing a designer with the aim of eliciting knowledge to describe

a particular design process. In general, the company involved regarded the

designers who completed the KAP to have an increased understanding of the

design process and relevant design issues, in comparison to designers with

similar levels of experience, who had not participated in the KAP. Early in

the KAP, the discourses were unstructured, further into the project the

trainee designers used the discourses to fill in gaps in their descriptions of

the design processes under consideration. These later discourses consisted of

dialogues between the designers in addition to specific questions. At least

one of the designers from each pair was to work with the team, for which the

design process described was relevant, for example, the process of designing

a turbine blade was elicited by the team with one designer who worked with

the Turbine Systems team. Hence, the knowledge obtained during the

discourses would be beneficial to that designer, e.g. the knowledge of the

process involved in designing a turbine blade.

The discourses took place in the environment of the interviewees, i.e. the

offices of the experienced designers and lasted between 90 and 120 minutes

each. These discourses were audio-recorded by the trainee designers. The

transcripts from these discourses were structured by the trainee designers

12
using knowledge acquisition software to generate web-pages (refer to for

further details about the KAP and software). The process of generating web-

pages was independent of the research described and the transcripts analysed

7
during this research study were the raw data, i.e. prior to being structuring

for the purpose of generating web-pages. The KAP included training in

interview techniques, and the use of the software. The trainee designers had

only recently taken up employment within the company, with the exception

of one designer. The level of experience of both the trainee designers and the

interviewees is discussed later, in section 1.2.

Each team of trainee designers carried out between five and twelve

interviews during the KAP. The complete transcripts from the discourses

conducted at various points during the project were provided for this

research. As the KAP was conducted for purposes other than this research,

no additional time was required from the participants other than their

involvement in the KAP.

At the time of the discourses, the participants were unaware that the

transcripts of their discourses would be analysed for research purposes.

Hence they could not bias their questions or answers as no expected results
13
were communicated to the participants (referred to as subject bias ). In

addition, as no researcher was present during the discourses, the participants

could not alter their behaviour as a result of being observed. However, they

were aware that they were being audio-recorded and this may have affected

the discourses. The transcripts were obtained with the permission of the

designers involved after the discourses had been completed.

As stated earlier, the aim of the study was to understand the knowledge

needs of novice designers and, the awareness of the novice designers of their

8
knowledge needs was also of interest. Ahmed found from a previous

observational study of novice and experienced designers that experienced

designers predominantly used eight design strategies whilst carrying out

14
design tasks . During this observational study, the novice designers and

experienced designers were observed carrying out design tasks. The novice

designers were not observed to use these design strategies, although two of

the more experienced novice designers were observed to use one or two

strategies very occasionally. Therefore based upon these findings, prior to

the analysis of the discourses, it was hypothesised that:

ƒ The topics of the queries made by the trainee designers would vary with

their level of experience.

ƒ The trainee designers would not ask questions related to all of the eight

design strategies, but may ask questions related to some of them.

ƒ The novice designers would ask questions in topics not related to the

eight design strategies.

1.2 PARTICIPANTS

Sonnentag states that experience is thought to contribute to being a good

designer, but once a particular level of experience has been reached other

factors become more important, for example, field studies in software design

have demonstrated the importance of co-operation for high performance 15.

The period necessary to become experienced, defined as having attained an

international level, in fields such as chess, arts, sports and sciences, is

9
 16-18 in 19
thought to be 10 years . Designers with under two-and-a-half years

experience in the company were classed as novices and those with over ten

years of experience as experienced designers. It was not assumed that an

experienced designer was necessarily a good designer: no attempt was made

to assess the quality of any design work. By classing novices with less than

two-and-a-half years experience and experienced designers with over ten

years experience, it was assumed that the novice designers had not reached

the level of experience required to be good designers. The participants in this

research were therefore classed based upon the number of years of relevant

industrial experience and not on their design skills.

The participants of this study have been described as trainees, those asking

the questions and interviewees, those responding. The amount of relevant

experience in the aerospace industry of the trainees varied: members of two

of the three pairs had under nine months of experience and members of the

third pair had one-and-a-half and eight years of experience. The third pair

were significantly more experienced than the other two teams. The varying

levels of experience allowed the types of questions being asked by the

novice pairs and the more experienced pair to be compared respectively. The

level of experience of the interviewees varied between 2 years and over 30

years. All of the trainees and interviewees were British Caucasians males,

however this was not a deliberate choice. The background and experience of

the trainees together with their interviewees is summarised in Table 1, along

with the projects under discussion. Two of the interviewees, interviewee 3

10
and interviewee 10, had less than ten years experience i.e. were not classed

as experienced designers. However interviewee 3 with two-and-a-half years

experience was significantly more experienced compared to the trainees who

interviewed him, the trainees had two months and nine months experience

each. Interviewee 10 was interviewed as he was considered to be

knowledgeable with a certain design processes, namely plastic analysis.

11
Background Level of Experience of Participants: Background of

of Trainee Interviewee

Trainees (in months) Interviewees (in years)

Team A Project: Fuel Systems & Heat Management

Fuel Systems Trainee 1: 2 months Interviewee1: Over 30 years Fuel system

Military Trainee 2: 9 months Interviewee 2: Over 30 years Fuel system

Interviewee3: 2.5 years Fuel system

Team B Project: Turbine Systems New Part Introduction

Combustion Trainee 1: 8 months Interviewee 4: 10 years Turbine systems

Engine Interviewee 5: 10 years Turbine systems

systems Interviewee 6: 12 years Manufacture

Turbine Key Trainee 2: 8 months Interviewee 7: Over 30 years Manufacture

Systems Interviewee 8: Over 30 years Manufacture

Team C Project: Structural Analysis

Military Trainee 1: 18 months Interviewee 9: 15 years Static structures

Static Trainee 2: 96 months Interviewee 10: 2 years Static structures

Structures (i.e. 8 years) Interviewee 11: 20 years Preliminary design

Table 1 Experience of participants and projects discussed

1.3 Analysis method

The discourses of the three teams were all audio-recorded by the trainees and

then transcribed by the research group at Nottingham University. These

transcripts were analysed for this project to identify all the queries made by

12
the trainees. The method of analysis consisted of two phases: (1) an

encoding phase and (2) an analysis phase.

During the encoding phase all of the queries were identified. Each query was

categorised by the type of query, e.g. either a question or a statement. In total

633 queries were identified. An example of a statement is ‘there is

aerodynamic, thermal and stress work to be done’. An example of a

question is ‘where do you get that information?’

The response to each query was also categorised by its type, e.g. if a query

was simply answered or if it was rephrased. The encoding scheme for the

types of response was developed from the data, with new categories being

added as the need arose until no further categories were required. All the

discourses were reanalysed using the complete set of categories.

The second part of the encoding phase was to encode the queries and

responses by their topic, for example, a query related to terminology, e.g.

‘what is water hammer?’. The responses of the experienced designers were

categorised using the same encoding scheme as that used for the queries of

the trainees. The experienced designers sometimes talked about several

things and their answers therefore fell into more than one category. The

encoding scheme was developed from the data and, as before, new

categories were added as the need arose until no further new categories were

required. All the discourses were reanalysed using the complete set of

categories. The discourses were also analysed to identify any patterns in the

13
interactions between the trainees and the interviewees, described in section

2.2.

During the analysis phase, all the queries and their responses were analysed

with respect to the level of experience of the trainees. One person carried out

all of the categorisation of the data. However, the coding scheme was tested

for reliability using Cohen’s Kappa coefficient of reliability (described in 20).

The Kappa coefficient takes the actual percentage agreement and subtracts

the percentage agreement that can be expected from chance. A section of

transcript containing 45 queries was categorised independently by three

people: the researcher and two coders who were considered ‘naïve’. These

two coders were provided with a transcript of the interview and also a form

with the queries and the coding scheme consisting of eleven categories. They

were asked to encode the queries of the novice designers using the coding

scheme. An agreement was defined as the coders placing a query within the

same category as the researcher (and also agreeing that a query did not

belong to a particular category). Prior to categorising the transcript, both the

coders were provided with a description of each category and with examples

of queries that fell within each category. The transcripts of each of the

coders were compared to that of the researcher. The Kappa coefficients

calculated were 0.92 and 0.89, which indicate high inter-coder reliability.

The Kappa coefficient does not reveal details of the disagreements, for

example if all the disagreements belong to the same category. However, by

inspecting the method used to calculate the Kappa coefficients, it was clear

14
that the disagreements of the coder who had a Kappa coefficient of 0.92

were mainly between whether a query was ‘design process’ or ‘company

process’. The disagreements of the coder with a Kappa coefficient of 0.89

were mainly between the categories ‘how does it work’ and ‘what issues’.

When these disagreements were discussed with the coder, it was evident that

the definitions of these categories were not clear to the coder. The

definitions were improved to clarify the categories for further use.

2. Findings

The queries and responses were encoded depending on their topic and their

type. The encoding scheme and findings related to the topic of the queries

and responses are presented first as these are then used in the findings

related to the type of queries and responses.

2.1 Topics of queries and responses

The queries of the trainees and the responses received were encoded

according to their topic. The following encoding scheme consisting of

eleven topic categories was developed from the discourses.

Topic 1: Obtaining information: queries and responses regarding where

specific information in the form of documents, numerical data, etc, could be

obtained. The trainees required the experienced designers to provide details

15
of how to obtain information that was required to carry out a design task,

e.g. Do you get that information from previous engines?

Topic 2: Typical value: queries and responses requesting or stating typical

numerical values, as well as minimum and maximum values, e.g. What sort

of peak power does the high-pressure pump put out?

Topic 3: Terminology: queries and responses regarding what a particular

term meant, e.g. What is suction pressure?

Topic 4: Trade-offs: queries and responses regarding the effects of one

issue on another, e.g. If the pipe diameter is reduced, would that have a

negative effect by sucking against a small pipe?

Topic 5: How does it work: queries and responses related to how a

particular part of the engine functioned, e.g. Does the fuel go back through

the drains? Many of the queries were answered using domain knowledge, i.e.

knowledge of how the engine functions, knowledge of manufacturing

processes, etc.

Topic 6: Why: queries and responses about the reasons why the design is

carried out in a particular way, i.e. the rationale behind the design, e.g. Is it

because fuel flow doesn't take heat from the oil?

Topic 7: What issues to consider: queries and responses about issues that

should be considered during particular stages of the design process and also

the importance of issues, e.g. Does one of the issues take priority? Does one

expect the panels on the intercasing to be excited by vibration?

16
Topic 8: When to consider issues: queries and responses referring to when

issues should be considered, e.g. If it was a centrifugal compressor would it

be different?

Topic 9: How to calculate: queries and responses about the methods used

by a designer to achieve a task. The methods may involve knowledge of

principles of physics, for example, how to calculate the stresses in a

component, e.g. How do you calculate the variation in pump delivery

pressure from these graphs?

Topic 10: Design process: queries and responses referring to aspects of the

design process including: the information provided during the design

process; what is expected to be produced, etc, e.g. When starting a new

design, are you given a specification to work to?

Topic 11: Company process: queries and responses related to the

distribution of design work between departments; the relevant company

procedures; information on relevant people; and other aspects of company

procedures fell into this category, e.g. Who is your primary customer? What

is your job role? Who do you work with?

The queries of the trainees were encoded into the eleven topic categories

described above and are summarised for each of the three teams of trainees

in Table 2. In this table the queries are shown as percentages of the total

number of queries asked by that particular team. Nine percent of the 633

queries were not categorised by topic for the following reasons:

ƒ the query was repeating the answer of the experienced designer

17
 ƒ the query asked the experienced designer to repeat his or her answer

ƒ the query commented on the answer of the experienced designer.

Five percent of the queries were categorised as ‘other’. A high percentage of

the queries were about the design process (16%) or the company process

(20%). These may be due to one of the aims of the discourses being to

produce web-pages that map out a particular design process. The team with

the least amount of experience (Team A) had far fewer queries about the

design process (only 5%) and the company process (4%). The reason for this

is unknown.

The breakdown of queries for each team was used to identify topics that

correspond to the differing levels of experience of the teams (refer to Table

1). The first hypothesis stated that the topics of the queries made would vary

with experience, and was found to be true. The percentage of queries

categorised as ‘how does it work’ and ‘typical value’ decreased as the

experience of the team increased. The most experienced team (Team C)

made no queries about ‘how does it work’ and ‘typical values’. It is

presumed they had a greater understanding of how the engine functions and

of manufacturing processes and, hence, did not have a need to ask such

questions. Team C had greater experience of design tasks and presumably of

typical values than the other teams and, hence, did not need to ask queries on

this topic. The queries related to ‘what issues’ increased with the experience

level of the team. Team C asked twice as many queries on ‘what issues’ than

the less experienced teams.

18
Categories Team A Team B Team C Average

Experience of each 2 months 8 months 18 months for the three

trainee of the team 9 months 8 months 96 months (8 yrs) teams

Obtaining information 8 5 8 6

Typical value 9 2 0 4

Terminology 5 5 5 5

Trade-offs 3 0 0 1

How does it work 27 5 0 10

Why 11 3 5 5

What issues to consider 9 8 16 9

When to consider issues 8 1 9 3

How to calculate 6 6 4 6

Design process 5 20 20 16

Company process 4 28 13 20

Other 2 6 7 5

Total 97 89 87 90

Table 2 Percentage of total queries by team and topic

2.2 Types of queries and responses

The discourses were analysed for the type of query, i.e. a question or a

statement. All the possible types of responses from the interviewees to a

19
query from the trainees were identified from the transcripts and are

summarised in Table 3.

Type of query Type of response

Question Answer the question

Answer the question and provide additional information

Rephrase or state question as irrelevant

Rephrase or state question as irrelevant and provide additional

information

Statement Rephrase or state as irrelevant

Rephrase or state as irrelevant and provide additional

information

Provide additional information

Table 3 Type of query (from trainees) and types of possible responses (from

interviewees)

The discourses were analysed to understand the interaction between the

trainees and the interviewees. The interaction tended to fall into five main

patterns. These patterns focus upon the type of query; the type of response;

or the type of query and the type of response. These patterns are described in

the following sections and are summarised in Table 4.

Pattern 1: Explicit question and answer: The trainees asked a question

and the interviewee simply answered it. Approximately 35% of the queries

fell into this category (refer to Table 4). This pattern related to the following

20
topics: ‘obtaining information’, ‘terminology’, ‘company process’ and

‘typical values’. Questions were specifically aimed at obtaining this

information, since the experienced designers were not observed to offer this

information of their own accord. The trainees were aware of what they

needed to know and which questions to ask. An example that is related to the

topic ‘terminology’ is: The novice designer asked: ‘What is delta P?’, to

which the experienced designer replied: ‘The change in the pressure.’ The

experienced designer offered no additional information other than that

necessary to answer the question.

Queries about ‘how to calculate’ were also explicit questions and answers,

except those from Team C, the most experienced pair. Team C received

additional information from their interviewees. A possible explanation for

this maybe that the experienced designers expected Team C to be able to

understand the additional information offered, and hence provided additional

detailed information.

21
Patterns Team A Team B Team C Percentage

Experience of each trainee 2 months 8 months 18 months of all queries

9 months 8 months 96 months

Explicit question and answer 10% 29% 27% 35%

Rephrased or irrelevant queries 17% 8% 7% 10%

Additional information provided 18% 35% 21% 29%

Statements 28% 8% 32% 16%

Confirming queries 27% 20% 13% 21%

Table 4 Patterns of interaction

Pattern 2: Rephrased or irrelevant queries: The interviewee rephrased a

question or statement, or described it as irrelevant. Approximately 10% of

the queries fell into this category (refer to Table 4). An example of a

rephrased query is when a novice designer asked: ‘What is the relevance of

ambient pressure to the upper pumping level pressure?’, the response of the

experienced designer was: ‘We are only interested in the ambient pressure

under normal operation of the system - we should be considering the typical

pressure.’ The experienced designer rephrased the question, as in this case

the issue of ambient pressure was not relevant.

Figure 1 summarises the topics of the irrelevant and rephrased queries for

each of the three teams. Kuffner and Ullman also found that protocols of

three mechanical designers found 11% of all conjectures were false,

although a further 45% could be classified as either true or false 6. The

22
majority of the irrelevant and rephrased queries for the three teams were

about ‘how does it work’ (22%) followed by ‘design process’ (9%) and

‘terminology’ (8%). It is interesting to note that the most experienced team

(Team C) had no rephrased or irrelevant queries about ‘how does it work’,

‘terminology’ and ‘how to calculate’. When expressed as a percentage of

each teams’ queries there seems to be little difference between the number of

queries and the teams experience (refer to Table 4). However, in terms of

actual number of queries, Team C had five irrelevant or rephrased queries

whereas, in comparison, the less experienced teams had 28 and 30.

irrelevant or rephrased queries

20%
18%
Percentage of all

16%
14%
12%
10%
8%
6%
4%
2%
0%
rk
in e

e
n

s
pr s
fs

s
y
y

es

at
u

to s ue

es
es
io

W Wh
og

do -of

wo
Te v a l

W ssu

ul
Ty mat

oc
c
ol

mp pro
de

De calc
s
it

i
ti
l
ca
r

Ho en
ra

es
fo

rm

ha

y
Co gn
pi

h
in

an
si
w
g

w
in

Ho
ain

Topics
t
Ob

Team A Team B Team C

Figure 1 Irrelevant and rephrased queries categorised by topic

The interviewees were categorised by their number of years of

experience, into three groups: 2-3 years; 10-20 years; and 30+ years (see

23
Table 1). The number of times an experienced designer rephrased or stated a

query as irrelevant was found to increase with the experience of the

designer, except those related to the company process (refer to Figure 2).

These queries were more likely to be rephrased by an experienced designer

with 10-20 years of experience than one with 30 years or more. This

suggests that designers with 10-20 years of experience were more up to date

about the company process and, hence, were able to rephrase or state queries

as irrelevant in this topic.

24
 Number of rephrased or

2.5
irrelevant queries

2
1.5

1
0.5
0
y

e
s

yp s
hy

s
lue
n

ss
k
s
do -off

es
ue
log

W sue

lat
or
Ty atio

ce
W
va

oc
ss

lcu
it w
ino

t is

ro
m

Ho en i

pr
al

ca
or

rm

ha
ra
pic

es

n
h
inf

to

sig
W

an
Te
ing

mp
De
w
ain

Ho

Co
bt
O

Topics

2-3yrs 10-20yrs 30+yrs

Figure 2 Rephrased or irrelevant queries categorised by the experience of the

interviewees

Pattern 3: Additional information provided: In 16% of the queries, the

experienced designers offered additional information to that necessary to

simply answer a question or respond to a statement (refer to Table 4). As a

consequence, the number of queries was far less than the number of

responses in topics where a lot of additional information was provided. The

experienced designers provided additional information about what issues to

consider (60 queries and 178 responses), when to consider issues (22 queries

and 104 responses) and, to a lesser extent, trade-offs (5 queries and 12

25
responses). Experienced designers were clearly aware of issues that needed

to be considered and when these issues were relevant, e.g. ‘if the skin is

1mm or less on a panel that is fairly thick, it is quite highly loaded, which

could be a problem’. The more experienced pair of trainees asked a

significantly higher number of questions about relevant issues than the less

experienced teams. However, no clear correlation was found between the

experience of the trainees and the percentage of additional information. An

example of an experienced designer providing additional information is

when a novice designer asked: ‘Is it a case of saying we want a pump

delivery pressure rise?’, and the experienced designer answered ‘That's right

for simple checks.’ The designer then provided additional information by

explaining how to read graphs that showed how pump delivery pressure

varies: ‘Graph shows how the pressure varies, you can make an assumption

from the graph if the data is not available from suppliers. A twenty-five

percent increase in pressure decreases the flow by fifty percent. This is what

is plotted on the graph’.

Pattern 4: Statements: A statement was defined when no explicit question

was asked by the trainees, instead they simply expressed a statement.

Queries were expressed as statements in 29% of all cases suggesting that the

trainees required further information but were unsure of the questions (refer

to Table 4). The majority of the statements were spread over five topics:

‘company process’ (27%), ‘design process’ (14%), ‘why’ (12%), ‘how does

it work’ (11%), and ‘when issues’ (11%). All the statements regarding ‘how

26
does it work’ were made by the less experienced teams (Teams A and B). An

example of a statement is: ‘The actual high pressure pump requirements

would be quite high.’

Pattern 5: Confirming queries: The trainees were observed to make

queries to confirm the answers of the experienced designers (this does not

refer to confirming answers from previous interviews). The majority of

confirming queries were about ‘how does it work’ and ‘company process’.

Queries about ‘how does it work’ and ‘why’ were made by the less

experienced teams only (Teams A and B). This suggests that the less

experienced designers found it necessary to ask for confirmation of their

understanding of how the engine functions and the reasons why particular

components or manufacturing processes were used. An example of a

confirming query is when an experienced designer said: ‘ This normally

coincides with the maximum temperature and is towards the descent of the

flight’ and the novice designer asked the following question to confirm: ‘So,

this is at the descent of the flight and at maximum temperature?’

The number of queries that aimed to confirm the answers of the experienced

designers was found to decrease as the experience of the team increased.

Team C made queries to confirm the answers of the experienced designers

half as often as the least experienced team (refer to Table 4). Marsh also

found that novice designers often asked experienced designers questions to

confirm their beliefs 3.

27
The number of responses was found to increase with the level of experience

of the experienced designer (refer to Figure 3). However, the responses

related to the topic ‘company process’ were slightly higher for experienced

designers with 10-20 years of experience than for designers with over 30

years of experience.

25
of responses per person

20
Average number

15

10

0
ha hy
H Tr logy

pr s
a n

Co sign late
es ffs

D calc s

s
H hen es

er
rm lue

ny es

es
to ssue
pic tio

or

th
W issu
W W
do e-o
Te l va

pa roc
w

oc
u
Ty rma

ino

O
i
it
ow ad

p
t
o
inf
ing

ow

e
m
ain

Topics
bt
O

2-3yrs 10-20yrs 30+yrs

Figure 3 Responses categorised by topic and years of experience of the experienced

designers

2.3 Awareness of knowledge needs

In order to understand the awareness of the trainees of their knowledge

needs, the topics of the queries were compared to design strategies that are

employed by experienced designers 7. The design strategies were identified

28
from observations of novice and experienced designers working in the same

aerospace company as the designers participating in the discourses. Twelve

observations together with thinking-aloud were used to study the differences

between how novices and experienced designers approached their design

tasks. The transcripts were analysed and a total of twenty-two categories

describing the thoughts and actions of the designers were generated. Eight of

these were identified as design strategies, and were observed to be

predominantly experienced designer behaviour. The strategies were:

Strategy 1: Consider issues: experienced designers tended to consider

several relevant issues, and decided which were the most important. They

were also aware when issues were not relevant.

Strategy 2: Question data: experienced designers questioned data they

obtained from any source. They questioned the accuracy of the data; how

components were modelled or tested; how much accuracy was required;

customer specifications; and the applicability of standards.

Strategy 3: Question is it worth pursuing: the experienced designers asked

themselves how much they could expect to achieve if they continued a

particular approach and if it was worthwhile.

Strategy 4: Aware of reason: experienced designers were often aware of

the reasons behind the use of a particular design solution or manufacturing

process. The reasons why a component or process was used may be due to a

specific function or the capability of a particular supplier or manufacturing

29
process. The experienced designers assessed the reasons and their

applicability in the current situation.

Strategy 5: Aware of limitations: experienced designers were aware of the

limitations of the current design task and hence of the amount of time to

spend on it. The following reason was identified to limit the task: the

expected achievement of the current task versus further design tasks, and

incompleteness of information.

Strategy 6: Aware of trade-offs: experienced designers were aware of the

relationships between issues. They were aware that many decisions were

based on compromises and when they were aware of the trade-offs, they

would question whether it was worthwhile continuing to pursue the task or

implementing a decision.

Strategy 7: Refer to past designs: experienced designers referred to past

projects to find similar designs; designs in similar environmental and

functional conditions; and where similar problems had been encountered and

how they were resolved.

Strategy 8: Keep options open: experienced designers rejected an option or

delayed a decision on an option if it limited later options in the design task.

They were aware of what needs had to be considered further down the

design process.

The most experienced team (Team C) was found to have a greater

understanding of how the engine works and why a particular component or

manufacturing process was needed. They had no rephrased or irrelevant

30
queries in these topics (refer to Figure 1). This finding is inline with the

observation that experienced designers were observed to be ‘aware of

reason’ 7. Team C made more queries about what issues to consider and

when to consider them than the other teams. This confirmed the finding from

the observations that experienced designers were found to ‘consider issues’

more than novice designers 7. Team C asked questions to establish the

accuracy of data and demonstrated a need to establish the degree of accuracy

of the data before accepting it, e.g. How accurate is the technique of plastic

analysis? This finding is also supporting by the observations as the category

‘question data’ was described as experienced designer behaviour. Team C

was also found to make fewer queries to confirm the experienced designers

answers than the less experienced teams.

When comparing the topics of queries made by the trainees to the design

strategies employed by experienced designers, the trainees were aware of the

need to find out ‘how the engine works’ (part of the category ‘aware of

reason’) and to ‘consider issues’. However, during the discourses

experienced designers provided additional information over and above that

necessary to answer these questions. This suggests that the trainees were

only partly aware of the need to find out how the engine works and about

issues to consider, and still required guidance from experienced designers.

Thus confirming the confirming the second hypothesis, i.e. the trainee

designers would not ask questions related to the all of the eight design

strategies, but may ask questions related to some of them. As the trainees did

31
ask questions in topics that were not related to the design strategies, as

described in section 2.1, the third and final hypothesis was also found to be

true.

2.4 Limitations

The discourses were conducted with the aim of mapping out a particular

design process. The questions asked were not part of an ongoing design

project and, hence, the frequency of questions referring to the rationale

behind concepts or the rejection of alternatives, etc. was not assessed as the

number of queries in these topics was not a true representation. As the

trainees worked in teams of two, they could not be assessed individually but

only as a pair. Although not a direct limitation, each of the two-hour

discourse analyses required a further 32 hours in travel, transcription and

analysis and, hence, was a time consuming approach.

3. Conclusions

Eleven discourses between novice and experienced engineering designers

have been analysed. In total, 633 queries were analysed in an attempt to

identify the knowledge needs of novice designers. The analysis found that

the queries and their responses were related to eleven different topics. Five

different patterns of interactions between the novice and experienced

32
designers were identified. The analyses also provided an insight into the

awareness novice designers have of their knowledge needs.

Novice designers tended to be unaware of the design strategies employed by

experienced designers. The findings indicated that experienced designers

had greater domain knowledge (how the engine works), regularly questioned

the accuracy and suitability of data, considered a fuller range of issues and

were aware of trade-offs between issues. The trainees asked a question and

the interviewee simply answered it in only 35% of all their queries. This

suggests that the novice designers were only aware of the specific

information that they needed to know in about a third of all their queries.

They were aware of the specific information they required in some topics,

such as domain knowledge. 29% of all queries were statements and this

suggests the novice designers were aware of the topic that they needed to

know about, but they did not know the exact question to ask. The

experienced designers rephrased queries or stated them as irrelevant in about

10% of all the queries.

Overall, the findings suggest that supporting novice designers by simply

supplying knowledge may not be enough, they also require support in

identifying what they need to know. Supplying novice designers with a

question-based approach may help to inform them of what they do not know.

A question-based method to help novice designers be aware of what they

need to know has been proposed and is described separately 21.

33
4. Acknowledgements

The authors acknowledge the support for this research from the Engineering

and Physical Sciences Research Council (EPRSC) and from Rolls-Royce

plc. The authors acknowledge Prof. Lucienne Blessing for her contribution

to the research project and Dr Michael Moss (Rolls-Royce plc.) for his

support throughout the project. The authors acknowledge the support of the

University Technology Partnership for Design, a collaborative research

project between the universities of Cambridge, Sheffield and Southampton;

and with industrial partners BAE SYSTEMS and Rolls-Royce.

References

[1] Studer R, Benjamins V R and Fensel D Knowledge Engineering: Principles and

Methods Data and Knowledge Engineering Vol 25. (1998) pp 161-197

[2] Cao Q and Protzen J P Modelling Design Reasoning In: Computer Aided Conceptual

Design. Lancaster University (1998) pp 1-22

[3] Marsh J R The Capture and Utilisation of Experience in Engineering Design PhD.

Thesis, Cambridge University (1997)

[4] Cantamessa M Design best Practices at Work: An Empirical Research upon the

Effectiveness of Design Support Tools In: International Conference of Engineering Design

(ICED 97). Schriftenreihe WDK 25, Tampere, (1997) pp 541-546

[5] Baya V Information Handling Behaviour of Designers During Conceptual Design: Three

Experiments Ph.D. Thesis, Stanford University, Palo Alto (1996)

[6] Kuffner T A and Ullman D G The information requests of mechanical design engineers

Design Studies Vol 12. No 1 (1991) pp 42-50

34
[7] Ahmed S, Wallace K M and Blessing L S Understanding the differences between how

novice and experienced designers approach design tasks Research in Engineering Design

Vol 14 No 1 (2003) pp 1-11

[8] Coyle A Discourse Analysis. In: G. M. Breakwell, S.Hammond, C. Fife-Shaw (eds)

Research Methods in Psychology. Sage Publications, London (2000) pp 251-268

[9] Foucault M The Archaeology of Knowledge and Discourse on Language Pantheon

Books, New York (1972)

[10] Bullock A, Stallybrass O and Trombley S The Fontana Dictionary of Modern Thought

Fontana Press, London (1988)

[11] Potter J and Wetherell M Discourse and Social Psychology: Beyond Attitudes and

Behaviour Sage Publications, London (2002)

[12] Milton N and Shadbolt N Towards a knowledge technology for knowledge

management Human-Computer Studies Vol 51. (1999) pp 615-641

[13] Orne M T Demand Characteristics and the Concept of Quasi-Controls In: R. Rosenthal

and R. Rosnow (eds) Artifacts In Behavioral Research Academic Press, New York (1969) pp

143-179

[14] Ahmed S Understanding the Use and Reuse of Experience in Engineering Design, Ph.D.

Thesis, Cambridge University, Cambridge (2001)

[15] Sonnentag S Expertise in Professional Software Design: A Process Study Journal of

Applied Psychology Vol 83. No 5 (1998) pp 702-715

[16] Simon H A and Chase W G Skill in Chess American Scientist Vol 61. (1973) pp 394-

403

[17] Hayes J R The Complete Problem Solver Franklin Institute Press, Philadelphia (1981)

[18] Bloom B S Developing Talent in Young People Ballantine Books, New York (1985)

[19] Ericsson K A and Smith J Empirical Study of Expertise: Prospects and Limits In: K.

A. Ericsson and J. Smith (eds) Towards a General Theory of Expertise Cambridge University

Press, Cambridge (1991) pp 1-38

35
 [20] Bakemann R and Gottam J M Observing Interaction: An Introduction to Sequential

Analysis Cambridge University Press, Cambridge UK (1997)

[21] Ahmed S and Wallace K M Developing and Evaluating a Method of Support for

Novice Designers in the Aerospace Industry Journal of Engineering Design (forthcoming)

36

View publication stats