Data Collection Methods

1
Research Design and How Data Collection Methods
Fit In (Sekaran, 2010, p.185)

2
 Where do data come from?
• We’ve seen our data for this lab, all nice and
collated in a database – from:
– Insurance companies (claims, medications,
procedures, diagnoses, etc.)
– Firms (demographic data, productivity data, etc.)

3
 Where do data come from?
• Take a step back – if we’re starting from
scratch, how do we collect / find data?
– Secondary data
– Primary data

4
 Secondary Data
• Secondary data – data someone else has
collected

5
 Secondary Data – Examples of Sources
• County health departments
• Vital Statistics – birth, death certificates
• Hospital, clinic, school nurse records
• Private and foundation databases
• City and county governments
• Surveillance data from state government programs
• Federal agency statistics - Census, NIH, etc.

6
 Secondary Data – Limitations
• What did you find on the frustrating side as
you looked for data on the state’s websites?

7
 Secondary Data – Limitations
• When was it collected? For how long?
– May be out of date for what you want to analyze.
– May not have been collected long enough for
detecting trends.
– E.g. Have new anticorruption laws impacted
Russia’s government accountability ratings?

8
 Secondary Data – Limitations
• Is the data set complete?
– There may be missing information on some
observations
– Unless such missing information is caught and
corrected for, analysis will be biased.

9
 Secondary Data – Limitations
• Are there confounding problems?
– Sample selection bias?
– Source choice bias?
– In time series, did some observations drop out
over time?

10
 Secondary Data – Limitations
• Are the data consistent/reliable?
– Did variables drop out over time?
– Did variables change in definition over time?
• E.g. number of years of education versus highest
degree obtained.

11
 Secondary Data – Limitations
• Is the information exactly what you need?
– In some cases, may have to use “proxy variables” –
variables that may approximate something you really
wanted to measure. Are they reliable? Is there correlation
to what you actually want to measure?
– E.g. gauging student interest in U.W. by their ranking on
FAFSA – subject to gamesmanship.

12
 Secondary Data – Advantages
• Saving Time and Cost
• It may be very accurate.
– When especially a government agency has
collected the data, incredible amounts of time and
money went into it. It’s probably highly accurate.
• It has great exploratory value
– Exploring research questions and formulating
hypothesis to test.

13
 Primary Data
• Primary data – data you collect

14
 Primary Data - Examples
• Surveys
• Focus groups
• Questionnaires
• Personal interviews
• Experiments and observational study

15
 Primary Data - Limitations
• Do you have the time and money for:
– Designing your collection instrument?
– Selecting your population or sample?
– Pretesting/piloting the instrument to work out
sources of bias?
– Administration of the instrument?
– Entry/collation of data?

16
 Primary Data - Limitations
• Uniqueness
– May not be able to compare to other populations

17
 Primary Data - Limitations
• Researcher error
– Sample bias
– Other confounding factors

18
 Interviewing is…
• Talking and listening to people
• Verbally asking program participants the program
evaluation questions and hearing the participant’s
point of view in his or her own words. Interviews can
be either structured or unstructured, in person or
over the telephone.
• Done face-to-face or over the phone
• Individual; group

19
 Interviews are useful…
• When the subject is sensitive
• When people are likely to be inhibited in
speaking about the topic in front of others
• When people have a low reading ability
• When bringing a group of people together is
difficult (e.g., in rural areas)

20
 Interviews
Verbally asking program participants the program
evaluation questions and hearing the participant’s
point of view in his or her own words.
Interviews can be either structured or unstructured, in
person or over the telephone.

21
 Interviews
• Advantages • Disadvantages
– deep and free – costly in time and
response personnel
– flexible, – requires skill
adaptable – may be difficult to
– glimpse into summarize
respondent’s responses
tone, gestures – possible biases:
– ability to probe, interviewer,
follow-up respondent,
situation 22
 Types of interviewing

Structured Conversational

23
 Type: Structured interview
• Uses script and questionnaire
• No flexibility in wording or order of questions
• Closed response option
• Open response option

24
 Type: Guided interview
• Outline of topics or issues to cover
• May vary wording or order of questions
• Fairly conversational and informal

25
Type: Conversational interview
• May not know that an interview is taking
place
• Spontaneous
• Questions emerge from the situation and
what is said
• Topics or questions are not predetermined
• Individualized and relevant to situation

26
 Probing
Interview question:
“What did you like best about this program?”
Response: “I liked everything.”
Probe 1: “What one thing stood out?”
R: “Being with my friends.”
Probe 2: “What about the program activities?”
R: “I liked it when we worked as a team.”
Probe 3: “How come?”
R: “It was neat to hear each other’s perspectives. I heard
some things I hadn’t considered before.”
Probe 4: “What is one thing that you learned?”

27
 Interviewing tips
• Keep language pitched to that of respondent
• Avoid long questions
• Create comfort
• Establish time frame for interview
• Avoid leading questions
• Sequence topics
• Be respectful
• Listen carefully

28
 Recording responses
• Write down response
• Tape record
• Key in on computer
• Work in pairs
• Complete notes after interview

29
 Questionnaires are…
• Data collection instruments used to collect
standardized information that can be expressed
numerically or through short answers
• Basic instruments of surveys and structured
interviews
• Appropriate when…
– you want information from many people
– you have some understanding of the situation and
can ask meaningful questions
– information is sensitive or private − anonymous
questionnaires may reduce bias

30
 Questionnaires
• Advantages • Disadvantages
– can reach large – might not get
numbers careful feedback
– provide for – wording can bias
anonymity client’s response
– relatively – response rate is
inexpensive often low
– easy to analyze – literacy demands

31
When should a questionnaire be used?

• Respondents can provide useful information

about the topic.
• You know what it is you want to know and are
reasonably sure that you can ask standardized
questions to get the information.
• Respondents can be relied upon to provide the
information you need (perhaps with
incentives). This means they can comprehend
the questions and respond properly, they are
truthful, and they are motivated enough to
respond carefully.

32
Good questionnaires are NOT EASY!
• Developing a good questionnaire, takes
time, time, and more time.
• Multiple (even a dozen!) drafts may be
involved before the questionnaire is
ready.
• It’s important to involve others in
writing the questionnaire.

33
Principles of Questionnaire Design (Sekaran,
2010, p.199)

34
 Choosing Question Structure –
Unstructured Questions

• Unstructured questions are open-ended questions

that respondents answer in their own words.

What is your occupation?

Who is your favorite actor?
What do you think about people who shop at high-
end department stores?
35
 Choosing Question Structure –
Structured Questions

• Structured questions specify the set of

response alternatives and the response
format. A structured question may be
multiple-choice, dichotomous, or a scale.

36
 Choosing Question Structure –
Multiple-Choice Questions
• In multiple-choice questions, the researcher provides a choice
of answers and respondents are asked to select one or more of
the alternatives given.

Do you intend to buy a new car within the next six months?
____ Definitely will not buy
____ Probably will not buy
____ Undecided
____ Probably will buy
____ Definitely will buy
____ Other (please specify)

37
 Choosing Question Structure –
Dichotomous Questions

• A dichotomous question has only two response alternatives: yes

or no, agree or disagree, and so on.
• Often, the two alternatives of interest are supplemented by a
neutral alternative, such as “no opinion,” “don't know,” “both,”
or “none.”

Do you intend to buy a new car within the next six months?
_____ Yes
_____ No
_____ Don't know

38
 Choosing Question Structure –
Scales

• Scales were discussed in detail in Chapters 7:

Do you intend to buy a new car within the next six months?

Definitely Probably Undecided Probably Definitely

will not buy will not buy will buy will buy
1 2 3 4 5

39
 Choosing Question Wording –
Define the Issue
• Define the issue in terms of who, what, when, where,
why, and way (the six Ws). Who, what, when, and
where are particularly important.

Which brand of shampoo do you use?

(Incorrect)

Which brand or brands of shampoo have you

personally used at home during the last month?
In case of more than one brand, please
list all the brands that apply. (Correct)
40
 Choosing Question Wording
The W's Defining the Question

Who The Respondent

It is not clear whether this question relates to
the individual respondent or the respondent's
total household.

What The Brand of Shampoo

It is unclear how the respondent is to answer
this question if more than one brand is used.

When Unclear
The time frame is not specified in this question.
The respondent could interpret it as meaning
the shampoo used this morning, this week, or
over the past year.

Where At home, at the gym, on the road?

41
 Choosing Question Wording –
Use Ordinary Words

“Do you think the distribution of soft drinks is

adequate?” (Incorrect)

“Do you think soft drinks are readily available when

you want to buy them?”
(Correct)

42
 Choosing Question Wording –
Use Unambiguous Words
In a typical month, how often do you shop in department
stores?
_____ Never
_____ Occasionally
_____ Sometimes
_____ Often
_____ Regularly
(Incorrect)

In a typical month, how often do you shop in department

stores?
_____ Less than once
_____ 1 or 2 times
_____ 3 or 4 times
_____ More than 4 times (Correct)

43
 Choosing Question Wording –
Avoid Leading or Biasing Questions
• A leading question is one that clues the respondent to what the answer
should be, as in the following:
 
Do you think that patriotic Americans should buy imported automobiles
when that would put American labor out of work?
_____ Yes
_____ No
_____ Don't know
(Incorrect)

Do you think that Americans should buy imported automobiles?

_____ Yes
_____ No
_____ Don't know
(Correct)

44
 Choosing Question Wording –
Avoid Implicit Alternatives

• An alternative that is not explicitly expressed in the options is

an implicit alternative.
 
1. Do you like to fly when traveling short distances?
(Incorrect)

2. Do you like to fly when traveling short distances, or would

you rather drive?
(Correct)

45
 Choosing Question Wording –
Avoid Implicit Assumptions

• Questions should not be worded so that the answer is

dependent upon implicit assumptions about what will
happen as a consequence.
 
1. Are you in favor of a balanced budget?
(Incorrect)

2. Are you in favor of a balanced budget if it would

result in an increase in the personal income tax?
(Correct)

46
 Choosing Question Wording –
Avoid Generalizations and Estimates

“What is the annual per capita expenditure on groceries in

your household?” (Incorrect)
 
“What is the monthly (or weekly) expenditure on groceries in
your household?”

and

“How many members are there in your household?”

(Correct)
47
 Choosing Question Wording
Dual Statements: Positive and Negative

• Questions that are in the form of statements

should be worded both positively and negatively.

48
 Determining the Order of Questions
Opening Questions
• The opening questions should be interesting, simple, and non-
threatening.
 
Type of Information
• As a general guideline, basic information should be obtained
first, followed by classification, and, finally, identification
information.
 
Difficult Questions
• Difficult questions or questions which are sensitive,
embarrassing, complex, or dull, should be placed late in the
sequence.
49
Determining the Order of Questions
Effect on Subsequent Questions
• General questions should precede the specific
questions (funnel approach).
 
Q1: “What considerations are important to you in
selecting a department store?”

Q2: “In selecting a department store, how important

is convenience of location?” (Correct)
50
Determining the Order of Questions
Logical Order

The following guidelines should be followed for

branching questions:

• The question being branched (the one to which the

respondent is being directed) should be placed as close as
possible to the question causing the branching.

• The branching questions should be ordered so that the

respondents cannot anticipate what additional information
will be required.
51
 Observation…
• Is watching people, programs, events,
communities, etc.
• Involves all 5 senses: sight, hearing, smell,
touch, and taste
– observation includes more than just “seeing”

52
 Observation is used…
• To provide information about real-life
situations and circumstances
• To assess what is happening
• Because you cannot rely on participants’
willingness and ability to furnish
information

53
 When is observation useful?
• When you want direct information
• When you are trying to understand an
ongoing behavior, process, unfolding situation,
or event
• When there is physical evidence, products, or
outcomes that can be readily seen
• When written or other data collection
methods seem inappropriate

54
 Observations
• Advantages • Disadvantages
– May require training
– Most direct
– Observer’s presence may
measure of
create artificial situation
behavior – Potential for bias
– Provides direct – Potential to overlook
information meaningful aspects
– Easy to complete, – Potential for
misinterpretation
saves time
– Difficult to analyze
– Can be used in
natural or
experimental
55
settings
 Types of observation
Structured Unstructured
Looking for Looking at

Observing what does not happen may be as

important as observing what does happen.

56
Steps in planning for observation
• Determine who/what will be observed.
• Determine aspects that will be observed
(characteristics, attributes, behaviors, etc.).
• Determine where and when observations will be
made.
• Develop the observation record sheet.
• Pilot test the observation record sheet.
• Train the observers and have them practice.
• Collect the information.
• Analyze and interpret the collected information.
• Write up and use your findings.

57
 Who/what to observe
• People (individuals, groups,
communities)
– Characteristics
– Interactions
– Behaviors
– Reactions
• Physical settings
• Environmental features
• Products/physical artifacts

58
 Observation – Example
If you want
information about… You would record…
Who uses a particular Total number of users
service broken down by gender,
age, ethnicity, etc.

Interactions between # and types of questions

youth and adults asked by each
Neighborhood safety ???
 What to observe − Example
Exhibit on tobacco use at a county fair

Information needed:
Number of youth who visit the exhibit: age, gender,
cultural background

Can the information be observed accurately? E.g.,

gender may be more obvious than age or cultural
background.
Will the observer affect the situation?

60
 Example – Plans for observing
participation in an after school program
• Who: youth attending the program
• What:
– approximate age
– gender, cultural background
– length of time student stays in the program
• When: all hours the program is open for one week each
month during 2013

61
 Recording your observations
Observations need to be recorded to be credible. You
might use:
– Observation guide
– Recording sheet
– Checklist
– Field note
– Picture
– Combination of the above

62
 Observational rating scales
• Written descriptions – written explanations of
each gradation to observe
• Photographs – series of photos that
demonstrate each of the grades on the rating
scale
• Drawings, sketches, etc. – other visual
representations of conditions to be observed

63
 Who are the observers?
• You – program staff
• Participants
• Stakeholders
• Colleagues
• Volunteers
• College students

64
 Training observers
Training is often necessary:
– To learn what to look for
– To learn how to record observations
– To practice
– When want standardized observations
across sites: important that all observers
use same methods, rate same observation
in same way

65
 Practice
Imagine you are sitting in a room where
ten youth are participating in a computer
demonstration. If you were looking for
indicators of student interest and learning
from the demonstration, what would you
look for?
(Remember to include verbal and nonverbal indicators.)

66
 Data collection choice
• What you must ask yourself:
– Will the data answer my research question?

67
 Data collection choice
• To answer that
– You much first decide what your research question
is
– Then you need to decide what data/variables are
needed to scientifically answer the question

68
 Data collection choice
• If that data exist in secondary form, then use
them to the extent you can, keeping in mind
limitations.
• But if it does not, and you are able to fund
primary collection, then it is the method of
choice.

69