SURVEY DESIGN

• You’ve read your literature (lit review)

• You’ve created your question (research question)
• You’ve got your people (sampling)
• You’ve figured how to measure your variables
Design: what (measures)
are you • Now you need to figure out what you’ll do with
your participants when they come into your study
actually
going to do… This is called design
• Quantitative: experiment, survey, secondary
data analysis
• Qualitative: interview, focus group, secondary
data analysis
 Agenda

1.Introduction to 2. Steps in surveying

surveying

3.Sampling methods 4.Questionnaire

development

5. Survey
6. Data analysis
implementation
 1. Introduction to surveying
• What is a survey?
• “A systematic collection and analysis of data
on some aspect of an area or group” (Ross et
al, 2002)
• What is asked in a survey?
Questions Categories
How frequently do you brush your teeth?
Did you vote in the last national election? When did you last visit a close
relative?

What type of job do you think the Prime Minister is doing? Do you think other
people say many negative things about you when you are not there? What is
the biggest problem facing the nation these days?

Are you married, never married, single, divorced, separated, or widowed? Do

you belong to a union? What is your age?

Do you plan to buy a new car in the next 12 months? How much schooling do
you think your child will get? Do you think the population in this city will grow,
decrease, or stay the same?

Do you consider yourself to be liberal, moderate, or conservative? Into which

social class would you put your family? Would you say you are highly religious
or not religious?

Who was elected Prime Minister in the last election? About what percentage
of the people in this city are non-Asian? Is it legal to own a personal copy of
Karl Marx’s Communist Manifesto in this country?
 Types of surveys
• Complete survey (Census): involve the total
population

• Sample survey: involve a representative part

of population
 Types of surveys
Factors Complete survey Sample survey

Budget Large Small

Time available Long Short

Population size Large Small

Cost of High Low

sampling error
Attention to No Yes
individual cases
 2. Steps in surveying
6. Edit and
1.Define the
revise 7. Gather data
purpose
questionnaire

2. Define the 5. Pretest 8. Analyze and

hypothesis instrument evaluate data

4. Develop
3. Develop 9. Prepare
survey
survey plan report
questions
 2. Steps in surveying
• Step 1: Define the purpose of the survey
– Be specific and clear
– Determine future actions
Example:
Consumer satisfaction with services provided by
Company X.
Exercise: Define the purpose of the survey
 2. Steps in surveying
• Step 2: Define the hypothesis
– Be as specific as possible
– Avoid biased statements
Ex: “All consumers are satisfied with the services
quality of Company X”
Exercise: Write the hypothesis for your research
 HYPOTHESIS
• A hypothesis is a ‘tentative answer’ to the
problem you are addressing.

• It can help you to move from the DESCRIPTIVE to

the EVALUATIVE as it provides a base or
standard against which to test an idea.

• It also encourages you to set out your own

assumptions or values in approaching an issue –
these can often go unsaid!
 HYPOTHESIS: Decide what you
want to do?

Theory-then-research Research-then-theory

Testing an idea; deductive in Investigating to come up with an

nature idea; more inductive in nature
What sort of approach would you What sort of approach would you
associate this with? associate this with?
Problems? Problems?

HYPOTHESIS – BUILDING A THEORY –

TESTING AN IDEA EXPLORING A QUESTION
 2. Steps in surveying
• Step 3: Develop the survey plan
– Data collection plan
– Data coding plan
– Data analysis plan
 2. Steps in surveying
• Step 4: Develop the survey questions
– Connectedness between the questionnaire and
the research objectives.
 2. Steps in surveying
6. Edit and
1.Define the
revise 7. Gather data
purpose
questionnaire

2. Define the 5. Pretest 8. Analyze and

hypothesis instrument evaluate data

4. Develop
3. Develop 9. Prepare
survey
survey plan report
questions
 3. Sampling methods
3.1 Type of samples
3.2 Sampling techniques
3.2 Sampling size
 Important statistical terms
A sample is “a smaller (but
hopefully representative)
collection of units from a
population used to determine
Population: truths about that population”
(Field, 2005)
a set which includes all
measurements of interest
to the researcher
(The collection of all responses,
measurements, or counts
that are of interest)

Sample:
A subset of the population
 3.1 Types of sampling

• Non-probability samples
➢ Convenience samples (ease of access) sample is
selected from elements of a population that are easily
accessible (friends, colleagues, family…)
➢ Snowball sampling (friend of friend….etc.)
➢ Purposive sampling (judgemental)
– You chose who you think should be in the study

• Probability samples: random sampling

 3.2 Sampling techniques
• Random sampling (probability)
– Simple random sampling
– Systematic random sampling

• Non-random sampling
– Convenience sampling
– Snowball sampling…
Simple random sampling
Systematic sampling
 3.3 Sample size
• Formula:
 3.3 Sample size
Example:
N=10.000
d=0,05
Z=1,96
 3.3 Sample size
• Usual sample sizes used in survey

Sources: Malholtra (2008)

 3.3 Sample size
Authors Sample size Evaluation
Gorsuch (1983) and 100 Minimum
Kline
Comrey and Lee 50 Very poor
100 Poor
200 Fair
300 Good
500 Very good
1000 Exellent
Sources: Mundfrom, D.J., Shaw, D.G. and Ke, T.L., 2005. Minimum sample size
recommendations for conducting factor analyses. International Journal of
Testing, 5(2), pp.159-168.
 3.3 Sample size
• Variables-to-factors ratio: the ratio of number
of participants to number of variables is a better
way to determine a minimum sample size (Kline,
1983)
Authors Recommendations
Cattell (1978) 3-6 times
Everitt (1975) At least 10 : 1
Hair, Anderson, Tatham, and Black 20 : 1
(1995)
 4. Questionnaire development
4.1 What is a questionnaire?
4.2 Questionnaire structure
4.3 Question principles
4.4 Questionnaire design
4.5 Incentives
 4.1 What Is A Questionnaire?

A questionnaire is a
research instrument
consisting of a series
of questions and other
prompts for the
purpose of gathering
information from
respondents
 4.2. Questionnaire structure
• Introduction: is called a “cover letter” if the
introduction is written to accompany a mail
survey or online survey.
• Screening questions: are used to ferret out
respondents who do not meet research study
qualifications.
• Questions flow: pertains to the sequencing of
questions or set of questions.
 4.2 Questionnaire organization
• Functions of questionnaire introduction:
– Identifies the surveyor/sponsor
– Indicates the purpose of the survey
– Motivates the respondents to reply truthfully and
quickly
• Explain why the survey is important to them
• Explain how they were chosen to participate
• Pronounce the anonymity and the confidentiality
Introduction
Identify the surveyor

Inform the
respondents of the
topic and the reason
for the call

Motivate the respondents

to answer truthfully and
quickly
Step 4: Develop the survey questions

B. Questionnaire organization
• Introduction: is called a “cover letter” if the
introduction is written to accompany a mail
survey or online survey.
• Screening questions: are used to ferret out
respondents who do not meet research study
qualifications.
• Questions flow: pertains to the sequencing of
questions or set of questions.
Screening questions:
 Screening questions
• Example:
Topic: Factor affecting consumer satisfaction in
retail services: Evidence from Winn-Dixie.
Q: Have you shopped at Winn-Dixie in the last
month?
– Yes (Continue)
– No (Stop)
Screening or qualifying questions
are asked to ensure you are talking
to your target respondents.
 4.2 Questionnaire organization
• Introduction: is called a “cover letter” if the
introduction is written to accompany a mail
survey or online survey.
• Screening questions: are used to ferret out
respondents who do not meet research study
qualifications.
• Questions flow: pertains to the sequencing of
questions or set of questions.
 Warm-up questions

• Ex: How many major grocery

shopping trips do you do in a
month?
Questions Major questions: Complicated
& Difficult-to-Answer
flow • Rate each of the following aspects
of Winn-Dixie on how satisfactory it
is for you.

Classification and
demographic questions
Questions flow
 Questions flow
• Classification and demographic questions
 Types of Questions in a Questionnaire

• Open questions
– more information but difficult to codify, enter, and
analyze

• Closed questions
– less information but easy to codify, enter, and analyze
 Open-ended questions
• Help collect qualitative data in a questionnaire
where the respondent can answer in a free form
with little to no restrictions.

– What are your expectations/ requirements for this

product (Information gathering)
– What do you see as the next action steps?
(Qualifying)
– What’s the most important priority to you with this?
Why? (Establishing rapport)
 Close-ended questions
• They are a multiple-choice questions type in which a
respondent has to select one (single select multiple choice
question) or many (multiselect multiple choice question)
responses from a given list of options.

– Have you been stressed lately?

❑Yes
❑No
Types of Multiple Choice Questions
• Likert Scale
Types of Multiple Choice Questions

• Checklist type
Types of Multiple Choice Questions
• Rank Order
Types of Multiple Choice Questions

• Rating Scale
 Which Scale Do I Use?

If you are using a closed-ended question, which type of scale should you use?
• It depends on what you are measuring

If your focus is… then consider using…

yes/no
presence or absence or
true/false

importance, agreement,
frequency, quality, rating scale
satisfaction, or likelihood

48
 How Many Rating Scale Categories Should I Use?

• More than 7 is not recommended; 5 is

usually sufficient
• Consider whether you need a midpoint
• What are some advantages of having a
midpoint?
• What are some disadvantages?

1 2 3 4 5

Strongly agree Agree Neither Disagree Strongly

disagree

49
 Common mistakes
What might
be wrong
with this
question?

others
 Common mistakes
What might
What is your age group?
be wrong
with this 0-18 18-30 30-45 45+
question?

mutually exclusive
0-18 19-30 31-45 46+
 Common mistakes
What might
What did you think about the
be wrong
waiting time?
with this
question? Excellent Very Good Average Poo
good r
 Common mistakes
What might
How would you rate the
be wrong
appointment booking service
with this
using the:
question?
Good Average Poor
Website
Phone
Reception
4.2 Questionnaire structure
• Place demographic questions at the end
of the questionnaire.
• Have your opening questions arouse
interest.
• Ask easier questions first.
• To minimize conditioning, have general
questions precede specific ones.
• Group similar questions together.
• If you must use personal or emotional
questions, place them at the end of the
questionnaire.
 4.3 Question principles
1. Keep the language simple
2. Keep the questions short
3. Limit each question to one idea or
concept
4. Do not ask leading questions
5. Allow for all possible answers
6. Avoid vague quantifiers
 C. Question Principles
• Keep the language simple
tired………………………… exhausted
honest……………………… candid
work………………………… employment
most important………….top priority
free time……………………leisure
doctor………………………your responses to this
questionnaire
 4.3 Question Principles
1. Keep the language simple
tired………………………… exhausted
honest……………………… candid
work………………………… employment
most important………….top priority
free time……………………leisure
your answers……………….. your responses to this
questionnaire
 4.3 Question Principles
2. Keep the questions short
Ex:
Think about what makes FTU an ideal place to
learn and write down the 5 most important
aspects and rate them from 1-5 based on their
importance.
 4.3 Question Principles
3. Limit each question to one idea or concept
Ex: How many cups of coffee or tea do you
drink in a day?
Solution: Separate the question into two –
– (1) How many cups of coffee do you drink during a
typical day?
– (2) How many cups of tea do you drink during a
typical day?
 4.3 Question Principles
• To what extent do you agree or disagree with
the following statement?
 4.3 Question Principles
4. Do not ask the leading question
Do you support the use of green products?
- Yes
- No
 4.3 Question Principles
Solution:
 4.3 Question Principles
5. Allow for all possible answers
• Ex: What brand of computer do you own?
(A) IBM PC
(B) Apple
 4.3 Question Principles
5. Allow for all possible answers
• Solution:
 4.3 Question Principles
6. Avoid vague quantifiers
Ex: How often did you go to the cinema last
year?
❑ Never
❑ Rarely
❑ Occasionally
❑ Regularly
 4.3. Question Principles
6. Avoid vague quantifiers
Solution:
 4.4 Questionnaire design
• Stimuli on page consists of two languages:
– Written words
• Font size, color, style
• Question variety
– Graphic symbols and arrangements
• Bullets
• Images/Icon
• Space
 4.4 Questionnaire design
• Font size:
– For paper questionnaires the standard font is 12 pt
Times New Roman – also consider a larger font
for senior audiences
– For online questionnaires the standard font is 10
pt Arial
• Font color and underline:
Use italics or bolding or underline to emphasize
instructions and directions
 4.4 Questionnaire design
Start here:
1. Which of the following is your main work activity?

❑ Research
❑ Teaching
❑ Administration
❑ Other (Please Specify) _________________
START HERE:

 Which of the following is your main work activity?

❑ Research

❑ Teaching

❑ Administration

❑ Other (Please specify) ____________________

 4.4 Questionnaire design
• Stimuli on page consists of two languages:
– Graphic symbols and arrangements
• Bullets
• Push button
• Icon
Be consistent through out your
instrument
❑
Layout for Internet Questionnaires
• Graphical User Interface (GUI)
• Paging layout going from screen to screen
• Scrolling layout gives the respondent the
ability to scroll down
• Push buttons
• Status bar
 4.5 Incentives
• Incentives are offers to do something for the
respondent to increase the probability that
the respondent will participate in the survey.
• Incentives may be monetary or non-monetary
– Monetary incentives: cash, voucher, mobile card…
– Non-monetary incentives:
• Tangible rewards: useful documents (data, report,
book…) or results of the research
• Intangible rewards : show positive regard, say thank
you, ask for advice, make the questionnaire interesting
 Online Surveying Platforms

• docs.google.com/forms

• freeonlinesurveys.com

• surveymonkey.com
 Create a 5-10 question survey which addresses your
group topic though the specific research question is
up to you…
• Include questions and answers
• Follow best practices mentioned
Activity • Use at least one filter question
Everyone in the group should help write the survey
Take about 10 minutes to complete the google doc
Next we will try to complete each other’s surveys and
discuss!
 Class reflection

What questions worked What research questions What would you do

well? Didn’t work so could we answer? differently, if you could?
well? What would
univariate analysis Setting? Format?
look like here?
What bivariate
relationships could we
explore? Multivariate?
 5. Survey implementation
• Fieldwork:
– Select fieldworker (interviewer)
– Training survey fieldworker
– Quality control
• Online:
– Collect contacts
– Personalized correspondence
– Timeline to send the questionnaire (send ->
remind -> thank)
 6. Data analysis
6.1 Coding the data
6.2 Cleaning the data
6.3 Analyzing the data
 6. Data analysis
6.1 Coding the data
 6. Data analysis
6.2 Cleaning the data
• Data cleaning includes consistency checks and
treatment of missing responses.
Example:
Q: How old are you?
A: 150
• Treatment of missing data
– Substitute a neutral value
– Remove the missing responses
 6. Data analysis
6.3 Data analysis
• Descriptive analysis
• Regression analysis
 References
• Malhotra, N.K. and Birks, D.F.,
2010. Marketing research: An applied
approach. Pearson education.
• Janssens, W., De Pelsmacker, P., Wijnen, K. and
Van Kenhove, P., 2008. Marketing research
with SPSS. Pearson Education.