Statistical Model and Prediction

from Data

Primary data collection:

Survey Design
The survey research
in six steps
1) Identification of the reference population and sampling frame
2) Choice of sampling criteria
3) Choice of sample size
4) Choice of the data-collection method
5) Questionnaire design
6) Cost evaluation
Not necessarily in this order – rather interlinked decisions
Constructing questions
• Define target constructs
• Check related research & questionnaires
• Draft items (aim to have multiple indicators)
• Pre-test & revise

When drafting questions aim to:

• Focus directly on topic/issue

• Be clear
• Be brief
• Avoid big words
• Use simple and correct grammar
 Indagini con questionario

Formulazione del questionario:

• Prima parte (o ultima) riguardante le caratteristiche del

rispondente
• Parte centrale: domande sull’oggetto dell’indagine

Suggerimenti pratici
• Nel questionario utilizzare domande formulate con diversi
criteri
• Evitare l’effetto di trascinamento
• Eventualmente, sul problema di maggiore interesse
formulare due domande con criteri diversi (in posizioni
non consecutive)
• Utilizzare questionari non troppo lunghi
 Wording
• Avoid long and elaborate questions
• Use wording compatible with the measurement scale
• Use ordinary words
• Avoid ambiguous words (no generally, frequent etc.)
• Avoid phrasing which suggests the answer (Do you think people
should listen more to the radio and watch less television?)
• Avoid questions which need a particular effort for memory,
computing, etc. (How many hours per year do you listen to the radio?;
What is the frequency of your favourite radio station?)
• Avoid questions that are too generic (Why do you like radio
programs?)
 Sorting questions
• Use good opening (ice-breaking) questions (What is your current favourite
radio hit?)
• Place difficult and sensitive question towards the end (What is your
salary?)
• Ask basic information first (Do you own a radio?)
• Ask classification and identification questions at the end or at the
beginning (age, gender, etc.)
• General questions should precede specific questions
• Follow a logical order (a flow chart may help)
Code the question and test statistical processing

• Code the questions in a way that is functional for an electronic

data sheet (or statistical package)
• Try to anticipate potential problems in terms of lack of variability
(e.g. all respondents giving the same answers, number of items in a
measurement scale, etc.)
Introduction presentation

• Decide whether it is good to mention who promotes the research

(trust versus perception of vested interests)
• Consider pros and cons of provision of incentives to participate
(higher response rates vs. selection bias)

Layout
• Professional appearance in self-administered questionnaires
• Avoid splitting questions across pages
• Consider positioning key question close to the top of the page
 Pilot and pre-testing
• Pilot study: preliminary test on the questionnaire on a small number of
respondents to check for all previous issues and potential non-sampling
error
• Control on quality parameter (e.g. length and timing of the
questionnaire)
Overcoming problems in answering

• Factors that might lead to unanswered questions or inaccurate

answers:
• Lack of information
• Lack of memory
• Incapacity to articulate certain responses
• Unwillingness to answer (sensitive information, too much effort,
the question/context is perceived as inappropriate)
Techniques to get sensitive questions answered

• Hide the question among a group of innocent questions

• State that the behaviour of interest is common or the
usefulness of an answer
• Use the third-person technique
 Bias in questions
• Inapplicable (the questions must apply to all respondents)
• Over-demanding (recall of detail or time-consuming, unnecessary
questioning)
• Ambiguous (meaning must be clear to all respondents)
• Double negatives (e.g. do you not approve of tax reforms?)
• Double-barrelled (e.g. do you think speed limits should be lowered for
cars and trucks?)
• Leading (e.g. don’t you see some danger in the new policy?)
• Loaded (Do you advocate a lower speed limit to save human lives? vs
Does traffic safety require a lower speed limit?)

Bias in responding
• Social desirability
• Acquiescence or Yea- and Nay-saying (tendency to agree or disagree with
everything  use reversed items to control
• Self-serving bias (tendency to enhance self)
• Order effects (routine, fatigue)
 How to formulate questions
Objective vs subjective
• Objective:
How many times during last month did you visit a "coop" grocery?
• Subjective:
Think about the visits you made to a "coop" grocery during last month. How cheap
did you find the shopping?
very cheap reasonably cheap poorly cheap not at all

Open-ended vs close-ended

Open-ended
• rich information can be gathered
• useful for descriptive, exploratory work
• difficult and subjective to analyze
• time consuming
Close-ended
• important information may be lost
forever
• useful for hypothesis testing
• easy and objective to analyze
• time efficient
Open-ended questions - Examples
What are the main issues you are currently facing in your life?

How many hours did you spend studying this week? _________

Close-ended questions - Example

What are the main issues you are currently facing in your life? (please all
that apply)
 financial
 physical/health
 academic
 employment/unemployment
 intimate relations
 social relations
 other (please specify) ________________________________
 Close-ended questions - Example
How many hours did you spend studying this week?
 less than 5 hours
 > 5 to 10 hours
 > 10 to 20 hours
 more than 20 hours

Close-ended rating scales

• Likert scale: intensity of a single attribute, usually measured through
agreement with a sentence
• Graphic or continuous rating scale: tick in a continuous line which runs
between two extremes of a single attribute
• Semantic differential scale: tick in a continuous line which runs between two
extremes of a single attribute
• Stapel scale: unipolar scale on a single attribute with ten points from minus
five to plus five – no neutrality
• Non-verbal scale
• Frequency scale
Likert Scale
Pick a number from the scale to show how much you agree or disagree with each
statement:
1 2 3 4 5
strongly disagree neutral agree strongly
disagree agree

1 2 3 4 5
strongly agree neutral disagree strongly
agree disagree

Graphic Rating Scale

How would you rate your enjoyment of the movie you just saw? Mark with a cross

not enjoyable very enjoyable

Semantic Differential Scale
What is your view of smoking?
Tick to show your opinion.

Bad ___:___:___:___:___:___:___ Good

Strong ___:___:___:___:___:___:___ Weak
Masculine ___:___:___:___:___:___:___ Feminine
Unattractive ___:___:___:___:___:___:___ Attractive
Passive ___:___:___:___:___:___:___ Active

Non-verbal scale
Point to the face that shows how you feel about the economic situation in Italy
Verbal Frequency Scale
Over the past month, how often have you argued with your colleagues at work?

1. All the time

2. Fairly often
3. Occasionally
4. Never
5. Doesn’t apply to me at the moment

Sensitivity & Reliability

• Scale should be sensitive yet reliable
• Watch out for too few or too many options
General aim: How many options?

Maximise sensitivity (i.e. more options) Minimum= 2

Average = 3 to 7
Maximise reliability (i.e. less options) Maximum= 10
 Continuous rating scale
this cheese is
Extremely soft Extremely hard
0 50 100
Likert scale
this cheese is soft
Strongly disagree neither Strongly agree
1 2 3 4 5 6 7
Semantic differential scale: this cheese is
Extremely very slightly slightly very Extremely
soft soft neither hard hard hard
soft
1 2 3 4 5 6 7

Stapel scale: please evaluate this cheese

 Feeling about something
extremely positive extremely negative
2 categories
good not good
3 categories
good fair poor
4 categories
very good good fair poor
5 categories
excellent very good good fair poor
Watch out for too many or too few responses
"Capital punishment should be reintroduced 1 = very Strongly Agree
2 = strongly Agree
for serious crimes" 3 = agree
4 = slightly Agree
1 = Agree 5 = neutral
6 = slightly Disagree
2 = Disagree 7 = disagree
8 = strongly Disagree
9 = very Strongly Disagree
Scaling techniques

COMPARATIVE SCALING
Measurement is based on the comparison between objects

NON-COMPARATIVE SCALING
Measurement is based on individual assessment of each of the
objects
 Comparative scaling techniques
• Pairwise scaling: compare two objects
• Guttman scaling: measure one dimension through agreement
towards ranked set of items, ordered from least extreme to most
extreme position
• Rank order scaling: rank more than two objects
• Constant sum scaling: allocate a given number of points (e.g. 100)
to several objects
Q: Distribute 100 hours of leisure time
to the following activities, according to
your preferences:
1) Going out at night
2) Playing sports
3) Listening to music
4) Eating out
5) Studying statistics
… 2) I hate statistics
Q: Rank the following activities
according to your preferences:
1) Going out at night
2) Playing sports
3) Listening to music
4) Eating out
5) Studying statistics
…
Q: Do you prefer statistics or football?
1) Statistics
2) Football
3) No preference
Q: State the level of agreement with
the following sentences:
1) I do not like statistics
3) Statistics should be taken
off academic programs
4) Books about statistics
should be burnt
…
Modes of Survey Administration
Questionnaire
Interview
1. Mail survey
1. Personal interview
2. Email survey
2. Telephone interview
3. Computer Direct Interviews
• high demand characteristics 4. Internet/Intranet (Web Page) Surveys
• can elicit more information
• lower demand characteristics
• The interviewer must be trained
• information may be less rich
Personal Interviews
An interview is called personal when the Interviewer asks the questions face-to-face with the
interviewee. Personal interviews can take place in the home, at a shopping mall, on the street,
outside a movie theater and so on.
Advantages
 The ability to let the Interviewee see, feel and/or taste a product.
 The ability to find the target population. (e.g. people who have seen a film can be found
much more easily outside a theater in which it is playing than by calling phone numbers at
random.
 Longer interviews are sometimes tolerated. Particularly with in-home interviews that have
been arranged in advance. People may be willing to talk longer face-to-face than to someone
on the phone.
Disadvantages
 Personal interviews usually cost more per interview than other methods. This is particularly
true of in-home interviews, where travel time is a major factor.
 Each mall has its own characteristics. It draws its clientele from a specific geographic area
surrounding it, and its shop profile also influences the type of client. These characteristics may
differ from the target population and create a non-representative sample.
Personal interviews

In-home (interviewer visits respondent at home)

• Personal contact with interviewer • Highly Expensive
• Skilled interviewers increase quality • Interviewer influence/bias
• Longer duration • Wariness of respondents
• High response rate

Mall-intercept (respondent was stopped outside shops or in the street)

• Cheaper
• Easy use of stimuli
• Can be run without sampling frame
Computer-Assisted (CAPI) with interviewer
• Increased involvement of respondent
• On-screen and off-screen stimuli
• Interviews may last even longer
• Difficulties in obtaining sensitive
information (no anonymity)
• High social desirability
• Limited sampling control
• Slower (but time perception
varies)
Telephone Surveys
Surveying by telephone is the most popular interviewing method. This is made
possible by nearly universal coverage (96% of homes have a telephone)
Advantages
• People can usually be contacted faster over the telephone than with other methods. If
the Interviewers uses CATI (computer-assisted telephone interviewing) the results can be
available minutes after completing the last interview.
• You can dial random telephone numbers when you do not have the actual telephone
numbers of potential respondents.
• CATI software, such as The Survey System, makes complex questionnaires practical by
offering many logic options. It can automatically skip questions, perform calculations and
modify questions based on the answers to earlier questions. It can check the logical
consistency of answers and can present questions or answers choices in a random order
(the last two are sometimes important).
• Skilled interviewers can often elicit longer or more complete answers than people will
give on their own to mail, email surveys (though some people will give longer answers to
Web page surveys).
• Interviewers can also ask for clarification of unclear responses.
• Some software, such as The Survey System, can combine survey answers with pre-
existing information you have about the people being interviewed.
Disadvantages

• Many telemarketers have given legitimate research a bad name by

claiming to be doing research when they start a sales call. Consequently,
many people are reluctant to answer phone interviews and use their
answering machines to screen calls.
• The growing number of working women often means that no one is
home during the day. This limits calling time to a "window" of about 6-9
p.m. (when you can be sure to interrupt dinner or a favorite TV program).
• You cannot show or sample products by phone.
Telephone interviews

• Traditional interviewing (a phone, a pencil and a questionnaire)

• Computer Assisted Telephone Interviewing (CATI) –
computerised questionnaire administered to respondents
through the phone

• Software checks for consistency • Cannot reach those without a

and completeness
• Reduces the interviewers’ errors
• May control sampling procedures
(e.g. random dialling)
• Measures quality parameters (e.g.
duration)
• Data are ready to use
telephone
• It is expensive (at least 700-
1000 interviews to justify costs)
• Not very suitable for open
questions
• Interviews should be short
(twelve to fifteen minutes)
• Use of stimuli is not possible
Mail Surveys
Advantages
 Mail surveys are among the least expensive.
 This is the only kind of survey you can do if you have the names and addresses of the target
population, but not their telephone numbers.
 The questionnaire can include pictures - something that is not possible over the phone.
 Mail surveys allow the respondent to answer at their leisure, rather than at the often
inconvenient moment they are contacted for a phone or personal interview. For this reason,
they are not considered as intrusive as other kinds of interviews.
Disadvantages
 Time! Mail surveys take longer than other kinds. You will need to wait several weeks after
mailing out questionnaires before you can be sure that you have gotten most of the
responses.
 In populations of lower educational and literacy levels, response rates to mail surveys are
often too small to be useful. This, in effect, eliminates many immigrant populations that form
substantial markets in many areas. Even in well-educated populations, response rates vary
from as low as 3% up to 90%. As a rule of thumb, the best response levels are achieved from
highly-educated people and people with a particular interest in the subject (which, depending
on your target population, could lead to a biased sample).
Mail surveys

Mail interviews (Fax for businesses)

• Cheap • Very low response rate
• Optimal for sensitive • Selection bias / low sample
questions/anonymity/social desirability control
• No interviewer bias • Very slow

Mail panels
• Allow for longitudinal (time • More expensive
comparison) design • Low control of data collection
• Higher response rate environment
• Higher sample control
Computer Direct Interviews
The Interviewees enter their own answers directly into a computer. Can be used at malls, trade
shows, offices, and so on. Response rates are usually high
Advantages
 Elimination of data entry and editing costs.
 More accurate answers to sensitive questions. Recent studies of potential blood donors have
shown respondents were more likely to reveal HIV-related risk factors to a computer screen than
to either human interviewers or paper questionnaires. Studies has also found that computer-
aided surveys among drug users get better results than personal interviews. Employees are also
more often willing to give more honest answers to a computer than to a person or paper
questionnaire.
 Elimination of interviewer bias: different interviewers ask questions in different ways leading
to different results while the computer asks the questions always in the same way.
 Skip patterns are accurately followed. The Survey System can ensure people are not asked
questions they should skip based on their earlier answers. These automatic skips are more
accurate than relying on an Interviewer reading a paper questionnaire.
Disadvantages
 The Interviewees must have access to a computer or one must be provided for them.
 Serious response rate problems in populations of lower educational and literacy levels.
Email Surveys

Email surveys are both very economical and very fast. More people have email
than have full Internet access. This makes email a better choice than a Web page
survey for some populations. On the other hand, email surveys are limited to
simple questionnaires, whereas Web page surveys can include complex logic.

Advantages
 Speed. An email questionnaire can gather several thousand responses within a
day or two.
 There is practically no cost involved once the set up has been completed.
 You can attach pictures and sound files.
 The novelty element of an email survey often stimulates higher response levels.
Disadvantages
 You must possess (or purchase) a list of email addresses.
 Some people will respond several times or pass questionnaires along to friends to
answer. Many programs have no check to eliminate people responding multiple
times to bias the results. The Survey System\92s Email Module will only accept one
reply from each address sent the questionnaire. It eliminates duplicate and pass
along questionnaires and checks to ensure that respondents have not ignored
instructions (e.g., giving 2 answers to a question requesting only one).
 Many people dislike unsolicited email even more than unsolicited regular mail.
You may want to send email questionnaires only to people who expect to get email
from you.
 You cannot use email surveys to generalize findings to the whole populations.
People who have email are different from those who do not, even when matched
on demographic characteristics, such as age and gender.
 Email surveys cannot automatically skip questions or randomize question or
answer choice order or use other automatic techniques that can enhance surveys
the way Web page surveys can.
Internet/Intranet (Web Page) Surveys
Rapidly gaining popularity. Have major speed, cost, and flexibility advantages, but also
significant sampling limitations. These limitations make software selection especially
important and restrict the groups you can study using this technique.
Advantages
 Extremely fast. A questionnaire posted on a popular Web site can gather several thousand
responses within a few hours. Many people who will respond to an email invitation to take a
Web survey will do so the first day, and most will do so within a few days.
 There is practically no cost involved once the set up has been completed. Large samples do
not cost more than smaller ones (except for any cost to acquire the sample).
 You can show pictures. Some Web survey software can also show video and play sound.
 Web page questionnaires can use complex question skipping logic, randomizations and
other features not possible with paper questionnaires or most email surveys. These features
can assure better data.
 Can use colors, fonts and other formatting options not possible in most email surveys.
 A significant number of people will give more honest answers to questions about sensitive
topics, such as drug use, when giving their answers to a computer, instead of to a person.
 On average, people give longer answers to open-ended questions on Web page
Disadvantages
 While growing every year, Internet use is not universal. Internet surveys do not
reflect the population as a whole. This is true even if a sample of Internet users is
selected to match the general population in terms of age, gender and other
demographics.
 People can easily quit in the middle of a questionnaire. They are not as likely to
complete a long questionnaire on the Web as they would be if talking with a good
interviewer.
 If your survey pops up on a web page, you often have no control over who replies -
anyone from Antartica to Zanzibar, cruising that web page may answer.
 Depending on your software, there is often no control over people responding
multiple times to bias the results.
Internet for surveys is recommended mainly when the target population consists
entirely or almost entirely of Internet users. Business-to-business research and
employee attitude surveys can often meet this requirement. Surveys of the general
population usually will not. That said, Internet surveys did about as well, and in
some cases better, than other methods in predicting the outcome of the 2012 U.S.
presidential election.
Electronic interviews
E-mail (ASCII/text message)
• Very cheap
• Quick
• No interviewer bias
Web-based (HTML/Java)
• Allow for (some type of) stimuli
• Logic/consistency checks (CAWI)
• Higher sample control
• Anonimity/Sensitive questions
• Selection bias
• Requires data entry before analysis
• Low quality of data
• Low sample control
• Low response rate (and decreasing)
• (Very) low sample control
• Selection bias
• Problems in compiling lists
• Even lower response rate
The choice of survey method will depend on several factors.
These include: