DESCRIPTIVE METHODS II

Topic 4: Survey Research

By: Chie Qiu Ting @ Karen
Learning Outcomes
Explain The strength and weakness of survey method

Distinguish Probability and non-probability samplings

Clarify Reliability and validity

Describe The concept of path analysis

 2

Survey Research
• Survey research
• Describe thoughts, opinions, feelings
• Measure naturally occurring variables
• Allow predictions based on correlations
• Questionnaires
• Predetermined set of questions
• Sample represents a population
• Examine survey procedures and analyses for sources of bias
 3

Sampling in Survey Research

• Use sample to represent the larger population
• Requires careful selection of a sample
• Goal: Generalise survey findings from representative sample to
the population

A sample is representative of the

population to the extent that it
exhibits the same distribution of
characteristics as the population.
 4

Basic Terms of Sampling

• Population
• Set of all cases of interest (Psychology students, UTAR)
• Sampling Frame
• List of the members of a population (get the list of enrolled
students)
• Sample
• Subset of population drawn from sampling frame (300 students)
• Element
• Each member of the population
Sampling Method
 7

Approaches to Sampling
• “Sampling”
• Procedures used to obtain a sample
• Two basic approaches Sample

• Probability sampling
• Nonprobability sampling Population
 11

Approaches to Sampling (Cont.)

• Probability sampling
• All members of population have a specified chance of
being selected for the survey
• Simple random sample
• Random selection, random-digit dialing
• Stratified random sample
• Divide population into strata and sample proportionally
• Improves representativeness of sample
 8

Approaches to Sampling (Cont.)

Simple random sampling

• Example
1. Upon identifying target population for
research: Employees of food and
beverage industry in Selangor
2. Identify all companies in the F & B
industry in Selangor.
3. Generate a list of employees from
each company
4. Use a random number generator to
select the required number of
employees for your sample from the
list
Stratified Random Sampling
• Example:
1. Upon identifying target population for
research: Undergraduate students from
private universities
2. Consider the proportion that is representative
of undergraduate university students (e.g.,
60% First-year students, 30% Second-year
students, 10% Third-year students).
3. Thus, you need to calculate the proportion of
1st year, 2nd year and 3rd year students
required out of a total of N students (targeted
sample size)
4. Out of a sample of 200 students, stratified
sample obtained = 120 First-year students, 60
Second-year students and 20 First-Year
students
Systematic random sampling

• Example:
1. Upon identifying target population
for research: Employees of food
and beverage industry in Selangor
2. Identify all companies in the F & B
industry in Selangor.
3. Generate a list of employees from
each company
4. In each list, select every 10th
employee in sequence until you
obtain the required number of
samples for your sample
Cluster random sampling

• Example:
1. Upon identifying target population for
research: Employees of food and beverage
industry in Selangor
2. Divide sample into clusters: Director, Senior
Manager, Manager, Executive, Steward
3. Randomly select the clusters from all
companies in the F & B industry in Selangor –
assign number, random number generator
4. Select until you achieve the required number
of samples that is adequate for study
 12

Approaches to Sampling (Cont.)

• Nonprobability sampling
• No guarantee each member of population has an equal
chance of being in the sample
• “Convenience sampling”
• Individuals are available and willing to respond to the survey
• Example: magazine surveys, call-in radio surveys
• Sample likely not representative of population
 9

Approaches to Sampling (Cont.)

Convenience Sampling
• Example:
1. Upon identifying the target
population for sampling –
members of public
2. Choosing location for sampling
that is able to easily provide
required numbers – shopping mall,
grocery store, nearby university
campus, community centre etc.
Quota Sampling
• Example:
1. Identify target population to research
preference of cosmetic brand in Perak,
ages between 30 – 40 years
2. Decide on two subgroups (male and
female consumers) and allocating equal
proportions (e.g., 100 for each group) or
based on gender proportion of cosmetic
consumers
3. Utilise convenience sampling or
interview people who meet consumer
criteria to recruit required sample
proportions
Judgemental/Purposive Sampling
• Example:
1. Identify the targeted sample for
research: cancer patients undergoing
clinical trial treatment
2. Investigate the population of cancer
patients and calculate target sample size
3. Screen participants based on the criteria
of having medical diagnosis of Stage 1
and 2 cancer for at least 1 year and are
enrolled in a clinical treatment trial for
past 6 months
Snowball Sampling
• Example:
1. Upon identifying the target
population for sampling: academic
staff from private higher education
institutions in Perak
2. Approach a participant from each
higher education institution in Perak
3. Request their participation and ask
them to recruit other colleagues and
subsequently, ask them to recruit
other people
4. The recruitment is repeated until
required sample size is obtained
 5

Biased Samples
• A biased sample
• Characteristics of the sample differ systematically from those of
the population.
• Sample over-represents or under-represents segment(s) of a
population
• Eg. (300 students) 70% female and 30% male
 6

Biased Samples
• Two sources
• Selection bias
• Researcher’s procedures for selecting sample cause bias
• Response-rate bias
• Individuals selected for the sample do not return or complete
the survey
 13

Survey Methods
• Four methods for obtaining survey data
• Mail surveys

• Personal interviews
• Telephone interviews
• Internet surveys

• Each method has advantages and disadvantages.

• Choose method based on research question.
1. Mail surveys

• Quick and convenient, self-

administered, best for highly personal or
embarrassing topics.

• May have the problem of response

bias when people selected for the
survey sample don’t complete and
return their survey.
Mail surveys (cont.)
• Due to response bias, the final sample may not be
representative of the population.

• Because mailed surveys are self-administered, respondents are

free to interpret questions as they see fit, leading to possible
differences in how people respond to questions.
2. Personal Interviews

• Costly, but researchers gain more

control over how the survey is
administered, and how people
interpret survey questions.

• Interviewers can seek clarification

of answers.
Personal Interviews (cont.)
• Potential problem: interviewer bias

• Interviewer bias occurs when interviewer records only

selected portions of respondents’ answers, or interviewer
words questions differently to fit particular respondents.

• Interviewers must be highly motivated, carefully trained, and

supervised.
3. Telephone Interviews
• Brief surveys can be completed
efficiently and with greater
access to the population.

• Random-digit dialing technology

allows researchers to select
random samples.
Telephone Interviews (cont.)
• Interviewers can be supervised easily from one location.

Potential problems include:

• selection bias (only people with phones can be included)
• response bias (people may refuse solicitations to complete
surveys over the phone)
• interviewer bias
4. Internet Surveys
• The Internet allows for efficient, low-cost means to survey very large
samples.

• Samples can be very diverse and access typically underrepresented samples.

Potential problems include:

• selection bias (access to computers and Internet required)
• response bias
• lack of control over the research environment
 14

Survey Methods (Cont.)

• Ways to increase response rate
• Responding requires minimal effort
• Topic of survey is interesting to
respondents
• Respondents identify with
organisation or sponsor of survey
 15

Survey Research Designs

• “Research design”
• A plan for conducting a research project
• Choose method best suited for answering a particular
question
• Three types of survey research designs
• Cross-sectional design
• Successive independent samples design
• Longitudinal design
 16

Survey Research Designs (Cont.)

1 . Cross-sectional survey design
• Select sample from one or more populations at one time
Example: adolescent, young adults, senior adults
• Survey responses are used to
• Describe population (descriptive statistics)
• Make predictions for the population (correlations)
at that one point in time
• Compare populations
• Cannot assess change over time
 17

Survey Research Designs (Cont.)

2 . Successive independent samples design
• A series of cross-sectional surveys over time
• A different sample from the population completes the
survey each time.
• Each sample is selected from the same population.
Example: Jan 2017: 13 years old; Jan 2018: 13 years old;
Surveying independent samples at 1-month, 3-month and 6-
month milestones
• Responses from each sample are used to describe
changes in the population over time.
Survey Research Designs (Cont.)
• Problem:
Noncomparable successive samples – Are responses differing
because of true changes over time, or because different
populations were sampled?
 18

Survey Research Designs (Cont.)

3 . Longitudinal survey design
• Same sample of individuals completes the survey at
different points in time
• Assess how individuals change over time
• Responses from the sample are generalised to describe
changes over time in the population.
• Problems: attrition (loss of participants/dropout) and
reactivity (Participants may strive to be consistent or
become sensitised to the topic)
 19

Measures in Survey Research

• Questionnaires
• Most frequently used to collect survey data
• Measure different types of variables
• Demographic variables using checklists
• Preferences, opinions, and attitudes
• Self-report scales
• Rating scales (assume interval level of measurement)
• All measures must be reliable and valid.
Steps to Develop a Questionnaire
1. Decide what information should be investigated
2. Decide how to administer the questionnaire
3. Write a first draft of the questionnaire
4. Reexamine and revise the questionnaire
5. Pretest the questionnaire
6. Edit the questionnaire and specify the procedures for its use
Quiz!
 20

Reliability
• Reliability refers to consistency of
measurement.
• Test-retest reliability
 Administer measure two times to same
sample
 High correlation between the two sets of
scores indicates good reliability (r > .80)
• Inter-rater reliability
 Administer same measure to different
raters and compare responses
 21

Reliability (cont.)
• How to improve reliability?
• More items
• Greater variability among individuals on the factor being
measured
• Testing situation to be free of distractions
• Clear instructions
• A measure can be reliable but not valid.
 22

Validity
• Validity refers to the truthfulness of a measure.
• Assesses what it is intended to measure
• Construct validity
• Instrument measures the theoretical construct it was designed to measure.

• Convergent validity
• Extent to which two measures of the same construct are correlated (go together)
• Discriminant validity
• Extent to which two measures of different constructs are not correlated (do not go
together)
 24

Validity (cont.)
• Face validity: Subjective evaluation of whether test measures what it claims to
measure
 Recruiting people who understand the research topic – check if the items
captured the concepts accurately
 Enlist a field expert to check the construct and structure of questions to
eliminate double, confusing and leading questions

Statistical method of checking validity?

 Run a Principal Component Analysis (Factor Analysis) – pilot study
 Degree of Extent in
confidence which
that findings can
relationship be
tested is not generalised
due to other to other
factors context
 25

Thinking Critically About Survey Research

• Correspondence Between Reported and Actual behaviour
• Survey responses may not be truthful.
• Reactivity
• Social desirability
• Accept people’s responses as truthful unless there’s reason to
suspect otherwise.
• Use a multimethod approach to answering research questions.
 26

Thinking Critically About Survey Research (Cont.)

• Correlation and causality
• “Correlation does not imply causality”
• Three possible causal inferences for any correlation
• A causes B
• B causes A
• Variable C causes both A and B
 27

Thinking Critically About Survey Research (Cont.)

• Path analysis
• Statistical procedure to tease apart complex correlational
relationships among variables

• Mediators
• Variables used to explain a correlation
between two variables
• Moderators
• Variables that affect direction or
strength of correlation between two
variables
 29

Thinking Critically About Survey Research (Cont.)

• Path analysis example
• A moderator variable may affect the direction and strength of
these relationships.
• Possible moderators:
• Sex of the child
• Population density
(e.g., rural, urban)
• Personality features of children
(e.g., resilience)
 28

Thinking Critically About Survey Research (Cont.)

Diagram of direct and indirect relationships

Creative Process
Engagement
path a path b

Shyness Creativity
path c’ (direct)
path c(total effect: without mediator)
Creative process engagement mediates the relationship between shyness and
creativity among undergraduate students OR

Shyness has an indirect effect on creativity via creative process engagement

 30

Thinking Critically About Survey Research (Cont.)

• Path analysis
• Helps us to understand relationships among variables
(e.g., why are they related? How is motivation related to performance?)
• But these relationships are still correlational
• Cannot make definitive causal statements
• Other untested variables may be important
The End of Topic
4 lecture!
•Disclaimer:

The information provided are part

of educational materials created
for the course UAPG1004 and
are not to be redistributed to any
parties outside of this course.