Metode Penelitian

Dr. Dedi Walujadi, MA

Sesi 5, Goodness of Measures and The Nature of
Questionnaire Survey
 Outcome pembelajaran
• Mahasiswa dapat mengimplementasikan konsep validitas dan reliabilitas
dalam penelitian yang akan dilakukan.
• Mahasiswa dapat menerapkan teknik pengumpulan data yang sesuai, tepat,
dan menghasilkan data yang berkualitas sesuai tujuan penelitian melalui
perancangan questionnaire, meminimumkan bias, dan menerapkan cara
pengumpulan data.

Metode Penelitian Sesi 5 2019 2

 Outline
• Goodnees of Measures: Discuss validity and reliability of measures
• The Nature of Questionnaire Survey
• Questionnaire Design:
• design questionnaires to tap different variables
• evaluate questionnaires, distinguishing the ‘good’ and ‘bad’ questions therein
• identify and minimise the biases in collecting data with questionnaires
• discuss how a pilot study can be used to prove the structure of a questionnaire
• discuss the advantages and disadvantages of the different ways of
administering questionnaires

Metode Penelitian Sesi 5 2019 3

Goodness of
Measures
4
 Goodness of Measures
• Reliability
• A matter of whether a particular technique, applied repeatedly to the same
object, yields the same result each time.
• How stable and consistent is the measuring instrument?
• Validity
• The extent to which an empirical measure adequately reflects the real meaning
of the concept under consideration.
• Are we measuring the right thing?
Metode Penelitian Sesi 5 2019 5
 Reliability
• Stability
• refers to the ability of a measure to maintain consistency over time, despite
uncontrollable testing conditions or the state of the respondents themselves
• Internal consistency
• indicates how well the items ‘hang together as a set’ and can independently measure the
same concept, so respondents attach the same overall meaning to each of the items

Metode Penelitian Sesi 5 2019 6

 Forms of validity
• Face validity
• That quality of an indicator that makes it seem a reasonable measure of a variable.
• Criterion related validity
• The degree to which a measure relates to some external criterion. For example, the
validity of the VCE tests is shown in their ability to predict the college success of
students.

Metode Penelitian Sesi 5 2019 7

 Forms of validity (cont.)
• Construct validity
• The degree to which a measure relates to other variables as expected within a system of
theoretical relationships.
• Content validity
• Refers to how much a measure covers the range of meanings included within a concept.

Metode Penelitian Sesi 5 2019 8

 The Nature of
Questionnaire Survey
9
 The Nature of Questionnaire Surveys
(Veal)
• Data collected via a printed questionnaire or interview
schedule
• Usually based on a sample selected from a population
(see Chapter 11)
• Reliance on information from individuals – self-reported
data
• Often large samples require computer-aided analysis

Metode Penelitian Sesi 5 2019 10

Merits of the Questionnaire Survey Method

• Quantified data for decision making

• Ideal for providing quantified information for organisations which need quantified
information for decision-making.
• Transparency
• Provide a 'transparent' set of research procedures.
• Procedures clear for all to see.
• Data can be re-analysed for alternative interpretation.

Metode Penelitian Sesi 5 2019 11

 Merits of the Questionnaire Survey Method
(continued)
• Succinct presentation
• Quantification can provide complex information in a succinct form.
• Comparability
• Longitudinal and annual surveys enable the study of change over time.
• Capturing complexity
• An effective means of gathering a wide range of complex information on individuals or
organisations.

Metode Penelitian Sesi 5 2019 12

 Why do people answer questionnaires?
Social exchange theory (Blau, 1964, Dillman 1978)
Likelihood of response is a function of:
• Rewards: positive regard, verbal appreciation, helping,
group values, interest, social validation
• Costs: subordination, embarrassment, inconvenience,
complexity, privacy
• Trust: tokens of appreciation, sponsorship,
importance, other relationships
Metode Penelitian Sesi 5 2019 13
 When to use questionnaires
GOOD FOR
• Descriptive research
• Explanatory research

BUT NOT FOR

• Exploratory research

• Consider MULTIMETHOD approaches

Metode Penelitian Sesi 5 2019 14
 Interviewer-completion vs Respondent-completion
Figure 8.1 Interviewer Respondent
Advantages -More accuracy -Cheaper

-Higher response rates -Quicker

-Fuller, more complete -Relatively anonymous

answers
-Design can be less ‘user-
friendly’
Disadvantages -Higher cost -Patchy response
-Less anonymity -Incomplete response

-Exaggerated desire to please -Risk of frivolous response

interviewer More care needed in
layout/presentation

Metode Penelitian Sesi 5 2019 15

 Types of questionnaire
(Veal, Figure 8.2)

Self-administered
• Mail surveys - questionnaires sent and
returned by mail;
• E-surveys - questionnaires sent and
returned by email/Internet;
• Captive group surveys - members of
groups are surveyed
• Delivery and collection
• Organisation surveys - members of an
organisation/ organisations are surveyed
Interviewer-administered
• Household surveys – respondents selected
on the basis of where they live and
interviewed in their home
• Street/quota/intercept surveys -
respondents selected by stopping in
street, shopping malls, etc
• Telephone surveys - interviews are
conducted by telephone
• Customer/visitor on-site intercept surveys –
users/customers surveyed on site

Metode Penelitian Sesi 5 2019 16

 Household surveys
Advantages Disadvantages
• Rapport and motivation for • Take personal time
resp.
• Costs are higher
• Clarification possible
• Confidentiality concerns
• Read non-verbal cues
• Interviewer training vital
• Visual aids possible
• Interviewer biases possible
• Richer data
• CAPI available

Metode Penelitian Sesi 5 2019 17

 Telephone Interviews
• Less costly and faster than • Respondents can terminate at
mail or personal any time
• Wide geographical area • Non-verbal cues not available
• Higher anonymity than • Need to be kept short
personal (AMSRS - 20 min.)
• Can be completed using • Representativeness of
CATI - Computer-aided telephone sampling lists?
Telephone Interviewing
• Automatic sampling and
dialling
• Answers keyed directly into
computer (quick, cost-
effective)

Metode Penelitian Sesi 5 2019 18

 Mail Questionnaires
• High anonymity
• Wide geographic regions
• Can include token gifts
• At respondent convenience
Metode Penelitian Sesi 5 2019
• Relatively low response rate
• No opportunity for
clarification
• Follow-up procedures
required
• Total design method (Dillman,
1978)
19
 Mail survey response rates
• Response rate affected by:
• Interest of the respondent in the survey topic
• Length of the questionnaire
• Questionnaire design, presentation, perceived complexity
• Style, content and authorship of accompanying letter
• Provision of a reply-paid envelope
• Rewards for responding
• Number/timing of reminders/follow-ups – see Fig. 8.3
Metode Penelitian Sesi 5 2019 20
 Mail survey response pattern

Metode Penelitian Sesi 5 2019 21

 Electronic Questionnaires

• Easy to administer • Require computer literacy

• Global reach • Respondents must have
• Inexpensive computer access

• Fast • Must be willing to complete

• At respondent’s convenience

Metode Penelitian Sesi 5 2019 22

 Captive group survey
• Survey in an organised setting – participants may have little choice
• But research ethics requires voluntary participation …
• Respondent completion
• Very cheap and quick

Metode Penelitian Sesi 5 2019 23

Questionnaire Design

24
 Questionnaire design (Veal)
• Topics:
• Research problems and information requirements
• Types of information
• Open-ended and pre-coded questions
• Wording of questions
• Measuring attitudes and opinions
• Ordering of questions
• Layout
• Filters
• Introductory remarks

Metode Penelitian Sesi 5 2019 25

 Research problems & information
requirements
• Questionnaire design/content should arise from the research problem and its
information needs – see Figure 8.4
• Respondent characteristics - Who?
• Activities and behaviour - What?
• Attitudes and motivations - Why?
• Essential data – specified in a data requirements table

Metode Penelitian Sesi 5 2019 26

 Questionnaire design process

Metode Penelitian Sesi 5 2019 27

 Type of Questions
• Open-ended questions
• Freedom to answer the question any way one wants
• Could pose problems for researcher in coding the response
• Closed questions
• Alternatives that are mutually exclusive and collectively exhaustive, given
• Respondent can make a quick decision
• Easy to intepret and code

Metode Penelitian Sesi 5 2019 28

 Open-ended and pre-coded questions

Metode Penelitian Sesi 5 2019 29

 Wording of questions
• Avoid jargon
• Simplify wherever possible: Would the respondent understand the words in the
questionnaire?
• Avoid ambiguity
• Avoid leading questions
• Ask only one question at a time (avoid multi-purpose questions) - Are there double-barrelled
questions?
• Are there any recall-dependent questions?
• Is there any potential of social desirability?
Metode Penelitian Sesi 5 2019 30
 Validity of questionnaire surveys
• Validity: the extent to which the data truly reflect what they are thought to
reflect
• Validity may be affected by:
• Exaggeration or under-statement (to give good impression)
• Halo effects
• Inaccuracy of recall (recency bias)
• Desire to please the interviewer
• Order effects
• Some checks:
• Include ‘dummy’ categories in some questions– eg. books that do not
exist, events that have not happened
• As the same question twice – in different way
• Repeat interviews – some time later
Metode Penelitian Sesi 5 2019 31
 Possible sources of questions
• ADOPT questions used in other questionnaires
• Generally already tested for validity
• Can compare consistency of responses (i.e. reliability)
• ADAPT questions used in other questionnaires
• DESIGN your own questions

Metode Penelitian Sesi 5 2019 32

 Forms of closed questions
(SLT p.292)

• List (tick all that apply)

• Category (which one applies to you?)
• Ranking (relative importance)
• Rating (agree-disagree, good-bad, …)
• Quantity (how many – record a number)
• Grid (multiple question responses)

Metode Penelitian Sesi 5 2019 33

 Measuring attitudes and opinions
• Formats
a.Open-ended or direct questions
b.Checklist
c.Ranking
d.Likert scales
e.Attitude statements
f. Semantic differential
Metode Penelitian Sesi 5 2019 34
 Classification Data –
Personal Information or Demographic Data
Decisions on:
• Name, number required? (anonymity)
• What info required? (income, marital status, etc needed?)
• Age, income, etc, should a range be given? If so, what are the
appropriate ranges?
• Eg, <20, 20 – 40, >40
• Should personal info be at beginning or end of questionnaire?
Decisions should be based on the research questions you need to answer!!
Metode Penelitian Sesi 5 2019 35
 Ordering of questions
• Start with easy questions
• Start with 'relevant' questions
• Leave sensitive questions until later

Metode Penelitian Sesi 5 2019 36

 Sequencing of Questions
The Funnel Approach
• From general to specific
questions
• From easy to difficult questions

Metode Penelitian Sesi 5 2019 37

 Questionnaire layout
• Be aware of reader/user – interviewer or respondent?
• Special care with mail survey questionnaires
• Compactness (eg. single page) = ease of handling
• Two-column layout often helps

Metode Penelitian Sesi 5 2019 38

 Filtering

Metode Penelitian Sesi 5 2019 39

 Introductory remarks
• Eg. ‘Hallo, my name is _____ : we are conducting a survey of
_______: would you mind answering a few questions?
• Different content/formats for different survey types
• Interviewer-completion: include in interviewer instructions – additional
information available if required
• Respondent-completion – printed on questionnaire
• Mail survey: this is dealt with in covering letter
Metode Penelitian Sesi 5 2019 40
 Introduction to respondents
Consider social exhange – increase benefits and trust, reduce
costs
• identity of researcher and purpose of survey
• confidentiality & anonymity
• distribution and/or publication of results
• courteous note, thanks
• Aim is to provide opportunity for INFORMED CONSENT
Options: covering letter, introductory statement

Metode Penelitian Sesi 5 2019 41

 Instructions
• Define a desired navigational path for each page
• Create visual navigational guides to assist respondents to adhere to
navigational path
• Develop additional visual guides for disruption where redirection is required

(Dillman, 2000: 96)

Metode Penelitian Sesi 5 2019 42

 Pre-testing Questionnaires
• Face validity
• Content validity
• Pilot study
• With sample of respondents from target population
• Ideal research design
• Factor analysis, reliability analysis

Metode Penelitian Sesi 5 2019 43

Alhamdulillah

44