Basics of survey

construction
Surveys
• Surveys are tools that measure variables through questions
• Censuses are surveys that tries to get an image of an entire population
• Self-reports are surveys that focus on the psychological measurements
of an individual
• Beliefs, feelings, intentions, behaviors
• Personality
• Three main types of surveys (in terms of structure):
• Structured
• Semi-structured
• Unstructured
Crafting a self-report
• The most common rating scales used in measurement are the likert
scale and the semantic differential scale
• The likert scale aims to quantify variables by asking questions with
answers of varying strength or valence
• Strongly disagree, Disagree, Agree, Strongly Agree
• Semantic differential scale uses two polar opposite options, and lets
the participant locate themselves within the scale
• (1) Afraid , 2 , 3, 4, 5 (Brave)
• Either way, the important part is to assign numbers for each option.
Other types of rating
• Checklists
• Usually better for quantifying behaviors or the manifestations of a particular
phenomenon (e.g. sickness)
• Forced ranking
• Used when there are different options or preferences
• Not recommended because different rankings are not equally distant from
each other
• The difference between rank 1 and 2 might different from rank 2 and 3. This varies from
participant to participant
Considerations for test quality
• Validity – How accurate your survey is
• Content validity – does the test measure everything that needs to be measured?
• Construct validity – do the items really measure the variable you are examining
• Criterion validity – does the test agree/disagree with gold standard
measurements
• Reliability – How stable the survey is
• Internal consistency – do the items in a variable set all measure the same thing
• Test-retest reliability – do the items exhibit relatively similar results over time?
Things to think about when crafting a test
• Clarify: What is the variable that you’re measuring?
• Every variable should have a set of questions. One item per variable will lead
to unreliable results
• What rating scale will you use?
• The typical scale used in studies is 1-5, however Filipinos have a tendency to
answer neutral in order to avoid any problem
• Using 1-6 will force the respondent to at least agree or disagree
• Strongly Disagree, Moderately Disagree, Slight Disagree etc.
• Do not use a scale more than 7 items, because it will lead to unreliable results
• How many items?
• Get the most amount of reliable information with the least amount of effort
Questions to avoid
• Leading questions – these are questions that prime the respondent to
a particular biased response
• This creates a skewed response set
• “I am satisfied with the excellent work of the government”
• Double-barreled questions – questions that put two questions and/or
contexts in one
• This creates confusion because the participants might have different
responses for the two questions/contexts
• “I am usually talkative in parties and in classes”
Questions to avoid
• Assumptive questions – questions that need assumptions or
hypothetical situations
• Problematic because they might not reflect the true attitudes of respondents
• “When I am tired, I tend to feel emotional”

• Absolute qualifiers – Avoid putting qualifiers that create absolutes or

extremes
• This creates a skewed response set, or worse, unreliable answers
• “I always disagree with my partner”
• In general, avoid qualifiers altogether
Questions to avoid
• Sensitive questions – as much as possible, do not push people to
answer sensitive questions.
• Creates discomfort, and unwanted bias because of social acceptability
• “I can be emotionally unhinged at times”
• Ambiguous questions – vague questions are hard to answer
• Will lead to unreliable answers
• “Being able to express who I am is an important part of who I think I am”
What kind of questions to include
• Cover all possible content, but do not be redundant
• Multiple angles or dimensions
• Questions that are related to the variable you are measuring
• As much as possible, include multiple items measuring various aspects of your
outcome instead of just one catch-all item
• Instead of asking “nakatulong po ba…” ask specific items on different variables that
capture the goals of the program
• Simple questions
General Tips
• Make it easy for the respondent to answer – the assumption is that
the respondent has no time for your data-gathering
• For short sessions, 10 questions per variable will suffice
• Make use of reversed questions – these questions measure the
opposite of what you want to know.
• They still measure your variable (but will be negatively correlated) and will
keep the participants on their toes and prevent shotgun responses