RESEARCH IN EDUCATION
Constructing the Survey Instruments
BEEd 3-1 — Angeles, Cerono, Dela Cruz, Jalbuena, Saludades, Santos
Definition of Survey
▪ Survey research is one of the most important
areas of measurement in applied social research.
▪ The broad area of survey research encompasses
any measurement procedures that involve asking
questions of respondents.
▪ A “survey” can be anything from a short paper-
and-pencil feedback form to an intensive one-on-
one in-depth interview.
Types of Survey
▪ Surveys can be divided into two (2) categories:
o Questionnaire
o Interview
Questionnaires are usually paper-and-pencil instruments
that the respondent completes. Interviews are completed
by the interviewer based on the respondent says.
▪ For instance, some people think that
questionnaires always ask short closed-ended
questions while interviews always ask broad
open-ended ones.
▪ But you will see questionnaires with open-ended
questions although they do tend to be shorter than
in interviews and there will often be a series of
closed-ended questions asked in an interview.
Questionnaire
Mail Survey
▪ There are many advantages to mail surveys.
o They are relatively inexpensive to
administer.
o You can send the exact same instrument
to a wide number of people.
o They allow the respondent to fill it out at
their own convenience.
▪ But there are some disadvantages as well.
o Response rates from mail surveys are
often very low.
o Mail questionnaires are not the best
vehicles for asking for detailed written
responses.
Group Administered Questionnaire
▪ A sample of respondents is brought together and
asked to respond to a structured sequence of
questions.
▪ Traditionally, questionnaires were administered
in group settings for convenience. The researcher
could give the questionnaire to those who were
present and be fairly sure that there would be a
high response rate. If the respondents were
unclear about the meaning of a question they
could ask for clarification. And, there were often
organizational settings where it was relatively
easy to assemble the group.
o In a company or business, for instance.
Difference between Group Administered
Questionnaire and Group Interview/Focus
Group?
▪ In the group administered questionnaire, each
respondent is handed an instrument and asked to
complete it while in the room.
▪ Each respondent completes an instrument. In the
group interview or focus group, the interviewer
facilitates the session.
▪ People work as a group, listening to each other’s
comments and answering the questions.
 ▪ Someone takes notes for the entire group – people facilitates the exchange of information between the
don’t complete an interview individually. research participants and the person or organization
carrying out the research.
Interviews
To give you a number of questions you might ask that can
Personal Interview
help guide your decision.
▪ The interviewer works directly with the
respondent. Unlike with mail surveys, the Population Issue
interviewer has the opportunity to probe or ask
The first set of considerations have to do with the
follow-up questions. And, interviews are
population and its accessibility.
generally easier for the respondent, especially if
what is sought is opinions or impressions. Is the population literate?
Interviews can be very time consuming and they ▪ Questionnaires require that your respondents can
are resource intensive. read.
▪ The interviewer is considered a part of the ▪ While this might seem initially like a reasonable
measurement instrument and interviewers have to assumption for many adult populations, we know
be well trained in how to respond to any from recent research that the instance of adult
contingency. illiteracy is alarmingly high.

Telephone Interview What are the geographic restrictions?

▪ Telephone interviews enable a researcher to ▪ Is your population of interest dispersed over too
gather information rapidly. broad a geographic range for you to study feasibly
▪ Most of the major public opinion polls that are with a personal interview?
reported were based on telephone interviews. ▪ It may be possible for you to send a mail
▪ Like personal interviews, they allow for some instrument to a nationwide sample. You may be
personal contact between the interviewer and the able to conduct phone interviews with them.
respondent. And, they allow the interviewer to
Are there language issues?
ask follow-up questions. But they also have some
▪ We live in a multilingual world. Virtually every
major disadvantages.
society has members who speak other than the
▪ Many people don’t have publicly-listed telephone
predominant language.
numbers. Some don’t have telephones. People
▪ Some countries (like Canada) are officially
often don’t like the intrusion of a call to their
multilingual. And, our increasingly global
homes. And, telephone interviews have to be
economy requires us to do research that spans
relatively short or people will feel imposed upon.
countries and language groups.
Selecting the Survey Method

A survey method is a process, tool, or technique that you

can use to gather information in research by asking
questions to a predefined group of people. Typically, it
 Sample Issue
The sample is the actual group you will have to contact in
some way.
There are several important sampling issues you need to
consider when doing survey research.
What data is available?
▪ What information do you have about your
sample?
▪ Do you know their current addresses? Their
current phone numbers?
▪ Are your contact lists up to date?
Can respondents be found?
▪ Can your respondents be located?
▪ Some people are very busy. Some travel a lot.
Some work the night shift. Even if you have an
accurate phone or address, you may not be able to
locate or make contact with your sample.
Who is the respondent?
▪ Well, respondent has been for many years the
usual term for an individual who takes part in
research project. However, this is increasingly
replaced by the term 'participant' as researchers
and clients recognize the value of a more
collaborative interviewing relationship.
Sample Issue
Sometimes the nature of what you want to ask
respondents will determine the type of survey you select.
What types of questions can be asked?
▪ Are you going to be asking personal questions?
▪ Are you going to need to get lots of detail in the
responses?
▪ Can you anticipate the most frequent or important
types of responses and develop reasonable
closed-ended questions?
How complex will the questions be?
▪ Sometimes you are dealing with a complex
subject or topic.
▪ The questions you want to ask are going to have
multiple parts.
▪ You may need to branch to sub-questions.
Will lengthy questions be asked?
▪ If your subject matter is complicated, you may
need to give the respondent some detailed
background for a question.
▪ Can you reasonably expect your respondent to sit
still long enough in a phone interview to ask your
question?
Content Issue
The content of your study can also pose challenges for the
different survey types you might utilize.
Can the respondents be expected to know about the
issue?
▪ If the respondent does not keep up with the news
(e.g., by reading the newspaper, watching
television news, or talking with others), they may
not even know about the news issue you want to
ask them about.
Will the respondent need to consult records?
▪ Even if the respondent understands what you are
asking about, you may need to allow them to
consult their records in order to get an accurate
answer.
o For instance, if you ask them how much
money they spent on food in the past
month.
 ▪ In this case, you do not want to
be involved in an interview
where they would have to go
look things up while they keep
you waiting.
Bias Issue
People come to the research endeavor with their own sets
of biases and prejudices. Sometimes, these biases will be
less of a problem with certain types of survey approaches.
Can social desirability be avoided?
▪ Respondents generally want to “look good” in the
eyes of others. None of us likes to look like we do
not know an answer.
Can false respondents be avoided?
▪ With mail surveys it may be difficult to know
who actually responded.
▪ It is hard to convince respondents to go against
what is acceptable even when things are clearly
good. The respondents will have a propensity to
provide the socially acceptable response over the
true response.
Administrative Issue
Last, but certainly not least, you have to consider the
feasibility of the survey method for your study.
Cost
▪ This often the major determining factor in
selecting survey type.
▪ You might prefer to do personal interviews, but
cannot justify the high cost of training and paying
for the interviewers.
▪ You may prefer to send out an extensive mailing
but cannot afford the postage to do so.
Time
▪ Some types of surveys take longer than others.
▪ Do you need responses immediately (as in an
overnight public opinion poll)?
Facilities
▪ Do you have the facilities (or access to them) to
process and manage your study?
▪ In phone interviews, do you have well-equipped
phone surveying facilities?
▪ For focus groups, do you have a comfortable and
accessible room to host the group?
Constructing Survey
Constructing a survey instrument is an art in itself.
There are numerous small decisions that must be made –
about content, wording, format, placement – that can have
important consequences for your entire study.
There are three areas involved in writing a question:
▪ Determining the question content, scope and
purpose.
▪ Choosing the response format that you use for
collecting information from the respondent.
▪ Figuring out how to word the question to get at
the issue of interest.
Close-ended Question
▪ Closed-ended questions have a limited set of
options that the respondents need to select from.
▪ They can be either questions that can be answered
with a Yes/No or a multiple-choice format like A,
B, C or D.
Open-ended Question
▪ These types of questions are completely opposite
of close-ended questions.
 ▪ Open-ended questions allow the respondents to
freely express their opinion in their own words.
▪ There are no options, but just a question to which
the respondent must write an answer as clearly as
possible.
Types of Survey Questions
Survey questions are classified in to two:
▪ Structured questions
o This is when the questions provided has
a fixed answer like multiple choice,
yes/no questions etc.
▪ Unstructured questions
o These are questions with open-ended
answers, and the respondent does not
have a fixed answer.
Dichotomous Questions
▪ This refers to questions with two possible
answers.
o Closed-ended questions
▪ Requires a one-word answer.
▪ Example: Are you a
resident of Manila? Yes
or No
Question Based on Level of Measurement
▪ Rating questions
o These are questions that answers are map
onto a numeric scale.
▪ Likert Scale questions
o A question that uses a 5 or 7-point scale,
sometimes referred to as a satisfaction
scale, that ranges from one extreme
attitude to another.
o Typically, the Likert survey question
includes a moderate or neutral option in
its scale.
Filter or Contingency Question
▪ Questions that are only to be answered by some
subgroup(s) of respondents.
o Example: Are you a resident of Manila?
Yes or No. If yes, how long have you
been staying in Manila?
Open-ended Questions
▪ Questions that cannot be answered with a simple
'yes' or 'no', and instead require the respondent to
elaborate on their points.
▪ Open-ended questions help you see things from a
customer's perspective as you get feedback in
their own words instead of stock answers.
o Example: Where do you see yourself in 5
years?
Multiple Choice Questions
▪ An assessment item consisting of a stem, which
poses the question or problem, followed by a list
of possible responses, also known as options or
alternatives.
Question Content
Each question in the survey should be addressing the
needed information that needs to be gathered for the
research. Here are some questions that we may consider
when creating survey questions.
Is the question necessary?
▪ Decide whether each question is really needed to
avoid fillers and questions that don't really
address the topic.
Does the respondent possess the necessary/needed
information?
▪ This is important because if the respondents don’t
possess the needed information/ characteristics
that we need for the research, then the answers
may not be as reliable.
Does the question need to be more specific?
▪ This is important to ensure that the survey
questions would really target the needed
information.
Is the question biased?
▪ This has to be considered for we want the
research to be unbiased and reliable.
Question Wording
The choice of words and phrases in a question is critical
in expressing the meaning and intent of the question to the
respondent and ensuring that all respondents interpret the
question the same way. Even small wording differences
can substantially affect the answers people provide.
▪ Answer the following questions:
o Can the question be misunderstood?
o What assumptions does the question
make?
o Is the time frame specified?
o How personal is the wording?
o Is the wording too direct?
▪ Identical question wording should be used when
the intention is to compare results to those from
earlier surveys.
Question Placement
Question placement is important because this determines
the order or the placement of the questions and to ensure
that the important questions would really be given the
attention that it needs without being introduced too early
in the survey that it makes the respondents uncomfortable
to answer difficult questions or too far in the end that it
gets neglected.
Question Placement Components
▪ Opening Questions
o These are the easy and general questions
that the respondents will have to answer
first to allow them to be comfortable
before answering the much harder or
sensitive questions.
▪ Sensitive Questions
o Sensitive questions are inevitable in
some survey topics therefore, knowing
how to properly address this is important.
o This kind of questions must be put near
the end but not too far that it gets
neglected for the respondents are already
tired of answering.
o A good transition between general and
easy questions to sensitive questions
would be helpful wherein a good
transition could be something like,
▪ “In this part of the survey, we’d
like to ask more personal
questions regarding (a certain
topic).” and to assure them that
they don’t have to answer the
questions if they are
uncomfortable.
Practices for Creating an Effective Survey
Instrument
The secret to designing a survey that works is to
keep validity, reliability, replicability, and
generalizability in mind.
Knowing what you want to learn is the first step in
any study project.
Determine the Objectives
▪ Start with your hypothesis
▪ Have a clear idea of what decision you need to
make, and what information you need to make it.
o What is the purpose of the survey?
o What are we trying to measure?
o How many questions should we include?
o What type of rating scale should we use?
o How will we know that the survey
worked — what will make the data
actionable?
Design the Questionnaire
The most crucial step in survey instrument design is
making sure you can acquire the information and
outcomes your organization need.
Ask forthright, clear-cut, straightforward, and unbiased
inquiries. Make sure every query is objectively
"measuring" something.
▪ Begin with title and a preamble that explains the
overall aim of the survey
o This alerts respondents up front about
things like the length of the survey and
the degree of confidentiality, whether it
is part of the survey invitation or at the
start of the survey itself.
▪ Balance white space to improve readability
o If you want to make the survey easier to
read without unnecessarily extending its
perceived size, you should utilize white
space sparingly between the questions
and the sections.
▪ Include easy to understand instructions and
questions
o Keep instructions and questions at the
eighth-grade level or lower, without
being condescending.
All instructions, question directions, and response
categories need to be clear, especially with written
surveys.
Use as few words as possible in both the question and the
alternatives and avoid the use of polysyllabic words.
Provide general instructions
▪ Provide general instructions to the respondents at
the beginning of each section and clearly define
specific instructions associated with the different
question types to order to aid in the correct
completion of that question.
▪ This includes phrases such as
o “Please check one box only”
o “Please rank in order of 1 to 5” with 1
being most important.
Ask only one question at a time
▪ It is not uncommon to ask what we call double-
barrel questions, such as “how satisfied are you
with cost and convenience of….?”
o With this question, you will not know
whether the response applied to cost or
convenience.
▪ Use the language of your respondents.
Avoid biasing questions
▪ Avoid questions like “what did you dislike
about…?” and then only offering response
categories to support the questions, which then
forces the respondent to bias their answer.
 ▪ A better way to approach this question is to ask a
qualifying question first such as, “Did you dislike
anything about…? Yes or No.”
o Those that say yes go on to the question
that asks them what did they dislike.
▪ These questions are important to the design of any
type of survey, whether it be a customer
satisfaction survey, a product evaluation survey,
▪ Determine a rating scale
or a program evaluation survey.
▪ It often helps to involve multiple departments in
the process in order to gain consensus. When all
units are involved upfront, your chances of asking
the right questions and not collecting unnecessary
data are greatly improved.
Pre-test your surveys
Ask the tester to let you know if there are any words that
they do no regularly use, or did not understand.
o Organize the survey into sections or
topics with the easier items first.
o Keep sentences to 8 words or less and
limit the number of questions per page.
o Extensively long surveys lead to fatigue
and high abandonment rates.
o The shorter the better.
o Scales are critical to the success of your
research.
o Well-designed scales are easy to
understand and accurately represent the
respondent’s true attitude, preference or
opinion.
o Clear and well thought out rating scales,
as well as clearly defined instructions, are
key to minimizing rating errors.

▪ Select the sample and define your respondents

▪ Use filter questions
o It is important to have a sufficient sample
o Filter questions make it possible for
size if you want the results to be statically
respondents to bypass questions (or
meaningful.
whole sections) that are not relevant to
o Factors that affect a sample size are how
them
large the group difference you wish to
▪ “don’t know” and “not
detect, how variable your measure is in
applicable”
the sample or populations, and how
o This will help ensure people don’t select
precise you want the results to be.
something just because there is no other
o In order to achieve an accurate sample,
option available to them.
you must decide on a level of granularity
o However, when a large number of
to reach a conclusion.
respondents choose such options, it is
o Develop a process for finding these
time to examine whether the question is
respondents and engaging them.
badly worded, or in the wrong place in
the questionnaire, or if the question is
Survey Implementation
reaches the objectivity of the study.
A study's reliability is impacted by its low response rate.
▪ Keep it short, simple, and focused Before launching your survey, always test it.
o Keep in mind what you want to know,
and who you need to know it from.
The response rate is the single most significant indicator
of how much trust may be put in survey results.

What the survey author may consider to be obvious may

not make any sense to the usual receiver. Nevertheless,
even worse, a challenging question may be misconstrued
or skipped, and a challenging survey is almost guaranteed
to be ignored.

▪ Plan ahead for survey reminders.

▪ Make sure the respondents are qualified to answer
the questions, otherwise they may feel compelled
to make up an answer.

Analyze and Report the Results

▪ When gathering and analyzing data, follow a

methodical process.
▪ Describe your findings in objective, fact-based
sentences.
▪ Validity should be taken into consideration while
reviewing research findings. Analyzing the
proportion of respondents who selected each
response option allows one to judge the quality of
the questions.
▪ No answer should receive more than 85% of the
votes, and no answer should receive less than 5%.

Research is both a science and an art. The goal is to derive

insights from the data to foster more confident customer,
market, and product decisions. Be sure you have the
science under your belt.