Professional Documents
Culture Documents
Questionnaire Design: Asking Questions With A Purpose: Program Development and Evaluation
Questionnaire Design: Asking Questions With A Purpose: Program Development and Evaluation
Program Development
and Evaluation
Questionnaire Design:
Asking questions with a purpose
Ellen Taylor-Powell
Program Development and Evaluation Specialist
May 1998
Originally published with Mary G. Marshall
Texas Agricultural Extension Service
The Texas A&M University System
College Station, Texas
■ ■ ■ TABLE OF CONTENTS
Constructing a questionnaire. . . . . . . . . . . . . . 2
Kinds of information . . . . . . . . . . . . . . . . . . . . . 2
KNOWLEDGEÑWhat people know . . . . . . . . . . . . . . 2
BELIEFSÑATTITUDESÑOPINIONS . . . . . . . . . . . . . 2
BEHAVIORÑWhat people do . . . . . . . . . . . . . . . . . . 3
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2 ■ ■ ■ P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N
Constructing a Kinds of
questionnaire information
A questionnaire provides a tool for eliciting A questionnaire can help you obtain informa-
information which you can tabulate and tion about what people do, what they have,
discuss. In many evaluations, a questionnaire what they think, know, feel, or want.
serves as the major source of information.
Four different types of information may be dis-
Writing questions and constructing a question-
tinguished. Any one or a combination of these
naire takes time and attention. Before you
types may be included in a questionnaire.
begin, it is essential to know what kind of evi-
dence you need to fulfill the purpose of the KNOWLEDGE—What people
study and to know how the information will be
know; how well they understand
used.
something
■ Make a list of what you want to know. his type of question asks about what people
What do you really want to find out?
What do you want to achieve with the
T know. Knowledge questions offer choices
such as correct vs. incorrect or accurate vs.
questionnaire?
inaccurate. They may ask what respondents
■ Check to see if the information you need is believe is true or factual, or about awareness.
already available somewhere else.
For example:
■ DonÕt ask a question unless it has a use;
that is, unless it relates to the purpose of What is the major cause of accidental deaths
the study. Collecting too much information among children inside the home?
adds to your time and expenses and can The most effective weight loss plan includes
produce an information overload. exercise.
Eliminate all the Ònice to knowÓ items that The ideal refrigerator temperature is ________.
arenÕt really essential. Eliminate ambigu-
ous questions as well. BELIEFS—ATTITUDES—
■ From the beginning, think through what OPINIONS
you will do with each piece of information.
hese terms refer to psychological statesÑthe
What do you want to be able to say? Do
you expect to use frequencies, percentages,
T
perceptions people hold, their thoughts,
feelings, ideas, judgements, or ways of think-
rankings, multivariate analysis, narrative
ing. Questions may elicit respondentsÕ percep-
remarks?
tions of past, present or future reality, their
■ As you write questions, try to view them feelings about a subject, or their opinions.
through your respondentsÕ eyes.
For example:
Will the question be seen as
reasonable? Do you favor or oppose the reclassification of
Will it infringe on the respondentÕs forestry land?
privacy? In your opinion, does positive self-esteem
Will the respondent be able and willing prevent drug abuse among adolescents?
to answer the question? Do you think that lower prices would increase
■ Be selective and realistic. Know what beef consumption?
information is needed, why, and how you What do you consider the biggest challenge
plan to use it. facing our community in the next five years?
Q U E S T I O N N A I R E D E S I G N ■ ■ ■ 3
people about who they are, rather than what ■ Use simple wording. Adapt wording to
they do. the vocabulary and reading skills of your
respondents but donÕt talk down to them.
For example:
Are any words confusing? Do any words
Where do you currently live? have double meanings?
How many children do you have? ■ Avoid the use of abbreviations, jargon,
What percentage of your household income or foreign phrases. We use a lot of jargon
comes from off-farm employment? in Extension. Will the respondents under-
stand terms such as ÒCNRED,Ó Òlearning
To write meaningful questions, be clear about experiences,Ó Òlife skills,Ó or Òfocus
the objectives and type of information groupsÓ?
desiredÑwhether it is information about
■ Be specific. A question about older youth
knowledge, attitudes/beliefs/opinions, behav-
should specify what age or grade is con-
ior, or attributes. Otherwise, the questionnaire
sidered Òolder.Ó Likewise, in the question
may elicit opinions when the actual intent is to
ÒHow many times did your 4-H club meet
document behavior.
last year?Ó specify whether Òlast yearÓ
Likewise, questions related to each type of refers to 1994, 1994Ð1995, the last 12
information present different writing prob- months, or September 1994ÐAugust 1995.
lems. Questions concerning attitudes tend to ■ Use clear wording. Words such as Òregu-
be more difficult to write given the complexity larlyÓ and ÒoccasionallyÓ mean different
underlying most attitudes. Pay careful atten- things to different people. Some vague
tion to wording. In contrast, questions about terms include: majority (more than half of
knowledge, behaviors and attributes are more what?); often (daily? twice weekly?
straightforward. weekly?); government (state? federal?
local?); older people (how old?).
Remember, the response or information you
obtain is only as good as the question. If you
donÕt get the type of information you want, it
is probably because you didnÕt ask the right
question!
4 ■ ■ ■ P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N
Open-ended questions
n open-ended question is often the easiest
A question on one topic structured
in different ways
A way to ask for information, but the
responses are not easy to analyze. Answers are
1. Open-ended likely to be varied so you will need to catego-
What would you like to see as the main program rize and summarize them. Think about how
emphasis next year? you will analyze the narrative data.
Open-ended responses can be used to:
2. Close-ended with ordered responses
How important to you are each of the following ■ Stimulate free thought, solicit creative sug-
possible program emphases? (Circle one for each gestions, or recall information learned.
item.) ■ Probe for more detail.
None Little Some Much Examples:
A. Effective parenting 1 2 3 4 1. What do you think should be done to improve
B. Child development 1 2 3 4 the 4-H program in this county?
C. Guidance & discipline1 2 3 4 2. Name the five basic food groups.
D. Communications 1 2 3 4 3. Please indicate how you intend to use the infor-
mation obtained during the workshop.
3. Close-ended with unordered response
4. We are interested in any other comments you
choices
might have concerning your role as a volunteer
Which of these four topics would you most like to leader. Please write in the space below any
see as the primary program emphasis next year? thoughts youÕd like to share with us.
(Circle number of your answer.)
When asking for a numeric response, include
1 Effective parenting the unit of measurement to be used.
2 Child development 5. Please list the number of acres (if any) of tem-
3 Guidance and discipline porary pasture you planted in 1988.
4 Communication A. ____ Acres of wheat
4. Partially close-ended B. ____ Acres of oats
What topic do you feel should be the main program C. ____ Acres of rye grass
emphasis for next year? (Circle number of your D. ____ Acres of clover
answer.) E. ____ Acres of summer annuals
1 Effective parenting F. ____ Other, please specify
2 Child development
Open-ended questions are also appropriate
3 Guidance and discipline when respondents are asked to supply a spe-
4 Communication cific answer and a large number of responses
5 Other (please specify) are possible (see example 3 above) or when all
___________________ the possible answers are not known. They are
often used at the end of a questionnaire to ask
(Dillman, 1978)
respondents for additional comments
(example 4 above).
Q U E S T I O N N A I R E D E S I G N ■ ■ ■ 7
here are a variety of ways to write close- 1. What does the word ÒnutritionÓ mean to you?
T ended questions. Some require answers that
fall along an implied continuum (as in rating
(Circle one number.)
1 Getting enough vitamins
scales); others supply answers in no particular 2 The food you eat and how your body
order (lists). Some questions employ multiple uses it
choice options (Òcheck all that applyÓ); others
3 Having to eat foods I don’t like
provide relevant answers but allow respon-
dents to add others not in the list. The follow- 4 Having good health
ing section gives examples of close-ended Rating scale
questions. Often, respondents are asked to indicate their
choice at the most appropriate point on a scale.
Two-option responses
Whether you use a scale of three, four, five or
This is the simplest response format. Options
more categories depends on the question, the
may include: NoÐYes, DisagreeÐAgree,
amount of differentiation that is possible and
FalseÐTrue, OpposeÐFavor.
desirable, and the respondentsÕ capacity to
Example: answer.
1. Do you remove the clippings from your lawn Examples:
after mowing?
1. To what extent do you agree or disagree with
1 No the new zoning code? (Circle one.)
2 Yes 1 Strongly disagree
Depending upon the information you desire, 2 Mildly disagree
this may be the most appropriate format. It is
3 Neither agree or disagree
often used as the first question in a series of
questions on one topic. However, using a 4 Mildly agree
rating scale or a ranking (when appropriate) 5 Strongly agree
elicits more information. 2. When purchasing new herd bulls, how impor-
Starting with either positive or negative tant are the following traits in your selection
response options appears to have little effect process? (Circle one number for each selection
on response. Neither does it matter whether trait.)
Yes or No is listed first. But you do need to be Selection Of little Highly
consistent in the order you follow throughout trait importance important
the questionnaire. A. Performance 1 2 3 4 5 6 7
One best answer B. Conformation 1 2 3 4 5 6 7
These questions can be used to solicit informa-
tion or to test knowledge. They are appropriate C. Pedigree 1 2 3 4 5 6 7
when all relevant choices are known and D. Breed 1 2 3 4 5 6 7
listed. Respondents are provided with a list of
answers and asked to check or circle the choice For greater differentiation, use a numerical scale
they feel is the best. Responses are indepen- from 0 or 1 to some number (see the second
dent of each other, rather than gradations example above). A five-point option series
along a continuum. seems to be best for measuring attitudes; a
four-point option series appears useful for
ratings (excellent, good, fair, poor). Some
people may relate best to a 10-point scale.
8 ■ ■ ■ P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N
■ g. Applied for resale tax number 2. From the list provided, select THREE adjec-
■ f. Other tives which best describe you.
________________________________ (Place the letter of the adjective on the lines
provided.)
2. What information would you like covered in
1. _____ a. Ambitious
the next Extension workshop? (Check all
choices.) 2. _____ b. Happy
■ c. Disease control
■ d. Nursery layout
■ e. Weed control practices
■ f. Greenhouse management practices
Q U E S T I O N N A I R E D E S I G N ■ ■ ■ 11
Ranking
Rank ordering is a multiple-choice option.
Formatting the
Respondents are given various responses and questionnaire
asked to rank them in order of importance or After you have selected your questions, youÕll
indicate a Òtop three.Ó need to make a series of decisions about the
questionnaire formatÑits appearance, length,
Examples:
and the order in which the questions will
1. What would you like to know more about? appear. The questionnaire should be pleasing
Select three responses from the list and rank to the eye and easy to complete.
them in order of 1, 2, 3.
The following guidelines offer some tips to
1. ___ a. What to eat to look better
help you put the questionnaire together.
2. ___ b. How food affects you
■ Begin with an introduction that includes
3. ___ c. Weight control
the questionnaireÕs purpose, identifies its
d. Health foods source, explains how the information
e. Physical conditioning obtained will be used, and assures respon-
through diet dents of confidentiality. In mailed ques-
f. Vitamins
tionnaires, reinforce the points you made
in the cover letter.
2. What would you like to know more about?
Select three responses from the right hand ■ The first questions should be easy, avoid-
column and rank them in order of first, second, ing controversial topics. Write interesting
and third choice. questions that are clearly related to the
questionnaireÕs purpose. DonÕt use open-
1. __ 1st choice a. What to eat to ended or long questions with lengthy
look better answer choices in the beginning of the
2. __ 2nd choice b. How food affects questionnaire.
you ■ Address important topics early, rather than
3. __ 3rd choice c. Weight control late, in the questionnaire.
d. Health foods ■ Arrange questions so that they flow natu-
e. Physical conditioning rally. Keep questions on one subject
grouped together. Start with general ques-
f. Vitamins
tions and then move to those that are
specific.
■ Try to use the same type of question and
response throughout a series of questions
on a particular topic. For example, donÕt
needlessly break a respondentÕs concentra-
tion by using a multiple choice format fol-
lowed by a yes/no question, followed by
an open-ended question.
■ Place demographic questions (age, sex,
income level, etc.) at the end of the ques-
tionnaire.
■ Print it in an easy-to-read typeface.
■ A numbered response should mean the
same thing throughout the questionnaire.
12 ■ ■ ■ P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N
2 No
■ Use transitional statements to enhance
continuity. Transitional statements serve
■ Avoid making respondents turn a page in three functions: 1) to signal that a new
the middle of a question or between a topic is about to begin; 2) to start new
question and answer. pages; and 3) to break up the monotony of
■ Be sure that the question is distinguishable a long series of questions.
from the instructions and the answers. You Examples:
could put the instructions in boldface or
italics. Dillman (1978) suggests using Next, we would like to ask you several
lower case letters for questions and upper questions about the organizations you
case letters for answers. belong to in your community.
■ Questions and answers are easiest to read Another important purpose of this survey
if they flow vertically. By placing answer is to learn how you feel about the work of
choices under questions (rather than side service organizations in your community.
by side), the respondent moves easily Finally, we would like to ask a few ques-
down the page. If you feel this format tions about you to help us interpret the
results in too much wasted space, you may results.
wish to reorganize your questions. ■ Make sure that the respondent is referring
Example: to the same program mentioned in the
1 Excellent questionnaire and defining it similarly. A
Òvalidation itemÓ (Bennett, 1982) at the
2 Good beginning of the questionnaire identifies
3 Fair the program and sets the stage for the
4 Poor questions to follow. It is a brief summary
of the programÕs activities and the people
Rather than: __ Excellent __ Good __ Fair
involved.
__ Poor
Example:
■ Give directions about how to answer.
Include directions in parentheses immedi- The Dell County Extension family living
ately following questions. It is better to program included a variety of activities
repeat directions too often than not during 1989 focusing on teaching money
enough. Here are some examples of spe- management and budgeting skills to help
cific instructions you might use: Circle the families manage their resources better.
number of your choice; circle only one; check These activities included lunch and learn
all that apply; please fill in the blank; enter programs, computer budgeting workshops,
whole numbers; please do not use decimals or letter series and short courses. Consumers
fractions; etc. from across the county attended these
activities on Money Management Skills.
Q U E S T I O N N A I R E D E S I G N ■ ■ ■ 13
Q-5 Do you own or rent the home in which you now live? (Circle the number of your answer.)
1 Own home
2 Rent home
(if you own your home) (if you rent your home)
Q-6a How much is your monthly house Q-6b How much is your monthly rent?
payment (without property taxes)?
1 Less than $100 1 Less than $100
2 $100 to $199 2 $100 to $199
3 $200 to $299 3 $200 to $299
4 $300 or more 4 $300 or more
Q-7a How much per month do you Q-7b Which, if any, of these is
pay for electricity, garbage included in your monthly rent?
collection, heat and water? (Circle all that are included.)
1 Less than $25 1 Electricity
2 $25 to $74 2 Garbage
3 $75 to $124 3 Heat
4 $125 or more 4 None of the above
14 ■ ■ ■ P R O G R A M D E V E L O P M E N T A N D E V A L U A T I O N
(Adapted from Rohs, 1985; Sudman and Bradburn, 1986; Salant and Dillman, 1994)
Q U E S T I O N N A I R E D E S I G N ■ ■ ■ 15
References
Babbie, Earl R. Survey Research Methods.
Belmont, CA: Wadsworth Publishing Co.,
1973.
Bennett, Claude F. Reflective Appraisal of
Programs (RAP): An Approach to Studying
Clientele-Perceived Results of Cooperative
Extension Programs. Ithaca, NY: Cornell
University Media Services, 1982.
Dillman, Donald A. Mail and Telephone
Surveys: The Total Design Method. New
York: John Wiley and Sons, Inc., 1978.
Fischer, Christy and Jeffrey D. Layman. Edge:
Constructing a Questionnaire. Ohio State
University: Ohio Cooperative Extension
Service, 1986.
Newsome, Bob W., et al. Northeast Area
Evaluation Process. Kansas State University,
Kansas State Cooperative Extension
Service, n.d.
Rohs, F. Richard. Questionnaire Construction.
Athens, GA: Cooperative Extension
Service, 1985.
Salant, Priscilla and Don Dillman. How to
Conduct Your Own Survey. New York: John
Wiley & Sons, Inc. 1994.
Sawer, Barbara J. Evaluating for Accountability.
Corvallis: Oregon State University
Extension Service, 1984.
Sudman, Seymour and Norman M. Bradburn.
Asking Questions: A Practical Guide to
Questionnaire Design. San Francisco:
Jossey-Bass Publishers, 1982.
Appendix
Attribute information ■ Phrase income questions carefully. Do
hen you collect attribute data (personal you want to know the householdÕs (or
W characteristics such as age, education,
income,) it is often a good idea to formulate
respondentÕs) annual income? Monthly
income? Net? Gross? Taxable? Take-home
pay? Some people are sensitive to income
responses you can compare to existing data;
questions. Instead of asking ÒWhat was
for example, categories from a previous study,
your total household income in 1995?Ó you
or the same categories as reported in U.S.
could ask, ÒWhich of the following cate-
Census data. You can then compare your data gories best describes your total household
to the census to demonstrate how closely your income for 1995?Ó Offer a choice of several
sample reflects the general population. broad dollar amounts.
Even if you have no comparisons currently in ■ Use appropriate employment
mind, collecting this type of information about categories. Employment status is one
respondents can contribute to a Òdata bank.Ó It attribute that is hard to describe. The fol-
will allow you to build a Òparticipant lowing example is a fairly lengthy one.
profileÓthat describes participants in terms of (The respondent is asked to read the entire
key variables, compares the characteristics of list, then select the one response which
those who participate and those who donÕt, best describes his/her employment status.)
assesses the degree to which target audiences A. Employed full-time in the work force
are being reached, and plan strategies for pro- B. Employed part-time in the work force
moting future programs effectively.
C. Unpaid full-time in farm, family or home
Here are a few tips for collecting such data. business
(Adapted from Sawer, 1984) D. Unpaid part-time in farm, family or home
business
■ Place attribute questions at the end of
E. Unemployed, seeking work
the questionnaire.
F. Unemployed, not seeking work
■ Avoid overlapping categories. Use exclu-
sive categories; for example 25Ð34, 35Ð44, G. Student, employed part-time
45Ð54, etc., not 25Ð35, 35Ð45, 45Ð55. H. Student, not seeking work
■ Ask age questions directly, but be diplo- I. Retired, employed part-time
matic. Some respondents may perceive J. Retired, not seeking work
ÒHow old are you?Ó as a little blunt.
A simpler breakdown is:
ÒWhat is your age?Ósoftens the request.
ÒPlease circle the number of the category A. Employed
which includes your ageÓ is inoffensive. If B. Unemployed, seeking work
you need more precisely refined figures,
C. Not in the labor force
you could ask ÒWhat was your age at your
most recent birthday?Ó(Most people hope
they havenÕt yet celebrated their ÒlastÓ
birthday.) Also avoid asking for a month
and/or year of birth unless itÕs absolutely
necessary, since such information will
require many extra calculations.
Q U E S T I O N N A I R E D E S I G N ■ ■ ■ 17
Some examples of attribute questions follow. 7. What is your race? (Circle one number.)
1. What is your age? (Circle one number.) 1. Anglo or white
1. Under 18 2. Black
2. 18–24 years 3. Hispanic
3. 25–34 years 4. Asian
4. 35–44 years 5. Native American
5 45–54 years 6. Other ____________
6. 55–64 years 8. Is your current agricultural operation a
single-family operation, a partnership, a
7. 65 or older
family-held corporation, or a nonfamily-
2. Are you married? held corporation? (Circle one number.)
1. No 1. Single-family operation
2. Yes 2. Partnership
3. How many people live in your household, 3. Family-held corporation
counting yourself?
4. Non-family-held corporation
_____________________
5. Other (specify)____________
4. Are you employed outside the home?
9. What was the approximate gross value of
1. No
farm sales from your operation last year
2. Yes If you work outside the including crops sold, animals sold, and
home, do you work: dairy products? (Circle one number.)
1 Full-time? 1. Less than $2,500
2 Part-time? 2. $2,500 to $4,999
5. What is the highest level of education that 3. $5,000 to $9,999
you have completed? (Circle one number.)
4. $10,000 to $19,999
1. Some grade school (1–8)
5. $20,000 to $39,999
2. Some high school (grades 9–12)
6. $40,000 to $99,999
3. High school graduate
7. $100,000 to $199,999
4. Some college or technical school
8. $200,000 to $499,999
5. College graduate or beyond
9. $500,000 or more
6. Where do you live? (Circle one number.)
10. In the past year did you work off-the-farm
1. Rural farm in a part-time or full-time job? (Circle one
2. Rural community (less than 2,500 number.)
people) 1. Did not work off the farm
3. Town between 2500–25,000 people 2. Was employed part-time off the farm
4. City between 25,000–50,000 people 3. Was employed full-time off the farm
5. City over 50,000 people
Author: Ellen Taylor-Powell is a program development and evaluation specialist for Cooperative Extension,
University of WisconsinÐExtension.
An EEO/Affirmative Action employer, University of WisconsinÐExtension provides equal opportunities in
employment and programming, including Title IX and ADA requirements. Requests for reasonable accommoda-
tion for disabilities or limitations should be made prior to the date of the program or activity for which they are
needed. Publications are available in alternative formats upon request. Please make such requests as early as
possible by contacting your county Extension office so proper arrangements can be made. If you need this infor-
mation in an alternative format, contact the Office of Equal Opportunity and Diversity Programs or call
Extension Publications at (608)262-2655.
© 1996 by the Board of Regents of the University of Wisconsin System doing business as the division of
Cooperative Extension of the University of WisconsinÐExtension. Send inquiries about copyright permission to:
Director, Cooperative Extension Publications, 201 Hiram Smith Hall, 1545 Observatory Dr., Madison, WI 53706.
You can obtain copies of this publication from Cooperative Extension Publications, Room 170, 630 W. Mifflin
Street, Madison, WI 53703, or by phoning (608)262-3346.