Developing Effective Questionnaires

CHRISTOPHER E. BORK
and J. BRUCE FRANCIS

Many research questions are effectively answered by collecting data about the
opinions, beliefs, and perceptions of large numbers of persons in widely dis-
persed areas. The questionnaire is an economical means of gathering such
information. Properly designed questionnaires can collect valid and reliable data
for analyzing a research problem. This article reviews fundamental principles of
questionnaire design and presents a matrix to guide the construction of a
questionnaire. Methods to avoid common problems and pitfalls in developing a
questionnaire are also presented. Field testing is recommended to ensure validity
and reliability.
Key Words: Physical therapy, Research design.

The need for knowledge that goes beyond the basic and
clinical sciences increases as the physical therapy profession
continues to develop and mature. An understanding of the
feelings, attitudes, and concerns of the members of the profes-
sion, the patients served, and the public in general is necessary
when clinicians plan, implement, and assess changes in the
practice of physical therapy. The realm of survey research
includes concepts and strategies that enable the researcher to
accumulate data concerning the attitudes, opinions, and
knowledge of a population or sample.12 Survey research is
also useful to describe a particular knowledge, such as what
physical therapists know about the legal implications of in-
dependent practice.1
One of the most potent survey research tools is the ques-
tionnaire, which can be administered directly, by telephone,
or through the mail.3 We will discuss only questionnaires sent
through the mail in this article. Because a questionnaire allows
a researcher to gather data conveniently from large samples,
it is a useful tool to collect and analyze data from groups that
may be affected by changes in professional practice or edu-
cation. Unfortunately, many questionnaires contain major
flaws that render them incomprehensible to respondents or
do not collect the intended data. If a questionnaire is incom-
prehensible to the recipient, it is unlikely to be returned; a
questionnaire that does not gather valid information does not
allow the researcher to address the research problem being
studied.3
A researcher's attention to several fundamental principles
for developing a questionnaire greatly enhances the likelihood
that valid and reliable data will be gathered. This article is
intended to assist the reader to develop questionnaires that
collect valid and reliable data. Generally, the term validity
refers to the ability of an instrument, such as a mailed
questionnaire, to measure actually what the researcher intends
to measure. Reliability refers to the consistency of a question-
naire in gathering data. Optimally, each time an individual
responds to an item he or she should choose the same re-
sponse. Thus, carefully developed questionnaires tend to
gather data that accurately represent the knowledge, attitudes,
or opinions of the respondents and address the researcher's
Dr. Bork is Associate Professor and Director of Graduate Studies, Depart-
ment of Physical Therapy, College of Allied Health Professions, Temple Uni-
versity, Philadelphia, PA 19140 (USA).
Dr. Francis is Academic Vice-President, Walden University, Minneapolis,
MN 55401.
This article was submitted September 6, 1983; was with the authors for
revision 51 weeks; and was accepted December 12, 1984.
hypotheses. Before developing a questionnaire, however, a
researcher must decide whether its use is appropriate for a
particular study.
ADVANTAGES AND LIMITATIONS OF A
QUESTIONNAIRE
We are assuming the researcher has identified the problem
to be studied, translated it into a testable form, and defined
the variables operationally. The next tasks are to determine
what data must be collected and analyzed, identify sources
where the data can be obtained, and describe how data will
be gathered. As a tool to collect data, the questionnaire has
advantages and limitations that must be taken into account.
The main advantages of using a questionnaire are the
following:
1. Data are gathered economically because the cost for each
completed response is low.2-6
2. Distributionalflexibilityis provided because the question-
naire can be sent almost anywhere.4-6
3. Data can be gathered from a large number of subjects.4-6
4. The data gathered can be standardized, especially if an
item requires a response from a group of choices provided
by the researcher.4
5. The process is time efficient because it generally involves
less of the researcher's time to gather data in comparison
with an equivalent number of interviews.2, 5, 6
The main limitations of a mail questionnaire are the follow-
ing:
1. The subject's motivation while completing a questionnaire
is unknown.4
2. Data cannot be gathered from subjects who are illiterate.5
3. The researcher must assume all subjects have sufficient
knowledge to complete the questionnaire.3,4
4. The researcher must assume that all subjects are willing to
answer all questions.3, 4
5. All subjects who receive questionnaires may not complete
and return them.2, 3, 5, 6
6. Questionnaires do not allow follow-up questions such as
an interviewer may pose to clarify a given response.5
7. Although each subject receives the same items in a ques-
tionnaire, test conditions cannot be standardized.5
8. Subjects may not interpret items contained in a question-

Volume 65 / Number 6, June 1985 907


Fig. 1. Questionnaire item construction matrix.
naire in the same manner. Perceptions of what an item
asks may differ.4, 6
After carefully reviewing the advantages and limitations of
using a questionnaire for a particular study, a researcher must
decide if the advantages outweigh the limitations. The poten-
tial effects of using a questionnaire on the validity and relia-
bility of the study must also be considered.4 Using a question-
naire simply because it is economical and easily administered
may result in the collection of meaningless data. For example,
suppose a researcher wishes to compare, one year after dis-
charge, the functional ability of individuals who received a
particular exercise regimen with individuals who did not.
Using a questionnaire to ascertain individuals' functional
abilities probably would not yield valid and reliable data with
the possible exception of what the subjects perceived their
functional ability to be. In this example, the researcher should
decide that a questionnaire is inappropriate for the study.
USING A MATRIX TO TRANSLATE THE
RESEARCH PROBLEM INTO ITEMS
Translating a research problem into items suitable for a
questionnaire requires an understanding of the kinds of in-
formation necessary to validate or refute each hypothesis.2,4
Equally important is a knowledge of the characteristics of the
persons who will be answering the questionnaire, the respond-
ents.2-4 A questionnaire can be designed in a logical and
efficient manner if the following concerns are addressed: 1)
What information must be elicited from the respondent to
test the hypothesis? 2) Are the respondents likely to be willing
and able to provide this information? and 3) How will the
data be analyzed?
We have developed a matrix that has been helpful in
constructing questionnaires and checking items (Fig. 1). The
following sections of the article correspond to the major
sections of the matrix. Each section provides information to
908
enable the reader to fill in cells of the matrix and to avoid
pitfalls frequently encountered when developing a question-
naire.
Deciding What Information to Collect
A common tendency in survey research is to gather more
data than necessary.3,4 This results in questionnaires that are
too long and discourage the respondents from completing
them.4-6 Gathering more data than necessary also makes the
data analysis more complicated because decisions must be
made about which data are essential to the study and which
are superfluous. Justifying each item by the matrix ensures
that only items that gather information for testing the hy-
potheses of the study are included in the questionnaire.
To help refute or validate a hypothesis,3,4, 7 include demo-
graphic information that can be used to test the relationship
of sample to population and possibly the relationship of
variables to one another. Sometimes more than one item for
one hypothesis is necessary to elicit the information the
researcher seeks.3, 5 For example, multiple items are needed if
detailed information is sought or if the information is of a
sensitive nature and must be elicited in a manner to keep the
respondents inclined to provide the information.
Knowledge of the Respondents
The characteristics of the potential respondents are impor-
tant to the design of the questionnaire and the items. The
researcher must be confident that the respondents possess
adequate knowledge to complete the questionnaire.3, 4 If the
questionnaire must use technical language or jargon, the
respondents must be able to comprehend fully this language.3
A knowledge of the characteristics of the respondents is im-
portant to avoid asking unanswerable questions. An example
of an unanswerable item is, Do you prefer TENS or cold laser
therapy for pain reduction? If asked of physical therapists
with access to both modalities, the question would be accept-
able, but if asked of physical therapists in general, the question
might not be answerable to those clinicians without access to
both modalities. If the researcher incorrectly assumes that all
respondents will be able to answer an item and no provision
is made for the respondents to indicate that they are unable
to provide the information, the data gathered might be inac-
curate or misleading. Some simple guidelines to avert this
problem include avoiding complicated syntax (ie, lengthy,
complex statements), omitting technical jargon unfamiliar to
any potential respondents, choosing only topics familiar to
respondents, and adding a response to indicate insufficient
knowledge for respondents who are not able to complete the
item.3, 5
A second important reason for understanding the respond-
ents is to determine whether the respondents will volunteer
the desired information. A researcher must be cognizant of
social and cultural mores that may affect the willingness of
the respondents to provide information.2-4 For these reasons,
questions about the potential respondents are important to
answer in the matrix. When the researcher has completed this
section of the matrix, she or he is ready to construct question-
naire items.
PHYSICAL THERAPY

CONSTRUCTING ITEMS
A researcher must decide how many items are needed to
gather data to test each hypothesis. The appropriate number
depends on the complexity of what is being studied and the
willingness of the respondents to provide the information. In
some cases, more than one item is necessary to gather relevant
data. For example, consider how a researcher would approach
gathering data to test the hypothesis that physical therapists'
attitudes about a specific therapeutic technique are related to
their knowledge of the technique. Several items relating to
various aspects of the technique may be appropriate to ascer-
tain the respondents' knowledge of the technique. In contrast,
only one item may be necessary to ascertain therapists' atti-
tudes concerning the technique. In some cases, more than
one item may be necessary to approach a potentially sensitive
issue gradually. The decision of whether more than one item
is necessary is thus dependent on each hypothesis and the
nature of the data being sought.2-4
Because the purpose of a questionnaire is to elicit infor-
mation from respondents in a consistent, and optimally, a
standardized manner, the utmost care must be exercised when
constructing each item. The information elicited by an item
may be a fact, an attitude, or an opinion. If an item is poorly
constructed, it may gather erroneous information that will be
of questionable use to the researcher. Flawed items result in
communication errors by respondents. These can be avoided
by attention to the information desired and the format of the
item.
Common Errors in Item Construction
Ambiguity in an item forces the respondent to interpret the
item and make decisions based on those interpretations. If
several interpretations are possible, the researcher may draw
erroneous conclusions from the data because the respondents'
interpretations differ substantially from the researcher's.3 An
example of an ambiguous question requiring an interpretation
is, Do you use transcutaneous electrical nerve stimulation
(TENS) frequently? Respondents answering this item must
interpret what is meant by "frequently." Consequently, what
the researcher considers "frequently" and what the respond-
ents consider "frequently" may vary significantly. Thus, the
information provided by this item is likely to be misleading
and almost useless to the researcher.
To avoid errors of ambiguity, the researcher must be as
precise as possible when developing items.7"9 The less subject
to interpretation an item is, the more likely it accurately
represents the respondent's view. Optimally, an item should
be reliable, that is, convey the same meaning to all respond-
ents. In the previous example, we can correct the ambiguity
by changing the nonspecific words: Do you use transcutane-
ous electrical nerve stimulation (TENS) more than three times
per week? Note that the term "frequently" has been specified
to mean "more than three times per week" so that respondents
are less likely to interpret the question differently.
Another common error in item construction is an item that
asks the respondent to address two or more questions or
statements.9 Consider this poorly worded example: Physical
therapists should reevaluate their patients frequently. The
declarative statement asks for the respondent's opinion on
two issues: 1) Whether physical therapists should reevaluate
patients, and 2) whether patients should be reevaluated fre-
Volume 65 / Number 6, June 1985
PRACTICE
quently or infrequently. Consider how a respondent will
complete the item if she or he strongly agrees that patients
should be reevaluated but not necessarily frequently. By en-
suring that an item makes only one declarative statement or
asks only one question, this pitfall may be avoided.
A final common error in item construction is an implicit
or explicit bias in wording.2, 3 Because the researcher is seeking
information from the respondents, any statement or phrase
that tends toward favoring a particular response must be
avoided. An accurate portrayal of each respondent is sought.
Consider the following examples of an item to determine the
happiness of Californians:
1. Do you think that people residing in sunny California are
happier than people residing elsewhere in the country?
2. Do you think that people residing in California are happier
than people residing elsewhere in the country?
The first sample item biases the respondents by using the
word "sunny," which may tend to evoke positive feelings
toward the sunshine and climate of California.
Poorly constructed or biased items decrease the likelihood
that the information gathered will be reliable and valid. By
conscientiously examining each item for construction errors,
the researcher can avoid needless frustration later on when
analyzing the data.
Response Formats
The response format provides the respondent with a means
to communicate facts, attitudes, or opinions. An inappro-
priate response format generates incomplete or erroneous
information. An earlier example requested information on
whether clinicians preferred TENS or cold laser treatment for
patients with pain. We pointed out that many respondents
might not be familiar with one or both of the modalities and
would lack the knowledge to complete the item. If the re-
sponse format included no choice to indicate a lack of famil-
iarity with the modalities, respondents would have no way to
communicate this important information. Therefore, the re-
searcher must select response formats that best convey the
information each questionnaire item seeks from the respond-
ents.
One means of classifying response formats is open- or
closed-ended. The criterion for classifying response formats is
whether the respondent is limited in response choices. If the
respondent is required to choose from a limited set of re-
sponses, the format is closed-ended. If there is no limit, the
response format is open-ended.
The open-ended format is often called the essay response
because the respondent is asked to respond by writing the
answer in her or his own words. By allowing the respondent
such freedom, the researcher is provided a response that is
rich in content. The open-ended format has the advantage of
allowing for all possible responses to a given item.10 Many
individuals designing questionnaires use the open-ended for-
mat to determine the range of responses to an item.2, 3 If a
researcher is interested in the respondents' knowledge about
a question, the respondents' frames of reference, or their
motivations, the open-ended format should be chosen.
Open-ended formats are not without several drawbacks.
When the respondent is free to choose the frame of reference,
the response may not provide the information anticipated by
the researcher.3 Completing a large number of items with an
909
 Please place a check in t h e box which
b e s t r e p r e s e n t s your r e s p o n s e t o t h e
s t a t e m e n t below.
There should be certification for
physical therapy specialists.
Fig. 2. Item with a scale response format.
open-ended response format can also be tiresome to respond-
ents. Many may not complete and return the questionnaire.6
Some respondents may have difficulty expressing themselves
in written language, and words chosen by a respondent to
convey a certain meaning may represent an entirely different
meaning to the researcher.6 Written communications, unless
carefully worded, tend to be imprecise and are interpreted in
ways other than those intended. Finally, the diversity, individ-
uality, and richness of open-ended responses make them
difficult and time-consuming to code for analysis.6, 10
Open-ended response formats simply provide space to allow
the respondent to answer the item. The respondent may be
asked to provide any one of the following:
1. One bit of information—What do you feel is the principal
value of independent practice by physical therapists?
2. More than one bit of information—What qualities should
a person possess if interested in becoming a physical ther-
apist?
3. The ranking of information—The types of patients, edu-
cational benefits, and salary are three important factors
influencing initial job selection by entry-level graduates.
Please rank these factors from the most important to the
least important.
When the researcher is aware of all the respondents' frames
of reference and all potential responses, the closed-ended
format may be used. Advantages of a closed-ended response
format include ease of answering and coding items because
of the limited and simultaneously focused choices.2, 3 On the
other hand, the closed-ended response format may sacrifice
richness; by selecting this format, the researcher assumes that
all possible responses are represented by the choices given.2
Finally, because of different interpretations by respondents
and the researcher, the response chosen by a respondent may
not truly represent the information the respondent meant to
convey.
Closed-ended response formats may be structured in several
different ways. The designs called dichotomies or two-way
answers ask the respondent to choose between two alternatives
that, optimally, are mutually exclusive.9 Some authors also
suggest including an alternative for respondents to designate
that they lack the information to answer the item.9 An ex-
ample of this response format is illustrated by the item, Do
you think all physical therapists should have a master's degree
to work in a specialty? The responses include Yes, No, or
Don't Know.
Another form of the closed-ended response format is the
multiple-choice design. In a multiple-choice design, the re-
spondent must choose either one or a number of alternatives
910
that best represent her or his response to the item. Often
respondents are asked to rank their choices to provide more
information for the researcher.
Another variation of the closed-ended response format is
the scale, whereby respondents must select a point on a
continuum of possible responses (Fig. 2). Scales are generally
used to gain information on the relative intensity of attitudes
or opinions.
The question of whether to use a closed- or open-ended
response format depends largely on the information the re-
searcher is seeking. Most experts recommend the thoughtful
use of both response formats so that all essential information
can be obtained.2, 3 An important consideration in selecting
response formats is how the responses will be analyzed. The
investigator must be certain that the data collected are appro-
priate for the statistical tests available to test each hypothesis.
Data Analysis
The matrix requires the researcher to describe the type of
data to be collected and the proposed method of analysis. The
potential usefulness of the data and the appropriateness of the
method of analysis can be evaluated. Hence, the matrix serves
not only to identify what information should be sought but
also to determine the appropriateness of the information.
SEQUENCING ITEMS
The final series of cells in the matrix begins with the
sequence of items in the questionnaire. To fill these cells, the
investigator must determine and justify the sequence of the
items. The arrangement of items should follow a sequence
that appears logical from the respondent's viewpoint.3, 5 A
sequence that appears logical to the investigator for providing
information to test a hypothesis may confuse and frustrate
the respondents, who are then less likely to complete and
return the questionnaire. Arranging items in a sequence that
appears logical to the respondents increases the probability
that questionnaires will be completed and returned.5
To reduce frustration and increase interest, begin the ques-
tionnaire with items that seem to be the easiest to answer and
the most interesting.1 Although some authors suggest that the
initial items should be objective and impersonal to establish
a mind-set, whereby the respondents are encouraged to re-
main objective, holding the respondents' interest remains a
primary concern.8
The items should be sequenced in ways that focus the
respondents' attention to help them provide information.1
Items that are dependent on the ability of the respondent to
recall an event from the recent past are best sequenced by
asking first about current events and gradually about events
up to two months in the past.10 Generally, researchers are
concerned about the accuracy of individuals' memories of
events after two months have passed. Items calling for specific
information should be preceded by related items that ask for
general information and gradually increase in specificity.3
Items that ask for personal information and items that are
the most difficult to answer should appear near the end of the
questionnaire.10 Respondents who are frustrated or are made
defensive by difficult questions may elect to skip them; yet,
they have already provided substantial information elsewhere
in the questionnaire. Respondents who have answered all
previous items may not be as likely to be intimidated or
PHYSICAL THERAPY
 PRACTICE

frustrated by difficult or personal items. In effect, what has
taken place is the establishment of a type of rapport between
the researcher and the respondents through the medium of
the questionnaire.1, 5 When this happens, respondents may
not feel threatened and will provide the information re-
quested.
Instructions
sequentially. Response formats must be constructed to reflect
accurately the information sought, and instructions to the
respondent must be clear. Before printing, a questionnaire
should be field tested and revised if necessary.
Survey research can illuminate many problems and issues
of concern to the physical therapy profession. This article
serves as a starting point for those who wish to contribute to
the body of knowledge and elect to use a questionnaire.

The matrix concludes with cells requiring the researcher to REFERENCES

identify and justify instructions. Instructions given the re-
1. Babbie ER: The Practice of Social Research, ed 3. Belmont, CA, Wads-
spondents for completing the mail questionnaire should be worth Inc, 1983, pp 209-228
clear and concise. If the researcher wishes to emphasize the 2. Nisbet JD, Entwistle NJ: Educational Research Methods, New York, NY,
instructions, several methods are available, such as printing Elsevier Science Publishing Co Inc, 1970, pp 44-53
3. Sheatsley PB: Questionnaire construction and item writing. In Rossi PH,
the instructions in a different color from the items or printing Wright JD, Anderson PB (eds): Handbook of Survey Research. New York,
key words or phrases in capital letters.5 In either case, the NY, Academic Press Inc, 1983, pp 195-230
objective is to present the instructions as clearly as possible. 4. Mouly GJ: Educational Research: The Art and Science of Investigation.
Boston, MA, Allyn & Bacon Inc, 1978, pp 189-201
When the researcher changes topics or response formats, a 5. Brook L: Postal Survey Procedures. In Hoinville G, Jowell R: Survey
transition statement should accompany the change.1, 3 A tran- Research Practice. London, England, Heinemann Educational Books,
1977, pp 125-143
sition statement helps respondents change their frames of 6. Dillman DA: Mail and other self-administered questionnaires. In Rossi PH,
reference. The investigator who changes topics without in- Wright JD, Anderson PD (eds): Handbook of Survey Research. New York,
cluding transition statements risks confusing, frustrating, or NY, Academic Press Inc, 1983, pp 359-376
7. Francis JB, Bork CE: Design and Administration of Mail Questionnaires,
antagonizing the respondents, who may then become unwill- ed 2. Naples, FL, Action Research Associates, 1983, pp 3-37
ing to provide useful information. 8. Sax G: Principles of Educational Evaluation and Measurement. Belmont,
CA, Wadsworth Inc, 1974, pp 435-442
9. Payne SL: The Art of Asking Questions. Princeton, NJ, Princeton University
Press, 1951, pp 228-237
FIELD TEST 10. Courtenay G: Questionnaire construction. In Hoinville G, Jowell R: Survey
Research Practice. London, England, Heinemann Educational Books,
Before printing and mailing a questionnaire, the investiga- 1977, pp 27-54
tor should test the usefulness of the questionnaire.3, 5, 10 The
questionnaire should be completed by a group of individuals
who are most nearly like the target population for the ques-
tionnaire. The purpose of this trial is to discover and correct
any unforeseen problems with the questionnaire. In addition,
the researcher will gather preliminary data that will provide a
sense of the quality of the forthcoming study data.
The field test provides the researcher an opportunity to
validate the questionnaire by examining whether the data
gathered were appropriate to test the hypotheses and if the
data accurately represented the respondents' knowledge, opin-
ions, and attitudes. A researcher can also assess the reliability
of items by interviewing the respondents in the field test and
comparing their verbal responses with their responses to the
questionnaire item. If the questionnaire is reliable, the differ-
ences will be minimal.
After the questionnaire has been field tested and any nec-
essary changes incorporated, the researcher can have it printed
and mailed to the target population. One fundamental maxim
must be remembered when designing and administering ques-
tionnaires: No matter how excellent a questionnaire appears,
some oversights are likely. Therefore, although a researcher
should, and indeed must, work conscientiously to construct a
questionnaire, the instrument produced will not always be
perfect.

SUMMARY
The questionnaire, when properly designed, is a useful
approach for survey research. In constructing a questionnaire,
the researcher should include items that gather data to address
specific hypotheses. Using a matrix in developing a question-
naire can help the researcher ensure that important aspects of
construction are addressed and that all items included are
justified. Items should be worded concisely and arranged

Volume 65 / Number 6, June 1985 911