A Tool Kit for Questionnaire

Development
Dr. Nadir Kheir PhD MPS FNZCP
Assistant Professor & Coordinator of Continuing
Professional Pharmacy Development
College of Pharmacy, Qatar University, Doha, Qatar

15-Feb-12

Questionnaire-OFID 2012-NKheir

The great popularity with questionnaires
is they provide a “quick fix” for research
methodology. No single method has been
so abused.
Gillham B. Developing a questionnaire (real world research). London: Continuum, 2000.

A study is unethical if it is scientifically
unsound, causes undue offence or trauma,
breaches confidentiality,
or wastes people’s time or money
Boynton P, Greenhalg T. Selecting, designing, and developing your questionnaire. BMJ 2004, 328: 1312-1315
15-Feb-12

Questionnaire-OFID 2012-NKheir

Learning Outcomes
After the completion of this session, the participant will be
able to:
1.
Describe the main uses of survey questionnaires
2.
list the major stages of questionnaire development
process
3. discuss the known pitfalls that could affect questionnaire
development
4. Explain what is meant by reliability and validity
5. Examine questionnaire face validity
6. Develop a short questionnaire with consideration to the
main components of the development processes
15-Feb-12

Questionnaire-OFID 2012-NKheir

Useful Resources
• A guide to good survey design. Statistics New Zealand 1995.
Publishing & Media Services Division
• Research methods knowledge base. Survey Research. Available in:
http://www.socialresearchmethods.net/kb/survey.php
• QuestionPro (Survey software). Available in:
http://www.questionpro.com/akira/logout.do

15-Feb-12

Questionnaire-OFID 2012-NKheir

Survey Research      Can be a questionnaire OR an interview Important area of measurement in applied social research Considered branch of social science Any measurement procedures that involve asking questions Can be a short paper-and-pencil feedback form to an intensive oneon-one in-depth interview  Study samples chosen from target population to discover inferences  Study relations between behaviors and other sociological variables  Different from a census (where the whole population is studied) 15-Feb-12 Questionnaire-OFID 2012-NKheir .

Purposes of Surveys: Description • Making descriptive assertions about a population • Answering the ‘what’ questions: .what is the distribution of certain traits? Examples: -the attitude towards new service -the quality of life of diabetics -the skill level of vocational workers 15-Feb-12 Questionnaire-OFID 2012-NKheir .

Purposes of Surveys: Explanation  Making explanatory assertions  For example: -why some persons abuse pain killers? -why do students resort to cramming in exams? -what are the reasons for the increased visits to the staff clinic during 2010 compared to 2009?  Explanatory objective needs multivariate analysis -Cramming could be explained by quality of teaching. employment status. class size . relaxed laws . an endemic.Abusing pan-killers can be associated with issues around access.Increased visits to the clinic in 2010 could be explained by a big conference at the University. number of assessments. etc 15-Feb-12 Questionnaire-OFID 2012-NKheir .

cost. you need to refine your methodology through a ‘dress-rehearsal’ survey  Focus group to explore deep meaning and complex phenomena  No attempt to select representative sample  Might raise some questions and possibilities  They identify logistical. sampling. Focus Group. qualitative research  Before a large study. design issues  In pilot study: a small sample size means its results might not useable as stand alone data 15-Feb-12 Questionnaire-OFID 2012-NKheir .Purposes of Surveys: Exploration  Also: pilot survey.

online surveys 3. telephone 2. drop-off surveys. Interviews: face-to-face. Mail surveys. The Focus Group All Utilize a Questionnaire or ‘questions’ 15-Feb-12 Questionnaire-OFID 2012-NKheir . Groups.Types of Surveys 1.

high response rate.Questionnaire • Questionnaires are usually paper-and-pen or online instruments that the respondent completes Type of Questionnaire Characteristics Mail survey questionnaires Adv: inexpensive. costly. the interviewer makes personal contact with the respondent Disadv: response may have to be posted back or collected in person 15-Feb-12 Questionnaire-OFID 2012-NKheir . low errors Disadv: difficult to assemble. inaccuracies Group-administered questionnaire Adv: convenience. can send the exact instrument to a wide number of people. when it's convenient. missing data. inconvenience to participants Household drop-off survey Adv: the respondent can fill in private. allow the respondent to fill it out at their own convenience Disadv: Low response rate. not suitable for detailed responses. errors.

Developing a Questionnaire • A complex and iterative process • This means the designer needs to go back and forth to resolve issues (change questions to suit respondents leading to reduce/change objectives. etc) • Multiple evaluations during the developmental phases to reduce burden at the end 15-Feb-12 Questionnaire-OFID 2012-NKheir .

beliefs. and woolly recommendations 15-Feb-12 Questionnaire-OFID 2012-NKheir .A Questionnaire: • Offer an objective means of collecting information about people’s knowledge. misleading conclusions. will lead to poor quality data. and behaviour • Can be used as the sole research instrument (such as in a cross sectional survey) or within clinical trials • If inappropriately developed. attitudes.

range of possible responses • Seek information available in the literature • If such information are not available. conduct a focus group or pilot study 15-Feb-12 Questionnaire-OFID 2012-NKheir .Before Settling on a Questionnaire: • Consider: is a questionnaire the right tool? • Map the area for logistical problems. sampling issues.

When can an Existing Questionnaire be Used? • If the questionnaire address the same objective • If the questionnaire is validated in the same cultural context • First and best option! BUT BE CAREFUL: Importing an existing questionnaire Translating an existing questionnaire • Examples: Quality of Life Questionnaires 15-Feb-12 Questionnaire-OFID 2012-NKheir .

328: 1312-1315 15-Feb-12 Questionnaire-OFID 2012-NKheir . BMJ 2004. Greenhalg T. designing.Why develop your own questionnaire if one is available? Boynton P. and developing your questionnaire. Selecting.

15-Feb-12 Questionnaire-OFID 2012-NKheir .

main question • Conduct a scoping exercise to ensure that you clarify goals and if necessary reach an agreed compromise • This will also flag up potential practical problems (e. The Background • A thorough understanding of the problem through literature search • Establish clear objectives. how long the questionnaire will be and how it might be administered) • Decide the process used to select the respondents • Seek statistical advice if generalizability of results is required.g. purpose.Steps in Developing a Questionnaire: 1. differences measured. for comparing results 15-Feb-12 Questionnaire-OFID 2012-NKheir .

or several might be done by one individual (list tasks as reminder) • Respondents: VIP group Why: cannot be trained. data entry staff. coders. researchers who analyze • Tasks might be shared. program writers.Steps in Developing a Questionnaire: The Background • Consider all people who would use the questionnaire • Participants: general public? Consider literacy • Interviewers. and they give their time and effort free of charge 15-Feb-12 Questionnaire-OFID 2012-NKheir .

attitudes. perceptions. opinions) • Generate statements/questions for the questionnaire • Start with broad statements • Follow with a list of detailed information needs • Looks like a list of tables in the final result section • Involve other parties who are within the research team • A link among the objectives of the study and their translation into content is established 15-Feb-12 Questionnaire-OFID 2012-NKheir .Steps in Developing a Questionnaire: 2: Questionnaire Conceptualization • Decide: what will the questionnaire measure (knowledge.

and Academics on Pharmacokinetic Teaching and Application in Clinical Practice Aim: To assess the attitudes and practices Hypothesis: PK is a knowledge-based service that is poorly utilized in practice by pharmacists Identify Populations and questions : Students of undergraduate pharmacy students. practicing pharmacists. need for change Questionnaire-OFID 2012-NKheir . challenges (exploration) •Practices: teaching strategies. relevance. satisfaction • Attitudes: satisfaction with application. satisfaction • Barriers. Pharmacists.Attitudes and Practices of Pharmacy Students. challenges (exploration) students attitudes towards an undergraduate courses in pharmacy? Pharmacists Faculty/Teachers 15-Feb-12 • Attitude: importance. relevance Need of validation exercise? • Practice: application. and pharmacy academics on the on PK teaching and learning as well as application in clinical practice • Attitude: importance. etc Existing questionnaire on • Barriers.

Steps in Developing a Questionnaire: 3: Format and Data Analysis Turning questions into a questionnaire • The focus is on writing statements/questions • Selection of appropriate scales of measurement • Questionnaire layout • Question ordering • • Number questions • Add cover note or introduction. most done automatically (to the scoring and producing results) 15-Feb-12 Questionnaire-OFID 2012-NKheir . routing instructions. page numbers • In online survey. codes.

usually. • “I Regularly.” “frequently” implies frequency. and hardly ever 15-Feb-12 Questionnaire-OFID 2012-NKheir . some. many. twice a week. engage in exercise”” One person can regularly engage in exercise once a month whereas another person can regularly do so four times a week •Other words to avoid in question stems include: commonly. and so on) would be more appropriate.Poorly Designed questions •“I frequently engage in exercise” :“strongly agree” through to “strongly disagree. so a frequency based rating scale (with options such as at least once a day.

Steps in Developing a Questionnaire: 4: Establishing Validity • Development of a valid and reliable questionnaire is a must to reduce measurement error • Measurement error is the "discrepancy between respondents' attributes and their survey responses " • This requires pre-testing the questionnaire and correlating its results with other variables • Poorly constructed questions and wrong response scaling lead to poor validity 15-Feb-12 Questionnaire-OFID 2012-NKheir .

Is it appropriate for the sample/population? 4. Does the instrument look like a questionnaire? 15-Feb-12 Questionnaire-OFID 2012-NKheir . Does it represent the content? 3.Steps in Developing a Questionnaire: 4: Establishing Validity • Validity is the amount of systematic or built-in error in measurement • The following questions are addressed in Step 4: 1. Is the questionnaire comprehensive enough to collect all the information needed to address the purpose and goals of the study? 5. Is the questionnaire valid? 2.

missing data.Steps in Developing a Questionnaire: 4: Establishing Validity Desk Evaluations -Looking for potential problems -The designer before user evaluation -Simplify. Social and Face Validity 15-Feb-12 User Evaluations The designer knows too much about the questions -Other users also need to try the questionnaire -Self-Admin: Observing respondents fill a questionnaire -Also: check completed questionnaire and ask about errors. after a change -Done by more than 1 person Content analysis . but careful -Conduct at each stage. changes required -3rd person observes and take notes -Follow-up questions to check understanding and difficulties Questionnaire-OFID 2012-NKheir .

net/kb/quesword.socialresearchmethods. Hard to understand. Overly educated: Question: How often do you engage in exercise or use other nonpharmacological remedies? • Place your comments here: 15-Feb-12 Questionnaire-OFID 2012-NKheir . badly written • 2.Pitfalls with Questions Also see: http://www.php • 1.

 Q. What was your income last year? ‘Last year’ may mean 12 months ending today. Which type of diabetes do you have:  T2DM  T1DM • 4. before or after tax? etc 15-Feb-12 Questionnaire-OFID 2012-NKheir . Abbreviations that respondents may not be aware of:  Q. financial year. Use of undefined terms:  Q. What is your ID No.php • 3.net/kb/quesword.socialresearchmethods. Income: poorly defined. calendar year.Pitfalls with Questions Also see: http://www.

net/kb/quesword. Was your performance better in your ability to do daily exercise or your sleep quality this year? • 7.Pitfalls with Questions Also see: http://www. Questions that are too long: Includes questions with too many options that do not suit phone interviews or can be burdensome to respondents 15-Feb-12 Questionnaire-OFID 2012-NKheir .php • 5. Ambiguity: what exactly do you mean? Q. Questions that do not fit the objectives: Irrelevance.socialresearchmethods. undue effort and burden • 6.

php 8. did you have to see your physician or another healthcare professional?  Yes  No Q.net/kb/quesword.Pitfalls with Questions Also see: http://www.socialresearchmethods. Do you agree that insulin injections should be taken before food and stored in a fridge to avoid harmful effect of temperature?  Yes  No 15-Feb-12 Questionnaire-OFID 2012-NKheir . Double or triple questions: Q. During the past 2 weeks.

In the last 12 months.Double negative: Q. Do you agree or disagree that pharmacy graduates from the College of Pharmacy in Qatar University should not be asked to register in order that they practice Pharmacy in Qatar? 15-Feb-12 Questionnaire-OFID 2012-NKheir . Unreasonable recall period: Q.socialresearchmethods.php • 9. how many times did you feel numbness in your left arm? • 10.Pitfalls with Questions Also see: http://www.net/kb/quesword.

net/kb/quesword.socialresearchmethods. What is the your spouse’s qualification:  No education Primary school Secondary school  College University graduate  Post-graduate qualification Assumption: Respondent has a spouse ♣ Adding a Not Applicable is not enough (might not be read) ♣ Problematic: if no spouse and next questions about spouse ♣ A respondent might tick (No education) as an option when no spouse! 15-Feb-12 Questionnaire-OFID 2012-NKheir .Questions that are based on an assumption:  Q.php • 11.Pitfalls with Questions Also see: http://www.

Pitfalls with Questions Also see: http://www. unbalanced): Q. loaded.socialresearchmethods.php • 12. Physicians in this hospital are:  Very caring  Fairly caring  Not very caring 15-Feb-12 Questionnaire-OFID 2012-NKheir . how often do you miss taking your pills? Q.net/kb/quesword. Social Desirability and other bias (leading. In a week.

Which of these over-the-counter classes of medicine have you sold in the last two weeks from your pharmacy:      15-Feb-12 Non-steroid anti-inflammatory Drugs Analgesics Laxatives Anti-hypertensives Drugs acting on the cardiovsacular system Questionnaire-OFID 2012-NKheir .php • 13.net/kb/quesword. Responses that are not mutually exclusive: Q.Pitfalls with Questions Also see: http://www.socialresearchmethods.

Pitfalls with Questions Also see: http://www. Response options that are not exhaustive Q.php • 14.socialresearchmethods.net/kb/quesword. How to resolve this? 15-Feb-12 Questionnaire-OFID 2012-NKheir . To which of the following do you most refer your patients for further investigation:  Hamad General Hospital  Primary health care clinic  Private health clinic ♣ There might be other referral sites.

Crowded. untidy (in typed questionnaire) • 2.Pitfalls with Questions • 1. Inconsistent use of terms (medicine and drug) • 4. respondents may skip sections (avoid) • 5. Hard to read (in typed questionnaire) • 3. Routing instructions that are hard to follow 15-Feb-12 Questionnaire-OFID 2012-NKheir . Inappropriate titles: can be confusing.

Lay out that is likely to cause mistakes: No  Yes  Don’t know Formatting issues resolved in online surveys.surveymonkey.Pitfalls with Questions • 6. Inconsistent lay out: Yes before No then changes • 7.com/Default. Try: http://www.aspx 15-Feb-12 Questionnaire-OFID 2012-NKheir .

Variability of the characteristic : greater variation= greater error 3. Selecting a frame that does not match the target population 4. Non-responses* 6. Poor definition of objectives 7. Sample design: use of information known about the population reduces sampling error 15-Feb-12 Questionnaire-OFID 2012-NKheir . Including or excluding groups 3. Sample size Examples: 1. Quality of data collection 9. Poor definition of target population 2. Inadequate interviewer training 2. Poor design that does not give every respondent the same chance of being selected 5. Time frame: seasonality factors 8.Survey Errors Sampling Error Non Sampling Error -Resulting from collecting information from a fraction of the population -These are hard to measure -May be larger than sampling errors can result in bias Extent of error depends on: 1.

make a flowchart 15-Feb-12 Questionnaire-OFID 2012-NKheir . nonthreatening questions Put more difficult.socialresearchmethods. use a transition Reduce response set (the tendency of respondent to just keep checking the same response) • For filter or contingency questions.php • Some important issues to be considered in designing a questionnaire: • • • • • • Start with easy. follow chronological order Ask about one topic at a time When switching topics.A Checklist of Considerations From: http://www.net/kb/quesplac. threatening questions near end For historical demographics.

socialresearchmethods.net/kb/quesplac.Ethics of Questionnaire Survey From: http://www. and ensure to fulfill your promise ) 15-Feb-12 Questionnaire-OFID 2012-NKheir .only include what is absolutely necessary • Be sensitive to the needs of the respondent • Be alert for any sign that the respondent is uncomfortable • Thank the respondent at the end for participating • Assure the respondent that you will send a copy of the final results (if s/he desires.php • Thank the respondent at the beginning for allowing you to conduct your study • Keep your survey as short as possible -.

errors.com/s/5JCKYXX 15-Feb-12 Questionnaire-OFID 2012-NKheir . weaknesses) in online survey: https://www.Finally… • An exercise • Find the pitfalls (mistakes.surveymonkey.

Questions? 15-Feb-12 Questionnaire-OFID 2012-NKheir .