MSc Applied Psychology

Survey Design

Survey Design
By the end of this class you should be able to: apply principles of survey design design a questionnaire code a questionnaire Surveys are one of the key research tools in social sciences, and are often an invaluable method of measuring a variety of beliefs, attitudes and behaviours related to health. While they undoubtedly have a number of strengths, they also have a number of drawbacks advantages cheap produce quick results convenient for respondent anonymity stable and consistent can limit experimenter bias disadvantages do not allow prompting conditions of answering not monitored dont know if question order was followed possible partial response no opportunity for additional info lack of motivation for respondent

format of questionnaires
presentation the questionnaire should have as professional appearance as possible to encourage respondents to answer it. layout Questions should be easy to read, and easy to follow Questions and responses should be easy to find instructions Make no assumptions about how respondents will answer. Each answer should state whether to tick the box or circle the alternative length This is difficult to gauge, and many authors go for the truism of as short as possible, but as long as necessary, whatever that means.

the covering letter

This is attached to the questionnaire for the respondent to read before completing the survey. state the purpose of the study show how the respondent can help indicate why they should help explain how the findings will be used offer a chance to view the findings give details of your name and organisation provide an SAE/return address include a return by date

Types of questions
Primary questions are questions related directly to the research topic. Secondary questions are not directly related to the research topic, but can be used to check related issues, or consistency of attitudes. Direct questions deal directly with the respondent on the issue to be addressed, e.g. how many cigarettes do you smoke each day. Indirect questions are useful when tackling sensitive, threatening or embarrassing issues, for example How many cigarettes do you think people of your age smoke each day? closed vs. open questions

MSc Applied Psychology

Survey Design

This choice depends on a number of factors. Kahn and Kennel (57) claim that closed questions should be employed if the respondents have already formed an opinion, they can communicate, and you are interested in classifying either responses or respondents. Open questions are more suited to complex issues, but can be difficult to code, unlike closed questions. Closed questions may require considerable piloting. Schuman and Scott (1987) asked respondents to name the most important problem facing the US. problem quality of public schools pollution legalised abortion energy shortage all other responses open question 1 1 0 0 98 closed question 32 14 8 6 40

Actually stating an option does affect its frequency, but it is unclear if this makes the response more or less valid. filter questions, can be used followed by a more specific contingency question, for example, do you smoke? and then How many cigarettes per day?

content of questions
You should organise the content of your questionnaire according to the following criteria: 1. relevance: questions must be related to the research topic. 2. symmetry: the number of questions for each area of the research topic should be balanced 3. clarity and simplicity: avoid ambiguous, vague and double-barrelled questions. 4. leading questions should be avoided e.g. When did you stop beating your wife? or Dont you think that Rugg (1941) asked respondents Do you think that the United States should allow public speeches against democracy? (62% said no) Do you think that the United States should forbid public speeches against democracy? (46% said yes) These results seem to be based on the emotional meaning of forbid. Schuman and Presser replicated these studies, and found similar results. There appears to be a difference even between not allow and forbid. Hippler and Schwarz examining opinions on peep shows, Xrated films, and salt to melt ice on roads, found that more people would not allow these, compared to forbidding these. Loftus (1975) Do you get headaches frequently, and if so how often Do you get headaches occasionally, and if so how often In terms of the total number of headache products, how many have you tried? 1? 5? 10? In terms of the total number of headache products, how many have you tried? 1? 2? 3? Harris (1973) How long was the movie? How short was the movie? How tall was the basketball player? How short was the basketball player? mean answer 2.2/week 0.7/week 5.2 products 3.3 products 130 minutes 100 minutes 79 inches 69 inches

question order
in the beginning interesting questions & easy questions

MSc Applied Psychology

Survey Design

in the middle important questions & questions requiring thought at the end open-ended questions, threatening questions & autobiographical data the organisation of questions Questions should be in a logical order. A smooth transition should be made from one topic to another. There are 2 different approaches, the funnel sequence, where the flow moves from broad questions to more specific and personal ones, and its opposite the inverted funnel, which starts with more specific questions and ends with broad questions. order effects Schuman and Presser (1981) Do you think a Communist Country like Russia should let American newspaper reporters come in and send back to America the news as they see it? 1st 55% yes, 2nd 75% yes Do you think the United States should let Communist newspaper reporters from other countries come in and send back to their papers the news as they see it? 1st 82% yes, 2nd 55% yes recency effects Schuman and Presser (1981) Should divorce in this country be easier to obtain, more difficult to obtain or stay as it is now? (Easier: 23%; difficult: 36%; stay 41%) Should divorce in this country be easier to obtain, stay as it is now, or more difficult to obtain? (Easier: 26%; stay 29%; difficult 46%)

response format
The responses should be exhaustive, i.e. that all possible options can be handled, this is often ensured by the use of a catch-all other category, although you should ensure that only a low proportion of respondents use this category. Possible responses should also be mutually exclusive and unidimensional (only measuring one construct). There are a number of different response formats which can be selected. 1.) Likert Scales very satisfactory 5 4 3 2 1 very unsatisfactory 2.) very good good neutral bad very bad 1 1 () () () () () 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10

3.) Acceptable Efficient 4.) fair 5.)

|_________________________________________________| ( ( ( ( ( ( ) ) ) ) ) ) +3 +2 +1 -1 -2 -3

unfair

6.) various graphical scales have used devices such as thermometer scales, face scales, ladder scales

MSc Applied Psychology

Survey Design

7.) Constant-sum scales Two or more objects must sum to a pre-determined total, usually 100. For example, after watching a football match you might assign Middlesbrough 84 and Newcastle 16 on their performance. 8.) Ranking scales Rank items in order of importance or preference. the middle position Many surveys require respondents to chose between two fixed positions, and do not allow a middle answer. The reasoning is that it allows a safe position, and offering an explicit choice increases responses in that category, on most issues of around 10-20%, and a decline in the extreme polar positions (Schuman and Presser, 1981). the assessment of no opinion Order effects are lessened in magnitude when respondents feel strongly about an issue, when they know little such effects are more powerful, and when they know virtually nothing many respondents give pseudo-opinions. Tide magazine in the 40s asked its readers what they thought about the fictitious Metallic Metals Act. Over 70% expressed an opinion, even though the act did not exist! Schuman and Presser, and Bishop et al. (1980) found that this figure has fallen to around 25% for obscure legislation. One approach to tackling Dont Knows is simply not to allow it as an option in closed questions, unless the respondent spontaneously offers it. DKs are strongly related to educational background (Baron, 67), and may often reflect a lack of knowledge. Pseudo opinions can be filtered out by making a statement, and then asking whether the respondent has an opinion, and if the answer is yes, asking if they agree or disagree. These are termed filter questions. acquiescence Sometimes termed agreeing bias, is the predisposition people have to answering yes to questions. Certain psychologists even regard it as a personality trait. Others believe a more likely explanation to be based on educational level (e.g. Lenski and Leggett, 1960), but is still to be found among university undergraduates (Schuman and Presser, 1981).

types of administration
1. 2. 3. What are the options? face-to-face questionnaire telephone questionnaire mail questionnaire face x x ~ x x phone ~ x ~ ~ ~ ~ mail ~ ~ x ~ ~

response rates long questions complex questions boring questions item non-response social desirability interviewer bias interviewer's opinions speed cost

interview behaviour
When approaching potential respondents you should Possess a confident manner Use a clip-board

MSc Applied Psychology

Survey Design

dress in a neat, but neutral manner Have a common understanding of the project facilitate a relaxed atmosphere (don't doorstep) discourage the presence of 3rd parties Ask questions in the same way Don't deviate from the written text. Probe for more information in a neutral manner: e.g. tell me more, how do you mean exactly?, mmm Coding questionnaire data There are a number of different methods: What is your favourite sport: 1 [ ] football (tick box) 2 [ ] cricket 3 [ ] hockey 4 [ ] rugby union .... 32 [ ] (other) Favourite sport 1 = football 2 = cricket 3 = hockey 4 = rugby union year 1 = 1st year 2 = 2nd year 3 = 3rd year 4 = post-grad gender 1= male 2 = female mode of study 1 = full-time 2 = part-time

We can place this data directly into a stats package: fav_sp gender year mode 1 1 1 1 1 2 3 1 2 1 3 2 2 1 1 4 4 1 3 1 5 1 2 4 2 It is always a good idea to build in a coding frame in the questionnaire. pilot testing a questionnaire this is essential it checks that: respondents understand the instructions respondents understand the questions they are able to answer the questions you have included all relevant questions you have deleted irrelevant questions

age 19 21 18 20 28

Steps to survey design step one: preparation Decide on the need for a questionnaire, its type, find out if a similar tool already exists. step two: constructing the first draft Formulate questions, usually more than needed, of primary and secondary questions step three: self-critique check for clarity, simplicity, relevance and symmetry step four: external scrutiny give draft to expert/colleague step five: re-examination and revision make amendments in light of aboves comments step six: pilot study on a small sample step seven: revision

MSc Applied Psychology

Survey Design

in the light of results of pilot step eight: pre-test the revised elements of the questionnaire are piloted step nine: the final draft samples Sampling is quite a large topic in itself. Im not going to discuss it in much depth, but there are certain factors that you should take into account. I have already discussed the difference between a sample and a population. There are 2 types of sampling probability : simple random sampling here selections are independent from each other, and are randomly selected e.g. the lottery method, random number tables, computer methods systematic sampling usually takes the form of every nth selection, dependent on the sampling frame stratified random sampling used when the population to be studied can be divided into strata, e.g. male and female. cluster sampling used when no sampling frame is available, and when the cluster criteria is significant, for example a hospital multi- stage a sample is selected from a pre-existing sample, on some variable relevant to the study. non-probability convenience sampling all respondents that the experimenter comes into contact with snowball highly useful when it is difficult to find members of the population, you use informal links between members to contact new respondents.