The Validity of Surveys: Online and Ofine

Wybo Wiersma

The Validity of Surveys: Online and Ofine

Wybo Wiersma Oxford Internet Institute mail@wybowiersma.net

Introduction

and ofine surveys are set out across two crucial moments in conducting them: solicitation

As more types of media have entered peoples homes, surveys can be conducted in ever more ways. In the thirties and forties, going door to door, or mailing surveys out, were the only options. From the seventies onwards, phone interviews became a popular alternative, thanks to the near universal adoption of the telephone. And now, as the internet is ever more ubiquitous, web-surveys are becoming an interesting option as well. Online surveys are generally considered cheaper, faster, and more convenient.42 In addition, they also have a potential for international reach, allow for elaborate skip-logic, and eliminate errors in data-entry.10 Yet online surveys bring new problems as well, especially in terms of validity. Therefore the most important threats to the validity of both online and ofine surveys will be assessed in this paper. First some limits of the paper are explicated. Then the notion of validity that is used here, will be claried. And after that, a range of distinctions between online
1

(inviting participants) and delivery (administering the actual survey). When this foundation has been laid, it will be used for categorizing and assessing threats to validity, such as low response rates, and interviewer effects. In this assessment some thoughts will be given to determining which of these issues are likely to remain a problem in the long run as well, as importance is in part a function of permanence. The paper will then be concluded with an overview of the threats, and some nal words on their practical implications for survey research.

1.1

Limits

First of all, there are many types of websurveys, and they can have many different aims. Some just serve as entertainment, others aim to aid webmasters at soliciting feedback on their specic sites, and yet others are used to mine for hypotheses, or to pilot (try out) a new questionnaire.9,30 All of these are

The Validity of Surveys: Online and Ofine

Wybo Wiersma

valid aims in themselves, but this paper will only be concerned with scientic surveys that aim to allow for inferences across large populations. A thing that follows from this, is that this paper will focus on quantitative research. Secondly, there are many temptations in cheap, fast, online surveys: bad sampling such as posting an open invitation link on a forum, or sending out invitations to ones whole target population (a census); bad questioning, by formulating questions without much thought, or by adding too many meaningless questions; and nally, adding in multimedia and colour indiscriminately.30,4,13 While these can, and often do, threaten the validity of online surveys, due diligence will be assumed here, not what is cheap, seductive, or easy. In addition, what will be evaluated here, is objective, as opposed to perceived validity (many on-line surveys are done badly, giving them a bad name).41 Thirdly, there are many more threats to the validity of surveys, than can be discussed in the space of this paper. Some of these are serious threats to surveys regardless of the medium, while others have only small effects in most cases. A few examples are: bias from answers being self-reported; surveys being administered in articial settings; ways in which options are categorized, or can constrain answers; and bias or errors in
2

question-formulation, data-entry and analysis.15,1 These will not be further discussed. Nor will things that might amend them, such as mixed research designs, or gathering complementary data from other sources.32,41 This paper is about one-off (cross-sectional) surveys only, and mainly discusses threats that differ between online and ofine surveys.

1.2

Validity

Validity is not a simple notion, and it, and its components, are dened differently by different authors.26,6,3,40 The denition of validity that is relied on here, is: that a survey represents what it intends and claims to represent.26 Then there are two sub-types of validity: external-, and internal validity. External validity refers to the validity of the survey beyond the study: its generalizability, both to the population, and across contexts. Internal validity, for surveys, refers to the rigour of measurement: that the concepts one sets out to measure, are actually measured (and completely). Validity is often further subdivided, but for reasons of space that will not be done here.6,40 Finally, validity can be contrasted with reliability: a study giving stable results across trials. Though different, validity presupposes reliability: if (sets of) questions (instruments) are not reliable indicators

The Validity of Surveys: Online and Ofine

Wybo Wiersma

of what they try to measure, they cannot guarantee that one measures what one thinks one does.40

solicitation (inviting participants), and delivery (administering the actual survey), and along these, differentiations were made between types of media (see table 1).30

1.3

Online/Ofine

While the distinction between the online and ofine realms might seem straightforward, it is not. First of all, online is broader than the web, and also covers e-mail and Skype calls, for example. Secondly, there is a whole range of intermediate cases between online and ofine. Such as: an interviewer overseeing you entering data on an iPad, which is then stored on a server; a survey on CDrom sent through the mail; a scripted phoneinterview with touch-tone data-entry; or a web-survey that includes a video-link with the interviewer?41 Are these online, ofine, or both? In an effort to clarify things, a number of dimensions can be distinguished, such as: by what means the participant is invited; whether the participant enters the data himself; whether an interviewer is present; the medium used in the survey, such as text or voice; whether answers are recorded in realtime, and so on. But even that would be This schema accommodates for surveytechniques that are moderately complicated, such as phone solicitations for a web-survey, while still allowing for a quick overview. Another feature of dissecting things this way, is that the moments roughly correspond to the difference between external- and internal validity. This will become clearer in the following sections, as the threats to validity are discussed and classied in accordance with it (table 1).
3

Table 1: Modes of solicitation and delivery: The archetype ofine survey is the doorto-door survey, solicited and administered in person (left column), while the typical online survey is at the other end of the table: a portal-site survey solicited and delivered via the web (rightmost column).
Moment Solicitation Delivery Person Ofine Ofine Phone Mail E-mail Web Online Online

= intermediate = = intermediate =

too much granularity for the scope of this paper. Therefore, the two most important moments in the conduct of surveys were selected:

The Validity of Surveys: Online and Ofine

Wybo Wiersma

2 2.1

Solicitation problems Limited coverage

graphics, match those of the overall population.41 Naturally, this approach is not bulletproof, as removing skews on general demographic variables only allows you to say you have excluded these known biases, not the many unknown ones (in other answers).3,31 The only real solution will come with time. As adoption continues to grow in early 2010 the US adoption-rate has already risen to 80% , web-coverage will be less of a problem in the future.28 Though differences in connection-quality will likely remain, and at least for now, coverage is still a problem for e-mail and the web (see table 2). Table 2: Coverage as a threat to external validity: A threat to e-mail and web solicitation and delivery
Moment Solicitation Delivery Person Phone Mail E-mail C C Web C C

For a long time, the most obvious threat to external validity for web-surveys was, that large sections of the population did not have access to the internet.9,30 In 1998, only 33% of the US population had an internet-connection at home, and in 2000, though growth was phenomenal, it was still a mere 50%.9 This low rate of adoption was accompanied by a bias towards certain demographic groups. Households earning more than $75,000 were 20 times more likely to be online.9 Differences between web- and mail/phone surveys have been observed for other demographics (young, male, urban, educated), and for answers to various types of questions (slightly more liberal answers, if any pattern) as well.34,9 More recently, several studies have found smaller differences, or even no differences at all (for certain populations, such as students and lesbians).33,17,29,27,22

2.2 One early solution to the problem of coverage, was that offered by the Dutch Telepanel: people were recruited by phone, and then given a computer with internet-access, if they did not already have one.9 Another approach to the problem is post-stratication. This

Lack of a sampling frame

A more fundamental problem with the internet, has to do with sampling. Random sampling is important, as without it, introducing biasses is almost inevitable.3 Moreover, random samples are required for condenceintervals and other statistics to be valid.30,13 In order to take a random sample, one rst needs
4

comes down to re-weighting respondents so their weighted answers to questions on demo-

The Validity of Surveys: Online and Ofine

Wybo Wiersma

a list of ones population of interest, called a frame. But no such list exists for internet users.31 And where it is possible to randomly generate phone-numbers, this does not work for e-mail addresses.9 Only in very special cases is it possible to get, or compile, a list. Such as when ones population of interest consists of web-hosting companies, members of a club, or students at a certain university.5,42 Panels again have been proposed as a solution to this problem. Either self-selected panels consisting of millions of members, such as those of Harris Interactive, or large panels (randomly) recruited by phone. For both types of panel, biases can be introduced by selfselection and panel-effects (peoples answers changing as a result of taking many surveys).9 Yet a potential problem with panels recruited by phone, and with phone-surveys in general, is that land-line phone numbers (land-lines are traditionally used in phone-surveys) dont provide a perfect frame either. Some households have multiple lines, and some people, especially students and the young, have no land-line at all. Increasingly, they have one or more mobile phones instead.10,23 So, while the sunrise internet offers no frame, the frame offered by phone-lines is becoming less solid than it used to be. It is unclear how this will play out. Yet for now, and the near future, the lack of frames is a fundamental problem
5

for web- and especially e-mail-surveys, and a growing problem for phone-surveys (table 3). Table 3: Lack of a frame as a threat to external validity: A fundamental threat to e-mail and web solicitation, but increasingly problematic for phone-surveys as well (lowercase letter f indicates a modest threat).
Moment Solicitation Delivery Person Phone f Mail E-mail F Web F

2.3

Low response rates

Another fundamental problem for online surveys, and phone surveys to a lesser degree, is a low response rate. Low response rates are problematic because the people that dont participate, are usually different from those that do. For mail-surveys, response-rates between 40 and 70% are quite normal, while for e-mail and web-surveys, response-rates often fall (far) below 30%.7,37,41 Moreover, the kind of non-response that web-surveys receive, can be different from that in face to face settings. In the ofine world people can refuse to participate, or walk away, while on the internet, people can read (part of) the survey, and only then decide whether they want to continue or not.41 And where a physically present interviewer could gently motivate somebody

The Validity of Surveys: Online and Ofine

Wybo Wiersma

to continue lling out a survey, this is much harder to do online. The effects of a low response-rate might be mitigated by receiving higher quality data from online surveys, as some studies have reported things such as longer answers and less item non-response for on-line surveys.12 Yet other studies present conicting evidence.1,44 And much of the reported improvement, could have been due to differences in coverage as well (education level, age, etc.). Directly circumventing response-rateproblems (and frame-problems) has also been tried by inviting people to web-surveys by phone. However, these studies have had response rates that were even worse: around 10% (of internet users).41 The only thing that seems to have worked, so far, is offering multiple delivery modes. But even in that case, the online version was chosen by only 2 to 10% of respondents.35,14 So people still seem to have a strong preference for paper and phone surveys. Though it is likely that this problem will diminish over time (with better, and more user friendly devices). A more pressing issue behind low response-rates, is that (e-mail) invita-

from making it more difcult to harvest e-mail addresses (as people hide them), many invitation e-mails will now also end up in SPAM-lters.9 In addition, norms condemning SPAM can, for some people, come to cover e-mail invitations as well.36 One study even reports scholars having their servers hacked, after posting invitations to several news-groups.2 Certain professional associations, such as ESOMAR, MRA and MRS, even discourage sending invitations through e-mail.11,24,25 Yet the problem of overload stretches even further: people not only receive a lot of junk, they also have ever greater possibilities to engage in (more) entertaining activities (including entertaining surveys).9,20 Over time such crowding out effects could become great threats to scientic surveys across all modes, though low response rates are mostly limited to the web and phone for now (table 4). Table 4: Low response-rates as a threat to external validity: A threat to e-mail and web solicitation, but problematic for phone surveys as well (lowercase letter r indicates a modest threat).
Moment Solicitation Delivery Person Phone r Mail E-mail R Web R

tions are being crowded out by SPAM. This is a problem similar to that of telemarketing, which two decades earlier caused a drop in response-rates for phone-interviews.37 Apart
6

The Validity of Surveys: Online and Ofine

Wybo Wiersma

3 3.1

Delivery problems Controlling access

them back to their particular answers.34,4 Perceived anonymity is important, as it improves response-rates, and reduces social desirability biasses. E-mail surveys have this problem to an even greater extent, because they are sent to, and received from, particular addresses (so identities are stored).4 Another problem that remains, but that postal mail-surveys have as well, is that security can only be achieved up to the address-level. Nothing can stop people from asking their children, or a friend, to ll out the survey. Only in-person, and phonesurveys guard against this (table 5). Table 5: Access control as a threat to internal validity: Access can only be restricted up to the address-level for web-, e-mail, and mailsurveys (lowercase indicates a modest threat).
Moment Solicitation Delivery a a a Person Phone Mail E-mail Web

A commonly mentioned threat to the (mostly internal) validity of online surveys is, that it would be hard to control access to them.30,4 People could ll them out twice, post a link to them on a forum, or worse, even bots (computer-programs) could be stufng the ballot with bogus-answers. Traditional means of limiting control are indeed ineffective, such as limiting answers to one per IP address, and setting cookies in participants browsers. This because peoples IP-addresses change, and cookies can be removed, or circumvented by simply using another machine. Yet there is a solution that does work: providing each person in the sample with an unique password that can only be used to ll out the survey once (coupled with a properly congured server). The password can optionally be embedded in the url that is provided with the invitation, so it does not even have to be entered manually

3.2 (though requiring manual entry reportedly improves the quality of responses).4,19 Among the problems that remain, are that participants could feel insecure about whether their answers are being treated anonymously. Especially as participants can never know for sure what the survey-software records, and the password could always be used to link
7

Interviewer effects

So called interviewer-effects: the presence of an interviewer inducing socially desired answers (or other biasses), are a great threat to phone- and in-person surveys. White people are, for example, more likely to give politically correct answers, when they are being interviewed by a black person, on racial

The Validity of Surveys: Online and Ofine

Wybo Wiersma

issues.18,8 Other things that are known to be played down, besides racism, are smoking, drinking, and gambling habits, and reported sexual activities. One study has even found age-interaction-effects, with older people being more susceptible to interviewer effects.16 Another danger, is that an interviewer might (unknowingly) give out subtle cues about how he wants people to answer. This can happen, for example, through pronunciation, or by giving people more time for certain questions (more time is perceived as more important).15 One partial solution to interviewer effects, is working with multiple interviewers, where each takes care of a limited number of participants. This should at least cancel out some of the effects that differ between interviewers, and thereby increase validity. Another way to improve things, is training interviewers. As skills do vary. Also, it could be argued that, while using interviewers introduces interviewer-effects, them being able to steer things, has many benets as well. Both face to face, and on the phone, they can, for example, motivate people to continue, reassure participants trust in the anonymity of their responses, or detect and question bogus answers.9 Moreover, they can especially face to face pick up the respondents mood, or other contextual factors that might inuence
8

responses.

In addition, interviewer-effects

have been shown to be slightly smaller in phone-interviews.16 Some experiments even suggest that certain interviewer-effects might be mitigated with the clever use of virtual reality (such as differences in body-height, tone of voice, and strength of emotional expressions).41,43,38 Yet, traditional, in-person, and phone interviews, will always be affected interviewer effects (table 6). Table 6: Interviewer effects as a threat to internal validity: Fundamental for in-person and phone surveys.
Moment Solicitation Delivery I I Person Phone Mail E-mail Web

3.3

Display effects

Finally, display effects can be seen as the online equivalent of interviewer effects. Differences in how the survey is displayed across devices, screen-sizes and operating-systems, can, in more or less subtle ways, inuence how participants interpret questions. But

technology can have other effects as well. A slow internet-connection can discourage people from taking a survey, or missing plugins (such as a Flash-plugin) may even make a survey inaccessible to large portions of the sam-

The Validity of Surveys: Online and Ofine

Wybo Wiersma

ple.10,4

Table 7: Display effects as a threat to internal validity: A modest threat to e-mail surveys

One workaround, is sticking to basic technology (HTML forms).4 Though with the use of Javascript, it has become possible to make forms appear exactly the same across operating systems, by replacing OS-specic formbuttons and check-boxes with images (as SurveyMonkey does), so there is a trade-off here.
39,21

and web-surveys (lowercase indicates a modest threat).

Moment Solicitation Delivery d d Person Phone Mail E-mail Web

Table 8: Overview of threats to the validity of surveys. External: coverage (C), lack of a frame (F) and low response rates (R). Internal: Access (A), interviewer effects (I), and display effect (D) (lowercase letters indicate modest threats).
Moment Person Phone fr I I Mail a E-mail CFRa Cd Web CFRa Cd

As with interviewers, digital forms

also bring many benets, such as taking care of skip-logic, validating answers, and providing drop-down selection-menus, and contextual help.10 In addition, they allow one to randomize the order of questions, cancelling out effects that question-order might have in paper-surveys.9,41 Moreover, multimedia might be used to motivate, or guide people (increasing response-rates).41 Multimedia can bias responses as well, of course, but it can be argued (though this has not been studied) that in a world where people are increasingly used to multimedia, a sober, simple design, can bias responses as well (towards sober answers). Nevertheless, on overall, the differences between browsers on personal computers seems to be decreasing (apart from screensize). Though mobiles show a bit more variety, and differences in the way e-mails are displayed, still exist (table 7, and an overview of all threats in table 8).
9

Solicitation Delivery

Conclusion

To conclude, the notions of on- and ofine have been split across solicitation and delivery, and were divided into various types of media, along these. This schema was then used to categorize, and clarify threats to validity. It was found that, though coverage remains a threat to online surveys for now, lack of a sampling frame is a more fundamental problem for e-mail and web-surveys. Another important problem, and one that seems to be growing, is the low response-rate for online

The Validity of Surveys: Online and Ofine

Wybo Wiersma

and phone surveys, especially where caused by crowding out effects. Then for delivery, while access control is often cited as a problem for web surveys, there is a technical solution available that works up to the addresslevel (beyond which mail-surveys have similar problems). For in-person and phone surveys, interviewer effects are the most fundamental threats. While for online surveys, display effects can cause moderate to minor problems, especially across devices that are very different, such as personal computers and mobile phones. The overall picture that has emerged, is that all options have advantages and disadvantages (also see table 8). Even though some options, such as soliciting participants through websites (by leaving an open link), are generally not a good idea, unless there are specic reasons for taking this route, such as contacting populations that are hard to reach otherwise. Besides this, experiments with solicitation through one medium for a survey delivered through another (such as phone to web), did not seem to work well. Thus it follows, that web surveys are mainly a good alternative when a frame can be obtained for ones population of interest (and coverage is expected to be good for them). In most other cases, in-person, phone- and mail-surveys, still seem superior: both in terms of response-rates, cov10

erage, and the availability of frames. The sunset over land-lines might, at some point, make the web seem more favourable in comparison. Yet only if lady fortune is with survey researchers, and the adoptionrate of smartphones increases sufciently, it might become possible to send invitations to web-surveys by text-message. A clickable

link (already supported by the iPhone) could then take people directly to the survey, in the phones browser. Over the span of a decade or more, this could make smartphones into a very powerful frame for both phone- and websurveys. Until such times, the perfect modes for solicitation and delivery do not exist, and being aware of, and acknowledging the limits of each mode, is the best we can aspire to when designing and elding a survey.

Bibliography 1. Alwin, D. F, & Krosnick, J. A, The Reliability of Survey Attitude Measurement., Sociological Methods & Research 20, 139 (1991). 2. Andrews, D., Nonnecke, B., & Preece, J., Electronic Survey Methodology: A Case Study in Reaching Hard-to-Involve Internet Users., International Journal of Human-Computer Interaction 16, 185210 (2003).

The Validity of Surveys: Online and Ofine

Wybo Wiersma
org / uploads / professional _ standards / guidelines / ESOMAR _ Codes & Guidelines _ Conducting - research - using - Internet . pdf>.

3. Babbie, E. R, The Practice of Social Research. (Wadsworth Pub Co, 2007). 4. Best, S. J., Internet Survey Design., The SAGE handbook of online research methods 217 (2008). 5. Braithwaite, D., Emery, J., de Lusignan, S., & Sutton, S., Using the Internet to Conduct Surveys of Health Professionals: A Valid Alternative?, Family Practice 20, 545 551 (2003). 6. Bryman, A., Social Research Methods. (CSIRO, 2001). 7. Cook, C., Heath, F., & Thompson, R. L., A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys., Educational and Psychological Measurement 60, 821 836 (2000). 8. Cotter, P. R, Cohen, J., & Coulter, P. B, Race-of-Interviewer Effects in Telephone Interviews., Public Opinion Quarterly 46, 278 (1982). 9. Couper, M. P., Web-Based Surveys: A Review of Issues and Approaches., Public Opinion Quarterly 64, 464494 (2000). 10. Dillman, D. A, Mail and Internet Surveys: The Tailored Design Method. (2007). 11. ESOMAR, Guideline on Conducting Market and Opinion Research Using the Internet.
<http : / / www . esomar .

12. Evans, J. R, & Mathur, A., The Value of Online Surveys., Internet Research 15, 195219 (2005). 13. Fink, A., How to Conduct Surveys: A Step-by-Step Guide. (2008). 14. Fricker, R. D, & Schonlau, M., Advantages and Disadvantages of Internet Research Surveys: Evidence from the Literature., Field Methods 14, 347 (2002). 15. Gaskell, G. D, OMuircheartaigh, C. A, & Wright, D. B, Survey Questions About the Frequency of Vaguely Dened Events: The Effects of Response Alternatives., Public Opinion Quarterly 58, 241 (1994). 16. Groves, R. M, & Magilavy, L. J, Measuring and Explaining Interviewer Effects in Centralized Telephone Surveys., Public Opinion Quarterly 50, 251 (1986). 17. Handwerk, P. G, Carson, C., & Blackwell, K. M, On-Line Vs. Paper-andPencil Surveying of Students: A Case Study., 40th Annual Meeting of the Association of Institutional Research (2000). 18. Hatchett, S., & Schuman, H., White Respondents and Race-of-Interviewer Ef-

11

The Validity of Surveys: Online and Ofine

Wybo Wiersma

fects., Public Opinion Quarterly 39, 523 (1975). 19. Heerwegh, D., & Loosveldt, G., Web Surveys: The Effect of Controlling Survey Access Using PIN Numbers., Social Science Computer Review 20, 10 (2002). 20. Jones, S., Studying the Net: Intricacies and Issues., Doing Internet research: Critical issues and methods for examining the Net 127 (1999). 21. Kaczmirek, L., Internet Survey Software Tools., The SAGE handbook of online research methods 236 (2008). 22. Koch, N. S, & Emrey, J. A, The Internet and Opinion Measurement: Surveying Marginalized Populations., Social Science Quarterly 82, 131138 (2001). 23. Ling, R., New Tech, New Ties: How Mobile Communication Is Reshaping Social Cohesion., (2008). 24. MRA, Use of the Internet for Conducting Opinion and Marketing Research: Ethical Guidelines.
<http://www.mra- net.org/

26. Parry, H. J., & Crossley, H. M., Validity of Responses to Survey Questions., Public Opinion Quarterly 14, 61 80 (1950). 27. Picavet, H. S. J., National Health Surveys by Mail or Home Interview: Effects on Response., Journal of epidemiology and community health 55, 408 (2001). 28. Pierce, J., WIP USA: 2010 USC Annenberg Digital Future Study Finds Strong Negative Reaction to Paying for Online Services.
<http : / / www . digitalcenter .

org / pdf / 2010 _ digital _ future _ final _ release.pdf>.

29. Query, J. L, & Wright, K., Assessing Communication Competence in an Online Study: Toward Informing Subsequent Interventions Among Older Adults with Cancer, Their Lay Caregivers, and Peers., Health communication 15, 203218 (2003). 30. R. D., J. F., Sampling Methods for Web and E-Mail Surveys., The SAGE handbook of online research methods 195 (2008). 31. Rasmussen, K. B, General Approaches to Data Quality and Internet-generated Data., The SAGE handbook of online research methods 79 (2008). 32. Riegelsberger, J., & Nakhimovsky, Y., Seeing the Bigger Picture: A Multi-

pdf/internet_ethics_guidelines.pdf>.

25. MRS, Internet Research Guidelines.

<http : / / www . mrs . org . uk / standards / downloads / revised / active / internet _ mar06.pdf>.

12

The Validity of Surveys: Online and Ofine

Wybo Wiersma

Method Field Trial of Google Maps for Mobile., CHI08 extended abstracts on Human factors in computing systems 22212228 (2008). 33. Riva, G., Teruzzi, T., & Anolli, L., The Use of the Internet in Psychological Research: Comparison of Online and Ofine Questionnaires., CyberPsychology & Behavior 6, 7380 (2003). 34. Sax, L. J, Gilmartin, S. K, & Bryant, A. N, Assessing Response Rates and Nonresponse Bias in Web and Paper Surveys., Research in Higher Education 44, 409432 (2003). 35. Schonlau, M., Fricker, R. D, & Elliott, M. N, Conducting Research Surveys Via EMail and the Web. (Rand Corp, 2002). 36. Sheehan, K. B, & Hoy, M. G, Using E-Mail to Survey Internet Users in the United States: Methodology and Assessment., Journal of Computer-Mediated Communication 4 (1999). 37. Sheehan, K. B., E-Mail Survey Response Rates: A Review., Journal of ComputerMediated Communication 6 (2006). 38. Shklovski, I., Kiesler, S., & Kraut, R., The Internet and Social Interaction: A Meta-Analysis and Critique of Studies, 1995-2003., (2006).

39. SurveyMonkey: Free Online Survey Software & Questionnaire Tool. <http://www.
surveymonkey.com/>.

40. Vaus, D. A. D., Research Design in Social Research. (2001). 41. Vehovar, Overview: V., & Online Manfreda, K. L, The

Surveys.,

SAGE handbook of online research methods 177 (2008). 42. Wright, K. B, Researching InternetBased Populations: Advantages and Disadvantages of Online Survey Research, Online Questionnaire Authoring Software Packages, and Web Survey Services., Journal of Computer-Mediated Communication 10 (2005). 43. Yee, N., & Bailenson, J., The Proteus Effect: The Effect of Transformed SelfRepresentation on Behavior., Human Communication Research 33, 271290 (2007). 44. Yun, G. W, & Trumbo, C. W, Comparative Response to a Survey Executed by Post, E-mail, & Web Form., Journal of Computer-Mediated Communication 6 (2000).

13