Quantitative Data Collection

Research Methods in applied linguistics

1/45 By Zoltán Dörneyi
 Introduction
 The most common instrument for collecting
quantitative data is the test. tests have several
types: language test or psychological test
2/45
(aptitude or personal batteries)

 A frequent method of collecting quantitative

data is through conducting a survey using
some sort of questionnaire.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Sampling in quantitative research
The most frequent questions asked by novice
researchers before starting investigation:
 How many people do I need to include in my
3/45
study?
 Hoe large should my sample be?
 What sort of people shall I select?

Sampling decisions affect the necessary

arrangements, timing and scheduling of the
project as well as the costs.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
 Sample, population,
representativeness
 Sample: The group of participants whom the
researcher actually examines in investigation.
4/45  Population: The group of people whom the study is
about.
 The sample is a subset of the population that is
representative of the whole population.
 The strength of the conclusions we can draw
from the results obtained from a selected small
group depends on how accurately the particular
sample represents the larger population.
Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet
 Sampling procedures
 Probability Sampling

 Random Sampling: selecting members of population on

5/45 a completely random basis
 Stratified random Sampling: combination of
randomization and categorization
 Systematic Sampling: selecting every nth member of the
target group
 Cluster Sampling: random selection of larger groupings
or units of population(especially when the population is
widely dispread)

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Sampling procedures
 None-probability Sampling
 Quota Sampling and Dimensional Sampling:
Quota Sampling is similar to proportional stratified random
6/45 sampling without the ‘random’ element. In Dimensional
Sampling at least one representative of every combination of the
various parameters is included in sample.
 Snowball Sampling: a few people having criteria are
asked to identify further members.
 Convenience or opportunity
Sampling:members are selected if they meet certain pratical
criteria.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 How large should the sample be?
 Rule of thumb: a range of between 1% to 10% of the
population, with a minimum of 100 participants.
 Statistical consideration: sample should have a
7/45 normal distribution.
 Sample composition: identify any distinct sub groups
that may behave differently in advance.
 Safety margin: leave a decent margin for unforeseen or
unplanned circumstances.
 Reverse approach: first approximate the expected
magnitude of results then determine the sample size.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 The problem of respondent self-selection

The actual composition of sample is not only the

function of some systematic selection but also of
8/45 factors related to respondents' own willingness to
participate.
The problem can arise, for example when:
 Researcher invite volunteers to take part in study
 The design allows for high degree of dropout
 Participants are free to choose to be in study or not

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Questionnaire Survey
Main methodological issues:
a) How to sample the participants
b) How to design and administer the research tool
9/45
Positive point of questionnaires:
 Easy to construct
 Extremely Versatile
 Capable of gathering a large amount of
information quickly in a form that is readily
processible

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 What are questionnaires? what do they
measure?
It is used in at least 2 broad senses

1. Interview schedules/guides
10/45

2. Self–administered pencil-and-paper questionnaire

“any written instruments that present respondents with a
series of questions to which they are to react either by
writing out or selecting their answers”

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 What do questionnaires measure?
Questionnaires can yield 3 types of data:
 Factual questions: finding certain facts about the
respondents such as demographic characteristic,
11/45 occupation, residential location, marital and socio-
economical status, educational level,etc.
 Behavioral question: finding out what the
respondents are doing, have done in the past, habits,
lifestyles, focusing on action and personal history.
 Attitudinal question: finding out what people think,
covering attitudes, opinions, beliefs, iterests and values

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 The difference between test and questionnaire
 A test takes a sample of the respondent`s
behavior/knowledge for the purpose of evaluating
the individual`s more general underlying
12/45 competence/abilities/skills. It measures how well
some one can do something.
 Questionnaire items do not have good/bad answers.
They elicit information in a non-evaluative manner,
without gauging their performance against a set of
criteria.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Multi-item scales
Multi-item scales: a cluster of several
differently worded items that focus on the
13/45
same target.

Item wording in general has a substantial

impact on the responses.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Closed-ended items
 There`s no production of free writing;
respondents choose one of the given
alternatives.
14/45

 Most professional questionnaires are made up

of ‘closed-ended’ items.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Common closed-ended item formats
 Likert scale: consisting of a characteristic statement.
Respondents are asked to indicate the extent to which they
’agree’ or ’disagree’.
15/45
 Semantic differential scales: by using it researchers can
avoid writing statements.

 Numerical rating scale: ’Giving so many marks out of so

many’

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Other closed-ended item types
 True-false items: Problems: simplifying things too much;
resulting highly reduced and distorted

16/45
 Multiple-Choice items: When asking about personal
information, such as level of education

 Rank order: Ordering items by assigning a number to them

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Open-ended question

 Here questions are not followed by response

17/45
options.
 They permit greater freedom of expression.
 We use the when we do not know the range of
possible answers.
 They can work well if they are not completely
open but contain guidance.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Four question types in open-ended questions:
 Specific open questions: asking about concrete pies
of information
 Clarification questions: can be attached to answers
18/45 with special importance or after the “Other” in a MC item
 Sentence completion: completing an unfinished
sentence
 Short-answer questions: more than a phrase less than
a paragraph

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Rules about item wording
 Aim for short and simple items
 Use simple and natural language
 Avoid ambiguous or loaded words and sentences (non
19/45 specific adj/adv, universals, modifying words, words having
more than one meaning,Loaded words )
 Avoid negative constructions
 Avoid double-barrelled questions
 Avoid items that are likely to be answered the same way
by everybody
 Include both positively and negatively worded items

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 The format of the questionnaire
 Main parts
 Title: for identifying the domain of investigation, providing
initial orientation, activating relative background knowledge
20/45 and content expectations
 General introduction: describing the purpose,
sponsoring/conducting organization, emphasizing that there`s
no right or wrong answer, promising confidentiality,
requesting honest answers, saying ‘thank you’
 Specific instructions: explaining how respondents should
answer the questions

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 The format of the questionnaire
Main parts

21/45  Questionnaire items: main body of questionnaire

with the use of different typefaces and font styles.
 Additional information: including a contact name
with a phone number or address. we can include a
note promising to send a summary of findings,
inviting for follow-up interviews
 Final ‘thank you’

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 The format of the questionnaire
 Length
 The length depends on how important the topic of
questionnaire is for the respondent.
22/45
 Dörneyi agrees on a four-page well-designed
questionnaire that takes half an hour to be
completed.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 The format of the questionnaire
Layout

 Booklet Format: not only does the questionnaire have to be

23/45 short but also it has to look short.

 Appropriate density: we need to achieve a compromise on

how much material need to be put.

 Sequence marking: marking each main section with roman

numbers, each question with Arab figures and lettering
subparts of questions.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Item sequence
Four principles:
 Mixing up the scales(items of different scales)

24/45  Opening questions: need to be easy, interesting and

focusing on important aspects
 Factual(‘personal’,’classification’) Questions:
leave personal questions at the end (especially culture issues)
 Open-ended questions ate the end: So other items
won`t be affected by negative consequences of this question.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Developing and piloting questionnaire
Developing and piloting questionnaire is a
stepwise process:
Drawing up an item pool:
25/45
‘Item pool’: letting our imagination to go free and create as
many potential items for each scale as we can think of.
2 sources for drawing ideas in doing so:
a) Qualitative, exploratory data gathered in interviews or student
essays focusing on the content of the questionnaire
b) Established/published questionnaires in the area

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Developing and piloting questionnaire
Initial piloting of the item pool: for reducing the
large of questions in the item pool to the intended final
number it is useful to ask 3-4 trusted and helpful colleagues
or friends to go through the items.
26/45

Final piloting(dress rehearsal): there`s only one

way to find out whether respondents will reply in intended
manner and it is by administrating the questionnaire to about
50 respondents similar to the target population.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Developing and piloting questionnaire
Item analysis: it involves checking three
aspects:
 Missing responses that were not understood correctly
27/45  The range of responses elicited by each item
 The internal consistency of multi-item scales

Post hoc item analysis: conducting a final item

analysis after administration of the final questionnaire screen
out any items that have not worked properly

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Administering the questionnaire
In social research by mail
In educational research by hand
In applied linguistics by group administration
28/45

People in general do not mind answering the

questions as long as they think that the survey is
related to a worthy cause and that their opinion
matters.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Strategies for achieving the cooperation of our
informants:
 Advance notice: announcing a few days in advance and
explaining the purpose and nature of the survey
 Attitudes conveyed by teachers, parents, and
29/45 other authority figures: participants are quick to pick
up superiors’ attitude towards the survey. Win the support of
all these authority figures
 Respectable sponsorship: if we represent an
organization that is esteemed highly by respondents, the
positive reputation is likely to be projected onto the survey

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Strategies for achieving the cooperation of our
informants:
 The behavior of the survey administrators: 1.business-
like cloths 2.friendlyness 3.smile for breaking the ice
4.professional overall conduct
30/45  Administrator attitudes: their behavior should exhibit
keen involvement in the project and show an obvious interest
in the outcome.
 Communicating the purpose and significance of the
survey: cover the following points in introductory speech1.
greeting and thanking 2. the purpose of the survey 3.the
reason of the selection of these participants 4.assurance of
confidentiality 5.the duration 6.’any question?’ 7.’thank you’

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Strengths of questionnaires

 Collecting a huge amount of information in

31/45
less than an hour
 Straightforward and fast data processing
 Useable with a variety of people in variety of
situations targeting variety of topics
 Offering anonymity if needed

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 weaknesses of questionnaires
 Easy to produce unreliable and invalid data
 Need of simple and straightforward items
32/45  Unsuitable for probing deeply into an issue
 Usually resulting in superficial data
 Including possible respondent literacy
problems
 Including social desirability bias

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Experimental study
 Experimental study can establish
unambiguous cause-effect relationships.

33/45  First take a group of learners and do something

special to them while measuring their progress. Then
compare the data with another group that is similar
to first group but it did not receive special treatment.

 First group is “treatment” or “experimental group”

and the Second one is “control group”

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Quasi-experimental study
They are similar to experiments except they
do not use random assignment to create the
comparisons
34/45

Two ways of improving its design:

 Avoiding students self-selection to be in treatment
group
 Minimizing pre-test differences between the two
groups

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Methods for minimizing pre-test differences
 Matching participants in the treatment and
control groups: first determine particular
individual difference variables then identify
35/45 participants in two comparison groups with similar
parameters
 Using analysis of covariance(ANCOVA):it
offers statistical method for adjusting the post-test
scores for any pre-test differences; we can
statistically screen the unwanted effects out of the
outcome measure

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Analyzing the results
Two ways of analyzing data obtained with a
‘pre-test-post-test control group design’:
1. ANOVA: computing ‘gain scores’ separately(by
36/45
subtracting the pre-test scores from post-test) then
comparing with T-test or ‘analysis of variance’
2. ANCOVA: comparing the post-test scores by
controlling from the pre-test scores

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 ANCOVA offers more precise results because:
 Gain scores are not sufficiently reliable as
they are systematically related to any random
error of measurement
37/45

 ANCOVA helps to reduce the initial group

differences(especially in quasi-experimental
studies)

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Experimental studies in educational and applied
linguistic research
American Educational Research Journal(AERJ) reports
“not only has educational intervention research been
decreasing in quantity but there also has been a
38/45 decline in quality”.

Reasons for losing popularity in experiments:

a) Many of topics are not directly related to ‘treatment’ or
’intervention’
b) Narrow scope(only one or a few variables can be altered at a
time)

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Strengths and weaknesses of experimental
design
Strengths
 Best method of establishing cause-effect relationship
and evaluating educational innovations
39/45  ‘pre-test-post-test control group design’ can control
the threats to its validity
Weaknesses
 High price for implementation
 We may end up with artificial framework in
laboratory conditions which reduce the external
validity

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Strengths and weaknesses of quasi-experimental
design
Strengths
 We do not have to worry about external

40/45
validity

Weaknesses
 ‘selection bias’: inequality of the initial
treatment and control group

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Collecting quantitative data via the
Internet
A web based study offers some tempting benefits:
 Reduced costs
 Convenience of administration
41/45  Automatic coding(by using ‘CGI script’)
 High level of anonymity(high level of honesty)
 International access (+for cross-cultural research)
 Access to specialized population(small, scattered, or
specialized)
 Limitation of this approach:
1.Technial issues 2.Sampling issues

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Technical issues
Internet users have different computers, systems,

browsers, and monitors, so the actual stimulus
received may differ from what the investigator
42/45 has intended. For this reason we can employ a
single HTML page and improve the user
interface.
 Participation is limited by technical issues like
connection speed and quality of installed
software but given the speed of progress these
restrictions are likely to be temporary.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Sampling issues

 There's a lack of control over who will

43/45
eventually participate in the study

 The actual sample that completes the web-

based survey or experiment may be much
more heterogeneous than in traditional
research.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

 Strategies offering a partial solution for
sampling issues
1. Analysis the question separately within each sub stream of
the sample. If the same conclusions are reached in each
subgroups, this might lend some external validity to the
result.
44/45
2. Comparison the web-based results with the outcomes of a
similar, non-web-based survey or experiment. Convergence
of the findings can help to validate the results.

3. Sampling in traditional way and asking them to log on and

complete the survey online at home or in computer lab.

Introduction|Sampling |Questionnaire|Experimental and quasi-experimental|via Internet

45/45 Thank you for paying
attention