Data collection instruments

Dr Azleena Mohamad
Signpost
• Questionnaires
• Field notes
• Educational records
• Quantitative research methods
• Attitude scales
• Digital research tools
• Triangulation
What is a
Questionnaire?
A set of questions
designed to generate the
statistical information
from a specific
demographic needed to
accomplish the research
objectives
• How questions are specified and put
together will influence the respondents’
willingness to participate & the
responses they provide
Purposes of the Questionnaire
 Ensures standardisation and comparability of
the data across interviews – everyone is asked
the same questions
 Increases speed and accuracy of recording
 Facilitates data processing
 Allows the researcher to collect the relevant
information necessary to address the
management decision problem
Improper design can
lead to…
 Incomplete information
 Inaccurate data
 Higher costs
Criteria to consider
 Does it provide the necessary
information?
 Does it consider the respondent?
 Does it meet editing, coding and
data processing requirements?
Designing the
Questionnaire
 Determine objectives: Plan what to
measure.
 Decide on format. E.g. personal interview,
telephone, self.
 Formulate questions to obtain the needed
information
 Decide on the wording of questions
 Decide on the question sequence and
layout of the questionnaire
 Using a sample, test the questionnaire for
omissions and ambiguity
 Correct the problems (pretest again, if
necessary)
The Major Decisions in
Questionnaire Design
1. Content - What should be asked?
2. Wording - How should each
question be phrased?
3. Sequence - In what order should
the questions be presented?
4. Layout - What layout will best serve
the research objectives?

The most difficult step is specifying
exactly what information is to be
Content: Questions must meet 5 requirements
1. Are the questions relevant? Do
they pertain to the research
problem?
2. Are the questions accurate? Do
they accurately depict the
attitudes, behaviors, etc. intended
to investigate?
3. Do respondents have the
necessary information?
• Qualify respondents
4. Do respondents understand and
interpret the question correctly?
5. Will respondents give the
How should the questions be asked?
Format: How much freedom do we give
respondents in answering questions

•Open-ended questions
• What do you look for most in a job?
• Is there anything else you would like to add about
the product?
Open Ended Questions:
Advantages Disadvantages
Wide range of responses and Ability and/or willingness of
information can be obtained respondent to answer

Answers based on respondent’s not Interviewer’s ability to record
researcher’s frame of reference – answers quickly or summarize
consumer’s terms accurately & probe effectively

Lack of influence. Don't channel Interviewer’s attitude influences
respondents thinking response

Can help interpret closed-ended Time consuming (interview sessions,
questions - why tabulation, classification,
assignment, validation)

Particularly useful as introduction to Difficulty in coding
survey or topic

When it’s important to measure the Require respondents to be articulate
salience of an issue

Respondents may miss important
points

Non-response
Closed-ended questions
(Fixed-alternative responses)
What do you look for most in a job?
____ Work that pays well
____ Work that gives a sense of
accomplishment
____ Work where you make most decisions
by yourself
____ Work that is steady with little chance
of being laid
off.
Closed ended questions
Advantages Disadvantages
Ease of understanding Middle/Neutral categories
often selected inappropriately
(ignorance, safety)

Requires less effort on part of Less opportunity for self-
interviewer and respondent expression or subtle
qualifications

Ease of tabulation & analysis Less involving for respondents

Less error prone Order of response categories
can have major impact on
results

Less interviewer bias

Less time consuming

Answers directly comparable
Key tradeoff

• Want to get respondent to address
issues our research is concerned
with (Forced response) and at
same time give respondent
opportunity to honestly opt out of
question
• (i.e., Don’t Know, No Answer,
Neither Agree nor Disagree) so as
not to dilute data collected
Dichotomous Questions
Should the government give
consumers an energy rebate?
1.Agree
2.Disagree

Advantages Disadvantages
Easy to administer and Prone to large amounts
tabulate of error since polarized
responses prevent
gaining information on
the range of variation

Fail to communicate any
intensity of feeling
Multiple Choice Questions
Are all possible alternatives included?
Too many alternatives
Position Bias

Scaled Response Questions
Closed ended questions where the response
choices are designed to capture an intensity of
feeling (Likert, Staple, Semantic differential)
Easy to code and more powerful statistical tools
Main problem: Respondent misunderstanding
Question Wording
Can have major impact on how
respondent interprets question
All respondents should interpret
in the same way
Things to avoid
 Complexity: use simple, direct, conversational
language
 leading questions -- that suggest or imply certain
answers
 loaded questions -- suggest social desirability, or
are emotionally charged. Have you purchased a
high quality Sony TV this year
 Ambiguity and vagueness: Words such as “often”,
“occasionally”, “usually”, “regularly”,
“frequently”, “many”, should be used with
caution. If these words have to be used, their
meaning should be explained properly.
 Which district is bigger? Seremban or Nilai? Would
your answer be based on population or area?
More things to avoid
long-worded questions

 double-barreled questions.
Questions that refer to two or more
issues within the same question.
Where respondent may agree with
only 1 part of multipart statement.
 Do you think Nike offers better
pricing and variety than other
brands?
 Making implicit assumptions
More things to avoid
 burdensome questions - that may
tax the respondent’s memory
 How many tubes of toothpaste
have you purchased in the last 3
months?
 Have you purchased toothpaste
in the past week?
Embarrassing, sensitive, or
threatening questions:
 Have you charged more on your
credit card than you should?
Question Sequence & Layout Decisions
Initial stages
 Screening or qualifying questions:
 Have you been to the movie theatre this
month?
 Need to gain & maintain respondent’s
cooperation
 Make questionnaire simple for
interviewer to administer
Opening questions should be interesting,
simple, and easy to answer.
 Which theatre did you go to last?
Transition Questions
 What aspects of the theatre did you like
best?
Questions directly related to research
objectives which require more effort and
get respondent thinking about topic
Difficult and Complicated Question
The following 10 questions relate to the
characteristics of theatres
Respondent now committed and can see
an end in sight
Classifying and Demographic Questions
 What is your average annual household
income?
May not be answered (sensitive or
threatening) but most questions have
been answered
Other Considerations
Questions should flow logically from one to
the next
General questions should be asked before
more specific ones
Earlier questions should not influence
response to later ones
Questions should flow from factual and
behavioural questions to attitudinal and
opinion questions
Questions should flow from the least
sensitive to the most sensitive.
Questions should flow from unaided (which
brand do you prefer) to aided questions (which
brand do you prefer Tide, Cheer, Gain)
Demographic questions should come at
the end.
1. initial questions - screening and rapport
questions
2. Middle - product specific questions
3. End demographic questions.
Other Considerations
Use multiple questions instead of one
 Similar questions together
 consistent mindset for
respondents
 Develop a logical flow
 Use transitions between sections (E.g.
In this section we ask questions about
X)
Distinguish between instructions,
questions and responses
Distinguishing question and responses:
CAPS or BOLD or Underline versus lower case or unformatted

INSTRUCTIONS LAYOUT A:
Do you agree, disagree or have no opinion that this
company has:
• A good vacation policy - disagree/not sure/agree.
• Good management feedback - disagree/not sure/agree.
• Good medical insurance - disagree/not sure/agree.

INSTRUCTIONS LAYOUT B:
Does this company have:
___  Disagree Not Sure Agree    
• A good vacation policy   1       2       3   
• Good management feedback   1       2    3   
• Good medical insurance     1      2  3   
Filter and pivot questions should
be used as necessary.

 A FILTER question is one that
screens out respondents who are
not qualified to answer a second
question.
A PIVOT question is a type of
filter question that is used to
determine what version of a
second question to ask.
Developing a logical flow
 If the questionnaire deals with several topics,
complete questions on a single topic before
moving on to a new topic
 If topics are related, ask questions on related
topics before asking questions about unrelated
topics
 If you ask questions about behaviors over some
time period, follow chronological order
backward in time
 When changing topics, use some transitional
phrase
Questionnaire Layout
The layout and physical attractiveness
of a questionnaire are important
aspects
 Questionnaires should be designed
to appear as short as possible
 Questionnaires should not appear
overcrowded
Leave lots of space for open ended
questions
 Questionnaires in booklet form are
often recommended
Opening
 Provide first name (at least)
 Provide name of organisation doing
research
 Provide reason for survey and topic
(objective)
 State that no selling will be involved and
no personal data other than for
statistical purposes
 Tell respondent approximate time to
complete
 Reinforce that respondent’s time is
Closing
 Thank for time
 Ask if they had a positive
experience and remind them that
their opinions count
Pretesting and Correcting Problems
Purpose of pretest: To ensure that
the questionnaire meets the
expectations in terms of the
information that will be obtained
Is question necessary – does it serve
a purpose, will info be used?
Missing important variables – does it
provide the info needed?
Match questions to objectives
 Pretest Specific Questions For
• Variation
• Meaning
• Task difficulty
• Respondent interest and attention
• Ambiguous, ill-defined, loaded, double-barreled
questions
 Pretest the Questionnaire
• Flow of the questionnaire
• Skip patterns
• Length
 Respondent Interest and Attention
Questionnaire Design Flow Chart
Step 1: Specify what information will be sought
Step 2: Determine type of questionnaire and method of
administration
Step 3: Determine the content of individual questions
Step 4: Determine form of response for each question
Step 5: Determine wording for each question
Step 6: Determine sequence of questions
Step 7: Determine physical characteristics of
questionnaire
Step 8: Revise steps 1-7, revise if necessary
Step 9: Pretest questionnaire, revise if necessary
Step 10: implement
A QUESTIONNAIRE
IS ONLY AS GOOD
AS THE QUESTIONS
IT ASKS
Field notes
• Records of what the observer has seen or heard
• Contain literal descriptions as well as personal
reactions and comments on what the observer has
experienced and thought during an observation
session
• May be guided by a protocol developed before the
observation session
• May take the form of anecdotal records compiled as
part of a more systematic authentic assessment or
portfolio effort
• What you write depends on what you are looking for.
Positioning
Components of Effective Observation
Degrees of Participation
Participant • Becomes a part of, and a participant
Observer in the situation under observation
• Engage in activities
• Observe activities, people &
physical aspects
Privileged • A teacher’s aide during specialist’s
Observer time
Passive Observer • Watches but does not participate in
the situation while observing it
• Present only to observe what’s
going on
How to write field notes
(Wolcott, 1994)
Components of Effective Observation
Field notes
Observe and Attune to what you actually
record everything record through verbatim
conversations, maps and
illustrations, photos, and video
and audio recordings
Observe and look Try to see beyond the routine
for nothing and look with a fresh
perspective
Look for What are the unintended
paradoxes consequences of action?
The ‘don’t’
• Do not remove yourself from the site when
observation is supposed to take place
Educational records
(Mills, 2014)
• Archival documents
• Journals
• Maps
• Video and audio recordings
• Artifacts
Archival documents
• Sources that teacher researcher can access:
Student records
Minutes of meetings (faculty, PTA, school board)
Newspaper clippings about significant events in
the community
Students’ work
Journals/ Diaries/ Logs
• Students’ journals – provide researcher with a
valuable window into the students’ world ( as how
homework assignments provide parents with
insights of their children’s daily experiences)
• Can also use a daily journal to keep a narrative
account of their perspectives of what is happening
in their classrooms
Maps
• Extremely helpful for teacher researchers in trying
to monitor movements in a classroom – data that
are not easily recorded in a narrative form
Video and audio recordings
• Provide teacher researchers with another data
source when the teacher is fully engaged in
teaching but still wants to capture:
Classroom events
Interactions
Artifacts
• Classrooms – rich sources
• Written or visual sources of data that contribute to
our understanding of what is happening in our
classrooms and schools
Online focus group
(Stewart & William, 2005)
• Synchronous (real-time) and asynchronous (non
real-time)
• The interaction among participants will likely to
yield the best information when
• They are similar and cooperative with each other
• Time to collect information is limited
• Participant may be hesitant to provide information
on one-to-one basis
• (Kruger, 1994; Morgan, 1988, Stewart & Shamdasani, 1990)
Advantages & Disadvantages

• Advantages
 Participants can be questioned over long periods of
time
 Larger number can be managed
 More heated and open exchanges occur
• Disadvantages
 Obtaining complete informed consent
 Recruiting individuals to participate
 Choosing time to convene given different
international time zones
Photo elicitation
(Pink, 2001; Clandinin, 2006)
• Participants are shown pictures (their own/ those
taken by researcher)
• Participants are asked the content of the pictures
(Denzin & Lincoln, 1994)
Quantitative Data
Collection Tools
• Teacher-made tests
• Standardised tests
• School-generated report cards
• Attitude scales
• Likert scales
• Semantic differentials
Teacher-made tests

• Used to aid teachers in their ability to monitor and
adjust instruction (Creswell, 2014)
• Should not rely solely on the unit tests provided
by textbook companies to determine whether
students have achieved mastery of specific goals
and objectives
• Teachers are the experts, thus must make their
own test from scratch
• Data from this instrument provides accessible
information about how well their students are
responding to a particular teaching or curriculum
innovation
Standardised tests
• Intend to provide teachers, principals, parents and
other stakeholders with individual student
achievement data
• Provide teachers with a snapshot of how their
students are performing on a given subject test
relative to all other students taking the test
• Can be a source that contributes to our
understanding of how teaching practices affect
our students
School-Generated Report Cards
• Valuable data source for
 teachers (who can quantify student achievement)
 Students and parents (who can interpret the data
and set goals accordingly)
Attitude Scales
• Many teacher researchers are curious about the
impact of their work on students’ attitudes
• Scales – used to measure attitudes, such as Likert
scales and semantic differentials
• Allows teacher researchers to determine “what an
individual believes, perceives, or feels” (Gay, Mills
& Airasian, 2012, p.150)
Likert scale
Ask students to respond to a series of statements
indicating whether they:
 Strongly agree (SA)
 Agree (A)
 Undecided (U)
 Disagree (D)
 Strongly disagree (SD)

Each point corresponds to a point value and a score is
determined by adding the point values for each
statement
A high point value on a positively stated item would
indicate a positive attitude and a high total score on the
test would be indicative of a positive attitude (Gay,
Mills & Airasian, 2012, p. 130)
Semantic differential

• Asks a student (or parent) to give a quantitative
rating to the subject of the rating scale on a
number of bipolar adjectives
• Boring __ __ __ __ __ __ __ Exciting
-3 -2 -1 0 1 2 3
• Each location on the continuum between the
bipolar words has an associated score
• Totaling the score – teacher researcher can
determine whether a child’s attitude is positive or
negative
• Usually have five to seven intervals, with a neutral
attitude being assigned a value of zero
Digital research tools
• Web-Based survey tools
 SurveyMonkey.com
 Zoomerang
 LimeSurvey
 eSurveyspro
Triangulation
• The desire to use multiple sources of data
• Researchers should not rely on any single source of
data, interview, observation or instrument
• Action researchers complete a “triangulation matrix” –
a simple grid that shows the various data sources that
will be used to answer each research question (Sagor,
2000, p. 19-20)
• Pelto and Pelto (1978, p. 122) called this as the “multi-
instrument” approach
• Wolcott (1988, p. 192) suggests that the teacher is the
research instrument who, in collecting data, utilises a
variety of techniques over an extended period of time
• Can involve both qualitative and quantitative
approaches
Triangulation Matrix
Exercise
Research Data Source
Questions 1 2 3

1.
2.
3.
4.
5.
Example
Research Data Source
Questions 1 2 3

1. Student Survey Computer
Preexisting Knowledge
knowledge? Pretest
2. Pretest Posttest Teacher help
Keyboarding
speed?
3. Pretest Posttest Timed typing
Appropriate software software teacher
use? constructed
4. Time on School lab Student survey Parent survey
computers? records
Any questions?