BUS5301

Research Methods in Education

Week 7
Data Collection
Assoc. Prof. Dr. Nihal Yurtseven
nihal.yurtseven@es.bau.edu.tr
Quantitative Data Collection
Data Collection Procedures
1. Identify the Varible

• Research questions and hypotheses contain variables. To determine

what data need to be collected, you need to identify clearly the
variables in your study (independent, dependent, and control
variables).
2. Operationally Define Each Varible
• An operational definition is the specification of how you will define
and measure the variable in your study.
• You can find definitions in published research studies on your topic.
Sometimes published studies have sections titled “Definition of
Terms.”
3. Locate Data
• With operational definitions for your variables, you next need to
identify types of data that will measure your variables. Researchers
collect data on instruments.
• An instrument is a tool for measuring, observing, or documenting
quantitative data. Identified before the researchers collect data, the
instrument may be a test, questionnaire, observational checklist,
inventory, or assessment instrument.
• Researchers use instruments to measure achievement, assess
individual ability, observe behavior, develop a psychological profile of
an individual, or interview a person.
 4. Collect Data

Standardization
Ethical Issues

Protecting the anonymity of individuals by assigning

it is important to use standard procedures. When
numbers to returned instruments and keeping the identity
procedures vary, you introduce bias into the study and
of individuals confidential offers privacy to participants.
the data for individuals may not be comparable for
During data collection, you must view the data as
analysis. Written procedures, as well as other data
confidential and not share it with other participants or
collectors assisting in the process, help keep you on
individuals outside of the project.
track.
 Types of Data

Performance Measures

Attitude Measures

Behavioral Observations

Factual Information
Performance Tests
• They are used to assess an individual’s ability to perform on an
achievement test, intelligence test, aptitude test, interest inventory,
or personality assessment inventory. Participants take tests that
measure their achievement (e.g., the Iowa Test of Basic Skills), their
intelligence (e.g., Wechsler), or their aptitude (e.g., Stanford–Binet).

• Through past research, researchers have developed “norms” for these

tests (conducted the tests with a number of individuals, averaged
their scores, and looked at the differences in their scores) so that they
can compare individual scores with typical scores for people who
have taken the test.
Attitude Tests
• Researchers use these tests when they measure feelings toward
educational topics (e.g., assessing positive or negative attitudes
toward giving students a choice of school to attend).
Behavioral Observations
• To collect data on specific behaviors, you can observe behavior and
record scores on a checklist or scoring sheet.
• Behavioral observations are made by selecting an instrument (or
using a behavioral protocol) on which to record a behavior, observing
individuals for that behavior, and checking points on a scale that
reflect the behavior (behavioral checklists).
Factual Information

• Factual information or personal documents consist of numeric,

individual data available in public records. Examples of these types of
data include grade reports, school attendance records, and student
demographic data.
Absence Records for
EDT5002
How to Decide What Types to Choose
• What am I trying to learn about participants from my research questions
and hypotheses? If you are trying to learn about individual behaviors of
parents at a student–parent conference meeting, then you could use a
behavioral checklist and record observations. If you are trying to measure
the attitudes of teachers toward a bond issue, attitudinal questions or an
attitudinal instrument will be required.
• What information can you realistically collect? Some types of data may
not be collectible in a study because individuals are unwilling to supply it.
For example, precise data on the frequency of substance abuse in middle
schools may be difficult to collect; identifying the number of student
suspensions for substance abuse is much more realistic.
• How do the advantages of the data collection compare with its
disadvantages? In our discussion of each data source, we have talked
about the ideal situations for data collection. Given the ease or difficulty of
collecting data, each type needs to be assessed.
What instrument will you use to collect data?

Using an already existing instrument directly

Using an already existing instrument by modifying

Developing your own instrument

• Of the choices, using an already existing instrument is the easiest option. It
is more difficult to develop an instrument than to locate one and modify it
for use in a study.
• Modifying an instrument means locating an existing instrument, obtaining
permission to change it, and making changes in it to fit your requirements.
Typically, authors of the original instrument will ask for a copy of your
modified version and the results from your study in exchange for your use
of their instrument.
• An instrument to measure the variables in your study may not be available
in the literature or commercially. If this is the case, you will have to
develop your own instrument, which is a long process.
• Developing an instrument consists of several steps, such as identifying the
purpose of the instrument, reviewing the literature, writing the questions,
and testing the questions with individuals similar to those you plan to
study.
How to search for an instrument?
• If you decide to use an existing instrument, the publisher or author
will typically charge you a fee for use of the instrument. Finding a
good instrument that measures your independent, dependent, and
control variables is not easy.
• Look in published journal articles. Often authors of journal articles
will report instruments and provide a few sample items so that you
can see the basic content included in the instrument. Examine
references in published journal articles that cite specific instruments
and contact the authors for inspection copies. Before you use the
instrument, seek permission from the author. With limited space in
journals, authors are including fewer examples of their items or
copies of their instruments.
How to search for an instrument?
• Run an ERIC search. Use the term instruments and the topic of the
study to search the ERIC system for instruments. Use the online
search process of the ERIC database. Use the same search procedure
to locate abstracts to articles where the authors mention instruments
that they have used in their studies.
• Examine guides to tests and instruments that are available
commercially. More than 400 commercial firms develop instruments
that are available for sale to individuals and institutions.
 Scale Development
Expert
opinion

1. Identify 2. Create 3. For the 4. Applt the

the purpose items trial version scale, analyze
and interpret
results

Literature Expert

Interview Observation

23
How can we decide whether the instrument
we choose is a good one?
Criteria for Choosing a Good Instrument
• Have authors developed the instrument recently, and can you obtain
the most recent version? (older than 5 years old is outdated)
• Is the instrument widely cited by other authors?
• Are reviews available for the instrument?
• Is there information about the reliability and validity of scores from
past uses of the instrument?
• Does the procedure for recording data fit the research
questions/hypotheses in your study?
 Validity

Reliability

Usability
Validity
• Validity is the level in which a measuring tool can measure precisely
and accurately the property it wants to measure without mixing other
features. That is, the more accurate a measuring instrument can
measure the property it wants to measure, the more valid it is.

• Validity is the degree to which all of the evidence points to the

intended interpretation of test scores for the proposed purpose.
Thus, a focus is on the consequences of using the scores from an
instrument
Types of Validity
Content Validity

Construct Validity

Face validity

Criterion Validity

Predictive Validity
Reliability
• The measuring instrument should always be able to measure the
desired feature in the same way, be stable and consistent.
• Scores should be nearly the same when researchers administer the
instrument multiple times at different times. Also, scores need to be
consistent. When an individual answers certain questions one way,
the individual should consistently answer closely related questions in
the same way. Several factors can result in unreliable data, including
when:
ü Questions on instruments are ambiguous and unclear
ü Procedures of test administration vary and are not standardized
ü Participants are fatigued, are nervous, misinterpret questions, or
guess on tests
Reliability Methods
1. Methods Based on Single Application (Internal Consistency)
a. Kuder Richardson (KR20-KR21)
b. Cronbach Alpha
c. Gutman’s Split Half

2. Methods Based on Two Applications

a. Alternative forms reliability
b. Test-retest reliability
Usability

• The instrument should be easy to apply, easy to score, and time and
cost effective.
Closing Remarks on Quantitative Data Collection
• When collecting data for a study, plan to engage in five steps: (1)
selecting participants, (2) obtaining permissions, (3) selecting types of
data, (4) identifying instruments, and (5) administering data
collection.
• Identify the population and sample for a study. There are several
types of probability and nonprobability sampling. The most rigorous
sampling will be simple random sampling. However, the research
circumstances may dictate a form of non-probability sampling.
• Select as large a sample as possible. Use sampling formulas to be
systematic in selecting the size of the sample.
• Obtain permission to conduct a study.
Closing Remarks on Quantitative Data Collection
• Consider how the research questions or hypotheses will be answered
when deciding on what data type(s) to use. Then identify your
variables, operationally define them, and select measures (e.g.,
performance and attitudes, observations of behavior, and factual and
personal data) that fi t the operational definitions.
• Decide whether to develop your own instrument or to use an existing
instrument for your research.
• Consider the types of scales you plan to use on your instruments.
These scales will affect the types of statistics to be used in analyzing
the data. Make sure they relate to your questions.
• Before deciding on an instrument to use, make sure that the scores
from past uses of it are reliable and valid.
Qualitative Data Collection
Qualitative Data Collection

What types of data

Observation are you going to
collect?

Interview

Document

Audiovisual Materials
 Observation
• Observation is the process of gathering open-ended, firsthand
information by observing people and places at a research site. As a
form of data collection, observation has both advantages and
disadvantages.
• Advantages include the opportunity to record information as it
occurs in a setting, to study actual behavior, and to study individuals
who have difficulty verbalizing their ideas (e.g., preschool children).
Some of the disadvantages of observations are that you will be
limited to those sites and situations where you can gain access, and
in those sites, you may have difficulty developing rapport with
individuals.
 Interview
• Equally popular to observation in qualitative research is
interviewing. A qualitative interview occurs when researchers ask
one or more participants general, open-ended questions and
record their answers. The researcher then transcribes and types
the data into a computer file for analysis.

• In qualitative research, you ask open-ended questions so that the

participants can best voice their experiences unconstrained by any
perspectives of the researcher or past research findings. An open-
ended response to a question allows the participant to create the
options for responding.
Forms of Interview

Individual

Focus
Telephone
Group

E-mail
Documents
• Documents consist of public and private records that qualitative
researchers obtain about a site or participants in a study. They can
include newspapers, minutes of meetings, personal journals, and
letters. These sources provide valuable information in helping
researchers understand central phenomena in qualitative studies.
Audiovisual Materials

• Audiovisual materials consist of images or sounds that researchers

collect to help them understand the central phenomenon under
study. Used with increasing frequency in qualitative research, images
or visual materials such as photographs, videotapes, digital images,
paintings and pictures.
• Images provide an opportunity for the participants to share directly
their perceptions of reality. Images such as videotapes and films, for
example, provide extensive data about real life as people visualize it.
A potential disadvantage of using images is that they are difficult to
analyze because of the rich information.
Recording Data
• An essential process in qualitative research is recording data.
• For documents and visual materials, the process of recording
information may be informal (taking notes) or formal (optically
scanning the material to develop a complete computer text file). For
observations and interviews, qualitative inquirers use specially
designed protocols.
• Data recording protocols are forms designed and used by qualitative
researchers to record information during observations and interviews.
• Audiotaping of interviews provides a detailed record of the interview.
Validity & Reliability in Qualitative Studies
(Lincoln & Guba, 1984, Naturalistic Inquiry)

Credibility

Transferability

Dependability

Confirmability
Credibility

• The confidence that can be placed in the truth of the research

findings. Credibility establishes whether the research findings
represent plausible information drawn from the participants’ original
data and is a correct interpretation of the participants’ original views.
Credibility

Prolonged Persistent Member

Triangulation Audit trail
engagement observation check
Transferability
• The degree to which the results of qualitative research can be transferred
to other contexts or settings with other respondents. The researcher
facilitates the transferability judgment by a potential user through thick
description.

• Describing not just the behaviour and experiences, but their context as
well, so that the behaviour and experiences become meaningful to an
outsider
Dependability
• The stability of findings over time. Dependability involves participants’
evaluation of the findings, interpretation and recommendations of the
study such that all are supported by the data as received from participants
of the study.

• Transparently describing the research steps taken from the start of a

research project to the development and reporting of the findings. The
records of the research path are kept throughout the study.
Confirmability

• The degree to which the findings of the research study could be

confirmed by other researchers. Confirmability is concerned with
establishing that data and interpretations of the findings are not
figments of the inquirer’s imagination, but clearly derived from the
data.
Closing Remarks for Qualitative Data Collection
• Plan five phases when designing qualitative data collection: selecting
participants, gaining permissions, selecting and weighing different
data types, designing recording protocols, and administering data
collection.
• Use a purposeful sampling strategy that matches your intent for the
study. Consider using sampling at different levels (e.g., organization,
site, individual, parent). It is better to select a few, rather than many,
individuals or sites to study, to provide an in-depth understanding of
the phenomenon.
Closing Remarks for Qualitative Data Collection

• Weigh the advantages and disadvantages of each major form of

qualitative data and each subtype (i.e., observations, interviews,
documents, and audiovisual material) before selecting the type(s) to
use in your study. Also consider what type of data will best address
your research questions.
• Consider how you will record information during data collection. Use
interview and observational protocols to structure the data collection
as well as to record information.