Session 1

Research Method
Finals Review
Session 1
Table of contents
01 Review of Key Concepts

02 Review of APA (7th Edition)
01
Review of
Key Concepts
Research
Definition: Research is defined as a systematic approach to finding answers
to questions (Hatch and Farhady, 1982, p. 1).
Three key words: → We begin our research by asking

1. Questions questions and by using a systematic
2. Systematic approach approach in our investigation, and end
3. Answers with answers to the questions
Research design
Definition: Research design is a logical plan of getting from here
to there, where here may be defined as the initial set of questions to
be answered, and there is some set of conclusions (answers) about
these questions. Between “here” and “there” may be found a number
of major steps, including the collection and analysis of relevant data.
(Yin, 2003, p. 20)
Research design
Research questions
Definition: Research questions refer to the relationship among a
small number of variables. Identify a limited number of variables,
and specify the relationships among them. (Neuman, 1994, p. 111).
You need a narrowly focused research question before you design
a research project.
Hypothesis
• Rather than including specific information and directions in the research
question, which would make it cumbersome and unmanageable, to
obtain more specificity and direction, we develop hypotheses.
• Definition: Hypothesis is a tentative statement about the outcome of
the research.
Hypothesis
• Hypotheses may be
• derived directly from the statement of the problem
• they may be based on the research literature
• or in some cases, such as ethnographic research, they may (at least in part) be
generated from data collection and analysis.
• Hypotheses range from quite formal statements (in the presence of adequate literature
and facts) to your current best guess (in the absence of adequate information)
Null hypothesis
• Definition: Null hypothesis (usually characterized as H0) predicts
neither a positive nor a negative relationship between two variables.
• Usually we try to reject the null hypothesis and support either the
negative or the positive hypothesis (called the alternative hypothesis
and characterized as H1).
Internal Validity
• Definition: The degree to which a study measures what it intends to measure.
• Factors that threaten internal validity: selection bias, history, maturation, testing,
instrumentation, regression to the mean, and experimental mortality.
• Enhancing internal validity: can be achieved through strategies such as random

assignment, matching, statistical control, and holding variables constant.
• Significance: Internal validity is crucial for establishing cause-and-effect relationships

between variables in a research study.
External Validity
• Definition: the extent to which the findings of a study can be generalized
to other populations, settings, and time periods beyond the specific sample
and context of the study.
• Factors that threaten external validity: the representativeness of the

sample, the similarity of the study context to the real-world context, and
the timing of the study relative to other historical events or changes.
External Validity
• Enhancing external validity: can be achieved through strategies such as
using a diverse and representative sample, replicating the study in different
settings or with different populations, and using multiple methods or
measures to confirm the findings.
• Significance: External validity helps to ensure that research findings have

practical relevance and can be applied to real-world situations beyond the
immediate study context.
Reliability
• Definition: The consistency and stability of research findings over time, across different researchers,
and under different conditions.
• Different types of reliability:

o Test-retest reliability: measures the consistency of results when the same test is administered to the same group of
participants at two different times.
o Inter-rater reliability: measures the consistency of results when different raters or observers rate the same behavior or event.
o Parallel forms reliability: measures the consistency of results when two versions of a test are administered to the same group
of participants.
o Internal consistency reliability: measures the consistency of results across items within a test.
Reliability
• Factors that threaten reliability: participant variability, measurement error, and experimenter bias.
• Enhancing reliability: can be achieved through strategies such as using standardized procedures,
training raters or observers, minimizing measurement error, and using multiple measures of the
same construct.
• Significance: Research reliability is essential for making comparisons between groups or across
time, as it ensures that any observed differences are not due to chance or measurement error. By
ensuring research reliability, we can have confidence in the validity of research findings, which is
crucial for making informed decisions and advancing knowledge in a field.
Validity vs. Reliability
Ensures that the conclusions drawn from Ensures that the results of a study are
a study are accurate and trustworthy. consistent and dependable.
→ The relationship between research validity and research reliability is complex. A study can be both
reliable and valid, or it can be reliable but not valid, or valid but not reliable. For example, a bathroom
scale that consistently measures weight with a high degree of precision is reliable but might not be valid
if it measures only the weight of one leg. On the other hand, a test that accurately measures a construct
but produces inconsistent results is valid but not reliable.
Validity vs. Reliability
Ensures that the conclusions drawn from Ensures that the results of a study are
a study are accurate and trustworthy. consistent and dependable.
• The relationship between research validity and research reliability is complex. A study can be both reliable
and valid, or it can be reliable but not valid, or valid but not reliable.
• In general, research validity is more important than research reliability. This is because a study that is not
valid is fundamentally flawed and cannot be trusted, while a study that is not reliable may still be useful.
Variable
• Definition: A variable can be defined as an attribute of a person or of
an object which “varies” from person to person or from object to object.
• Functions: Variables can be classified as dependent, independent, or
moderator variables. It is also possible to have intervening and control
variables as well
Dependent Variable
• Definition: The dependent variable is the variable which you observe and
measure to determine the effect of the independent variable.
Independent Variable
• Definition: The independent variable is the major variable which you hope to
investigate. It is the variable which is selected, manipulated, and measured by
the researcher.
Dependent vs. Independent Variable
Determine the dependent and independent variable in the following study:
Investigating the Effectiveness of Flashcards on

Vocabulary Acquisition among High School English
Language Learners.
Investigating the Effectiveness of Flashcards on

Vocabulary Acquisition among High School
English Language Learners.
The Impact of Grammar Exercises on English

Writing Skills among English Language Learners
The Impact of Grammar Exercises on English

Writing Skills among English Language Learners
A Comparative Study of Authentic and Simplified

Texts on Reading Comprehension among English
Language Learners
A Comparative Study of Authentic and Simplified

Texts on Reading Comprehension among
English Language Learners
Enhancing Listening Skills among English

Language Learners through Cooperative
Learning Strategies
Enhancing Listening Skills among English

Language Learners through Cooperative
Learning Strategies
Exploring the Relationship between Teacher

Self-Efficacy and English Language Learners'
Academic Achievement
Exploring the Relationship between Teacher

Self-Efficacy and English Language Learners'
Academic Achievement
Moderator Variable
• Definition: A moderator variable is a special type of
independent variable which you may select for study in order to
investigate whether it modifies the relationship between the
dependent and the major independent variables.
Intervening Variable
• Definition: An intervening variable is a process underlying the behavior we
are measuring which is usually neither observable nor measurable.
Control Variable
• Definition: A control variable is a variable which is held constant in order to
neutralize the potential effect in might have on behavior.
Practice: Determine the variables
Determine the variables in the following study:
A study examines the effects of a reading intervention program

on English language learners' reading comprehension skills.
The study also examines whether the effects of the program
are moderated by students' English proficiency level.
A study examines the effects of a reading

• Independent variable: reading intervention program
intervention program on English language
learners' reading comprehension skills. • Dependent variable: English language learners’ reading
The study also examines whether the
comprehension skills
effects of the program are moderated by
students' English proficiency level. • Moderator variable: students’ English proficiency level
Researchers are interested in whether the use of a vocabulary-

building app improves English language learners' vocabulary
acquisition. They also examine whether the effects of the app
are mediated by students' motivation to learn.
Researchers are interested in whether the

• Independent variable: use of a vocabulary-building app
use of a vocabulary-building app
improves English language learners' • Dependent variable: English language learners'
vocabulary acquisition. They also examine
vocabulary acquisition
whether the effects of the app are
mediated by students' motivation to learn. • Intervening variable: Students’ motivation
A study investigates the effects of teacher feedback on English

language learners' writing skills. The study also examines
whether the effects of feedback are influenced by the
students' level of self-efficacy.
A study investigates the effects of • Independent variable: teacher feedback

teacher feedback on English language
learners' writing skills. The study also • Dependent variable: English language learners'
examines whether the effects of writing skills
feedback are influenced by the
students' level of self-efficacy. • Moderator variable: Students’ level of self-efficacy
Researchers are interested in whether the use of a speaking

practice app improves English language learners' speaking
skills. They also examine whether the effects of the app are
influenced by the amount of practicing time.
Researchers are interested in whether the

use of a speaking practice app improves • Independent variable: use of a speaking practice app
English language learners' speaking skills.
• Dependent variable: English language learners'
They also examine whether the effects of
the app are influenced by the amount of speaking skills
practicing time.
• Moderator variable: amount of practicing time
A study investigates the effects of explicit grammar instruction on

English language learners' grammar skills. The study also examines
whether the effects of instruction are mediated by students'
working memory capacity. Additionally, the study holds constant
students' prior English language proficiency level.
A study investigates the effects of explicit • Independent variable: explicit grammar instruction
grammar instruction on English language
learners' grammar skills. The study also
• Dependent variable: grammar skills
examines whether the effects of instruction are • Intervening variable: Students’ level of self-efficacy
mediated by students' working memory capacity.
• Control variable: Students’ prior English language
Additionally, the study holds constant students'
prior English language proficiency level. proficiency level
Researchers are interested in whether the use of peer feedback

improves English language learners' writing skills. The study also
examines whether the effects of feedback are influenced by
students' cultural background. Additionally, the study keeps students'
prior English language proficiency level at a constant level.
Researchers are interested in whether the use of • Independent variable: use of peer feedback
peer feedback improves English language
• Dependent variable: English language learners'
learners' writing skills. The study also examines
writing skills
whether the effects of feedback are influenced
by students' cultural background. Additionally, • Moderator variable: Students’ cultural background
the study keeps students' prior English language • Control variable: Students’ prior English language
proficiency level at a constant level. proficiency level`
Quantitative Data Categories
1. Nominal – categories (e.g., Hindu, Catholic)
2. Ordinal – ranked categories (e.g., strongly agree, agree, undecided, disagree, etc.)
3. Interval – quantifiable difference between categories (e.g., grade or IQ)
4. Ratio – Categories can be related as proportions (e.g., income, years at school)

means to name; hence a nominal scale does not actually measure but rather names. Observations
are simply classified into categories with no necessary relationship existing between the categories.
… The relationship between the categories is that they are different from each other.

implies the ability to put data into rank order. Ordinal numbers convey more information,
in that their relative magnitude is meaningful.


Scales in which equal differences between scores or intervals can be treated as equal units … Interval scales, in
addition to incorporating orderings, have the property that there is a specific distance between each pair of levels.
Hence, we can compare values not only in terms of which is larger or older but also in terms of how much larger or
how much older. In interval scales, the distance between all adjacent levels are equal.

Allows subtraction and addition but does not necessarily allow multiplication or division.
Length and weight are ratio scales.
Triangulation
• It is good research practice to obtain data related to a given construct or
situation from a variety of sources.
• 4 types of triangulation
1. Variety of data source
2. Use of several evaluators
3. Use of multiple theories to interpret a set of data
4. Use of multiple method
Sampling
• Apparent conclusions of our research are
• less by the social reality under investigation
• more by the nature of the samples we use to collect data
→ Sampling is the basis of all research
Sampling
• Purpose: To use a relatively small number of cases to find out about a much larger number
• ‘Population’: group we wish to study
• ‘Sample’: group we actually involve in our research
• Representation:
• Sample drawn from a previously defined population
• sample being representative of this population
→ The researcher only sets out to generalise to the population from which the sample
was drawn
Sampling
1. Decide whether to use a sample, and why. 5. Choose a method of selecting population
2. Define the population of interest (be as elements (consider random, systematic,
precise as possible). stratified, clustered, or non-probability).
3. List the inhabitants of the population 6. Decide on methods of correction (consider
(create a sampling frame), or characterize response rate, refusal rate).
the population. 7. Characterize the achieved sample &
4. Estimate the size of sample you need. compare with the ideal (or the population).
Make it large 8. Apply corrections if necessary.

Sampling
→ In summary, a good sample is representative of a wider
population, and large, with a high participation rate. It is
risky to accept the generalizations made in previous work,
without first considering their sampling strategy and the
potential biases this introduces.
Sampling
• Sampling Frame: The list of all cases [in the study] because in real life your sampling frame
will be an incomplete list of population. This is not to be confused with population list.
• How large a sample should be? → To have as large a sample as possible
• Confidence interval: an indication of accuracy of findings as estimates for population. This

is required for findings since the intention of sampler is to generalize results to population
of interest → larger sample, more accurate results, smaller confidence interval
Probability Sampling
1. Simple: Select people based on a true random procedure.
2. Systematic: Select every kth person (quasi-random)
3. Stratified: Randomly select people based on predetermined groups
4. Cluster: Take multi-stage random samples in each of several levels




Probability Sampling - Practice
Determine which probability sampling method the following study used:
A language school wants to conduct a study on the effectiveness of a new

teaching method for improving students' speaking skills. They obtain a list
of all students who have signed up for the course and randomly select 50
students from the list to participate in the study. What type of probability
sampling method is being used in this study?
A language school wants to conduct a study on the effectiveness of a new

teaching method for improving students' speaking skills. They obtain a list
of all students who have signed up for the course and randomly select 50
students from the list to participate in the study. What type of probability
sampling method is being used in this study?
→ Simple random sampling

A researcher wants to conduct a study on the use of corrective feedback in English

language writing classes. She obtains a list of all English language writing classes in
the state and randomly selects 20 classes from the list. She then randomly selects
10 students from each selected class to participate in the study. What type of
probability sampling method is being used in this study?
A researcher wants to conduct a study on the use of corrective feedback in English
language writing classes. She obtains a list of all English language writing classes in
the state and randomly selects 20 classes from the list. She then randomly selects
10 students from each selected class to participate in the study. What type of
→ Cluster random sampling

A researcher wants to conduct a study on the use of authentic materials in

English language teaching. They obtain a list of all English language teachers in
the state and use a systematic sampling method to select every 10th teacher
from the list until they reach a sample size of 100 teachers. What type of
A researcher wants to conduct a study on the use of authentic materials in
English language teaching. They obtain a list of all English language teachers in
the state and use a systematic sampling method to select every 10th teacher
from the list until they reach a sample size of 100 teachers. What type of
→ Systematic random sampling

4A language school wants to conduct a study on the effectiveness of a new

vocabulary teaching approach for different language proficiency levels. They obtain
a list of all students who have enrolled in their courses and divide them into three
strata based on their proficiency level (beginner, intermediate, advanced). They
then randomly select 20 students from each stratum to participate in the study.
What type of probability sampling method is being used in this study?
4A language school wants to conduct a study on the effectiveness of a new
vocabulary teaching approach for different language proficiency levels. They obtain
a list of all students who have enrolled in their courses and divide them into three
strata based on their proficiency level (beginner, intermediate, advanced). They
then randomly select 20 students from each stratum to participate in the study.
What type of probability sampling method is being used in this study?
→ Stratified random sampling

A researcher wanted to find out something about the degree of anomie
suffered by foreign students enrolled in universities in U.S. They pulled together
a master list of foreign students from every university in U.S and pulled all the
names in the hat & picked 200 as the sample. However, they listed the names
and noticed that 174 of them happen to be at universities in New York.
→ In order to prevent this, what type of probability sampling should the
researcher use?
A researcher wanted to find out something about the degree of anomie suffered by foreign
students enrolled in universities in U.S. They pulled together a master list of foreign students from
every university in U.S and pulled all the names in the hat & picked 200 as the sample. However,
they listed the names and noticed that 174 of them happen to be at universities in New York.
→ In order to prevent this, what type of probability sampling should the researcher use?
→ Stratified random sampling

Non-probability Sampling
1. Haphazard (Convenience): Select anyone who is convenient
2. Quota: Select anyone from pre-determined groups
3. Snowball: Select people connected to one another
4. Purposive: Select anyone from a hard-to-find target population





Determine which non-probability sampling method the following study used:
A researcher wants to conduct a study on the experiences of English language learners

who have participated in a language exchange program. She obtains a list of all English
language learners who have participated in the program in the past year and contacts
the first 5 learners on the list to participate in the study. She then asks these learners to
refer other learners who have participated in the program to participate in the study.
What type of non-probability sampling method is being used in this study?
A researcher wants to conduct a study on the experiences of English language learners
who have participated in a language exchange program. She obtains a list of all English
language learners who have participated in the program in the past year and contacts
the first 5 learners on the list to participate in the study. She then asks these learners to
refer other learners who have participated in the program to participate in the study.
→ Snowball sampling
A language testing company wants to conduct a study on the reliability of a new

speaking test. They obtain a list of all English language learners who have taken the
test in the past year and select 200 learners who are available to take the test again.
A language testing company wants to conduct a study on the reliability of a new

speaking test. They obtain a list of all English language learners who have taken the
test in the past year and select 200 learners who are available to take the test again.
→ Quota sampling
A researcher wants to conduct a study on the attitudes of English language

learners towards the use of mobile devices in language learning. She obtains a
list of all English language learners in the city and selects the first 50 learners
who agree to participate in the study. What type of non-probability sampling
method is being used in this study?
A researcher wants to conduct a study on the attitudes of English language

learners towards the use of mobile devices in language learning. She obtains a
list of all English language learners in the city and selects the first 50 learners
who agree to participate in the study. What type of non-probability sampling
method is being used in this study?
→ Convenience sampling
A researcher wants to conduct a study on the experiences of English

language teachers who use online teaching platforms. She obtains a list of all
English language teachers in the country and selects 20 teachers who have
experience using online platforms and are willing to participate in the study.
A researcher wants to conduct a study on the experiences of English language teachers who use
online teaching platforms. She obtains a list of all English language teachers in the country and selects
20 teachers who have experience using online platforms and are willing to participate in the study.
→ Purposive sampling
Questionnaires
• A questionnaire is a self-report instrument designed to gather ‘valid’ and ‘reliable’ information
• Questions may be structured (specified) or unstructured (open or unspecified)
• The use of questionnaires as data-gathering research instruments is based on 3 key assumptions:
1. Respondents can read and understand items
2. Respondents possess the necessary information to answer the items
3. Respondents are willing to answer the items honestly

Principles of questionnaire design 1:
The basics
• Select only those potential items (variables) that are consistent with: (1) specific and
theoretically-driven research questions, (2) operationally-defined constructs, AND
(3) the target sample or population
• Limit questions/items to only those of major interest
• Item development is crucial and should be based on extensive:
(a) interviewing (b) piloting (c) pre-testing

Principles of questionnaire design 2:
Constraints
• Time:
o Not more than 30 minutes for adults (about 15 mins for mail questionnaire)
o Not more than 10-15 minutes for children and adolescents
• Sensitivity or delicacy of content
• Design → The attributes of a well-designed questionnaire do not arise

naturally out of the process of questionnaire construction, but
• Avoid ‘non-response’ bias
are the results of careful developmental work
Principles of questionnaire design 3
• Decide major research question(s) variables and their possible multiple indicators
• Decide key demographic and classificatory variables [e.g., age, sex, education, occupation, residential
(or work) location, etc.]
• Design response categories to avoid potential ‘non-response bias’

• Use simple nomenclature, direct statements/questions
• Avoid, at all costs, double- or multiple-barreled questions
• Wherever possible, avoid the use of negatively-worded questions/items; i.e., state all items in the POSITIVE
• TRYOUT potential items with respondents drawn from the target
sample/population
• Revise wording and ordering of items
• Pay particular attention to LAYOUT
• Conduct an ‘empirical’ TRIAL of the instrument and be prepared to revise the

instrument drastically
• ADMINISTER the revised questionnaire

02
Review of
APA (7 Edition)
th
Punctuation
• Ensure that a period appears after each reference element-
that is, after the author, date, title, and source. However, do not
put a period after a DOI or URL because it may interfere with link
functionality.
Punctuation
• Use punctuation marks (usually commas or parentheses) between parts of
the same reference element. For example, in a reference for a journal article,
use a comma between each author's last name and initials and between
different authors' names, between the journal name and the volume number,
and between the journal issue number and the page numbers. Do not use a
comma between the journal volume and issue numbers; place the issue
number in parentheses instead (see Section 9.25 and Chapter 10, Example 1).
Identification of Specialized Roles
• Use the abbreviation "(Ed.)" for one editor and the abbreviation "(Eds.)" for
multiple editors. In the case of multiple editors, include the role once.
Group Authors
• Spell out the full name of a group author in the reference list entry, followed by
a period: National Institute of Mental Health.
Thank you
&
Good luck

Session 1

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Session 1

Uploaded by

Copyright:

Available Formats

Research Method

01 Review of Key Concepts

Three key words: → We begin our research by asking

• Enhancing internal validity: can be achieved through strategies such as random

• Significance: Internal validity is crucial for establishing cause-and-effect relationships

• Factors that threaten external validity: the representativeness of the

• Significance: External validity helps to ensure that research findings have

• Different types of reliability:

Investigating the Effectiveness of Flashcards on

Investigating the Effectiveness of Flashcards on

The Impact of Grammar Exercises on English

The Impact of Grammar Exercises on English

A Comparative Study of Authentic and Simplified

A Comparative Study of Authentic and Simplified

Enhancing Listening Skills among English

Enhancing Listening Skills among English

Exploring the Relationship between Teacher

Exploring the Relationship between Teacher

A study examines the effects of a reading intervention program

A study examines the effects of a reading

Researchers are interested in whether the use of a vocabulary-

Researchers are interested in whether the

A study investigates the effects of teacher feedback on English

A study investigates the effects of • Independent variable: teacher feedback

Researchers are interested in whether the use of a speaking

Researchers are interested in whether the

A study investigates the effects of explicit grammar instruction on

Researchers are interested in whether the use of peer feedback

3. Interval – quantifiable difference between categories (e.g., grade or IQ)

4. Ratio – Categories can be related as proportions (e.g., income, years at school)

3. Interval – quantifiable difference between categories (e.g., grade or IQ)

4. Ratio – Categories can be related as proportions (e.g., income, years at school)

3. Interval – quantifiable difference between categories (e.g., grade or IQ)

4. Ratio – Categories can be related as proportions (e.g., income, years at school)

3. Interval – quantifiable difference between categories (e.g., grade or IQ)

4. Ratio – Categories can be related as proportions (e.g., income, years at school)

3. Interval – quantifiable difference between categories (e.g., grade or IQ)

4. Ratio – Categories can be related as proportions (e.g., income, years at school)

2. Define the population of interest (be as elements (consider random, systematic,

precise as possible). stratified, clustered, or non-probability).

3. List the inhabitants of the population 6. Decide on methods of correction (consider

(create a sampling frame), or characterize response rate, refusal rate).

the population. 7. Characterize the achieved sample &

Make it large 8. Apply corrections if necessary.

• How large a sample should be? → To have as large a sample as possible

• Confidence interval: an indication of accuracy of findings as estimates for population. This

2. Systematic: Select every kth person (quasi-random)

3. Stratified: Randomly select people based on predetermined groups

4. Cluster: Take multi-stage random samples in each of several levels

2. Systematic: Select every kth person (quasi-random)

3. Stratified: Randomly select people based on predetermined groups

4. Cluster: Take multi-stage random samples in each of several levels

2. Systematic: Select every kth person (quasi-random)

3. Stratified: Randomly select people based on predetermined groups

4. Cluster: Take multi-stage random samples in each of several levels

3. Stratified: Randomly select people based on predetermined groups

4. Cluster: Take multi-stage random samples in each of several levels

A language school wants to conduct a study on the effectiveness of a new

A language school wants to conduct a study on the effectiveness of a new

→ Simple random sampling

A researcher wants to conduct a study on the use of corrective feedback in English

→ Cluster random sampling

A researcher wants to conduct a study on the use of authentic materials in

→ Systematic random sampling