Research Methodology

1
1
RESEARCH
METHODOLOGY
(Module I I I )
Dr. M.N.Vinodkumar
Professor
Division of Safety and Fire Engineering
School of Engineering
CUSAT.
2
WHAT I S RESEARCH ?
Research is a structured enquiry that utilizes
acceptable scientific methodology to solve
problems and create new knowledge that is
generally applicable.
A study conducted to find answers to a
question makes it a research study if -
1. It is being undertaken within a framework of a set of
philosophies
2. It uses procedures, methods & techniques that have
been tested for validity & reliability
3. It is designed to be unbiased & objective
3
Philosophiesmeans approaches e.g. qualitative,
quantitative and the academic discipline in which
you have been trained.
Validitymeans correct procedures have been applied
to get answers
Reliabilityrefers to the quality of a measurement
procedure that provides repeatability and accuracy.
Unbiased and objectivemeans that each step is
undertaken without any deliberate attempt to
conceal or highlight something and each conclusion is
drawn to best of your ability without introducing any
vested interest.
4
OBJECTI VES OF RESEARCH
n To gain familiarity with a phenomenon or to
achieve new insight into it.
n To portray accurately the characteristics of a
particular individual, situation, group,
material or a machine.
n To determine the frequency with which
something occurs.
n To test a hypothesis of a causal relationship
between variables.
5
MOTI VATI ON I N RESEARCH
n To get a research degree and its
consequential benefits.
n Desire to face the challenge to solve an
unsolved problem.
n Desire to get intellectual joy of doing some
creative work.
n Desire to be of service to society.
n Desire to get respectability.
6
Characteristics of good research
To qualify as research, the process must be
Controlled - when relationship between 2
variables are studied, minimize the effect of
other factors.
Rigorous - ensuring that procedures
followed to find answers are relevant,
appropriate and justified.
Systematic - the investigation follow some
logical sequence
Generated by Foxit PDF Creator Foxit Software
http://www.foxitsoftware.com For evaluation only.
2
7
Contd..
Valid and verifiable - whatever you conclude
on the basis of our findings is correct and
can be verified by you and others.
Empirical - any conclusion drawn are based
upon hard evidence gathered from
information collected from real life
experiences or observations.
Critical - the process and procedure of
investigation must be foolproof and free
from drawbacks and should withstand
critical scrutiny.
8
Types of Research
1.Exploratory research (foundation for hypothesis)
Literature survey research, Experience survey
research, Study of problems to have an insight.
2.Conclusive research (tests the hypothesis)
Descriptive research, Experimental research.
3. Modelling research
Symbolic model, Mathematical model,
Simulation model
4. Algorithmic research
9
Types of research
Research can also be classified based on the
application, objectives and methods used.
1. Application of research study - pure
research, applied research.
2. Objectives in undertaking the research
descriptive, correlational, explanatory,
exploratory
3. I nquiry mode employed quantitative
research, qualitative research.
10
Research process
1. Formulating the research problem
2. Extensive literature review
3. Developing the objectives
4. Preparing the research design
5. Data collection/ Performing the experiment
6. Data analysis
7. I nterpretation of results
8. Validation of results
9. Preparation of report
11
Step 1. Formulating the research
problem
l Main function is to decide what you want to
find out about.
l The way you formulate a problem
determines almost all steps that follows.
12
Contd.
l Sources of research problem
People - a set of individuals
Problems - existence of some issues or problems
related to their lives or attitudes toward certain issues.
Programs - to evaluate the effectiveness of an
intervention.
Phenomena - to establish the existence of regularity.
Performance - to evaluate the performance of a
machine or an assembly of elements or a device either
existing or newly designed
Properties - to determine various properties of
materials.
3
13
Each research study has two aspects
1. Study population/ material (We collect
information from them)- individuals,
organizations, groups, communities,
materials, equipments etc.
2. Subject area of study (I nformation we
need to collect to find answers to our
research problem)- I ssues, associations,
consumers, thermal conductivity, flow
patterns etc.
14
Considerations in selecting a
research problem
1. I nterest to sustain the required motivation.
2. Magnitude narrow the topic down to
something manageable, specific and clear.
3. Measurement of concepts have clear idea
about the measurements to be taken.
4. Level of expertise make sure that you select
topic where you have sufficient level of
expertise.
15
Contd
5. Relevance ensure that the study adds to the
existing body of knowledge, bridges current
gaps and is useful in policy formulation/
decision making.
6. Availability of data before finalizing the
topic, make sure that data are available/ can
be made available.
7. Ethical issues How ethical issues can affect
the study population and how ethical
problems can be overcome should be
thoroughly examined.
16
Steps in formulation of research problem
(Pre-requisite - Reasonably good knowledge in the
broad subject area within which study is
undertaken.)
1. I dentify a broad subject area of interest.
2. Dissect broad area into sub areas.
3. Select what is of most interest to you.
4. Raise research questions.
5. Formulate objectives.
6. Assess your objectives.
7. Double check.
17
Step 2. Reviewing the literature
Absolutely indispensable
Can be time consuming, daunting,
intimidating and frustratingbut is also
rewarding
Functions of literature review :-
1. Bring clarity and focus to your research
problem.
2. I mprove your methodology.
3. Broaden your knowledge base in your
research area.
4. Contextualize your findings.
18
Procedure for reviewing the literature
1. Search for existing literature in your area of study
in books and journals.
2. Review the literature selected and make a theme
wise listing. Note theories and methodologies
and look for criticisms. Look for gaps and
suggestions for future research. Make a write up
of literature reviewed.
3. Develop a theoretical framework (until you go
through the literature you cannot develop a
theoretical framework and until you have
developed a theoretical framework, you cannot
effectively review the literature)
4. Develop a conceptual framework (when writing
about your findings).
4
19
Step 3. Developing the objectives
Word the objectives (goals) of your study clearly and
specifically so that a reader can understand what
you want to attain in your study.
v The main objectives or aims - I ts an overall
statement of thrust of your study. It gives the main
associations and relationships you seek to discover
or establish.
v The sub-objectives are the specific aspects of the
topic that you want to investigate within the main
frame work of your study.
20
Contd.
v Objectives should start with words like-
To determine, To find out, To measure, To ascertain,
To compare, To demonstrate etc.
v The wording of objectives determines the type of
research (descriptive, correlational, and experimental)
and the type of research design you need to adopt to
achieve them.
v The objective should contain clear, complete and
specific main variable and its direction of relationship
to another.
v Example- The effect of cutting speed on surface
roughness in machining XYZ composite using ABC
cutting tool.
21
I dentifying variables
Operationalise concepts into measurable
variables. Knowledge about variables help to
fine tune the research problems.
Judgments
Switzerland is the best country to live
LNG Petronet is an excellent corporate
model
I IM A is the best business school in I ndia
22
Variable and concept
n An image, perception or concept that can be
measured -hence capable of taking on
different values- is called a variable.
n Concepts are mental images or perceptions
and hence their meanings varies markedly
from individual to individual.
n Concept cannot be measured while variable
can be measured by some method or other.
n So it is important for concept to be
converted into variable.
23
Concepts, indicators and variables
n If you use a concept in a research, identify its
indicators and then convert them into variables.
>600000 pa
>5000000
Income
Value of house,
car, total
investments.
Income
Assets
Rich
Working
Definition
Variable Indicator Concept
24
Step 4. Preparing the research
design
n Research design is the conceptual structure
within which research would be conducted.
n The function of research design is to
provide complete guidelines for the
collection of relevant information with
minimal expenditure of time, effort and
money.
5
25
Contd..
n The following points are to be considered
while preparing a research design for a
particular research problem-
1. Objectives of the research study
2.Method of data collection to be adopted
3.Source of information (Sample design)
4.Tool for data collection
5.Data analysis (Quantitative and
qualitative)
26
1. Objectives of the research study
Objectives identified to answer the
research question have to be listed
making sure that they are numbered and
statement starts with an action verb.
2. Method of data collection to be
adopted- There are 2 types of data
Primary data- collected for the first time.
Secondary data- those which already
have been collected and analyzed by
someone else.
27
Methods of Primary data collection
1. Observation method
Used in behavioural science research
This can yield info that people are usually unwilling to provide
n Types of observations-
structured - for descriptive research
unstructured for exploratory research
participant observation
non participant observation
disguised observation
Limitations of observation method
beliefs, feelings and attitudes motivating the behaviour
cannot be observed.
expensive method
28
2. Survey method
Types
Structured survey, Unstructured survey
Direct survey approach, I ndirect survey
approach
Advantages
Can collect many different kind of information.
Quick and cheap compared to observational
and exptl.
29
Contd.
Limitations
n Reluctance to answer, buzypeople may not
spend time, may give pleasant answers, unable
to answer, may give wrong answers to prove
smart and well informed.
Contact methods
n Mail, telephone, personal interview
30
Mailing questionnaire
Advantages
n Can collect large amount of information at
low cost per respondent.
n May get more honest answers to personal
questions.
n I nterviewer wont bias the respondent.
n Can answer when they have free time
n Good way to reach people who often travel.
6
31
Contd
Limitations
Not flexible
Take more time than telephonic or personal
interview
Low response rate
Researcher has no control over the person
who answers.
32
Telephonic interviewing
Advantages
n Quick method
n I nterviewer can explain difficult questions
n Can make changes with answers of
respondent
n Better sample control
n Higher response rate compared with
mailing.
33
Contd
Drawbacks
Higher cost per respondent
Some dont like to discuss personal
questions
I nterviewers manner of speaking will affect
the answers
I nterpretation depends on interviewer
Under time pressure, data may be entered
without actual interview
34
Personal I nterviewing
n Flexible, can get large amount of information, can
attract attention, clarify doubts, can guide
interview, explore issues and probe as the situation
requires.
n Suitable for any questionnaire and can be done
quickly.
n Can show actual products, packages or advt and
observe and collect their reactions and behaviour.
35
Contd
n Can be of 2 types-
n I ntercept interviewing
n Tourism research, Quick in reaching known people, Can
reach unknown people anywhere, Flexible time duration,
Judgment sampling.
n Error possibility if interviewer fails to make correct
judgment, may find some difficulty with certain groups.
Focus group interviewing
Group of 6or 7 people called for discussion with the help of
a moderator.
Sampling bias, costly, findings cannot be generalized.
Can be used as pilot interview to help for a later large scale
structured direct interview.
36
3. Experimental method
n (Also called Empirical research or Cause and Effect
method)
n Databased research, comes up with conclusions that
can be verified with observation or experiment.
n Suitable when proof is sought for relationship b/ n
variables.
n Researcher makes aworking hypothesis and then work
to get enough facts to prove or disprove the hypothesis
n Experimental design is manipulated so as to bring forth
the desired information.
n Evidence fromexperimental or empirical studies gives
the most powerful support for agiven hypothesis.
7
37
Secondary data collection
n Less cost, time, and effort
n No control over data
n May sometimes be outdated
n I nternal sources
n Sales records, marketing activity, cost
information, distributor reports and feedback,
customer feedback.
External sources
Govt. publications, journals, books, magazines,
newspapers, annual reports, research reports,
census data, publications of UNESCO, WHO,
I LO, UNO etc.
38
Determining sample design
n A sample is segment of the population
selected to represent the population as a
whole.
n Designing the sample calls for 3 decisions
n The sample (who will be surveyed)
n Sample size (how many people will be surveyed)
n Sampling (how should the sample be chosen)
n At random(probability sample)
n Select people who are easier (non-probability
sample)
n The needs of the research project will determine
which method is most effective..
39
Probability sampling methods
1. Simple random sampling- every member
of the population has a known and equal
chance of being selected.
2. Systematic sampling- spl type of random
sampling where the first sample is selected by
randomization and the remaining units
selected at fixed interval.
3. Stratified sampling- population is divided
into mutually exclusive groups such as age
groups and random samples are drawn from
each group. 40
Contd
4. Cluster sampling- population is divided
into mutually exclusive groups such as blocks
and take a block as the sample.
5. Multi-stage sampling- When the
population is too big and sample size is large,
more than one stage is used to sample the
population depending upon the reality. A
combination of sampling methods will be used
in each stage to match the reality.
41
Non-probability sampling methods
1. Convenience sampling- the researcher
selects the easiest population members from
which to obtain information.
2. Judgement sampling- researcher uses
his/ her judgement to select sample members
who are good prospects for accurate
information.
42
Contd.
3. Quota sampling- Population is classified
into a number of groups based on some
criterion. Members are selected from each
group proportional to the strength of the
group.
4. Snowball sampling- Restricted multi stage
sampling. Initially some respondents are taken
randomly and additional sampling units are
selected based on referral process.
8
43
Tool for data collection
(Research instruments)
n Guidelines to construct a research tool
n Clearly define and individually list all the
specific objectives or research questions for your
study.
n For each objective, list all the associated
questions you want to answer through your
research.
n Take each research question listed above and
list the information required to answer it.
n Formulate questions to obtain this information.
44
Example
n To determine the factors contributing to the high
divorce rate among I ndian software professionals
n Do personal variables, social variables, organizational
variables etc influence divorce rate ?
n Personal variables-
n I s there an ego clash between them due to salary ?
n I s there some problem between them due to moral
westernization of (mainly) females ?
n I s there dispute among them about taking care of
their children ?
n Do ambitions make one ignore his/ her partner in
any manner?
45
Contd.
One typical question that can be formed:-
Do you believe that the husbands salary
must always be more than the wifes ?
46
Questionnaire
n Must be developed and tested before using
in a large scale
n There are 3 types of questionnaires
n Closed ended
n Open ended
n Combination of both
47
Contd
n Closed ended- answers given, multiple
choice, can choose from given answers,
scaled questions, used to generate
statistics in quantitative research,
response can be easily entered into a
computer, large number of response can
be taken.
n Examples.
48
Contd.
n Open ended- used to know what people
have to say about something, answer in
their own words, interviewer writes
answers, blank space in questionnaire to
write answer, no standard answers and
hence data analysis is complex, opinions
are collected rather than numbers and so
lesser number may be preferred
n Example.
9
49
Contd..
n Combination of both types - from the
same instrument we can know how many
people use a service and what they think
of the service, start with closed ended
items and ends with open ended ones.
50
How to construct questionnaire
n Decide type- open or closed ended, self or
interviewer administered.
n Wording and structure
n Short and simple, avoid double barreled qns
n Avoid negative questions
n Should not contain prestige bias
n Try to cover all possible answers using closed ended
questionnaire and also include dont know answer
n Use indirect questions for sensitive issues
n Avoid leading questions which leads the respondent to
an answer.
51
Contd.
n Length and ordering of the questions
n Short Q
n Easy to answer
n I f combined keep open ended Qs at the end
n Make qnsinteresting by changing length and
type
n Group them topic wise for clarity
n Make layout and spacing appealing.
52
Piloting the questionnaire
n You check if you get the result you require
n Ask the respondent to read and find ambiguities
which you have not noticed
n Ask for comments about length, structure and
wording
n Change questions accordingly.
n If the piloting sample is good, check validity and
reliability.
53
Step 5. Collecting data
Ethical issues concerning research participants
n Collecting information
n Your request for information will createanxietyin
respondentis it ethical ?
n If theresearch is not relevant..is it ethical ?
n Gettingconsent of respondent is necessary ..otherwise
unethical
n Seeking consent - voluntarily without pressure
n Providing incentives
n Seeking sensitive information
n Possibility of causing harmto participant
n Maintaining confidentiality
54
Ethical issues relating to the researcher
n Avoid bias
n Provision or deprivation of a treatment
n Using inappropriate research methodology
n Incorrect reporting
n Inappropriate use of the information
10
55
Step 6. Data processing and analysis
Data processing(editing, classification, tabulation)
n Editing a process of examining the collected
raw data for errors and omissions and to correct
them when possible
n Classification- process of arranging data in
groups or classes based on some common
characteristics
n According to attributes that are descriptive or
numerical
n Simpleclassification
n Manifoldclassification
56
Contd
Simple classification one attribute and divide the
whole universe into two classes - example
Teachers with Ph.D
103 92 11 Teachers with Ph.D.
Total No Yes
57
Contd..
Manifold classification two or more attributes and
divide data into a number of classes - example
Qualification of bank officers
45 40 43 35 2 5 MBA
45 40 35 28 10 12 Engg
degree
45 40 44 38 1 2 Engg+
MBA
Female Male Female Male Female Male
Total No Yes
58
Contd..
n Classification according to class intervals- done for data
related to income, weight, height, age etc
Weight of students in class A
100 60 Total
25 15 61-70 kg
31 19 51-60 kg
44 26 41-50 kg
Percentage Frequency Range of weight
59
Contd
n Tabulation-Process of summarizing the raw data
and displaying it in compact form (in columns
and rows) for further analysis
n It conserves space and eliminates descriptive and
explanatory statements
n Facilitates process of comparison
n Helps summation of items and detection of errors
and omissions
n Provides basis for various statistical computations
60
Data analysis methods
n Qualitative data analysis- Analysis of content of
interview to identify the main themes that
emerge from responses given by respondents.
This is done in 4 steps
n Identify main themes.
n Assign codes to the main themes.
n Classify responses into the main themes.
n Integrate themes and responses into the text of your
report.
11
61
Quantitative data analysis
n Used in well designed and well administered surveys
using properly constructed and worded questionnaire.
n Datacan be analyzed manually or with acomputer
(SPSS for Windows)
n Before analysis, it becomes necessary to makecertain
transformations of data
n Identifyingand codingmissingvalues
n Computingtotals and new variables
n Reversingscaleitems
n Recodingand categorization
62
Missing value imputation techniques
n Hot deck imputation (Values taken from
matching respondents)
n Predicted mean imputation (Values predicted
using certain statistical procedures)
n Last value carried forward (Based on previously
observed values)
n Group means (Values determined by calculating
variables group mean)
63
Computing totals and new variables
Example-
Total income I = I1+I2+I3+I4
Expenditure vide Chapter VI= E
Taxable income = TI= I-E
Tax=T= (TI-200000)*0.1 if TI<500001
Tax=T=(30000+(TI-500000)*0.2) if TI<1000001
Tax=T=(130000+(TI-10000000)*0.3) if TI>1000000
64
Reversing scale items
Responses of negative questions are to be reverse
scored before analysis- see example
*
I get veryangry when
shesnorein thenight
*
Shemakes tasty food
Strongly
disagree
Disagree Neither
agreeor
disagree
Agree Strongly
agree
65
Recoding variables
n Turning a continuous variable into categorical
variable or collapsing some observations into
ranges
Example age, income etc
66
Data analysis
n Descriptive statistics- It allow the researcher to
describe the data and examine relationship
between the variables. Used to summarize a
study sample prior to analyzing a studys primary
hypotheses (frequency tables, histograms,
measures of central tendency, dispersion,
correlation, skewness)
n Inferential statistics- It allow the researcher to
examine causal relationships (t-test, ANOVA,
chi-square and regression)
12
67
Contd.
Generally, analysis of data involves one or more of
the following tasks
1. Computation of descriptive statistics
2. Regression analysis
3. Correlation analysis
4. Testing hypotheses
5. Factor analysis
6. Discriminant analysis
7. Conjoint analysis
68
Step 7. I nterpretation of results
n If a hypothesis is tested and upheld several
times, it may be possible for the researcher to
come to generalization, i.e., to build a theory.
n The real value of research lies on its ability to
arrive at some generalizations.
n If there is no hypothesis to start with, the
researcher may try to explain his findings based
on some existing theory. This is called
interpretation. This may trigger off new
questions which may lead to further researches.
69
Critical issues to be considered while
interpreting data and drawing inferences
from findings
I n Survey based research
1. Are you fully powered? Statistical power is a measure
of the probability that a statistical test will reject a
false null hypothesis or the probability of finding a
significant result when there is really one (0.8 is
ideal). Statistical power is largely determined by three
factors-
n significance criteria (e.g., 0.05, 0.01)
n effect size (magnitude of the difference b/ w group means)
n sample size
70
Contd
Example- test the null hypothesis that NI T
teachers are as intelligent as II T teachers. I f the
data hold the null hypothesis, Type I I error has
occurred due to some reason.
We say, the study lacked sufficient statistical
power to detect the actual difference b/ w NIT and
I IT teachers.
Usually necessary sample size is found out to obtain
sufficient power (0.8) based on the set
significance criterion and the anticipated effect
size.
71
Contd
2. Are your distributions in good shape?
Failure to consider the characteristics of the
distribution can lead to faulty interpretation of
statistical findings. Virtually all statistical tests have
certain basic assumptions. Parametric tests (e.g., T-
tests, ANOVA, linear regression) require to data to
meet certain requirements (i.e., normality and
independence)
Example- testing the effect of two teaching methods
on final grades
Data outliers can skew the shape of distribution.
72
Contd
3. Measurement error- Psychometrically sound
instrument - valid and reliable
4. Multiple comparisons. The probability of
committing a Type I error for a set of statistical
tests in the same expt is high. I f we make enough
comparisons, one or some of the results will
undoubtedly will be significant (Fishing)
5. Clinical significance- real value of research finding
lies in its clinical or real world significance and not
in the statistical significance. Example- a study
may reveal that two different studying methods
lead to significantly different test scores, but that
neither method results in passing scores.
13
73
Contd
6. Are there alternate explanations?- Randomization is
the best way to obtain scientific control over the
study and to rule out alternate explanations. Make
sure that there are no systematic differences between
the exptl groups or conditions, and that the only
thing that varies is the independent variable that you
are manipulating. However, there always is a chance
that there are between-group differences on
variables other than the one you are manipulating.
So a good researcher should look at his findings with
a good degree of suspicion and always consider
alternate explanations for those findings.
74
Contd..
7. Are you confusing correlation with causation
n Can we say cold temperature cause colds?
n Can we say rock music leads to drug abuse?
n Can we say eating ice-cream cause drowning?
8. How significant is your non-significance?
Non-significant findings can be as important,
if not more important, than significant ones.
Common beliefs or expected outcomes can
change with population and cant be
generalized.
75
I n modellingresearch
The results are inferred from the model on
which the research is done and traced back
to the real world problem with assumptions
and necessary transformations.
76
I n algorithmic research
n This type of research give optimal or near
optimal solution as per the assumed
transformation of the real life problem in to
a form suitable for the algorithm. This
solution should be inferred in such a way
that it is possible to implement it to bring
about the desired results for the real life
situation.
77
Step 8. Validation of results
n I n modellingand algorithmic research, the results
after interpretation must be validated by using past
data.
n Validation gives credibility of the results
n I f there is mismatch, assumptions and modelling
exercise must be revisited the results are validated
n I n testing hypotheses, validity is built-in in the
significance level. Inferences from research can act as
guidelines for forming policies or bringing changes in
the system for betterment.
78
Contd.
Example of validation
Assessing Quality of Engineering colleges
Teachers experience, qualifications, spl
attention to weak students, infrastructural
facilities like class rooms, labs, library,
seminars and conferences conducted
What all the above should produce ??????
GOOD PASS PERCENTAGE OF
STUDENTS
14
79
Step 9. Preparation of report
n Preliminary pages
n Main text
n End matter
80
Validity in research
n Refers to the conceptual and scientific
soundness of a research study. I ts primary
purpose is to increase the accuracy and
usefulness of findings by eliminating or
controlling as many confounding variables as
possible, which allows for greater confidence in
the findings of a given study.
n There are 4 distinct types of validity
n I nternal validity
n External validity
n Construct validity
n Statistical conclusion validity
81
I NTERNAL VALI DI TY
n Refers to the ability of a research design to
rule out or make implausible (incredible)
alternative explanations of the results, or
plausible rival hypotheses, thus
demonstrating that the independent variable
was directly responsible for the effect of the
dependent variable and, ultimately, for the
results found in the study.
82
Example
n To find the effectiveness of two different parental
skills and training and education programs on
improving symptoms of depression in
adolescents.
n Take families with an adolescent having depression
n Allot them (parents) to 2 teachers with 2 skills
training programs for 3 months
n Sent the parents home to apply the skills they have
learned
n Re-evaluate the adolescents 6 months later to see
whether there has been improvement in the
adolescents symptoms of depression.
83
Contd
n Findings..BOTH GROUPS I MPROVED
Can we conclude that both parental skills training
interventions were effective for treating
depression in adolescents?
n No.This study has poor internal validity
n There are a number of alternative rival hypotheses
that have not been controlled for and could just
easily explain the results of the study.
n So many things might have changed during this 6
months (were some of them given medicines? did
their life circumstances change?)
84
Threats to internal validity
n History
n Maturation
n Instrumentation
n Testing
n Statistical regression
n Selection biases
n Attrition
n Diffusion or imitation of treatment
n Special treatment or reactions of controls
15
85
1. History
n Refers to events or incidents that take place during the
course of the study that might have an unintended and
uncontrolled impact on the studys final outcome
n These events tend to be global affecting all
participants in a study
n They can occur inside or outside study and typically
occur between pre and post measurement phases of
the dependent variable
n Example- Studying relationship between parental care
and end semester marks
n Event- World cup cricket during study leave..
played in India..
86
2. Maturation
n Due to intrinsic changes within the participants due
to passage of time.
n Can be both biological and psychological changes
like aging, learning, fatigue, and hunger.
n Occurs between pre and post measurement phases
of the study and interferes with interpretations of
causation regarding the independent and dependent
variables.
n Example- 6 months of interaction with parents,
regardless of skill training, may improve the
depression level of children.
87
3. I nstrumentation
n This threat is unrelated to participant characteristics
and refers to changesin theassessment of the
independent variable, which are usually related to
changes in the measuring instrument or measurement
procedures over time.
n Some of the measurement and assessment tools
available to researchers are more susceptible to
instrumentation effects than others.
n The susceptibility of a measure to instrumentation
bias is usually a function of standardization.
n Usage of standardized and psychometrically sound
instruments to measure the variables eliminate
instrumentation effects. 88
4. Testing
n Refers to the effects that taking test on one
occasion may have on subsequent administrations
of the same test.
n When participants are measured several times on
the same variable, their performance might be
affected by factors such as practice, memory,
sensitization, and participant and researcher
expectancies.
n This threat in internal validity is most often
encountered in longitudinal research where
participants are repeatedly measured on the same
variable over time.
89
Example
I mpact of guided imagery techniques on the
retention of a series of random symbols
Recall of random symbols - dependent variable
Series of guided imagery techniques - independent
variable
Make initial baseline measure of retention capacity by
exposing participants to random symbols and asking
them to reproduce as many as possible from memory
after 15 minutes.
Expose them to a series of interventions (guided imagery
techniques) before exposing to random symbols and
measure recall after each intervention.
Are imagery techniques the cause of improved retention
of random symbols? No.. May be due to practice
effect.
90
5. Statistical regression
n This refers to a statistical phenomenon whereby
extremely high or low scores on a measure tend
to revert toward the arithmetic mean or average
of the distribution with repeated testing.
n This phenomenon is particularly prevalent in
research in which a pre- and post test design is
used to assess the variable of interest or when
participants are assigned to experimental groups
based on extreme scores.
16
91
Example
Effect of new treatment on anxiety patients
n People with low, medium and high anxiety levels
(G1,G2,G3) as determined by (after pre-testing) a
score on a standardized measure of anxiety (12, 29,
42) and group mean of 32.
n Subjected to treatment hoping that the new
treatment will reduce symptoms of anxiety across
each of the three conditions.
n After 10 weeks, anxiety scores of low group
increased, medium group almost remained the
same, high group reduced.
92
Contd
n Is the treatment effective only for cases G3, has no
effect on G2 and damaging effect on G1?
n This finding may be accuratebut it is also
possible that they are the result of statistical
regression.
n The scores in G3 might have reverted to the overall
group mean over the 10 weeks giving the
impression that symptom reduction resulted from
the intervention
n The scores of G1 might have moved towards the
overall group mean over 10 weeks giving the
impression that symptom increase resulted from the
intervention
93
Contd..
n The mean scores of both these groups might
have extreme scores, or outliers, which were
then influenced by regression to the mean.
n It is possible that we would have seen the same
results even without the impact of the
independent variable.
n G2 did not change and its mean was closest to
the overall sample mean which makes it least
susceptible to the effects of statistical regression.
94
6. Selection biases
n This threat of internal validity refers to systematic
differences in the assignment of participants to
experimental condition.
n Randomization is the solution for this issue.
n This is prevalent in quasi-exptl research in which
participants are assigned to exptl conditions or
comparison groups in a nonrandom fashion (Here
groups are already formed and cannot be altered. In
other words, randomization is not feasible or
possible).
95
Example
Take a design to test the effectiveness of a classroom
intervention to improve Mathematics skills in two
classes of Std.III
n Since the students are already assigned to classes,
randomization is not possible, and the study is
therefore quasi-exptl in nature.
n Give pretest in both classes.
n Give intervention in C1 and keep C2 as control
group.
n Conduct posttest in both classes.
96
Contd
n If C1 perform better, is it safe to conclude that the
intervention, or independent variable, is responsible
for the improvement ?
n May be.but there are a number of plausible rival
hypotheses that have not been controlled for
n There might have been preexisting differences
between the two groups before the intervention was
delivered.
n Any preexisting differences (motivation, maturity)
can provide a valid explanation for the results.
17
97
7. Attrition
n This threat to internal validity refers to the
differential and systematic loss of participants
from exptl and control groups.
n Participants drop out of study in a systematic
and non-random way that can affect the original
composition of groups formed.
n Seen in longitudinal study and is a direct
function of time.
98
Example
Study of effectiveness of a new drug on symptoms of
anxiety.
n Randomization is used to assign participants to either a
medication (exptl) group or control group.
n During the course of study, participants in exptl group
experiencesevere sideeffects and increase in anxiety
and drop out.
n Participants in control group are unaffected and hence
continue.
n At the end of study, if the average anxiety level of exptl
group is less than that of control group, does it mean
that the treatment is effective?
n Does the research has good internal validity?
99
8. Diffusion or I mitation of Treatment
n Common in medical and psychotherapy treatment
effectiveness research.
n It manifests itself in 2 distinct but related sets of
circumstances.
n In the first case - unintended exposure of the
control group to the actual or similar intervention
(independent variable) intended only for the exptl
group.
n In the second case - exptl group does not receive
the intended intervention at all.
100
Example case 1
Study examining the relative benefits of exercise and
nutritional counselingon weight loss.
n G1- Exercise group, G2- Nutritional counseling group,
G3- Control group.
n During the course of study, a well intentioned, but
misguided, nutritional expert talks about the benefits of
exercise to members of G2.
n Participants in G2 start doing exercise along with
nutritional control without the knowledge of
researchers.
n Exercise condition has diffused into nutritional group
which might equalize performance across groups.
101
Example - case 2
Study to determine which treatment is more effective for
diabetics, yoga or naturopathy?
Two groups G1and G2
Yoga expert give yoga treatment to G1
Naturopathy expert either intentionally or
unintentionally strayed from the specific protocol and
gave yoga treatment itself (or a hybrid of both yoga
and naturopathy) to G2 without the knowledge of the
researcher.
This might equalize the performance of the two
groups which could have the effect of distorting or
clouding the results of the study.
102
9. Special treatment or reactions of
controls
n This threat to internal validity may be caused by the
special, often compensatory, treatment or attention
given to the control group.
n Participants in control group may feel that they are in
lesser condition and react by competing or
improving their performance.
n Their periodical monitoring using structured interview
itself may act as an intervention.
n These can equalize the performance of the exptl and
control conditions and thereby washout the
between-group differences on the dependent variable.
18
103
Conclusion
n Threat to internal validity of a research study are
common and, at times, unavoidable.
n They occur alone or in combinations, and they can
create unwanted plausible alternate hypotheses for the
results of the study.
n These rival hypotheses may make it difficult to
determine causation.
n Some of these threats can be handled effectively
through design components.
n Accounting for these threats is a critical aspect and
function of research methodology that should take
place, if possible, at the design stage of the study.
104
EXTERNAL VALI DI TY
n Refers to the degree to which research results
generalize to other conditions, participants, times,
and places.
n A study has more external validity when the results
generalize beyond the study sample to other
populations, settings, and circumstances.
105
Example
Study of effectiveness of a new intervention for
exam anxiety.
n Study conducted at SOE I Year B.Techstudents.
n Students are assessed to determine their levels of
exam anxiety and then assigned to control group or
exptl group that receives intervention.
n If the therapy is effective and remarkably reduce
exam anxiety, can it be marketed as an effective
treatment for exam anxiety?
n Nothe study has poor external validity..
106
Threats to external validity
n Sample characteristics
n Stimulus characteristics and settings
n Reactivity of experimental arrangements
n Multiple treatment interference
n Novelty effects
n Reactivity of assessment
n Test sensitization
n Timing of measurement
107
1. Sample characteristics
n The phenomenon whereby the results of a study apply
only to a particular sample.
n The key question is whether the studys results can be
applied to other samples that vary in a variety of
demographic and descriptive characteristics, such as
age, gender, sexual orientation, education, and socio-
economic status.
n Example-study of intervention for exam anxiety in I
year B.Techstudents in SOE.
108
2. Stimulus characteristics and settings
n This threat to external validity refers to an
environmental phenomenon in which particular
features or conditions of the study limits the
generalizability of the findings so that the
findings from one study do not necessarily apply
to another study, even if the other study is using
a similar sample.
19
109
Example
n If similar study (Exam anxiety) conducted at
another college can give a different result ie, no
effect for the intervention
n Settings are different- (difference in comfort
level of participants)
n Stimulus characteristics are different- a different
person has handled the intervention who may be
less competent or less approachable.
110
3. Reactivity of the experimental
arrangements
n This refers to a potentially confounding variable that
is a result of the influence produced by knowing that
one is participating in a research study.
n This awareness can have an impact on their attitudes
and behavior during the course of the study.
n When the participants know the purpose of research
and hypotheses, there is high probability that their
responses will make the study problematic.
111
Example
Study of effectiveness of counseling against drug
usage amongst prisoners in Keralajail
n Get a base line record of drug usage of each
prisoner
n Apply intervention (counseling) with an expert
n Measure the dependent variable again
n Suppose it has improved
n What does it mean?
112
Contd.
n Sometimes, the participants might have taken
compliant attitude towards the intervention to
avoid troublewhich obviously might have
resulted in improving the dependent variable.
n So, we can say, the outcome is the result of
reactivity to exptl arrangements and not because
of the intervention itself.
113
4. Multiple treatment interference
n This threat to external validity refers to
research situations in which
1. Participants are administered more than one
exptl intervention (independent variable) within
the same study.
2. Same individuals participate in more than one
study.
n Common in treatment outcome studies.
114
Contd
n In the first situation, independent variables
administered simultaneously or sequentially may
produce an interaction effect.
n It makes a confounding effect that makes it difficult to
determine which one is responsible for the observed
results.
n The second situation refers to the relative experience
and sophistication of the participants.
n Familiarity with research can affect the behavior and
responses of participants, which makes it difficult to
interpret the results.
20
115
Example
Effectiveness of psychodynamic and cognitive
treatment for depression
n Control group and exptl group, initial assessment
made for symptoms of depression.
n 24 weeks treatment, first 12 weeks-psychodynamic
therapy followed by 12 weeks of cognitive therapy.
n At the end of 24 weeks, both groups are reassessed.
n If we find significant symptom reduction in exptl
group and no change in control group, what do we
say?
116
Contd.
n Can we say that a combined psychodynamic
cognitive therapy model is an effective approach
for treating depression?
n May be true, but there are so many unanswered
questions.
n Why not change the order?
n Why 2 treatments? One may be enough.
n Would they have got better improvement without
any of the treatments ?
n Multiple treatment effects can be controlled
through specific research designs.
117
5. Novelty effects
n This refers to the possibility that the effects of
the independent variable may be due in part to
the uniqueness or novelty of the stimulus or
situation and not to the intervention itself.
n This effect disappear once the novelty wears off.
n Novelty of the intervention acts as a
confounding variable, and it is that novelty (and
not the independent variable) that is the real
explanation for the results.
118
Example
n Two organizations, org1 and org2
n An intervention is designed to increase teamwork
and related productivity for top level managers.
n Administered to top managers of both org.
n Researchers find that intervention increases
productivity and teamwork in org1 only.
n The reason is that org2 was familiar to this since
they were adopting TQM principles for a while.
n Since these policies were quite new for org1, its
novelty has improved productivity and not the
intervention.
119
6. Reactivity of Assessment
n A Phenomenon whereby participants awareness
that their performance is being measured can
alter their performance from what it otherwise
would have been.
n There can be under-reporting as well as over-
reporting depending upon the situation.
120
7. Pretest and posttest sensitization
n These threats refer to the effects that pre-testing
and post-testing might have on the behavior and
responses of study participants.
n Pre-testing done to quantify some variable of
interest to form a baseline measure against
which the effects of the exptl intervention can
be evaluated.
n Would the results of the study have been the
same if the pretest have not been administered?
21
121
Contd
n A post test measurement (to measure the
effectiveness of the independent variable) can
have a similar effect on external validity.
n In both cases the question is whether the
participants were sensitized by either measure.
n If so, the findings might be less generalizable.
122
8. Timing of assessment and
measurement
n This threat refers to whether the same results
would have been obtained if measurement had
occurred at a different point in time.
n Important in longitudinal treatment-outcome
study.
n Symptoms are assessed after a pre-decided
period depending upon the therapy
administered.
123
CONSTRUCT VALI DI TY
n It relates to interpreting the basis of the causal
relationship, and it refers to the congruence
between the studys results and the theoretical
underpinnings guiding the research.
n The focus of construct validity is usually on the
studys independent variable.
n In essence, the construct validity asks the question
of whether the theory supported by the findings
provides the best available explanation of the
results.
124
Contd.
n In other words, is the reason for the relationship
between the exptl intervention (independent
variable) and the observed phenomenon
(dependent variable) due to the underlying
construct or explanation offered by the
researchers.
n If yes, construct validity is good
n If no, construct validity is poor.
125
Example
n To study the factors that contribute to mortality
rates in a number of different countries.
n Hypothesis Education level and family income
will be significantly related to mortality rate.
(Mortality rate will drop as education level and
family income rise)
n The researchers have to prove that education
level and family income are negatively correlated
to mortality rate.
126
Contd
n If they find that their hypothesis is upheld, they can
conclude that educational level and family
income are protective factors that reduce the
likelihood of mortality.
n Is there something inappropriate in this research?
n Higher level of education may lead to higher
income and better access to wider variety of
privileges and services, such as access to higher
quality health care
n So, access to health care could have been a better
independent variable than the other two selected by
the researchers.
22
127
Threats to construct validity
n The number and types of threats are related to the
unique aspects and design of the study itself.
n These threats are features of a study that interfere with
the researchers ability to draw causal inferences from
the studys results.
n They are difficult to classify because they can be
anything that relates to the design of the study and the
underlying theoretical construct under consideration.
n They are closely parallel to some of the threats to
external validity such as conditions surrounding the
experimental situation, experimenter expectancies and
characteristics of the participants
128
Methods to improve construct validity
1. Provide a clear operational definition of the
abstract concept or independent variable.
2. Collect data to demonstrate that the empirical
representation of the independent variable
produces the expected outcome.
3. Collect data to show that the empirical
representation of the independent variable
does not vary with measures of related but
different conceptual variables.
4. Conduct manipulation checks of the
independent variable.
129
STATI STI CAL VALI DI TY
n It refers to aspects of quantitative evaluation that
affect the accuracy of the conclusions drawn from
the results of a study.
n Statistical procedures are typically used to test the
relationship between two or more variables and
determine whether an observed statistical effect is
due to chance or is a true reflection of a causal
relationship.
n It addresses the question of whether the statistical
conclusions drawn from the results of a study are
reasonable.
130
Threats to statistical validity
1. Low statistical power
2. Variability in the exptl procedures
and participant characteristics
3. Unreliability of measures
4. Multiple comparisons and error
rates
131
1. Low statistical power
The presence of this threat produces a low
probability of detecting a difference between
experimental and control conditions even when
a difference truly exists. Low statistical power is
directly related to small effect and sample sizes,
with the presence of each increasing the
likelihood that low statistical power is an issue in
research design. This can cause a researcher to
conclude that there are no significant results
even when significant results actually exist.
132
2. Variability
n Variability in methodological procedure
n This concept includes a wide array of differences
and questions that relate to the actual design
aspects of the study.
n These differences can be found in the delivery of
the independent variable, the procedures related
to the execution of the study, variability in
performance measure over time, and a host of
other examples that are directly dependent on
the unique design of a particular study.
23
133
Contd.
n Variability in participant characteristics
n Participants in a research study can vary along
a variety of characteristics and dimensions,
such as age, education, socio-economic status,
and race.
n As the diversity of participant characteristics
increases, there is less likelihood that a
difference between the control and exptl
condition can be detected.
134
Contd.
n When the variability across these two sources is
minimized, the likelihood of detecting a true
difference between the control and exptl
conditions increases.
n This threat must be considered at the planning
stage of the study and is usually controlled
through the use of
n Homogenous samples.
n Strict and well-defined procedural protocols.
n Statistical controls at the data analysis stage.
135
RELI ABI LI TY
n Refers to the consistency or dependability of a
measurement technique, and is concerned with the
consistency of the score obtained from a measure
or assessment technique over time and across
settings or conditions.
n If a measurement is reliable, then there is less
chance that the obtained score is due to random
factors and measurement error.
n A score obtained from data collection has truescore
andmeasurement error components
136
Contd
n Can be expressed as a correlation coefficient
between the scores on an assessment instrument
taken on two different occasions.
n Adequate reliability exists when the correlation
coefficient is 0.80 or higher.
n Construct like personalitywhich is a relatively
stable construct do not change drastically with
time and hence its measurement will give reliable
result if the instrument is a standardized one.
n Certain constructs like emotional traitsvary with
time and hence the reliability may be less.
137
Strategies to improve reliability
n The administration of the instrument or
measurement strategy should be standardized.
n The researchers should make certain that the
participants understand the instructions and content
of the instrument or measurement strategy.
n Every researcher involved in data collection should
be thoroughly trained in the use of the measurement
strategy.
n Every effort should be made to ensure that data are
recorded, compiled, and analyzed accurately.
138
Assessing reliability
1.Test-retest reliability-This refers to the stability of
test scores over time and involves repeating the same
test on at least one other occasion. Time interval is
important as the test-retest correlations tend to
decrease as the time interval increases.
2. Split-half reliability This refers to the
administration of a single test that is divided into two
equal halves. Theoretically, the items on both forms
measure the same construct. This method is much less
susceptible to time interval effects because all the
items are administered at the same time and then split
into separate items pools afterward.
24
139
Contd.
3. Alternate form reliability It is expressed as the
correlation between different forms of the same
measure where the items on each measure
represent the same item content and construct.
This approach requires two different forms of the
same instrument which are then administered at
different times. The two forms should cover
identical content and have a similar difficulty level.
The two test scores are then correlated.
140
Contd
4. I nter-rater reliability this is used to
determine the agreement between different
judges or raters when they are observing or
evaluating the performance of others.
Example- we have two evaluators assessing the the
lack of interestof a student in a subject in a class.
We operationalizelack of interest as the number of
times he yawn or doze in class. The extent to
which the evaluators agree on whether or when
the behavior occurs reflects this type of
reliability.
141
Validity of instrument
n Validity indicates the degree to which an instrument
measures what it is supposed to measure. Validity is
concerned not only with what is being measured,
but also how well it is being measured.
n A measurement cannot be valid unless it is reliable.
But a measurement can be reliable without being
valid (Think of someexample)
n There are 3 types of validity
n content - related
n criterion - related
n construct - related.
142
Content validity
n Refers to the relevance of the instrument or
measurement strategy to the construct being
measured.
n Applicable to most forms of measurement
strategies used in research.
n The approach for determining content validity
starts with the operationalizationof the construct
of interest.
n The developer defines the construct and then
attempts to develop item content that will
accurately capture it.
143
Criterion validity
n This is determined by the relationship between the
measureand performance on an outside criterion or
measure.
n The outside criterion or measure should be related to the
construct of interest, and it can be measured at the same
time the measure is given or some time in future.
n If the measure is compared to an outside criterion that is
measured at the same time, it is then referred to as
concurrent validity.
n If the outside criterion is measured in the future, it is then
referred to as predictivevalidity.
n Example- job satisfaction and attrition rate of employees.
144
Construct validity
n Construct validity assesses the extent to which the
test or measurement strategy measures a theoretical
construct or trait.
n There is a variety of approaches for determining
construct validity.
n These approaches focus on the extent to which the
measurement of a certain construct converges or
diverges with the measurement of similar or
different constructs.
25
145
Contd.
n If an instrument measure depression accurately,
we would expect that there would be a strong
relationship between the measurement in
question and other measures of depression,
expressed by a strong positive correlation
coefficient - indicating good convergent validity.
146
Contd.
n If an instrument measure depression accurately,
we would expect that there would be a strong
relationship between the measurement in
question and an opposite measure of depression
(Say happiness), expressed by a strong negative
correlation coefficient - indicating good
divergent validity.
147
THANKS FOR YOUR
PATI ENCE..
AND
GOODLUCK FOR THE
TEST PAPER

Research Methodology

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Research Methodology

Uploaded by

Copyright:

Available Formats

1

You might also like