Professional Documents
Culture Documents
in educational planning
Series editor: Kenneth N.Ross
1
Module
T. Neville Postlethwaite
Institute of Comparative Education
University of Hamburg
Educational research:
some basic concepts
and terminology
These modules were prepared by IIEP staff and consultants to be used in training
workshops presented for the National Research Coordinators who are responsible
for the educational policy research programme conducted by the Southern and
Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ).
iiep/web doc/2005.01
The designations employed and the presentation of material throughout the publication do not imply the expression of
any opinion whatsoever on the part of UNESCO concerning the legal status of any country, territory, city or area or of
its authorities, or concerning its frontiers or boundaries.
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any
form or by any means: electronic, magnetic tape, mechanical, photocopying, recording or otherwise, without permission
in writing from UNESCO (International Institute for Educational Planning).
Content
1. Introduction 1
1
Module 1 Educational research: some basic concepts and terminology
6. Conclusion 29
Appendix A
Terminology used in educational research 30
Formative and summative evaluation 31
Assessment, evaluation, and research 33
Measurement 33
Surveys and experiments 34
Tests 36
1. Test items 36
2. Sub-scores/Domain scores 37
Variable 37
1. Types of variables 38
Indicator 42
Attitude scales 43
Appendix B
Further reading suggestions 46
Introductory texts 46
Examples of educational research studies that aimed
to have an impact on educational planning 47
Encyclopedias and handbooks 48
Journals 48
Appendix C
Exercises 49
II
Introduction 1
© UNESCO 1
Module 1 Educational research: some basic concepts and terminology
2
2. Descriptive research provides information about conditions,
situations, and events that occur in the present. For example,
a survey of the physical condition of school buildings in order
to establish a descriptive profile of the facilities that exist in a
typical school.
© UNESCO 3
Module 1 Educational research: some basic concepts and terminology
4
Types of education research
© UNESCO 5
Module 1 Educational research: some basic concepts and terminology
Descriptive questions
In the field of educational planning, the research carried out on
descriptive questions is often focused on comparing the existing
conditions of schooling with: (i) legislated benchmark standards,
(ii) conditions operating in several other school systems, or (iii)
conditions operating in several sectors of a single school system.
6
• Do the supplies and equipment in classrooms in the schools
match the legislated standards set by the Ministry? (The
supplies and equipment might be textbooks, exercise books,
pencils, erasers, seats and desks. The Ministry may have norms
that each student in a particular grade must have one mother
tongue textbook, one math textbook, one science textbook and
one social studies textbook, four exercise books, three pencils
and one eraser in a year, and that each student must have
one seat and one writing place. The research required in this
situation then consists of undertaking a count of the supplies
and equipment in all schools, or in a scientific sample of schools
that can be used to estimate the situation in all schools, and
then matching every school or classroom against the Ministry’s
norms. The main aim of this kind of research study would be to
examine whether there are particular districts or regions which
are under-supplied or over-supplied.)
© UNESCO 7
Module 1 Educational research: some basic concepts and terminology
Correlational questions
Behind these kinds of questions, there is often an assumption
that if an association is found between variables then it provides
evidence of causation. However, care must be exercised when
moving between the notions of association and causation. For
example, an ‘association’ may be discovered between the incidence
of classroom libraries and average class reading scores. However,
the real ‘cause’ of higher reading scores may be that students from
high socio-economic backgrounds, while they tend to be in classes
with classroom libraries, read better than other students because
their home environments (in terms of physical, emotional, and
intellectual resources) facilitate the acquisition of reading skills.
8
Three types of research questions in educational planning
Causal questions
Causal questions are usually the most important to educational
planners. For example, in some schools it is considered normal for
children to have a desk at which to sit. In other schools the children
sit on the ground and write on their laps. It is important to know if
schools (with a particular socio-economic background of children)
with a shortage of desks and seats achieve less well than schools
(with a similar socio-economic background of children) with an
adequate supply of desks and chairs. Or, to put the question in a
different way, is it the desks and chairs, or something else, which
really cause the better achievement? It may be a better supply
of books or better qualified teachers or, or, or.... It is, therefore,
important to disentangle the relative influence of each of the many
input and process factors in schools on achievement.
Thus, causal questions take one of two forms. Some examples are:
© UNESCO 9
Module 1 Educational research: some basic concepts and terminology
10
Identifying research issues 4
for educational planning
The reason why all educational planners should be prepared to
undertake research is that it is important to be sure of the facts
before making suggestions for changes in educational policies and
practices. The maxim must always be “when in doubt, find out”.
The questions listed above are only examples and are very general
in nature. It is up to each Ministry and educational planning office
to pose its own questions in order to remove doubt. The formulation
of research questions is, however, not an easy matter.
© UNESCO 11
Module 1 Educational research: some basic concepts and terminology
12
Identifying research issues for educational planning
© UNESCO 13
Module 1 Educational research: some basic concepts and terminology
14
Identifying research issues for educational planning
© UNESCO 15
Module 1 Educational research: some basic concepts and terminology
Literature review
The review of literature aims to describe the ‘state of play’ in the
area selected for study. That is, it should describe the point reached
by the discipline of which the particular research study will form
a part. An effective literature review is not merely a summary
of research studies and their findings. Rather, it represents a
‘distillation’ of the essential issues and inter-relationships associated
with the knowledge, arguments, and themes that have been
explored in the area. Such literature reviews describe what has been
written about the area, how this material has been received by other
scholars, the major research findings across studies, and the major
debates in terms of substantive and methodological issues.
16
Research design
Given the specific research questions that have been posed, a
decision must be taken on whether to adopt an experimental design
for the study or a survey design. Further, if a survey design is to
be used, a decision must be taken on whether to use a longitudinal
design, in which data are collected on a sample at different points
of time, or a cross-sectional design, in which data are collected at a
single point of time.
© UNESCO 17
Module 1 Educational research: some basic concepts and terminology
Instrumentation
Occasionally, data that are required to undertake a research study
already exist in Ministry files, or in the data archives of research
studies already undertaken, but this is rarely the case. Where
data already exist, the analysis of them is known as “secondary
data analysis”. But, usually, primary data have to be collected.
From the specific research questions established in the first step
of a research study it is possible to determine the indicators and
variables required in the research, and also the general nature of
questionnaire and/or test items, etc. that are required to form these.
Decisions must then be taken on the medium by which data owe
to be collected (questionnaires, tests, scales, observations, and/or
interviews).
18
Sequential stages in the research process
â
Search for, and review of, other previous studies that (a) identify
controversie debates, and knowledge gaps in the field; (b) elucidate
Stage 2
theoretical foundations that need to be tested empirically; and/or (c)
Literature
provide excellent models in terms of design, management, analysis,
reporting, and policy impact
â
Stage 3 Development of overall research design including specification of the
Research design information that is to be collected from which individuals under what
"review" research conditions
â
Construction of operation definitions of key variables and selection/
Stage 4
preparation of instruments (tests, questionnaires, observation
Instrumentation
schedules, etc.) to be employed in the measurement of these variables
â
Pilot testing of instruments/data collection and recording procedures
Stage 5
and techniques. Use of results to revise instruments and to refine all
Pilot testing
data collection procedures
â
Stage 6
Data collection and data preparation prior to main data analysis
Data collection
â
Stage 7
Data summarization and tabulation
Data analysis
â
Stage 8
Writing of research report(s)
Research report
© UNESCO 19
Module 1 Educational research: some basic concepts and terminology
Pilot testing
At the pilot testing stage the instruments (tests, questionnaires,
observation schedules, etc.) are administered to a sample of the
kinds of individuals that will be required to respond in the final
data collection. For example, school principals and/or teachers and/
or students in a small number of schools in the target population.
If the target population has been specified as, for example, Grade
5 in primary school, knowledge should exist in the Ministry, or in
the inspectorate, about which schools are good, average, and poor
schools in terms of educational achievement levels or in the general
conditions of school buildings and facilities. A ‘judgement sample’
of five to eight schools can then be drawn in order to represent a
range of achievement levels and school conditions. It is in these
schools that the pilot testing should be undertaken.
20
Sequential stages in the research process
The same can be said about the procedures for entering data,
cleaning data, and merging files. This work is usually undertaken
by the planning office data processing unit, but again the results
of the pilot testing experience can help to ‘de-bug’ the procedures.
Once the instruments and procedures have been finalized, the main
data collection can begin.
© UNESCO 21
Module 1 Educational research: some basic concepts and terminology
Data collection
When a probability sample of schools for the whole of the target
population under consideration has been selected, and the
instruments have been finalized, the next task is to arrange the
logistics of the data collection. If a survey is being undertaken in
a large country, this can require the mobilization of substantial
resources and many people.
22
Sequential stages in the research process
© UNESCO 23
Module 1 Educational research: some basic concepts and terminology
After the data have been entered, cleaned, and merged – which
often requires the student, teacher, and school files for a particular
school to be joined into one record – the data analysis can begin.
Data analysis
If there are unequal probabilities of selection for members of the
sample, or if there is a small amount of (random) non-response,
then the calculation of sampling weights has to be undertaken.
For teacher and school data there are choices which can be made
about weighting. For example, if one is conducting a survey and
each student in the target population had the same probability of
entering the sample, then the school weights can either be designed
to reflect the probability of selecting a school, or school weights
can be made proportional to the weighted number of students in
the sample in the school. In this latter case, the result for a school
variable means the school value given is what the ‘average student’
experiences. This matter has been discussed in more detail in the
module on ‘Sample Design’.
24
Sequential stages in the research process
a. Descriptive
Typically, the first step in the data analyses is to produce descriptive
statistics separately for each variable. These statistics are often
called univariates. Some variables are continuous – for example
‘size of school’ which can run from, say, 50 to 2,000. In this case
the univariate statistics consist of a mean value for all schools,
the standard deviation of the values, and a frequency distribution
showing the number of schools of different sizes. Other variables
are proportions or percentages. Such a variable could be the
percentage of teachers with different types of teacher training.
© UNESCO 25
Module 1 Educational research: some basic concepts and terminology
Rural Urban
All schools
Educational provision Schools schools
Mean S.D. Mean S.D. Mean S.D.
Desks per classroom X X X X X X
Chairs per classroom X X X X X X
Floor space per student X X X X X X
Pens/Pencils per student X X X X X X
The first pair of columns presents the mean value and standard
deviation for all schools in the sample for desks per classroom,
chairs per classroom, floor space per student, etc. However, the total
sample is ‘broken down’ in the second and third pairs of columns
into rural and urban schools.
b. Correlational
In this case product moment correlations or cross tabulations can
be calculated. There are statistical tests which can be applied to
determine whether the association is more than would occur by
chance. When the association between two variables is examined,
this is known as ‘bivariate’ analysis.
c. Causal
If the research design used is an experimental one, then tests can be
applied to see if the performance of the experimental group (that is,
the group subjected to the new treatment) is better than the control
group.
26
Sequential stages in the research process
© UNESCO 27
Module 1 Educational research: some basic concepts and terminology
Research report
There are three major types of research reports. The first is the
Technical Report written in great detail and showing all of the
research details. This is typically read by other researchers. It is
this report that provides evidence that the research was conducted
soundly. This is usually the report which is written first.
The second report is for the senior policy makers in the Ministry
of Education. It is in the form of an Executive Summary of about 5
or 6 pages. It reports the major findings succinctly and explains, in
simple terms, the implications of the findings for future action and/
or policy.
28
Conclusion 6
Each system of education has its political goals, its general
educational goals, and its specific educational objectives. For
example, some political goals stress equality of opportunity, others
stress quality of education, and many stress both.
© UNESCO 29
Module 1 Educational research: some basic concepts and terminology
Appendix A
30
The operational research aims emerging from these general and
specific research questions could include the following:
© UNESCO 31
Module 1 Educational research: some basic concepts and terminology
are trained to teach the new units, and that the units are tried out
in a range of schools. Examples of the kinds of questions asked in
formative evaluation would be: Do the specific objectives which
have been developed cover the general objectives to be learned
which are in the curriculum? Can the teachers cope with the new
units? Are there any ‘gaps’ in the curriculum units which result in
a poor coverage of some of the specific objectives? Can the layout of
the curriculum units be changed so as to make the material more
interesting for students?
32
Appendix A
Measurement
Measurement is a process that assigns a numerical description to
some attribute of an object, person, or event. Just as rulers and stop-
watches can be used to measure, for example, height and speed, so
can other quantities of educational interest be measured indirectly
through the use of achievement tests, questionnaires and the like.
© UNESCO 33
Module 1 Educational research: some basic concepts and terminology
34
Appendix A
© UNESCO 35
Module 1 Educational research: some basic concepts and terminology
Tests
A test is an instrument or procedure that proposes a sequence of
tasks to which a student is to respond. The results are then used to
form measures to define the relative value of the trait to which the
test refers.
1. Test items
A test may be an achievement, intelligence, aptitude, or practical
test. A test consists of questions, known as items.
An item is divided into two parts: the stem and the answer. Stems
pose the question. For example, a stem could be:
• What is the sum of 40 and 8?
• or in Reading Comprehension it could be a reading passage
followed by specific questions.
The answer could be an ‘open-ended’, a ‘closed’, or a ‘fill-in’ answer.
For example, in the first stem given above an open or fill in answer
could require the student to write the answer in a box. Or, it could
be put into multiple choice format as follows:
• What is the sum of 40 and 8?
a. 84 b. 50 c. 48 d. 408
In this closed format the student is requested to tick the correct
answer.
36
Appendix A
2. Sub-scores/Domain scores
The score of a student on the whole test is known as the “total test
score”. A sub-score refers to the achievement of the students on a
sub-set of items in the overall test. Thus, for example, in a Science
test it may be considered desirable to classify the items into Biology
items, Chemistry items, and Physics items. Each of these constitutes
a domain of the test and the scores on each are known as ‘sub-
scores’ or ‘domain scores’.
Variable
The term variable refers to a property whereby the members of a
group being studied differ from one another. Labels or numbers
may be used to describe the way in which one member of a group is
the same or different from another.
© UNESCO 37
Module 1 Educational research: some basic concepts and terminology
1. Types of variables
Variables may be classified according to the type of information
which different classifications or measurements provide. There are
four main types of variables: nominal, ordinal, interval, and ratio.
a. Nominal
This type of variable permits statements to be made only about
equality or difference. Therefore we may say that one individual is
the ‘same as’ or’ ‘different from’ another individual. For example,
colour of hair, religion, country of birth.
b. Ordinal
This type of variable permits statements about the rank ordering
of the members of a group. Therefore we may make statements
about some characteristics of an individual being ‘greater than’ or
‘less than’ other members of a group. For example, physical beauty,
agility, happiness.
c. Interval
This type of variable permits statements about the rank ordering
of individuals. It also permits statements to be made about the
‘size of the intervals’ along the scale which is used to measure the
individuals and to compare distances at points along the scale.
It is important to note that interval variables do not have true
zero points. The numbering of the years in describing dates is an
interval scale because the distance between points on the scale is
comparable at any point, but the choice of a zero point is a socio-
cultural decision.
38
Appendix A
d. Ratio
This type of variable permits all the statements which can be made
for the other three types of variables. In addition, a ratio variable
has an absolute zero point. This means that a value for this type of
variable may be spoken of as ‘double’ or ‘one third of’ another value.
For example, physical height or weight.
1. Validity
Validity is the most important characteristic to consider when
constructing or selecting a test or measurement technique. A valid
test or measure is one which measures what it is intended to measure.
Validity must always be examined with respect to the use which
is to be made of the values obtained from the measurement
procedure. For example, the results from an arithmetic test may
have a high degree of validity for indicating skill in numerical
calculation, a low degree of validity for indicating general reasoning
ability, a moderate degree of validity for predicting success in future
mathematics courses, and no validity at all for predicting success in
art or music.
© UNESCO 39
Module 1 Educational research: some basic concepts and terminology
a. Content validity
This type of validity refers to the extent to which a test measures
a representative sample of subject-matter content and behavioural
content from the syllabus which is being measured. For example,
consider a test which has been designed to measure “Competence
in Using the English Language”. In order to examine the content
validity of the test one must initially examine the subject-matter
knowledge and the behavioural skills which were required to
complete the test, and then after this examination compare these
to the subject-matter knowledge and behavioural skills which
are agreed to comprise correct and effective use of the English
language. The test would have high content validity if there was a
close match between these two areas.
b. Criterion-related validity
This type of validity refers to the capacity of the test scores to
predict future performance or to estimate current performance
on some valued measure other than the test itself. For example,
‘Reading Readiness’ scores might be used to predict a student’s
future reading achievement, or a test of dictionary skills might be
used to estimate a student’s skill in the use of the dictionary (as
determined by observation).
40
Appendix A
c. Construct validity
This type of validity is concerned with the extent to which test
performance can be interpreted in terms of certain psychological
constructs. A construct is a psychological quality which is assumed
to exist in order to explain some aspect of behaviour. For example,
“Reasoning Ability” is a construct. When test scores are interpreted
as measures of reasoning ability, the implication is that there is
a quality associated with individuals that can be properly called
reasoning ability and that it can account to some degree for
performance on the test.
2. Reliability
Reliability refers to the degree to which a measuring procedure
gives consistent results. That is, a reliable test is a test which would
provide a consistent set of scores for a group of individuals if it was
administered independently on several occasions.
Note that reliability refers to the nature of the test scores and not to
the test itself. Any particular test may have a number of different
reliabilities, depending on the group involved and the situation in
which it is used. The assessment of reliability is measured by the
‘Reliability Coefficient’ (for groups of individuals) or the “Standard
Error of Measurement” (for individuals).
© UNESCO 41
Module 1 Educational research: some basic concepts and terminology
Indicator
An indicator generally refers to one or more pieces of numerical
information related to an entity that one wishes to measure. In
some cases, it consists of information about only one variable
and this information may be gathered by only one question on a
questionnaire. For example, consider an indicator of classroom
library availability. In this case the indicator may be assessed by a
single variable (which has only two values) that is measured by one
question on a questionnaire:
Possession No Yes
Car
Refrigerator
T.V.
Video
etc.
42
Appendix A
Attitude scales
Probably the least debated definition of an attitude is: “a moderate
to intense emotion that prepares or predisposes an individual to
respond consistently in a favourable or unfavourable manner when
confronted with a particular object” (Anderson, 1985). In education,
attitudes which are often measured are ‘Like School’, ‘Interest
in Subject-matter’, and ‘Teacher Satisfaction with Classroom
Conditions’. Each of these titles implies a high to low measure.
Thus, ‘Like School’ implies a measure that ranks students from
those who love school to those who hate school.
© UNESCO 43
Module 1 Educational research: some basic concepts and terminology
Strongly Strongly
Disagree Incertain Agree
disagree agree
Since the Likert Scale is the one most frequently used in educational
research, a short explanation is given here. For example, consider
the development of a ‘Like School’ scale to be used for 14-year-
students. The researcher must first of all listen carefully to how
14-year-old students describe their like or dislike of schools. Both
positive and negative statements are used, after editing, to form a
set of statements about ‘Like School’. An example is given below:
44
Appendix A
© UNESCO 45
Module 1 Educational research: some basic concepts and terminology
Appendix B
46
Examples of educational research studies
that aimed to have an impact on educational
planning
Asmah bt Mohd Taib, Ahmad, Siatan, Solehan bin Remot, &
Nordin, Abu Bakar. (1982). Moral education in Malaysia.
Evaluation in Education, 6 (1), 109-136.
Jiyono & Suryadi, Ace. (1982). The planning, sampling, and some
preliminary results of the Indonesian Repeat 9th Grade survey.
Evaluation in Education, 6 (1), 5-30.
© UNESCO 47
Module 1 Educational research: some basic concepts and terminology
Journals
American Educational Research Journal
Applied Measurement in Education
Assessment in Education
Comparative Education Review
Educational Assessment
Educational Evaluation and Policy Analysis
International Journal of Educational Research
International Journal of Educational Development
International Review of Education
Journal of Education Policy
Research Papers in Education: Policy and Practice
Review of Educational Research
Studies in Educational Evaluation
48
Appendix C
Exercises
© UNESCO 49
Module 1 Educational research: some basic concepts and terminology
Select one of the five general aims above that you believe would
probably receive a high priority in your country. For that general
aim write five specific research questions. For each of the five
specific research questions, prepare several operationalized
research aims that focus on the performance of the education
system in meeting these aims. Then, write down a broad outline
of the sequence of activities that would need to be undertaken in
order to assess the system’s performance with respect to these
aims.
When this has been completed, discuss and write down, in outline
form only, the sequence of activities to be undertaken in the
research study or studies.
Collate the work of the small groups. Again refine the wording.
Then write down in detail the sequential activities to be
undertaken in a research study for each general aim covered to
provide valid, reliable, and useful information for decision-makers
to assess to which extent the general aims have been addressed.
50
Since 1992 UNESCO’s International Institute for Educational Planning (IIEP) has been
working with Ministries of Education in Southern and Eastern Africa in order to undertake
integrated research and training activities that will expand opportunities for educational
planners to gain the technical skills required for monitoring and evaluating the quality
of basic education, and to generate information that can be used by decision-makers to
plan and improve the quality of education. These activities have been conducted under
the auspices of the Southern and Eastern Africa Consortium for Monitoring Educational
Quality (SACMEQ).
In 2004 SACMEQ was awarded the prestigious Jan Amos Comenius Medal in recognition
of its “outstanding achievements in the field of educational research, capacity building,
and innovation”.
These modules were prepared by IIEP staff and consultants to be used in training
workshops presented for the National Research Coordinators who are responsible for
SACMEQ’s educational policy research programme. All modules may be found on two
Internet Websites: http://www.sacmeq.org and http://www.unesco.org/iiep.