Keep Rising In Zeal, Ignite Ambition’s Heart (KRIZIAH)!

CHAPTER 1: PSYCHOLOGICAL TESTING AND ASSESSMENT evaluation evaluation

Skill of Requires an educated selection of Requires technician-like skills in
TESTING AND ASSESSMENT Evaluator tools of evaluation, skill in terms of administering and scoring
The roots of contemporary psychological testing and assessment can be found in early 20th evaluation, and thoughtful a test as well as in interpreting a
century France. organization and integration of data test result
 Alfred Binet and Theodore Simon (1905) published a test designed to help place Outcome Entails a logical-problem solving Yields a test score or series of
Paris schoolchildren in appropriate classes. They developed questions that would approach the brings to bear many test score
predict children’s future progress in the Paris school system sources of data designed to shed
light on a referral question
 Within a decade, an English-language version of Binet’s test was prepared for the
use in schools in the US
PROCESS OF ASSESSMENT
 During WWI, the military needed a way to screen large numbers of recruits quickly
1. The process of assessment begins with a referral for assessment from a source (e.g.,
for intellectual and emotional problems, and psychological testing provided this
teacher, judge, counselor). One or more referral questions are put to the assessor about
method
the assessee (e.g., can this child function in a general educational environment?). The
 During WWII, the military would depend even more on psychological test to screen
assessor may meet with the assessee or others before the formal assessment in order to
recruits for service
clarify the reason for referral.
2. The assessor prepares for the assessment by selecting the tools of assessment to be
used. The assessor’s own past experience, education, and training play a key role in the
PSYCHOLOGICAL TESTING AND ASSESSMENT DEFINED specific tests or other tools to be employed in the assessment.
Testing is a term used to refer to everything from the administration of a test to the 3. Finally, the formal assessment will begin.
interpretation of a test score. 4. After the assessment, the assessor writes a report of the findings that is designed to
answer the referral question.
By WWII, a distinction between testing and a more inclusive term “assessment began to 5. More feedback sessions may also be scheduled.
emerge. The US Office of Strategic Services used a variety of procedures and measurement
tools in selecting military personnel for highly specialized positions involving espionage, APPROACHES USED IN ASSESSMENT
intelligence gathering, and the like. The OSS model would later inspire what is now referred Different assessor may approach the assessment task in different ways.
as the assessment center approach to personnel evaluation.  Collaborative Psychological Assessment, the assessor and assessee may work as
partners from initial contact through final feedback. One variety of this approach is
Assessment acknowledges that tests are only one type of tool used by professional assessors. the Therapeutic Psychological Assessment where, therapeutic self-discovery and
A test’s value is intimately linked to the knowledge, skill, and experience of the assessor. understandings are encouraged throughout the assessment process.
 Most notably used in educational settings is the Dynamic Assessment which is an
Psychological Assessment is the gathering and integration of psychology related data for the interactive approach to psychological assessment that usually follows a model of (1)
purpose of making a psychological evaluation that is accomplished through the use of tools evaluation, (2) intervention of some sort, and (3) evaluation. It provides a means for
such as tests, interviews, case studies, behavioral observations, and specifically designed evaluating how the assessee processes or benefits from some type of intervention
apparatuses and measurement procedures. during the course of evaluation.
Psychological Testing is the process of measuring psychology related variables by means of
TOOLS OF PSYCHOLOGICAL ASSESSMENT
devices or procedures designed to obtained a sample of behavior.
TESTS
 A test is a measurement device or technique used to quantify behavior or aid in the
PSYCHOLOGICAL ASSESSMENT PSYCHOLOGICAL TESTING
understanding and prediction of behavior.
Objective Answer a referral question, solve a To obtain some gauge, usually
 Psychological test is a set of items that are designed to measure characteristics of
problem, or arrive at a decision numerical in nature, with regard to
human beings that pertain to behavior (intelligence, personality, etc.). It always involves
through the use of tools of an ability or attribute
analyses of a behavior.
evaluation
 Format pertains to the form, plan, structure, arrangement, and layout of test items as
Process Usually individualized which May be individual or group in
well as to related considerations of such as time limit. It can also used to refer to the
focuses on how an individual nature. After test administration,
form in which a test is administered. Format can also be used to denote the form or
processes rather than simply the the tester typically adds up the
structure of other evaluative tools and processes.
results of that processing number of correct answers or the
 Tests differ in their administration procedure. Tests for one-to-one basis may require
number of certain types of
an active and knowledgeable test administrator. Tests designed for administration to
response
groups may not even require the test administrator to be present while the test takers
Role of Assessor is the key to the process The tester is not the key to the
independently complete the required tasks.
Evaluator of selecting tests and/or other process; one tester may be
 Tests differ in their scoring and interpretation procedures. Score is a code or summary
tools of evaluation as well as substituted for another tester
statement, usually but not necessarily numerical in nature, that reflects an evaluation of
drawing conclusions from the entire without appreciably affecting the
Keep Rising In Zeal, Ignite Ambition’s Heart (KRIZIAH)!
performance on a test, task, interview, or some other sample of behavior. Scoring is the
process of assigning such evaluative codes or statements to performance on tests, tasks,
interviews, or other behavior sample. A cut score is a reference point, usually
numerical, derived by judgement and used to divide a set of data into two or more
classifications.
 Psychometrics is the science of psychological measurement. Psychometrists and
psychometrician refers to a professional who uses, analyzes, and interprets
psychological test data.
 Tests differ with their psychometric soundness or technical quality, which refers to
how consistently and how accurately a psychological test measures what it purports to
measure.
 Psychometric utility refers to the usefulness or practical value that a test or other tool
of assessment has for a particular purpose.
INTERVIEW
 Interview is a method of gathering information through direct communication
involving a reciprocal exchange. It is more than talk.
 If conducted face-to-face, interviewer is taking note of both verbal and non-verbal
behavior. Non-verbal behaviors may include the interviewee’s body language
such as movements and facial expressions in response to the interviewer, the extent
of eye contact, the way they are dresses, and apparent willingness to cooperate.
 Can be conducted through telephone, online interviews, e-mail interviews, and text
messaging.
 An interview may be used to help professionals in human resources. A panel
interview can be employed where, more than one interviewer participates in the
assessment to minimize biases of a lone interviewer.
 Interviewing in the context of clinical and counseling setting, has as its objective not
only the gathering of information, but also target change in the interviewee’s
thinking and behavior. Motivational Interviewing is a therapeutic dialogue that
combines person-centered listening skills such as openness and empathy, with the
use of cognitive-altering techniques designed to positively affect motivation and
effect therapeutic change.
PORTFOLIO
 Portfolio are work products – whether retained on paper, canvas, film, video, audio,
or some other medium. It is a sample of one’s ability and accomplishment. The
appeal of portfolio assessment as a tool of evaluation extends to many other fields,
including education.
CASE HISTORY DATA
 Refers to records, transcripts, and other accounts in written, pictorial, or other form
that preserve archival information, official and informal accounts, and other data
and items relevant to an assessee.
 It can shed light to an individual’s past and current adjustment as well as on the
events and circumstances that may have contributed to any changes in adjustment.
BEHAVIORAL OBSERVATION
 It is the monitoring the actions of others or oneself by visual or electronic means
while recording quantitative and/or qualitative information regarding the actions.
 Naturalistic Observation is a variety of this tool which is used in order to observe
behavior of humans in a natural setting, setting in which the behavior would be
expected to occur.
ROLE-PLAY TESTS
 A role-play test is a tool of assessment wherein assessees are directed to act as if
they were in a particular situation. Assessees may then be evaluated with regard to
their expressed thoughts, behaviors, abilities, and other variables.
 Used in situations where it is too time-consuming, too expensive, or simply to
inconvenient to assess in real situation.
 Role play can be used as both a tool of assessment and a measure of outcome.
COMPUTERS AS TOOLS
 Computers can serve as test administrators (online or offline) and as highly
efficient test scorers. The account of performance spewed out can range from a
mere listing of score to the more detailed extended scoring report, which includes
statistical analyses of the test taker’s performance.
 Computer Assisted Psychological Assessment (CAPA) provides assistance to the
test user, not the test taker.
 Computer Adaptive Testing (CAT), the computer has the ability to tailor the test to
the test taker’s ability or test taking pattern.
WHO, WHAT, WHY, HOW, AND WHERE
WHO ARE THE PARTIES IN THE ASSESSMENT ENTERPRISE?
The Test Developers and publishers create tests or other methods of assessment. The APA
has estimated that more than 20,000 new psychological tests are developed each year.
The Test User are the persons responsible for the selection, administration, and scoring of
tests for the analysis, interpretation, and communication of test results and for any decisions
or actions that are based on test scores. Individuals who simply administer tests, score
tests, and communicate simple or “canned” test results are not test users.
Test User Qualification:
 Level A – There are no special qualification to purchase the products (psychological
tests)
 Level B – A master’s degree and formal training in the ethical administration,
scoring, and interpretation of clinical assessment; Certification by or full active
membership in a professional that requires training and experience in the relevant
area of assessment; A degree or license to practice in the healthcare or allied
healthcare field
 Level C – A doctorate degree with formal training in the ethical administration,
scoring, and interpretation of clinical assessments related to the intended use of the
assessment; Licensure or certification to practice in your state in a field related to
the purchase; Certification by or full active membership in a professional that
requires training and experience in the relevant area of assessment
The Test Taker can be anyone who is the subject of an assessment or an evaluation. The
amount of anxiety they are experiencing and the degree to which that test anxiety might be
significantly affect their test results.
Keep Rising In Zeal, Ignite Ambition’s Heart (KRIZIAH)!
As society evolves and as the need to measure different psychological variables emerges, test
developers respond by devising new tests. Society at large exerts its influence on various
aspects of the testing and assessment enterprise.
WHAT TYPES OF SETTINGS ARE ASSESSMENT CONDUCTED?
Educational Settings. As mandated by law, tests are administered early in school life to help
identify children who may have special needs. Achievement tests evaluates accomplishment
or the degree of learning that has taken place. Diagnostic tests are used to assess the need
for educational intervention as well as to establish or rule out eligibility for special education
program.
Clinical Setting. Used to help screen for or diagnose behavior problems. The hallmark of
testing in clinical settings is that the test is employed with only one individual at a time.
Group testing is used primarily for screening in identifying those individuals who require
further diagnostic evaluation.
Counseling Setting. Regardless of the tools used, the ultimate objective of many such
assessment is the improvement of the assessee in terms of adjustment, productivity, or some
related variables.
Geriatric Setting. This relates to old people, especially with regard to their healthcare. Such
assessments are used in the extent to which assessees are enjoying as good a quality of life
as possible. Quality of life is the variables related to perceived stress, loneliness, sources of
satisfaction, personal values, quality of living conditions, and quality of friendships and other
social supports.
Business and Military Setting. Most notable in the selection and decision making about the
careers of personnel.
Governmental & Organizational Setting. One of the applications is in governmental
licensing, certification, or general credentialing of professionals. Before one is legally entitled
to practice something, they must pass an examination. Members of some professions have
formed organizations with requirements for membership that go beyond licensing or
certification.
Academic Research Setting & Other Settings. Many different kinds of measurement
procedures find application in a wide variety of settings.
WHY IS THERE A NEED FOR PSYCHOLOGICAL ASSESSMENT?
Psychological tests and assessments allow a psychologist to understand the nature of the
problem, and to figure out the best way to go about addressing it. Psychologists use test and
other assessment tools to measure and observe a client’s behavior to arrive at a diagnosis
and guide treatment.
HOW ARE TEST/ASSESSEMENTS ADMINISTERED?
Responsible test users have obligations before, during, and after a test or any measurement
procedure is administered.
Pre-Test Obligations:
 Ethical guidelines dictate that when test users have discretion with regard to the
test administered, they should select and use the tests that are most appropriate for
the individual being tested
 Tests should be stored in a way that ensures its specific contents will not be made
know in advance
 It should be ensured that a prepared and suitably trained person administers the
test properly
 The test administrator (or examiner) must be familiar with the test materials and
procedures and must have at the test site all the materials needed t properly
administer the test
 Test users have the responsibility of ensuring that the room in which the test will be
conducted is suitable and conducive to the testing
During-Test Obligations:
 Rapport between the examiner and the examinee can be critically important.
Rapport is the working relationship between the examiner and the examinee.
After-Test Obligations:
 Safeguarding the test protocols to conveying the test results in a clearly
understandable fashion is one of the obligations of the test user
 Test users who have responsibility for interpreting scores or other test results have
an obligation to do so in accordance with the established procedures and ethical
guidelines
Assessment of people with disabilities can be done through accommodation and alternate
assessment. Accommodation is the adaptation of a test, procedure, or situation, or the
substitution of one test for another, to make the assessment more suitable for an assessee
with exceptional needs. Alternate assessment is an evaluative or diagnostic procedure or
process that varies from the usual, customary, or standardized way a measurement is
derives, either by virtue of some special accommodation made to the assessee or by means of
alternative methods designed to measure the same variable(s).
WHERE TO GO FOR ATHORATIVE INFORMATION: REFERENCE SOURCES
 Test catalogues is one of the most accessible sources of information distributed by
the publisher of the test. Can be tapped by a simple telephone call, e-mail, or note.
 Test manuals have detailed information concerning the development of a particular
test and technical information relating to it.
 Reference volumes
 Journal Articles may contain reviews of the test, updated or independent studies of
its psychometric soundness, or examples of how the instrument was used in either
research or an applied context.
 Online databases
Keep Rising In Zeal, Ignite Ambition’s Heart (KRIZIAH)!
CHAPTER 2: HISTORICAL AND CULTURAL PERSPECTIVE OF
ASSESSMENT
HISTORICAL PERSPECTIVE OF ASSESMENT
EARLY ANTECEDENTS
Tests and testing programs first came into being in China as early as 2200 B.C.E. Testing
was used as a means of selecting who, of many applicants, would obtain government jobs.
Evry third year in China, oral examinations were given to help determine work evaluations
and promotion decisions.
 Han Dynasty (206 B.C.E. to 220 B.C.E.). Test batteries (two or more tests used in conjunction)
were used. Topics were civil law, military affairs, agriculture, revenue, and geography
 Ming Dynasty (1368-1644 C.E.). National Multistage Testing Programs involved local and
regional testing centers equipped with special testing booths. Only those who passes this third
set of tests were eligible for public office .
Reports by British missionaries and diplomats encouraged the English East India Company
in 1832 to copy the Chinese system as a method of selecting employees for overseas
duty. Because testing programs worked well for the company, the British government
adopted a similar system of testing for its civil service in 1855. After the British
endorsement of a civil testing system, the French and German governments followed. In
1883, the US government established the American Civil Service Commission, which
developed and administered competitive examinations for certain government jobs.
 Charles Darwin suggested that to develop a measuring device, we must understand what we want to
measure.
 Sir Francis Galton continued Darwin’s works and he initiated a search for knowledge concerning
human individual differences which is now one of the most important domains in scientific
psychology.
 Galton’s work was extended by James McKeen Cattel, who coined the term mental test. Cattel
perpetuated and stimulated the forces that ultimately led to the development of modern tests.
 intelligence testing. It yielded several scores, permitting an analysis of an individual’s pattern
EXPERIMENTAL PSYCHOLOGY AND PSYCHOPHYSICAL MEASUREMENT
 J.E. Herbart used mathematical models as the basis for educational theories that strongly influenced
19th-century educational practices.
 E.H. Weber followed and attempted to demonstrate the existence of a psychological threshold; the
minimum stimulus necessary to activate a sensory system.
 G.T. Fechner devised the law that the strength of a sensation grows as the logarithm of the stimulus
intensity.
 Wilhelm Wundt set up a laboratory at the University of Leipzig in 1879, and was credited with
founding the science of psychology.
 E.B. Titchner succeeded the works of Wundt and founded structuralism
 G. Whipple, a student of Titchner provided the basis for immense changes in the field of testing by
conducting a seminar at the Carnegie Institute in 1919. The seminar came up with the Carnegie
Interest Inventory and later the Strong Vocational Interest Blank.
Psychological testing developed form at least two lines of inquiry:
 One based on the work of Darwin, Galton, and Cattell on the measurement on individual
differences; and
 The other (more theoretically relevant and probably stronger) based on the work of the German
psychophysicist Herbart, Weber, Fechner, and Wundt (experimental psychology developed from
this)
There are also tests that arose in response to important needs such as classifying and
identifying the mentally and emotionally handicapped.
 Seguin Form Board Test was developed in an effort to educate and evaluate the mentally
disabled.
 Kraepelin devised a series of examination for evaluating emotionally impaired people.
THE EVOLUTION OF INTELLIGENCE AND STANDARDIZED ACHIEVEMENT TEST
The Binet-Simon Scale had its first version published in 1905. The instrument contained
30-item of increasingly difficult and was designed to identify intellectually subnormal
individuals. It is designed to help identify Paris schoolchildren with intellectual disability The
1908 Binet-Simon Scale has been improved and introduced the significant concept of a
child’s mental age.
L.M. Terman of Stanford University had revised the Binet test for the use in the US. The
Stanford-Binet Intelligence Scale was the only American version of the Binet test that
flourished.
Robert Yerkes was requested for assistance by the army. He headed a committee who
developed two structured group tests of human abilities:
1. The Army Alpha was a verbal test, measuring such skills as ability to follow
directions; it required reading ability
2. The Army Beta presented nonverbal problems to illiterate subjects and recent
immigrants who were not proficient in English It measured the intelligence of
illiterate adults.
Standardized Achievement Tests provide multiple-choice questions that are standardized
on a large sample to produce norms against which the results of new examinees can be
compared. It became famous because of the ease of administration and scoring and the lack
of subjectivity/favoritism that can occur in essay or other written tests.
Two years after the revision of 1937 Stanford-Binet Test, David Wechsler published the
Wechsler-Bellevue Intelligence Scale, which contained several interesting innovations in
or combination of abilities. It can produce the performance IQ score.
Personality Tests measured presumably stable characteristics or traits that theoretically
underlie behavior.
 The first structured Personality Test was the Woodworth Personal Data Sheet,
which was developed during WWI and was published in final form just after the war.
 Projective test is also developed which provides an ambiguous stimulus and
unclear response requirement (Rorschach Inkblot Test and Thematic Apperception
Test was the most famous).
 The Minnesota Multiphasic Personality Inventory (MMPI) published in 1943
began a new era for structured personality tests. It is currently the most widely
used and referenced personality test.
 Factor analysis is a method of finding the minimum number of dimensions. Called
factors, to account for a large number of variables. R.B. Cattel had introduced the
Keep Rising In Zeal, Ignite Ambition’s Heart (KRIZIAH)!
Sixteen Personality factor Questionnaire (16PF), which remains as one of the
most well-constructed structured personality tests and an important example of a
test developed with the aid of factor analyses.
SOME ISSUES REGARDING CULTURE AND ASSESSMENT
Individualist Culture is characterized by value being placed on traits such as self-reliance,
autonomy, independence, uniqueness, and competitiveness. Collectivist culture value is
placed on traits such as conformity, cooperation interdependence, and striving toward group
goals. Culture-specific tests are designed for use with people from one culture but not from
another.
CHAPTER 3: A STATISTICS REFRESHER
SAMPLING AND SAMPLING TECHNIQUES
A population is the set of all individuals of interest in a particular study. A sample is a set of
individuals selected from a population, usually intended to represent the population in a
research study.
Sampling is the process of selecting observations to provide an adequate description and
inferences of the population. Probability sampling is a method of selecting a sample wherein
each element in the population has a known, nonzero chance of being included in the
sample; otherwise, it is nonprobability sampling.
 Simple random sampling is a sampling method wherein all elements of the
population have the same probability of inclusion in the selected sample (e.g., draw
lots, fishbowl method).
 Stratified sampling is a probability method where we divide the population into
nonoverlapping subpopulation or strata (a group whose members are of the
characteristics), and then randomly pick samples from each stratum (population –
strata – random selection – sample).
 Cluster sampling divide the population into nonoverlapping groups or clusters.
And then randomly select clusters (a group whose members are not of the same
characteristics).
 Systematic sampling is where researchers select members of the population at a
regular interval (or k) determined in advance.
 Multistage sampling draws a sample from a population using smaller and smaller
groups (units) at each stage. It’s often used to collect data from a large,
geographically spread group of people in national surveys.
Multistage cluster sampling is where the researcher divides the
population into groups at various stages for a better data collection,
management, and interpretation. The groups are called clusters.
Multistage random sampling is where the researcher chooses the
samples randomly at each stage. The researcher does not create clusters,
but narrow down the convenience sample by applying random sampling.
 In convenience sampling, units are selected for inclusion in the sample because
they are easiest for the researcher to access.
 In snowball sampling, new samples are recruited by other sample to form part of
the sample. This can be a useful way to conduct research about people with specific
traits who might otherwise be difficult to identify.
 Purposive sampling/Judgement sampling involves the researcher using their
expertise to select a sample that is most useful to the purpose of the research. An
effective purposive sample must have clear criteria and rationale for inclusion.
 Quota sampling is done until a specific number of samples for various
subpopulations have been selected.
Descriptive statistics are methods used to provide a concise description of a collection of
quantitative information. Inferential statistics are methods used to make inferences from
observations of a small groups of people known as a sample to a larger group of individuals
known as population.
The levels of measurements are:
 Nominal scale - categorical data and numbers that are simply used as identifiers
or names represent a nominal scale of measurement (e.g., gender).
 Ordinal scales - represents an ordered series of relationships or rank order (e.g.,
Likert-type scale, rank in a contest).
 Interval scales - represents quantity and has equal units but for which zero
represents simply an additional point of measurement is an interval scale. In
addition, zero does not represent the absolute lowest value. Rather, it is point on
the scale with numbers both above and below it.
 Ratio scale - similar to the interval scale in that it also represents quantity and
has equality of units. However, this scale also has an absolute zero (no numbers
exist below the zero).
A frequency table is an ordered listing of number of individuals having each of the different
values for a particular variable. A frequency distribution shows the pattern of frequencies
over the various values. They can be illustrated through a histogram, bar graph, or frequency
polygon.
Skewness is the nature and extent to which symmetry is absent. Skewed distribution in the
distribution in which the scores pile up one side of the middle and are spread out on the
other side. Can be positively skewed or negatively skewed.
Floor effect is a situation in which many scores pile up at the low end of a distribution
(creating skewness to the right; positively skewed distribution) because it is not possible to
have any lower score. Ceiling effect is a situation in which many scores pile up at the high
end of a distribution (creating skewness to the left; negatively skewed distribution) because it
is not possible to have a higher score.
The term testing professionals use to refer to the steepness of a distribution in its center is
kurtosis. It is the peakedness/flatness. It can be platykurtic (relatively flat), leptokurtic
(relatively peaked, and mesokurtic (somewhere in the middle).
The central tendency of a distribution refers to the middle of the group of score.
 Mean is the sum of the scores divided by the number of scores.
 Mode is the value with the greatest frequency in a distribution
 Median is the middle score.
Variability is an indication of how scores in a distribution are scattered or dispersed.
 Range indicates the distance between the two most extremes scores in distribution
(range = highest score – lowest score)
 Variance is the average of each score’s squared difference from the mean
 Standard deviation is simply the square root of the variance. It indicates the
average deviation from the mean, the consistency in the scores, and how far scores
are the spread out around the mean.
A standard score is a raw score that has been converted from one scale to another scale,
where the latter scale has some arbitrarily set mean and standard deviation.
Keep Rising In Zeal, Ignite Ambition’s Heart (KRIZIAH)!
Correlation is an expression of the degree and direction of correspondence between two
things. The coefficient of correlation is the numerical index that expresses the relationship,
it tells us the extent to which X and Y are corelated . Can be positive or negatively
correlated.
Code of ethics: Beneficence & Nonmaleficence, Fidelity & Responsibility, Integrity,
Justice, and Respect for People's Rights & Dignity
CHAPTER 5: RELIABILITY
Reliability – consistency in measurement; proportion of the total variance attribute to true
variance. The greater the proportion of the total variance attributed to true variance, the
more reliable the test.
In psychological assessment context, a reliable test means that it gives a dependable and
consistent results even though it will be used again and again although, error would always
be present. Error implies that there will always be some inaccuracy in our measurement. One
of the goals in psychological assessment is to minimize, as much as possible, the presence of
error in testing.
The reliability can be measured through its reliability coefficient – an index that indicates
the ratio between the true score variance on a test and the total variance. Reliability
estimates in the range of .70 and .80 are good enough for most purposes in basic research.
Variance from true differences is true variance, and variance from irrelevant, random sources
is error variances.
[observed score = true ability + random error]
According to the Classical Test Theory, each person has a true score that would be obtained
if there were no errors in measurement. Difference between the true score and the observed
score results from measurement error. The true score of an individual will not change with
repeated applications of the same test.
validity. Typical cause of systematic error includes observational error, imperfect
instrument calibration, and sampling bias.
SOURCES OF ERROR VARIANCE
Aside from measurement error, we also have the error variance (also called residual error,
residual variance, or unexplained variance). It is the element of variability in a score that is
produced by extraneous factors, such as measurement imprecision, and is not attributable to
the independent variable or other controlled experimental manipulation. There are 3 sources
of error variance:
1. Construction
Example of this is the item sampling or content sampling – refer to variation among
items within a test as well as to variation among items between test.
2. Administration
Can be affected by the (1) Test Environment (room temperature, level of lighting,
and the amount of ventilation and noise); (2) Testtaker variables (emotional
problems, physical discomfort, lack of sleep): and (3) Examiner-related Variables
(examiners physical appearance and demeanor – even the presence or absence of an
examiner).
3. Scoring and Interpretation
Despite rigorous scoring criteria set forth in many better-known tests of intelligence,
the scorer (or rater) can be a source of error variance. If subjectivity is involved in
scoring, then the scorer can be a source of error variance.
RELIABILITY ESTIMATES/MEASURES
1. Test-Retest Reliability (Time-Sampling Reliability)
It uses the same instrument to measure the same thing/factor at two points in time. It is
an estimate of reliability obtained by correlating pairs of scores from the same people on
two different administrations of the same test. It is appropriate when evaluating a test
that purports to measure something that is relatively stable over time (e.g., personality
traits). Poor test-retest correlations do not always mean that a test is unreliable –
suggest that the characteristic under study has changed. Pearson r or Spearman Rho is
usually used to measure the test-retest reliability where: 1 – perfect reliability; ≥0.5 –
poor reliability; and 0 – no reliability.

Purpose: To evaluate the stability of a measure

THE CONCEPT OF RELIABILITY
Measurement error – all the factors associated with the process of measuring some variable,
other than the variable being measured. In other terms, the variables that we are supposed
to measure are affected by the extraneous variable. There are two types of measurement
error:
2.
Random error is a source of error in measuring a targeted variable caused by
unpredicted fluctuations and inconsistencies of other variables in the measurement
process. This source of error fluctuates from one testing situation to another with
no discernible pattern that would systematically raise or lower scores thus,
affecting reliability. The main reasons for random error are limitations of
instruments, environmental factors, and slight variations in procedure.
Systematic Error refers to a source of error in measuring a variable that is typically
constant or proportionate to what is presumed to be the true value of the variable
being measured. Once a systematic error becomes known, it becomes predictable –
as well as fixable. Systematic errors primarily influence a measurement’s
Typical use: When assessing the stability of various personality traits
Number of Testing Sessions: 2
Sources of Error Variance: Administration
Statistical Procedure: Pearson r or Spearman rho
Parallel-Forms and Alternate-Forms Reliability
This reliability compares two equivalent forms of a test that measure the same attribute.
Used to assess the consistency of the results of two tests constructed in the same way
from the same content domain. In order to call the forms “parallel”, the observed score
must have the same mean and variances. If the test are merely different versions
(without the sameness of observes score), they are called alternate forms.
Parallel forms reliability can avoid some problems inherent with test-retest like
practice and fatigue effects but you have to create a large number of questions that
measure the same construct. Spearman rho or Pearson r can be used to measure.
Purpose: To evaluate the relationship between different forms of a measure
Typical use: When there is a need for different forms of a test
Keep Rising In Zeal, Ignite Ambition’s Heart (KRIZIAH)!
Number of Testing Sessions: 1 or 2
Sources of Error Variance: Test Construction or Administration
Statistical Procedure: Pearson r or Spearman rho
3. Internal Consistency
Assesses the correlation between multiple items in a test that are intended to measure
the same construct. It measures how related the items in the test that measures a
certain construct or characteristic. You can calculate the internal consistency without
repeating the test or involving other researchers, so it’s a good way of assessing
reliability when you only have one data set. In effect, we judge the reliability of the
instrument by estimating how well the items that reflect the same construct yield similar
results. There are three ways to compute the internal consistency:
A. Split-half Reliability
Correlating two pairs of scores obtained from equivalent halves of a single test
administered once. This is useful when it is impractical to assess reliability with two
tests or to administer test twice. Results of one half of the test are then compared
with the results of the other.
Steps in performing split-half reliability:
(1) Divide the test into equivalent halves. Do not divide in the middle because it
would lower the reliability, instead use the odd-even system – where one sub
score is obtained for the odd-numbered items in the test and another for the
even-numbered items.
(2) Calculate the Pearson r between scores on the two halves of the test.
(3) Adjust the half-test reliability using the Spearman-Brown formula. This allows
a test developer or user to estimate internal consistency reliability from a
correlation of two halves. It can also be used to determine the number of items
needed to attain a desired level of reliability.
B. Kuder-Richardson Formula 20 (KR-20)
A measure of internal consistency reliability for measures with dichotomous
choices. It is a special case of Cronbach’s Alpha, computed for dichotomous scores.
It is often claimed that a high KR-20 coefficient (e.g., > 0.90) indicates a
homogeneous test. However, like Cronbach’s Alpha, homogeneity (that is,
unidimensionality) is actually an assumption, not a conclusion, or reliability
coefficients.
C. Cronbach’s Alpha/Coefficient alpha
The most famous and commonly used among reliability coefficients because it
requires only one administration of the test. Cronbach’s Alphha is mathematically
equivalent to the average of all possible split-half estimates. The general rule of
thumb is that a Cronbach’s alpha of .70 and above is good, .80 and above is better,
and .890 and above is best. This is calculated to help answer questions about how
similar sets of data are.
Although it does have some limitations; scores that have a lower number of items
associated with them tend to have lower reliability, and sample size can also
influence the results.
D. Average Proportional Distance (ADP)
Rather than focusing on similarity between scores in items of a test, the ADP is a
measure that focuses on the degree of differences that exists between item scores.
Average Proportional Distance Method is a measure used to evaluate the internal
consistency of a test that focuses on the degree of difference that exist between item
scores.
One advantage of ADP over Cronbach’s alpha is that APD’s index is not connected to
the number of items on a measure unlike the later method.
Purpose: To evaluate the extent to which items on a scale relate to one another
Typical use: When evaluating the homogeneity of a measure
Number of Testing Sessions: 1
Sources of Error Variance: Test Construction
Statistical Procedure: Pearson r with Spearman Brown/ KR-20/ Cronbach’s alpha/
ADP
4. Inter-scorer Reliability
Inter-scorer reliability is the degree of agreement or consistency between two or more
scorers (or judges or raters) with regards to a particular measure. It is often used when
coding nonverbal behavior.
Purpose: To evaluate the level of agreement between raters on a measure
Typical use: Interviews or coding of behavior. Used when researchers need to show that
there is consensus in the way that different raters view a particular behavior pattern
(and hence no observer bias)
Number of Testing Sessions: 1
Sources of Error Variance: Scoring and Interpretation
Statistical Procedure: Cohen’s kappa/ Pearson r /Spearman rho
USING AND INTERPRETING A COEFFICENT OF RELIABILITY
As a rule of thumb, .90s rates a grade of A, ,80s rates a B, and anywhere from .65 through
the .70s rates a weak grade that borders on failing.
THE PURPOSE OF THE RELIABILITY COEFFICIENT. If the purpose of determining
reliability is to break down the error variance into its parts, then a number of reliability
coefficients would have to be calculated.
NATURE OF TESTS
1. Homogenous vs Heterogeneous Items
Homogenous (equal difficulties and equal intercorrelations) items have high
reliability. The more homogenous a test is, the more inter-item consistency (refers to
the degree of correlation among all the items on a scale) it can be expected to have.
2. Dynamic vs Static Characteristics
Dynamic Characteristics are traits, state, and/or ability presumed to be ever-
changing as a function of situational and cognitive experiences. Static
Characteristics on the other hand, are the trait, state, and ability which are
relatively unchanging.
3. Restriction or Inflation of Range
Restricted variance results in a correlation coefficient that tends to be lower.
Inflated variance results in a correlational coefficient that tends to be higher.
4. Speed Test vs Power Tests
A power test has a time limit that is long enough to allow test takers to attempt all
items, some items have a higher degree of difficulty. A speed test has items of
Keep Rising In Zeal, Ignite Ambition’s Heart (KRIZIAH)!
uniform level of difficulty, and when given time limits, all test takers should be able
to complete all the test items correctly (based on performance speed).
5. Criterion-referenced Tests
Designed to provide an indication of where a testtaker stands with respect to some
variable or criterion. Scores on this test tends to be interpreted in pass-fail terms,
and any scrutiny of performance on individual items tends to be for diagnostic and
remedial purposes.
THE STANDARD ERRROR OF MEASUREMENT
Provides an estimate of the amount of error inherent in an observed score or measurement.
The relationship between the SEM and the reliability of a test is inverse; the higher the
reliability of a test, the lower the SEM.
SEM is the tool used to estimate or infer the extent to which an observed score deviates from
a true score. It is the standard deviation of a theoretically normal distribution of test scores
obtained by one person on equivalent tests. It is an index of the extent to which one
individual’s scores vary over tests presumed to be parallel.
The SEM is useful in establishing the confidence interval – a range or band of test scores
that is likely to contain the true score. The standard error of the difference is the statistical
measure that can aid a test user in determining how large a difference should be before it is
considered statistically significant.
What to do about low reliability?
1. Increase the numbers of items. The larger the sample, the more likely that the test
will represent the true characteristics.
2. Factor and Item Analysis. Tests are more reliable if they are unidimensional:
measuring a single ability, attribute, construct, or skill. Examine the correlation
between each item and the total score for the test.
CHAPTER 6: VALIDITY
A measurement can be reliable but not valid, however, if a measure is valid, then it is also
reliable.
Validity – as applied to a test, is a judgement or estimate of how well a test measures what it
purports to measure in a particular context.
 Judgment is based on evidences about the appropriateness of inferences drawn
from test scores. Validity of test must be shown from time to time to account for
culture and advancement.
 Validation is the process of gathering and evaluating evidence about validity. Test
user and testtaker both have roles in validation of test.
 Test users can conduct a (1) validation studies to yield insights regarding a
particular population of as compared to the norming sample or (2) local validation
studies which is absolutely necessary when test user plans to alter elements of the
test.
One way measurement specialists have traditionally conceptualized validity is according to
three categories of types of validity (Trinitarian view) – believes that evaluations of validity
should focus on (1) how well a measurement’s quantitative value represents the attribute
being measured and how well the measurement gives scores relevant to what the test is
trying to measure.
1. Content Validity
Is based on an evaluation of the subjects, topics, or content covered by the items in
the test
2. Criterion-related Validity
Is obtained by evaluating the relationship of scores obtained on the test to scores on
other tests or measures
3. Construct Validity
“Umbrella Validity” because every other variety of validity falls under it. It assesses
whether a test is representative of all aspects of the construct. This is a measure of
validity that is arrived at by executing a comprehensive analysis of
a. how scores on the test relate to other test scores and measures; and
b. how scores on the test can be understood within some theoretical framework
for understanding the construct that the test was designed to measure
Keep Rising In Zeal, Ignite Ambition’s Heart (KRIZIAH)!
ECOLOGICAL VALIDITY. Refers to a judgment regarding how well a test measures what it
purports to measure at the time and place that the variable being measured (typically a
behavior, cognition, or emotion) is actually emitted.
FACE VALIDITY. Relates more to what a test appears to measure to the person being tested
than to what the test actually measures. Judgment concerning how relevant the test items
appear to be, which is usually from testtaker’s, not test user’s perspective. Lack of face
validity may result to a lack of confidence in perceived effectiveness of test which decreases
testtaker’s motivation or cooperation. A test’s lack of face validity could contribute to a lack of
confidence in the perceived effectiveness of the test – with a consequential decrease in the
testtaker’s cooperation or motivation to do his/her best. Ultimately, face validity may be more
a matter of public relations than psychometric soundness.
VALIDITY, BIAS, AND FAIRNESS
 Rating error is a judgment resulting from the intentional or unintentional misuse of a
rating scale.
 Leniency error (generosity error) is an error in rating that arises from the tendency on
the part of the rater to be lenient in scoring, marking, and/or grading.
 Severity error lies on the opposite extreme of the leniency error where, a rater would pan
just about everything they review
 Central tendency error is where a rater exhibits a general and systematic reluctance to
giving ratings at either the positive or the negative extreme, thus the ratings would
cluster in the middle of the rating continuum

CONTENT VALIDITY. A judgment of how adequately a test samples a behavior

representative of the universe of behavior that the test was designed to sample. In the test of
ensuring content validity, test developers strive to include key components of the construct
targeted for measurement, and exclude content irrelevant to the construct targeted for
measurement. One example of content validity is an achievement test where, the test items
approximate the proportion of materials covered in the course. CHAPTER 7: UTILITY
Utility refers to how useful a test is specifically, to the practical value of using a test to aid in
When making a test, we usually have a test blueprint – a plan regarding the types of
decision making; usefulness/practical value of testing to improve efficiency. An index of
information to be covered by the items, the number of items tapping each area of coverage,
utility can tell us something about the practical value of the information derived from scores
the organizations of the items in the test, etc. It is also important to note that a content valid
on the test.
test may not be universally valid – it depends on the politics, culture and history of the
country.
FACTORS THAT AFFECT A TEST’S UTILITY:
A. Psychometric Soundness – a test is said to be psychometrically sound for a
CRITERION-RELATED VALIDITY. A judgment of how adequately a test score can be used to
particular purpose if reliability and validity coefficients are acceptably high. The
infer an individual’s most probable standing on some measure of interest – the measure of
higher the criterion-related validity of test scores for making a particular decision,
interest being the criterion. Whatever the criterion, ideally it is relevant, valid, and
the higher the higher the utility of the test is likely to be however, some tests may be
uncontaminated.
psychometrically sound, but has a little utility – particularly to the testtakers who
A. Concurrent validity – index of the degree to which a test score is related to some
fail to follow the test’s directions.
criterion measure obtained at the same time.
B. Costs – one of the most basic elements in any utility analysis; refers to
B. Predictive validity – index of the degree to which a test score predicts some criterion
disadvantages, losses, or expenses in both economic and noneconomic terms.
measure.
C. Benefits – utility of a test may take into account whether the benefits (profits, gains,
or advantages) of testing justify the costs of administering, scoring, and interpreting
Judgments of criterion-related validity, whether concurrent or predictive, are based on two
the test.
types of statistical evidence: the validity coefficient and expectancy data. Validity coefficient is
a correlation that provides a measure of the relationship between test scores and scores on
UTILITY ANALYSIS
the criterion measure. There is no rule for determining the minimum acceptable size of a
Utility analysis – family of techniques that entails a cost-benefit analysis designed to yield
validity coefficient. Essentially, it should be high enough to result in the identification and
information relevant to a decision about the usefulness and/or practical value of a tool of
differentiation of testtakers with respect to target attribute.
assessment. It is an umbrella term covering various possible methods that can be employed
to yield various kinds of outputs. Undertaken for the purpose of evaluating whether the
CONSTRUCT VALIDITY. Viewed as the unifying concept for all validity evidence. It is a
benefits of using a test outweighs the costs. The endpoint is typically an educated decision
judgment about the appropriateness of inferences drawn from test scores regarding
about which of many possible courses of action is optimal.
individual standings on a variable called a construct. Evidences that a test has construct
validity are (1) the test is homogeneous, measuring a single construct, (2) test scores
Utility analysis can be conducted through:
increase/decrease as a function of age, the passage of time, or an experimental manipulation
1. Expectancy data. Converting a scatterplot of test data to an expectancy table which can
as theoretically predicted, (3) test scores obtained after some event or the mere passage of
provide an indication of the likelihood that a testtaker will score within some interval of
time (or posttest scores) differ from pretest scores as theoretically predicted, (4) test scores
scores on a criterion measure – an interval that may be categorized as “passing”,
obtained by people from distinct groups vary as predicted by the theory, and (5) test scores
“acceptable”, or “failing”.
correlate with scores on other tests in accordance with what would be predicted from a
theory that covers the manifestation of the construct in question.
The Taylor-Russell tables provide an estimate of the percentage of employees hired by
the use of a particular test who will be successful at their jobs, given different
combinations of three variables: (1) the test’s validity (the value assigned is the validity
Keep Rising In Zeal, Ignite Ambition’s Heart (KRIZIAH)!
coefficient), (2) the selection ratio used (relationship between the number of people to be
hired and the number of people available to be hired, and (3) the base rate (percentage of
people hired under the existing system for a particular position).
Use of Naylor-Shine table can compensate for the limitations of the Taylor-Russell
tables. It entails obtaining the difference between the means of the selected and
unselected groups to derive an index of what the test is adding to an already established
procedures.
2. The Brodgen-Cronbach-Gleser formula. Used to calculate the dollar amount of utility
gain (estimate of the benefit, monetary or otherwise, of using a particular test/selection
method) resulting from the use of a particular selection instrument under specified
conditions.
Modification of the BGC formula is the productivity gain, which exist for researchers
who prefer their findings in terms of productivity gains (estimated increase in work
output) rather than the financial ones.
METHODS FOR SETTING CUT SCORES
ANGOFF METHOD. Setting fixed cut scores can be applied to personnel selection tasks as
well as to questions regarding the presence or absence of a particular trait, attribute, or
ability. The judgment of the experts is averaged to yield cut scores for the test.
KNOWN GROUPS METHOD. Collecting of data on the predictor of interest from groups
known to possess, and not to possess a trait, attribute, or ability of interest. Based in an
analysis of this data, a cut score is set on the test that best discriminates the two groups’ test
performance.
IRT-BASED METHODS. Each item is associated with a particular level of difficulty, and
testtakers must answer items that are deemed to be above some minimum level of difficulty
to pass the test, which is determined by experts and serves as the cut score. Techniques can
be item-mapping method, and bookmark method.
CHAPTER 8: TEST DEVELOPMENT
The creation of a good test is the product of the thoughtful and sound application of
established principles of test development – an umbrella term for all that goes into the
process of creating a test. The process of developing a test occurs in five stages:
1. TEST CONCEPTUALIZATION
Ideas for a test
2. TEST CONSTRUCTION
Stage in the process of test development that entails writing
test items (or re-writing or revising existing items), as well
as formatting items, setting scoring rules, and otherwise
designing and building a test.
3. TEST TRYOUT
Once a preliminary form of the test has been developed, it
is administered to a representative sample of testtakers
under conditions that simulate the conditions that the final
version of the test will be administered.
4. ITEM ANALYSIS
Statistical procedures are employed to assist in making
judgements about which items are good as they are, which
items need to be revised, and which items should be discarded. The analyses may
include item reliability, item validity, and item discrimination.
5. TEST REVISION
Action taken to modify a test’s content or format for the purpose of improving the
test’s effectiveness as a tool of measurement.
TEST CONCEPTUALIZATION
PILOT WORK. The preliminary research surrounding the creation of a prototype of the test.
Test items may be pilot studied to evaluate whether they should be included in the final form
of the instrument. The test developer typically attempts to determine how best to measure a
targeted construct. Pilot work is a necessity when constructing tests or other measuring
instruments for publication and wide distribution.
TEST CONSTRUCTION
SCALING. The process by which a measuring device is designed and calibrated and by which
numbers (or other indices) – scale values – are assigned to different amounts of the trait,
attribute, or characteristic being measured.
L.L. Thurstone introduced the notion of absolute scaling – a procedure for obtaining a
measure of item difficulty across samples of test takers who vary in ability.
Scaling methods are as follows:
 Rating Scale. Can be defined as a grouping of words, statements, or symbols on which
judgement of the strength of a particular trait, attitude, or emotion are indicated by the
test taker. The use of rating scales of any type results in ordinal-level data.
 Method of Paired Comparisons. Test takers are presented with pairs of stimuli which
they are asked to compare and must select one.
 Guttman scale. Items on it range sequentially from weaker to stronger expressions of
the attitude, belief or feeling being measured. All respondents who agree with the
stronger statements of the attitude will also agree with milder statements. The resulting
data are then analyzed by means of scalogram analysis.
WRITING ITEMS. It is usually advisable that the first draft contain approximately twice the
number of items that the final version of the test will contain. An item pool is the reservoir
or well from which items will or will not be drawn for the final version of the test.
Items presented in a selected-response format require test takers to select a response from
a set of alternative responses.
 Multiple-choice formats have three elements: (1) stem, (2) correct alternative/option,
and (3) several distractors/foils.
 In matching items has two columns: premises on the left and responses on the
right.
 True-False items requires the test taker to indicate whether the statement is or is
not a fact.
Items presented in a constructed-response format require test takers to supply or to create
the correct answer, not merely to select it.
 Completion item
 Short-answer item
 Essay item
Computerized Adaptive Testing (CAT) refers to an interactive, computer-administered test-
taking process wherein items presented to the test taker are based in part on the test taker’s
performance on previous items. It tends to reduce floor effects and ceiling effects. It also has
Keep Rising In Zeal, Ignite Ambition’s Heart (KRIZIAH)!
the ability to tailor the content and order of presentation of test items in the basis of
responses to previous items referred to as item branching.

SCORING ITEMS. Many different test scoring models have been devised and some of them
are as follow:
 Cumulative model – the higher the score on the test, the higher the test taker is on
the ability, trait, or other characteristic that the test purports to measure.
 Class/Category scoring – test taker responses earn credit toward placement in a
particular class or category with other test takers whose pattern of responses is
presumably similar in some ways.
 Ipsative scoring – comparing a test taker’s score on one scale within a test to
another scale within that same test.

TEST TRYOUT
The test should be tried out on people who are similar in critical respects to the people for
whom the test was designed. A good test item is reliable and valid, and it helps to
discriminate test takers.

ITEM ANALYSIS
ITEM-DIFFICULTY INDEX. It is obtained by calculating the proportion of the total number
of test takers who answered the item correctly. The result can range from 0 to 1, and the
larger the item-difficulty index, the easier the item. For example, 50 out of 100 students got
the item correct so 50 divided by 100 = .5 (p)

ITEM-RELIABILITY INDEX. Provides an indication of the internal consistency of a test. The

higher the index, the greater the tests internal consistency. One statistical that can be used
is factor analysis, where items get factored and items belonging to several factors can be
revised or eliminated.

ITEM-VALIDITY INDEX. Provides an indication of the degree to which a test is measuring

what it purports to measure. The higher the item-validity index, the greater the test’s
criterion-related validity.

ITEM-DISCRIMINATION INDEX. How adequately an item separates or discriminates

between high scorers and low scorers on an entire test.