Test Construction

Week 5 & 6
Ms. Saira Javed
 Context for Test Construction
• no scale exists measuring a specific construct
existing scales do not represent the construct
adequately: –lack of reliability
–lack of reliability –lack of validity –outdated
(old words; meaning of words changed;
attitudes changed)
–insensitive for changes
-Stake holders
 Construct Definition

• scope of the scale (level of abstraction)

• what is to be measured? –broad or narrow
construct?
• definition of the construct in literature?
–adopt a definition vs. work out an own
definition
–clearly describe what is meant by your
construct (and what is not meant)
• aspects of construct
•continuum vs. categories –frequency / intensity
of experiencing emotions (joy, anger)
•scores on items can be combined using
(weighted) means
•response patterns can be analyzed yielding
classes of reaction styles
• unidimensional vs. multidimensional –
frequency of experiencing specific positive
emotions
• facets of a construct
 Recommendation

•write out a brief, formal description of the

construct
•relate it to other constructs
•search literature for information
-Subject Matter Experts
 Design Scale

• •item writing
• •operationalization of construct
• •sample systematically all content that is
potentially relevant to the construct
–one may always discard items...
• the initial item pool
–is broader than one‘s own theoretical view about the construct
• –includes multiple items for each potential facet / dimension
• dimension –includes also items that will finally proof to
be distinct or tangential to the construct –search for
aspects in literature (scientific, but also fiction,
dictionnaries, etc.)
focus groups
interviews
item generation is a „creative act“
• basic principles of item writing
• Selected-response format
• Constructed-response format
•simple, straightforward, and appropriate
language –adequate to reading level
–no trendy expressions
Ex.: Do you feel happy? never –sometimes –
always
• basic principles of item writing
•one aspect at a time
–I feel happy and beloved
–I do not insult people because it is morally
wrong
•individuals must differ on items
–constants are useless
–I am the happiest individual alive
• basic principles of item writing
• •avoid frequencies in item wording
– –Sometimes, I am happy

• avoid negatives to reverse meaning of an item,

but include negative aspects
–I am not happy
• I am sad
• response format / scaling
• •depends on introduction
– –How often do you feel ….
– •frequency format (never, always)
–Do you agree to….
•Agreement (not at all, very much so)
–How much does … correspond to you
•similarity (not at all like me, very much like me)
–How do you judge…
•evaluation (terrible, excellent)
• response format
• •avoid middle category to increase variability
– –not always desirable
• •avoid too many categories
– respondents cannot differentiate adequately
differentiate adequately
– •avoid check-lists
• response bias
– Item Sequencing
• Pilot Test
• •small sample
• •critique scale
• –ambiguous or confusing items
• –mismatch of items and scale
 Administration and Item Analysis

• large set of items (all potential items) considerable sample size

• •correlation with criteria (nontest-criteria)
• •internal consistency (interitem structure)
• Item-difficulty index
• Item-discrimination index
• Analysis of item alternatives
• Item-characteristics curve
• Other considerations
– Guessing
– Item fairness
– Speed test
 Validate and Norm

• Validity: does the scale measure, what it

intends to measure?
• •content validity
• •convergent validity
• •criterion validity
• •discriminant validity
• Norms: what are the properties of the
distribution of scores for a given population?
 Test Revision
• Cross-validation
• Co-validation