Unidimensional and Multidimensional

IRT Analysis for a Competency Test in

Guidance & Counseling

Adonis P. David
Philippine Normal University
What to Expect:

1. Brief overview of the unidimensional and

multidimensional approach in IRT analysis.

2. Introduction to the development of a competency

test in guidance and counseling.

3. Examination of the test using unidimensional and

multidimensional approaches.
Background:

In the Philippines, there have been recent efforts to develop

cognitive measures like achievement and aptitude tests
using the framework of Item Response Theory (IRT).

There are instances were a measure is not measuring one

single trait or ability (unidimensional) but a set of traits or
abilities (multidimensional).

The probability of successfully answering a test item in a

unidimensional model depends on one underlying ability
dimension, while the probability of success in a
multidimensional model depends on multiple ability
dimensions (Hartig & Hohler, 2009).
Unidimensional IRT Analysis

Refers to the traditional and standard IRT

models commonly used in the development of
cognitive measures.

In this approach, items are expected or

assumed to be unidimensional- that is, items
are supposed to measure a similar, single
latent trait.

In Rasch analysis, a single latent trait is

assumed to determine individuals'
performances on a test (Wang, Yao, Tsai,
Wang, & Hsieh, 2006).
Unidimensional IRT Approach
1 ITEM1

ITEM1 ABILITY 1
1
ITEM2

ITEM2

ABILITY
1 ITEM3
ITEM3

ABILITY 2

ITEM4
ITEM4

1A 1B

Composite Approach Consecutive Approach

Multidimensional IRT Analysis

An extension of the traditional IRT models.

MIRT has been described as a highly

interesting methodology for assessing
competencies within educational contexts
since it is capable of taking into account
multiple abilities simultaneously (Hartig &
Holer, 2009).

Done through assuming correlations and

simultaneously calibrating all the items and all
the subtests of a given measure.
Multidimensional IRT Approach
1 ITEM1
1 ITEM1
ABILITY 1
ABILITY 1
ITEM2
ITEM2

1 ITEM3
1 ITEM3

ABILITY 2 ABILITY 2

ITEM4 ITEM4

2A 2B

Between-Item Within-Item
The Guidance and Counseling
Competency Test (GCCT)

a. A competency test for undergraduate and

graduate guidance and counseling & psychology
students.
b.A paper-and-pencil test in a multiple-choice
format.
c. Has six subtests corresponding to the six areas
covered by the licensure examination in guidance
and counseling.
d. Has a total of 50 items.
e. Designed to measure evidence of minimum
competency expected in a guidance counselor.
Development of the GCCT

DEVELOPMENT AND EXPERT

REVIEW OF LITERATURE/
SURVEY OF EXISTING VALIDATON OF THE INITIAL
LICENSURE/ACHIEVEMENT TEST ITEM POOL
IN GUIDANCE AND COUNSELING (100 ITEMS)

GCCT
FOR
IRT ANALYSIS
(50 ITEMS)
Participants:

Ninety-six (96) college and master’s level students from two

universities in Manila participated in this study
(M age=25.23 yr.); 64 (66.7%) were females, 32 (33.3%) were
males. All of the college level participants were psychology
majors while majority of the master’s level participants
were guidance and counseling majors; the others are from
allied courses (e.g. school psychology).
IRT ANALYSIS:
The program Construct Map, a graphical, menu-driven
software package combining a multidimensional IRT
engine for estimating item and person parameters
(Kennedy, Wilson, Draney, Tutunciyan, & Vorp, 2008), was
used in all IRT analysis.

In the unidimensional model, all 50 items of the GCCT were

assumed to load in one single latent dimension (composite
approach).

In the multidimensional model, the items were assigned into

one particular dimension corresponding to one of the
six subtests of GCCT (between-item multidimensionality).
FINDINGS:

1. The item response data was represented by the

multidimensional model better than the
unidimensional model.
MODEL FIT:

Deviance Number of
Parameters

Unidimensional 5,484.578 51

Multidimensional 5,444.657 71
ADDITIONAL ANALYSIS:

Given that the multidimensionality of the GCCT was

justified, the six subtest of the GCCT were
examined in terms of their item and ability
estimates.

Unidimensional modeling refers to estimation of

items that load to one specific dimension
independent of the other dimensions (consecutive
approach).

Multidimensional modeling of the six dimensions

where the dimensions are assumed to load to
one latent trait or competency.
FINDINGS:

1. The item response data was represented by the

multidimensional model better than the
unidimensional model.

2. The estimated difficulty of the items in each

dimension was similar regardless if the dimension
is considered as unidimensional or
multidimensional.
Comparison of Item Difficulty and Item Fit Statistics
Dimension Item Standard Item Fit No. of
Model Difficulty Error Misfits
Counseling
Unidimensional .03 .15 1.02 4
Multidimensional .03 .16 1.06 5
Group Process
Unidimensional -.10 .17 1.01 1
Multidimensional -.10 .17 .99 0
Testing
Unidimensional .11 .17 .95 2
Multidimensional .11 .17 .97 2
Foundations
Unidimensional .17 .15 .95 1
Multidimensional .18 .16 1.00 0
Career Guidance
Unidimensional .29 .17 .95 1
Multidimensional .31 .17 .99 1

Organization
Unidimensional .30 .19 1.01 1
Multidimensional .30 .17 .93 1
FINDINGS:

1. The item response data was represented by the

multidimensional model better than the
unidimensional model.

2. The estimated difficulty of the items in each

dimension was similar regardless if the dimension
is considered as unidimensional or
multidimensional.

3. The results obtained showed moderate to

substantial difference in the ability estimation of
several cases.
Comparison of the Unidimensional and Multidimensional
Ability Estimates of Selected Participants Across the 6 Dimensions.

Student 1 Student 2 Student 3

GCCT Raw UD MD Raw UM MD Raw UD MD

Score Score Score
Dimension

Counseling 12 .37 1.15 8 -.07 .15 2 -.89 -1.53

Group 5 -.36 -.02 2 -.99 -.88 4 -.56 -.52

Testing 4 -.54 -.31 3 -.87 -.57 4 -.54 -.31

Foundations 3 -.01 .32 4 .04 .58 4 .04 .89

Career 2 -.87 -.12 2 -.87 -.22 2 -.87 -.18

Organization 1 -2.23 -.82 1 -2.23 -.82 0 -2.39 -.99

CONCLUSIONS:

The analysis provided evidence on multidimensionality of

the GCCT.

The GCCT would benefit from structural validation (e.g.

Confirmatory Factor Analysis)

There were some observed differences in the outcomes of

the IRT analysis as a function of the type of modeling
approach (unidimensional vs multidimensional).

There is evidence that results of IRT analysis depend a lot

on the choice of model.