Psychotherapy Research

ISSN: 1050-3307 (Print) 1468-4381 (Online) Journal homepage: https://www.tandfonline.com/loi/tpsr20

The therapeutic factor inventory-8: Using

item response theory to create a brief scale
for continuous process monitoring for group
psychotherapy

Giorgio A. Tasca, Christine Cabrera, Elizabeth Kristjansson, Rebecca

MacNair-Semands, Anthony S. Joyce & John S. Ogrodniczuk

To cite this article: Giorgio A. Tasca, Christine Cabrera, Elizabeth Kristjansson, Rebecca MacNair-
Semands, Anthony S. Joyce & John S. Ogrodniczuk (2016) The therapeutic factor inventory-8:
Using item response theory to create a brief scale for continuous process monitoring for group
psychotherapy, Psychotherapy Research, 26:2, 131-145, DOI: 10.1080/10503307.2014.963729

To link to this article: https://doi.org/10.1080/10503307.2014.963729

Published online: 08 Oct 2014. Submit your article to this journal

Article views: 717 View related articles

View Crossmark data Citing articles: 13 View citing articles

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=tpsr20
Psychotherapy Research, 2016
Vol. 26, No. 2, 131–145, http://dx.doi.org/10.1080/10503307.2014.963729

EMPIRICAL PAPER

The therapeutic factor inventory-8: Using item response theory to

create a brief scale for continuous process monitoring for group
psychotherapy

GIORGIO A. TASCA1, CHRISTINE CABRERA2, ELIZABETH KRISTJANSSON2,

REBECCA MACNAIR-SEMANDS3, ANTHONY S. JOYCE4, & JOHN S. OGRODNICZUK5
1
Department of Psychology, The Ottawa Hospital – General Campus, Ottawa, ON, Canada; 2Department of Psychology,
University of Ottawa, Ottawa, ON, Canada; 3Department of Psychology, University of North Carolina at Charlotte,
Charlotte, NC, USA; 4Department of Psychiatry, University of Alberta, Edmonton, AB, Canada & 5Department of
Psychiatry, University of British Columbia, Vancouver, BC, Canada
(Received 14 November 2012; revised 29 July 2014; accepted 25 August 2014)

Abstract
Objective: We tested a very brief version of the 23-item Therapeutic Factors Inventory-Short Form (TFI-S), and describe
the use of Item Response Theory (IRT) for the purpose of developing short and reliable scales for group psychotherapy.
Method: Group therapy patients (N = 578) completed the TFI-S on one occasion, and their data were used for the IRT
analysis. Of those, 304 completed the TFI-S and other measures on more than one occasion to assess sensitivity to change,
concurrent, and predictive validity of the brief version. Results: Results suggest that the new TFI-8 is a brief, reliable, and
valid measure of a higher-order group therapeutic factor. Conclusion: The TFI-8 may be used for continuous process
measurement and feedback to improve the functioning of therapy groups.

Keywords: group psychotherapy; statistical methodology; test development; therapeutic factors; item response theory

The 99-item Therapeutic Factors Inventory (TFI;
Lese & MacNair-Semands, 2000; MacNair-Semands
& Lese, 2000) was initially developed to assess the
11 group psychotherapy factors defined by Yalom
(1995). These group therapeutic factors have been
described as the key mechanisms by which change
occurs in all types of therapy groups (Corsini &
Rosenberg, 1955; Yalom & Leszcz, 2005). How-
ever, the length of the original TFI was a barrier to
its widespread use (Roy, Turcotte, Montminy, &
Lindsay, 2005). The TFI short form (TFI-S) was
subsequently developed with 23 items (MacNair-
Semands, Ogrodniczuk, & Joyce, 2010) and later
with 19 items (TFI-19; Joyce, MacNair-Semands,
Tasca, & Ogrodniczuk, 2011) in order to improve
its reliability and validity, and to make it more
accessible. The TFI-S and TFI-19 had parallel four
factor solutions of instillation of hope, secure emo-
tional expression, awareness of relational impact, and
social learning. Joyce and colleagues reported good
evidence for predictive validity, as the TFI-19 scales
measured early in group therapy predicted treatment
outcomes at posttreatment. The authors also re-
ported good concurrent validity with a measure of
group engagement and good discriminant validity
evidenced by a low correlation with a scale of social
desirability (Joyce et al., 2011).
Researchers have argued that group therapeutic
factors can be represented by a higher-order single or
essential factor (Burlingame, Fuhriman, & Johnson,
2002). This single factor may represent expressed
emotion (Castonguay, Pincus, Agras, & Hines, 1998)

Correspondence concerning this article should be addressed to Giorgio A. Tasca, Department of Psychology, The Ottawa Hospital –
General Campus, 501 Smyth Road – Room 4428, Ottawa ON K1H 8L6, Canada. Email: gtasca@ottawahospital.on.ca
This article was originally published with errors. This version has been updated. Please see Corrigendum (http://dx.doi.org/10.1080/1050
3307.2014.981446).

© 2014 Society for Psychotherapy Research

132 G. A. Tasca et al.
or cohesion (Budman, Soldz, Demby, Davis, &
Merry, 1993; Burlingame et al., 2002) in groups. In
fact, although the construct validity of the four-
factor TFI-19 was supported by using confirmatory
factor analysis, the correlations among the four
TFI-19 factors were high, suggesting that a single
over-arching therapeutic factor construct may be
present (Joyce et al., 2011).
The goals of the current study were to (i) Develop
and test a very brief version of the TFI in order to
adapt the measure to make it more feasible for
clinical purposes or for research purposes in routine
clinical settings and (ii) describe the use of Item
Response Theory (IRT; Baker, 2001; Furr & Bachar-
ach, 2008) for the purpose of developing short and
reliable scales for group psychotherapy. Similar
pragmatic concerns motivated the development of
brief versions of individual therapeutic alliance mea-
sures such as the Agnew Relationship Measure-5
(ARM-5; Cahill et al., 2012). This trend is consistent
with recent approaches to repeatedly assess processes
and outcomes in psychotherapy in order to track
client progress and report these back to therapists
(e.g., Lambert & Shimokawa, 2011). Lambert and
Shimokawa completed a meta-analysis in which
they found that such feedback improves therapists’
responsiveness to clients, especially for the cases in
which clients are deteriorating. A similar argument
can be made for group psychotherapy processes.
If group therapists are continually aware, through
repeated assessment and feedback, of the state of
therapeutic factors in the group, then therapists
may be able to respond immediately to address any
problematic issues in the group. Such repeated
assessments would be simplified if a brief, valid, and
reliable measure of group psychotherapeutic factors
was available. Accordingly, a main goal of our study
was to use IRT analysis to develop a very brief TFI.
IRT Data Analysis
To achieve this with the TFI-S, we first reduced the
number of scale items by using an IRT (Baker, 2001;
Furr & Bacharach, 2008) approach. IRT comprises
modeling techniques that provide a great deal of
item-level information; it has many advantages over
classical test theory (CTT). For example, item dis-
crimination and difficulty parameters derived from
IRT analyses (described below) are unaffected by
sample variability. Furthermore, IRT allows re-
searchers to calculate standard errors of measure-
ment at each point along a latent trait scale so that
one can select items that provide maximum precision
at different levels of the trait (Embretson & Reise,
2000). IRT can be used to evaluate the properties of
an existing scale, to optimally shorten this scale when
necessary, and to evaluate the performance of this
reduced scale (Edelen & Reeve, 2007).
Although IRT models have a long history (e.g.,
Lord, 1953), their use in psychotherapy research is
relatively new (Doucette & Wolf, 2009). Our review
of the literature did not reveal any IRT-based study in
the group psychotherapy literature. As Doucette and
Wolf (2009) argued, psychotherapy researchers often
assume measurement precision rather than carefully
assessing the quality of measurements used to support
theories and decisions about psychotherapy.
CTT considers a test score to consist of the trait
level that is being measured and measurement error.
This is depicted in the following equation:
Observed score ¼ True score þ error:
IRT, on the other hand, looks at each individual
scale item and considers the level of the measured
trait, measurement error, and characteristics of the
item, including item discrimination, item difficulty,
and guessing (Furr & Bacharach, 2008). This addi-
tion of detailed item characteristic information sets
IRT apart from CTT.
The item characteristic curve is the “basis of IRT”
(Edelen & Reeve, 2007, p. 6). It characterizes the
relationship between the probability of correct
response on the item, the amount of the latent trait,
and the item characteristics. The equation below
depicts one of the most common IRT models, the
two parameter or 2PL model (Harris, 1989), which
is used with dichotomous (1/0) items.
e aðHbÞ
PðX ¼ 1Ih; a; bÞ ¼
1 þ e aðHabÞ
According to this model, the probability of a correct
response is dependent on the respondent’s underlying
ability and two item characteristics, item difficulty
and item discrimination. Item difficulty (β) is an
index of how high one’s trait level (Θ or theta) must
be to achieve a particular score on an item. The term
“difficulty” is used because IRT models were origin-
ally developed to assess educational abilities. There,
an item was “difficult” if a high level of educational
ability was required to answer the item. In our
context, difficulty refers the amount of the trait (e.g.,
group therapeutic factor) required to score highly on a
TFI item. In dichotomous items, the β parameter in an
IRT model represents the trait level necessary to
respond above a certain threshold with at least a .50
probability (Baker, 2001; Furr & Bacharach, 2008).
Item discrimination (α) is an index of how well the
item distinguishes between people with contiguous
trait levels, especially those who are high as opposed
to those who are low on a trait. For example, a highly
discriminatory item will differentiate a participant

who is high in a group therapeutic factor trait from a
participant who is moderate or low on that group
therapeutic factor trait. Item discrimination is ana-
logous to an item-factor loading or item-test correla-
tion in CTT. Hence, an item that has a high
discrimination value is a better indicator of the latent
trait. A scale with many items with low discrimina-
tion is likely to have inaccurate total scores (Baker,
2001; Furr & Bacharach, 2008).
Polytomous items have multiple response options.
Common polytomous IRT models include the
Nominal Model for nominal or nonordered
responses (Bock, 1972), the Partial Credit Model
(PCM; Masters 1982), the Generalized Partial
Credit Model (GPCM; Muraki, 1992), and the
Rating Scale Model (Andrich, 1982) for items which
may be ordered. Finally, the Graded Response
Model (GRM; Samejima, 1969) is designed for
items which are clearly ordered along a response
continuum (e.g., Likert items; Hays, Morales, &
Reise 2000; Ostini & Nering, 2006; Templin, n.d.).
The PCM produces a “dichotomous Rasch model
for each pair of adjacent item categories” (Ostini &
Nering, 2006, p. 34). The GPCM is an extension of
the PCM that allows discrimination to vary. The
Rating Scale Model is a version of the PCM for
items with the same format (Ostini & Nering, 2006;
Templin, n.d.). Currently, there are no well-known
1PL models for ordered polytomous data (Templin,
n.d.). The GRM for ordinal items assesses both
location (difficulty) and discrimination; it is regularly
used in practical testing situations as it is more stable
and has fewer data demands than other polytomous
models (Ostini & Nering, 2006; Templin, n.d.). We
selected the GRM because the items in the TFI
are clearly ordered as well as the aforementioned
advantages.
The GRM assesses the probability that the item
response will be in category k (i.e., representing a
response option) or higher. In this model, each item
has one discrimination and several (i.e., the number
of k response categories—1) β parameters, which are
termed “threshold” parameters. These thresholds
represent the place on the scale on which the pro-
bability of selecting a response is .50, compared to
all other responses that are ordinally higher (Van
Dam, Earleywine, & Borders, 2010).
Combining information from discrimination and
difficulty parameters allows one to assess an item’s
performance, or reliability, at different levels of the
trait. This is represented by the IRT concept of item
information which is graphically displayed on an
item information curve (IIC). These curves display
the range of trait levels over which the item provides
the most information (measures most reliably); items
can be very reliable at some levels of ability or trait
Psychotherapy Research 133
and less reliable at other levels (Edelen & Reeve,
2007). For example, an item may discriminate well
between people who have high levels of the group
therapeutic factor trait, but not discriminate as well
between people at the lower end of group thera-
peutic factor trait dimension. Furthermore, some
items provide more overall information than others.
IICs can be summed to produce an information
curve for the full scale called the test information
curve (TIC), which represents the relative precision
of a scale across different levels of the trait con-
tinuum. Such detailed and nuanced information
about items and tests represent a major advance
over CTT methods which only provide one estimate
of reliability for all ability levels.
Prior to conducting IRT analyses, two assump-
tions must be met: (i) The underlying latent trait
must be unidimensional, and (ii) at a given level of
Φ, or ability, the response of any one item must not
depend on the response to any other item (i.e., local
independence; DeMars, 2010).
As described above, one of the main purposes of
the current study was to develop and test a very brief
version of the TFI in order to adapt the measure for
clinical purposes or for research purposes in routine
clinical settings. Since Joyce and colleagues (2011)
suggested that the TFI-S factors could be repre-
sented by a single higher-order construct, we
hypothesized that the TFI scale could be reduced
to a small set of highly reliable items drawn from the
original TFI-S factors but presented as a unidimen-
sional scale.
Method
Participants
The IRT analysis was conducted on data combined
from three previously published studies that used
similar versions of the TFI scales (Joyce et al.,
2011; Lese & MacNair-Semands, 2000; MacNair-
Semands et al., 2010). For the multilevel modeling
(MLM) analyses to assess concurrent, discriminant,
and predictive validity, only data from the study
published by Joyce and colleagues were used. Parti-
cipants from the first study (Lese & MacNair-
Semands, 2000) were 77 undergraduate and gradu-
ate students who participated in counseling and
support groups at university counseling centers in
three major universities in the Northeast, the South-
east, and the Southwest of the USA; 59 were
women; 70 were White, 1 African-American, and 4
were Hispanic/Latino. The mean age was 25.50 years
(SD = 8.34). Groups included open-ended, struc-
tured, and support therapy groups. Participants from
the second TFI study (MacNair-Semands et al., 2010)
134 G. A. Tasca et al.
were 174 patients consecutively admitted to a day
treatment program (DTP) at a university hospital in
Edmonton, Canada. Their average age was 37.20
years (SD = 10.65). Most (65%) were women, 91%
were White, and 71% received a DSM-IV (Amer-
ican Psychiatric Association, 2000) Axis II disorder
diagnosis. The most prevalent Axis I disorder was
major depression (70%). Participants attended the
DTP all day, five days a week for 18 weeks. The
data were collected during the self-awareness group
within the DTP, a small, insight-oriented group
(approximately eight members) that individual
patients attended throughout their tenure in the
DTP (Piper, Rosie, Joyce, & Azim, 1996). Parti-
cipant data from the third TFI study (Joyce et al.,
2011) were collected at eight sites across North
America. These sites included two urban university
counseling centers (one including six groups with
43 participants; one including eight groups with 84
participants); outpatient services at three local
hospitals in Vancouver, Canada (one including
four groups with 53 participants; two including
two groups with 15 participants); an outpatient
clinic system in Calgary, Canada (six cohorts
including 35 participants); and an outpatient
binge-eating disorder treatment program in a gen-
eral hospital in Ottawa, Canada (12 groups with
103 participants). After exclusions due to non-
attendance and missing data, a total of 51 different
groups with 380 participants provided data. The
sample from Joyce et al.’s (2011) study included
267 women and 93 men (missing data on gender
for 20 members), with a mean age of 36.28 years
(SD = 13.6). Of the available sample, 85.5% were
Caucasian, 5.0% were African-American, 2.8%
were Asian/Asian-American, 1.7% were multi-
racial, 1.1% were Persian/Arabic, 1.4% were
Latino/Hispanic, 1.1% were East Indian, and
1.4% reported “other” ethnicity.
Measures
Therapeutic factors inventory—short form
(TFI-S). The 23-item TFI-S is a self-report meas-
ure designed to assess individual group members’
perceptions of four broad therapeutic factors:
Instillation of hope, secure emotional expression,
awareness of relational impact, and social learning
(MacNair-Semands et al., 2010). The TFI-S items
are rated on a 7-point Likert scale that ranges from 1
(Strongly Disagree) to 7 (Strongly Agree). In the
previous work, the four factors had coefficient alpha
values between 0.71 and 0.91 (MacNair-Semands
et al., 2010), indicating adequate internal consist-
ency of the scales (Clark & Watson, 1995).
Group climate questionnaire—Short form
(GCQ-S). The GCQ-S (MacKenzie, 1983) is a
self-report measure designed to assess individual
members’ perceptions of a group’s therapeutic
environment. The GCQ-S has 12 items rated on a
7-point Likert scale indicating extent of agreement
ranging from 0 (not at all) to 6 (extremely). The
items provide for the scoring of three subscales:
Engagement (five items), avoidance (three items),
and conflict (four items). The engagement scale was
used in this study to assess group cohesion. It
consists of items that call for ratings on the degree
of self-disclosure, cognitive understanding, and con-
frontation occurring in the group. In the present
study, based on administration of the measure at
Week 4 in all groups of Joyce et al.’s (2011) sample,
the engaged scale had a coefficient alpha of .74
(mean inter-item r = .41). The mean inter-item
correlation and coefficient alpha indicate adequate
internal consistency for the engaged scale (Clark &
Watson, 1995).
Brief symptom inventory (BSI). The BSI-18 is
an 18-item instrument that measures psychological
distress (Derogatis, 2000). The BSI-18 is composed
of six items each for the dimensions of somatization,
depression, and anxiety, scored on a Likert scale of
0–4 to reflect the degree of distress experienced
during the preceding two weeks. An overall score
representing general level of symptom distress (Glo-
bal Severity Index) was calculated. For Joyce et al.’s
(2011) sample at pretreatment, the Global Severity
Index had a coefficient alpha of .93.
Social desirability. We assessed social desirabil-
ity using the desirability scale of the Personality
Research Form (Jackson, 1984). The desirability
scale is a 16-item, true–false, self-report measure.
Higher scores reflect a greater tendency to provide
socially desirable responses. At pretreatment in Joyce
et al.’s (2011) study sample, coefficient alpha
was .73.
Procedure
After a description of the project by the research
coordinator at each site, participants provided writ-
ten informed consent to participate. The original
projects and data collection were approved by the
affiliated research ethics board at each site. Prior to
the start of their groups, participants completed a
demographics questionnaire. For data collected in
Lese and MacNair-Semands (2000) study, partici-
pants completed the TFI-S at one time point during
their attendance in their group. For MacNair-
Semands et al.’s (2010) study, patients completed

the TFI-S on three occasions—Weeks 4, 10, and 16
of the 18-week program. Ratings from Week 4 were
used for the data analysis. For data collected in Joyce
et al.’s (2011) study, the TFI-S and GCQ-S were
completed by most participants at Weeks 4, 8, and
12, but individuals at sites with rolling membership
(i.e., day hospitals) only completed the measures at
Weeks 4 and 12. Participants at all sites in Joyce
et al.’s (2011) study also completed the social
desirability and BSI-18 measures at pretreatment,
and the BSI-18 at postsession 12.
Checking Assumptions
PRELIS, a component of the Lisrel 8 program
(Mels, 2006), was used to examine the assumption
of unidimensionality1. To do this, we ran a principal
components analysis (PCA) on the 23-item scale and
checked the ratio of the first to second eigenvalue.
According to Lord (1980), this ratio should be at
least 3 to 1 in order for the scale to be considered
unidimensional. We also ran a PCA on the shor-
tened scale (TFI-8) developed after our first IRT
analysis. Finally, we ran a confirmatory factor ana-
lyses (CFA) comparing the model fit of the original
four-factor model (Joyce et al., 2011) to a model
with a higher-order single factor. Chi-square differ-
ence test, and the information criteria Bayesian
information criterion (BIC) and corrected Akaike
information criterion (CAIC) were used to assess the
comparative fit of the models.
The assumption of local independence for the
eight items on the TFI-8 was checked through an
analysis of the residual correlation matrix produced
as an output from the IRT-FIT program (Bjorner,
Smith, Stone, & Sun, 2007). This analysis is a test of
whether or not there are substantial dimensions
leftover after the IRT analysis is performed (Linacre,
2014).
IRT Analysis
The IRT analysis was conducted on all available TFI-
S data from Lese and MacNair-Semands (2000) and
MacNair-Semands et al.’s (2010) studies, and the
Session 4 TFI-S data from Joyce et al.’s (2011) study.
As TFI-S has clearly ordinal response categories, the
GRM was used for IRT analysis (Embretson & Reise,
2000; Samejima, 1969). The analysis produced one
item discrimination (α) and six item difficulty values
(β; k–1) for each item, and three graphics: an IIC and
an item category characteristic curve (ICC) for each
item, and the TIC for the scale as a whole. Generally,
items with discrimination values (α) ranging from
0 to .24 are considered to be very poorly discrim-
inating, those with values of .25–.64 are poorly
Psychotherapy Research 135
discriminating, those with values of .65–1.34 are
moderate, items with values of 1.35–1.69 are highly
discriminating, and items with values above 1.70 are
very highly discriminating (Baker, 2001). Typically
item difficulty values (β) range between –3 and +3,
with values rising from the lowest to the highest
response category within an item indicating a greater
amount of latent trait needed to endorse a higher-level
response category (i.e., higher levels of the underlying
trait should correspond to endorsing “7” as opposed to
“4” on the TFI; DeMars, 2010). An ICC depicts the
level on the trait scale where a specific item is most
reliable. The ICC graphically depicts the relationship
between the particular response to an item and the
latent trait, each potential response being contingent
on trait level, item discrimination, and item difficulty
(Baker & Kim, 2004). The TIC indicates whether the
scale is more reliable at certain levels of the latent trait
(in this case, group therapeutic factor) and depicts the
degree of measurement error across the scale. Also,
the output generates a marginal reliability index (MRI)
that serves as the IRT-based measure of internal
consistency (similar to Cronbach’s alpha in CTT;
Florida Department of Education, 2005). As in CTT,
a minimal acceptable level for the MRI is 0.80
(de Ayala, 2009).
IRT analysis with this data-set was conducted
using the MULTILOG 7.03 program (Thiessen,
1991). Maximum likelihood methods were used to
estimate item discrimination (α) and item difficulty
(β) parameters. For further information regarding
the item response procedure used by the program,
see the MULTILOG manual (Thiessen, 1991).
Multilevel Modeling
First, we tested the group effect with the intra-class
correlation coefficient (ρ) to evaluate dependence in
the hierarchically nested data with a hierarchical
linear modeling (HLM) approach using methods
described by Tasca, Illing, Joyce, and Ogrodniczuk
(2009). To assess sensitivity to change of the new
brief TFI scale, data were analyzed from Weeks 4, 8,
and 12 of each group from Joyce et al.’s (2011) study
(N = 304 individuals and 47 groups with available
data). To evaluate the degree of change across
assessments while accounting for group dependen-
cies in the data, a three-level longitudinal HLM
approach was undertaken (Tasca et al., 2009). One
advantage of HLM is that all the data are used, i.e.,
cases were not deleted if ratings from an assessment
were missing. Level 1 of the model represented the
repeated measurement of the TFI scale scores across
the three time points within each individual group
member. Level 2 represented estimated initial scores
(intercepts) and rates of change (slopes) of
136 G. A. Tasca et al.
individuals nested within the distinct therapy groups
in the sample. Level 3 represented intercepts and
slopes for the distinct groups themselves. The time
parameter at Level 1 was log transformed to model a
more pronounced change in TFI-8 scores from
Sessions 4–8, and less pronounced change from
Sessions 8–12. To assess if change in TFI-8 scores
were related to change in symptoms, we used BSI-18
residual change scores calculated by regressing BSI-
18 scores at Session 12 on to BSI scores at pretreat-
ment. BSI-18 residual change scores were entered as
group-centered predictors at Level 2, and their
group level relationship with TFI-8 slopes was
examined at Level 3 of the model. Models were
developed and effect sizes (pseudo-R2) were assessed
by adding predictors in sequential models (Tasca
et al., 2009). Appendix 1 shows the full three-
level HLM.
To assess predictive validity, the new brief TFI at
Week 4 was evaluated as a predictor of post-treat-
ment status on the BSI-18 outcome variable. Status
at Week 12 (post-treatment) was assessed with the
BSI-18, while controlling for status at Week 4 (i.e.,
BSI-18 at baseline). A Level 2, hierarchically struc-
tured HLM approach (participants nested within
groups) was employed (Tasca et al., 2009). Error
terms for the intercept coefficients at Level 2 (group
level) were left free to vary. Appendix 1 also shows
the HLM for this two-level model.
Results
The TFI-S was administered to 621 therapy group
participants from three studies (Joyce et al., 2011;
Lese & MacNair-Semands, 2000; MacNair-Semands
et al., 2010); 578 of these provided sufficient data on
the TFI-S. This instrument was originally composed
of 23 items; however, only 22 items were examined
in this study. Item 20 (“This group helps empower
me to make a difference in my own life”) from the
instillation of hope therapeutic factor was removed
from the IRT analysis due to insufficient cases caused
by a clerical error in one of the study samples.
Checking Assumptions for IRT
Unidimensionality. Joyce and colleagues (2011)
proposed the TFI-S as a measure of four latent
therapeutic factors, which may preclude unidimen-
sionality. However, the estimates provided by Joyce
and colleagues (2011) for the final structural model
of the scales revealed high correlations among the
four factors, suggesting a common higher-order
construct. First, we conducted a nonlinear PCA to
test the unidimensionality of the TFI-S (Hambleton
& Rovinelli, 1986; Hattie, 1984, 1985) using
PRELIS from the LISREL 8 program (Du Toit,
Du Toit, Mels, & Cheng, n.d.). We found that the
first component accounted for 46.32% of the vari-
ance in the scores, which was larger than the
recommended criterion of 20% (Reckase, 1979).
The second component was substantially smaller
than the first, accounting for only 10.29% of the
variance. In our analyses, the first eigenvalue (36.7)
had a ratio of 4:1 to the second eigenvalue (8.19),
suggesting that there was one dominant dimension
(Lord, 1980). A test of the 8-item scale also showed
unidimensionality; the ratio of the first (14.5) to the
second eigenvalue (3.42) was also 4:1. Second, we
compared a model with a higher-order single factor
by using CFA to the original four-factor model
tested by Joyce et al. (2011). The difference in chi-
square test between the two CFA models was not
significant, Δχ2(2) = 2.25, p = .33, however, the
information criteria BIC (1174.51) and CAIC
(1224.51) for the single-factor higher-order model
were smaller than the respective BIC (1183.98) and
CAIC (1235.98) for the four-factor model. Taking
all of this evidence together suggests that the TFI
can be represented by a single higher order factor.
Local independence. The items of the TFI-8
were tested for local independence using the residual
correlation matrix produced by IRT-FIT.1 First, we
assessed the size of the correlations; the largest one
was .25, which is small using Cohen’s criteria.
Second, we ran a linear factor analysis on the residual
correlation matrix using SPSS. Linacre (2009) sug-
gested that an eigenvalue of 2.00 or more indicates
local dependence. The largest eigenvalue was 1.44,
which was well below Linacre’s criterion of 2.00.
Third, following Bjorner’s suggestion (personal com-
munication), we ran a one-factor confirmatory factor
analysis with M-PLUS using unweighted least squares
estimation. The CFA results also supported the
hypothesis of local independence. The comparative
fit index (CFI) was 0.56 and the Tucker-Lewis index
(TLI) was 0.39, which are well below the currently
recommended cut-off of 0.95 (Hooper, Coughlan, &
Mullen, 2008). Furthermore, the loadings on the
factor were relatively low, ranging from 0.036 to
0.377. The RMSEA, however, was 0.057, which is
just below the recommended cut-off of 0.06 for good
fit. Altogether, the results indicated that there was not
an important dimension remaining after using IRT.
IRT Analysis
TFI-S. Discrimination values for all items are
presented in Table I. The discrimination parameters
ranged from α = 0.63 to α = 2.27. Item 15 had poor
discrimination; items 1, 3, 4, 5, 7, 8, 13, 21, and 23
 Psychotherapy Research 137
Table I. Discrimination values of 22-item of the TFI-S.

Standard
Item (and group therapeutic factor) Discrimination Error

1. Because I have got a lot in common with other group members, I am starting to think that I may have 1.48 0.16
something in common with people outside group too. (Social Learning)
2. Things seem more hopeful since joining group. (Instillation of Hope) 1.98 0.18
3. I feel a sense of belonging in this group. (Secure Emotional Expression) 2.08 0.17
4. I find myself thinking about my family a surprising amount in group. (Awareness of Relational Impact) 0.92 0.15
5. Some times I notice that in group I have the same reactions or feelings as I did with my sister, brother, 0.74 0.12
or a parent in my family. (Social Learning)
6. In group I have learned that I have more similarities with others than I would have guessed.(Instillation 1.83 0.20
of Hope)
7. It is okay for me to be angry in group. (Secure Emotional Expression) 0.92 0.14
8. In group I have really seen the social impact my family has had on my life. (Awareness of Relational 1.2 0.17
Impact)
9. My group is kind of like a little piece of the larger world I live in: I see the same patterns, and working 1.77 0.19
them out in group helps me work them out in my outside life. (Social Learning)
10. Group helps me feel more positive about my future. (Instillation of Hope) 2.88 0.23
11. It touches me that people in group are caring of each other. (Secure Emotional Expression) 1.63 0.23
12. I pay attention to how others handle difficult situations in my group so I can apply these strategies in my 1.76 0.20
own life. (Awareness of Relational Impact)
13. In group sometimes I learn by watching and later imitating what happens. (Social Learning) 0.95 0.13
14. This group helps me recognize how much I have in common with other people. (Instillation of Hope) 2.35 0.23
15. In group, the members are more alike than different from each other. (Secure Emotional Expression) 1.19 0.14
16. It is surprising, but despite needing support from my group, I have also learned to be more self- 1.73 0.16
sufficient. (Awareness of Relational Impact)
17. This group inspires me about the future. (Instillation of Hope) 2.70 0.22
18. Even though we have differences, our group feels secure to me. (Secure Emotional Expression) 1.85 0.22
19. By getting honest feedback from members and facilitators, I have learned a lot about my impact on 1.81 0.18
other people. (Awareness of Relational Impact)
21. I get to vent my feelings in group. (Secure Emotional Expression) 1.49 0.16
22. Group has shown me the importance of other people in my life. (Awareness of Relational Impact) 2.12 0.18
23. I can “let it all out” in my group. (Secure Emotional Expression) 1.48 0.15

Note. Item 20 data were not available due to clerical error in one of the samples.

had moderate discrimination; items 2, 6, 11, 12, 16,
18, 19, and 22 had high discrimination; and items 9,
10, 14, and 17 were very highly discriminatory.
Notably, all of the factors had some items that were
moderately discriminatory with the exception of the
instillation of hope factor. This factor had a majority
of highly discriminatory items (three of the five items
for this factor).
On the basis of this original analysis, eight items
were selected for further IRT analysis using a
polytomous GRM (Embretson & Reise, 2000;
Samejima, 1969). The selected items were the two
most discriminatory items from each of the four
therapeutic factors. Consequently, the conceptual
basis of the TFI-S was preserved while the discrim-
inatory capacity was maximized.
TFI-8. Residual correlation values for the eight
items appear in Table II. Discrimination values
from a separate run for the final eight items ranged
from α = 1.19 to α = 2.66 (see Table III). Items 1, 6,
9, and 18 were highly discriminatory and items 3,
12, 17, and 19 were very highly discriminatory.
Table IV contains the difficulty values (β) for the
final eight items. Across all items, increasing levels
of the underlying trait were associated with choosing
a higher-level response category (e.g., a participant
perceiving more of the group therapeutic factor was
more likely to choose a “6” over a “3” response).
Within each item, the distance between the highest
and lowest difficulty value was between 3.6 and 5.3
units, indicating that response categories were
spread reasonably well across the trait range. How-
ever, there were somewhat more difficulty values
around and below the midpoint (0) of the group
therapeutic factor range, so this scale will measure
average and lower levels of the group therapeutic
factor particularly well.
For illustration purposes, Figure 1 displays the
ICCs and IICs for items 3 (α = 1.85) and 18 (α =
1.42). In an ICC, the x-axis represents the trait level
(i.e., the group therapeutic factor) and the y-axis
represents the probability of a response. The average
amount of latent trait is set at 0, with a standard
deviation of 1. When an item is more discriminat-
ory, the slope of the response curves is quite steep
while those of less discriminatory items slope more
gradually (DeMars, 2010). The ICC graph for item
138 G. A. Tasca et al.
Table II. Residual correlation values from the IRT analysis of the TFI-8.

Item TFI-1 TFI-3 TFI-6 TFI-9 TFI-12 TFI-17 TFI-18

TFI-1 1
TFI-3 0.05 1
TFI-6 0.19 0.06 1
TFI-9 0.11 –0.02 0.09 1
TFI-12 –0.12 –0.13 –0.09 –0.07 1
TFI-17 0.03 0.02 0.04 –0.02 –0.09 1
TFI-18 0.01 0.25 0.03 –0.01 –0.01 0.07 1
TFI-19 –0.01 –0.05 –0.02 0.05 –0.06 0.04 0.06

3 depicts steeper slopes for all potential item
responses (i.e., ranging from 1 “Strongly Disagree”
to 7 “Strongly Agree”) compared to the more
gradual slopes of the item response curves for item
18. On the IICs, the x-axis represents the trait level
and the y-axis represents the reliability of score
information gathered by the specific item at distinct
levels of the trait. Item 18 functions across the entire
range of the trait. Item 3 functions more reliably
than item 18; however, not at the highest level of the
trait. Correspondingly, the final difficulty parameter
for item 18 is β6 = 2.47, indicating that a higher level
of the trait corresponds with a 50% chance of
endorsing this response. ICC and IICs for all other
items are presented in Appendix 2.
The graph of the TIC for the TFI-8 is presented in
Figure 2. The x-axis represents the trait level (group
therapeutic factor) and the y-axis represents the
reliability of score information that has been gath-
ered by the test items at particular levels of the latent
group therapeutic factor trait. In Figure 2, the solid
line represents the test information and the hatched
line represents the standard error of measurement of
the latent trait. The average amount of latent trait is
set at 0 on the x-axis, with a standard deviation
of 1 (DeMars, 2010). The TIC is a broad-based
unimodal curve that indicates that the TFI-8 scale is
reliable across an extensive range of the latent trait,
including for those who demonstrate below average
and average levels of the group therapeutic factor. The
TFI-8 is less sensitive at high levels of the latent trait
(>2 SD above the mean trait level), as indicated by the
steep downward slope at the high end of the x-axis
(Figure 2). Finally, the MRI for the TFI-8 was found
to be 0.88 indicating acceptable internal consistency.
Hierarchical Linear Modeling
The intra-class correlation coefficient (ρ; Tasca et al.,
2009) indicated that 12% of the variance in TFI-8
scores and 9% of the variance in TFI-8 slopes were
attributable to the group effect. These results suggest
that the use of three-level HLM to analyze these data
was appropriate.
Sensitivity to change. The item means and
standard deviations for the TFI-8 scale scores were
as follows: At Week 4, M = 4.71, SD = 1.06, n =
312; at Week 8, M = 5.10, SD = 1.02, n = 186;
and at Week 12, M = 5.19, SD = 1.17, n = 226.
Due to differences in data collection across sites
(e.g., certain groups collected TFI-S ratings only at

Table III. TFI-8 items with discrimination and standard error.

Discrimination Standard
Item (and original TFI-S factor) α error

1. Because I’ve got a lot in common with other group members, I’m starting to think that may have 1.19 0.36
something in common with people outside group too. (Social Learning)
3. I feel a sense of belonging in this group. (Secure Emotional Expression) 1.85 0.14
6. In group I’ve learned that I have more similarities with others than I would have guessed. (Instillation 1.47 0.37
of Hope)
9. My group is kind of like a little piece of the larger world I live in: I see the same patterns, and working 1.64 0.13
them out in group helps me work them out in my outside life. (Social Learning)
12. I pay attention to how others handle difficult situations in my group so I can apply these strategies in my 2.66 0.52
own life. (Awareness of Relational Impact)
17. This group inspires me about the future. (Instillation of Hope) 2.17 0.16
18. Even though we have differences, our group feels secure to me. (Secure Emotional Expression) 1.42 0.37
19. By getting honest feedback from members and facilitators, I’ve learned a lot about my impact on other 1.75 0.14
people. (Awareness of Relational Impact)
 Psychotherapy Research 139
Item characteristic curve Item information curve
1.0 2.5

0.8 2.0

Information
Probability

0.6 1.5

1
0.4 2 3 1.0
4
5

0.2 6 0.5
7

0 0
-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3
Ability Scale score
Item characteristic curve Item information curve
1.0 2.5

0.8 2.0

1
1.5
Information

0.6
Probability

0.4 2 1.0

4
0.2 3 0.5
5 6 7

0 0
-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3

Ability Scale score

Figure 1. IICs (right) and item response category characteristic curve (left) for items 3 (top) and 18 (bottom).

Week 4 and at Week 12, while other groups collected These findings suggest that the TFI-8 is sensitive to
ratings at each of the three assessments), missing change across time.
data, and treatment noncompletion, the individual
n available at each assessment varied. As indicated, Predictive validity. To assess predictive validity,
one advantage of HLM is that reliable parameters can a two-level random effects HLM was used
be estimated even with missing data. The three-level (Appendix 1). Greater TFI-8 scores at Week 4
HLM models of change appear in Appendix 1. significantly predicted reductions in the BSI-18
Results indicated that there was a significant increase General Severity Index, γ20 = –0.11, t(40) = 3.54,
in TFI-8 scores from Week 4 to Week 8 that was p < .001, accounting for 42% of the symptom
maintained to Week 12, γ100 = 0.92, t(45) = 7.82, outcome variance.
p < .001, and the size of the effect was large, pseudo-
R2 = .33. Further, BSI residual change scores were Concurrent and discriminant validity. Con-
negatively related to the significant slope in TFI-8 current validity was assessed by correlating the TFI-
scores over time, γ120 = –0.74, t(45) = 3.01, p = .00, 8 given at Week 4 from Joyce et al.’s (2011) sample
pseudo-R2 = .11. This result indicates that at the with available data (n = 300) from the Engaged
group level a greater increase in therapeutic factors GCQ scale. As expected, there was a large positive
during therapy was associated with better symptom relationship between the TFI-8 and the engaged
outcomes, and conversely a lesser increase in TFI-8 scale, r = .55, p < .001. We also found evidence of
was associated with poorer symptom outcomes. discriminant validity in that the correlation between
140 G. A. Tasca et al.
10 0.61
9

8 0.49

Standard error
6 0.37

Information
5

4 0.25

2 0.12

0 0
-3 -2 -1 0 1 2 3
Scale score

Test information curve: solid line Standard error curve: dotted line

Figure 2. TIC of Therapeutic Factor Inventory–8.

the TFI-8 and the social desirability scale was small,
r = .18, p = .003.
Discussion
The major goal of this study was to reduce the length
of the TFI-S in a psychometrically sound manner in
order to make it even more efficient for clinical and
research use in practice settings and to assess the
shortened scale’s reliability and validity. The brief
TFI-8 items demonstrated very good to excellent
item discrimination. Although the TFI-8 performed
well overall, the TIC indicated that the measure was
better at assessing mid- and lower-levels of the group
therapeutic factor. The TFI-8 had very good mar-
ginal internal consistency, indicating that the items
measure the group therapeutic factor construct
reliably.
The TFI-8 also demonstrated good concurrent
validity by correlating highly with the engaged scale
of the GCQ-S. This suggests that the TFI-8 is
closely related to group cohesion, but does not
completely overlap the cohesion construct, thus
representing a related but separate group therapeutic
factor. The TFI-8 is not greatly affected by social
desirability, which supports the TFI-8’s discriminant
validity. We made efforts to use both the IRT
method and to select items that reflected each of
the four original factors from the TFI-S so that not
only does the TFI-8 perform well psychometrically
but it also continues to cover the content of all
aspects of the original TFI-S scale constructs.
Hence, we conceptualize the single TFI-8 factor as
feeling hopeful about the processes of emotional
expression and relational awareness, which then
translate into and promote social learning.
A key finding of this study was that assessment of
the group therapeutic factor by the TFI-8 early in
group therapy was associated with positive change
in general symptom severity, and the effect was large.
This lends support to theTFI-8’s predictive validity
and to the use of the TFI-8 as a tool for regular
feedback to therapists about the ongoing function-
ing of a group. The TFI-8 changed over time so that
most groups felt more hopeful about emotional
expression leading to interpersonal learning. In
addition, greater change in the group therapeutic
factor was associated with greater positive changes in
symptoms, and conversely, less positive change in
the TFI-8 was associated with poorer outcomes.

Table IV. Difficulty values for TFI-8.

Item β1 β2 β3 β4 β5 β6

1 –2.85 (0.88) –1.76 (0.61) –0.7 (0.34) 0.54 (0.32) 1.13 (0.41) 2.39 (0.73)
3 –2.69 (0.25) –1.86 (0.15) –1.05 (0.10) –0.25 (0.08) 0.72 (0.09) 1.62 (0.14)
6 –3.03 (0.89) –1.48 (0.43) –1.22 (0.38) –0.19 (0.23) 0.85 (0.30) 2.24 (0.54)
9 –3.19 (0.33) –2.23 (0.21) –1.44 (0.14) –0.72 (0.10) 0.24 (0.09) 1.41 (0.14)
12 –2.15 (0.41) –1.47 (0.29) –0.82 (0.20) –0.03 (0.17) 0.45 (0.17) 1.40 (0.27)
17 –2.70 (0.25) –2.03 (0.16) –1.49 (0.12) –0.90 (0.08) –0.02 (0.07) 0.98 (0.09)
18 –2.18 (0.65) –0.93 (0.37) –0.27 (0.29) 0.92 (0.32) 1.68 (0.43) 2.47 (0.70)
19 –2.86 (0.28) –1.91 (0.17) –1.27 (0.12) –0.32 (0.08) 0.58 (0.09) 1.48 (0.41)

One of our motivations for reducing the length
of the TFI-S was to make the TFI-8 more user-
friendly for use in busy clinical settings. The TFI-8
could help to flag problematic group therapy pro-
cesses that are prognostic of poorer treatment
outcomes among group participants. The procedure
of frequently assessing outcomes and providing
feedback to therapists can reduce the number of
patients who deteriorate in the context of individual
therapy (Lambert & Shimokawa, 2011). Repeated
assessments with the TFI-8 may also allow the group
therapist to immediately identify a rupture in the
relationship with the group or an individual that
requires a specific intervention in order to repair
group functioning (Safran, Muran, & Eubanks-
Carter, 2011). For frequent and repeated administra-
tions, the TFI-8 represents an efficient use of
resources, and it may function in clinical and research
contexts to identify problematic group sessions or
participants who are not benefiting from the group.
This may be particularly useful for continuous feed-
back to group therapists in training to supplement
supervisory discussions.
There are now a few user-friendly descriptions of
the IRT method (e.g., Doucette & Wolfe, 2009) and
demonstrations of the use of IRT for scale develop-
ment (e.g., Fraley, Waller, & Brennan, 2000). There
are also relatively easy-to-use computer programs to
conduct IRT, such as the MULTILOG program
(Thiessen, 1991). Despite this, we could not find
any published examples of the application of IRT to
develop group therapy scales. IRT is an important
method for developing and testing group psycho-
therapy measurement tools in order to make meas-
urement more precise and reliable. As demonstrated
here, IRT is also an excellent method of reducing
longer unidimensional scales to a more concise
measure consisting of the best functioning items. In
the case of the TFI-8, the scale is particularly good at
discriminating among those with low-to-moderate
levels of the factor and is less effective at the higher
end of the group therapeutic factor dimension.
Hence, the TFI-8 may be most useful for newer
groups or groups working toward achieving an
adequate level of group therapeutic factor and may
not work as well for well-functioning groups who
have achieved a high level of group hopefulness,
interpersonal learning, and emotional awareness and
expression.
This study is consistent with recent research that
reflects a move away from highly overlapping factors
that are broad and unevenly balanced to a single- or
higher-order conceptualization of a group thera-
peutic mechanism. One of the implications is that
trainees who are taught about the 11 therapeutic
factors as delineated by Yalom and Leszcz (2005)
Psychotherapy Research 141
need education to begin to intentionally facilitate the
overall factor of hope, emotional expression, aware-
ness of relational impact, and social learning. Clin-
ically, this may require group therapists to highlight
and reflect upon these factors when they emerge in
order to improve therapist skills and create better
patient retention and outcome.
Limitations
This study had some limitations. First, the samples
were drawn from groups in a variety of settings,
including day hospitals, outpatient departments,
counseling centers, and community agencies. Nev-
ertheless, these may not represent the full range of
groups offered in the community, nor are the
individuals entirely representative of all those who
attend intervention groups, and thus the results may
be limited in generalizability by the sample of
groups. However, the number and variety of groups
represented in this study is larger and more varied
than is common for group therapy research, there-
fore, enhancing the ecological validity of this study.
Second, sample participants were predominantly
White, which may have resulted in biased ratings
and may also reduce generalizability. Replicating the
results with a more diverse sample would enhance
the generalizability to individuals from minority
groups. Third, the IRT analyses were conducted on
individual-level data, but the data were nested within
therapeutic groups. Nested data may result in
dependence in the data and inflated Type I error.
We used HLM to model change in the outcomes at
the group level for this reason. IRT assumes that the
item responses are solely dependent on the underly-
ing trait. Therefore, it is possible that the therapeutic
group membership may be influencing responses on
the TFI-8 (Fox, 2007). For this reason, the IRT
results may need to be interpreted with some
caution. The HLM results for the sensitivity to
change and predictive validity analyses of the TFI-8
were conducted at the group level, and so those
results are likely reliable. Fourth, unfortunately, one
item from the original TFI-S was not available for all
samples and so this may have limited the pool of
items representing the TFI construct. However, the
item was missing from the TFI-S hope factor, which
has several items with excellent ICC values. Finally,
although shorter tests provide practical advantages
over longer tests especially for continuous monitor-
ing, some have argued that shorter tests may be less
reliable due to larger measurement error (Emons,
Sijtsma, & Meijer, 2007). This problem may be
mitigated somewhat with the TFI-8 as only items
with optimal discriminatory and difficulty values
were selected. Nevertheless, in contexts where the
142 G. A. Tasca et al.
TFI-8 might be used for clinical decision-making
purposes, we suggest that the TFI-8 be combined
with other sources of information to improve reliab-
ility of the decisions.
Conclusions
The use of very brief and reliable repeated measure-
ments in clinical and research contexts is growing due
to the benefits of systematic feedback to therapists
about their clients’ functioning and the status of
group processes and outcomes. IRT allows the group
psychotherapy researcher to scrutinize the quality of
their measurements and potentially to reduce longer
unidimensional scales to more practical lengths. IRT
is a psychometric approach that is only recently being
used in individual psychotherapy research (Doucette
& Wolf, 2009), and this study is the first to use IRT in
a group therapy context. IRT may result in better
measurements of group therapy processes and out-
comes with the goal to reduce measurement error,
increase precision in defining a construct that sup-
ports a theory, and inform clinical decisions regard-
ing the functioning of a group. Results from this study
using IRT suggest that the new TFI-8 is a brief,
reliable, and valid measure of a higher-order group
therapeutic factor. The TFI-8 may be used for
continuous process measurement and feedback to
improve the functioning of therapy groups.
Acknowledgement
This study was supported by a grant from the Group
Psychotherapy Foundation and by a grant from the
Canadian Institutes for Health Research. Giorgio A.
Tasca holds the Research Chair in Psychotherapy
Research, University of Ottawa and the Ottawa
Hospital.
Note
1 989X.12.1.105
This was done as the TFI-8 is the final scale and thus it should
show no local dependence.
References
American Psychiatric Association. (2000). Diagnostic and statistical
manual of mental disorders (4th ed., text rev.). Washington, DC:
Author.
Andrich, D. (1982). An extension of the Rasch model for ratings
providing both location and dispersion parameters. Psychome-
trika, 47(1), 105–113. doi:10.1007/BF02293856
Baker, F. (2001). The basics of item response theory. College Park,
MD: ERIC Clearinghouse.
Baker, F. B., & Kim, S.-H. (2004). Item response theory: Parameter
estimation techniques (2nd ed.). New York, NY: Dekker.
Bjorner, J., Smith, K., Stone, C., & Sun, X. (2007). IRT-FIT: A
macro for item fit and local dependence tests under IRT models.
Lincoln, RI: Quality Metrics.
Bock, R. D. (1972). Estimating item parameters and latent ability
when responses are scored in two or more nominal categories.
Psychometrika, 37(1), 29–51. doi:10.1007/BF02291411
Budman, S. H., Soldz, S., Demby, A., Davis, M., & Merry, J.
(1993). What is cohesiveness? An empirical examination Small
Group Research, 24(2), 199–216. doi:10.1177/10464964932
42003
Burlingame, G. M., Fuhriman, A., & Johnson, J. E. (2002).
Cohesion in group psychotherapy. In J. C. Norcross (Ed.),
Psychotherapy relationships that work (pp. 71–88). New York,
NY: Oxford University Press.
Cahill, J., Stiles, W. B., Barkham, M., Hardy, G. E., Stone, G.,
Agnew-Davies, R., & Unsworth, G. (2012). Two short forms of
the Agnew Relationship Measure: The ARM-5 and ARM-12.
Psychotherapy Research, 22(3), 241–255. doi:10.1080/105033
07.2011.643253
Castonguay, L., Pincus, A., Agras, W., & Hines, C. (1998). The
role of emotion in group cognitive-behavioral therapy for binge
eating disorder: When things have to feel worse before they get
better. Psychotherapy Research, 8, 225–238. doi:10.1080/105033
09812331332327
Clark, L. A., & Watson, D. (1995). Constructing validity: Basic
issues in objective scale development. Psychological Assessment,
7, 309–319. doi:10.1037/1040-3590.7.3.309
Corsini, R. J., & Rosenberg, B. (1955). Mechanisms of group
psychotherapy: Processes and dynamics. The Journal of Abnor-
mal and Social Psychology, 51, 406–411. doi:10.1037/h0048439
de Ayala, R. J. (2009). The theory and practice of Item response
theory. New York, NY: Guilford.
DeMars, C. (2010). Item response theory: Understanding statistics
New York, NY: Oxford University Press.
Derogatis, L. R. (2000). Brief symptom inventory 18: Administra-
tion, scoring, and procedures manual. Minneapolis, MN: National
Computer Systems Pearson.
Doucette, A., & Wolf, A. W. (2009). Questioning the measure-
ment precision of psychotherapy research. Psychotherapy
Research, 19, 374–389. doi:10.1080/10503300902894422
Du Toit, S., Du Toit, M., Mels, G., & Cheng, Y (n.d.). LISREL
for windows – PRELIS user’s guide. Lincolnwood, IL: Scientific
Software International.
Edelen, M. O., & Reeve, B. B. (2007). Applying item response
theory (IRT) modeling to questionnaire development, evalu-
ation, and refinement. Quality of Life Research, 16(S1), 5–18.
doi:10.1007/s11136-007-9198-0
Embretson, S. E., & Reise, S. (2000). Item response theory for
psychologists. Mahwah, NJ: Erlbaum Publishers.
Emons, W. H. M., Sijtsma, K., & Meijer, R. R. (2007). On the
consistency of individual classification using short scales.
Psychological Methods, 12(1), 105–120. doi:10.1037/1082-
Florida Department of Education. (2005). FCAT handbook –
A resource for educators. Tallahassee, FL: Harcourt.
Fox, J. P. (2007). Multilevel IRT modelling in practice with the
package mlirt. Journal of Statistical Software, 20(5), 1–16.
Fraley, R. C., Waller, N. G., & Brennan, K. A. (2000). An item
response theory analysis of self-report measures of adult
attachment. Journal of Personality and Social Psychology, 78,
350–365. doi:10.1037/0022-3514.78.2.350
Furr, R., & Bacharach, V. (2008). Item response theory and Rasch
models. In Authors (Ed.), Psychometrics: An introduction.
Thousand Oaks, CA: Sage.
Hambleton, R. K., & Rovinelli, R. J. (1986). Assessing the
dimensionality of a set of test items. Applied Psychological
Measurement, 10, 287–302. doi:10.1177/014662168601000307
Harris, D. (1989). Comparison of 1-, 2-, and 3-parameter IRT
models. Educational Measurement: Issues and Practice, 8(1),
35–41. doi:10.1111/j.1745-3992.1989.tb00313.x

Hattie, J. (1984). An empirical study of various indices for
determining unidimensionality. Multivariate Behavioral
Research, 19(1), 49–78. doi:10.1207/s15327906mbr1901_3
Hattie, J. (1985). Methodology review: Assessing unidimension-
ality of tests and itenls. Applied Psychological Measurement, 9(2),
139–164. doi:10.1177/014662168500900204
Hays, R., Morales, L., & Reise, S. (2000). Item response theory
and health outcomes measurement in the 21st century. Medical
Care, 38(9 suppl), 1128–1142.
Hooper, D., Coughlan, J, & Mullen, M. R. (2008). Structural
equation modelling; Guidelines for determining model fit.
Electronic Journal of Business Research Methods, 6, 53–60.
Jackson, D. N. (1984). Personality research form manual (3rd ed.).
Port Huron, MI: Research Psychologists.
Joyce, A. S., MacNair-Semands, R., Tasca, G. A., & Ogrodnic-
zuk, J. S. (2011). Factor structure and validity of the
Therapeutic Factors Inventory–Short Form. Group Dynamics:
Theory, Research, and Practice, 15, 201–219. doi:10.1037/
a0024677]
Lambert, M. J., & Shimokawa, K. (2011). Collecting client
feedback. Psychotherapy, 48(1), 72–79. doi:10.1037/a0022238
Lese, K. P., & MacNair-Semands, R. R. (2000). The therapeutic
factors inventory: development of a scale. Group, 24, 303–317.
doi:10.1023/A:1026616626780
Linacre. (2014). A user’s guide to Winsteps, Ministeps Rasch Model
Computer Programs. Retrieved from http://www.winsteps.com/
winman/principalcomponents.htm
Linacre, J. M. (2009). Local Independence and residual covar-
iance: A study of Olympic figure skating ratings. Journal of
Applied Measurement, 10, 157–169.
Lord, F. M. (1953). On the statistical treatment of football
numbers. American Psychologist, 8, 750–751. doi:10.1037/
h0063675
Lord, F. M. (1980). Applications of item response theory to practical
testing problems. Hillsdale, NJ: Erlbaum.
MacKenzie, K. R. (1983). The clinical application of a group
measure. In R. R. Dies & K. R. MacKenzie (Eds.), Advances in
group psychotherapy: Integrating research and practice (pp. 159–
170). New York, NY: International Universities.
MacNair-Semands, R. R., & Lese, K. P. (2000). Interpersonal
problems and the perception of therapeutic factors in group
therapy. Small Group Behavior, 31(2), 158–174. doi:10.1177/
104649640003100202
MacNair-Semands, R. R., Ogrodniczuk, J. S., & Joyce, A. S.
(2010). Structure and initial validation of a short form of the
Therapeutic Factors Inventory. International Journal of Group
Psychotherapy, 60, 245–281. doi:10.1521/ijgp.2010.60.2.245
Masters, G. N. (1982). A Rasch model for partial credit scoring.
Psychometrika, 47, 149–174. doi:10.1007/BF02296272
Mels, G. (2006). LISREL for windows: Getting started guide.
Lincolnwood, IL: Scientific Software International, Inc.
Muraki, E. (1992). A generalized partial credit model: Application
of an EM algorithm. Applied Psychological Measurement, 16(2),
159–176. doi:10.1177/014662169201600206
Ostini, R., & Nering, M. (2006). Polytomous item response theory
models. Thousand Oaks, CA: Sage.
Piper, W. E. Rosie, J. S., Joyce, A. S., & Azim, H. F. A. (1996).
Time-limited day treatment for personality disorders: Integration of
research and practice in a group program Washington, DC:
American Psychological Association.
Reckase, M. D. (1979). Unifactor latent trait models applied to
multifactor tests: Results and implications. Journal of Educa-
tional Statistics, 4, 207–230. doi:10.2307/1164671
Roy, V., Turcotte, D., Montminy, L., & Lindsay, J. (2005).
Therapeutic factors at the beginning of the intervention process
in groups for men who batter. Small Group Research, 36(1),
106–133. doi:10.1177/1046496404270261
Psychotherapy Research 143
Safran, J. D., Muran, J. C., & Eubanks-Carter, C. (2011). Repairing
alliance ruptures. Psychotherapy, 48(1), 80–87. doi:10.1037/a00
22140
Samejima, F. (1969). Estimation of latent ability using a response
pattern of graded scores. Psychometrika Monograph Supple-
ments, 17.
Tasca, G. A., Illing, V., Joyce, A. S., & Ogrodniczuk, J. S. (2009).
Three-level multilevel growth models for nested change data: A
guide for group treatment researchers. Psychotherapy Research,
19, 453–461. doi:10.1080/10503300902933188
Templin, J. (n.d.). IRT models for polytomous data. Lecture 4.
ICPSR item response theory workshop.
Thiessen, D. (1991). Multilog (Version 6) [Computer program].
Mooresville, IN: Scientific Software.
Van Dam, N. T., Earleywine, M., & Borders, A. (2010).
Measuring mindfulness? An item response theory analysis of
the Mindful Attention Awareness Scale. Personality and Indi-
vidual Differences, 49, 805–810. doi:10.1016/j.paid.2010.07.020
Yalom, I. D. (1995). The theory and practice of group psychotherapy
(4th ed.). New York, NY: Basic Books.
Yalom, I. D., & Leszcz, M. (2005). The theory and practice of group
psychotherapy (5th ed.). New York, NY: Basic Books.
Appendix 1. Hierarchical linear models
Three-level longitudinal model (for sensitivity to change)
Level 1 : Y tij ¼ p0ij þ p1ij  ðlog timeÞ þ etij
Level 2 : p0ij ¼ b00j þ b01j  ðindividual prescoresÞ þ b02
 ðBSI residualÞ þ f0ij
p1ij ¼ b10j þ b11j  ðindividual prescoresÞ þ b12
 ðBSI residualÞ þ f1ij
Level 3 : b00j ¼ c000 þ c001  ðgroup prescoreÞ þ k00j
b01j ¼ c010 þ k01j
b02j ¼ c020 þ k02j
b10j ¼ c100 þ c101j  ðgroup prescoreÞ þ k10j
b11j ¼ c110 þ k11j
b12j ¼ c120 þ k12j
The dependent variable (Ytij) in this model is the TFI-8 scores.
The growth models shown here used a log transformation for
“time” to model a more pronounced change from Session 4 to 8,
and less pronounced change from Sessions 8 to 12. Individual
scores were group mean centered and group scores were grand
mean centered.
Two-level hierarchically nested models (for predictive validity)
Level 1 : Y ij ¼ b0j þ b1j  ðpreBSI scoreÞ þ b2j
 ðTFI8 Week 4Þ þ fij
Level 2 : b0j ¼ c00 þ k0j
b1j ¼ c10 þ k1j
b2j ¼ c20 þ k2j
The dependent variables (Yij) in this model are Brief Symptom
Index (BSI) General Severity Index scale scores at Session 12.
Individual BSI prescores and TFI-8 scale scores at Week 4 were
grand mean centered.
144 G. A. Tasca et al.

Appendix 2
Item Characteristic and Item Information Curves for Six of the Eight Group Therapeutic Factor Inventory (TFI-8) Items (Curves for Items
3 and 18 Appear in Figure 1).

Item characteristic curve: 1 Item 1 Item information curve: 1

1.0 2.5

0.8 2.0

Information
Probability

0.6 1.5

1
0.4 1.0
2 3
4

0.2 6 0.5
5 7

0
–3 –2 –1 0 1 2 3 0 –3 –2 –1 0 1 2 3
Ability Scale score

Item characteristic curve: 3

Item 6 Item information curve: 3
1.0 2.5

0.8 2.0
Probability

Information

0.6 1.5
2

0.4 4 1.0
1
5
6
0.2 0.5
3
7

0 0
–3 –2 –1 0 1 2 3 –3 –2 –1 0 1 2 3
Ability Scale score
Item 9
Item characteristic curve: 4 Item information curve: 4
1.0 2.5

0.8 2.0
Probability

Information

0.6 1.5

0.4 5 1.0
6
2 3 4
1
7
0.2 0.5

0 0
–3 –2 –1 0 1 2 3 –3 –2 –1 0 1 2 3
Ability Scale score
 Psychotherapy Research 145

Item 12
Item characteristic curve: 5 Item information curve: 5
1.0 2.5

0.8 2.0
1
Probability

Information
0.6 1.5

2
0.4 3
1.0
4

6
0.2 5 0.5
7

0 0
–3 –2 –1 0 1 2 3 –3 –2 –1 0 1 2 3
Ability Scale score

Item characteristic curve: 6

Item 17 Item information curve: 6
1.0 2.5

0.8 2.0
Information
Probability

0.6 1.5

5
1 6
0.4 1.0
2 4
3 7

0.2 0.5

0 0
–3 –2 –1 0 1 2 3 –3 –2 –1 0 1 2 3
Ability Scale score

Item characteristic curve: 8

Item 19 Item information curve: 8
1.0 2.5

0.8 2.0
Information
Probability

0.6 1.5

0.4 4 1.0
1 2
5
3
6
0.2 7 0.5

0 –3 –2 –1 0 1 2 3 0 –3 –2 –1 0 1 2 3
Ability Scale score