See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/230068121

The Frustration Discomfort Scale: development and psychometric

properties

Article  in  Clinical Psychology & Psychotherapy · September 2005

DOI: 10.1002/cpp.465

CITATIONS READS

114 11,094

1 author:

Neil Harrington
University of Stirling
16 PUBLICATIONS   395 CITATIONS   

SEE PROFILE

All content following this page was uploaded by Neil Harrington on 01 December 2015.

The user has requested enhancement of the downloaded file.

Clinical Psychology and Psychotherapy
Clin. Psychol. Psychother. 12, 374–387 (2005)

Assessment The Frustration

Discomfort Scale:
Development and
Psychometric
Properties
Neil Harrington*
University of Edinburgh and Fife Health Board, UK

Frustration intolerance beliefs are hypothesized by rational–emotive

behaviour therapy (REBT) to form one of two major categories of
psychological disturbance, along with those referring to self-worth.
Although frustration intolerance is a central REBT concept its content
and factor structure are unclear. Therefore, a multidimensional
measure, the Frustration Discomfort Scale (FDS), was developed
based on REBT literature and theory. An exploratory factor analysis,
using a combined clinical and student population, indicated a four-
factor structure. These dimensions were labelled (I) emotional intol-
erance, involving intolerance of emotional distress; (II) entitlement,
involving intolerance of unfairness and frustrated gratification; (III)
discomfort intolerance, involving intolerance of difficulties and
hassles, and (IV) achievement, involving intolerance of frustrated
achievement goals. From these results, a simplified scale was devel-
oped, with confirmatory factor analysis supporting a four-factor solu-
tion. Both the preliminary and revised scales showed good internal
reliability, and evidence of discriminative validity. Copyright © 2005
John Wiley & Sons, Ltd.

INTRODUCTION However, although frustration intolerance1 is a

Rational–emotive behaviour therapy (REBT) is
unique in clearly distinguishing between two
categories of dysfunctional belief (Ellis, 1979, 1980).
The first category involves the intolerance of frus-
tration and discomfort, whilst the second involves
the global rating of self-worth. Whilst these two
categories of belief interact, they are also hypo-
thesized to have an independent and distinct
relationship with specific psychological problems.
central concept in REBT theory, and one of REBT’s
distinguishing features, investigation of these
beliefs has been limited (Neenan & Dryden, 1999).
Evidence from existing scales indicates that frus-
tration intolerance and self-worth do form separate
categories (DiGiuseppe, Leaf, Exner, & Robin,
1988). Yet beyond this, the content and factor struc-
ture of frustration intolerance beliefs, and their
relationship with specific problems, has remained
largely unexplored.

* Correspondence to: Dr. Neil Harrington, Stratheden Hos-
pital, Cupar, Fife, KY15 5RR, UK
E-mail: Neil@nharr.freeserve.co.uk
1
Traditionally the term ‘low frustration tolerance’ has
described this belief process. Since this has been criticized as
being ambiguous (Neenan & Dryden, 1999) ‘frustration
intolerance’ is used in this paper.

Copyright © 2005 John Wiley & Sons, Ltd.

Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/cpp.465
Development of the Frustration Discomfort Scale
One reason is that many earlier irrational belief
measures, such as the Belief Scale (Malouff &
Schutte, 1986), were based on inadequate and out-
dated definitions of irrational belief, and gave
only an overall score. Contemporary REBT theory
focuses on four belief processes, with demands
(e.g. ‘I must do well’) as the central process, and
self-worth, frustration intolerance, and awfulizing
secondary to this belief. Some assessment scales,
such as the Survey of Personal Beliefs (Demaria,
Kassinove, & Dill, 1989), have employed this
model. However, by focusing on belief processes
the content of beliefs has been largely ignored,
although recent evidence suggests that both belief
processes and their content are important. In par-
ticular, the series of experimental studies by Bond
and Dryden indicates that the content of self-worth
and frustration intolerance beliefs, rather than
demandingness, is central in determining psycho-
logical disturbance (Bond & Dryden, 2000).
The focus on belief content, rather than ‘irra-
tional’ processes, brings REBT closer to other
cognitive therapy approaches. However, whilst
assessment scales in the general literature have
focused on belief content, they do not clearly dis-
tinguish between self-worth and frustration intol-
erance. For instance, achievement beliefs on the
Dysfunctional Attitude Scale (DAS) (Weissman &
Beck, 1978) almost entirely refer to issues of self-
worth. Likewise, the Multidimensional Perfection-
ism Scale (Frost, Marten, Lahart, & Rosenblate,
1990) fails to separate self-worth and frustration
intolerance beliefs. In regard to perfectionism,
REBT would argue that high standards may be
independently associated with self-worth (‘If I fail
it means I’m a failure as a person’) and frustration
intolerance beliefs (‘I can’t stand not achieving my
absolute best’).
Even if irrational belief processes are fundamen-
tal to emotional problems, it may be argued that
these are too generalized and all encompassing to
enable specific predictions. Furthermore, although
frustration intolerance has been treated as unidi-
mensional, the range of frustration intolerance
beliefs described in the REBT literature suggests a
multidimensional construct. These beliefs include
the intolerance of hassles, effort, unfairness, and
uncomfortable emotions (Dryden & Gordon, 1993).
However, there is no existing belief scale specifi-
cally aimed at assessing the content of frustration
intolerance beliefs. The General Attitude and Belief
Scale (GABS) (DiGiuseppe et al., 1988) was de-
signed to assess both belief content and process,
and factor analysis of the GABS indicated that
Copyright © 2005 John Wiley & Sons, Ltd.
375
items were best grouped in terms of content (Shaw,
unpublished master’s thesis). In line with theoret-
ical models (Walen, DiGiuseppe, & Dryden, 1992),
these content factors reflected themes regarding
the need for achievement, approval, and comfort,
with further items added later to form a fairness
sub-scale (Bernard, 1998). However, the General
Attitude and Belief Scale has significant limita-
tions. First, the sub-scales do not clearly differenti-
ate between frustration intolerance and self-worth
beliefs, with some sub-scales including items from
both categories. Second, the frustration intolerance
content range is relatively limited with many
similar worded items, features that are likely to
distort the factor structure.
Therefore, Neenan and Dryden (1999) have high-
lighted the need to develop a multidimensional
measure of frustration intolerance. Likewise, from
a cognitive therapy perspective, Power et al. (1994)
have recommended the development of multi-
dimensional belief scales, with content areas based
on theoretical as well as statistically grounds. The
authors argue that the failure to adequately assess
specific belief domains is one reason for inconsis-
tent evidence linking core beliefs with enduring
vulnerability to depression. This is because only a
small sub-set of beliefs will be relevant to specific
problems, and therefore global measures of beliefs
will merely act as general indicators of distress.
Similarly, a fundamental weakness of REBT theory
has been the difficulty in distinguishing between
distinct psychological problems in terms of specific
irrational beliefs (Kendall et al., 1995).
The purpose of the present study was to develop
a multidimensional scale regarding the content
of frustration intolerance beliefs, and to clarify its
factor structure. Two separate studies were
involved in the development of the scale, both
employing clinical and student samples. The first
study reports the construction and exploratory
factor analysis of a preliminary Frustration Dis-
comfort Scale (FDS). Based on these results, a
revised scale was developed in the second study,
and the factor structure tested by confirmatory
factor analysis.
STUDY 1: PRELIMINARY FRUSTRATION
DISCOMFORT SCALE: DEVELOPMENT
AND ANALYSIS
Method
Participants
The clinical group comprised 242 respondents
from a group of consecutive therapy referrals to an
Clin. Psychol. Psychother. 12, 374–387 (2005)
376 N. Harrington

adult clinical psychology department (39% men tionally described by the phrase ‘I can’t stand it’)
and 61% women). The mean age of the clinical and self-worth beliefs were separately rated for
group was 39 years (range 18–74). The sample was each demand statement:
recruited from a diverse geographical and social
area, and represented a broad range of non- I absolutely must not be kept waiting
psychotic problems, with anxiety disorders (32%),
Because I can’t tolerate waiting
depression (23%), and anger (14%) the most
0 1 2 3 4
common classifications. Problems were largely
It would totally lower my self-esteem
moderate to severe, with 50% of clients experienc-
0 1 2 3 4
ing these difficulties for three years or more
and 44% rating the effect on their everyday life
Reverse-scored items were not used, since the use-
as ‘severe’. There was a response rate of 44%.
fulness of this method of reducing response bias
A previous pilot study had indicated that non-
has been questioned (Schriesheim & Hill, 1981). In
responders did not differ from responders on
particular, it has been noted that only ten percent
diagnostic or demographic characteristics. The
of respondents are required to mistakenly answer
non-clinical group consisted of undergraduate
the reversed questions the ‘wrong’ way to produce
psychology students from three separate courses
a spurious reverse-scored factor (Schmitt & Stults,
in abnormal psychology (30% male and 70%
1985). Items referring to affect were avoided to
female).
reduce cross contamination and to insure the scale
Missing data was randomly spread across the
primarily assessed cognition rather than emotion
items, with a mean percentage missing values for
(Smith, 1989).
each item of 0.32%. Nine clinical cases and one
To enable adequate investigation of the underly-
student case with more than 10% of missing values
ing factor structure, and satisfy content validity,
were removed. With remaining cases containing
care was taken to include all relevant aspects
only a small percentage of missing data, a group
of frustration intolerance beliefs. Possible content
mean was employed. Six extreme outliers were
domains were suggested by REBT theory, and
also removed due to problematic patterns of
included demands for achievement, approval,
responses. Thus, following preliminary data analy-
comfort, fairness, immediate gratification, and
sis, 329 cases were included in the factor analysis:
control/certainty (Neenan & Dryden, 1999; Walen
242 in the clinical group and 87 in the student
et al., 1992). An item pool was created using exam-
group.
ples of frustration intolerance in the REBT litera-
ture (e.g. Dryden & Gordon, 1993; Neenan &
Dryden, 1999). Seventy-four items were selected
Scale Development
from this larger group to form the preliminary
It was important that the structure of items con-
Frustration Discomfort Scale. A small pilot study
formed to current REBT theory. Irrational beliefs
(n = 30) was conducted to allow revision of items
have been described as being best represented as a
that had poor comprehension or extreme fre-
compound sentence involving both a demand and
quency distributions. Content validity of the scale
a secondary belief (DiGiuseppe, 1996). It has also
was reviewed by Albert Ellis (personal communi-
been argued that the ‘absolute’ nature of the
cation, 1999), who commented that the important
demand statement needs to be emphasized,
aspects of the concept had been thoroughly
and that the word ‘should’ is insufficient in this
described.
regard (Robb & Warren, 1990). Following these
recommendations, the items were designed as
compound sentences with demand statements
Measures
prefaced by the phrase ‘absolutely must’. It was
also necessary to differentiate frustration intoler- Frustration Discomfort Scale (preliminary version).
ance from self-worth beliefs. This presented some This contained 74 items, rated on a five-point
difficulty, since these two categories of belief are Likert scale. Individuals were asked to rate the
closely interrelated. Therefore, Dryden (1996) has strength of belief, with the following anchors: (0)
argued that the strength of self-worth relative absent, (1) mild, (2) moderate, (3) strong, (4) very
to frustration intolerance needs to be assessed to strong. Using examples, the instructions high-
determine which type of belief is most salient. To lighted the difference between frustration and self-
address this, both frustration intolerance (tradi- esteem beliefs.

Copyright © 2005 John Wiley & Sons, Ltd. Clin. Psychol. Psychother. 12, 374–387 (2005)
Development of the Frustration Discomfort Scale
Rosenberg Self-Esteem Scale (Rosenberg, 1965). This
scale was only completed in study I by the clinical
group. It is a 10-item Likert-type scale with satis-
factory internal reliability (0.87) and temporal
stability (0.86). Schmitt and Stults (1985) have
highlighted the importance of careful preliminary
screening and the deletion of ‘careless responses’
due to reversed item response error. Following this
recommendation, the present study identified out-
liers in terms of the difference between the nega-
tive and positive self-esteem sub-scale scores,
eliminating nine cases and leaving 232 for analy-
sis. The mean and standard deviation of the clini-
cal group was 23.39 (5.79), with higher scores
representing high self-esteem.
Design and Procedure
A cross-sectional design was used. For the clini-
cal group, questionnaires were included with
notification of first appointment with a stamped
addressed envelope for reply. For the student
group, questionnaires were completed during a
psychology workshop.
Results
Preliminary Analyses
A series of preliminary analyses were conducted
to identify items that were inappropriately in-
cluded in the scale. Frequency analysis examined
the distribution of responses across the rating scale
for each frustration question. Items were deemed
to have frequency problems if any two adjacent
scale points had fewer than 10% of responses. Reli-
ability analyses, carried out to identify items
not correlated with the scale, took the form of an
iterative process. That is, items with the lowest
reliability were removed and corrected item–total
correlations recalculated. Examination of these cor-
relations indicated two items below the 0.30 crite-
rion, these items involved being unable to ‘risk
upsetting others’ and the ‘need for excitement’.
It had been intended, by using both self-worth
and frustration intolerance statements, to identify
the most salient belief category. Several methods
of weighting frustration intolerance scores relative
to self-esteem scores were investigated. However,
these did not show meaningful results, and a sim-
plified scoring system was finally employed using
only the frustration intolerance statement scores.
Nevertheless, since a central aim of the FDS was
to measure frustration intolerance independently
from self-worth, other methods of determining
Copyright © 2005 John Wiley & Sons, Ltd.
377
discriminative validity were used. First, sub-
statement scores on the FDS were examined to
screen for those items with higher self-worth sub-
statement scores relative to frustration intolerance
scores. Paired t-tests indicated that only four items
had significantly higher self-worth sub-statement
scores. Second, correlations between FDS items
and the Rosenberg Self-Esteem Scale were exam-
ined. Comparison of corrected item–total correla-
tions showed that three items had significantly
lower correlations with remaining items on the
FDS than with the Rosenberg Self-Esteem Scale.
Two of these items (intolerance of personal flaws
and of upsetting others) were clearly associated
with the concept of self-worth. These two items
and six further items displaying multiple problems
were removed, leaving 66 items to be included in
the factor analysis.
Exploratory Factor Analysis
Since there is little empirical evidence regarding
the structure of frustration intolerance beliefs,
the preliminary study used exploratory factor
analysis. Theoretically, frustration intolerance
dimensions were likely to be closely associated,
suggesting that an oblique rotation would be most
appropriate. However, this can make interpreta-
tion more complex. Therefore, a principal com-
ponents analysis employing a varimax orthogonal
rotation was conducted first, and this was then
compared with an oblique rotation.
Careful consideration was given to the number
of factors to retain. The eigenvalue greater than one
criterion can greatly overestimate the number of
factors (Cattell, 1978), and with 15 factors above
this criterion this appeared to be the case in the
present study. A scree plot analysis is a more accu-
rate decision method (Zwick & Velicer, 1982),
and this indicated four factors. Therefore, solutions
involving four, five, and six factors were examined.
The eigenvalues of these six factors were 19.43,
3.46, 2.89, 2.31, 1.58 and 1.45. Examining the
five-factor solution, the fifth factor was found to
account for only a very small amount of additional
cumulative variance (2.7%), with just one variable
loading above 0.40. Likewise, in the six-factor solu-
tion the last two factors were poorly defined, with
the final factor having only one item with a sub-
stantial loading, and little psychological coherence
between items.
Therefore, a four-factor solution was considered
to best represent the data. The cumulative variance
accounted for was 42.58%, with factor I accounting
for 13.96%, factor II for 11.35%, factor III for 10.25%,
Clin. Psychol. Psychother. 12, 374–387 (2005)
378
and factor IV for 7.02%. All items had salient load-
ings on at least one factor. Reassuringly, the results
of the four-factor oblique rotation closely matched
the orthogonal rotation. The pattern of factors and
item order was essentially the same in both rota-
tions. Attempts at oblique rotations with five and
six factors failed to converge after 50 iterations,
further supporting the reliability and robustness of
the four-factor solution.
Factor Labelling
The first factor was labelled emotional intolerance,
with items reflecting intolerance of emotional dis-
tress. The item with the highest loading on this
factor was ‘I absolutely must be free of distressing
feelings as quickly as I can. I can’t stand for them
to continue’ (0.716). Items included beliefs regard-
ing the uncertainty, controllability, and aversive-
ness of emotion. REBT has long emphasized the
importance of secondary disturbance, such as
anxiety about anxiety, and its relationship to frus-
tration intolerance (Ellis, 1979).
The second factor was labelled entitlement. The
highest loading item was ‘I absolutely must not be
taken for granted. I can’t stand being unappreci-
ated’ (0.661). Included within this factor were
demands for immediate gratification and for fair-
ness. Whilst many of the items referred to personal
relationships, analysis indicated that these beliefs
were not associated with self-worth or gaining
approval. Thus, the item regarding upsetting
others had a negligible correlation (r = 0.09) with
entitlement, suggesting that individuals scoring
high on this factor were less concerned with
Table 1. Correlations between the revised FDS and Rosenberg Self-Esteem
Scale (preliminary FDS scores in brackets)
Discomfort intolerance
Entitlement
Emotional intolerance
Achievement
Full four-factor scale
Gratification
Fairness
Full five-factor scale
N = 232 (self-esteem), N = 242 (preliminary FDS).
N = 248 (self-esteem), N = 254 (revised FDS).
*** p < 0.001.
** p < 0.01.
* p < 0.05.
Copyright © 2005 John Wiley & Sons, Ltd.
N. Harrington
approval and more with the control and sub-
servience of others. Indeed, compared with the
other FDS sub-scales, entitlement had the lowest
correlation with self-esteem (Table 1). The central
theme of this factor might be summarized as the
belief that one’s desires must be met and that other
people should indulge and not frustrate these
desires.
The third factor, labelled discomfort intolerance,
referred to demands that life should be easy, com-
fortable, and free of hassle. The highest loading
item was ‘Tasks that I attempt absolutely must
not be too difficult. Otherwise, I can’t stand doing
them’ (0.680). Such beliefs have traditionally been
central to the REBT concept of frustration intoler-
ance. Whilst in the GABS these beliefs were com-
bined with emotional intolerance items to form a
‘need for comfort’ sub-scale, the present analysis
indicated they were best represented by a separate
factor.
Factor four was labelled achievement, with items
referring to task related frustration. The highest
loading item was ‘I absolutely must not leave work
unfinished. I can’t bear to leave work unfinished’
(0.745). These types of belief have been included in
several belief scales, including the DAS achieve-
ment sub-scale and the multidimensional perfec-
tionism scales. However, the present achievement
items aimed to assess the intolerance of frustration,
as opposed to loss of self-worth, following achieve-
ment goal failure. The relatively weak correlation
of the achievement sub-scale with self-esteem,
compared with its correlation with other FDS sub-
scales, supports the argument that this aim was
successful (Table 1).
Rosenberg FDS
Self-Esteem Scale
-0.43*** (-0.36***) 0.65*** (0.72***)
-0.20*** (-0.17*) 0.67*** (0.73***)
-0.49*** (-0.44***) 0.70*** (0.69***)
-0.29*** (-0.18**) 0.64*** (0.57***)
-0.43*** (-0.36***)
-0.28*** 0.79***
-0.16* 0.68***
-0.40***
Clin. Psychol. Psychother. 12, 374–387 (2005)
Development of the Frustration Discomfort Scale
Sub-Scale Formation
The final selection of items to form sub-scales
was primarily based on relative loadings, using a
cut-off point of >0.45. The need to take into account
communalities and to include a representative
content range led to three items below this loading
criterion being retained. Only three of the final sub-
scale items were classed as complex variables, with
loadings greater than 0.45 on other factors. Cor-
rected item–total correlations were computed to
highlight those items with poor reliability regard-
ing designated sub-scales. All items showed corre-
lations above 0.45 on their own scale, and above
0.30 with the full scale. Alpha values were (with the
mean inter-item correlation in brackets) emotional
intolerance 0.91 (0.43), entitlement 0.88 (0.37), dis-
comfort 0.90 (0.40), achievement 0.82 (0.36), and
full-scale score 0.95 (0.29). The full scale comprised
47 items, with each of the first three sub-scales con-
taining 13 items, and the achievement sub-scale
eight items. As expected, the FDS dimensions were
substantially intercorrelated (Table 2).
Divergent and Discriminative Validity
There were weak to moderate correlations
between the Frustration Discomfort sub-scales
and the Rosenberg Self-Esteem Scale (Table 1).
However, these correlations with self-esteem were
substantially lower compared with the correlations
between the Frustration Discomfort sub-scales and
Table 2. Revised FDS: sub-scale intercorrelation matrix (preliminary scale in
brackets)
Sub-scales 1 2
1 Discomfort intolerance
2 Entitlement 0.56 (0.65)
3 Emotional intolerance 0.62 (0.68)
4 Achievement 0.48 (0.47)
All correlations p < 0.001.
Table 3. Preliminary FDS: mean scores and standard deviations
Clinical
Full scale 95.1 (34.5)
Discomfort intolerance 22.4 (11.5)
Entitlement 25.0 (10.9)
Emotional intolerance 30.8 (12.2)
Achievement 16.9 (7.0)
N = 242 (clinical). N = 87 (student).
*** p < 0.001.
** p < 0.01.
* p < 0.05.
Copyright © 2005 John Wiley & Sons, Ltd.
379
the remaining FDS items (i.e. minus the sub-scale
in question). The discomfort and emotional intol-
erance sub-scales had the strongest correlations
with self-esteem and entitlement and achievement
the weakest.
Significant differences in FDS mean scores were
found between the clinical and student groups
(Table 3). All sub-scales, except entitlement, were
higher in the clinical population. There was no sig-
nificant correlation between the full scale score and
age (r(240) = -0.02, ns) or gender (t(240) = 0.62, ns),
or between these variables and individual sub-
scales. Although 15 final scale items did not sig-
nificantly differentiate between the two groups, it
was expected that some beliefs would have ele-
vated scores in the student group. Thus, items
referring to task obstruction, task hassle, task
interest, and task persistence all showed elevated
scores, possibly reflecting increased concerns
regarding academic work.
Discussion
The results of this study support a model of
frustration intolerance as a multidimensional
construct. The strong intercorrelations between
sub-scales, and the weaker relationships with self-
esteem, are consistent with the separation of frus-
tration intolerance beliefs from those of self-worth.
Factor analysis indicated that a four-factor solution
3 4
0.56 (0.56)
0.60 (0.55) 0.60 (0.51)
Student t
72.6 (27.1) 5.50***
16.6 (8.4) 4.31***
24.1 (9.3) 0.69
18.5 (9.4) 8.64***
13.5 (5.9) 4.04***
Clin. Psychol. Psychother. 12, 374–387 (2005)
380
best described the data, and yielded meaningful
sub-scales showing high internal consistency.
Additional factors appeared residual and poorly
defined. However, the original content categories
suggested by REBT theory were not entirely
replicated. Control beliefs were associated with a
number of content areas rather than forming a sep-
arate certainty/control factor. Discomfort intoler-
ance and emotional intolerance were also found to
form separate factors.
It could also be argued that the equivalent
dimension to the ‘need for approval’ within the
domain of frustration intolerance was that of enti-
tlement. A further finding was that, rather than
forming separate factors, items referring to imme-
diate gratification and fairness mainly loaded on
entitlement. Interestingly, whilst loading on enti-
tlement, the two positive gratification items (the
need for a ‘buzz’ or excitement) had weak rela-
tionships with remaining items on the FDS (0.34
and 0.24 respectively). This was unexpected, since
short-term hedonism is described as a core aspect
of frustration intolerance (Dryden & Gordon,
1993). On the other hand, the item referring to
intolerance of gratification delay had a relatively
strong correlation (0.58) with the remaining
scale. One possible explanation is that the two
‘excitement’ items reflected functional extraversion
rather than dysfunctional impulsiveness. It might
further suggest that frustration intolerance items
were better worded in terms of negative states,
such as deprivation or aversiveness, rather than as
a need for positive gratification.
STUDY II: REVISED FRUSTRATION
DISCOMFORT SCALE: DEVELOPMENT
AND ANALYSIS
Scale Redesign
The preliminary scale had been designed follow-
ing REBT theory that proposed irrational beliefs
were best represented by combining a primary
demand with a secondary self-worth or frustration
intolerance belief. However, subsequent research
indicated that secondary beliefs may constitute the
primary mechanism leading to disturbance, and
were best used on their own in assessment (Bond
& Dryden, 2000). The preliminary scale analysis
also indicated that all but two frustration intoler-
ance items had substantially higher correlations
with the FDS than with the Rosenberg Self-Esteem
Copyright © 2005 John Wiley & Sons, Ltd.
N. Harrington
Scale. This suggested that the overlap with self-
esteem was relatively small. Therefore, it was
decided to use a single frustration intolerance
statement for each item and dispense with the
additional demand and self-worth statements:
I can’t stand doing things that involve a lot
of hassle
1 2 3 4 5
These changes substantially simplified the ques-
tion structure and improved readability (Flesch
reading score of 84% compared to 64%). There
was also a reduction in missing data and spoiled
responses (2% compared to 5%). Furthermore, con-
siderably fewer responses in the ‘absent’ category
(12% compared to 21%) suggested greater ‘careless
responding’ on the preliminary scale, possible due
to item complexity.
All but two preliminary scale items were trans-
ferred to the new scale. Of 47 items included in the
revised scale, 37 were used with relatively minor
changes in wording. Overlapping and redundant
item content was reduced and the achievement
scale was lengthened to make it equal with other
sub-scales. Since it was unclear whether fairness
and gratification beliefs were separate dimensions
or facets of entitlement, further items were gener-
ated to enable entitlement to be split into two
smaller sub-scales. Study II aimed to examine how
well the hypothesized factor structure fitted the
data using a confirmatory factor analysis.
Method
Participants and Procedure
The participants and procedure followed that
of the preliminary study. The clinical sample con-
sisted of 254 patients, with the response rate (45%)
comparable to the previous study. Mean age of the
clinical group was 38.6 (41% male and 59% female).
Six cases were removed due to spoilt or missing
data. The non-clinical population (n = 79) came
from two classes of undergraduate psychology
students (17% male and 83% female). This gave an
overall sample of 333 cases for inclusion in the con-
firmatory factor analysis. The mean percentage of
missing values for each item was 0.48%, and these
were replaced by the group mean.
Measures
Frustration Discomfort Scale (revised version). The
initial scale consisted of 47 items rated on a five-
Clin. Psychol. Psychother. 12, 374–387 (2005)
Development of the Frustration Discomfort Scale
point Likert scale with the following anchors: (1)
absent, (2) mild, (3) moderate, (4) strong, (5) very
strong. The instructions were as follows: ‘Listed
below are a number of common thoughts and
beliefs that people may have when they are dis-
tressed or frustrated. Please read each statement
and decide how well this usually describes your
own beliefs. Circle the number that best indicates
the strength of this belief’. Following analysis, the
final number of items was reduced to 28 items on
the four-factor scale, with each sub-scale contain-
ing seven items. The five-factor scale contained
seven additional items to form separate gratifica-
tion and fairness sub-scales.
Rosenberg Self-Esteem Scale. The Rosenberg scale
was completed by clinical and student groups.
Preliminary screening resulting in the removal of
three clinical cases showing ‘careless responding’,
leaving 248 clinical, and 77 student replies, for
further analysis.
Preliminary Analyses and Sub-Scale Formation
Inappropriate items and cases were eliminated
using the same criteria as in the preliminary study.
No items showed frequency problems and all
items had corrected item–total correlations above
0.45. For the initial scale, internal reliabilities were
high with a full-scale alpha of 0.96 and sub-scale
alpha values between 0.92 and 0.85. However,
a high alpha may simply reflect high inter-
correlations among items, indicating overlapping
or redundant item content. Therefore, particular
importance was placed on reducing redundancy
and conceptual overlap when deciding item selec-
tion for the final scale. Examination of the sub-scale
intercorrelation matrix indicated only one pair of
items with an intercorrelation >0.70, referring to
intolerance of disturbed feelings and of disturbing
thoughts (r = 0.73). However, it was decided that
conceptually these beliefs were sufficiently distinct
to warrant inclusion. Five items with intercorrela-
tions above 0.60 were removed, and two other
items with overlapping content. Similarly, five
items with relatively low correlations with their
designated sub-scale were also removed. Cor-
rected item–total correlations were recalculated
after the removal of each item.
As in the preliminary study, the relationship
between individual items and the Rosenberg Self-
Esteem Scale was examined. Only two items (intol-
erance of disturbing thoughts and continuing
situation) had correlations above 0.50, and these
two items, along with all other items on the scale,
Copyright © 2005 John Wiley & Sons, Ltd.
381
had substantially higher correlations with the
overall FDS than with self-esteem. Final selection
for sub-scales took into account the strength of cor-
rected item-total correlations and the need to have
an acceptable range of item content.
Sub-scale alpha coefficients were (mean inter-
item correlation in brackets) discomfort intolerance
0.88 (0.50), entitlement 0.85 (0.42), emotional intol-
erance 0.87 (0.49), and achievement 0.84 (0.43),
with gratification 0.84 (0.43), and fairness 0.83
(0.42). The full-scale mean inter-item correlation
for the four-factor scale was 0.94 (0.34) and the five-
factor scale 0.95 (0.33). These values indicate good
internal consistency, with the mean inter-item cor-
relation giving a clearer measure of item homo-
geneity since this is not influenced by scale length.
Although the sub-scales were designed to measure
relatively narrow factors, no mean inter-item cor-
relation exceeded 0.50, which represents the item
redundancy cut-off point suggested by Briggs and
Cheek (1986). Furthermore, the full-scale scores are
within the optimal range of item homogeneity
(0.20–0.40) that Briggs and Cheek suggest offers the
best compromise between bandwidth and fidelity.
Finally, all items had corrected item–total correla-
tions above 0.48 with the full scale, and above 0.50
with sub-scales (Table 4). Overall, these results
suggest the revised scale has captured the range
and complexity of frustration intolerance beliefs
within an acceptable level of item redundancy.
Confirmatory Factor Analysis
Both clinical and student FDS scores were
normally distributed, and examination of individ-
ual item distributions did not indicate marked
deviations from normality. Therefore, confirmatory
factor analysis was conducted employing the
maximum likelihood method and using the AMOS
4 program (Arbuckle & Wothke, 1999). Five alter-
native models were compared. The first model was
a unidimensional model in which all the items
were assumed to load on a single frustration intol-
erance dimension. A two-factor model had dis-
comfort intolerance and emotional intolerance
forming a ‘discomfort’ factor, and entitlement
and achievement forming a ‘frustration’ factor. The
four-factor model was the solution most strongly
supported by the exploratory factor analysis. A
five-factor model split the entitlement sub-scale
into two further sub-groups representing fairness
and immediate gratification.
The adequacy of the competing models was
evaluated using five different fit indices: (1) the
model chi-square, a measure of overall fit, with a
Clin. Psychol. Psychother. 12, 374–387 (2005)
382
non-significant c2 indicating good fit; (2) the c2
divided by the degrees of freedom, with a ratio of
between two and three suggesting a good fit
(Carmines & McIver, 1981); (3) the comparative fit
index (CFI; Bentler, 1990), with values above 0.95
indicating a good fit; (4) the root mean square of
approximation (RMSEA; Browne & Cudeck, 1993),
with values over 0.10 leading to rejection of the
model, those from 0.05 to 0.08 acceptable, and
values below 0.05 indicating a good fit; (5) the
Tucker–Lewis Index (TLI; Tucker & Lewis, 1973),
which takes into account the degree of parsimony,
with scores of above 0.90 regarded as a reasonable
fit.
Fit indices are presented in Table 5. Although all
models had significant c2 values, this frequently
overstates the lack of fit in larger samples.
However, the one- and two-factor models also dis-
played a poor fit on the other fit indices, with both
the c2 ratio and the RMSEA values failing to reach
acceptable levels. On the other hand, the four- and
five-factor models were both good fits to the data.
The CFI and TLI values indicate a better fit for the
four-factor model, although the difference was rel-
atively small. All items loaded on their designated
factors. The standardized factor loadings for both
the five- and four-factor models were all substan-
tial and significant, ranging from 0.554 to 0.826
(Table 4).
Sub-Scale Intercorrelations
The examination of the intercorrelations between
sub-scales was helpful in comparing the four- and
five-factor models (Table 2). The correlation cor-
rected for attenuation due to unreliability should
be larger than the sub-scale reliabilities, and if this
is not the case then the two scales lack discri-
mination. All four sub-scales, although moderately
intercorrelated, met this criterion. However, the
fairness and gratification scales had corrected
correlation (0.87) higher than their respective reli-
abilities (0.83 and 0.84), suggesting that these two
scales are practically indistinguishable.
Divergent and Discriminative Validity
Correlations between the FDS and Rosenberg
Self-Esteem Scale are presented in Table 1. Com-
parison with the preliminary scale results shows an
increased association with between the revised
scale and self-esteem, with the full-scale correla-
tion increasing from r = 0.36 to 0.43. However,
apart from the achievement sub-scale, which con-
tained a wider range of items in the revised scale,
the increase is relatively small. Moreover, sub-scale
Copyright © 2005 John Wiley & Sons, Ltd.
N. Harrington
correlations with remaining FDS items were sub-
stantially higher than with self-esteem, supporting
the divergent validity of the scale. Whilst the com-
parisons in Table 1 refer to clinical groups, the rela-
tionship between the FDS sub-scales and student
self-esteem scores showed evidence of even greater
separation (correlations with remaining FDS items
in brackets): discomfort intolerance -0.38 (0.57),
entitlement -0.11 (0.67), emotional intolerance
-0.38 (0.70), achievement -0.24 (0.68).
Means and standard deviations for both the four
and five-factor models are presented in Table 6.
Independent t-tests indicated significant differ-
ences in FDS scores between the clinical and
student groups. As might be expected, emotional
intolerance showed the greatest separation and
achievement frustration the least. Analysis of indi-
vidual items indicated that nine items did not
distinguish between the two groups. As with the
preliminary scale, these items largely reflected
the increased importance of task orientation in a
student population (Q2, 4, 5, 8, 9, 10, 16, 24, and
25). Four of the seven additional items included
in the fairness and gratification sub-scales also
failed to discriminate between clinical and student
groups (Q30, 31, 32, and 33). Interestingly, the grat-
ification sub-scale had the strongest correlation
with remaining FDS items, underlining the impor-
tance of these beliefs.
There were no gender differences for the FDS
total score in the clinical or student groups. How-
ever, the emotional intolerance sub-scale was
significantly higher for females in the clinical
group (t(252) = 2.53, p < 0.05). Unlike the prelimi-
nary scale, there was a weak but significant nega-
tive relationship between the FDS total score and
age (r(248) = -0.15, p < 0.05), with age significantly
correlating with entitlement (r(248) = -0.24, p <
0.001) and emotional intolerance (r(248) = -0.15, p
< 0.05). This indicates that entitlement and emo-
tional intolerance decrease with greater maturity.
Discussion
The confirmatory factor analysis supported the
multidimensional model of frustration intolerance
described in study I. Although the four- and five-
factor models both fitted the data well, the magni-
tude of the difference in fit between these two
models was small and the degree of interrelation-
ship large. Furthermore, the fairness and gratifica-
tion sub-scales had corrected correlations higher
than their reliability coefficients. This suggests
Clin. Psychol. Psychother. 12, 374–387 (2005)
 Table 4. Revised FDS: Standardized factor loadings and corrected item-total correlations

Factor Factor Factor Factor Factor Factor Corrected

I II III IV II (a) II (b) item–total
correlation
Factor I Discomfort intolerance
21 I can’t stand doing things that involve a lot of hassle 0.784 0.724
13 I can’t stand having to push myself at tasks 0.758 0.701
25 I can’t stand having to persist at unpleasant tasks 0.704 0.637
5 I can’t stand doing tasks that seem too difficult 0.702 0.666
17 I can’t stand the hassle of having to do things right now 0.672 0.633
1 I need the easiest way around problems; 0.669 0.602

Copyright © 2005 John Wiley & Sons, Ltd.

I can’t stand making a hard time of it
9 I can’t stand doing tasks when I’m not in the mood 0.666 0.623
Development of the Frustration Discomfort Scale

Factor II Entitlement
10 I can’t bear it if other people stand in the way 0.756 0.683
of what I want
6 I can’t stand it if other people act against my wishes 0.716 0.632
18 I can’t stand having to give in to other people’s demands 0.707 0.625
22 I can’t stand having to change when others are at fault 0.697 0.665
26 I can’t tolerate criticism especially when I know I’m right 0.627 0.581
14 I can’t tolerate being taken for granted 0.600 0.547
2 I can’t stand having to wait for things I would like now 0.559 0.499

Factor III Emotional intolerance

19 I can’t bear disturbing feelings 0.826 0.756
11 I can’t bear to have certain thoughts 0.799 0.728
3 I must be free of disturbing feelings as quickly as possible; 0.702 0.646
I can’t bear if they continue
15 I can’t stand situations where I might feel upset 0.685 0.613
27 I can’t stand to lose control of my feelings 0.636 0.585
7 I can’t bear to feel that I am losing my mind 0.632 0.600
23 I can’t get on with my life, or be happy, 0.619 0.570
if things don’t change

Clin. Psychol. Psychother. 12, 374–387 (2005)

383
 384

Factor IV Achievement
28 I can’t tolerate any lapse in my self-discipline 0.672 0.588
24 I can’t stand feeling that I’m not on top of my work 0.666 0.602
20 I can’t stand doing a job if I’m unable to do it well 0.660 0.581
4 I can’t stand being prevented from achieving 0.654 0.604
my full potential
8 I can’t bear the frustration of not achieving my goals 0.652 0.604
16 I can’t bear to move on from work 0.651 0.591
I’m not fully satisfied with
12 I can’t tolerate lowering my standards even when 0.617 0.565

Copyright © 2005 John Wiley & Sons, Ltd.

it would be useful to do so

Factor II (a) Gratification

10 I can’t bear it if other people stand in the way of what I want 0.743 0.677
6 I can’t stand it if other people act against my wishes 0.713 0.621
18 I can’t stand having to give into other people’s demands 0.688 0.591
29 I can’t tolerate being overlooked 0.658 0.594
34 I can’t bear being deprived now of things I lacked in the past 0.612 0.563
32 I can’t stand giving up immediate pleasures for 0.587 0.536
the sake of a distant goal
2 I can’t stand having to wait for things I would like now 0.579 0.559

Factor II (b) Fairness

22 I can’t stand having to change when others are at fault 0.757 0.691
14 I can’t tolerate being taken for granted 0.716 0.651
33 I can’t tolerate being treated with disrespect 0.689 0.635
26 I can’t tolerate criticism especially when I know I’m right 0.641 0.552
35 I can’t tolerate other people’s bad or stupid behaviour 0.592 0.529
31 I can’t stand being left in the dark with no explanations 0.585 0.521
30 I can’t bear to have been treated unjustly 0.554 0.494

Clin. Psychol. Psychother. 12, 374–387 (2005)

N. Harrington
Development of the Frustration Discomfort Scale 385

Table 5. Confirmatory factor analysis: fit indices

Model c2 df c2/df CFI TLI RMSEA

Single factor 1531.51 350 4.38 0.949 0.940 0.101
Two factors 1224.90 349 3.51 0.962 0.956 0.087
Four factors 758.46 344 2.21 0.982 0.979 0.060
Five factors 1208.37 550 2.20 0.977 0.974 0.060

Table 6. Revised FDS: means and standard deviations

Clinical Student t
Four-factor score 90.8 (20.7) 75.7 (15.9) 6.82***
Discomfort intolerance 20.4 (6.5) 17.6 (4.7) 4.22***
Entitlement 22.1 (6.2) 19.6 (4.4) 3.87***
Emotional intolerance 25.6 (6.6) 18.3 (5.1) 10.39***
Achievement 22.7 (6.1) 20.2 (5.2) 3.28***
Five-factor score 113.2 (25.3) 95.4 (19.4) 6.57***
Gratification 19.9 (6.3) 17.2 (4.1) 4.43***
Fairness 24.6 (5.9) 22.2 (5.1) 3.27***
N = 254 (clinical group). N = 79 (student group).
*** p < 0.001.

that they are correlated to such an extent that little
would be gained by their separation, and that they
are best interpreted as highly related facets of enti-
tlement (John & Benet-Martinez, 2000). However,
whilst the four-factor model is preferred on the
grounds of parsimony it may be premature to con-
clude that the distinction between gratification
and fairness is unimportant. One limitation of the
present study is that, whilst representing a wide
range of disorders, the numbers of individuals in
some specific diagnostic categories was relatively
small. Problems expected to have particular
associations with frustration intolerance, such as
alcohol and substance abuse, were underrepre-
sented, and it would be useful for further research
to determine the generalizability of these factors to
different populations.
The revised scale did show an increased correla-
tion with self-esteem, suggesting that the more
complex preliminary scale structure helped to sep-
arate these belief categories. However, the increase
in correlation is relatively small, and balanced
against this the revised scale shows improvement
in readability, acceptability, and psychometric
properties. Since individuals with low frustration
tolerance would be less likely to complete difficult
tasks, these are not minor considerations. Discrim-
ination between the clinical and student groups
was also improved, with all sub-scales significantly
higher in the clinical group.
These results are of clinical and theoretical
importance. Regarding the REBT theory of psy-
chological disturbance, they support a model of
irrational belief in which frustration intolerance
and self-worth beliefs form separate categories of
disturbance (DiGiuseppe, 1996). However, com-
parison of the preliminary and revised scales indi-
cates that restricting the wording of irrational
beliefs to frustration intolerance statements, and
omitting absolute demandingness, did not change
the factor structure or reduce the relationship
with psychological disturbance. Whilst this does
not imply that imperative demands are irrelevant
to disturbance, it does suggest that other belief
processes may be equally important. This is also
consistent with recent evidence that irrational
beliefs may be best phrased in terms of secondary
belief content (Bond & Dryden, 2000).
From a clinical perspective, effective therapy
depends on being able to identify which cognitions
are related to what problems. However, although
frustration intolerance beliefs are hypothesized as
being central to many difficulties, such as therapy
resistance (Ellis, 1985), the empirical evidence is

Copyright © 2005 John Wiley & Sons, Ltd. Clin. Psychol. Psychother. 12, 374–387 (2005)
386
sparse. In particular, it is unclear whether the
dimensions of frustration intolerance are uniquely
related to particular problems or if these beliefs
have a more general influence. There is initial evi-
dence from further validation data that specific
dimensions of frustration intolerance are associ-
ated with distinct emotional problems, indepen-
dent of any shared relationship with self-esteem
(Harrington, unpublished doctoral thesis). It is
hoped that the development of a multidimensional
measure will encourage the further investigation of
these beliefs.
ACKNOWLEDGEMENTS
This paper is based on research submitted to the
University of Edinburgh in part fulfilment of a
Doctorate of Philosophy degree. I would like to
thank Professor Mick Power for his help and
encouragement throughout the research.
REFERENCES
Arbuckle, J.L., & Wothke, W. (1999). Amos 4.0 users guide.
Chicago, IL: SPSS.
Bernard, M.E. (1998). Validation of the General Attitude
and Belief Scale. Journal of Rational–Emotive and
Cognitive–Behavior Therapy, 16, 183–196.
Bentler, P.M. (1990). Comparative fit indices in structural
equation models. Psychological Bulletin, 107, 238–246.
Bond, W.B., & Dryden, W. (2000). How rational beliefs
and irrational beliefs affect people’s inferences: An
experimental investigation. Behavioural and Cognitive
Psychotherapy, 28, 33–43.
Briggs, R.S., & Cheek, J.M. (1986). The role of factor
analysis in the development and evaluation of per-
sonality scales. Journal of Personality, 54, 106–148.
Browne, M.W., & Cudeck, R. (1993). Alternative ways
of assessing model fit. In K.A. Bollen and J.S. Long
(Eds.), Testing structural equation models (pp. 136–162).
Newbury Park, CA: Sage.
Carmines, E.G., & McIver, J.P. (1981). Analyzing models
with unobserved variables. In G.W. Bohrnstedt, & E.F.
Borgatta (Eds), Social measurement: Current issues.
Beverly Hills: Sage.
Cattell, R.B. (1978). The scientific use of factor analysis. New
York: Plenum.
Demaria, T.P., Kassinove, H., & Dill, C.A. (1989). Psy-
chometric properties of the survey of personal beliefs:
A rational–emotive measure of irrational thinking.
Journal of Personality Assessment, 53, 329–341.
DiGiuseppe, R. (1996). The nature of irrational and
rational beliefs: Progress in rational emotive behavior
therapy. Journal of Rational Emotive and Cognitive–
Behavior Therapy, 14, 5–28.
DiGiuseppe, R., Leaf, R.C., Exner, T., & Robin, M. (1988).
The development of a measure of irrational/rational think-
Copyright © 2005 John Wiley & Sons, Ltd.
N. Harrington
ing. Poster session presented at the World Congress of
Behaviour Therapy, Edinburgh.
Dryden, W. (1996). When musts are not enough. In
W. Dryden (Ed.), Inquiries in rational emotive therapy
(pp. 47–50). London: Sage.
Dryden, W., & Gordon, W. (1993). Beating the comfort trap.
London: Sheldon.
Ellis, A. (1979). Discomfort anxiety: A new cognitive
behavioral construct. Part I. Rational Living, 14, 3–8.
Ellis, A. (1980). Discomfort anxiety: A new cognitive
behavioral construct. Part II. Rational Living, 15, 25–30.
Ellis, A. (1985). Overcoming resistance: Rational emotive
therapy with difficult patients. New York: Springer.
Frost, R.O., Marten, P., Lahart, C., & Rosenblate, R.
(1990). The dimensions of perfectionism. Cognitive
Therapy and Research, 14, 449–468.
John, O.P., & Benet-Martinez, V. (2000). Measurement:
Reliability, construct validation, and scale construc-
tion. In H.T. Reis, & C.M. Judd (Eds), Handbook of
research methods in social and personality psychology (pp.
339–369). Cambridge: Cambridge University Press.
Kendall, P.C., Haaga, D.A.F., Ellis, A., Bernard, M.,
DiGiuseppe, R., & Kassinove, H. (1995). Rational–
emotive therapy in the 1990 and beyond: Current
status, recent revisions, and research questions. Clini-
cal Psychology Review, 15, 169–185.
Malouff, J.M., & Schutte, N.S. (1986). Development and
validation of a measure of irrational belief. Journal of
Consulting and Clinical Psychology, 54, 860–862.
Neenan, M., & Dryden, W. (1999). Rational emotive behav-
iour therapy: Advances in theory and practice. London:
Whurr.
Power, M.J., Katz, R., McGuffin, P., Duggan, C.F., Lam,
D., & Beck, A.T. (1994). The Dysfunctional Attitude
Scale (DAS): A comparison of forms A and B and pro-
posals for a new sub-scaled version. Journal of Research
in Personality, 28, 263–276.
Robb, H.B., & Warren, R. (1990). Irrational belief tests:
New insights, new directions. Journal of Cognitive Psy-
chotherapy, 4, 303–311.
Rosenberg, M. (1965). Society and the adolescent self-image.
Princeton, NJ: Princeton University Press.
Schmitt, N., & Stults, D.M. (1985). Factors defined
by negatively keyed item: The result of careless
respondents? Applied Psychological Measurement, 9,
367–373.
Schriesheim, C.A., & Hill, K.D. (1981). Controlling acqui-
escence response bias by item reversals: The effect
on questionnaire validity. Educational and Psychological
Measurement, 41, 1101–1114.
Smith, T.W. (1989). Assessment in rational emotive
therapy: Empirical access to the ABCD model. In
M. Bernard, & R. DiGiuseppe (Eds), Inside rational
emotive-therapy: A critical appraisal of the theory and
therapy of Albert Ellis (pp. 135–153). San Diego:
Academic.
Tucker, L.R., & Lewis, C. (1973). A reliability coefficient
for maximum likelihood factor analysis. Psychometrika,
38, 1–10.
Walen, S.R., DiGiuseppe, R., & Dryden, W. (1992). Prac-
titioner’s guide to rational–emotive therapy (2nd ed.).
Oxford: Oxford University Press.
Clin. Psychol. Psychother. 12, 374–387 (2005)
Development of the Frustration Discomfort Scale 387

Weissman, A., & Beck, A.T. (1978). Development and vali- Zwick, W.R., & Velicer, W.F. (1982). Factors influencing
dation of the Dysfunctional Attitude Scale. Paper pre- four rules for determining the number of components
sented at the Annual Convention of the Association for to retain. Multivariate Behavioral Research, 17, 253–269.
Advancement of Behavior Therapy, Chicago, IL.

Copyright © 2005 John Wiley & Sons, Ltd. Clin. Psychol. Psychother. 12, 374–387 (2005)
View publication stats