Work & Stress

ISSN: 0267-8373 (Print) 1464-5335 (Online) Journal homepage: http://www.tandfonline.com/loi/twst20

Measuring exposure to bullying and harassment at

work: Validity, factor structure and psychometric
properties of the Negative Acts Questionnaire-
Revised

Staale Einarsen , Helge Hoel & Guy Notelaers

To cite this article: Staale Einarsen , Helge Hoel & Guy Notelaers (2009) Measuring
exposure to bullying and harassment at work: Validity, factor structure and psychometric
properties of the Negative Acts Questionnaire-Revised, Work & Stress, 23:1, 24-44, DOI:
10.1080/02678370902815673

To link to this article: http://dx.doi.org/10.1080/02678370902815673

Published online: 20 May 2009.

Submit your article to this journal

Article views: 3589

View related articles

Citing articles: 178 View citing articles

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=twst20

Download by: [University of Otago] Date: 20 March 2016, At: 18:23

 Work & Stress
Vol. 23, No. 1, January
March 2009, 24
44

Measuring exposure to bullying and harassment at work: Validity, factor

structure and psychometric properties of the Negative Acts
Questionnaire-Revised

Staale Einarsena*, Helge Hoelb and Guy Notelaersa

a
Bergen Bullying Research Group, Faculty of Psychology, University of Bergen, Norway;
b
Manchester Business School, University of Manchester, UK
Downloaded by [University of Otago] at 18:23 20 March 2016

This study investigates the psychometric properties, factor structure and validity of the revised
Negative Acts Questionnaire-Revised (NAQ-R), an instrument designed to measure exposure
to bullying in the workplace. By reanalyzing data based on a heterogeneous sample of 5288
UK employees, the results show that the 22-item instrument has a high internal stability,
with three underlying factors: personal bullying, work-related bullying and physically inti-
midating forms of bullying, although the instrument may also be used as a single factor
measure. Criterion validity was explored by relating the scores on the NAQ-R to a single-item
measure of perceived victimization from bullying, showing high correlations with both the
total NAQ-R and scores on the three factors. Targets of bullying scored significantly higher on
all 22 items compared to non-targets. The NAQ-R correlated as expected with measures of
mental health, psychosocial work environment and leadership, indicating a good construct
validity of the instrument. Furthermore, a latent class cluster (LCC) analysis showed that the
instrument may be used to differentiate between groups of employees with different levels of
exposure to bullying, ranging from infrequent exposure to incivility at work to severe
victimization from bullying and harassment. The more commonly used operational criteria
can also be used to detect targets of bullying. Hence, the NAQ-R is proposed as a standardized
and valid instrument for the measurement of workplace bullying.
Keywords: bullying; harassment; victimization; survey; inventory; prevalence; Negative Acts
Questionnaire

Introduction
Workplace bullying (the persistent exposure to interpersonal aggression and
mistreatment from colleagues, superiors or subordinates) is a prevalent problem in
contemporary working life, with devastating effects on both targets and organiza-
tions (Einarsen, Hoel, Zapf, & Cooper, 2003; Rayner & Keashly, 2005). Much effort
has, therefore, been put into the documentation of this serious problem by
determining its frequency and risk groups within different occupational groups
and organizations (see Zapf, Einarsen, Hoel, & Vartia, 2003), and a number of
studies addressing the issue of prevalence of bullying within different countries and
different occupational groups have been published (e.g. Mikkelsen & Einarsen, 2001;

*Corresponding author. Email: stale.einarsen@psysp.uib.no

ISSN 0267-8373 print/ISSN 1464-5335 online

# 2009 Taylor & Francis
DOI: 10.1080/02678370902815673
http://www.informaworld.com
 Work & Stress 25

Niedhammer, David, & Degioanni, 2006). However, Keashly and Harvey (2005)
argue that this field of research has often been influenced by a desire to discover
substantive issues at the expense of the development of sound methodology.
A range of self-report inventories and scales measuring exposure to bullying and
harassment have been introduced in this field; however, most have been used only in
one or a few studies (e.g. Dawn, Cowie, & Ananiadou, 2003), with the Negative Acts
Questionnaire (Einarsen & Raknes, 1997) and the Leymann Inventory of Psycho-
logical Terror (LIPT; Leymann, 1997) as two notable exceptions. Many studies have
also used instruments that are derivations, expansions or shortened versions of other
original scales portraying unknown validity and reliability (e.g. Salin, 2001). Some
scales are overly long and, consequently, difficult to use in standard organizatio-
nal surveys. For instance, the WAR-Q (see Keashly & Neumann, 2004) contains 60
items in its original version, while the LIPT (Leymann, 1997) contains 48 items.
While some instruments are based on the responses of undergraduate students (e.g.
Downloaded by [University of Otago] at 18:23 20 March 2016

Keashly, Trott, & MacLean, 1994), others, such as the LIPT, are designed as
diagnostic tools to identify victims of severe bullying likely to be traumatized from
their experience, and hence are less suitable in surveys of the general working
population. However, a main problem is that hardly any studies have been published
in peer review journals explicitly outlining and describing a proposed instrument
while simultaneously scrutinizing its validity and psychometric properties. Our paper
addresses these shortcomings, proposing the Negative Acts Questionnaire-Revised as
a valid, comprehensive, yet short scale to be used in organizational surveys of
exposure to workplace bullying. In this paper we initially describe the theoretical
foundation on which the NAQ-R rests, followed by a description of the instrument
and its development. Subsequently, we provide empirical evidence on psychometric
properties, factor structure and validity of the instrument, before we conclude by
pointing out some of the limitations and strengths of the instrument and include its
practical applications.

Dimensions and defining characteristics of bullying

The concept of workplace bullying refers to situations where an employee is
persistently exposed to negative and aggressive behaviours at work primarily of a
psychological nature (Leymann, 1996), with the effect of humiliating, intimidating,
frightening or punishing the target. Many of these single acts may be relatively
common in the workplace (Leymann, 1996) and, when occurring in isolation, may be
seen as signs of uncivil behaviour or ‘‘incivility’’ at work (Lim & Cortina, 2005).
When persistently directed towards the same individual(s) over a longer period of
time, they may turn into an extreme source of social stress (Zapf, 1999), capable of
causing severe harm. Although the negative and unwanted nature of the behaviour
involved is essential to the concept of bullying, the concept’s core characteristic is not
the nature of the behaviours per se, but rather the persistency of the experience
(Einarsen et al., 2003). Thus, the emphasis is as much on the frequency and duration
of what is done as it is on what and how it is done.
Hence, bullying constitutes evolving and often escalating hostile workplace
relationships rather than discrete and disconnected events and is associated with
repetition (frequency), duration (over a period of time) and patterning (of a variety
of behaviours involved) as its most salient features (Einarsen et al., 2003). A simple
 26 S. Einarsen et al.

distinction exists between direct actions, such as accusations, verbal abuse and public
humiliation, on the one hand, and indirect acts of aggression, such as rumours,
gossiping and social isolation, on the other (e.g. O’Moore, Seigne, McGuire, &
Smith, 1998). With reference to the workplace setting, a further distinction has been
made between work-related behaviours and person-related behaviours (Einarsen,
1999), evident in an early Finnish study (Vartia, 1991) where slander, social isolation
and insinuation about someone’s mental health may be seen as examples of person-
related bullying, whilst giving a person too many, too few or too simple tasks, or
persistently criticizing a person or their work, may be associated with work-related
bullying. Although primarily concerned with negative behaviour of a psychological
nature, studies involving targets of bullying have revealed that physically intimidating
acts, and in some cases even physical violence or the threat of violence, form part of a
wider repertoire of aggressive acts applied in bullying cases (Leymann, 1990).
Another feature of some definitions of bullying is the imbalance of the power
Downloaded by [University of Otago] at 18:23 20 March 2016

relationships between the parties involved (Niedl, 1996). A pre-existing or evolved

imbalance of power between the parties is considered central to the bullying experience,
as this may limit targets’ ability to retaliate or successfully defend themselves. In many
cases the imbalance of power may simply mirror the formal power-structure of the
organizational context in which the bullying scenario unfolds, as would be the case
when someone is on the receiving end of negative acts from higher up in the
organizational hierarchy (Hutchinson, Vickers, Jackson, & Wilkes, 2006). Alterna-
tively, the source of power may be informal and related to factors such as knowledge
and experience as well as access to social support (Einarsen et al., 2003). Most
importantly, the nature of the bullying experience in terms of its frequency and long-
term duration of exposure to negative acts tends to drain the coping resources of the
target, thus in itself emphasizing the increasing powerlessness of targets (Leymann,
1996).
The distinction between subjective and objective experience has been essential
when considering how best to operationalize the definition of bullying. According to
Brodsky (1976), subjective experience of bullying refers to targets’ perceptions of their
experience. By contrast, objective experience of bullying has to be validated or verified
by third parties or observers, for example by means of peer nomination. However,
Björkqvist and colleagues (Björkqvist, Österman, & Hjeltbäck, 1994) argued against
the objectivity and, therefore, supposedly neutrality of such an approach, as economic
dependency could effectively prevent people from being honest in their assessment.
This would be particularly true when asked to assess one’s superiors or people in
formal positions of power. Nevertheless, according to Niedl (1996), the ‘‘definitional
core of bullying at work rests on the subjective perception made by the victim
that these repeated acts are hostile, humiliating and intimidating and that they are
directed at himself/herself’’ (p. 49). It follows that it is the subjective experience of the
behaviours or the pattern of behaviours that will manifest themselves in mental and
physical health problems.

The Negative Acts Questionnaire-Revised (NAQ-R): descriptions and development

The NAQ-R is based on the previous NAQ (Einarsen & Raknes, 1991, 1997;
Mikkelsen & Einarsen, 2001). Based on a review of the literature and a series of case
studies, the original scale had 23 items describing negative acts of a personal as well
 Work & Stress 27

as a work-related nature. Although this scale showed high internal consistency (as
measured by Cronbach’s alpha), containing items with good face validity and with
evidence of good construct validity, the scale also had some serious shortcomings.
Its items were overly influenced by the perspectives of severely affected targets,
developed at a time when most bullying research and debate was confined to the
Nordic countries. Its validity was only tested within a limited Scandinavian cultural
context. When translated into English, the face validity of some items was
questionable, with other items revealing a cultural bias. A further weakness was
found in its factor structure, although a two-factor solution associated with work-
related and a personal bullying respectively was used in some studies (see Einarsen &
Raknes, 1997; Matthiesen & Einarsen, 2001). Hence, a revised scale for use in other
national settings was clearly needed.
The NAQ-R was therefore created with the aim of establishing a reliable,
valid, comprehensive, yet relatively short scale, tailor-made for use in a variety of
Downloaded by [University of Otago] at 18:23 20 March 2016

occupational settings, and especially adapted to Anglo-American cultures. Items

were developed and refined based on the original scale, conceptual reasoning and a
focus group study employing 11 focus groups with 61 participants from a variety of
UK occupations and positions across organizational hierarchies (see Hoel, Cooper,
& Faragher, 2001, for an overview). This resulted first in a 29-item new version of the
NAQ (see Hoel, Cooper, & Faragher, 2001; Hoel, Cooper, & Faragher, 2004). Later,
on the basis of further analyses, a 22-item version was proposed (see Einarsen &
Hoel, 2001; Nielsen et al., 2008). In the present paper we test the 22-item scale, which
taps direct and indirect aspects of bullying and contains items that can be construed
as work-related bullying, person-related bullying or physical intimidation respec-
tively. All items are written in behavioural terms with no reference to the terms
‘‘bullying’’ or ‘‘harassment,’’ following recommendations by Arvey and Cavanaugh
(1995) in relation to sexual harassment. Although based on self-report, such an
approach is considered to provide a more objective estimate of exposure to bullying
behaviours than self-labelling approaches, as respondents’ need for cognitive and
emotional processing of information would be reduced.

Aim of the study

In the present study, the instrument’s psychometric properties, factor structure and
validity were scrutinized by utilizing and partly re-analyzing data from an existing
large-scale survey of UK employees that focused on the prevalence, antecedents and
consequences of workplace bullying (see also Hoel, 2002; Hoel, Cooper, & Faragher,
2001, 2004). Based on previous research on bullying (Einarsen, Hoel, Zapf, &
Cooper, 2003; Rayner & Keashly, 2005; Zapf & Einarsen, 2005), we hypothesized
that the NAQ-R would show negative correlations with measures on subjective
health and well-being, and negative correlations with perceptions of the quality of
the psychosocial work environment, including job satisfaction, commitment and
relationships with superiors and colleagues. According to Leymann (1990), bullying
exists in organizations characterized by deficiencies in both work design and
leadership and a negative social climate. Following Kelloway and colleagues (2004),
‘‘poor management’’ may be present in two forms: an active and abusive type of
leadership, and an indirect and passive form where managers have abdicated their
responsibilities. Where managers avoid taking charge or involving themselves with
 28 S. Einarsen et al.

interpersonal conflicts and tensions, it is argued that there is a breeding ground for
bullying (Leymann, 1996). In many cases, targets also report being bullied by their
manager or supervisor (Zapf et al., 2003). Hence, we expect that those exposed to
bullying will experience their immediate superior as an abusive and tyrannical leader.
The NAQ-R should also be, albeit more moderately, associated with raised levels of
sick-leave, reduced work performance and increased intention to leave. Furthermore,
a strong positive association with subjective feelings of victimization from bullying at
work should exist, as the NAQ-R should be able to differentiate clearly between
targets and non-targets of bullying. Finally, building on a theoretical notion of
bullying as a process and not an ‘‘either-or’’ phenomenon, NAQ-R should be able to
identify different groups of respondents having unique experiences of bullying
behaviours regarding the nature, frequency and severity of their experience (see also
Notelaers, Einarsen, De Witte, & Vermunt, 2006).
Downloaded by [University of Otago] at 18:23 20 March 2016

Methods
Participants
In total, 12,350 questionnaires were distributed to employees in 70 organizations
within the private, public and voluntary sectors across Great Britain, with a total of
just under one million employees. A total of 5288 were returned providing a response
rate of 42.8%, ranging from 27% to 57% in the different organizations (see also Hoel,
2002). A total of 2764 (52.4%) were males and 2508 (47.6%) females. The mean age
for the total sample was 40.2 years (standard deviation 9.8) and the median
40 years. A total of 84.9% of respondents worked full-time as opposed to 15.1% part
time. Approximately two thirds of respondents had been in their present job for
4 years or more (65.8%), as opposed to 11.8% who took up their current job within
the last 12 months.

Measurements
Exposure to bullying. The version of the NAQ-R tested in this study has 22 items,
measuring exposure to bullying within the last 6 months, with the response
alternatives: ‘‘Never,’’ ‘‘Now and then,’’ ‘‘Monthly,’’ ‘‘Weekly’’ and ‘‘Daily’’. As
the last category was seldom used, we collapsed the two latter categories in the
statistical analysis. A single-item measuring self-labelled victimization from bullying
during the last 6 months was then included after presenting the respondents with a
global definition of bullying (see Einarsen & Skogstad, 1996; Salin, 2001). This was
followed by a number of questions regarding the experience of bullying, such as
frequency of encounters, duration of experience and who the main perpetrators were,
etc. Participants were then given six alternatives: ‘‘no,’’ ‘‘yes, very rarely,’’ ‘‘yes, now
and then,’’ ‘‘yes, several times per month,’’ ‘‘yes, several times per week’’ and ‘‘yes,
almost daily.’’ By being provided with a relatively unambiguous definition of
bullying, respondents’ tendency to use their own definitions when considering this
question should be reduced.

Absenteeism, intention to leave and work performance. Respondents were asked to state
how many days they had been off work due to their own illness within the last
6 months, given the following options: no days off, 1
3 days, 4
10 days, 11
20 days
 Work & Stress 29

and more than 20 days. For the variable ‘‘intention to leave,’’ respondents were
provided with five options (never, rarely, sometimes, quite often and very often) in
response to the question: ‘‘Have you considered quitting your present job over the
last 6 months?’’ Respondents were also asked to rate their present performance or
productivity given five different options varying from 100% to less than 50% of
normal capacity.

Mental and psychosomatic health. Mental health was measured by the 12-item version
of the General Health Questionnaire (GHQ; Grayson, Bridges, Duncanjones, &
Goldberg, 1987). Internal consistency as measured by Cronbach’s alpha was 0.92
for these 12 items. Psychosomatic health complaints were measured by means of a
12-item scale taken from the Occupational Stress Indicator (Cooper, Sloan, &
Marshall, 1988), with a high internal consistency in the present study (Cronbachs’s
alpha .89).
Downloaded by [University of Otago] at 18:23 20 March 2016

Psychosocial work environment quality. Five subscales from the Pressure Manage-
ment Indicator (PMI) were employed (Williams & Cooper, 1998): ‘‘Workload’’
included six items measuring the amount of work as well as possible problems
encountered in the work situation (Cronbach’s alpha.86); ‘‘Relationships with
colleagues’’ included eight items focusing on how well one interacts with colleagues
and the support one may receive from them (Cronbach’s alpha .89); ‘‘Organiza-
tional climate’’ included four items measuring the ‘‘feel’’ or ‘‘atmosphere’’ within the
workplace (Cronbach’s alpha .84); ‘‘Organizational satisfaction’’ included six items
(Cronbachs’s alpha .88); while ‘‘Organizational commitment’’ measured the level
of commitment a person feels to their organization and the contribution work makes
to their quality of life, with five items (Cronbach’s alpha .78).
Leadership. Destructive leadership by one’s immediate superior was measured with
two scales devised for the purpose of this study (see Hoel, 2002, for more details).
‘‘Autocratic leadership,’’ where there is no room for employee involvement in the
decision-making process and where such involvement is considered unnecessary,
was measured by five items taken from item pools produced by the focus group
study (Cronbach’s alpha .76). Examples of the items were: ‘‘Keeps information to
him/herself’’ and ‘‘Insists that a task should be undertaken in a particular way
whether this is necessary or not.’’ ‘‘Laissez-faire leadership’’ refers to indifference
and lack of involvement in tasks as well as lack of interest in employees and was
measured with three items (Cronbach’s alpha .77), e.g. ‘‘Turns a blind eye to
conflicts and disputes among staff’’; ‘‘Is never around to deal with problems when
they arise.’’

Statistical design
In order to examine the dimensionality of the NAQ-R, a confirmatory approach
employing LISREL 8.80 (Jöreskog, Sörbom, & du Toit, 2007) was conducted.
Based on earlier research and theoretical notions we expected three underlying
factors of bullying to exist (here referred to as sub-factors), tapping person-related
bullying, work-related bullying and physical intimidation. Consequently, different
measurement models were defined and tested. As the response categories of the
NAQ-R strictly speaking are not interval measures, but rather may be conceived as
 30 S. Einarsen et al.

ordinal data, weighted least squares estimators in LISREL 8.80 were used. Hence,
next to the polychoric correlation matrix, an asymptotic covariance matrix was
estimated to ‘‘correct’’ for non
normality. A non-significant or small chi-square
value indicates that the model fits the data well. However, in large samples even
small and substantively unimportant differences between the estimated model and
the "true" underlying model will result in rejection of the model that is tested
(Bentler & Chou, 1987). Therefore, we also considered other indices in judging the
fit of our models. The root-mean square error of approximation or the RMSEA
has been put forward as a measure for approximate fit (Jöreskog, 2005). A non-
significant RMSEA indicates close fit. In addition we used more descriptive fit
measures like the goodness-of-fit index (GFI), based on a ratio of the squared
discrepancies to the observed variances (Jöreskog & Sorbom, 1993), the non-
normed fit index (NNFI) which represents the increase in fit when comparing any
hierarchical step-up comparison of two models (Bentler & Bonett, 1980), and the
Downloaded by [University of Otago] at 18:23 20 March 2016

comparative fit index (CFI) levels of .90 or better for CFI, GFI and NNFI (Byrne,
2002).
A latent class cluster analysis (LCC) (Magidson & Vermunt, 2001, 2004) using
Latent GOLD (Vermunt & Magidson, 2003) was conducted in order to investigate
whether different homogeneous groups of respondents exist, and which might differ
according to the nature and degree of their exposure to bullying (Notelaers et al.,
2006). LCC allows us to test empirically the number of different target groups as
well as the prevalence of each group. This method takes the nature, frequency and
severity of the different items into account (Vermunt, 2001), thus creating a more
nuanced picture of what is measured, thereby facilitating the understanding of high
correlations between the different dimensions (Notelaers et al., 2006). As with
other cluster method, LCC classifies respondents into mutually exclusive groups,
irrespective of the dimensionality of the measurement. Unlike other cluster
methods, LCC does not rely on the strict assumption that responses are measured
on an interval level and that data are normally distributed (Eid, Langeheine, &
Diener, 2003). Hence, LCC can handle measures of bullying that necessarily
produce highly skewed results with response alternatives, strictly speaking, being at
an ordinal rather than an interval level. LCC classifies respondents into mutually
exclusive groups with respect to a not directly observed (latent) trait (e.g. being a
target of bullying), starting with the assumption that there is only one group,
subsequently estimating more classes of respondents until a LCC model is found
that statistically fits the data (Magidson & Vermunt, 2001, 2004). To determine
how many clusters are needed in order to explain the associations in the multiway
table, the Bayesian Information Criterion (BIC) is used. In line with Magidson and
Vermunt (2004), the model with the lowest BIC was accepted. Next to BIC, and
equally important, is the question whether the model fits to the data: given L2 with
a given number of degrees of freedom and a p-value that has to exceed a value of
.01 in large samples. However, for very sparse tables, it is likely that L2 does not
follow a x2 distribution. Therefore, Langeheine, Pannenkoek, and VandePol (1996)
suggest a bootstrapping procedure. Even though response categories have been
recoded, such a bootstrapping procedure is still needed because bullying items are
necessarily highly skewed and many combinations of behaviours are still very
uncommon, which results in very sparse tables. Finally, the initial bivariate
 Work & Stress 31

associations between the indicators should be sufficiently explained by the latent

class model (Magidson & Vermunt, 2004).

Results
Reliability and factor solution
Cronbach’s alpha for the 22 items in the NAQ-R was .90, indicating excellent internal
consistency whilst also suggesting that it may be a reliable instrument with an
even fewer number of items. Yet, the ‘‘alpha if item deleted’’ analysis indicated that
internal consistency would not improve if any of the 22 items were deleted. To test
the proposed underlying dimensions of the NAQ-R we followed a confirmatory
approach with three distinguishable measurement models. These were a one-
dimension model with all items measuring the same latent variable, i.e. workplace
bullying, a two-dimension model with items loading exclusively on either a person-
Downloaded by [University of Otago] at 18:23 20 March 2016

related bullying dimension or a work-related bullying dimension, and a three-

dimension model including physical intimidation with three items loading on this
factor. The analysis showed that even though an overall workplace bullying mea-
sure (one dimension) fits the data (RMSEA .051, p
.05; WLSx2 3003.2 (209)
p B.000) and is associated with acceptable descriptive fit measures, the two-
dimension solution fits the data better since it is associated with a significantly
decreasing x2 (RMSEA .051, p
.05; WLSx2 2899.5 (208) p B.001). The three-
dimension structure, however, that additionally distinguishes physically intimidating
acts, has the best fit (RMSEA .049, p
.05; WLSx2 2741.4 (206) p B.001). The
more descriptive statistics CFI (.91 for model 1 and .92 for models 2 and 3), NNFI
(.91 for all models) and GFI (.98 for all models) also indicated that the fit is
satisfactory for all three models. Table 1 presents the 22 items of the NAQ-R and
their factor leadings on their respective sub-factor, along the correlations among the
three sub-factors.
All factor loadings exceed .70, with no cross loadings or error correlations. Yet,
although the three-factor solution is associated with the best fit, the correlations
between the factors or dimensions are very high: .96 between person-related and
work-related bullying, .89 between work-related and physically intimidating bullying
and .83 between person-related and physically intimidating bullying. Hence, several
underlying dimensions of reported workplace bullying can be distinguished, yet they
do not discriminate well between different types of bullying behaviours, suggesting
co-occurrence of these different types of bullying.

A latent class cluster approach

Latent class cluster (LC) was computed using Latent Gold to investigate whether the
NAQ-R may be used to identify and differentiate between groups of respondents
sharing a certain probability regarding the nature and frequency of their exposure to
bullying at work (see also Notelaers et al., 2006). As opposed to the factor models
which identify one or more latent variables on a continuum which is then translated
into a sum-score or a factor-score to be used for investigating relationships with
other variables, this method estimates homogeneous groups of respondents
regarding the nature and degree of their exposure to bullying. This method,
 32
Table 1. Items, factors loadings and correlations among sub-factors of the NAQ-R.

NAQ-R item
Factor number Item wording Factor loading

Work-related bullying 1 Someone withholding information which affects your performance .71
Downloaded by [University of Otago] at 18:23 20 March 2016

3 Being ordered to do work below your level of competence .77

14 Having your opinions ignored .88
16 Being given tasks with unreasonable deadlines .85
18 Excessive monitoring of your work .82
19 Pressure not to claim something to which by right you are entitled (e.g. sick leave, holiday .77
entitlement, travel expenses)
21 Being exposed to an unmanageable workload .81
Person-related bullying 2 Being humiliated or ridiculed in connection with your work .86

S. Einarsen et al.
4 Having key areas of responsibility removed or replaced with more trivial or unpleasant .86
tasks
5 Spreading of gossip and rumours about you .84
6 Being ignored or excluded .83
7 Having insulting or offensive remarks made about your person, attitudes or your private .87
life
10 Hints or signals from others that you should quit your job .93
11 Repeated reminders of your errors or mistakes .90
12 Being ignored or facing a hostile reaction when you approach .88
13 Persistent criticism of your errors or mistakes .95
15 Practical jokes carried out by people you don’t get along with .85
17 Having allegations made against you .92
20 Being the subject of excessive teasing and sarcasm .91
Physically intimidating 8. Being shouted at or being the target of spontaneous anger .88
bullying
9. Intimidating behaviours such as finger-pointing, invasion of personal space, shoving, .86
blocking your way
22 Threats of violence or physical abuse or actual abuse .83
 Table 1 (Continued)
NAQ-R item
Factor number Item wording Factor loading

Correlations among the factors

Downloaded by [University of Otago] at 18:23 20 March 2016

Work-related Person-related Physical

intimidation
Work-related 1.00
Person-related .96 1.00
Physical intimidation .83 .89 1.00

Work & Stress

33
 34 S. Einarsen et al.

therefore, allows us empirically to test the number and nature of different target
groups as well as the prevalence of each group.
The fit statistics from the LCC-analysis showed that the Bayesian Information
Criterion (BIC) dropped when adding an additional cluster up to the eighth cluster
model. (For more details on the results of this analysis, please contact the first
author.) BIC increased when a nine cluster solution was estimated. However, the
difference in BIC between a seven and an eight cluster model (in which the latter
solution only adds a new group that is not exposed to bullying) is much smaller than
the differences between the other models. Next, the bivariate residual analysis of the
seven and the eight cluster models showed that the association between items 3 and 4
was still not accounted for by the number of clusters. Since the association between
items four and three was also larger than three in the eighth cluster model, the
parsimonious principle urges us to free this association in the seventh cluster model.
The BIC of a seven cluster model with an (‘error’) correlation between the third
Downloaded by [University of Otago] at 18:23 20 March 2016

and the fourth indicator, was the lowest. Equally important, the seven cluster model
fits the data well: next to L2 also x2 that was bootstrapped is not significant (p .25).
Additionally, all bivariate residuals showed that the initial associations between the
indicators in the one cluster model were sufficiently explained. Hence, we concluded
that a model with seven clusters fits the data best.

The seven clusters

The meaning of the seven clusters can be derived from the latent profile output (cf.
Vermunt & Magidson, 2003). Table 2 summarizes this profile output. The first row
denotes the size of the clusters. The other cells in the body of the table refer to the
average conditional probability (CP) to respond ‘‘never,’’ ‘‘now and then,’’ ‘‘once a
month’’ or ‘‘once a week/daily’’ to any of the 22 negative acts of the NAQ-R.
Respondents from the first cluster were characterized by an average CP of .94
to report never having been subjected to any kind of negative acts during the
last 6 months. Because of the high average conditional probability that these
employees answer ‘‘never’’ having encountered any kind of negative acts measured

Table 2. The seven clusters, their prevalence and their average conditional probabilities
across items (ranging from 0 to 1) for each response category.

Some Occasional Work-

No work negative Occasional related Severe Physical
bullying criticism encounters bullying bullying bullying intimidation

Size of 28% 25% 15% 13% 10% 5% 3%

cluster
Never .94 .74 .72 .40 .62 .22 .51
Now and .06 .24 .24 .47 .21 .30 .33
then
Monthly .00 .01 .02 .07 .06 .13 .05
Weekly/ .00 .01 .02 .06 .12 .34 .12
daily
 Work & Stress 35

by the NAQ-R, this cluster was labelled as a ‘‘No bullying’’ cluster. Twenty eight
percent of the respondents belonged to this cluster.
The respondents of the second cluster are also characterized by a high probability
to answer ‘‘Never.’’ This time, however, the average CP dropped to .74. A closer
inspection of the profile output showed some exposure to work-related negative acts
of a frequency of ‘‘now and then’’ in this cluster. Other types of negative acts were
hardly reported. Since this clusters reports relatively few negative behaviours and
since those belonging to the cluster report limited exposure to almost exclusively
work-related negative acts, we conclude that this cluster was not bullied, although
they may be exposed to criticism of their work. This cluster is the second largest
cluster and covers about 25% of the respondents in this sample, and was labelled
‘‘Some work-related criticism.’’
The average probability structure over items for the respondents in the third
cluster was quite similar to that of the respondents in the second cluster, with work-
Downloaded by [University of Otago] at 18:23 20 March 2016

related negative acts not reported more frequently than in the second cluster. Yet, in
this cluster some person-related acts were reported more frequently, corresponding to
the answer ‘‘now and then.’’ However, some types of negative acts, e.g. social isolation
and physical intimidation, are ‘‘never’’ being reported. Again we conclude that these
employees, constituting some 15% of the sample, are probably not bullied. Yet, one
may still say that they are experiencing some ‘‘occasional negative encounters.’’
The fourth cluster was characterized by an average CP over items of almost .50.
In this cluster, participants reported having been subjected to a given negative act
while at work ‘‘now and then.’’ At the same time, the CP to report more frequent
exposure was only .13. The dominant feature of this cluster was that, compared to
the former clusters, the average probability to report exposure to other negative acts
‘‘now and then’’ concerning social isolation and other person-related bullying
behaviour as well as physically intimidating behaviour doubles or even triples or
quadruples. Yet, the CP of the reported negative acts referring to social isolation and
physical intimidation remained relatively small. In sum, respondents in this cluster
reported occasional but still clearly systematic exposure to workplace bullying,
although most acts happened typically ‘‘now and them.’’ Therefore, we label these
respondents as ‘‘Occasional bullying.’’ About 13% of the respondents belonged to
this cluster.
The fifth cluster was not really characterized by its average probability structure.
Although this cluster reported on average less workplace bullying (average CP for the
‘‘never’’ category equals .62), it is rather characterized by being more frequently
exposed to work-related types of bullying. The CP to report being monthly or more
frequently confronted with work-related bullying acts was particularly high, with the
CP to be exposed on a ‘‘weekly’’ basis to work-related bullying being .24. Hence we
label this cluster as the ‘‘Work-related bullying’’ cluster.
The sixth cluster was characterized by the highest average CP (.34) in that the
respondents had been subjected to bullying behaviours on a weekly or more frequent
basis. This is contrasted by the low average CP for the response alternative ‘‘never’’
(.22). Compared to the occasionally bullied cluster and the work-related bullied
cluster, the CP for reporting exposure to bullying on a ‘‘monthly’’ or more frequent
basis increased substantially. For work-related negative acts the CP doubled, whilst
for personal related and/or physical intimidation, the CP increased nearly tenfold. As
 36 S. Einarsen et al.

respondents in this cluster were clearly exposed to intense bullying, we label this as
the ‘‘Severe bullying’’ cluster, comprising about 5% of the sample.
The seventh and last cluster showed a high average CP over items to report
‘‘never’’ to have been confronted with bullying behaviours. However, a closer
inspection of the profile output shows that the CP to report to be exposed to some
specific kinds of negative acts ‘‘weekly/daily’’ is much higher than among any of the
‘‘Severe bullying,’’ the ‘‘Occasional bullied’’ and the ‘‘Work-related bullying’’ clusters.
Looking at the nature of these specific negative acts, we conclude that these
respondents are targets of persistent physical intimidation. We, therefore, label this
cluster, which represents 3% of the respondents, as the ‘‘Physical intimidation’’
cluster.

Validity of the NAQ-R

Downloaded by [University of Otago] at 18:23 20 March 2016

Table 3 shows Pearson product
moment correlations between the total NAQ-R

score and the three factors, on the one hand, and measures of health, sickness
absenteeism, work performance and intention to leave, on the other. To correct for
any skewed scales, we also calculated non-parametric correlations employing
Spearman rank order correlations for all correlations in Tables 3 and 4. These
correlations were either identical to or stronger than those reported except for sick
leave, where the results provided .01 to .02 lower correlations with the NAQ-scales.
Correlations between total NAQ-R and GHQ-12 and psychosomatic complaints
respectively were statistically significant and moderately strong. Correlations
between exposure to bullying and self-rated work performance, sick leave and
intention to leave were also significant, yet moderate to weak, and in the expected
direction. High scores on the NAQ-R were associated with more health complaints,
reduced performance, raised sickness absenteeism and higher scores on greater
inclination to leave one’s job. As far as the three factors of the NAQ-R are
concerned, the strongest correlations are found for work-related bullying (NAQ-
R-work), with the weakest correlations found for physical intimidation (NAQ-
R-Physical Intimidation).

Table 3. Pearson product-moment correlations between total NAQ-R, the three sub-factors
and measures of health, sick-leave, self-rated work performance and having considered leaving
one’s job during the last 6 months. Means and standard deviations for all scales.

Mean (SD) 5 6 7 8 9

1. NAQ-R 31.88 (10.15) .43 .41 .13 .22 .36

2. Person-related bullying 14.51 (5.04) .33 .34 .11 .18 .26
3. Work-related bullying 13.78 (5.2) .45 .42 .12 .23 .40
4. Physical intimidation 3.88 (1.85) .18 .20 .08 .09 .15
5. GHQ-12 25.01 (6.38) 1
6. Psychosomatic complaints 32.88 (12.07) .68 1
7. Sickness absenteeism last 6 months 1.88 (1.07) .23 .26 1
8. Self-rating of recent work performance 1.96 (1.08) .34 .28 .21 1
9. Considered leaving last 6 months 2.43 (1.3) .48 .44 .19 .29 1
All correlations are significant at the .001 level (2-tailed).
 Work & Stress 37
Table 4. Pearson product-moment correlations between NAQ-R and its sub-factors and
scales measuring: leadership style; work pressure from workload; climate and relationships
with colleagues; and organizational satisfaction, commitment and security. Means and
standard deviations for all scales.

Mean (SD) 5 6 7 8 9 10 11

1. NAQ-R 31.88 (10.15) .52 .45 .41 .53 .61 .35 .48
2. Person-related bullying 14.51 (5.04) .42 .35 .24 .33 .45 .24 .36
3. Work-related bullying 13.78 (5.2) .53 .45 .49 .52 .58 .38 .50
4. Physical intimidation 3.88 (1.85) .24 .20 .19 .22 .28 .18 .23
5. Autocratic leadership 11.83 (4.2 1
6. Laissez-faire leadership 7.20 (2.8) .55 1
7. Workload 19.49 (7.22) .21 .21 1
8. Organizational climate 14.83 (4.49) .37 .40 .50 1
9. Stress from relationships 27.19 (8.41) .50 .53 .46 .68 1
Downloaded by [University of Otago] at 18:23 20 March 2016

with colleagues
10. Organizational 19.04 (4.3) .31 .30 .19 .41 .43 1
commitment
11. Organizational 18.75 (5.45) .45 .52 .25 .60 .64 .54 1
satisfaction
All correlations are significant at the .001 level (2-tailed).

As shown in Table 4, significant and fairly strong correlations exist between the
scores on the NAQ-R and measures of work environment quality and ratings of the
leadership style of one’s immediate superior. All correlations are in the expected
direction, indicating that respondents with high scores on the NAQ-R tended to rate
their immediate superior as being high on autocratic leadership and/or high on
laissez-faire leadership. High scores on NAQ-R were also related to high workload
pressure, a negative organizational climate, negative relationships with colleagues,
low scores on organizational satisfaction and low scores on commitment. It is worth
noting that the strongest relationships were found between bullying and autocratic
leadership, and similarly between bullying and experiencing a negative relationship
with one’s colleagues, while the weakest relationships were revealed as being bet-
ween the NAQ-R and organizational commitment. Hence, reporting exposure
to behaviours measured by the NAQ-R was more closely associated with relation-
ships with colleagues and leaders than with one’s relationship with the organization
as such.
In total, 10.6% of respondents reported being avictim of workplace bullying during
the last 6 months. Pearson’s product
moment correlation between the total score on
the NAQ-R and the perception of being a victim of workplace bullying was strong
(r.54, p B.001), indicating that having high scores on the NAQ-R was strongly
associated with labelling oneself as a victim of bullying, yet also indicating that self-
labelling may be related to factors other than mere exposure to bullying behaviours.
Analysis of variance using one-way ANOVA revealed that self-reported victims of
bullying scored significantly higher than non-victims on all 22 items (p B.001).
Lastly, we investigated the validity of the latent class clusters by relating them to a
range of individual and work-related outcomes. Table 5 shows the mean standar-
dized scores (z-scores) on eight outcome variables for all the seven latent clusters.
 38 S. Einarsen et al.

Table 5. Comparisons of the mean scores of the seven latent clusters on eight individual and
work-related outcomes using z-values. Identical superscripts (a, b or c) indicate that the pairs of
means are not significantly different. Means with no identical superscripts differ significantly
from each other (a.05).

Some Occasional Work-

No work negative Occasional related Severe Physical
bullying criticism encounters bullying bullying bullying intimidation

Considered 0.4 0.04 0.18 0.37 0.57 0.75 0.14

leaving during
leave during
last 6 months
GHQ-12 0.47 0.05 0.15 0.39 0.63 0.99 0.07
Psychosomatic 0.49 0.02 0.16 0.47 0.51 0.92 0.04
complaints
Downloaded by [University of Otago] at 18:23 20 March 2016

Organizational 0.37 0.07a 0.12a 0.35b,c 0.44b 0.73 0.26c

commitment
Organizational 0.54 0.08a 0.12a 0.46 0.60 1.02 0.31
satisfaction
Workload 0.53 0.18a 0.29 0.31b 0.68c 0.67c 0.25a,b
Stress from 0.69 0.05a 0.08a 0.66b 0.56b 1.16 0.32
relationships
with colleagues
Autocratic 0.60 0.01a 0.07a 0.60 0.50 1.20 0.12
leadership

A negative Z-score represents an unfavourable score on organizational commitment

and satisfaction. For the other outcome variables, a positive Z-score represents an
unfavourable position. After performing an analysis of variance, post-hoc multiple
comparisons (pairwise Bonferroni tests) were administered. The one-way analysis of
variance showed that there were significant differences among the clusters on all
outcomes, although not all groups may be different from each other. The variation
between the latent cluster categories regarding the outcome variables varied from
about 1 to 1.80 standard deviations between the ‘‘severe bullying’’ cluster and the
‘‘not bullied at all’’ cluster, with the remaining clusters in between. It is worth
noticing that the seventh cluster ‘‘Physical intimidation’’ was less problematic
compared to the ‘‘severe bullying’’ cluster (see Table 5 for statistical details). Hence,
overall, these results indicate good content and discriminatory validity for the latent
class cluster solution.

Discussion
Based on the study presented, we conclude that the 22-item NAQ-R comprises a
reliable and valid measure of exposure to workplace bullying, with three inter-related
factors associated with person-related bullying, work-related bullying and physically
intimidating bullying, respectively. Yet, the NAQ-R may also be used as a one-factor
or even as a two-factor measurement of work-related and person-related bullying.
Identifying stable and easily interpretable and meaningful factors in comprehensive
measures of workplace aggression across samples has been difficult when a full
 Work & Stress 39

spectrum of abusive behaviours is considered (Einarsen & Raknes, 1997; Keashly &
Harvey, 2005), hence the instrument should also be tested in other cultural settings.
Yet, the high internal consistency of the scale, as measured by Cronbach’s alpha
suggested that the number of items could be further reduced if used as a uniform
one-factor measure of exposure to bullying at work. Furthermore, the latent class
analysis revealed that the NAQ-R discriminates well between different target groups
regarding the nature and the severity of their exposure, also revealing target groups
that probably face some occasional aggression and incivility, rather than bullying in
the strictest sense. The fact that at least 70% of the working population that we
studied did not appear to be exposed to bullying, with some 5% of respondents being
exposed to severe bullying, and with another 3% experiencing physical abuse or
intimidation, also bears witness to the validity of the NAQ-R, as bullying
theoretically, logically and empirically is considered to be an infrequent phenom-
enon, especially in its most severe forms.
Downloaded by [University of Otago] at 18:23 20 March 2016

Furthermore, six of the emerging clusters were identical to those identified by

Notelaers and colleagues in a Belgian sample using the original NAQ scale
(Notelaers et al., 2006), which in itself represents a further indication of good
validity of the NAQ-R as well as generalizability of these result across cultures. In the
present study, a seventh cluster not found in the latter study also emerged, associated
with a high probability for exposure to physically intimidating behaviour only.
However, this supports the results from the factor structure where this constituted a
separate factor.
The latent class analysis is particularly well suited for instruments such as the
NAQ-R, as this analysis has the advantage of handling both skewed data as well as
data which could be considered as categorical in nature. Highly skewed responses
with ‘‘never’’ as the most often chosen response pose a statistical problem in studies
on interpersonal aggression in the workplace, and especially so when measuring
bullying and harassment which by their very nature should be an infrequent
phenomenon. Although this skewedness may create some statistical problems, such a
result in itself also emphasizes the validity of the scale as it intends to measure
experiences outside or at the outer limits of the normal range of social interaction at
work. Therefore, the NAQ-R’s items are indicators of inappropriate behaviours that
one may experience, but normally should not experience at work, at least not in a
regular or systematic way.

Methodological limitations
The present study builds on self-report data obtained by means of a cross-sectional
design, which prevents us from drawing firm conclusions on the cause and effect
directions of the observed relationships. Yet, this does not question the fact that
NAQ-R showed the expected relationships with other relevant measures. Second, the
relationships may be inflated due to common source of variance, thus the observed
correlations between NAQ-R and the other related measures may be due to
dispositional factors (Keashly et al., 1994). However, we have good reasons to
believe that the latter is not the case. First, the strengths of the associations between
NAQ-R and the other measures reported in this study varied considerably, from a
correlation of .13 for sickness absenteeism to .58 for authoritarian leadership,
seriously questioning such an explanation. In addition, the strongest associations
 40 S. Einarsen et al.

were consistently found with other measures of social interaction at work, indicating
the discriminatory power of these analyses. Supporting such an interpretation, the
NAQ-R’s items form separate factors with similar, but still unique patterns of
relationships with other variables. Second, a range of earlier studies (some of which
have used the original NAQ), have shown that dispositional and attitudinal factors
do not explain away relationships between exposure to bullying behaviours, on the
one hand, and ratings of health and other critical job-related variables on the other
(e.g. Donovan, Drasgow, & Munson, 1998; Mikkelsen & Einarsen, 2002). Third, and
as has previously been argued, the NAQ-R contains well specified observable
behaviour with low requirements for participants’ cognitive and emotional proces-
sing when responding, thus reducing the influence of attitudinal and dispositional
factors. The use of well specified and observable behaviours in the items should also
to some extent reduce the possibility of reporting artefacts in the scale, e.g. of under
or over reporting, which may be a problem in sensitive topics such as bullying.
Downloaded by [University of Otago] at 18:23 20 March 2016

The NAQ-R was developed with the aim of creating a culturally anchored
measurement of bullying in Anglo-American cultures. Previous research on bullying
or similar concepts indicates a large degree of overlap of empirical findings across
cultures (see Einarsen et al., 2003). Still, important cultural differences manifesting
themselves in organizational behaviour and practices between different countries
do remain, with implications for the meaning of individual instrument items,
potentially both in the selection of items as well as in their wording. Consequently,
studies must be conducted in order to test the applicability of the NAQ-R across
cultural settings. For this purpose, the NAQ-R has recently been translated into a
number of languages and is currently being tested in studies both within and outside
the Anglo-American world (e.g. Lutgen-Sandvik, Tracy, & Alberts, 2007; Nielsen
et al., 2008). Yet further work needs to be done regarding the applicability of the
scale in different cultures, necessitating the need for systematic cross-cultural
validation studies.

Practical application of the NAQ-R

With a total of 22 short and easy-to-read items, thus easing the mental demands on
respondents, the technical application of the NAQ-R appears to be fairly simple. As
survey length is generally an issue in most organizational studies, the NAQ-R
appears to be a comprehensible, yet short and valid instrument. The scale
emphasizes experiences within the last 6 months, as recommended by Arvey and
Cavanaugh (1995) in their review of methodological problems in surveys of sexual
harassment. Using such a relatively short time-frame ensures the measurement of
repeated and on-going experiences, whilst simultaneously making responses less
vulnerable to recall problems, memory biases and distortions (Hoel, Rayner, &
Cooper, 1999). Measuring exposure to a range of repeated and systematic behaviours
happening very often over a long time period of course has the downside of not
tapping exactly who did what to whom in which situation, which may be possible in
measurements with a focus on infrequent aggression happening over a shorter time
period. Yet, this shortcoming may be overcome by also including a self-labelling
approach where respondents are presented with a definition of bullying before being
asked whether or not they perceive themselves as victims of bullying according to this
definition, as was the case in the present study (see also Einarsen & Skogstad, 1996;
 Work & Stress 41

Nielsen et al., 2008; Salin, 2001). While the approach used in the NAQ-R has the
potential of measuring a wide range of experiences where bullying may be seen to
exist on a continuum from ‘‘not exposed at all’’ to ‘‘highly exposed,’’ the self-labelling
approach, treating bullying as an ‘‘either-or’’ phenomenon, provides more con-
servative estimates of bullying generally limited to the more severe cases of perceived
victimization (Zapf et al., 2003). Combining the two approaches, as recommended
(Salin, 2001), opens many possibilities. First, following the self-labelling approach,
one may include items on who are the main perpetrators, the total duration of the
bullying situation, and so on (see also Einarsen & Skogstad, 1996; Mikkelsen &
Einarsen, 2001), which the NAQ-R does not provide in itself. Second, combining the
two approaches, the responses on the 22 items may be used to validate or qualify
subjective claims of being a target by providing examples in behavioural terms of
such experiences (Salin, 2001). When employing the 22 items of the NAQ-R in
combination with a self-labelling measure the NAQ-R is construed in accordance
Downloaded by [University of Otago] at 18:23 20 March 2016

with the views of researchers of sexual harassment (e.g. Arvey & Cavanaugh, 1995;
Fitzgerald & Shullmann, 1993) who argue for separating the two methods in the
questionnaire, allowing for participants to respond to the NAQ-R before being asked
to label their experience as bullying or not. Thus, a notable strength of the NAQ-R is
that it allows for the estimation of the prevalence of bullying without forcing
respondents to label themselves as targets.
When estimating the prevalence of bullying from the responses to the 22 NAQ-R
items, a method thought to provide a truer estimate of bullying than the self-labelling
approach (Notelaers et al., 2006), one may either use a statistical approach
employing latent class cluster analysis, as was done in this study, or the more
commonly used operational criteria (see also Nielsen et al., 2008, for a comparison).
The latter approach builds on the work of Leymann (1990, 1997) who defined targets
of bullying as those respondents who reported being subjected to at least one
negative act specified in the Leymann Inventory of Psychological Terror (LIPT) on
at least a weekly basis for a period of 6 months. Mikkelsen and Einarsen (2001),
applying the original NAQ, suggested using regular exposure to two negative acts as
a criterion of bullying, first as some acts may be particularly prevalent in certain
work settings and, second, as this criterion overlaps to a greater extent with the self-
labelling method. A range of studies have used such a cut-off score to differentiate
between targets and non-targets of bullying (Einarsen & Raknes, 1997; Lutgen-
Sandvik, Tracy, & Alberts, 2007; Niedhammer, David, & Degioanni, 2006; Salin,
2001).Yet, in principle, both these criteria and the self-labelling approach reduce
bullying to an ‘‘either-or’’ phenomenon. As bullying is considered to be a gradually
escalating process, taking a snapshot of this ongoing process is always a reduction of
the very nature of bullying as well as of its multi-dimensionality. The latent class
cluster analysis employed in our study shows exposure to bullying to be a
multifaceted phenomenon regarding both type and severity of exposure. Therefore,
in this respect we will argue that the latent class cluster approach provides a more
valid estimation (Nielsen et al., 2008; Notelaers et al., 2006). However, for practical
purposes in organizational settings where the application of such an advanced
approach is not feasible, the operational criterion approach may well be used.
To strengthen the applicability and practical value of the NAQ-R for researchers
and practitioners alike, data from various studies using the scale ought to be
collected and norms estimated accordingly. With access to normative data for the
 42 S. Einarsen et al.

general population as well as for particular countries, for particular demographic

groups, occupations and industries, the NAQ-R may become a useful tool in the
identification of high-risk groups and in benchmarking exercises. With access to a
valid, reliable and easy-to-use instrument, the NAQ-R could play an important role
in problem identification and measurement of bullying, thus hopefully contributing
to identifying, controlling and ultimately reducing the problem of workplace
bullying.

References
Arvey, R.D., & Cavanaugh, M.A. (1995). Using surveys to assess the prevalence of sexual
harassment: some methodological problems. Journal of Social Issues, 51(1), 39
52.
Bentler, P.M., & Bonnet, D.G. (1980). Significance tests and goodness of fit in the analysis of
covariance structures. Psychological Bulletin, 88, 588
606.
Downloaded by [University of Otago] at 18:23 20 March 2016

Bentler, P.M., & Chou, C.P. (1987). Practical issues in structural modeling. Sociological
Methods & Research, 16(1), 78
117.
Björkqvist, K., .Österman, K., & Hjeltbäck, M. (1994). Aggression among university
employees. Aggressive Behavior, 20, 173
184.
Brodsky, C.M. (1976). The harassed worker. Toronto: Lexington Books.
Byrne, B.M. (2002). Structural equation modeling with AMOS. Mahwah, NJ: Lawrence
Erlbaum.
Cooper, C.L., Sloan, S.J., & Marshall, J. (1988). Occupational stress indicator. Windsor:
NFER-Nelson.
Dawn, J., Cowie, H., & Ananiadou, K. (2003). Perceptions and experience of workplace
bullying in five different working populations. Aggressive Behavior, 29, 489
496.
Donovan, M.A., Drasgow, F., & Munson, L.J. (1998). The perceptions of Fair Interpersonal
Treatment Scale: Development and validation of a measure of interpersonal treatment in
the workplace. Journal of Applied Psychology, 83(5), 683
692.
Eid, M., Langeheine, R., & Diener, E. (2003). Comparing typological structures across
cultures by multigroup latent class analysis: a primer. Journal of Cross-Cultural Psychology,
34(2), 195
210.
Einarsen, S. (1999). The nature and causes of bullying at work. International Journal of
Manpower, 20, 16
27.
Einarsen, S., & Hoel, H. (2001). The Negative Acts Questionnaire: Development, validation and
revision of a measure of bullying at work. Oral presentation at the 10th European Congress
on Work and Organisational Psychology, Prague, May.
Einarsen, S., Hoel, H., Zapf, D., & Cooper, C.L. (Eds.). (2003). Bullying and emotional abuse in
the workplace. International perspectives in research and practice. London: Taylor & Francis.
Einarsen, S., & Raknes, B.I. (1991). Mobbing i arbeidslivet [Bullying in working life]. Bergen:
Institutt for samfunnspsykologi, Universitetet i Bergen.
Einarsen, S., & Raknes, B.I. (1997). Harassment in the workplace and the victimization of
men. Violence and Victims, 12(3), 247
263.
Einarsen, S., & Skogstad, A. (1996). Bullying at work: Epidemiological findings in public and
private organizations. European Journal of Work and Organizational Psychology, 5, 185
201.
Fitzgerald, L.F., & Shullman, S.L. (1993). Sexual harassment: a research analysis and agenda
for the 1990s. Journal of Vocational Behavior, 42(1), 5
27.
Grayson, D.A., Bridges, K., Duncanjones, P., & Goldberg, D.P. (1987). The relationship
between symptoms and diagnoses of minor psychiatric-disorder in general-practice.
Psychological Medicine, 17(4), 933
942.
Hoel, H. (2002). Bullying at work in Great Britain. Unpublished PhD thesis, University of
Manchester Institute of Science and Technology, Manchester.
 Work & Stress 43

Hoel, H., Cooper, C.L., & Faragher, B. (2001). The experience of bullying in Great Britain:
The impact of organizational status. European Journal of Work and Organizational
Psychology, 10, 443
465.
Hoel, H., Cooper, C.L., & Faragher, B. (2004). Bullying is detrimental to health, but all
bullying behaviours are not necessarily equally damaging. British Journal of Guidance and
Counselling, 32, 367
387.
Hoel, H., Rayner, C., & Cooper, C.L. (1999). Workplace bullying. In C.L. Cooper &
I.T. Robertson (Eds.), International review of industrial and organizational psychology
(pp. 195
230). Chichester: John Wiley & Sons.
Hutchinson, M., Vickers, M.H., Jackson, D., & Wilkes, L. (2006). Workplace bullying in
nursing: Towards a more critical organisational perspective. Nursing Inquiry, 13(2), 118
126.
Jöreskog, K. (2005). Structural equation modeling with ordinal variables using LISREL.
Chicago: Scientific Software International. Retrieved March 10, 2009, from http://www.
ssicentral.com/lisrel/advancedtopics.html
Jöreskog, K.G., & Sörbom, D. (1993). Lisrel-8 (user’s manual). Chicago: Scientific Software
Downloaded by [University of Otago] at 18:23 20 March 2016

International.
Jöreskog, K. Sörbom, D. &, du Toit, S. (2007). LISREL 8.88. Chicago: Scientific Software
International.
Keashly, L., & Harvey, S. (2005). Emotional abuse in the workplace. In S. Fox & P.E. Spector
(Eds.), Counterproductive behavior. Investigations of actors and targets (pp. 201
235).
Washington, DC: American Psychological Association.
Keashly, L., & Neuman, J. (2004). Bullying in the workplace. Its impact and management.
Employee Rights and Employment Policy Journal, 8(2), 335
373.
Keashly, L., Trott, V., & MacLean, L.M. (1994). Abusive behavior in the workplace: A
preliminary investigation. Violence & Victims, 9, 125
141.
Kelloway, E.K., Sivanathan, N., Francis, L., & Barling, J. (2004). Poor leadership. In
J. Barling, E.K. Kelloway & M.R. Frone (Eds.), Handbook of workplace stress (pp. 89
112).
Thousand Oaks, CA: Sage.
Langeheine, R., Pannekoek, J., & VandePol, F. (1996). Bootstrapping goodness-of-fit measures
in categorical data analysis. Sociological Methods & Research, 24(4), 492
516.
Leymann, H. (1990). Mobbing and psychological terror at workplaces. Violence and Victims,
5(2), 119
126.
Leymann, H. (1996). The content and development of mobbing at work. European Journal of
Work and Organizational Psychology, 5, 165
184.
Leymann, H. (1997). Explanation of the operation of the LIPT Questionnaire (Leymann
Inventory of Psychological Terror). Unpublished manuscript.
Lim, S., & Cortina, L.M. (2005). Interpersonal mistreatment in the workplace: The interface
and impact of general incivility and sexual harassment. Journal of Applied Psychology, 90(3),
483
496.
Lutgen-Sandvik, P., Tracy, S.J., & Alberts, J.K. (2007). Burned by bullying in the American
workplace: Prevalence, perception, degree, and impact. Journal of Management Studies, 44,
837
862.
Magidson, J., & Vermunt, J.K. (2001). Latent class factor and cluster models, bi-plots, and
related graphical displays. Sociological Methodology, 31, 223
264.
Magidson, J. & Vermunt, J.K. (2004). Latent class models. In D. Kaplan (Ed.), The Sage
handbook for quantitative methodology (pp. 175
198). Thousand Oaks: Sage Publications.
Matthiesen, S.B., & Einarsen, S. (2001). MMPI-2 configurations among victims of bullying at
work. European Journal of Work and Organizational Psychology, 32, 335
356.
Mikkelsen, E.G., & Einarsen, S. (2001). Bullying in Danish worklife: Prevalence and health
correlates. European Journal of Work and Organizational Psychology, 10, 393
414.
 44 S. Einarsen et al.

Mikkelsen, E.G., & Einarsen, S. (2002). Relationship between exposure to bullying at work
and psychological and psychosomatic health complaints: The role of state negative
affectivity and generalized self-efficacy. Scandinavian Journal of Psychology, 43, 397
405.
Niedhammer, I., David, S., & Degioanni, S. (2006). Economic activities and occupations at
high risk for workplace bullying: results from a large-scale cross-sectional survey in the
general working population in France. International Archives of Occupational and
Environmental Health, 80(4), 346
353.
Niedl, K. (1996). Mobbing and well-being: Economic and personnel development implica-
tions. European Journal of Work and Organizational Psychology, 5(2), 239
249.
Nielsen, M.B., Skogstad, A., Matthiesen, S.B., Glasø, L., Aasland, M.S. Notelaers, G., et al.
(2008). Bullying at work: Epidemiological findings from a representative study
of Norwegian employees. European Journal of Work and Organizational Psychology, 18,
81
101.
Notelaers, G., Einarsen, S., De Witte, H., & Vermunt, J. (2006). Measuring exposure to
bullying at work: The validity and advantages of the latent class cluster approach. Work &
Downloaded by [University of Otago] at 18:23 20 March 2016

Stress, 20(4), 288
301.

O’Moore, M., Seigne, E., McGuire, L., & Smith, M. (1998). Bullying at work in Ireland.
Journal of Occupational Health and Safety, 19(6), 569
574.
Rayner, C., & Keashly, L. (2005). Bullying at work: A perspective from Britain and North
America. In S. Fox & P.E. Spector (Eds.), Counterproductive behavior. Investigations of
actors and targets (pp. 271
296). Washington, DC: American Psychological Association.
Salin, D. (2001). Prevalence and forms of bullying among business professionals. A
comparison of two different strategies for measuring bullying. European Journal of Work
and Organizational Psychology, 10(4), 425
441.
Vartia, M. (1991). Bullying at workplaces. Paper presented at Towards the 21st century. Work
in the 1990s. International Symposium on Future trends in the Changing Working Life,
Helsinki.
Vermunt, J.K. (2001). The use of restricted latent class models for defining and testing
nonparametric and parametric IRT models. Applied Psychological Measurement, 25,
283
294.
Vermunt, J.K., & Magidson, J. (2003). Latent GOLD 3.0 User’s Guide. Belmont, MA:
Statistical Innovations Inc.
Williams, S., & Cooper, C.L. (1998). Measuring occupational stress: Development of the
pressure management indicator. Occupational Health Psychology, 306
321.
Zapf, D. (1999). Organizational work group related and personal causes of mobbing/bullying
at work. International Journal of Manpower, 20(1/2), 70
85.
Zapf, D., & Einarsen, S. (2005). Mobbing at work: Escalated conflicts in organizations. In
S. Fox & P.E. Spector (Eds.), Counterproductive behavior. Investigations of actors and targets
(pp. 237
270). Washington, DC: American Psychological Association.
Zapf, D., Einarsen, S., Hoel, H., & Vartia, M. (2003). Empirical findings on bullying in the
workplace. In S. Einarsen, H. Hoel, D. Zapf & C.L. Cooper (Eds.), Bullying and emotional
abuse in the workplace. International perspectives in research and practice (pp. 103
126).
London: Taylor & Francis.