You are on page 1of 16

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/256425091

The Missing Link: Delayed Emotional Development Predicts Challenging


Behavior in Adults with Intellectual Disability

Article  in  Journal of Autism and Developmental Disorders · September 2013


DOI: 10.1007/s10803-013-1933-5 · Source: PubMed

CITATIONS READS

60 1,408

6 authors, including:

Tanja Sappok Jan Budczies


Königin Elisabeth Herzberge Hospital Charité Universitätsmedizin Berlin
82 PUBLICATIONS   1,120 CITATIONS    279 PUBLICATIONS   15,422 CITATIONS   

SEE PROFILE SEE PROFILE

Isabel Dziobek Sven Bölte


Humboldt-Universität zu Berlin Karolinska Institutet
229 PUBLICATIONS   9,558 CITATIONS    507 PUBLICATIONS   18,319 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Vietnamese Migrants in Germany View project

Hannover medical School, Center for Systems Neuroscience, PhD Project View project

All content following this page was uploaded by Sven Bölte on 29 April 2017.

The user has requested enhancement of the downloaded file.


J Autism Dev Disord (2014) 44:786–800
DOI 10.1007/s10803-013-1933-5

ORIGINAL PAPER

The Missing Link: Delayed Emotional Development Predicts


Challenging Behavior in Adults with Intellectual Disability
Tanja Sappok • Jan Budczies • Isabel Dziobek •
Sven Bölte • Anton Dosen • Albert Diefenbacher

Published online: 4 September 2013


Ó Springer Science+Business Media New York 2013

Abstract Individuals with intellectual disability (ID) Keywords Intellectual disability  Autism spectrum
show high rates of challenging behavior (CB). The aim of disorders  Adults  Emotional development 
this retrospective study was to assess the factors underlying Challenging behavior
CB in an adult, clinical ID sample (n = 203). Low levels
of emotional development (ED), as measured by the
Scheme of Appraisal of ED, predicted overall CB, specif- Introduction
ically irritability and self-injury, high unemployment and
low occupation rates, while severity of ID controlled for Individuals with intellectual disability (ID) show high point
ED did not. Autism was the only mental disorder associ- prevalence rates of mental health problems and challeng-
ated with overall CB, stereotypy, lethargy, and predicted ing behaviors, approximately 40 and 20 %, respectively
antipsychotic drug usage. Given the persistence and clini- (Cooper et al. 2007). In ID, the presence of autism spec-
cal significance of CB, evaluation of autism and ED may trum disorders (ASDs) and the severity of ID are the most
suggest priority areas for diagnostics and therapy, to pro- important predictors for referrals to psychiatrists and pre-
vide the prerequisites for participation in society and living scription of psychotropic drugs, as well as the predominant
up one’s potentials. causes of challenging behavior (Bhaumik et al. 2008; Es-
bensen et al. 2009; Tsakanikos et al. 2007; Witwer and
Lecavalier 2005). A multitude of studies have shown that
T. Sappok (&)  A. Diefenbacher adults with ID and ASD demonstrate high levels of chal-
Department of Psychiatry, Psychotherapy and Psychosomatics, lenging behavior (Hill and Furniss 2006; Holden and Gi-
Königin-Elisabeth-Herzberge Hospital, University Affiliated tlesen 2006; Hove and Havik 2010; Kraijer 2000;
Hospital of the Charité, Herzbergstr.79, 10365 Berlin, Germany
McCarthy et al. 2010; McClintock et al. 2003; Underwood
e-mail: tanja.sappok@t-online.de
et al. 2010). In adults, according to McCarthy et al. (2010),
J. Budczies challenging behavior is approximately four times more
Department of Pathology, Charité University Hospital, likely in ASD/ID combined than in those with ID only.
Charitéplatz 1, 10117 Berlin, Germany
Challenging behaviors are already increased in childhood,
I. Dziobek (Murphy et al. 2009) and tend to persist over the life span,
Cluster of Excellence Languages of Emotion, Freie Universität especially in individuals with ASD and low verbal abilities
Berlin, Habelschwerdter Allee 45, 14195 Berlin, Germany (Matson and Rivet 2008; Murphy et al. 2005). Challenging
behavior is associated with other psychiatric morbidity and
S. Bölte
Department of Women’s and Children’s Health, Center of is an important barrier for access to education, participation
Neurodevelopmental Disorders at Karolinska Institutet (KIND), in society, and living up one’s personal and professional
17176 Stockholm, Sweden potentials (De Bildt et al. 2005; Emerson et al. 2000; Felce
et al. 2009; Holden and Gitlesen 2006). A reduction of
A. Dosen
Department of Psychiatry, University Hospital, Radboud challenging behavior and an increase of adaptive behavior
University, Comeniuslaan 4, 6525 HP Nijmegen, Netherlands seem to be important in improving mental health and the

123
J Autism Dev Disord (2014) 44:786–800 787

overall quality of life in affected individuals (Ciarrochi and for the onset of the concept of ‘‘self’’ and the formation of
Scott 2006; Rey et al. 2012). Finally, ASD-related chal- personality structures (Dosen 1997). This emotional mat-
lenging behaviors have been consistently associated with uration process is assumed to be a product of multiple
the psychological well-being and emotional problems of internal and external factors. First, cerebral circuits
caregivers (Lounds et al. 2007; Totsika et al. 2011). The responsible for survival (e.g., feeding, reproduction, or
mechanisms that lead to high levels of challenging social contact) are conducive for basic emotions (Damasio
behavior among ASD/ID individuals yet are poorly 2010). Second, ED is intertwined and closely accompanied
understood (McCarthy et al. 2010). by cognitive developmental changes (Pessoa 2010; Piaget
The severity of challenging behaviors and psychopa- 1953). ‘Object constancy’, for example, is defined as an
thology rises with the increasing severity of ID and ASD- internal representation of the outside environment (Piaget
defining symptomatology (Matson and Rivet 2008; Matson 1953) and results in the formation of ‘emotional object
et al. 2009a, b; McCarthy et al. 2010). Moreover, as constancy’ (Mahler et al. 1975). Emotional object con-
challenging behavior changes and tends to decrease with stancy is essential for the sense of self-boundaries and for
age, developmental factors may play a role (De Bildt et al. internalization of the representation of ‘love objects’,
2005; Felce et al. 2009; Maskey et al. 2013; Murphy et al. leading to secure attachment and consequently individu-
2009; Totsika et al. 2011). Alteration in emotion-process- alization (Bowlby 1969). Third, ED is anchored in social
ing such as difficulties in perception, recognition, under- interaction. Mirroring the emotions of a child in a secure
standing, expression, and regulation of emotions could be setting further supports emotion regulation, expression, and
found in ASD (Begeer et al. 2008; Hobson 1986; Kasari modulation (Kernberg 2012). Finally, environmental fac-
et al. 2012). Hence, the quality and severity of challenging tors and sensory abilities necessary for an appropriate
behavior may depend not only on intellectual but also on perception of the environment may influence emotional
emotional skills changing over a lifetime. The effect of the reactivity and emotion regulation strategies (Aldao and
overall emotional development (ED) on challenging Nolen-Hoeksema 2012). Thus, a multitude of cognitive,
behavior has not yet been examined in individuals with ID. social, sensorimotor, and environmental factors closely
interact with and stimulate each other, resulting in the
Emotional Development concept of overall ED. Each level of ED is associated with
specific basic emotional needs and motivations, certain
In this study, we consider the ‘concept of ED’ in terms of a coping strategies, and adaptive behavior outcomes. Gen-
developmental approach (Dosen 2005; Greenspan 1979; erally, individuals with ID pass through the same periods of
Izard et al. 2006; Sroufe 2009), i.e., acquisition of emo- ED as typically developing children do, although with an
tional competencies following a progressive sequence of increased risk of delay and incompleteness (Greenspan
qualitative changes incorporating predominantly emotional 1997; Hodapp and Zigler 1995).
but also social, sensorimotor, and cognitive functions on Currently, in addition to the treatment of physical ill-
the background of normal development in infants. The nesses and changes in psychosocial environmental factors,
various components of ED reciprocally interact and stim- applied behavior analysis is the favored assessment and
ulate each other for a further maturation of the individual therapy for challenging behavior among ID individuals
and an optimal adaptation to the environment. While (Didden et al. 1999; Matson et al. 2011). Combining these
newborns already show, perceive, and respond to a range strategies with the developmental approach might lead to
of simple emotions, within the first year of life, the emo- an increase in emotional capacities and thus further
tional response is modulated by the affect of the interaction improve functional life skills, quality of life, and well-
partner (Bertin and Striano 2006; Stern 2007). Around the being (Rey et al. 2012).
second year of life, affective sharing arises in the context of On the background of the developmental understanding
‘joint attention’ (Kasari et al. 1990; Trevarthen 1980). of ED, Dosen (1997) designed the Scheme of Appraisal of
Gradually, emotional responses and emotion regulation Emotional Development (SAED) to assess emotional
become more complex and around the third year of life, maturity according to the normative trajectory of typically
children are able to manipulate the emotional states of developing children. The SAED is a semi-structured inter-
others, e.g., by teasing or cooperating behaviors (Jackson view of a significant other classifying the present level of
and Tisak 2001). Preschoolers can differentiate, regulate, development in 10 domains (cf. ‘‘Methods’’ section) and an
and integrate their affective states and have a basic insight overall level of ED. Table 1 summarizes these different ED
into the causes and consequences of emotions. Further levels, the age-equivalents of typically developing children,
progress in empathy, pro-social behaviors, and moral and the major achievements within the assigned periods.
thinking is observed in school-aged children (Rieffe et al. In summary, individuals with ID show high prevalence
2005). These age-appropriate changes of ED are the basis rates of challenging behaviors leading to a considerable

123
788 J Autism Dev Disord (2014) 44:786–800

Table 1 Levels of ED, age equivalents and developmental achievements according to the SAED
Level of ED Age Developmental achievements

1: Adaptation 0–6 months Homeostasis; integration of internal; e.g., sensory stimuli; bowel movements; and external stimuli; e.g.,
time; place; caregivers
2: Socialization 6–18 months Attachment; emotional security; confidence
3: Individuation 18–36 months Differentiation of self from others; autonomy; management of separation from a bonding person; object
permanence
4: Identification 3–7 years Ego-formation; ego-centered; identification with significant others; little abstract thinking; evolving social
skills; learn by experience
5: Reality 7–12 years Reflective thinking; emerging moral thinking and self-respect; ego-differentiation; reasoning
awareness

health burden, reduction of quality of life as well as par- completed by a psychologist (H.K.), the MOAS and the
ticipation in social life. The aim of the present study was remaining SAED assessments were evaluated by the
(A) to assess the impact of various mental disorders attending psychiatrist; the ABC was assessed by a care-
including ASD, the level of ED, the degree of ID, age, and giver, all not involved in the study. The study was con-
gender on the severity of challenging behavior and ducted according to the recommendations of the
aggression in adults with ID and (B) on outcome variables declaration of Helsinki.
such as living and working situation and on medication
usage. Participants

In total, 203 participants (64 females, 139 males) were


Methods included in the study. None of the participants lived in a
long-term psychiatric institution; all lived with their fam-
Setting and Design ilies or in residential homes with a maximum group size of
8 inhabitants. Inclusion criteria in this naturalistic study
The study was conducted at a psychiatric department in a design were age [18 years, intellectual disability of any
general hospital with a specialized in- and outpatient unit degree, hospital referral from 1/2008 to 6/2012, and stan-
for adults with ID. This specialized unit has the responsi- dardized behavioral assessment with the MOAS and the
bility to assess and treat all adults with ID and comorbid ABC for clinical purposes. The mean age was 35.8
mental or severe behavioral problems in Berlin (popula- (SD = 12.6) years; the degree of ID was mild (n = 39),
tion 3.5 million), Germany. Aggressive and challenging moderate (n = 89), and severe-profound (n = 75). Eighty-
behaviors were assessed with the Modified Overt Aggres- six individuals were diagnosed with additional ASD, 117
sion Scale (MOAS) and the Aberrant Behavior Checklist were not (non-ASD). Table 2 shows the baseline charac-
(ABC) in 203 adults with ID referred to the center between teristics of the total study sample, including ASD and non-
1/2008 and 6/2012. This cohort was comprehensively ASD individuals.
examined with regard to the presence of pervasive or
emotional developmental disorders (cf. diagnostic proce- Diagnostic Procedures
dures and test materials), mental health, physical comor-
bidities, medication, employment, and living situation. Assessment of Neurological and Mental Disorders
Assessments of developmental disorders were conducted
after remission of the acute exacerbation of the mental All individuals were thoroughly evaluated by a psychiatrist
illness in the in- or outpatient clinic. The impact of the trained in medical assessment and treatment of individuals
mental disorders, level of ED, severity of ID, and baseline with ID. Evaluation comprised the current and past medical
variables on problem behaviors and outcome variables history and a comprehensive psychiatric and physical
were calculated by correlation and regression analysis. examination. In case of diagnostic uncertainty, laboratory
All assessments were part of the clinical routine and examinations, brain imaging (cCT, cMRT), X-rays, elec-
data analysis performed retrospectively from the hospital troencephalography, neurophysiological measures, and
database. Evaluation of existing diagnostic data is based on clinical assessment by specialists, e.g., a neurologist,
the legal requirements in Berlin, Landeskrankenhausgesetz ophthalmologist, ear-nose-throat physician, or surgeons
§ 25.1, version 18.09.2011. All data was anonymized. were applied. Neurological disorders leading to functional
The ASD measures and most SAED evaluations were impairments of daily living skills were grouped according

123
J Autism Dev Disord (2014) 44:786–800 789

Table 2 Baseline characteristics of the study population


Characteristic Total sample ID & ASD ID, no ASD p

General Information, n (%) 203 (100 %) 86 (42.4 %) 117 (57.6 %)


Gender (male), n (%) 139 (68.5 %) 63 (73.3 %) 76 (65.0 %) .209
Age in years, M (SD) 35.79 (12.57) 34.97 (11.48) 36.40 (13.34) .423a
Severity of ID n (%) 203 (100 %) 86 (42.4 %) 117 (57.6 %) .001*,b
Mild 39 (19.2 %) 10 (11.6 %) 29 (24.8 %) .019
Moderate 89 (43.8 %) 33 (38.4 %) 56 (47.9 %) .178
Severe-profound 75 (36.9 %) 43 (50.0 %) 32 (27.4 %) .001
Level of ED n (%) 203 (100 %) 86 (42.4 %) 117 (57.6 %) \.0005*,b
SAED 1 (0–6 months) 13 (6.4 %) 10 (11.6 %) 3 (2.6 %) .009
SAED 2 (6–18 months) 30 (14.8 %) 19 (22.1 %) 11 (9.4 %) .012
SAED 3 (18–36 months) 68 (33.5 %) 40 (46.5 %) 28 (23.9 %) .001
SAED 4 (3–7 years) 50 (24.6 %) 13 (15.1 %) 37 (31.6 %) .007
SAED 5 (7–12 years) 42 (20.7 %) 4 (4.7 %) 38 (32.5 %) \.0005
SAED level, M (SD) 3.40 (0.85) 2.88 (0.96) 3.79 (0.77) \.0005
Neurological disorders N (%) 203 (100 %) 86 (100 %) 117 (100 %)
Hearing deficits 11 (5.4 %) 5 (5.8 %) 6 (5.1 %) .831
Visual deficits 26 (12.8 %) 10 (11.6 %) 16 (13.7 %) .666
Movement disorder 24 (11.8 %) 5 (5.8 %) 19 (16.2 %) .023*
Seizure disorder 50 (24.6 %) 21 (24.4 %) 29 (24.8 %) .952
Mental disorders, N (%) 203 (100 %) 86 (100 %) 117 (100 %)
Dependency disorders (F1x.x) 12 (5.9 %) 7 (8.1 %) 5 (4.3 %) .248
Schizophrenia (F2x.x) 52 (25.6 %) 31 (36.0 %) 21 (17.9 %) .004*
Mood disorders (F3x.x) 64 (31.5 %) 23 (26.7 %) 41 (35.0 %) .209
Neurotic, stress-related and somatoform disorders (F4x.x) 35 (17.2 %) 17 (19.8 %) 18 (15.4 %) .414
Personality disorders (F6x.x) 12 (5.9 %) 0 (0 %) 12 (10.3 %) .002*
Medication, n (%) 202 (100 %) 86 (100 %) 116 (100 %)
Antipsychotics-high potency 116 (57.4 %) 60 (69.8 %) 56 (48.3 %) .002*
Antipsychotics-low potency 89 (44.1 %) 47 (54.7 %) 42 (36.2) .009*
Antidepressants 45 (22.3 %) 16 (18.6 %) 29 (25.0 %) .280
Anticonvulsive drugs 77 (38.1 %) 32 (37.2 %) 45 (38.8 %) .819
Benzodiazepines 32 (15.9 %) 18 (20.9 %) 14 (12.2 %) .093
Living situation, n (%) 200 (100 %) 86 (100 %) 114 (100 %) .423c
Alone 3 (1.5 %) 0 3 (2.6 %) .261c
Living with family 23 (11.5 %) 10 (11.6 %) 13 (11.4 %) .961
Residential home 174 (87.0 %) 76 (88.4 %) 98 (86.0 %) .616
Employment, n (%) 197 (100 %) 85 (100 %) 112 (100 %) .001c,*
Unemployed 40 (20.3 %) 24 (28.2 %) 16 (14.3 %) .016*
In School 6 (3.0 %) 5 (5.9 %) 1 (0.9 %) .087c
In Rehab program 67 (34.0 %) 32 (37.6 %) 35 (31.2 %) .348
Sheltered work 80 (40.6 %) 22 (25.9 %) 58 (51.8 %) \.0005*
Retirement 4 (2.0 %) 2 (2.4 %) 2 (1.8 %) 1.000
ABC, N (%) 203 (100 %) 86 (100 %) 117 (100 %)
Total score, M (SD) 46.83 (26.48) 56.55 (26.51) 39.68 (24.18) .001*,d
Irritability, M (SD) 14.65 (9.44) 16.65 (9.45) 13.17 (9.19) .008*,d
Lethargy, M (SD) 11.39 (9.07) 14.45 (9.48) 9.15 (8.08) .001*,d
Stereotypy, M (SD) 4.69 (4.78) 6.60 (4.71) 3.13 (4.29) .001*,d
Hyperactivity, M (SD) 13.16 (9.72) 15.67 (10.96) 11.32 (8.26) .007*,d
Inappropriate speech, M (SD) 3.04 (3.24) 3.20 (3.22) 2.92 (3.26) .516d

123
790 J Autism Dev Disord (2014) 44:786–800

Table 2 continued
Characteristic Total sample ID & ASD ID, no ASD p

MOAS, N (%) 203 (100 %) 86 (100 %) 117 (100 %)


Total score, M (SD) 4.93 (3.51) 4.62 (3.67) 5.35 (3.25) .110d
Aggression: verbal, M (SD) 1.27 (1.23) 1.10 (1.22) 1.39 (1.23) .047d
Aggression: objects, M (SD) 1.30 (1.22) 1.40 (1.27) 1.22 (1.17) .365d
Aggression: self, M (SD) 1.16 (1.17) 1.45 (1.09) 0.94 (1.18) .001*,d
Aggression: others, M (SD) 1.29 (1.28) 1.46 (1.26) 1.16 (1.28) .075d
Differences between ASD and non-ASD individuals were calculated by 2-sided Pearson’s v2 tests except a indicates t Test, b indicates v2 test for
trends; c indicates Fisher0 s Exact Test; d indicates Mann–Whitney Test
SD standard deviation
* Indicates p \ .05; p corrected for multiple testing analysis (ABC: p \ .010, MOAS: p \ .0125)

to hearing or visual deficits and movement disorders conducted (Lord et al. 1989, 1994, 2000). The ADOS is a
including for example spasticity, dystonia, cerebral palsy, semi-structured observational instrument assessing social
or extrapyramidal movement disorders. Seizure disorders and communicative abilities in individuals suspected of
were assessed and documented when present within the having ASD (Lord et al. 1989, 2000). The ADI-R is a semi-
past 10 years. Mental disorders were diagnosed based on structured parental interview assessing social reciprocity,
the ICD-10 diagnostic criteria and grouped according to communication and restrictive, repetitive behaviors and
the second variable, i.e., dependency disorders (F1x.x), interests over a lifetime (Lord et al. 1994). Recently, the
schizophrenia (F2x.x), mood disorders (F3x.x), neurotic, validity of these measures has been demonstrated in the
stress-related and somatoform disorders (F4x.x), person- adult ID population (Sappok et al. 2013). The final diag-
ality disorders (F6x.x), and ASD (F8x.x). The single indi- nostic classification was assigned by a second consensus
vidual could have no, only one, or more than one mental or conference.
neurological disorder.
Test Materials
Diagnosing ASD
Assessment of ED
Clinical ICD-10 diagnoses for autism and atypical autism
(F84.0/F84.1) were based on all available information from The SAED by Dosen (1997) is an interview conducted by a
the current and past medical history, a comprehensive psychologist or psychiatrist trained in developmental psy-
psychiatric and physical examination, a video-based chology with a close caregiver after remission of the acute
behavior analysis, and standardized psychodiagnostic exacerbation of the psychiatric illness. The SAED evalu-
assessments such as the Social Communication Question- ates the achieved developmental level in 10 basic aspects
naire (SCQ) and the Scale for Pervasive Developmental of ED:
Disorders in Mentally Retarded Persons (PDD-MRS;
Kraijer and Melchers 2003; Rutter et al. 2001). The SCQ is 1. Dealing with the own body
a categorical scale consisting of 40 items designed to 2. Interaction with a caregiver
screen for ASD in children and adolescents (4–18 years). 3. Experience of the self
The PDD-MRS is a diagnostic interview conducted by a 4. Object permanence
trained psychologist with a professional caregiver evalu- 5. Anxiety
ating autistic and problem behaviors of the patient in situ- 6. Interaction with peers
7. Handling of material objects
ations of daily living. It consists of 12 categorical items,
8. Verbal communication
some of which are weighted. After these screening
9. Affect differentiation
assessments were completed, a multidisciplinary case
10. Aggression regulation
conference was conducted that consisted of at least one
psychiatrist and one psychologist. In case of a concordant According to the 5 possible developmental levels (cf.
appraisal for or against ASD, no more assessments were Table 1), values from 1 to 5 can be scored on this ordinal-
applied. In case of diagnostic uncertainty, the Autism scaled measure. Certain statements of observed behavior
Diagnostic Observation Schedule (ADOS; n = 48) and/or that are typical for the developmental level guide the
the Autism Diagnostic Interview-R (ADI-R; n = 24) were interview. Statements in level one (adaptation) of the first

123
J Autism Dev Disord (2014) 44:786–800 791

domain (dealing with his/her own body) are for example: 1. speech) that have been found to have high internal con-
engages with own body (observing, playing, sucking, sistency with Cronbach’s alpha ranging from .86 to .94.
masturbating, picking, and injuring.), 2. moves in a ste-
reotypical manner (engages in particular movements in
the same manner repetitively), and 3. passive (shows no Assessment of Aggression
initiative). The behaviors of a person that can be observed
most often mark the developmental level at which the The secondary outcome measure to quantify aggression
person is functioning in a certain domain. In this study, was the MOAS, an instrument often used and validated in
the overall judgment of the developmental level was individuals with psychiatric illnesses (Knoedler 1989;
according to the mean level reached over the 10 different Yudofsky et al. 1986). Oliver et al. (2007) showed good
domains. The SAED interview lasted approximately reliability results in individuals with ID (interclass corre-
20–30 min. The scheme has shown adequate inter-rater lation coefficient = 0.93). It consists of 4 items (verbal
reliability (kappa = .75), internal consistency (Cronbach’s aggression, aggression to objects, self, and others) on a
alpha = .96) and concurrent validity with the Vineland five-point ordinal scale. It was applied during the first
Adaptive Behavior Scales (r = .51, p \ .002; La Malfa assessment after admission to hospital.
et al. 2009).
Assessment of Outcome Variables
Severity of ID
Living arrangements and employment were evaluated. The
Participants were grouped as having a mild, moderate, or living situation was categorized as either ‘alone’, ‘living
severe/profound ID according to the current/latest medical with family’, or ‘residential home’. With respect to
record. In accordance with the ICD-10, the IQ cut points employment, grouping possibilities were ‘unemployment’,
used were 50–70 (mild ID), 35–50 (moderate ID), and ‘school’, ‘rehab program’, ‘sheltered work’, or ‘retirement’.
below 35 (severe to profound ID). Assignment was based
upon the results of the Disability Assessment Schedule
(DAS) in 132 clients (Holmes et al. 1982; Meins and Data Analysis
Süssmann 1993). The DAS proofed high reliability and
good validity results as it highly correlated with the Col- (A) First, a correlation analysis (Pearson’s r) of all possible
oured Progressive Matrices (r = .75) and the Columbia predictors including several mental and neurological
Mental Maturity Scale (r = .77) (Holmes et al. 1982; disorders, age, gender, and level of ID and ED with the
Meins and Süssmann 1993). In the absence of a DAS, ABC and MOAS total scores and subscales was run.
clients were categorized according to their daily living and The multiple testing problem connected with simulta-
communication abilities by a physician experienced in the neous testing of 14 variables was corrected by the
field of ID. In case of uncertainties in ID assignments, Bonferroni method (p \ .05/14 = .004). Secondly,
structured IQ assessment was applied in selected patients, multivariate linear regression analysis was computed
e.g., the Coloured Progressive Matrices, the Kaufmann- for the ABC and MOAS total and subscale scores with
Assessment-Batteries for Children, the Snijders-Omen- forced entry of all variables for which substantial cor-
Nonverbal Intelligence Test, or the Wechsler-Intelligence relations have been found. Finally, the analysis was
Test for Adults. rerun to get an insight in the added variance of each
factor. In Model 1 relevant baseline variables were
Assessment of Challenging Behavior included and each correlating factor of interest was
added in a separate Model, respectively.
Challenging behavior was measured by the ABC, a widely (B) First, multivariate logistic regression analysis was
used and validated instrument for assessing behavior computed for living and working situations, which
problems in children and adults with ID, during the first provided at least 5 % of individuals in each group, i.e.,
assessments after hospital referral (Aman et al. 1985). The living with parents, living in a residential home,
ABC is a rating scale with 58 items rated on a four-point unemployment, rehab program, or sheltered work.
ordinal scale. It is completed by a caregiver who knows the The following predictors were included: Mental and
target individual well. It has demonstrated good discrimi- neurological disorders, medication, ABC and MOAS
nant and convergent validity and test–retest reliability total scores, severity of ID and level of ED, age and
(r = .96 to .99), but low inter-rater reliability (r = .55 to gender. Second, we computed a multivariate logistic
.69). It consists of five subscales (irritability, lethargy, regression analysis for medication (dependent variable)
stereotyped behavior, hyperactivity, and inappropriate and several mental disorders (independent variables).

123
792

123
Table 3 Correlation analysis of possible predictors for challenging and aggressive behaviors
ABC MOAS
ABC total Irritability Lethargy Stereotypy Hyper- Inappropr- MOAS total Aggression Aggression Aggression Aggression
activity iate speech verbal objects self others
r p r p r p r p r p r p r p r p r p r p r p

Age 2.25 .0005 2.24 .001 n.s. n.s. n.s. n.s. 2.30 .0005 n.s. n.s. 2.25 .0005 n.s. n.s. -.20 .005 2.21 .003 2.24 .001
Male gender n.s. n.s. n.s. n.s. .18 .011 .21 .003 n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. -.15 .037 n.s. n.s.
Severity of ID .16 .027 n.s. n.s. n.s. n.s. .25 .0005 .22 .002 2.20 .004 n.s. n.s. n.s. n.s. n.s. n.s. .18 .011 n.s. n.s.
Level of ED 2.28 .0005 2.20 .004 2.22 .002 2.30 .0005 2.28 .0005 .21 .003 n.s. n.s. .22 .002 n.s. n.s. 2.36 .0005 n.s. n.s.
Hearing deficits n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. -.14 .047
Visual deficits n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s.
Movement disorder n.s. n.s. n.s. n.s. n.s. n.s. -.14 .048 n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s.
Seizure disorder n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s.
Dependency disorders n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. .15 .039 .19 .008
Schizophrenia n.s. n.s. n.s. n.s. .17 .018 .15 .032 n.s. n.s. n.s. n.s. n.s. n.s. -.19 .007 n.s. n.s. n.s. n.s. n.s. n.s.
Mood disorders n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. .17 .02 n.s. n.s. n.s. n.s. n.s. n.s.
Neurotic disorders n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s.
(F4x.x)
Personality disorders n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. .20 .004 n.s. n.s. n.s. n.s. n.s. n.s.
ASD .32 .0005 .18 .009 .29 .0005 .36 .0005 .22 .0001 n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. .22 .002 n.s. n.s.
n.s. = no significant correlation; bold letters: p \ .05 adjusted for Bonferroni: .05/14 = .004
J Autism Dev Disord (2014) 44:786–800
J Autism Dev Disord (2014) 44:786–800 793

The study population was described in detail. The differences disorders, levels of functioning (ID and ED), and baseline
between the ASD and non-ASD groups with respect to variables (age, gender) were included in a Pearson’s r cor-
demographics and clinical characteristics were evaluated relation analysis with the ABC and MOAS total and sub-
using v2 tests, t-tests, and Fisher0 s Exact Test, where appro- scale scores. The results are shown in Table 3.
priate. ASD/non-ASD group differences for the ABC and With respect to the mental and neurological disorders
MOAS sum scores and various subscales were evaluated studied, ASD revealed associations with the ABC sum
using a Mann–Whitney test. The multiple testing problem score and various subscales, e.g., lethargy, stereotypy, and
connected with simultaneous testing of the five (ABC) or four hyperactivity. Moreover, ASD correlated with aggression
(MOAS) items was corrected by the Bonferroni method to the self. Beside from ASD, only personality disorders
(p \ .05/5 = .01 and p \ .05/4 = .0125, respectively). showed a correlation with the MOAS subscale ‘verbal
aggression’. Regarding the level of functioning, ED dis-
played several negative associations with the ABC total
Results and most of its subscales and ‘aggression to self’ indicating
more severe problem behaviors at lower levels of ED. The
Predictors for Challenging and Aggressive Behaviors ABC subscale ‘inappropriate speech’ and the MOAS sub-
scale ‘verbal aggression’, however, showed an inverse
Correlation Analysis of Possible Predictors pattern. Thus, individuals with higher levels of ED exhib-
for Challenging or Aggressive Behaviors ited more verbal challenging and aggressive behaviors.
Correspondingly, the severity of ID was negatively asso-
All possible predictors for challenging or aggressive ciated with ‘inappropriate speech’. Additionally, ‘hyper-
behaviors including several mental and neurological activity’ and ‘stereotypy’ were strongly correlating with

Table 4 Regression analysis predicting the effect of correlating factors on challenging behaviors and aggression
Dependent variable Predictor Multivariate regression—forced entry Regression analysis controlled for age/gender where
necessary
R2 b [95 % CI] p R2 b [95 % CI] p

ABC total Age .17 -0.46 [-0.73; -0.19] .001 .06 -0.52 [-0.80; -0.23] .0005
ED -3.84 [-7.55; -0.14] .042 .12 -6.56 [-9.97; -3.15] .0005
ASD 12.71 [5.12; 20.30] .001 .15 16.17 [9.29; 23.04] .0005
Irritability Age .09 -0.16 [-0.27; -0.06] .001 .06 -0.18 [-0.28; -0.08] .001
ED -1.63 [-2.38; -0.39] .01 .09 -1.63 [-2.87; -0.39] .01
Lethargy ASD .10 4.37 [1.67; 7.08] .002 .08 5.31 [2.87, 7.74] .0005
ED -1.03 [-2.34, 0.29] n.s. .05 -1.96 [-3.17; -0.76] .002
Stereotypy Males .20 1.89 [0.60; 3.19] .004 .04 2.13 [0.74; 3.53] .003
ASD 2.58 [1.22; 3.93] \.001 .16 3.33 [2.09; 4.56] .0005
ED -0.49 [-1.27; 0.30] n.s. .13 -1.39 [-2.00; -0.78] .0005
ID 0.84 [-0.18; 1.86] n.s. .11 1.67 [0.81; 2.53] .0005
Hyperactivity Age .17 -0.21 [-0.31; -0.11] .0005 .09 -0.23 [-0.33, -0.13] .0005
ASD 2.44 [-0.35; 5.23] n.s. .13 4.04 [1.49; 6.59] .002
ED -1.16 [-2.80; 0.49] n.s. .15 -2.34 [-3.58; -1.11] .0005
ID 1.56 [-0.55; 3.67] n.s. .14 2.92 [1.19; 4.64] .001
Inappropriate speech ED .05 0.43 [-0.10; 0.96] n.s. .04 0.65 [0.22; 1.09] .003
ID -0.53 [-1.28; 0.21] n.s. .04 -0.89 [-1.49; -0.28] .004
MOAS total Age .06 -0.52 [-0.8; -0.24] .0005
Aggression: verbal ED .07 0.21 [0.04; 0.38] .014 .05 0.26 [0.10; 0.43] .002
Personality disorders 0.82 [0.09; 1.54] .027 .04 1.05 [0.34; 1.76] .004
Aggression: self Age .16 -0.02 [-0.03; 0] .013 .04 -0.12 [-0.03; -0.01] .003
ASD 0.18 [-0.17, 0.52] n.s. .09 0.49 [0.17; 0.80] .003
ED -0.34 [-0.51, -0.18] .0005 .15 -0.38 [-0.53, -0.23] .0005
Aggression: others Age .06 -0.02 [-0.04; -0.01] .001
n.s. = no significant correlation; c.f. = controlled for; bold letters: p \ .05 in multivariate regression analysis

123
794 J Autism Dev Disord (2014) 44:786–800

ID. Strong correlations could also be seen for ‘age’ with the higher levels of ED were both independent predictors for
ABC and MOAS total score and several of their subscales ‘verbal aggression’ (cf. Table 3).
(c.f. Table 3).

Regression Analysis for Several Correlating Factors Predictors for Outcome Variables
on Challenging Behaviors and Aggression
Regression Analysis for Living and Working Situations
In a next step, several potential predictors that showed high
correlations for challenging or aggressive behaviors were Table 5 displays all predictors for the living and working
examined by multivariate regression analysis (c.f. Table 4). situations assessed in our study as indicated by multivariate
The most important predictors for overall challenging regression analysis.
behaviors as measured by the ABC total score were ASD, Low levels of ED were predictive for unemployment
lower levels of ED, and age. A second regression analysis and lower rates of a sheltered work. Moreover, individuals
for ED and ASD, controlled for ‘age’, revealed an added with a lower ED more often lived with their parents instead
variance of 6 and 9 %, respectively. The most important of a residential home. Unemployment was not only
predictor for overall aggression, aggression to others, and depending on ED, also higher scores of challenging
hyperactivity was age, explaining 6, 6, and 9 % of vari- behavior as indicated by the ABC total, presence of per-
ance. High scores for ‘irritability’ and ‘aggression to self’ sonality disorders, and absence of an affective disorder
depended on both, younger chronological ages and lower predicted unemployment in our clinical sample. The
levels of ED. ASD was the most important predictor for severity of ID was the most important predictor for par-
‘lethargy’ and ‘stereotypy’ explaining 8 and 12 % of var- ticipation in a rehab program. Individuals of older ages and
iance. Interestingly, independent from ASD, also male higher rates of aggressive behaviors (MOAS total score)
gender predicted stereotypic behaviors. Corresponding more often lived in a residential home and less often with
with the correlation analysis, personality disorders and their parents.

Table 5 Multivariate logistic


Dependent variable Predictor Multivariate regression—forced entry
regression analyses for living
and working situations and R2a OR [95 % CI] p
medication usage
Unemployment Mood disorders .36 .19 [0.05; 0.71] .013
Personality disorders 11.95 [1.71; 83.4] .012
ABC total score 1.02 [1; 1.04] .04
Level of ED .37 [.21; .68] .001
Rehab program Severity of ID .26 1.9 [1.02; 3.55] .045
Sheltered work Level of ED .45 2.29 [1.36; 3.88] .002
Living with parents Age .54 0.80 [0.71; 0.90] .0005
Mood disorders 0.11 [0.01; 0.80] .03
Movement disorder 15.96 [2.24; 113.56] .006
Level of ED 0.35 [0.13; 0.91] .032
MOAS total score 0.71 [0.55; 0.91] .006
Residential home Age .46 1.16 [1.08; 1.25] .0005
Movement disorder 0.12 [0.02; 0.69] .017
Level of ED 2.81 [1.23; 6.47] .015
Severity of ID 3.80 [1.15; 12.52] .028
MOAS total score 1.38 [1.11; 1.70] .003
Antipsychotics-high potency Schizophrenia .21 7.48 [3.02; 18.54] .0005
ASD 1.99 [1.04; 3.81] .037
Antipsychotics-low potency Schizophrenia .16 4.28 [2.04; 8.97] .0005
Antidepressants Mood disorders .24 7.40 [3.24; 17.00] .0005
Anticonvulsive drugs Mood disorders .05 2.19 [1.14; 4.21] .019
Benzodiazepines Dependency disorders .13 7.41 [2.00; 27.49] .003
OR odd’s ratio
a Schizophrenia 2.55 [1.02; 6.38] .045
Nagelkerkes R2

123
J Autism Dev Disord (2014) 44:786–800 795

Regression Analysis for Medication Usage in adults with ASD & ID compared to adults with ID only.
To determine, what type of challenging behavior was
As can be seen in Table 2, the prevalence of schizophrenia specific for ASD, we analyzed the ABC subscales sepa-
and prescriptions for antipsychotic drugs were significantly rately. Correlation analysis of the ABC subscales revealed
increased in the ASD sample. Accordingly, multivariate an association of ASD with the subscales ‘irritability’,
logistic regression analysis revealed that both, schizo- ‘hyperactivity’, ‘lethargy’, and ‘stereotypic behavior’.
phrenia and ASD were associated with the intake of high After controlling for several other correlating factors and
potency antipsychotic drugs (c.f. Table 5). Univariate age, ASD was still associated with elevated values in the
regression analysis showed an explained variance for ABC total score and the subscales ‘lethargy’ and ‘stereo-
antipsychotic drug usage of 17 % (OR 7.2) for schizo- typy’. High levels of stereotypic behaviors in individuals
phrenia and 6 % (OR 2.5) for ASD. Low potency anti- with ASD/ID combined versus ID only is a consistent
psychotics were only associated with schizophrenia, while finding in studies evaluating patterns of challenging
antidepressants and anticonvulsive drugs were associated behavior (Hill and Furniss 2006; Kraijer 2000; Matson and
with affective disorders. Benzodiazepines were predicted Rivet 2008; McTiernan et al. 2011). As stereotypic
by schizophrenia and dependency disorders explaining 4 % behavior is one of the core features of ASD, this result is
(OR 2.4) and 7 % (OR 6.3) of the total variance (univariate not surprising. Thus, factors other than the level of func-
regression analysis), respectively (c.f. Table 5). tioning may cause stereotypy, e.g., differences in sensory
processing or increased anxiety levels (Joosten and Bundy
2010). In summary, ASD is the major factor leading to
Discussion stereotypical and lethargic challenging behavior. Treatment
of social interaction and communication skills may be
This study aimed to examine the impact of various mental supportive in the reduction of these behaviors.
and neurological disorders, the level of ED, the severity of The level of ED predicted overall challenging behaviors
ID, age, and gender on (A) challenging and aggressive and showed associations with all 5 ABC subscales and
behavior and (B) certain outcome parameters in a repre- ‘aggression to self’. After controlling for other correlating
sentative clinical sample of adults with ID. ASD was the factors, delays in ED predicted high levels of the overall
only mental disorder predicting overall challenging challenging behavior, ‘irritability’ and ‘aggression to self’
behavior, in particular ‘lethargy’ and ‘stereotypic behav- and lower values of ‘verbal aggression’. This is supported
ior’. Personality disorders were associated with ‘verbal by studies concerning adaptive behavior, which is highly
aggression’. With regard to the level of functioning, a correlated with the level of ED (La Malfa et al. 2009),
lower level of ED was the major factor predicting chal- because it is an important predictor for the severity of
lenging behavior (ABC), in particular overall challenging challenging behavior (Matson et al. 2009a, b; Totsika et al.
behavior (ABC total) and ‘irritability’, and also aggression 2010). According to the developmental approach, the
(MOAS), in particular less ‘verbal aggression’ and more reduction in adaptive behaviors may be caused by lower
‘aggression to self’. Age war inversely associated with levels of ED. Moreover, co-morbidity of ASD and ID is
overall challenging and aggressive behaviors and most of associated with less secure bonding (Rutgers et al. 2004).
the subscales, while male gender was predictive for ster- Secure attachment is a developmental achievement attained
eotypic behaviors. (B) Unemployment was predicted by around the age of 3 years, which may not be reached in
high rates of challenging behaviors, low levels of ED, and case of developmental delay. Using the developmental
presence of personality disorders. Attendance at a sheltered approach, the increase in problem behaviors due to
working place was solely depending on higher levels of impaired attachment may be caused by the delay in ED (De
ED. Individuals showing aggressive behaviors more often Schipper and Schuengel 2010; Loeber et al. 2009; Robins
lived in a residential home and less likely lived with their and Price 1991). Analyzing the problem behavior resulting
parents. Antipsychotic drug usage was associated not only from attachment disorder, De Schipper and Schuengel
with schizophrenia, but also with ASD. (2010) also described high values of ‘irritability’, which
further supports our findings on this subscale. In a longi-
Predictors for Challenging Behaviors tudinal study, emotional difficulties at the age of 2 years
predicted specific behavioral problems at the age of 5 years
Overall challenging behaviors as measured with the ABC (Treyvaud et al. 2012). In summary, various surveys
total score was predicted by the presence of ASD, low revealed coherence between emotional developmental
levels of ED and younger ages. Our study confirms and delays and behavioral problems. However, this is not a
expands previous research conducted by McCarthy et al. one-way relationship, as an individual, who is temporar-
(2010) who showed that challenging behavior is increased ily experiencing behavioral problem will get a lower

123
796 J Autism Dev Disord (2014) 44:786–800

SAED-level than the level he is normally functioning at the severity of ASD characteristics, impulsivity, low verbal
(data not shown), implying that CB has an influence on the and cognitive abilities, hyperactivity, and negative affect
ED level as well. (Matson and Rivet 2008; Richards et al. 2012). The fun-
The severity of ID correlated with the ABC total score, damental importance of the delays in ED found in our
‘hyperactivity’, ‘stereotypy’, and ‘aggression to self’. study, however, may extend the understanding of mecha-
Multivariate regression analysis, however, revealed that nisms leading to self-injury and thereby may offer a further
factors such as age, presence of ASD, and the level of ED approach for treatment.
are more important predictors for these behaviors than the
severity of ID. ‘Inappropriate Speech’ was negatively Outcome Variables
correlated with severity of ID and inversely correlated with
level of ED. With increasing cognitive impairment and Lower levels of ED and higher levels of challenging and
lower emotional maturity, verbal abilities and thus chal- aggressive behaviors were important variables for the liv-
lenging behavior with ‘inappropriate speech’ decreased. ing and working situation of an individual. Attendance at a
According to our study results, a male gender was pre- ‘sheltered work’ was depending on higher levels of ED and
dictive for ‘stereotypic behaviors’ even when controlled for conversely, ‘unemployment’ was predicted by lower levels
other correlating factors. ‘Overall challenging and aggres- of ED. Moreover, ‘unemployment’ increased with higher
sive behavior’ and several subscales such as ‘irritability’, ABC total scores and presence of personality disorders.
‘hyperactivity’, and ‘aggression to the self and to others’ Regarding the examined variables for the living situation,
decreased with increasing age. This promising result for overall aggressive behavior and the level of ED were
many individuals, their families and carers confirms and crucial variables: Individuals with high MOAS total scores
expands previous research to adulthood (De Bildt et al. more often lived in residential homes and less often lived
2005; Maskey et al. 2013). Age is the most important with their parents, likely owing to the family burden of
predictor for ‘hyperactivity’, explaining 9 % of its vari- aggressive behavior. Interestingly, individuals with lower
ance. ED, ID, and ASD add another 6, 5, and 4 % of the chronological and emotional ages more often lived with
total variance, respectively. The impact of ASD and ID on their families and less often lived in residential homes,
‘hyperactivity’ has been described earlier (Totsika et al. while individuals with lower levels of ID more often lived
2011) and our study results add the relevance of the level of in a residential home. Once more, this finding supports the
ED. importance of measuring the emotional developmental
level and not to deduce it from the severity of ID.
Predictors for Aggression A multivariate regression analysis showed that—inde-
pendently form schizophrenia—ASD predicted antipsy-
A younger age was the only predictor for overall aggres- chotic drug usage, which agrees with previous studies
sive behaviors, neither any mental disorder nor levels of (Tsakanikos et al. 2007; Witwer and Lecavalier 2005).
ED or ID led to an increase in the total MOAS score. This Accordingly, the presence of ASD associated challenging
is consistent with other studies conducted in children and behavior may induce physicians to prescribe antipsychotics
young adults with ASD (Baghdadli et al. 2003; Kraijer in the absence of schizophrenia. Furthermore, our study
2000; McTiernan et al. 2011), but converse findings also revealed a significant decrease in disabling motor disorders
have been described (McClintock et al. 2003). Concerning in the ASD sample compared to the non-ASD sample with
the MOAS subscales, an association of age, the level of ED implications for the living situation. Interestingly, Klin
and—to a lesser extend—ASD with ‘aggression to the et al. (1992) found a similar pattern on the motor domain of
self’, personality disorders with ‘verbal aggression’, and the Vineland scale. This issue requires further clarification.
age with ‘aggression to others’ could be found. Self-injury
affects physical well-being, e.g., causing secondary infec- Clinical Impact
tions, malformations, or sensory impairments, tends to
persist throughout life, and is an important affliction for The high incidence of challenging behaviors presents a barrier
caregivers (Emerson et al. 2001). Self-injurious behavior is to attendance in designated programs (Holden and Gitlesen
thus a major impairment in affected individuals and ade- 2006). Accordingly, De Bildt et al. (2005) concluded adaptive
quate treatment is critical. The increase in self-injurious functioning was the most important factor in predicting the
behavior in the ASD population, especially in those with level of education in individuals with ID. Assessment and
additional ID, and in individuals with ID alone have been treatment of challenging behavior in the ID population is
reported previously (Baghdadli et al. 2003; McClintock complex and requires a multifaceted approach, especially as
et al. 2003; McTiernan et al. 2011). Positive associations psychopharmacology has not revealed an evidence-based
have been described between self-injurious behavior and treatment option (Brylewski and Duggan 2004). Over the past

123
J Autism Dev Disord (2014) 44:786–800 797

decades, behavioral based assessments, such as functional design of a clinical sample, the appraisal of the severity of
analysis (Didden et al. 1999; Grey and Hastings 2005), and ID was taken from the current/latest medical record. Psy-
treatments, such as applied behavioral analysis (Reichow chometric based ID assessment was available in many, but
et al. 2012; Warren et al. 2011), were the most favored and not in all individuals, which need to be considered when
best-studied methods. However, these programs are time- interpreting the results. Moreover, only a limited range of
consuming with limited availability, and funding remains a variables could be assessed in most participants; therefore
challenge for many families. Moreover, despite gains in important factors, such as the type of non-pharmacological
cognitive, verbal, and adaptive skills, treatments targeting treatment, life events, caregiver’s attitude, etc., were not
behavioral and social skill have only a limited effect on daily evaluated in the study. When interpreting relationships
life skills, behavioral and emotional problems (Begeer et al. between the different predictors and the severity of chal-
2011; Matson and Rivet 2008; McTiernan et al. 2011; Mur- lenging behavior, the interaction effects formally accoun-
phy et al. 2009). ted for only a small proportion of total variance. This may
Uncovering developmental mechanisms may be particu- be partially explained by the study design, as field studies
larly useful to prevent behavioral or mental health problems. often have less than 20 % of the efficiency of optimal
Loeber et al. (1993) demonstrated transformation of problem experimental tests (McClelland and Judd 1993). Moreover,
behavior across developmental periods, e.g., from opposi- the explained variance may be low due to the restricted
tional behavior to conduct problems to antisocial personality range of the predictors, as most were dichotomized or or-
disorder. This suggests that different levels of development dinally scaled ranging over 3 (severity of ID) to 5 (level of
may be associated with certain types of psychopathology. ED) grades. Nevertheless, the low percentage of explained
Janssen et al. (2002) provided with the ‘stress-attachment variance suggests that other factors still need to be con-
model’, a promising developmentally based explanatory sidered, for example behavioral phenotypes of certain
framework of challenging behavior among people with ID. genetic disorders, (Down, Fragile X, or Prader-Willi syn-
Within this developmental understanding of maladaptive drome; Loveland and Tunali-Kotoski 1998), and environ-
behavior, the impact for therapy has been shown in a number mental factors such as life events and social care (Hove and
of studies (Sappok et al. 2012a, b; Schuengel et al. 2009; Havik 2010). Recognition of the level of ED, however,
Smith et al. 2008). These developmentally derived interven- may extend and further improve the understanding of the
tions may lead to changes in emotional skills and subsequent complex circumstances causing challenging behavior
improvements in pro-social behavior and peer relationships, among people with ID and may have an impact on more
even in autistic individuals (Bergmann et al. 2011; Hofer et al. tailored treatment options.
2009; Schumacher and Calvet 2008). Analysis of etiological
factors should not be driven by linear causal thinking, but by a
multifaceted approach. Thus, an increasing number of inter- Conclusion
vention programs integrate applied behavior analysis with
developmental and relationship-based methods (Eapen et al. Aside from chronological age, presence of ASD and the
2013; Freitag et al. 2012) and report promising results with level of ED were the most important predictors for chal-
respect to adaptive skills, cognition and symptom severity. lenging and aggressive behaviors among people with ID
To conclude, the assessment and recognition of the level and have an impact on the living and working situation and
of ED may bring an additional perspective to the mecha- antipsychotic drug usage. Our study expands previous ASD
nisms leading to problem behaviors in the ID population. research to adulthood and adds the aspect of the overall
The developmental approach might serve as a diagnostic developmental level to the assessment of individuals with
aid for services supporting people with ID and lead to a severe problem behaviors. This detailed analysis of the
better understanding of the problem behaviors shown by origins of behavioral disorders may suggest priority areas
the individual. Subsequently, better-tailored treatment and for diagnostics and therapy, as these behaviors create
teaching options may arise. Considering the basic emo- obstacles to learning and productive labor. ED may be
tional needs connected with a certain level of ED may stimulated in individuals with ASD and ID to prevent or
increase well-being of the affected individual, stimulate a reduce problem behaviors, develop social and communi-
further development of personality and result in acquisition cation skills, and provide the prerequisites for participation
of functional life skills. in society and living a productive, satisfying life.

Limitations Acknowledgments We thank the foundation ‘‘von Bodelsch-


winghsche Stiftungen Bethel’’ for funding the study. Sven Bölte was
supported by the Swedish Research Council (Grant No. 523-2009-
A number of limitations need to be considered when 7054). We thank the clinicians who helped in data collection. We
interpreting our results. Due to the retrospective study appreciate Manuel Heinrich for his thorough data search from patient

123
798 J Autism Dev Disord (2014) 44:786–800

charts, data entry, support in data analysis, and layout. Thanks to De Schipper, J. C., & Schuengel, C. (2010). Attachment behaviour
Heika Kaiser who thoroughly performed the SAED, SCQ, PDD- towards support staff in young people with intellectual disabil-
MRS, ADOS, and ADI-R evaluations. We appreciate the compre- ities: Associations with challenging behaviour. Journal of
hensive elaboration of the reviewer as it substantially strengthened the Intellectual Disability Research, 54, 584–596.
methodology and intelligibility of the study. Didden, R., Duker, P. C., & Korzilius, H. (1999). Meta-analytic study
on treatment effectiveness for problem behaviors with individ-
Conflict of interest Sven Bölte receives royalties for the German uals who have mental retardation. American Journal of Mental
version of the ADOS and ADI-R from Hans Huber Publishers. Retardation, 101, 387–399.
Beyond that, the authors declare no conflicts of interest. Dosen, A. (1997). Psychische Störungen bei geistig behinderten
Menschen. Stuttgart: Gustav Fischer Verlag.
Dosen, A. (2005). Applying the developmental perspective in the
psychiatric assessment and diagnosis of persons with intellectual
disability: Part I-assessment. Journal of Intellectual Disability
References Research, 49, 1–8.
Eapen, V., Črnčec, R., & Walter, A. (2013). Clinical outcomes of an
Aldao, A., & Nolen-Hoeksema, S. (2012). The influence of context on early intervention program for preschool children with autism
the implementation of adaptive emotion regulation strategies. spectrum disorder in a community group setting. BMC Pediat-
Behavior Research and Therapy, 50, 493–501. rics, 13, 3.
Aman, M. G., Singh, N. N., Stewart, A. W., & Field, C. J. (1985). The Emerson, E., Kiernan, C., Alborz, A., Reeves, D., Mason, H.,
Aberrant Behavior Checklist: A behavior rating scale for the Swarbrick, R., et al. (2001). Predicting the persistence of severe
assessment of treatment effects. American Journal of Mental self injurious behavior. Research in Developmental Disabilities,
Deficancy, 89, 485–491. 22, 67–75.
Baghdadli, A., Pascal, S., Grisi, S., & Aussilloux, C. (2003). Risk Emerson, E., Kiernan, C., Turner, S., Hatton, C., & Alborz, A. (2000).
factors for self-injurious behaviors among 222 young children Psychiatric symptoms in adults with learning disability and
with autistic disorders. Journal of Intellectual Disability challenging behavior. The British Journal of Psychiatry, 177,
Research, 47, 622–627. 452–456.
Begeer, S., Gevers, C., Clifford, P., Verhoeve, M., Kat, K., Esbensen, A. J., Greenberg, J. S., Seltzer, M. M., & Aman, M. G.
Hoddenbach, E., et al. (2011). Theory of mind training in (2009). A longitudinal investigation of psychotropic and non-
children with autism: A randomized controlled trial. Journal of psychotropic medication use among adolescents and adults with
Autism and Developmental Disorders, 41, 997–1006. autism spectrum disorders. Journal of Autism and Developmen-
Begeer, S., Koot, H. M., Rieffe, C., Terwogt, M. M., & Stegge, H. tal Disorders, 39, 1339–1349.
(2008). Emotional competence in children with autism: Diag- Felce, D., Kerr, M., & Hastings, R. P. (2009). A general practice-
nostic criteria and empirical evidence. Developmental Review, based study of the relationship between indicators of mental
28, 342–369. illness and challenging behaviour among adults with intellectual
Bergmann, T., Dziobek, I., Reimer, S., Schumacher, K., Diefenbach- disabilities. Journal of Intellectual Disability Research, 53,
er, A., & Sappok, T. (2011). Jenseits des Dialogs—Bez- 243–254.
iehungsqualität in der Musiktherapie bei Menschen mit Freitag, C. M., Feineis-Matthews, S., Valerian, J., Teufel, K., &
Autismus und Intelligenzminderung [Beyond dialogue—quality Wilker, C. (2012). The Frankfurt early intervention program
of relationships in music therapy for people with autism and FFIP for preschool aged children with autism spectrum disorder:
intellectual disability]. Medizin für Menschen mit geistiger oder A pilot study. Journal of Neural Transmission, 119, 1011–1021.
mehrfacher Behinderung, 8, 14–20. Greenspan, S. (1979). Social intelligence in the retarded. In N.
Bertin, E., & Striano, T. (2006). The still-face response in newborn, 1. R. Ellis (Ed.), Handbook of mental deficiency: Psychological
5-, and 3-month-old infants. Infant Behaviour Development, 29, theory and research (pp. 483–531). Hillsdale, N.J.: Erlbaum.
294–297. Greenspan, S. (1997). Normal child development. In A. Freedman, H.
Bhaumik, S., Tyrer, F. C., McGrother, C., & Ganghadaran, S. K. Kaplan, & B. Sadock (Eds.), Comprehensive textbook of
(2008). Psychiatric service use and psychiatric disorders in psychiatry (pp. 1102–1130). Baltimore: Williams & Wilkins.
adults with intellectual disability. Journal of Intellectual Dis- Grey, I. M., & Hastings, R. P. (2005). Evidence-based practices in
ability Research, 52, 986–995. intellectual disability and behaviour disorders. Current Opinion
Bowlby, B. (1969). Attachment and loss. London: Hogarth Press. in Psychiatry, 18, 469–475.
Brylewski, J., & Duggan, L. (2004). Antipsychotic medication for Hill, J., & Furniss, F. (2006). Patterns of emotional and behavioural
challenging behaviour in people with learning disability. Coch- disturbance associated with autistic traits in young people with
rane Database Systematic Review, 3, CD000377. severe intellectual disabilities and challenging behaviours.
Ciarrochi, J., & Scott, C. (2006). The link between emotional Research in Developmental Disabilities, 27, 517–528.
competence and well-being: A longitudinal study. British Hobson, R. P. (1986). The autistic child’s appraisal of expressions of
Journal of Guidance & Counselling, 34, 231–243. emotion. Journal of Child Psychology and Psychiatry, 27,
Cooper, S. A., Smiley, E., Morrison, J., Williamson, A., & Allan, L. 321–342.
(2007). Mental ill-health in adults with intellectual disabilities: Hodapp, R., & Zigler, E. (1995). Past, present and future issues in the
Prevalence and associated factors. British Journal of Psychiatry, developmental approach to mental retardation and developmen-
190, 27–35. tal disabilities. In D. Cicchetti & D. Cohen (Eds.), Developmen-
Damasio, A. (2010). Self comes to mind. Amsterdam: tal psychopathology (pp. 299–331). New York: Wiley.
Wereldbibliotheek. Hofer, S. M., Gray, K. M., Piccinin, A. M., Mackinnon, A.,
De Bildt, A., Sytema, S., Kraijer, D., Sparrow, S., & Minderaa, R. Bontempo, D. E., Einfeld, S. L., et al. (2009). Correlated and
(2005). Adaptive functioning and behaviour problems in relation coupled within-person change in emotional and behavioral
to level of education in children and adolescents with intellectual disturbance in individuals with intellectual disability. American
disability. Journal of Intellectual Disability Research, 49, Journal of Intellectual and Developmental Disabilities, 114,
672–681. 307–321.

123
J Autism Dev Disord (2014) 44:786–800 799

Holden, B., & Gitlesen, J. P. (2006). A total population study of Lord, C., Risi, S., Lambrecht, L., Cook, E. H., Jr, Leventhal, B. L.,
challenging behaviour in the county of Hedmark, Norway: et al. (2000). The autism diagnostic observation schedule-
Prevalence, and risk markers. Research in Developmental generic: A standard measure of social and communication
Disabilities, 27, 456–465. deficits associated with the spectrum of autism. Journal of
Holmes, N., Shah, A., & Wing, L. (1982). The disability assessment Autism and Developmental Disorders, 30, 205–223.
schedule: A brief screening device for use with the mentally Lord, C., Rutter, M., Goode, S., Heemsbergen, J., Jordan, H., et al.
retarded. Psychological Medicine, 12, 879–890. (1989). Autism diagnostic observation schedule: A standardized
Hove, O., & Havik, O. E. (2010). Developmental level and other observation of communicative and social behaviour. Journal of
factors associated with symptoms of mental disorders and Autism and Developmental Disorders, 19, 185–212.
problem behaviour in adults with intellectual disabilities living Lord, C., Rutter, M., & Le Couteur, A. (1994). Autism diagnostic
in the community. Social Psychiatry and Psychiatric Epidemi- interview revised: A revised version of a diagnostic interview for
ology, 45, 105–113. caregivers of individuals with possible pervasive developmental
Izard, C. E., Youngstrom, E. A., Fine, S. E., Mostow, A. J., disorders. Journal of Autism and Developmental Disorders, 24,
Trentacosta, C. J., & Campbell, J. (2006). Emotions and 659–685.
developmental psychopathology. In D. Cicchetti & D. Lounds, J., Seltzer, M. M., Greenberg, J. S., & Shattuck, P. T. (2007).
J. Cohen (Eds.), Developmental psychopathology (pp. Transition and change in adolescents and young adults with
244–292). New York: Wiley. autism: Longitudinal effects on maternal well-being. American
Jackson, M., & Tisak, M. S. (2001). Is prosocial behaviour a good Journal on Mental Retardation, 112, 401–417.
thing? Developmental changes in children’s evaluations of Loveland, K., & Tunali-Kotoski, B. (1998). Development of adaptive
helping, sharing, cooperating, and comforting. British Journal behavior in persons with mental retardation. In J. A. Burack, R.
of Developmental Psychology, 19, 349–367. M. Hodapp, & E. Zigler (Eds.), Handbook of mental retardation
Janssen, C. G., Schuengel, C., & Stolk, J. (2002). Understanding and development (pp. 521–541). Cambridge, UK: Cambridge
challenging behaviour in people with severe and profound University Press.
intellectual disability: A stress-attachment model. Journal of Mahler, M. S., Pine, F., & Bergman, A. (1975). The psychological
Intellectual Disability Research, 46, 445–453. birth of the human infant. New York: Basic Books.
Joosten, A. V., & Bundy, A. C. (2010). Sensory processing and Maskey, M., Warnell, F., Parr, J. R., Le Couteur, A., & McConachie,
stereotypical and repetitive behaviour in children with autism H. (2013). Emotional and behavioural problems in children with
and intellectual disability. Australian Occupational Therapy autism spectrum disorder. Journal of Autism and Developmental
Journal, 57, 366–372. Disorders, 43, 851–859.
Kasari, L. L., Jahromi, L. B., & Gulsrud, A. C. (2012). Emotional Matson, J. L., Dempsey, T., & Rivet, T. T. (2009a). The interrela-
development in children with developmental disabilities. In J. tionships of psychopathology symptoms on social skills in adults
A. Burack, R. M. Hodapp, & E. Zigler (Eds.), The oxford with autism or PDD-NOS and intellectual disability. Journal of
handbook of intellectual disability and development (pp. Developmental and Physical Disabilities, 21, 39–55.
239–253). Cambridge, UK: Cambridge University Press. Matson, J. L., Kozlowski, A. M., Worley, J. A., Shoemaker, M. E.,
Kasari, C., Sigman, M., Mundy, P., & Yirmiya, N. (1990). Affective Sipes, M., & Horovitz, M. (2011). What is the evidence for
sharing in the context of joint attention interactions of normal, environmental causes of challenging behaviors in persons with
autistic, and mentally retarded children. Journal of Autism and intellectual disabilities and autism spectrum disorders? Research
Developmental Disorders, 20, 87–100. in Developmental Disabilities, 32, 693–698.
Kernberg, O. (2012). The inseparable nature of love and aggression. Matson, J. L., & Rivet, T. T. (2008). Characteristics of challenging
Washington, DC: American Psychiatric Publishing. behaviours in adults with autistic disorder, PDD-NOS, and
Klin, A., Volkmar, F. R., & Sparrow, S. S. (1992). Autistic social intellectual disability. Journal of Intellectual Developmental
dysfunction: Some limitations of the theory of mind hypothesis. Disabilities, 33, 323–329.
Journal of Child Psychology and Psychiatry, 33, 877–893. Matson, J. L., Rivet, T. T., Fodstad, J. C., Dempsey, T., & Boisjoli, J.
Knoedler, D. W. (1989). The modified overt aggression scale. A. (2009b). Examination of adaptive behaviour differences in
American Journal of Psychiatry, 146, 1081–1082. adults with autism spectrum disorders and intellectual disability.
Kraijer, D. W. (2000). Review of adaptive behavior studies in Research in Developmental Disabilities, 30, 1317–1325.
mentally retarded persons with autism/pervasive developmental McCarthy, J., Hemmings, C., Kravariti, E., Dworzynski, K., Holt, G.,
disorder. Journal of Autism and Developmental Disorders, 30, Bouras, N., et al. (2010). Challenging behavior and co-morbid
39–47. psychopathology in adults with intellectual disability and autism
Kraijer, D., & Melchers, P. (2003). Skala zur Erfassung von spectrum disorders. Research in Developmental Disabilities, 31,
Autismusspektrumstörungen bei Minderbegabten (SEAS-M). 362–366.
Frankfurt: Swets. McClelland, G. H., & Judd, C. M. (1993). Statistical difficulties of
La Malfa, G., Lassi, S., Bertelli, M., Albertinti, G., & Dosen, A. detecting interactions and moderator effects. Psychological
(2009). Emotional development and adaptive abilities in adults Bulletin, 114, 376–390.
with intellectual disability. A correlation study between the McClintock, K., Hall, S., & Oliver, C. (2003). Risk markers
scheme of appraisal of emotional development (SAED) and associated with challenging behaviours in people with intellec-
Vineland adaptive behavior scale (VABS). Research in Devel- tual disabilities: A meta-analytic study. Journal of Intellectual
opmental Disabilities, 30, 1406–1412. Disability Research, 47, 405–416.
Loeber, R., Jeffrey, D. B., & Dustin, A. P. (2009). Development and McTiernan, A., Leader, G., Healy, O., & Mannion, A. (2011).
etiology of disruptive and delinquent behavior. Annual Review of Analysis of risk factors and early predictors of challenging
Clinical Psychology, 5, 291–310. behavior for children with autism spectrum disorders. Research
Loeber, R., Wung, P., Keenan, K., Giroux, B., Stouthamer-Loeber, in Autism Spectrum Disorders, 5, 1215–1222.
M., Van Kammen, W. B., et al. (1993). Developmental pathways Meins, W., & Süssmann, D. (1993). Evaluation of an adaptive
in disruptive child behavior. Development and Psychopathology, behaviour classification for mentally retarded adults. Social
5, 103–133. Psychiatry and Psychiatric Epidemiology, 28, 201–205.

123
800 J Autism Dev Disord (2014) 44:786–800

Murphy, G. H., Beadle-Brown, J., Wing, L., Gould, J., Shah, A., & psychiatric assessment and treatment of adults with intellectual
Holmes, N. (2005). Chronicity of challenging behaviours in disability. Fortschritte Neurologie und Psychiatrie, 80, 154–161.
people with severe intellectual disabilities and/or autism: A total Schuengel, C., Sterkenburg, P. S., Jeczynski, P., Janssen, C. G., &
population sample. Journal of Autism and Developmental Jongbloed, G. (2009). Supporting affect regulation in children
Disorders, 35, 405–418. with multiple disabilities during psychotherapy: A multiple case
Murphy, O., Healy, O., & Leader, G. (2009). Risk factors for design study of therapeutic attachment. Journal of Consulting
challenging behaviors among 157 children with autism spectrum and Clinical Psychology, 77, 291–301.
disorders in Ireland. Research in Autism Spectrum Disorders, 3, Schumacher, K., & Calvet, C. (2008). Synchronization—music
474–482. therapy with children on the autistic spectrum. Göttingen:
Oliver, P. C., Crawford, M. J., Rao, B., Reece, B., & Tyrer, P. (2007). Vandenhoeck & Ruprecht.
Modified Overt Aggression Scale (MOAS) for people with Smith, L. E., Greenberg, J. S., Seltzer, M. M., & Hong, J. (2008).
intellectual disability and aggressive challenging behaviour: A Symptoms and behavior problems of adolescents and adults with
reliability study. Journal of Applied Research in Intellectual autism: Effects of mother–child relationship quality, warmth,
Disabilities, 20, 368–372. and praise. American Journal on Mental Retardation, 113,
Pessoa, L. (2010). Emergent processes in cognitive-emotional inter- 384–402.
actions. Dialogues Clinical Neuroscience, 12, 433–448. Sroufe, L. A. (2009). The concept of development in developmental
Piaget, J. (1953). The child’s construction of reality. London: psychopathology. Child Developmental Perspectives, 3,
Rotledge and Kegan Paul. 178–183.
Reichow, B., Barton, E. E., Boyd, B. A., & Hume, K. (2012). Early Stern, D. N. (2007). Die Lebenserfahrung des Säuglings. Stuttgart:
intensive behavioral intervention (EIBI) for young children with Klett-Cotta.
autism spectrum disorders (ASD). Cochrane Database System- Totsika, V., Felce, D., Kerr, M., & Hastings, R. P. (2010). Behavior
atic Reviews, 17, 10:CD009260. problems, psychiatric symptoms and quality of life for older adults
Rey, L., Extremera, N., Durán, A., & Ortiz-Tallo, M. (2012). with intellectual disability with and without autism. Journal of
Subjective quality of life of people with intellectual disabilities: Autism and Developmental Disorders, 40, 1171–1178.
The role of emotional competence on their subjective well- Totsika, V., Hastings, R. P., Emerson, E., Lancaster, G. A., &
being. Journal of Applied Research in Intellectual Disabilities, Berridge, D. M. (2011). A population-based investigation of
11, doi: 10.1111/jar.12015. behavioural and emotional problems and maternal mental health:
Richards, C., Oliver, C., Nelson, L., & Moss, J. (2012). Self-injurious Associations with autism spectrum disorder and intellectual
behaviour in individuals with autism spectrum disorder and disability. Journal of Child Psychology and Psychiatry, 52,
intellectual disability. Journal of Intellectual Disability 91–99.
Research, 56, 476–489. Trevarthen, C. (1980). The foundations of intersubjectivity: Devel-
Rieffe, C., Meerum Terwogt, M., & Cowan, R. (2005). Children’s opment of interpersonal and co-operative understanding in
understanding of mental states as causes of emotions. Infant and infants. In D. Olson (Ed.), The social foundations of language
Child Development, 14, 259–272. and thought (pp. 316–342). New York: WW Norton.
Robins, L., & Price, R. (1991). Adult disorders predicted by Treyvaud, K., Doyle, L. W., Lee, K. J., Roberts, G., Lim, J., et al.
childhood conduct problems: Results from the NIMH epidemi- (2012). Social-emotional difficulties in very preterm and term
ological catchment area project. Psychiatry, 54, 116–132. 2 year olds predict specific social-emotional problems at the age
Rutgers, A. H., Bakermans-Kranenburg, M. J., van Ijzendoorn, M. H., of 5 years. Journal of Pediatric Psychology, 37, 779–785.
& van Berckelaer-Onnes, I. A. (2004). Autism and attachment: A Tsakanikos, E., Costello, H., Holt, G., Sturmey, P., & Bouras, N.
meta-analytic review. Journal of Child Psychology and Psychi- (2007). Behaviour management problems as predictors of
atry, 45, 1123–1134. psychotropic medication and use of psychiatric services in
Rutter, M., Bailey, A., Berument, S. K., Lord, C., & Pickles, A. adults with autism. Journal of Autism and Developmental
(2001). Social Communication Questionnaire (SCQ). Los Ange- Disorders, 37, 1080–1085.
les: Western Psychological Services. Underwood, L., McCarthy, J., & Tsakanikos, E. (2010). Mental health
Sappok, T., Bergmann, T., Kaiser, H., & Diefenbacher, A. (2010). of adults with autism spectrum disorders and intellectual
Autism in adults with mental retardation. Der Nervenarzt, 81, disability. Current Opinion in Psychiatry, 23, 421–426.
1333–1345. Warren, Z., McPheeters, M. L., Sathe, N., Foss-Feig, J. H., Glasser,
Sappok, T., Diefenbacher, A., Bergmann, T., Zepperitz, S., & Dosen, A., & Veenstra-Vanderweele, J. (2011). A systematic review of
A. (2012a). Emotional developmental disorders in individuals early intensive intervention for autism spectrum disorders.
with intellectual disabilities (ID): A case-control-study. Psychi- Pediatrics, 127, e1303–e1311.
atrische Praxis, 39, 1–11. Witwer, A., & Lecavalier, L. (2005). Treatment incidence and
Sappok, T., Diefenbacher, A., Budczies, J., Schade, C., Grubich, C., patterns in children and adolescents with autism spectrum
Bergmann, T., et al. (2013). Diagnosing autism in a clinical disorders. Journal of Child and Adolescent Psychopharmacol-
sample of adults with intellectual disabilities: How useful are the ogy, 15, 671–681.
ADOS and the ADI-R? Research Developmental Disabilities, 34, Yudofsky, S. C., Silver, J. M., Jackson, W., Endicott, J., & Williams,
1642–1655. D. (1986). The Overt Aggression Scale for the objective rating of
Sappok, T., Schade, C., Kaiser, H., Dosen, A., & Diefenbacher, A. verbal and physical aggression. American Journal of Psychiatry,
(2012b). The importance of a developmental approach in 143, 35–39.

123

View publication stats

You might also like