Psychiatry Research 262 (2018) 63–69

Contents lists available at ScienceDirect

Psychiatry Research
journal homepage: www.elsevier.com/locate/psychres

Reliability of two social cognition tests: The combined stories test and the T
social knowledge test
Élisabeth Thibaudeaua,b, Caroline Cellarda,b, Maxime Legendrea,b, Karèle Villeneuvea,b,
⁎
Amélie M. Achima,c,
a
CERVO Brain Research Center, 2601 Chemin de la Canardière, G1J 2G3 Québec, Canada
b
École de Psychologie, Université Laval, Pavillon Félix-Antoine-Savard, 2325 Allée des Bibliothèques, G1V 0A6 Québec, Canada
c
Département de Psychiatrie et Neurosciences, Université Laval, Pavillon Ferdinand-Vandry, 1050 Avenue de la Médecine, local 4873, G1V 0A6 Québec, Canada

A R T I C L E I N F O A B S T R A C T

Keywords: Deficits in social cognition are common in psychiatric disorders. Validated social cognition measures with good
Psychometric properties psychometric properties are necessary to assess and target social cognitive deficits. Two recent social cognition
Test-retest reliability tests, the Combined Stories Test (COST) and the Social Knowledge Test (SKT), respectively assess theory of mind
Inter-rater reliability and social knowledge. Previous studies have shown good psychometric properties for these tests, but the test-
retest reliability has never been documented. The aim of this study was to evaluate the test-retest reliability and
the inter-rater reliability of the COST and the SKT. The COST and the SKT were administered twice to a group of
forty-two healthy adults, with a delay of approximately four weeks between the assessments. Excellent test-retest
reliability was observed for the COST, and a good test-retest reliability was observed for the SKT. There was no
evidence of practice effect. Furthermore, an excellent inter-rater reliability was observed for both tests. This
study shows a good reliability of the COST and the SKT that adds to the good validity previously reported for
these two tests. These good psychometrics properties thus support that the COST and the SKT are adequate
measures for the assessment of social cognition.

1. Introduction
Social cognition is defined as the mental processes underlying social
interactions, including the abilities involved in perceiving and inter-
preting social information in order to guide social interactions (Green
et al., 2015; Pinkham, 2014). Deficits in social cognition are common in
several psychiatric or neurological disorders, including bipolar disorder
(Bora et al., 2016; Samamé et al., 2015), autism (Chung et al., 2014),
traumatic brain injury (Martín-Rodríguez and León-Carrión, 2010;
Mcdonald, 2013) and schizophrenia (Achim et al., 2012; Savla et al.,
2013). Social cognitive deficits are well-documented in schizophrenia
and are associated with important difficulties in social and occupational
functioning (Achim et al., 2013b; Addington et al., 2006; Bell et al.,
2009; Fett et al., 2011; Lahera et al., 2012). Thus, there is an increased
interest to develop and validate treatments targeting these difficulties
(Pinkham et al., 2014; Schaafsma et al., 2015). The importance of social
cognition was for instance recognized over a decade ago by the
group of experts taking part in the Measurement and Treatment Re-
search to Improve Cognition in Schizophrenia (MATRICS) initiative
(Nuechterlein et al., 2008). However, this group of experts was faced
with the challenge that very few social cognition tests are standardized
or have had their psychometric properties documented.
A subsequent major initiative in schizophrenia, the Social Cognition
Psychometric Evaluation Study (SCOPE) (Pinkham et al., 2017, 2016,
2014), had the objectives (1) to achieve a consensus on the most im-
portant social cognitive domains in schizophrenia, (2) to identify the
most promising social cognition measures available in the experimental
psychology literature for each of these domains and (3) to test the
psychometric properties of the most promising tasks and make re-
commendations for the choice of social cognition measures, notably for
future clinical trials. This initiative led to a consensus on four domains
of social cognition that were important for schizophrenia: emotion
processing, attributional style/bias, social perception/knowledge and
theory of mind (ToM) (Pinkham et al., 2014). ToM and social knowl-
edge are two domains for which there is a significant lack of standar-
dized measures. ToM can be defined as the ability to represent or infer
the mental states of others (e.g. their intentions, emotions or beliefs)
(Achim et al., 2013a; Pinkham et al., 2014). Social perception can be

⁎
Correspondence to: 2601, de la Canardière (F-4561-4), Québec, QC G1J 2G3, Canada.
E-mail addresses: elisabeth.thibaudeau.2@ulaval.ca (É. Thibaudeau), Caroline.Cellard@psy.ulaval.ca (C. Cellard), maxime.legendre.1@ulaval.ca (M. Legendre),
karele.villeneuve.1@ulaval.ca (K. Villeneuve), amelie.achim@fmed.ulaval.ca (A.M. Achim).

https://doi.org/10.1016/j.psychres.2018.01.026
Received 30 September 2017; Received in revised form 23 December 2017; Accepted 12 January 2018
Available online 12 January 2018
0165-1781/ © 2018 Published by Elsevier B.V.
É. Thibaudeau et al.
defined as the ability to decode and interpret social cues in others,
which includes social knowledge and social context processing
(Pinkham et al., 2014).
The SCOPE initiative (Pinkham et al., 2014) targeted three pro-
mising ToM tasks for which the psychometric properties were assessed
in people with schizophrenia and in healthy participants: the Reading
the Mind in the Eyes Test (RMET) (Baron-Cohen et al., 2001), the
Hinting task (Corcoran et al., 1995), and The Awareness of Social In-
ferences Test, Part III (TASIT) (McDonald et al., 2003). In the sub-
sequent phases of SCOPE, improvements were made on the three ToM
tasks and the psychometric properties of these improved versions were
assessed (Pinkham et al., 2017). More specifically, for the RMET, de-
finitions were added for each of the words provided as response
choices. For the Hinting task, the scoring criteria were refined. For the
TASIT, a counterbalanced administration of test forms across sessions
was used and the response time was considered as an additional mea-
sure of interest. Despite these modifications, the RMET still showed
limited internal consistency and was classified “as acceptable with re-
servations” by the authors of SCOPE (Pinkham et al., 2017). The
Hinting task showed a practice effect, a moderate test-retest reliability
and a limited internal consistency for both patients and healthy parti-
cipants, and the task was considered acceptable (Pinkham et al., 2017).
The TASIT showed moderate test-retest reliability both in healthy
participants and in people with schizophrenia and was classified as
acceptable with reservations (Pinkham et al., 2017).
For social perception, the most promising task initially identified
through the SCOPE initiative was the Relationships Across Domains
(RAD) (Sergi et al., 2009). However, despite previous reports of ade-
quate psychometric properties for the RAD (Sergi et al., 2009), the re-
sults from SCOPE revealed limited test-retest reliability and internal
consistency in healthy participants, as well as a practice effect in people
with schizophrenia and a lower tolerability for both people with schi-
zophrenia and healthy participants (Pinkham et al., 2014). Therefore,
in the final phase of the SCOPE, two new measures of social perception
were evaluated; the Mini Profile of Nonverbal Sensitivity (MiniPONS)
(Bänziger et al., 2011) and the Social Attribution Task – Multiple
Choice (SAT-MC) (Bell et al., 2010). The MiniPONS showed limited
test-retest reliability in healthy participants, limited internal con-
sistency in both people with schizophrenia and healthy participants and
participants from both groups rated the MiniPONS as the least pleasant
task (Pinkham et al., 2017). Hence, the MiniPONS was classified as “not
recommended” (Pinkham et al., 2017). The SAT-MC showed limited
test-retest reliability and limited internal consistency both for people
with schizophrenia and healthy participants and was also classified as
“not recommended” by the authors of SCOPE (Pinkham et al., 2017).
While the SCOPE initiative improved our knowledge regarding the
psychometric properties of various social cognitive tasks, several lim-
itations were observed in relation to the tasks available to assess ToM
and social knowledge. Thus, further research is needed to identify social
cognition tasks with good validity and reliability.
In the context of previous studies, our team developed and used two
social cognition tests, the Combined Stories Test (COST) (Test des his-
toires combinées) and the Social Knowledge Test (SKT) (Test des
Situations) (Achim et al., 2012, 2013b; Lavoie et al., 2014, 2016). The
COST assesses the ability to infer different types of mental states
through open questions that require taking the mental states of the
characters into account (Achim et al., 2012). The COST was created by
combining items adapted from four well-known ToM tasks, as well as
adding new items. Two important advantages of the COST are that it
includes a greater number of items than other story-based tasks and that
it targets a wide range of mental states (e.g. beliefs, intentions, emo-
tions).
The SKT assesses social knowledge about mental states, or more
specifically about emotions typically experienced in given situations.
Contrary to ToM tasks, there is no character performing any action or
64
Psychiatry Research 262 (2018) 63–69
expressing anything in this test; participants are rather told about
generic situations that can happen in daily life and are asked to de-
termine how most people would feel in these situations.
In a previous paper with healthy participants and individuals with
recent onset schizophrenia, we reported good psychometric properties
for the COST, including a good convergent validity and internal con-
sistency, as well as an excellent inter-rater reliability (Achim et al.,
2012). Using the definition of ceiling effect used for the SCOPE study
(i.e. a perfect score on the test), no ceiling effects were observed. Fur-
thermore, additional studies showed normal distributions even in
healthy participants (e.g. Achim et al., 2012; Gaudreau et al., 2015;
Lavoie et al., 2014). We also recently documented that the COST is
sensitive to detect changes in performance following a cognitive re-
mediation therapy. Finally, a split-half analysis with the COST revealed
an excellent estimated test-retest reliability (unpublished results). For
the SKT, we reported an excellent inter-rater reliability and a moderate
internal consistency (Achim et al., 2012).
Overall, the COST and the SKT have shown good psychometric
properties and could represent informative tests to assess ToM and
social knowledge. However, the test-retest reliability of these tests re-
mains to be documented. Furthermore, in the initial evaluation of the
inter-rater reliability (Achim et al., 2012), one of the two judges was the
person who had built the scoring grid (i.e. AMA). Therefore, it would be
useful to examine the inter-rater reliability of the two tests with two
naïve judges.
The first objective of this study was to evaluate the reliability of the
COST and the SKT in a group of healthy adults, targeting both inter-
rater reliability and test-retest reliability. Based on pilot results (i.e. a
split-half analysis using previous data), we expected an excellent test-
retest reliability for the COST and a moderate to good test-retest re-
liability for the SKT. We also hypothesized that the excellent inter-rater
reliability previously reported in Achim et al. (2012) would be re-
plicated.
A second objective was to use the information about the reliability
of each item included in our test in order to improve our scoring grids,
which we identified as a useful step that could further improve the
reliability of the tests before documenting their reliability in psychiatric
populations.
2. Methods
2.1. Participants
Forty-three participants were recruited through an email sent to
students and employees at Université Laval. The inclusion criteria were:
1) age between 18 and 45 years (this range was chosen since we ob-
served different pattern of response in individuals above 50 years old
for the SKT in Lajoie et al. (2014); 2) French as the first language or the
language in which elementary school and high school were completed.
Exclusion criteria were: 1) current use of a psychoactive medication;
2) a diagnosis for a current psychiatric disorder; 3) a first degree re-
lative with psychosis or bipolar disorder; 4) a neurological disorder; 5)
a physical disability that could interfere with the testing; 6) an esti-
mated verbal intellectual quotient (IQ) less than 70; 7) a history of
traumatic brain injury; 8) drug or alcohol abuse or dependence.
One participant was excluded from the study because he presented
with an estimated IQ below 70 (estimated IQ of 65), leaving 42 parti-
cipants. Sociodemographic information is presented in Table 1. Two of
these participants could not be included in the test-retest reliability
analyses because they did not complete the second session, but they
were included in the analyses for the inter-rater reliability. This study
was approved by the ethics committee of the Centre intégré uni-
versitaire de santé et de services sociaux de la Capitale-Nationale
(CIUSSS-CN), and all participants provided written informed consent.
É. Thibaudeau et al. Psychiatry Research 262 (2018) 63–69

Table 1

26 questions for a maximum of 52 points. The answers to 18 questions are scored 0,

6 questions for a maximum of 12 points. The answers are scored 0, 1 or 2 points.

14 situations for a maximum of 14 points. The answers are scored 0 or 1 point.

29 questions for a maximum of 29 points. The answers are scored 0 or 1 point.
Sociodemographic information (N = 42).

COST = Combined Stories Test; SKT = Social Knowledge Test; QM = Second-order ToM questions; QR = Non-social reasoning questions; QC = Control questions; Q1 = First-order ToM questions; QSFP = Without faux-pas questions.
3 questions for a maximum of 6 points. The answers are scored 0 or 2 points.

2 questions for a maximum of 2 points. The answers are scored 0 or 1 point.

Mean (SD) Range

Sex (M/F) 15/27 –

1 or 2 points and answers to 8 questions are scored 0 or 2 points.

Age (years) 25.1 (5.8) 20–43
Education (years) 16.9 (2.7) 13–25
Estimated verbal IQ 99.5 (12.4) 78–129
Delay between assessments (days) 27.6 (4.2) 20–35

2.2. Material

2.2.1. The Combined Stories Test (COST) (Achim et al., 2012)

The COST includes 30 stories adapted from the Hinting Task
(Corcoran et al., 1995), the False Belief Task (Baron-Cohen, 1989; Frith
and Corcoran, 1996), the Faux-pas task (Baron-Cohen et al., 1999) and
the Strange Stories Test (Happé, 1994) as well as new items. Partici-
pants are to read each story out loud and are asked to answer ToM and
control questions. They are encouraged to refer back to the written

Scoring
stories if needed, which is done to minimize the impact of memory load.
The first story is a practice item, and feedback is provided only for that
first item. The COST takes approximately 30 min to complete. Here, the

Where is Jane going to look for her

Somebody who has a big argument

What was the color of the door in
Why did the painting fall off the
responses were rated by two independent judges who received a short

Has anyone said something he

What does Mary really mean?
training to use the scoring grid that was validated in a previous study
(Achim et al., 2012) and used in additional studies (Achim et al., 2013b;

should not have said?

Gaudreau et al., 2015; Lavoie et al., 2014, 2016; Tousignant et al.,
Fictitious example

2016). The COST was initially designed and validated in French, and

with his friend.

the French version is used for the current study. More details about the
different types of questions, as well as examples, are presented in

the story?
Table 2, and further information can be found in Achim et al. (2012).
wall?

doll?

Short situations are presented. The participant is to state the emotion that would be felt
Non-social reasoning items that require the inference of the cause of an event in the

2.2.2. The Social knowledge test (SKT) (Achim et al., 2012)

The QSFP are included so that the answer to a faux-pas question is not always yes.
Ability to extract information that is available in the text to control for memory or
Second-order ToM that requires taking the character's mental states into account.

Participants are verbally presented with a series of short hypothe-

First-order ToM to evaluate the capacity to link a mental state to a behavior.

tical situations inspired by those of Blair and Cipolotti (2000). For each
physical world, that is not linked to the actions or thoughts of a character.

situation, the participant is asked to state the emotion that would be felt
by most people in that situation. The participant thus needs to disregard
how he would feel and instead think about how most people would feel.
This test is classified as a social knowledge task since there is no action
or reaction by any character. This requirement to reflect about others
without being presented with any specific behavior from others dis-
tinguishes social knowledge tasks from emotion recognition tasks that
typically involve the recognition of other people's expressed emotion
attentional impairment that could impact ToM.

and from ToM tasks that systematically involves understanding the

actions or reactions of the characters. In the SKT, no action or verba-
lisation is presented that a character would have done or said, and
hence participants who complete the task have to rely on what they
by most people in this situation.

know about these types of situations in order to provide an answer

about the most likely reaction expected for each situation.
For the current study, the answers were rated by two independent
judges who received a short training to use the scoring grid validated in
Achim et al. (2012) and then used in subsequent studies (Achim et al.,
2013b, 2012; Gaudreau et al., 2015; Tousignant et al., 2016). The SKT
takes less than five minutes to complete. The details of the test, as well
Process

as an example, are presented in Table 2, and further information can be

Details of the COST and the SKT.

found in Achim et al. (2012).

Type of question

2.2.3. Mill-Hill, part B (Deltour, 2005)

Estimated verbal IQ was measured with the Mill-Hill part B. In this
QSFP
QM

QR

QC

test, participants are to select, from a list of six words, the synonym of a
Q1

target word. This test was administered only at the first session and was
Table 2

COST
Test

SKT

included to ensure that the participants presented with an estimated

verbal IQ above 70.

65
É. Thibaudeau et al.
2.3. Procedure
Participants completed two assessments separated by a period of
three to five weeks. These assessments included the COST, the SKT, the
Mill-Hill as well as additional measures that are not discussed in the
current paper.
2.4. Statistical analysis
For all analyses, the significance threshold was set at p < 0.05. For
the COST, our scores of interest were the QM and QR items (see
Table 2). The other control questions were not targeted by the analyzes
since they are included to verify that the participant is able to extract
relevant information from the stories (which is typically the case for
healthy participants), which leads to almost no variability in the scores.
For the COST-QM, COST-QR and SKT, distributions were first examined
to identify potential floor and ceiling effects, and the normality of the
distributions was evaluated with skewness and kurtosis statistics. Co-
hen's Kappas were then used to evaluate the inter-rater reliability of our
measures between the two independent judges (including data from
both testing sessions). The percentage of agreement was also calculated
for each individual item, and the scoring grid was clarified for the items
showing a percentage of agreement below a Z score of −2. We then
asked two naïve judges to re-score these items using the clarified grid,
and we updated our analyses. The final ratings were used as the con-
sensus score for the subsequent test-retest reliability analyses.
Test-retest reliability was evaluated with intra-class correlations
between the two sessions. Paired-sample t-tests and Cohen's ds were
calculated to examine potential practice effects. To ensure that the
consensus score did not inflate the test-retest reliability, we additionally
examined test-retest reliability using the score of only one of the two
judges (before consensus, but after correction of the scoring grids). In
addition, the percentage of consistency of each individual item between
the first and the second session was examined. The scoring grid was
clarified for items with a percentage below a Z score of −2, and we
asked two naïve judges to score the answers to these items using the
new grid. The same analyses were then repeated.
Finally, a correlation between the COST-QM and the SKT was con-
ducted to determine the extent of the association between the two
constructs.
3. Results
3.1. The COST
The COST-QM and the COST-QR showed normal distributions.
3.1.1. Inter-rater reliability
The Cohen's kappa for the COST-QM items with three possible
scores (2, 1 or 0) revealed a strong inter-rater agreement (κ = 0.75,
p < 0.001), while the COST-QM items with two possible scores (2 or 0)
revealed an excellent inter-rater agreement (κ = 0.97, p < 0.001). For
the COST-QR items, a strong inter-rater agreement was also observed (κ
= 0.76, p < 0.001). The individual item analysis revealed that only one
COST-QM item (item 11 – scored 0, 1 or 2 points) presented a weaker
inter-rater agreement (78.8%). Following the clarification of the
scoring grid, the agreement reached 90% for this item. The Cohen's
kappa for the COST-QM items with three possible scores (2, 1 or 0)
following this clarification was κ = 0.76, p < 0.001.
3.1.2. Test-retest reliability
For the COST-QM, the intra-class correlation between the two ses-
sions revealed an excellent test-retest reliability (ρ = 0.84, p < 0.01),
based on the classification proposed by Cicchetti (1994). The paired-
sample t-test revealed no practice effect for the COST-QM (see Table 3).
A percentage of consistency below two standard deviations was
66
Psychiatry Research 262 (2018) 63–69
observed for three COST-QM items (see Table 4). Following the clar-
ification of the scoring grid, a higher consistency was observed for these
three items (see Table 4) and a higher overall intra-class correlation was
observed (ρ = 0.86, p < 0.01). The test retest-reliability calculated
using the ratings of only one judge (ρ = 0.85, p < 0.01) was very si-
milar to the intra-class correlation observed after the consensus.
For the COST-QR, a poor, non-significant intra-class correlation was
observed, and the paired-sample t-test revealed no practice effect (see
Table 3). The percentage of consistency between the first and the
second session for the QR items ranged from 74.5% to 94.9%, with no
items below two standard deviations. The test retest-reliability calcu-
lated using the consensus score was in the low range (ρ = 0.15, p =
0.30), which suggests no inflation of the reliability compared to the
low-medium intra-class correlation obtained with the ratings of only
one judge (ρ = 0.30, p = 0.13).
3.2. The SKT
The distribution of the SKT scores was normal.
3.2.1. Inter-rater reliability
The inter-rater agreement for the SKT was almost perfect (κ = 0.99,
p < 0.001). Eleven items reached a 100% agreement between the two
judges, and agreement for the three other items ranged from 96.3% to
98.8%.
3.2.2. Test-retest reliability
A good test-retest reliability was observed for the SKT (ρ = 0.77,
p < 0.01). Furthermore, there was no evidence of practice effect be-
tween the two sessions (see Table 3). A percentage of consistency below
two standard deviations was observed for one item (see Table 4). Fol-
lowing the clarification of the scoring grid, a higher consistency for this
item (see Table 4) and a higher overall intra-class correlation were
observed (ρ = 0.78, p < 0.01). The test-retest reliability assessed using
the ratings of only one of the two judges revealed a similarly high intra-
class correlation (ρ = 0.76, p < 0.01).
3.3. Correlation between the COST-QM and the SKT
The correlation between the COST-QM and the SKT revealed a
moderate and significant relationship (r = 0.36, p = 0.021).
4. Discussion
This study assessed the test-retest reliability and the inter-rater re-
liability of the COST and the SKT in healthy adults. Before the clar-
ification of the scoring grid, good to excellent reliability was observed
for both tests. The clarification of the scoring grid for some items en-
abled us to further improve these psychometric properties, which re-
sulted in a good test-retest reliability, no practice effect and an excellent
inter-rater reliability for the COST-QM (a measure of ToM) and the SKT
(a measure of social knowledge). For the COST-QR (a control measure
included in the COST), a strong inter-rater agreement and no practice
effect were observed, but the non-significant intra-class correlation
suggests that the test-retest reliability of this control measure would
need to be improved. Combined with previous evidence of adequate
validity for the COST and the SKT, the results of the current study
nonetheless suggest that these two tests show good psychometric
properties at least in healthy participants.
4.1. The COST to assess ToM
The good test-retest reliability of the COST-QM indicates that per-
formance remains stable and that no practice effect is observed in the
short-term. Given the important association between ToM and func-
tioning and the difficulties in functioning in psychiatric populations
É. Thibaudeau et al.
Table 3
Intra-class correlations and practice effects for the COST and the SKT following the clarification of the scoring grid.
Measure T1 T2
Mean (SD) Range Mean (SD)
COST – QM (/52) 47.4 (2.9) 39–51 47.6 (3.8)
COST – QR (/12) 11.1 (1.0) 9–12 11.3 (0.9)
SKT (/14) 11.3 (1.2) 9–14 11.2 (1.3)
COST = Combined Stories Test; QM = second-order theory of mind questions; QR = Non-social reasoning questions; SKT = Social Knowledge Test; T1 = Session 1; T2 = Session 2; ICC
(ρ) = intra-class correlation; t = t-test; d = Cohen's d.
Table 4
COST-QM and SKT items for which a consistency below two standard deviations from the mean was initially observed between the two sessions.
Test and type of question Items Percentage of consistency for the item
COST – QM 11. Ravioli 60.0%
20. Annoyed 64.0%
26. Vase 62.5%
SKT 11. Anger 75.0%
COST = Combined Stories Test; QM = second-order theory of mind questions; SKT = Social Knowledge Test.
(Fett et al., 2011; Grove et al., 2016), it is important to have reliable
ToM measures that are relatively stable across administrations. The
inter-rater reliability was also evaluated and was excellent. These re-
sults confirmed the previous results of Achim et al. (2012) with healthy
participants and showed that the scoring grid is well-constructed and
exhaustive and allows naïve judges to use it correctly. These judges
received a short training before using the grid, and this is representative
of the type of training that researchers and clinicians that currently use
the test have received. Thus, these results indicate that a brief training
is sufficient for consistent scoring and proper use of the test.
The COST however may have some limitations. First, while the in-
tegration of non-social reasoning control questions is useful in order to
determine if potential deficits extend to reasoning about non-social si-
tuations, the COST-QR showed a poor and non-significant test-retest
reliability in the current study. A previous study of Achim et al. (2012)
showed that COST-QR items significantly contribute to the COST-QM
performance. Furthermore, non-social reasoning has been associated
with ToM in multiples studies (Abdel-Hamid et al., 2009; Fernandez-
Gonzalo et al., 2014; Pickup, 2008; Ziv et al., 2011). In the previous
study by Achim et al. (2012), the performance of healthy participants
on the COST-QR was similar to those in the current study, but the first-
episode psychosis participants showed a lower performance, which
suggests good discriminant validity between psychiatric and non-psy-
chiatric populations. In the current sample, a cut-off score between a
normal and an impaired performance can be set at 10, since a score
below that cut-off represents a performance beneath one-standard de-
viation from the mean (e.g. 16th percentile). Using a cut-off score for
the QR as a discriminant value between a psychiatric population and
healthy people will however have to be formally tested. It could also be
interesting to add more non-social reasoning questions or to increase
the difficulty of these items to get a larger range of performance that
could improve the test-retest reliability of these COST-QR. Another
limitation of the COST is that the administration time varies between 20
and 30 min. This could prevent clinicians or researchers from using the
test, given the limited time they often have with their patients. The
creation of a shorter version of the test could thus be interesting.
However, shorter tests with a limited number of items to assess ToM
often present with ceiling effects and are not sensitive enough to detect
subtler problems in ToM that could have an important impact on
functioning.
The good test-retest reliability and inter-rater reliability of the
COST-QM adds to the good internal consistency and convergent validity
Psychiatry Research 262 (2018) 63–69
ICC Practice effect
Range ρ p t p d
37–52 0.86 < 0.01 −0.598 0.553 −0.06
7–12 0.15 0.30 −1.070 0.291 −0.21
9–14 0.78 < 0.01 0.598 0.553 0.08
Percentage of consistency with the clarified grid
80.0%
97.5%
70.0%
82.5%
documented by Achim et al. (2012). Taken together, these results reveal
that the COST-QM is a valid and reliable measure of ToM with healthy
participants. The results of the SCOPE study suggested some limitations
in other ToM tasks such as limited internal consistency (RMET, Hinting
task), limited test-retest reliability (Hinting task, TASIT) and practice
effects (Hinting task) (Pinkham et al., 2017). Furthermore, few ToM
tasks are translated or validated in French. ToM is an important treat-
ment target in psychiatric populations, and clinical trials require the use
of reliable measures to identify the efficacy of various interventions.
The COST is therefore a suitable choice to measure ToM since it over-
comes these psychometric limitations, which will however have to be
confirmed in psychiatric populations. The COST is an interesting test for
psychiatric populations such as people with bipolar disorder or schi-
zophrenia, for which social cognition deficits are often observed si-
multaneously with deficits in various neurocognitive functions such as
memory, attention or executive functions (Bora et al., 2011, 2009;
Schaefer et al., 2013). The control questions included in the COST (i.e.
COST-QR and COST-QC) can provide useful cues regarding the source
of the difficulties observed during the ToM task. These questions can
help to control for potential neurocognitive problems that could inter-
fere with ToM performance such as difficulties with reasoning, problem
solving, attention or memory. The COST is thus a comprehensive test
that could be an interesting complement in a cognitive assessment to
evaluate ToM while taking into account other cognitive abilities.
4.2. The SKT to assess social knowledge
The SKT revealed good test-retest reliability and no practice effect
between the two sessions. As with the COST, the inter-rater reliability of
the SKT was evaluated in order to confirm, with different judges, the
results of the previous study of Achim et al. (2012). The inter-rater
agreement was almost perfect, which indicates that the scoring grid is
well-built and allows judges, with a short training, to correctly score the
test.
The findings from the SKT show some limitations of the task, or at
least some aspects that need to be considered. First, while the reliability
of the SKT is good, a modest internal consistency was observed in the
previous study of Achim et al. (2012). However, the composition of the
test (i.e. the evaluation of distinct component regarding the knowledge
of different mental states) and the dichotomous scoring could explain
this limited internal consistency. Second, there is an effect of age that
needs to be considered when interpreting the performance on the SKT.
67
É. Thibaudeau et al.
A previous study showed that adults above 50 years old present with
lower scores than younger adults (Lajoie et al., 2014). The validity and
reliability of the SKT thus need to be evaluated in other age groups in
the future.
In addition to the good psychometric properties documented in the
current study, the SKT showed a convergent validity with the COST-QM
and was found to significantly contribute to the COST-QM performance
in previous studies (Achim et al., 2012, 2013b). Furthermore, people
with schizophrenia exhibit moderate deficits in social knowledge, ac-
cording to the results of the meta-analysis of Savla et al. (2013). Taken
together, this suggests the importance of assessing social knowledge in
social cognition studies since deficits in this domain could impact ToM
performance. In addition, most social knowledge tasks assess the
knowledge about typical sequence in different social situations
(Corrigan and Addis, 1995), while the SKT interestingly assesses the
knowledge that people have about the link between typical situations
and mental states. Given the difficulty that people with psychiatric
disorder have to understand other peoples’ mental states, the SKT offers
a novel perspective on the impact that social knowledge can have on
ToM performance in these populations.
4.3. Limitations
Limitations of the tests were detailed in the previous sections. An
additional aspect that needs to be considered in the interpretation of the
results is the level of education of our participants. While it could have
led to ceiling effects or practice effects given that high levels of edu-
cation have been associated with the development of cognitive skills
and knowledge necessary to solve verbal tasks (Ritchie et al., 2015),
these problems were not encountered. It would nonetheless be useful to
document these psychometric properties with samples with different
sociodemographic characteristics, as well as in patients with different
psychopathologies.
4.4. Conclusion
The results of the current study add to the good psychometric
properties previously reported by Achim et al. (2012). The COST
overcomes several psychometric limitations of the currently available
ToM tasks, while the SKT provides a unique perspective on social
knowledge regarding mental states. While good psychometric proper-
ties were previously documented for these tasks in people with first-
episode psychosis, the test-retest reliability will also have to be assessed
in schizophrenia or other clinical populations before drawing firm
conclusions regarding the suitability of the tests for these populations.
An ongoing study will enable us to document the test-retest reliability
of these tests with individuals with schizophrenia, as well as the English
version of the tests. A copy of the COST and the SKT can be obtained by
contacting the corresponding author at the following email address:
amelie.achim@fmed.ulaval.ca.
Acknowledgment
ET, AMA and CC were involved in the design of the study, the
analysis, the interpretation of the data, the decision to submit the paper
and the writing of the manuscript. ET, ML and KV were involved in the
collection, the analysis, the interpretation of the data and the writing of
the manuscript.
Conflict of interest
None
Role of the funding source
This work was supported by the Natural Sciences and Engineering
68
Psychiatry Research 262 (2018) 63–69
Research Council of Canada (NSERC) [grant number #435556- 2013,
2013]. AMA is supported by a salary grant from Fonds de Recherche du
Québec en Santé (FRQS) and ET is supported by a studentship also from
FRQS. The funding sources were not involved in any step of this study,
from the design to the writing of the manuscript.
References
Abdel-Hamid, M., Lehmkämper, C., Sonntag, C., Juckel, G., Daum, I., Brüne, M., 2009.
Theory of mind in schizophrenia: the role of clinical symptomatology and neuro-
cognition in understanding other people's thoughts and intentions. Psychiatry Res.
165, 19–26. http://dx.doi.org/10.1016/j.psychres.2007.10.021.
Achim, A.M., Ouellet, R., Roy, M.A., Jackson, P.L., 2012. Mentalizing in first-episode
psychosis. Psychiatry Res. 196, 207–213. http://dx.doi.org/10.1016/j.psychres.
2011.10.011.
Achim, A.M., Guitton, M., Jackson, P.L., Boutin, A., Monetta, L., 2013a. On what ground
do we mentalize? Characteristics of current tasks and sources of information that
contribute to mentalizing judgments. Psychol. Assess. 25, 117–126. http://dx.doi.
org/10.1037/a0029137.
Achim, A.M., Ouellet, R., Lavoie, M.-A., Vallières, C., Jackson, P.L., Roy, M.-A., 2013b.
Impact of social anxiety on social cognition and functioning in patients with recent-
onset schizophrenia spectrum disorders. Schizophr. Res. 145, 75–81. http://dx.doi.
org/10.1016/j.schres.2013.01.012.
Addington, J., Saeedi, H., Addington, D., 2006. Influence of social perception and social
knowledge on cognitive and social functioning in early psychosis. Br. J. Psychiatry
189, 373–378. http://dx.doi.org/10.1192/bjp.bp.105.021022.
Bänziger, T., Scherer, K.R., Hall, J.A., Rosenthal, R., 2011. Introducing the MiniPONS: a
short multichannel version of the profile of nonverbal sensitivity (PONS). J.
Nonverbal Behav. 35, 189–204. http://dx.doi.org/10.1007/s10919-011-0108-3.
Baron-Cohen, S., 1989. The autistic child's theory of mind: a case of specific develop-
mental delay. J. Child Psychol. Psychiatry 30, 285–297. http://dx.doi.org/10.1111/j.
1469-7610.1989.tb00241.x.
Baron-Cohen, S., O’Riordan, M., Stone, V., Jones, R., Plaisted, K., 1999. Recognition of
faux pas by normally developing children with Asperger syndrome or high-func-
tioning autism. J. Autism Dev. Disord. 29, 407–418. http://dx.doi.org/10.1023/
A:1023035012436.
Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., Plumb, I., 2001. The “Reading the
Mind in the Eyes” Test revised version: a study with normal adults, and adults with
Asperger syndrome or high-functioning autism. J. Child Psychol. Psychiatry 42,
241–251. http://dx.doi.org/10.1111/1469-7610.00715.
Bell, M., Tsang, H.W.H., Greig, T.C., Bryson, G.J., 2009. Neurocognition, social cognition,
perceived social discomfort, and vocational outcomes in schizophrenia. Schizophr.
Bull. 35, 738–747. http://dx.doi.org/10.1093/schbul/sbm169.
Bell, M.D., Fiszdon, J.M., Greig, T.C., Wexler, B.E., 2010. Social attribution test - multiple
choice (SAT-MC) in schizophrenia: comparison with community sample and re-
lationship to neurocognitive, social cognitive and symptom measures. Schizophr. Res.
122, 164–171. http://dx.doi.org/10.1016/j.schres.2010.03.024.
Blair, R.J.R., Cipolotti, L., 2000. Impaired social response reversal A case of “acquired
sociopathy. Brain 123, 1122–1141. http://dx.doi.org/10.1093/brain/123.6.1122.
Bora, E., Bartholomeusz, C., Pantelis, C., 2016. Meta-analysis of Theory of Mind (ToM)
impairment in bipolar disorder. Psychol. Med. 46, 253–264. http://dx.doi.org/10.
1017/S0033291715001993.
Bora, E., Yucel, M., Pantelis, C., 2009. Cognitive endophenotypes of bipolar disorder: a
meta-analysis of neuropsychological deficits in euthymic patients and their first-de-
gree relatives. J. Affect. Disord. 113, 1–20. http://dx.doi.org/10.1016/j.jad.2008.06.
009.
Bora, E., Yücel, M., Pantelis, C., Berk, M., 2011. Meta-analytic review of neurocognition
in bipolar II disorder. Acta Psychiatr. Scand. 123, 165–174. http://dx.doi.org/10.
1111/j.1600-0447.2010.01638.x.
Chung, Y.S., Barch, D., Strube, M., 2014. A meta-analysis of mentalizing impairments in
adults with schizophrenia and autism spectrum disorder. Schizophr. Bull. 40,
602–616. http://dx.doi.org/10.1093/schbul/sbt048.
Cicchetti, D.V., 1994. Guidelines, criteria, and rules of thumb for evaluating normed and
standardized assessment instruments in psychology. Psychol. Assess. 6, 284–290.
http://dx.doi.org/10.1037/1040-3590.6.4.284.
Corcoran, R., Mercer, G., Frith, C.D., 1995. Schizophrenia, symptomatology and social
inference: investigating “theory of mind” in people with schizophrenia. Schizophr.
Res. 17, 5–13. http://dx.doi.org/10.1016/0920-9964(95)00024-G.
Corrigan, P.W., Addis, I.B., 1995. The effects of cognitive complexity on a social se-
quencing task in schizophrenia. Schizophr. Res. 16, 137–144. http://dx.doi.org/10.
1016/0920-9964(94)00072-G.
Deltour, J.J., 2005. Échelle de vocabulaire Mill Hill. Les Éditions du Centre de
Psychologie Appliquée, Paris, France.
Fernandez-Gonzalo, S., Jodar, M., Pousa, E., Turon, M., Garcia, R., Rambla, Hernandez,
Carla Palao, D., 2014. Selective effect of neurocognition on different theory of mind
domains in first-episode psychosis. J. Nerv. Ment. Dis. 202, 576–582. http://dx.doi.
org/10.1097/NMD.0000000000000164.
Fett, A.K.J., Viechtbauer, W., Dominguez, M., de, G., Penn, D.L., van Os, J., Krabbendam,
L., 2011. The relationship between neurocognition and social cognition with func-
tional outcomes in schizophrenia: a meta-analysis. Neurosci. Biobehav. Rev. 35,
573–588. http://dx.doi.org/10.1016/j.neubiorev.2010.07.001.
Frith, C.D., Corcoran, R., 1996. Exploring “theory of mind” in people with schizophrenia.
Psychol. Med. 26, 521–530. http://dx.doi.org/10.1017/S0033291700035601.
É. Thibaudeau et al.
Gaudreau, G., Monetta, L., Macoir, J., Poulin, S., Laforce, R.J., Hudon, C., 2015. Mental
state inferences abilities contribution to verbal irony comprehension in older adults
with mild cognitive impairment. Behav. Neurol. 2015. http://dx.doi.org/10.1155/
2015/685613.
Green, M.F., Horan, W.P., Lee, J., 2015. Social cognition in schizophrenia. Nat. Rev.
Neurosci. 16, 620–631. http://dx.doi.org/10.1038/nrn4005.
Grove, T.B., Tso, I.F., Chun, J., Mueller, S.A., Taylor, S.F., Ellingrod, V.L., McInnis, M.G.,
Deldin, P.J., 2016. Negative affect predicts social functioning across schizophrenia
and bipolar disorder: findings from an integrated data analysis. Psychiatry Res. 243,
198–206. http://dx.doi.org/10.1016/j.psychres.2016.06.031.
Happé, F.G.E., 1994. An advanced test of theory of mind: understanding of story char-
acters' thoughts and feelings by able autistic, mentally handicapped, and normal
children and adults. J. Autism Dev. Disord. 24, 129–154. http://dx.doi.org/10.1007/
BF02172093.
Lahera, G., Ruiz-Murugarren, S., Iglesias, P., Ruiz-Bennasar, C., Herreria, E., Montes, J.M.,
Fernandez-Liria, A., 2012. Social cognition and global functioning in bipolar dis-
order. J. Nerv. Ment. Dis. 200, 135–141. http://dx.doi.org/10.1097/NMD.
0b013e3182438eae.
Lajoie, M.-.P., Lavoie, M.-.A., Vistoli, D., Achim, A.M., 2014. La tâche de cognition sociale
Les Situations est-elle valide pour des adultes de différents groupes d’âge?, In: 36e
Congrès de La Société Québécoise Pour La Recherche En Psychologie. Montreal.
Lavoie, M.-A., Plana, I., Jackson, P.L., Godmaire-Duhaime, F., Bédard Lacroix, J., Achim,
A.M., 2014. Performance in multiple domains of social cognition in parents of pa-
tients with schizophrenia. Psychiatry Res. 220, 118–124. http://dx.doi.org/10.1016/
j.psychres.2014.07.055.
Lavoie, M.A., Vistoli, D., Sutliff, S., Jackson, P.L., Achim, A.M., 2016. Social re-
presentations and contextual adjustments as two distinct components of the Theory
of Mind brain network: evidence from the REMICS task. Cortex 81, 176–191. http://
dx.doi.org/10.1016/j.cortex.2016.04.017.
Martín-Rodríguez, J.F., León-Carrión, J., 2010. Theory of mind deficits in patients with
acquired brain injury: a quantitative review. Neuropsychologia 48, 1181–1191.
http://dx.doi.org/10.1016/j.neuropsychologia.2010.02.009.
Mcdonald, S., 2013. Impairments in social cognition following severe traumatic brain
injury. J. Int. Neuropsychol. Soc. 19, 231–246. http://dx.doi.org/10.1017/
S1355617712001506.
McDonald, S., Flanagan, S., Rollins, J., Kinch, J., 2003. TASIT: a new clinical tool for
assessing social perception after traumatic brain injury. J. Head Trauma Rehabil. 18,
219–238. http://dx.doi.org/10.1097/00001199-200305000-00001.
Nuechterlein, K.H., Green, M.F., Kern, R.S., Baade, L.E., Barch, D.M., Cohen, J.D., Essock,
S., Fenton, W.S., Frese, F.J., Gold, J.M., Goldberg, T., Heaton, R.K., Keefe, R.S.E.,
Kraemer, H., Mesholam-Gately, R., Seidman, L.J., Stover, E., Weinberger, D.R.,
Young, A.S., Zalcman, S., Marder, S.R., 2008. The MATRICS consensus cognitive
battery, Part 1: test selection, reliability, and validity. Am. J. Psychiatry 165,
69
Psychiatry Research 262 (2018) 63–69
203–213. http://dx.doi.org/10.1176/appi.ajp.2007.07010042.
Pickup, G.J., 2008. Relationship between theory of mind and executive function in
schizophrenia: a systematic review. Psychopathology 41, 206–213. http://dx.doi.
org/10.1159/000125554.
Pinkham, A.E., 2014. Social cognition in schizophrenia. J. Clin. Psychiatry 75, 14–19.
http://dx.doi.org/10.4088/JCP.13065su1.04.
Pinkham, A.E., Harvey, P.D., Penn, D.L., 2017. Social cognition psychometric evaluation:
results of the final validation study. Schizophr. Bull. http://dx.doi.org/10.1093/
schbul/sbx140.
Pinkham, A.E., Penn, D.L., Green, M.F., Buck, B., Healey, K., Harvey, P.D., 2014. The
social cognition psychometric evaluation study: results of the expert survey and
RAND Panel. Schizophr. Bull. 40, 813–823. http://dx.doi.org/10.1093/schbul/
sbt081.
Pinkham, A.E., Penn, D.L., Green, M.F., Harvey, P.D., 2016. Social cognition psycho-
metric evaluation: results of the initial psychometric study. Schizophr. Bull. 42,
494–504. http://dx.doi.org/10.1093/schbul/sbv056.
Ritchie, S.J., Bates, T.C., Deary, I.J., 2015. Is education associated with improvements in
general cognitive ability, or in specific skills? Dev. Psychol. 51, 573–582. http://dx.
doi.org/10.1037/a0038981.
Samamé, C., Martino, D.J., Strejilevich, S.A., 2015. An individual task meta-analysis of
social cognition in euthymic bipolar disorders. J. Affect. Disord. 173, 146–153.
http://dx.doi.org/10.1016/j.jad.2014.10.055.
Savla, G.N., Vella, L., Armstrong, C.C., Penn, D.L., Twamley, E.W., 2013. Deficits in do-
mains of social cognition in schizophrenia: a meta-analysis of the empirical evidence.
Schizophr. Bull. 39, 979–992. http://dx.doi.org/10.1093/schbul/sbs080.
Schaafsma, S.M., Pfaff, D.W., Spunt, R.P., Adolphs, R., 2015. Deconstructing and re-
constructing theory of mind. Trends Cogn. Sci. 19, 65–72. http://dx.doi.org/10.
1016/j.tics.2014.11.007.
Schaefer, J., Giangrande, E., Weinberger, D.R., Dickinson, D., 2013. The global cognitive
impairment in schizophrenia: consistent over decades and around the world.
Schizophr. Res. 150, 42–50. http://dx.doi.org/10.1016/j.schres.2013.07.009.
Sergi, M.J., Fiske, A.P., Horan, W.P., Kern, R.S., Kee, K.S., Subotnik, K.L., Nuechterlein,
K.H., Green, M.F., 2009. Development of a measure of relationship perception in
schizophrenia. Psychiatry Res. 166, 54–62. http://dx.doi.org/10.1016/j.psychres.
2008.03.010.
Tousignant, B., Jackson, P.L., Massicotte, E., Beauchamp, M.H., Achim, A.M., Vera-Estay,
E., Bedell, G., Sirois, K., 2016. Impact of traumatic brain injury on social cognition in
adolescents and contribution of other higher order cognitive functions.
Neuropsychol. Rehabil. 2011, 1–19. http://dx.doi.org/10.1080/09602011.2016.
1158114.
Ziv, I., Leiser, D., Levine, J., 2011. Social cognition in schizophrenia: cognitive and af-
fective factors. Cogn. Neuropsychiatry 16, 71–91. http://dx.doi.org/10.1080/
13546805.2010.492693.