Intelligence 90 (2022) 101614

Contents lists available at ScienceDirect

Intelligence
journal homepage: www.elsevier.com/locate/intell

Cognitive reflection, cognitive intelligence, and cognitive abilities: A

meta-analysis
Inmaculada Otero, Jesús F. Salgado *, Silvia Moscoso
Department of Political Science and Sociology, Faculty of Labor Relations, University of Santiago de Compostela, Campus Vida, 15782 Santiago de Compostela, Spain

A R T I C L E I N F O A B S T R A C T

Keywords: This paper presents a series of psychometric meta-analysis on the relationship between cognitive reflection (CR)
Cognitive intelligence and several cognitive abilities (i.e., cognitive intelligence, numerical ability, verbal ability, mechanical-spatial
Cognitive abilities ability, and working memory), and skills (i.e., numeracy skills). Also, the paper presents a bifactor analysis
Cognitive reflection
carried out to determine whether CR is a related but independent factor or a second-stratum factor in a hier­
Meta-analysis
Numeracy skills
archical model of cognitive intelligence. Finally, the study also tested a path meta-analytic model of the CR-
cognitive ability relationships. The results showed that CR correlated substantially with all the cognitive abili­
ties and skills (K ranged from 3 to 44 and N ranged from 624 to 20,307). The bifactor analysis showed that CR
variance was mainly accounted for by a general factor of cognitive intelligence plus a second-stratum factor of
numerical ability. The results of the bifactor analysis were similar for numerical-CRT and verbal-CRT. It was not
found evidence supporting the existence of a cognitive reflection factor. Finally, the path meta-analytic model
showed that the combination of cognitive intelligence and numerical ability accounted for 69% of CR variance.
The path model showed that cognitive intelligence and numerical ability have direct and indirect (through
numeracy skills) effects on CR. Theoretical and practical implications are discussed, and future research is
suggested.

Dual-Process (DP) models of human cognition suggest that the
human mind consists of two systems, System 1 (S1) and System 2 (S2;
Evans & Stanovich, 2013; Evans & Wason, 1976; Kahneman & Freder­
ick, 2002, 2005). S1 is characterized as intuitive, automatic, fast de­
cisions that require little or no effort. S2 is described as reflexive, slow,
and deliberative decisions that demand cognitive effort and concentra­
tion. Over two decades of research have shown that people vary in their
propensity to use S2 (Kahneman, 2011; Stanovich & West, 1998, 2000;
Tversky & Kahneman, 1974), and this has been partially attributed to
individual differences in cognitive reflection (CR; Böckenholt, 2012;
Frederick, 2005; Kahneman & Frederick, 2002, 2005; Stagnaro, Pen­
nycook, & Rand, 2018). According to Kahneman and Frederick (2002;
Frederick, 2005), CR refers to the individual ability or disposition to stop
the first impulsive response that our mind offers and to activate the
reflective mechanisms that allow us to find an answer, make a decision,
or carry out a specific behavior in a more thoughtful way. CR is inti­
mately associated with the popularity of the Cognitive Reflection Test
(CRT) developed by Frederick (Frederick, 2005; see also Kahneman,
2011; Kahneman & Frederick, 2002, 2005). The CRT consists of three
items that trigger individuals to answer immediately but incorrectly
(S1). Finding the correct answer requires the activation of reflective
mechanisms (S2) to reject the immediate-but-wrong answer and give a
more reflective-and-right one. The most famous item of Frederick’s CRT
is: A bat and a ball cost $1.10. The bat cost one dollar more than the ball.
How much does the ball cost? (intuitive answer: 10 cents; correct answer:
5 cents).
CR has been defined as both an “ability” and a “disposition” (Fred­
erick, 2005). Since then, researchers have explored whether CRT reflects
variance in cognitive abilities, variance in skills, or both. Nevertheless,
there is no broad consensus about which abilities CRT measures
(Blacksmith, Yang, Behrend, & Ruark, 2019; Campitelli & Gerrans,
2014). Empirical research used a variety of CR tests. For instance, some
studies used Frederick’s 3-item CRT. Others added new items to the 3-
item CRT (Finucane & Gullion; Primi, Donati, Chiesi, & Morsanyi,
2018). Several used a verbal test for assessing CR (Sirota, Kostovičová,
Juanchich, Dewberry, & Marshall, 2020; Thomson & Oppenheimer,
2016), and some other used new and more extensive numerical CRTs
(Salgado, Otero, & Moscoso, 2019).

* Corresponding author.
E-mail addresses: inmaculada.otero@usc.es (I. Otero), jesus.salgado@usc.es (J.F. Salgado), silvia.moscoso@usc.es (S. Moscoso).

https://doi.org/10.1016/j.intell.2021.101614
Received 24 February 2021; Received in revised form 8 November 2021; Accepted 9 December 2021
Available online 17 December 2021
0160-2896/© 2021 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
I. Otero et al.
Research about CRT has shown some mixed findings. For example,
previous studies have found that CRT correlated with cognitive intelli­
gence, (also termed as general mental ability, general intelligence,
general cognitive ability, and factor g), which has been defined as the
ability to learn a task, a subject, or a skill accurately and quickly, under
optimal instruction conditions (see, for instance, Carroll, 1993; Deary,
2012; Jensen, 1998; Hunt, 2011; Salgado, 2017). Previous studies have
also found that CRT correlated with specific cognitive abilities, such as,
numerical ability, verbal ability, mechanical-spatial ability, working-
memory capacity, perceptual speed, and numeracy skills (Avram,
2018; Blacksmith et al., 2019; Del Missier, Mäntylä, & De Bruin, 2011;
Finucane & Gullion, 2010; Gómez-Chacón, García-Madruga, Vila, Elo­
súa, & Rodríguez, 2014; Graffeo, Polonio, & Bonini, 2015; Koscielniak,
Rydzewska, & Sedek, 2016; Lado, Otero, & Salgado, 2021; Lindeman &
Svedholm-Häkkinen, 2016; Mækelæ, Moritz, & Pfuhl, 2018; Morsanyi,
McCormack, & O’Mahony, 2017; Pennycook, Cheyne, Barr, Koehler, &
Fugelsang, 2015; Poore, Forlines, Miller, Regan, & Irvine, 2014; Ruffle &
Tobol, 2017; Ståhl & van Prooijen, 2018; Szasci, Szollosi, Palfi, & Aczel,
2017; Teigen, Løhre, & Hohle, 2018; Teovanović, Knežević, & Stankov,
2015; Zonca, Coricelli, & Polonio, 2020, among others). Nevertheless,
an enormous variability was also found in the correlations, which cast
some doubts on how CRT is related to these cognitive abilities and skills.
The empirical evidence leads us to pose five crucial research ques­
tions. First, is CR an independent rationality factor (as posited by Top­
lak, West, & Stanovich, 2011, 2014) or a second stratum factor in a
hierarchical model of cognitive intelligence (e.g., Carroll’s model
[Carroll, 1993])? Second, what is the true score correlation between CR,
cognitive intelligence, other cognitive abilities (e.g., verbal and nu­
merical abilities), and skills? Third, is the observed variability among
the correlations real or artifactual? Fourth, is the relationship between
CR and cognitive abilities and skills moderated by the CRT type? Fifth,
how much variance of CR is explained by cognitive abilities and skills?
When, in a research domain, the primary studies show mixed findings,
psychometric meta-analysis (Schmidt & Hunter, 2015) is a method that
can provide a robust answer to these types of questions. However, so far,
no meta-analysis has examined how cognitive abilities and skills relate
to CR. Therefore, the purpose of this study is to shed further light on
these unresolved issues concerning the relationships of CR with several
cognitive variables and skills.
1. Cognitive reflection and cognitive abilities and skills: the
dual-process model vs. the hierarchical model of cognitive
intelligence
At present, there is no clear agreement about the nomological
network of CR. In other words, there is no unanimity about what CR tests
measure or reflect. Do CR tests reflect CR variance or do CR tests reflect
constructs that may be causes of CR, i.e., cognitive abilities (e.g.,
cognitive intelligence, and numerical ability) and skills (e.g., numeracy
skills)? In the first case, CR would be conceived of as a construct in a
reflective latent variable model which specifies a direct effect from a
construct to its measures (e.g., items) and, therefore, “the variance in
each measure is explained by a construct common to all measures and
error unique to each measure, and covariation among the measures is
attributed to their common cause” (Edwards & Bagozzi, 2000, pp. 162).
In the second case, CR would be conceived as a construct that would
indirectly reflect the variance of cognitive abilities and skills. This sec­
ond model would be a hierarchical reflective one (Edwards & Bagozzi,
2000), according to which the relationship between CR and the CR tests
may follow a more complex causal structure than was initially assumed.
Researchers have posited a variety of answers to this question, none of
them fully conclusive.
Initially, Kahneman and Frederick (2005) posited that the CRT
would directly reflect CR. Therefore, CRT would consist of CR variance
only. On the other hand, Stanovich and his associates (Stanovich, 2009;
Toplak et al., 2011, 2014) propose that the CRT assesses properties
2
Intelligence 90 (2022) 101614
relevant to rational thinking that go beyond those assessed by cognitive
intelligence tests. They sustain that the CRT, although it correlates
moderately with cognitive intelligence, appears to imply thinking dis­
positions, particularly those related to reflectivity, the tendency to
engage in fully disjunctive reasoning, and the tendency to seek alter­
native solutions. According to these researchers, the CRT predicts
rational thinking performance independently not only of cognitive in­
telligence, but also of executive functioning and thinking dispositions. In
a study, Toplak et al. (2011) found that the CRT accounted for more
unique explained variance than the block of cognitive intelligence
measures did. In another study, Toplak et al. (2014) found that an
expanded CRT was a substantial unique predictor of rational thinking
performance, and independent of cognitive intelligence and a set of
thinking dispositions. They speculated that the predictive power of the
expanded CRT is because the CRT is a strong indicator of the miserly
processing that, in dual process theory, is the source of much non-
normative responding. In their view, the CRT could be considered as
an actual measure of rational thought, rather than as cognitive intelli­
gence. More specifically, Toplak et al. (2014) sustain that the CRT is a
measure of the tendency towards the class of reasoning error that derives
from miserly processing. In their view, cognitive intelligence tests do not
assess the tendency towards miserly processing in the way that the CRT
does. Other studies have also shown that the CRT provided some in­
cremental variance over cognitive intelligence when predicting
decision-making tasks (Baron, Scott, Fincher, & Metz, 2015; Liberali,
Reyna, Furlan, Stein, & Pardo, 2012; Primi, Morsanyi, Chiesi, Donati, &
Hamilton, 2015).
However, other researchers found results that suggest a different
explanation for the predictive validity of the CRT. For instance, based on
the fact that CRT correlated with cognitive intelligence tests and
cognitive ability tests, some researchers have suggested that the CRT are
measures that mainly reflect the variance of cognitive abilities, partic­
ularly, cognitive intelligence (e.g., Avram, 2018; Baron, Scott, Fincher,
& Metz, 2015; Campitelli & Gerrans, 2014; Cokely, Galesic, Schulz,
Ghazal, & Garcia-Retamero, 2012; Del Missier et al., 2011; Frederick,
2005; Mækelæ et al., 2018; Morsanyi et al., 2017; Poore et al., 2014;
Primi et al., 2015; Ruffle & Tobol, 2017; Sinayev & Peters, 2015; Ska­
gerlund, Lind, Strömbäck, Tinghög, & Västfjäll, 2018; Szasci et al., 2017;
Toplak et al., 2011, 2014; Weller et al., 2013; Welsh, Burns, & Delfab­
bro, 2013; Zonca et al., 2020). In addition, one relevant characteristic of
most CRTs is that the items consist of arithmetical problems (Frederick,
2005; Kahneman & Frederick, 2002). This characteristic suggests that
numerical ability might explain some degree of the variance of the CRT
scores. For instance, Welsh et al. (2013) found that the predictive val­
idity of the CRT is explained mainly by quantitative reasoning. Kirke­
gaard and Nordbjerg (2015), and Sinayev and Peters (2015) found that
the CRT did not show incremental predictive validity over cognitive
intelligence for grade point average and normative decision-making
tasks. In their study, Blacksmith et al. (2019) found that the CRT
loaded 0.78 on a cognitive intelligence factor. Blacksmith et al. (2019)
suggested that the CRT score might also be capturing numerical ability
because the CRT content and the numerical ability test overlaps
considerably, and because they are assessed with very similar types of
items.
In order to control for the effects of numerical ability, new CRT-type
verbal items were developed. These items can also trigger a quick and
incorrect answer, but they do not require arithmetical operations to
generate the right answer, although most of them involve numbers in
their statements (see, Sirota et al., 2020; Thomson & Oppenheimer,
2016). An example of a verbal item would be the following: if you are
running a race and you pass the person in second place, what place are you
in? (intuitive answer: first place; correct answer: second place; Thomson
& Oppenheimer, 2016). Some studies have shown that the CR tests
composed of verbal items usually correlate moderately with the CR tests
composed of arithmetical items (Patel, 2017; Sirota et al., 2020;
Thomson & Oppenheimer, 2016).
I. Otero et al.
Some research has also examined the relationship between CR and
numeracy, this last variable has been defined as the quantitative skill
that permits us to comprehend and manipulate probabilistic and other
numerical information, conduct basic mathematical operations, under­
stand, and compare risks, and accurately work with decimal represen­
tations, proportions, and fractions (Ghazal, Cokely, & Garcia-Retamero,
2014; Graffeo et al., 2015; Peters et al., 2006; Reyna & Brainerd, 2008;
Reyna, Nelson, Han, & Dieckmann, 2009). Therefore, numeracy may be
considered as learned mathematical knowledge, i.e., the mathematical
knowledge acquired at school. Weller et al. (2013) have suggested that,
although the CRT was not explicitly defined as a numeracy test and only
speculation exists about the underlying dimensions of the CRT, the items
appear to require understanding, manipulating, and the use of numbers
to solve them. Research has supported this assertion. For instance, the
correlations between numeracy skills and CR showed a lower value of
0.29 (Graffeo et al., 2015) and a higher value of 0.65 (Mækelæ et al.,
2018).
Putting together the previous findings and confronting the posited
rationales, the CRT can be placed in a debate between cognitive intel­
ligence models vs. rationality and the dual-process models (Stanovich,
1999, 2009, 2011). In this sense, the proposal of the theorists of the dual-
process model regarding the CRT is that it reflects a different variable or
factor from the general factor of cognitive intelligence (although they
are correlated to some extent). On the other hand, some researchers
claims that the CRT mainly reflects cognitive intelligence, plus some
variance from numerical ability in the case of the numerical-CRTs or
some variance from verbal ability in the case of the verbal-CRTs. A third
position would be of those who suggest that the CRT might be a second
stratum factor in the hierarchical models of cognitive intelligence (e.g.,
in Carroll’s model). This paper aims to shed further light on this debate
by testing the competing models using meta-analysis and meta-analytic
based correlation matrices.
In summary, theoretical approaches and empirical findings have
suggested that CR might be related to a variety of cognitive abilities and
skills, but until now, no meta-analytic examination had been conducted.
Based on the reviewed research, we propose our first hypothesis:
Hypothesis 1. After corrections for measurement error and range re­
striction, CR correlates largely with cognitive intelligence, numerical
ability, verbal ability, mechanical-spatial ability, working memory ca­
pacity, and numeracy skills (and potentially with other cognitive tests
not examined in this meta-analysis).
2. Measures of cognitive reflection
As mentioned above, Frederick’s (2005) CRT was the first test
developed to assess cognitive reflection. The small number of items and
the brief time needed to complete the CRT (2–3 min) have increased its
popularity and use. Numerous studies have shown that CRT scores
correlate with relevant psychological variables, including moral judg­
ments, political conservatism, humor, economic decision-tasks,
dishonest behaviors, scientific and economic literacy, religious beliefs,
grade point average, and job performance (Campitelli & Labollita, 2010;
Deppe et al., 2015; Fosgaard, Hansen, & Piovesan, 2013; Gervais &
Norenzayan, 2012; Lado et al., 2021; Otero, 2019; Otero, Salgado, &
Moscoso, 2021; Pennycook, Cheyne, Barr, Koehler, & Fugelsang, 2014b;
Pennycook, Cheyne, Seli, Koehler, & Fugelsang, 2012; Royzman, Landy,
& Leeman, 2015; Ruffle & Tobol, 2017; Salgado et al., 2019; Shtulman &
McCallum, 2014; Skagerlund et al., 2018; Ventis, 2015; Yılmaz & Sar­
ıbay, 2016; Yılmaz & Sarıbay, 2017).
Nevertheless, Frederick’s CRT has shown several psychometric
weaknesses that must be mentioned. First, the CRT scores are not nor­
mally distributed, showing large kurtosis and skewness levels (see, for
instance, Frederick, 2005; Weller et al., 2013). That fact forces the re­
searchers to use appropriate statistical strategies to deal with non-
parametric data. Consequently, statistical and psychometric analyses
3
Intelligence 90 (2022) 101614
might not produce robust findings if researchers are not making good
data treatment choices. Second, we found that the average CRT’s in­
ternal consistency was 0.64 across 36 coefficients and a total sample of
16,609 (see Table A1 and Table A2 in the supplementary information).
This value agrees with the figures found by Skagerlund et al. (2018) and
Teovanović et al. (2015). This finding suggests that the shared variance
among items tends to be modest (average inter-item correlation = 0.37)
and, therefore, additional items would be required to have higher reli­
ability. Third, in many samples, the CRT’s mean score tends to approach
the minimum range of scores (see, Campitelli & Labollita, 2010; Toplak
et al., 2011), causing a “floor effect.” This effect may limit CRT’s ca­
pacity to differentiate the CR levels in many samples due to the lack of
variability in the scores and range restriction on the distribution of
scores. It is well-known that range restriction affects the correlation
magnitude and, in some cases, reverses the sign of the correlation (see
Salgado, 2016 for a description of these effects).
Moreover, recent research has suggested that the CRT’s growing
popularity may have a substantial influence on the average score. Baron,
Scott, Fincher, & Metz, 2015, Thomson and Oppenheimer (2016), and
Toplak et al. (2014) suggested that the CRT average score may be
currently approaching the maximum range of scores, causing a “ceiling
effect” due to the previous exposure of individuals to the CRT content
(through experimental studies, seminars, conferences, and magazines,
for instance). If this were correct, the lack of variability in the distri­
bution could further limit CRT’s capacity to differentiate the CR level in
many populations.
It may appear paradoxical that two opposite effects can be produced
with the CRT. A potential explanation for the “floor” and “ceiling” ef­
fects is that the different study samples might have different population
ability levels, or they have different exposure time to the original test.
For instance, lower ability level might produce a “floor” effect, but the
repeated exposure to the test might produce a “ceiling” effects, because
as the test consist of three items is easy to remember the correct answer
in repeated exposure of the CRT.
To overcome these previous limitations, researchers have developed
new tests of CR that show better psychometric properties than the
original CRT. For instance, the CRT-10 (Salgado, 2014), consists of 10
items that showed good internal consistency (rxx = 0.79), test-retest
reliability (rxx = 0.78), and a normal distribution (Lado et al., 2021;
Otero, 2019; Salgado et al., 2019). Other researchers developed tests
that consist of one to four items supplemented by the original three
items of the CRT (for instance, Baron, Scott, Fincher, & Metz, 2015;
Finucane & Gullion, 2010; Mata, Schubert, & Ferreira, 2014; Patel,
2017; Primi et al., 2015; Sirota et al., 2020; Ståhl & van Prooijen, 2018;
and Toplak et al., 2014). These additional items typically increased the
measure’s internal consistency, although other weaknesses of the orig­
inal CRT still remained.
Consequently, the variability in the relationships between CR and
cognitive abilities and skills might be a consequence of the diverse types
of CR measures and their different reliabilities. Low reliability produces
an underestimation of the true score correlation between X and Y
(Gulliksen, 1950; Lord & Novick, 1969). This fact suggests that the
variability found in the relationship between CR and cognitive abilities
and skills could be partially caused by the different reliabilities of CR
tests (Schmidt & Hunter, 2015). According to classical test theory
(Gulliksen, 1950; Lord & Novick, 1969), the use of different CRT types
(i.e., differences in the construct measurement) and their different re­
liabilities suggests that the observed correlation between CR and
cognitive abilities and skills might be lower in the studies that used the
original CRT than in the studies that used more reliable CR tests. In
addition, based on the validity generalization hypothesis (Schmidt &
Hunter, 1977, 2015), according to which most of the observed vari­
ability in cognitive ability tests is artifactual, it can be conjectured that
the observed variability in the correlation between CR and cognitive
abilities and skills could be totally or partially explained by artifactual
errors. Based on this rationale, we posit the following hypotheses:
I. Otero et al. Intelligence 90 (2022) 101614

Hypothesis 2. The observed variability in the correlations between CR

and cognitive abilities and skills is largely or totally artifactual (i.e., due
to sampling error, measurement error, and range restriction).
Hypothesis 3. The observed correlations between CR and cognitive
abilities and skills are lower for the studies which use the original CRT
than for the studies which use more reliable CR tests, but the true score
correlations are similar.

3. Aims of the study

As a whole, the empirical evidence suggests that CR is a complex

construct that may reflect both cognitive abilities (e.g., cognitive intel­
ligence, numerical ability, verbal ability, mechanic-spatial ability,
working memory) and cognitive skills (e.g., numeracy skills). Therefore,
this study’s first aim is to conduct a meta-analytic review of the rela­
tionship between CR and several cognitive abilities (i.e., cognitive in­
telligence, numerical ability, verbal ability, mechanic-spatial ability,
and working memory capacity) and skills (i.e., numeracy skills). The
second objective is to determine whether the variability found in the
correlations between CR and cognitive abilities and skills is real or is a
consequence of artifactual errors (e.g., sampling error, measurement
error in CR, and cognitive abilities and skills). The third objective is to
Fig. 1. PRISMA Flowchart of Information Through the Different Phases of a
examine whether the type of CR test (i.e., original 3-item CRT or other
Systematic Review.
CR tests) moderates the relationships between CR and cognitive abilities
and skills. The fourth objective is to determine whether CR is an inde­
of the relationship between CR and cognitive intelligence, cognitive
pendent factor or a second stratum factor in a hierarchical model of
abilities (i.e., numerical ability, verbal ability, mechanic-spatial ability,
cognitive intelligence (e.g., Carroll’s model). The fifth aim is to test a
and working memory) and numeracy skills or, alternatively, (2) other
path model of the relationships between cognitive intelligence, numer­
statistics that allowed for the estimation of the correlation. Perceptual
ical ability, numeracy skill, and CR.
speed was not included in the meta-analysis because we only found two
studies containing perceptual speed and CR measures. College admis­
4. Method
sion tests (e.g., SAT and ACT) were considered cognitive intelligence
measures if they provided a global score. When the study reported the
4.1. Literature search
correlation between CR and SAT and ACT sub-tests, those correlations
were regarded as estimates of CR’s relationships with verbal ability and
We used four strategies to identify published and unpublished pri­
numerical ability.
mary studies on the relationship between CR and cognitive abilities and
We used a single validity coefficient per sample. Therefore, when a
skills. The search covered studies carried out from September 2005 to
study provided two or more correlations because two or more tests
December 2019. First, we conducted electronic searches in the ERIC
evaluated the same ability, we considered these measures as conceptual
database and in Google Scholar using “Cognitive Reflection” and
replications of the same construct. A linear composite with unit weights
“Cognitive Reflection Test” as keywords. Second, we carried out an
for the components was formed in these cases, following Schmidt and
article-by-article search in the following scientific journals: Applied
Hunter’s (2015) suggestions because linear composites are a more
Cognitive Psychology, Cognition, Cognitive Science, Frontiers in Psychology,
construct-valid estimate than the average correlation. However, we used
Journal in Applied Research, Journal of Behavioral Decision Making, Jour­
the latter when the correlations between compound components were
nal of Economic Behavior and Operation, Journal of Experimental Psy­
not reported in the article.
chology: General, Journal of Operations Management, Judgment and
In the case of the conceptual replications, when the sample size was
Decision Making, Memory and Cognition, Mind and Society, Production and
different for each correlation, we applied two rules. First, if the differ­
Operations Management, The Journal of Economic Perspectives, The Journal
ences between sample sizes across the correlations were small, we used
of Socio-Economics (from 2005 to 2014), Journal of Behavioral and
the smallest sample size in the meta-analysis. Second, if the differences
Experimental Economics (from 2014) and Thinking and Reasoning. Third,
between sample sizes were considerable, we used the effect size of the
we examined the reference section of the collected papers to identify
largest sample only.
articles not covered in other search strategies. Fourth, we contacted
It must also be noted that a great deal of research used numeracy
some researchers on the topic to obtain new studies or supplementary
scales (Cokely et al., 2012; Lipkus, Samsa, & Rimer, 2001; Peters &
information.
Levin, 2008; Schwartz, Woloshin, Black, & Welch, 1997; Weller et al.,
We identified 95 records through the electronic searches and 300
2013). These scales were included in the numeracy skills category.
additional records through the other three strategies. Next, we checked
We also included in the meta-analysis studies that administered
the content of these 395 records. We discarded the studies that did not
numerical and verbal CR tests. As numerical-CR test we included: (a)
provide a correlation coefficient or did not permit the calculation of a
Frederick’s 3-item CRT, (b) Frederick’s extended version (i.e., Freder­
correlation between CR, cognitive intelligence, and cognitive abilities
ick’s 3-items plus new items), and (c) CR tests consisting entirely of new
and skills. The final database consisted of 83 documents. A PRISMA
items. The category of verbal-CR tests included only the tests composed
flowchart of information through the different phases of this systematic
exclusively of verbal items. The verbal-CR tests that included numerical
review appears in Fig. 1 (Moher et al., 2009).
items as well as verbal ones were excluded from this category.
When studies reported correlations for both the 3-item CRT and
5. Inclusion criteria and decision rules another CR test for the same sample, we included the correlation ob­
tained for 3-item CRT in the primary analyses. Next, we used the
To be included, the study had to provide (1) a correlation coefficient

4
I. Otero et al. Intelligence 90 (2022) 101614

correlation for the second CR test in the analysis of moderator variables.
As per MARS and PRISMA guidelines, the primary studies included in
the meta-analyses and the relevant information about them (i.e., sample
size, effect size, reliability, and u value of range restriction) can be found
in a supplementary information file.
6. Meta-analytic method
To conduct the meta-analyses, we used the psychometric meta-
analytic method of Schmidt and Hunter (2015). This method permits
us to estimate the average observed correlations (i.e., the bare-bones
meta-analysis) of the relationships and also estimate the correlations
corrected for artifacts (i.e., psychometric meta-analysis).The artifacts
considered in the current meta-analysis were sampling error, the error
measurement in independent and dependent variables (i.e., CR, cogni­
tive intelligence, numerical ability, verbal ability, mechanic-spatial
ability, working memory, and numeracy skills), and the indirect range
restriction in independent variable. Schmidt and Hunter’s (2015)
method provides an estimate of the corrected correlation called true
score correlation. It is obtained when the observed correlation is cor­
rected for measurement error in both independent and dependent var­
iables and for range restriction. In addition, Schmidt and Hunter’s meta-
analytic method permits the estimation of the variability in the observed
correlations due to artifactual errors (i.e., sampling error, measurement
error, and indirect range restriction, in the present case). The meta-
analyses were carried out using Schmidt and Le’s statistical software
(Schmidt & Le, 2014). Specifically, we used the INTNL-I program to
carry out a meta-analysis of correlations using artifact distributions.
Schmidt and Hunter (2015, pp. 189, 207) pointed out that only the
INTNL-I can be used in the case of indirect range restriction.
6.1. Reliability distributions
Not all the studies provided the reliability coefficient required to
individually correct the observed correlation for measurement error. For
this reason, we empirically developed a reliability distribution for each
variable included in this meta-analysis. The distributions were based on
the internal consistency coefficients reported in the primary studies. In
the case of CR, we developed three reliability distributions, one for the
CRT-3, another for the other numerical-CR tests, and a third for the
verbal-CR tests. Table 1 shows the reliability distributions.
6.2. Range restriction distributions
group (i.e., sd). Second, we took the normative standard deviation
provided by the manual of the instruments or by the paper, where the
measure was published, as the standard deviation value of the popula­
tion (i.e., SD). In the case of composites, we calculated the mean u of all
measures integrated in the composite to estimate their range restriction
coefficient.
After estimating the u coefficients of each sample, we developed
empirical distributions of range restriction in the predictors. We
assumed that all cognitive abilities (i.e., cognitive intelligence, numer­
ical ability, verbal ability, and mechanic-spatial ability) are range
restricted in the same way. There is no sign indicating that this sample is
more range restricted in some cognitive abilities than in others. Hence,
one empirical distribution was developed for all cognitive abilities.
Table 2 shows the obtained distributions.
7. Results
7.1. Meta-analyses for the relationship between CR and cognitive
intelligence, cognitive abilities, and numeracy skills
The meta-analytic results for the bivariate combination of CR and
cognitive intelligence, cognitive abilities, and numeracy skills appear in
Table 3. We will mainly focus our comments on the estimates of the true
score correlation.
The results showed that CR correlated highly with all the variables.
Therefore, the findings support Hypothesis 1. The true score correlations
between CR-cognitive intelligence, CR-numerical ability, CR-verbal
ability, CR-mechanic-spatial ability, and CR-working memory were
0.61, 0.62, 0.53, 0.62, and 0.53, respectively. Furthermore, the four
artifactual errors explain most of the observed variability (80% on
average). This last finding supported Hypothesis 2. The lower bound of
the 95% confidence intervals was positive in all the cases, which means
that the true score correlations were clearly different from zero. The
90% credibility values (i.e., the lower bound of the 80% credibility in­
terval) were all positive and notably different from zero, which means
that the correlation is generalizable to other potential studies.
With regard to numeracy skills, the results showed that CR correlated
0.79 with numeracy skills and was the largest true score correlation.
100% of the observed variability is explained by artifactual errors. These
two findings supported Hypotheses 1 and 2. The 95% confidence in­
terval’s lower bound was 0.75, and the 90% credibility value was 0.79.
Taken together, these findings indicate that numeracy skill is the vari­
able that shares most variance with CR.

To estimate the range restriction coefficient (u) of each study, i.e., 7.2. Meta-analyses for the combination of CRT type and cognitive
the ratio of the standard deviation of the restricted group (sd) and the intelligence, cognitive abilities, and numeracy skills
standard deviation of the population (SD), we used the following strat­
egy. First, we took the standard deviation of the cognitive ability scores The third objective of this research is to discover whether the CRT
from each primary study as the standard deviation value of the restricted type affects (moderates) the relationship between CR and the variables
analyzed in this meta-analysis.
Table 1 In order to test the moderator effects, we conveniently classified the
Reliability distributions of the cognitive abilities and cognitive reflection. studies into three groups. The first group consisted of the studies that
used Frederick’s 3-item CRT for assessing CR. The second group con­
K rxx SD Min. - Max.
sisted of the studies that assessed CR with other numerical tests, which
Cognitive Intelligence 14 0.80 0.07 0.68–0.90 typically consisted of a larger number of items. The third group
Numerical Ability 2 0.80 0.11 0.72–0.87
Numeracy Skills 28 0.64 0.09 0.43–0.82
Verbal Ability 6 0.76 0.07 0.65–0.83 Table 2
Mechanic-Spatial Ability 1 0.79 n.a. n.a. Range restriction distributions of the cognitive variables.
Working Memory 1 0.79 n.a. n.a. K u SD Min. - Max.
CRT-3 12 0.61 0.08 0.39–0.70
Other Numerical-CRT 11 0.73 0.08 0.64–0.92 Cognitive Abilities 21 0.83 0.16 0.62–1.20
Verbal-CRT 15 0.65 0.16 0.41–0.85 Numeracy Skills 22 0.92 0.15 0.66–1.28
Working Memory 3 0.84 0.08 0.76–0.90
Note. The average reliabilities are expressed in internal consistency reliability. K
= number of samples; rxx = average reliability coefficient; SD = standard devi­ Note. K = number of independent samples; u = average range restriction value;
ation of reliability; Min. – Max. = minimum and maximum values of reliability; SD = the standard deviation of u; Min. – Max. = minimum and maximum values
n.a. = not applicable; CRT-3 = CR test of Frederick (2005). of u.

5
I. Otero et al. Intelligence 90 (2022) 101614

Table 3
Meta-analytic results on the relationship between cognitive abilities, numeracy skills, and CR.
Meta-analysis of observed correlations Meta-analysis of corrected correlations

K N rw SDr %VESE rcxy ρ SDρ %VEART 90%CV 95%CIρ

Cognitive Intelligence 49 14,905 0.38 0.108 20.5 0.52 0.61 0.090 62.9 0.50 0.56–0.66
Numerical Ability 9 1810 0.39 0.138 18.9 0.54 0.62 0.138 45.8 0.44 0.47–0.76
Numeracy Skills 44 20,307 0.48 0.078 20.9 0.74 0.79 0.000 100 0.79 0.75–0.82
Verbal Ability 23 6330 0.31 0.085 41.3 0.44 0.53 0.000 100 0.53 0.47–0.58
Mechanic-Spatial Ability 3 624 0.38 0.028 100 0.53 0.62 0.000 100 0.62 0.57–0.67
Working Memory 14 1860 0.33 0.089 76.3 0.46 0.53 0.038 91.3 0.48 0.45–0.60

Note. K = number of independent samples, N = sample size; rw = the average observed correlation weighted by the study sample size; SDr = the standard deviation of
the observed correlation; %VESE = the percentage of observed variance explained by sampling error; rcxy = the observed correlation corrected by measurement error in
dependent and independent variables; ρ = true score correlation; SDρ = the standard deviation of ρ; %VEART = the percentage of observed variance explained by
artifactual errors; 90%CV = The 90% credibility value, i.e., the lower-bound of the 80% credibility interval; 95%CIρ = the 95% confidence intervals of ρ.

consisted of the studies that assessed CR with verbal tests, composed
exclusively of verbal items. Table 4 reports the meta-analytic results for
combining CR-test type and cognitive intelligence, specific cognitive
abilities, and numeracy skills.
Comparing the true score correlations found for the 3-item CRT with
those found for the other numerical-CRT, the results reveal that the
numerical-CRT type does not moderate the relationships between CR
and the variables included in this research. For example, the correlations
between CR and cognitive intelligence and verbal ability were virtually
the same for the two types of numerical-CRT. The true score correlation
was very similar for CR-numeracy skills and CR-working memory, but it
was greater for the 3-item CRT in the case of numerical ability. Never­
theless, when the width of the 95% confidence interval is considered, the
overlapping of the intervals is practically total for all variables and the
two types of numerical-CRT. However, the comparison of the observed
correlations obtained for the 3-item CRT and for the other numerical-
CRT shows that the correlations were generally smaller for the 3-item
CRT. Therefore, Hypothesis 3 was fully supported for the numerical-
CRT type.
The explanation of this finding resides in the effects of the lower
reliability of the 3-item CRT. It is well-known that the magnitude of the
reliability estimate depends on the number of parallel items, so that the
larger the number of items, the higher the reliability (Gulliksen, 1950;
Lord & Novick, 1969). Therefore, when the observed correlations are
corrected for measurement error, the attenuating effect of the 3-item
CRT’s reliability disappears.
When comparing the true score correlations found for both the
numerical-CRT and verbal-CRT, the results reveal that the CRT type
appears moderate the relationship between CR and the variables
included in this research. The true score correlation was greater for the
numerical-CRT than for the verbal-CRT in all cases, with the exception
of verbal ability, in which case the true score correlation was greater for
the verbal-CRT. In addition, considering the width of the 95% confi­
dence interval, it can be seen that the intervals do not overlap for all
variables and the two types of CR tests. Nevertheless, these results
should be taken with precaution as the number of studies with the
verbal-CRT is very small.
7.3. Bifactor analysis of CRT, cognitive intelligence, verbal and numerical
abilities and numeracy skills
After establishing the nomological network of CRT and the cognitive
ability variables, two important and related issues remain unanswered.
The two issues relate to the dual-process model vs. cognitive intelligence
model debate for explaining CR variance. First, it is necessary to know if
there is a CR factor that can account for some specific variance in the
CRTs. Second, it is important to know to what extent the common
variance among CRTs and cognitive ability measures (e.g., cognitive

Table 4
Meta-analytic results on the relationship between cognitive abilities, numeracy skills, and CR tests.
Meta-analysis of observed correlations Meta-analysis of corrected correlations

K N rw SDr %VESE rcxy ρ SDρ %VEART 90%CV 95%CIρ

CRT-3
Cognitive Intelligence 37 11,585 0.38 0.099 23.8 0.54 0.64 0.076 70.1 0.54 0.58–0.69
Numerical Ability 7 1488 0.41 0.144 17.6 0.59 0.66 0.153 38.9 0.47 0.49–0.84
Numeracy Skills 33 15,088 0.47 0.079 21.6 0.75 0.80 0.000 100 0.80 0.75–0.84
Verbal Ability 17 5522 0.31 0.079 40.5 0.46 0.55 0.000 100 0.55 0.48–0.62
Working Memory 10 1350 0.31 0.077 100 0.45 0.52 0.000 100 0.52 0.44–0.60
Mechanical-Spatial Ability 3 624 0.38 0.028 100 0.55 0.62 0.000 100 0.62 0.57–0.67

Other Numerical-CRT
Cognitive Intelligence 17 5433 0.42 0.121 14.8 0.55 0.63 0.110 49.4 0.48 0.54–0.71
Numerical Ability 4 518 0.31 0.053 100 0.41 0.49 0.000 100 0.49 0.41–0.57
Numeracy Skills 15 7123 0.50 0.083 17.6 0.73 0.77 0.000 100 0.77 0.71–0.84
Verbal Ability 8 2005 0.34 0.084 45.5 0.46 0.55 0.000 100 0.55 0.45–0.64
Working Memory 4 510 0.38 0.096 62.9 0.50 0.58 0.064 74.9 0.50 0.44–0.72

Verbal-CRT
Cognitive Intelligence 4 803 0.28 0.058 100 0.39 0.46 0.000 100 0.46 0.37–0.56
Numerical Ability 1 137 0.20 – – 0.28 0.34 – – – –
Numeracy Skills 8 2345 0.33 0.076 48.0 0.51 0.56 0.000 100 0.56 0.47–0.65
Verbal Ability 2 572 0.41 0.065 100 0.58 0.67 0.000 100 0.67 0.53–0.82

Note. K = number of independent samples, N = sample size; rw = the average observed correlation weighted by the study sample size; SDr = the standard deviation of
the observed correlation; %VESE = the percentage of observed variance explained by sampling error; rcxy = the observed correlation corrected by measurement error in
dependent and independent variables; ρ = true score correlation; SDρ = the standard deviation of ρ; %VEART = the percentage of observed variance explained by
artifactual errors; 90%CV = The 90% credibility value, i.e., the lower-bound of the 80% credibility interval; 95%CIρ = the 95% confidence intervals of ρ; CRT-3 = CR
test of Frederick (2005).

6
I. Otero et al.
intelligence tests, numerical ability tests, verbal ability tests, and
numeracy skill tests) can be explained by a general factor of cognitive
intelligence. In other words, do the CR tests consist mainly of CR vari­
ance, cognitive intelligence variance, or both? Bifactor analysis and
recently developed reliability coefficients can provide a means for
examining these two issues (Reise, 2012; Revelle, 2021; Rodriguez,
Reise, & Haviland, 2016).
In order to conduct a bifactor analysis, we used a meta-analytically
derived correlation matrix. The full corrected correlations between the
studied variables reported in the previous tables were used in this ma­
trix. In addition, using the correlations reported in the studies contained
in the database, we estimated the corrected correlation between
cognitive intelligence, numerical ability, verbal ability, and numeracy
skills. This meta-analytically-derived correlation matrix has two
important properties. First, they are the best estimates of the correla­
tions between the variable as they represent the weighted-size average
corrected correlation based on a large number of studies. Second, these
correlations have a smaller amount of error than the observed correla­
tions. Third, they have less error than any study-based estimates.
To overcome the problem that the sample size of the true score
correlations varies across the matrix cells, given that each correlation in
the matrix has been obtained from a different meta-analysis, Viswes­
varan and Ones (1995) suggested the use of the harmonic mean of the
sample sizes across the different cells. Also, other researchers recom­
mend the use of the harmonic mean, for example, Bergh et al. (2016),
Burke and Landis (2003), Eisend and Küster (2011), Landis (2013), Pick
and Eisend (2014), Podsakoff, Whiting, Podsakoff, and Blume (2009),
Ratcliff (1993), and Verbeke, Dietz, and Verwaal (2011). Other authors
have used the smallest N and others the arithmetic mean (see, Bergh
et al., 2016, for a review). What is important is that the use of the har­
monic mean, smallest N or the arithmetic mean does not substantially
change the findings regarding the model fit (Westrick, Schmidt, Le,
Robbins, & Radunzele, 2021). When a matrix of correlations is subject to
structural equation modelling (e.g., path analysis and factor analysis),
the sample size is typically used to estimate some model fit indexes
(Viswesvaran & Ones, 1995).
In this study, we have used a Schmid-Leiman bifactor analysis
(Lorenzo-Seva & Ferrando, 2021; Reise, 2012; Revelle, 2021) and six
reliability indexes (Rodriguez et al., 2016): omega, omega subscale,
omega hierarchical, omega hierarchical subscale, coefficient H and
explained common variance (ECV). Next, we succinctly describe each of
these indexes.
According to Revelle (2016, see also, Gignac, 2015; McDonald, 1999;
Rodriguez et al., 2016; Westrick et al., 2021), coefficient omega is based
on a factor analysis that permits the decomposition of the total variance
into four categories: (a) the variance attributed to a general factor with
the variance common to all the assessed variables (e.g., items or tests),
(b) the common variance attributed to group factors (e.g., groups of
items or groups of tests), (c) the unique variance attributed to specific
factors (e.g., an item or a single test) and (d) random error. The unique
variance cannot be separated from random error in a single test
administration (the separation requires at least two test administra­
tions). Therefore, the estimation of the omega coefficient combines both
variance types in a single estimate named uniqueness.
Rodriguez et al. (2016) suggest that two important advantages of the
omega coefficient are that it serves to inform of (a) the quality of the unit
weighted total and subscale score composites, and (b) the specification
and quality of a measurement model in structural equation modelling.
The partition of the variance into four categories permits the computa­
tion of four different omega coefficients (Rodriguez et al., 2016): (a)
omega total, (b) omega subscale (i.e., omega for the group factor), (c)
omega hierarchical, and (d) omega hierarchical subscale (i.e., omega
hierarchical for the group factor). The main difference between the four
omega coefficients is the amount of true (common) variance used in the
numerator and the amount of error variance used in the denominator.
Omega (ω) is the ratio of true over observed score, or the ratio of the
7
Intelligence 90 (2022) 101614
common variance to the total variance. Omega estimates the proportion
of variance in the observed total score which can be attributed to all the
modelled sources of common variance (Gignac, 2015; Rodriguez et al.,
2016). Omega subscale (ωS) estimates the proportion of the total vari­
ance of the subscale score which can be attributed to both the general
factor and the group factor.
Omega hierarchical (ωH) indicates the proportion of total score
variance that can be attributed to a single general factor. In other words,
omega hierarchical indicates the strength of a general factor on a stan­
dardized metric and ranges from 0 to 1 (Gignac, 2015). It is essentially
the ratio of common variance to total variance. For instance, an omega
hierarchical of 0.80 indicates that the general factor accounts for 80% of
the total variance in the scores. The comparison between omega and
omega hierarchical permits us to know if the observed scores are
essentially indicators of a general factor or if they are affected by group
factors (Rodriguez et al., 2016; Westrick et al., 2021). When omega and
omega hierarchical are similar, one can conclude that the observed
scores are essentially indicators of the general factor. Otherwise, the
conclusion is that the observed scores are affected by some or all the
modelled factors.
The omega hierarchical subscale (ωHS) represents the proportion of
scale (i.e., group factor) score variance that can be attributed to a group
factor, after removing the reliable variance due to the general factor
(Revelle, 2016; Rodriguez et al., 2016). If the omega subscale and the
omega hierarchical subscale are similar, then most of the variance of the
scale is accounted for by the factor group. Otherwise, the general factor
accounts for much of the reliable variance.
The coefficient H (Hancock, 2001; Hancock & Mueller, 2001) is an
estimate of the construct (i.e., factor) replicability that serves to judge
the extent to which a factor is represented by a set of indicators or items
(Rodriguez et al., 2016; Westrick et al., 2021). When H is high the factor
is well defined by the indicators and will tend to be stable across studies.
In connection with the coefficient H, Hancock and Mueller (2001) sug­
gested that a value of at least H = 0.70 must be obtained as a standard
criterion.
The last index we used is the proportion of explained common
variance (ECV) accounted for by the general factor. In other words, ECV
is the proportion of the variance explained by a general factor divided by
the sum of the explained variance of the general factor plus the
explained variance of the group factors (assuming that the group factors
are uncorrelated). A higher ECV value indicates a strong general factor
(Reise, 2012).
In order to calculate the six reliability indexes mentioned above, we
conducted an exploratory factor analysis followed by a Schmid-Leiman
transformation (Revelle, 2021; Schmid & Leiman, 1957). We used the
program FACTOR, version 11 (Lorenzo-Seva & Ferrando, 2021), with
unweighted least squared (ULS) as the method for factor extraction, and
Promin as the method of oblique rotation (Lorenzo-Seva, 1999). We
carried out two factor analyses. The first one in which cognitive
reflection was represented by the whole set of CR tests (i.e., both nu­
merical and verbal). The second factor analysis used six variables, as
cognitive reflection was assessed independently by numerical-CRTs and
verbal-CRTs. Tables 5 and 6 report the corrected correlation matrix of
variables used in the two factor analyses.
Tables 7 and 8 show the loadings of the variables on the group factors
and the general factor, as well as the indexes-of-fit for the two models.
Table 5
Corrected correlation matrix of cognitive variables and CRT.
Variables 1 2 3 4 5
1. CRT –
2. Cognitive Intelligence 0.61 –
3. Numerical Ability 0.62 0.55 –
4. Numeracy Skills 0.79 0.52 0.62 –
5. Verbal Ability 0.53 0.48 0.48 0.69
–
I. Otero et al. Intelligence 90 (2022) 101614

Table 6 87% of numerical-CRT and 43% of verbal-CRT variance was accounted

Corrected correlation matrix between cognitive variables and CRT type. for by the general factor (cognitive intelligence) and the numerical
Variables 1 2 3 4 5 6 factor.
An interesting finding is that the larger portion of the common
1. Numerical-CRT –
2. Verbal-CRT 0.64 – variance of both the numerical-CRT and the verbal-CRT is explained by
3. Cognitive Intelligence 0.64 0.46 – the general intelligence factor and, to a lesser extent, by the numerical
4. Numerical Ability 0.62 0.37 0.55 – factor, but that the verbal factor does not explain variance at all. This
5. Numeracy Skills 0.79 0.56 0.52 0.62 – fact could suggest that there are no differences between the two types of
6. Verbal Ability 0.55 0.67 0.48 0.48 0.69
CRT. However, this conclusion could be inaccurate if one take into ac­
–

count the communality (h2) for each CRT type, which can be calculated
from the figures of Table 8. In doing so, it can be seen that the com­
Table 7 munality for the numerical-CRT is 0.87 while the communality for the
Factor loadings after Schmid-Leiman transformation. verbal-CRT is 0.43. Therefore, the unexplained variance (1− h2) of the
Variable Factor 1 Factor 2 General verbal-CRT is noticeably larger than the unexplained variance of the
Cognitive Reflection 0.035 0.429 0.797 numerical-CRT (0.57 vs. 0.13). The difference in the amount of unex­
Cognitive Intelligence − 0.016 0.319 0.585 plained variance suggests that an additional specific factor, for instance,
Numerical Ability 0.326 0.250 0.695 a content factor, might explain a portion of the unexplained variance of
Numeracy Skills − 0.012 0.415 0.736
the verbal-CRT. Thus, although both kinds of CRT’s reflect the variance
Verbal Ability 0.798 0.000 0.603
of cognitive intelligence and numerical ability, the full explanation of
Indexes of fit: RMR = 0.032; CFI = 0.997; NNFI = 0.973; GFI = 0.998. the variance of the verbal-CRT would require the partial contribution of
one or more specific factors.
Table 9 reports the estimates for the six reliability indexes for the two
Table 8
group factors and the general factor obtained using the loadings re­
Factor loadings after Schmid-Leiman transformation.
ported in Table 7 (the results were essentially the same when we used
Variable Factor 1 Factor 2 General the loadings reported in Table 8). As can be seen in the second column,
Numerical-CRT 0.007 0.430 0.828 the omega coefficients for the general factor and the group factors are
Verbal-CRT 0.060 0.282 0.591 very high, ranging from 0.82 to 0.91. This means that the true variance
Cognitive Intelligence − 0.069 0.316 0.596 is 82% for the verbal factor, 89% for the numerical factor, and 91% for
Numerical ability 0.307 0.217 0.673
Numeracy Skills − 0.035 0.402 0.740
the general factor.
Verbal Ability 0.765 0.000 0.644 The third column reports the omega hierarchical and omega hier­
archical subscale coefficients. The comparison of the omega (variance
Indexes of fit: RMR = 0.044; CFI = 0.989; NNFI = 0.959; GFI = 0.988.
accounted for by the general and the group factors) and the omega hi­
erarchical (variance accounted for by the general factor alone) shows
The two factor analyses with the Schmid-Leiman Transformation that most of the true variance in the general factor can be attributed to
showed a general factor and two primary factors. In the first factor the general factor (0.81 / 0.91 = 0.89). In contrast, the comparison of
analysis, verbal ability shows its highest loading on the first factor and the omega subscale (variance accounted for by the general and the
an important secondary loading on the general factor. The other four group factor) and the omega hierarchical subscale (the variance
variables show their highest loading on the general factor and a sec­ accounted for the group factor alone) shows a different picture. In the
ondary loading on the second factor, except for numerical ability which case of the verbal factor, this factor accounts for 41% of the true variance
shows secondary loadings on both primary factors. Considering the (0.34 / 0.82 = 0.41), and the general factor accounts for 59% of the true
variables that load on each factor, the first one reflects a verbal ability variance. For the numerical factor, 75.3% of the true variance is
factor and the second one reflects a numerical ability factor. The cor­ accounted for by the general factor and only 24.7% of the true variance
relation between the verbal ability factor and the general factor was 0.60 is explained by the numerical factor (0.22 / 0.89 = 0.247).
and the correlation between the numerical ability factor and the general The results for coefficient H appear in the fourth column. The results
factor was 0.87. The indexes of fit, both absolute (RMR) and relative reveal that the general factor shows evidence of construct replicability
(CFI, GFI, NNFI) were excellent according to Hu and Bentler’s (1999) as its coefficient H is 0.86, which is noticeably greater than 0.70.
cut-off points. It is important to note that the correlation between However, neither of the two group factors reach an acceptable level of
cognitive reflection and the general factor was 0.80 (rounded) and it was construct replicability as they do not meet the 0.70 criterion.
0.43 (rounded) with the numerical ability factor. This finding indicates Finally, the results for the ECV ratio, reported in the last column,
that 80% of CRT variance was accounted for by these two factors. indicate that the general factor explains 69% of the common variance,
The second factor analysis shows very similar results. Again, verbal and that the explained common variance is only 17% and 14% for the
ability shows its highest loading on the first factor and an important verbal and numerical factors, respectively. In other words, most of the
secondary loading on the general factor. The other five variables show
their highest loading on the general factor and a secondary loading on
the second factor, except for numerical ability which shows secondary
Table 9
loadings on both primary factors. Considering the variables that load on Omega and H coefficients, and ECV ratio for the group factors and the general
each factor, the first one reflects a verbal ability factor and the second factor.
one reflects a numerical ability factor. The correlation between the
Variable ω/ωS ωH/ωHS H ECV
verbal ability factor and the general factor was 0.64 and the correlation
between the numerical ability factor and the general factor was 0.89. General Factor 0.91 0.81 0.86 0.69
Factor 1 (Verbal) 0.82 0.34 0.60 0.17
The indexes of fit, both absolute (RMR) and relative (CFI, GFI, NNFI) Factor 2 (Numerical) 0.89 0.22 0.40 0.14
were excellent according to Hu and Bentler’s (1999) cut-off points. With
regard to the correlation between cognitive reflection and the general Note. ω = omega coefficient for the total common variance; ωS = omega coef­
ficient for the common variance of the group factor; ωH = omega hierarchical
factor, it is larger for numerical-CRT than for verbal-CRT (0.83 vs. 0.59).
coefficient for the general factor; ωHS = omega hierarchical coefficient for the
Again, these correlations shows that CRT tests mainly reflect cognitive
group factor; H = H coefficient of construct replicability; ECV = proportion
intelligence plus numerical ability. Concerning the explained variance,
explained common variance.

8
I. Otero et al.
common variance is explained by the general factor, with small contri­
butions from the group factors.
In summary, the results of this set of analyses indicate that both the
numerical-CRT and the verbal-CRT consist of mainly cognitive intelli­
gence variance plus a small contribution from a numerical ability factor.
The results do not show evidence supporting the existence of a cognitive
reflection factor.
7.4. Path analysis model of the relationships between CR, cognitive
intelligence, numerical ability, and numeracy skills
The final objective of this research was to test the meta-analytic path
analysis presented in Fig. 2. As can be seen, the path model assumes that
CR is directly explained by cognitive intelligence, numerical ability, and
numeracy skills, that cognitive intelligence and numerical ability indi­
rectly explain CR through numeracy skills, and that cognitive intelli­
gence explains CR through numerical ability.
To conduct the path analysis, we used the meta-analytically derived
correlation matrix reported in Table 5 for the relationship between CR,
cognitive intelligence, numerical ability, and numeracy skills. The path
coefficients for the hypothesized model appear in Fig. 2.
As can be seen, the six path coefficients are positive. Still, the
magnitude of the path coefficients from cognitive intelligence to CR and
numerical ability to CR are small compared to the path coefficient from
numeracy skills to CR. The path coefficients from cognitive intelligence
to numeracy skills and numerical ability to numeracy skills are notice­
ably larger. The path coefficient from cognitive intelligence to numerical
ability is the largest. Fig. 2 shows that cognitive intelligence, numerical
ability, and numeracy skills explained 69% of CR variance (R = 0.83, p
< .001) and that cognitive intelligence and numerical ability explained
43% of numeracy skills variance (R = 0.66, p < .001). The fit was perfect
for the model.
The results showed that, when the direct and indirect effects are
jointly considered, the total effect of cognitive intelligence on CRT is
0.61 and the total effect of numerical ability is 0.25. The effect of nu­
merical skills on CRT is totally due to the indirect effects of cognitive
intelligence and numerical ability. In other words, numerical skills do
not contribute to the explanation of CRT variance. In terms of the
explained variance of CRT, cognitive intelligence accounts for by 71% of
the explained variance (0.71 × 0.69 = 0.49) and numerical ability ac­
counts for by 29% of the explained variance (0.29 × 0.69 = 0.20).
Therefore, cognitive intelligence explained two and a half times more
CRT variance than numerical ability.
Also, we carried out the path analyses for the numerical-CRT and
verbal-CRT separately, using the correlation matrix reported in Table 6.
The results for numerical CRT were essentially like those of the previous
model. We found that cognitive intelligence, numerical ability, and
numeracy skills explained 70% of CR variance (R = 0.84, p < .001) and
Fig. 2. Path Analysis Model of the Relationships between Cognitive Reflection,
Cognitive Intelligence, Numerical Ability, and Numeracy Skills.
Note. CI = cognitive intelligence; NA = numerical ability; NS = numeracy skills;
CR = cognitive reflection; R = multiple correlation; R2 = squared multiple
correlation; β = standardized regression weights. All β values are p < .001.
9
Intelligence 90 (2022) 101614
that cognitive intelligence and numerical ability explained 43% of
numeracy skills variance (R = 0.66, p < .001). Also, the fit was perfect
for this model. In the case of verbal-CRT, we added verbal ability as a
direct predictor of verbal-CRT, and we found that cognitive intelligence,
numerical ability, verbal ability, and numeracy skills explained 67% of
CR variance (R = 0.82, p < .001). The goodness-of-fit indexes were
SRMR = 0.071, GFI = 0.96, CFI = 0.96, NFI = 0.96, and IFI = 0.96.
According to Kline (Kline, 2011; see also Hu & Bentler, 1999; Fan &
Sivo, 2005; Marsh, Wen, & Hau, 2004), acceptable values of fit based on
these indexes are SRMR <0.10, CFI >0.95, GFI > 0.95 and NFI > 0.95.
8. Discussion
This study was a meta-analytic examination of the research carried
out over the last sixteen years on the relationship between cognitive
reflection and cognitive abilities and skills. We aimed to expand the
literature on the nomological network of CR as well as to provide an
answer to the debate between the dual-process model of human cogni­
tion and the hierarchical model of intelligence regarding the place of
cognitive reflection in the cognitive domain. In doing this research, this
study has made five contributions to the literature on CR.
The first goal of this meta-analysis was to provide an estimate of the
true score correlation between cognitive reflection, and cognitive in­
telligence, verbal ability, numerical ability, working memory,
mechanical-spatial ability, and numeracy skills. In connection with this
goal, the first contribution of this research has been to show that
cognitive reflection is highly correlated with those cognitive variables,
the correlations ranging from 0.53 to 0.79, with an average correlation
of 0.65. In addition, we showed that the observed variability found in
the primary studies was mostly artifactual, in line with the validity
generalization hypothesis (Schmidt & Hunter, 1977, 2015). The results
unequivocally showed that there is validity generalization for all
cognitive reflection-cognitive abilities relationships, as the 90% credi­
bility values were all positive and very different from zero.
The second contribution was the comparison of the relationship
between the two types of cognitive reflection tests (i.e., numerical and
verbal) and the cognitive abilities and skills. Our findings showed that
those relationships are not affected by the way in which cognitive
reflection is assessed. When we grouped the studies in two categories (i.
e., numerical-CRT vs. verbal-CRT) or three different categories (3-item
CRT, other numerical-CRT, and verbal-CRT), we found essentially the
same pattern of correlations, we found that the variability in the
observed correlations was totally artifactual for verbal-CRT and almost
totally artifactual for numerical-CRT, in accordance with the validity
generalization hypothesis.
The third, and probably, the most relevant contribution of this
research has been to show that cognitive reflection variance is mainly
due to the general factor of intelligence plus some variance from nu­
merical ability. We found that cognitive reflection correlates 0.80
(rounded) with the general factor of intelligence. In our bifactor anal­
ysis, we were not able to find a specific factor of cognitive reflection.
Therefore, we have to conclude that this meta-analysis did not find
support for the hypothesis of cognitive reflection as a cognitive factor
independent or distinct from cognitive intelligence and cognitive ability.
In fact, the findings strongly support that cognitive reflection consists of
cognitive intelligence variance plus numerical ability variance. Conse­
quently, placing our findings in the debate between the dual-process
model and the hierarchical intelligence model, our findings clearly
supported the position of the hierarchical intelligence model. Further­
more, the bifactor analysis showed that cognitive reflection cannot be
postulated as a specific factor of the second stratum in the hierarchical
model of intelligence. The findings also showed that numerical-CRT and
verbal-CRT reflect the same factors (i.e., general factor and numerical
factor), which rules out the existence of cognitive differences between
two CRT types. Despite of the appearance of the item (i.e., verbal or
numerical), numerical-CRT and verbal-CRT are essentially similar, and
I. Otero et al.
the item content is not a relevant feature with regard to the reflected
factors.
The fourth contribution has been the examination of the true score
variance using recently developed reliability coefficients based on factor
analysis. These analyses indicated that the true variance of the numer­
ical factor and verbal factor is mainly attributable to the general factor of
intelligence.
The final contribution was to show that around 70% of CRT variance
is explained by cognitive intelligence and numerical ability, which show
both direct and indirect effects (through numeracy skills) on cognitive
reflection. Also, the path model showed that the correlation between
numeracy skills and cognitive reflection is due to the direct and indirect
effects of cognitive intelligence and numerical ability on numeracy
skills. In other words, numeracy skill does not add incremental validity
over cognitive intelligence and numerical ability for explaining cogni­
tive reflection variance.
The findings of this study have implication for the theory, research,
and practice. From a theoretical point of view, the findings of this study
indicate that, contrary to the view of Stanovich, Toplak and associates
(Stanovich, 2009; Toplak et al., 2011, 2014), according to whom the
CRT assesses properties relevant to rational thinking that go beyond
those assessed on cognitive intelligence tests, our findings showed that
the CRTs are basically cognitive ability tests which reflect cognitive
ability variance and numerical ability variance fundamentally. We did
not find evidence of a separate and distinct factor that reflects thinking
dispositions related to reflectivity, the tendency to engage in fully
disjunctive reasoning, and the tendency to seek alternative solutions.
Again, our findings indicate that the CRTs are actually measures of
cognitive intelligence rather than measures of rational thought. As the
CRTs are mainly cognitive ability tests, the current findings also imply
that cognitive ability tests might also assess the tendency towards
miserly processing.
In the second place, the results suggest that the ability to learn a task,
a subject, or a skill accurately and quickly, under optimal instruction
conditions (i.e., cognitive intelligence), the ability to comprehend and
manipulate numerical information (i.e., numerical ability), and the
mathematical knowledge acquired at school (i.e., numeracy skills)
mainly determine the CR level of individuals.
From the point of view of research, our findings suggest that the
studies that found evidence of incremental validity of CRT for predicting
some dependent variables (e.g., rational thinking performance) may
have some flaws in the estimation of the incremental validity of the CRT.
As the CRT mostly consists of cognitive intelligence variance and nu­
merical ability variance, in order to know the true incremental validity
of the CRT, its specific variance must be residualized, i.e., CRT specific
variance must be separated from the variance of cognitive intelligence
and other cognitive abilities. None of the studies that examined the in­
cremental validity of the CRT residualized CRT variance prior to
entering it in the multiple regression analysis and this analysis does not
permit variance residualization. Three steps should be carried out in
order to properly examine the incremental validity of the CRT. The first
step should be to separate the variance of cognitive ability and numer­
ical ability from the CRT variance (specific variance). Bifactor analysis
and hierarchical factor analysis with a Schmid-Leiman transformation
can be used for this purpose. Next, researchers must carry out three
regression analyses, using three sets of factor scores as predictors: (a) the
factor scores of cognitive intelligence and numerical ability variance
alone; (b) the factor scores of the specific CRT variance (i.e., residualized
CRT) alone; and (c) the factor scores of cognitive intelligence and nu­
merical ability supplemented by the residualized CRT. The first regres­
sion analysis would permit the estimation of the validity of cognitive
intelligence and numerical ability for predicting rational thinking per­
formance. The second regression analysis would permit the estimation
of the validity of the residualized CRT. The third multiple regression
analysis would permit the estimation of the incremental validity of the
residualized CRT. Researchers have to be aware that factor score
10
Intelligence 90 (2022) 101614
regression can show some degree of bias in the estimation of regression
coefficients (e.g., Croon, 2002; Devlieger, Mayer, & Rosseel, 2016;
Hayes & Usami, 2020). Recent methodological developments provide
useful tools to avoid the bias and also to estimate fit indices when factor
score regression methods are used (Croon, 2002; Devlieger, Talloen, &
Rosseel, 2019; Hayes & Usami, 2020). As shown in several Montecarlo
studies, Croon’s formulas (Croon, 2002) and Croon’s modified formulas
(Hayes & Usami, 2020) showed to outperform typical factor score
regression and also SEM in some circunstance. Devlieger et al. (2016),
Devlieger et al. (2019), and Hayes and Usami (2020) provide recom­
mendations for using these formulas and the fit indices. Future research
should be conducted using this approach (or alternatively using SEM) to
clarify the extent to which CRT shows incremental validity over cogni­
tive intelligence and other cognitive abilities.
From an applied point of view, these findings indicate that, although
CR tests might be used in practical situations (i.e., situations in which a
decision on an individual must be made), as they correlated with prac­
tical criteria such as job performance, grade point average, rational
thinking performance (Otero et al., 2021; Toplak et al., 2014), CR tests
are not a substitute or a supplement for cognitive intelligence tests for
predicting those criteria.
The current research has some limitations that should be mentioned.
An initial limitation refers to the small number of studies and the small
sample size included in some CR-cognitive ability combinations. For
instance, this was the case for the CR-numerical ability, CR-mechanic-
spatial ability, and CR-working memory relationships. Therefore, new
primary studies are required to expand the current meta-analysis.
Another limitation is that we could not find enough studies examining
the relationship between CR and other cognitive variables such as
perceptual speed and the executive functions of monitoring and inhi­
bition. Therefore, a meta-analysis devoted to these relationships should
be conducted in the future.
In summary, this paper presents the first meta-analysis on the rela­
tionship between CR and several cognitive abilities (i.e., cognitive in­
telligence, verbal ability, numerical ability, mechanical-spatial ability,
and working memory) and skills (i.e., numeracy skills). The findings
showed that CR correlated with all these variables and that the rela­
tionship between CR and numeracy skills, cognitive intelligence, and
numerical ability is exceptionally high. Also, the findings showed that
the majority, and in some case all, of the observed variability in the
correlation between CRT and the cognitive abilities and skills is a sta­
tistical artifact (i.e., the variability is due to sampling error, measure­
ment error in X and Y, and range restriction). Furthermore, this research
shows that CR is a construct mainly accounted for by cognitive intelli­
gence, and numerical ability. Finally, a path meta-analytic model
showed that cognitive intelligence and numerical ability have direct and
indirect effects on CR.
Acknowledgements
All authors have made a substantial, direct, and equal contribution to
this research. We have no known conflict of interest to disclose.
This work was partially supported by the Spanish Ministry of Science
and Innovation [grant number PID 2020-116409GB-I00] to Jesús F.
Salgado and Silvia Moscoso.
Appendix A. Supplementary data
Supplementary data to this article can be found online at https://doi.
org/10.1016/j.intell.2021.101614.
I. Otero et al. Intelligence 90 (2022) 101614

References* Gignac, G. E. (2015). Estimating the strength of a general factor: Coefficient omega
hierarchical. Industrial and Organizational Psychology: Perspectives on Science and
Practice, 8(3), 434–438. https://doi.org/10.1017/iop.2015.59
*Avram, L. A. (2018). Gender differences and other findings on the cognitive reflection
*Gómez-Chacón, I. M., García-Madruga, J. A., Vila, J.Ó., Elosúa, M. R., & Rodríguez, R.
test. Studia Universitatis Babes-Bolyai Oeconomica, 63(3), 56–67. https://doi.org/
(2014). The dual processes hypothesis in mathematics performance: Beliefs,
10.2478/subboec-2018-0014
cognitive reflection, working memory and reasoning. Learning and Individual
Baron, J., Scott, S., Fincher, K., & Metz, S. E. (2015). Why does the cognitive reflection
Differences, 29, 67–73. https://doi.org/10.1016/j.lindif.2013.10.001
test (sometimes) predict utilitarian moral judgment (and other things)? Journal of
*Graffeo, M., Polonio, L., & Bonini, N. (2015). Individual differences in competent
Applied Research in Memory and Cognition, 4(3), 265–284. https://doi.org/10.1016/j.
consumer choice: The role of cognitive reflection and numeracy skills. Frontiers in
jarmac.2014.09.003
Psychology, 6, 844. https://doi.org/10.3389/fpsyg.2015.00844
Bergh, D. D., Aguinis, H., Heavey, C., Ketchen, D. J., Boyd, B. K., Su, P., … Joo, H.
Gulliksen, H. (1950). Theory of mental tests. John Wiley & Sons, Inc.
(2016). Using meta-analytic structural equation modeling to advance strategic
Hancock, G. R. (2001). Effect size, power, and sample size determination for structured
management research: Guidelines and an empirical illustration via the strategic
means modeling and MIMIC approaches to between-group hypothesis testing of
leadership-performance relationship. Strategic Management Journal, 37(3), 477–497.
mean son a single latent construct. Psychometrika, 66(3), 373–388. https://doi.org/
https://doi.org/10.1002/smj.2338
10.1007/BF02294440
*Blacksmith, N., Yang, Y., Behrend, T. S., & Ruark, G. A. (2019). Assessing the validity of
Hancock, G. R., & Mueller, R. O. (2001). Rethinking construct reliability within latent
inferences from scores on the cognitive reflection test. Journal of Behavioral Decision
variable systems. In R. Cudeck, S. du Toit, & D. Sörbom (Eds.), Structural equation
Making, 32(5), 599–612. https://doi.org/10.1002/bdm.2133
modelling: Present and future – A festschrift in honor to Karl Jöreskog (pp. 195–216).
Böckenholt, U. (2012). The cognitive-miser response model: Testing for intuitive and
Scientific Software International.
deliberate reasoning. Psychometrika, 77(2), 388–399. https://doi.org/10.1007/
Hayes, T., & Usami, S. (2020). Factor score regression in the presence of correlated
s11336-012-9251-y
unique factors. Educational and Psychological Measurement, 80(1), 5–40. https://doi.
Burke, M. J., & Landis, R. S. (2003). Methodological and conceptual challenges in
org/10.1177/0013164419854492
conducting and interpreting meta-analyses. In K. R. Murphy (Ed.), Validity
Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure
generalization: A critical review (pp. 287–309). Erlbaum.
analysis: Conventional criteria versus new alternatives. Structural Equation Modeling:
*Campitelli, G., & Gerrans, P. (2014). Does the cognitive reflection test measure
A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/
cognitive reflection? A mathematical modeling approach. Memory & Cognition, 42
10705519909540118
(3), 434–447. https://doi.org/10.3758/s13421-013-0367-9
Hunt, E. A. (2011). Human intelligence. Cambridge University Press.
Campitelli, G., & Labollita, M. (2010). Correlations of cognitive reflection with
Jensen, A. R. (1998). The g factor: The science of mental ability. Praeger.
judgments and choices. Judgment and Decision Making, 5(3), 182–191. http://journal.
Kahneman, D. (2011). Thinking fast and slow. Farrar, Straus, and Giroux.
sjdm.org/10/91230/jdm91230.pdf.
Kahneman, D., & Frederick, S. (2002). Representativeness revisited: Attribute
Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies.
substitution in intuitive judgment. In T. Gilovich, D. Griffin, & D. Kahneman (Eds.),
Cambridge University Press.
Heuristics and biases: The psychology of intuitive judgment (pp. 49–81). Cambridge
*Cokely, E. T., Galesic, M., Schulz, E., Ghazal, S., & Garcia-Retamero, R. (2012).
University Press.
Measuring risk literacy: The Berlin numeracy test. Judgment and Decision Making, 7
Kahneman, D., & Frederick, S. (2005). A model of heuristic judgment. In K. J. Holyoak, &
(1), 25–47.
R. G. Morrison (Eds.), The Cambridge handbook of thinking and reasoning (pp.
Croon, M. (2002). Using predicted latent scores in general latent structure models. In
267–293). Cambridge University Press.
G. Marcoulides, & I. Moustaki (Eds.), Latent variable and latent structure modeling (pp.
Kirkegaard, E. O., & Nordbjerg, O. (2015). Validating a Danish translation of the
195–223). Lawrence Erlbaum.
international cognitive ability resource sample test and cognitive reflection test in a
Deary, I. J. (2012). Intelligence. Annual review of Psychology, 63, 453–482. https://doi.
student sample. Open Differential Psychology, 4. https://doi.org/10.26775/
org/10.1146/annurev-psych-120710-100353
ODP.2015.07.31
*Missier, F., Mäntylä, T., & De Bruin, W. B. (2011). Decision-making competence,
Kline, R. B. (2011). Convergence of structural equation modeling and multilevel
executive functioning, and general cognitive abilities. Journal of Behavioral Decision
modeling. In R. B. Kline, M. Williams, & W. P. Vogt (Eds.), Convergence of structural
Making, 25(4), 331–351. https://doi.org/10.1002/bdm.731
equation modeling and multilevel modelling. Sage. https://doi.org/10.4135/
Deppe, K. D., Gonzalez, F. J., Neiman, J. L., Jacobs, C., Pahlke, J., Smith, K. B., &
9781446268261.n31.
Hibbing, J. R. (2015). Reflective liberals and intuitive conservatives: A look at the
Koscielniak, M., Rydzewska, K., & Sedek, G. (2016). Effects of age and initial risk
cognitive reflection test and ideology. Judgment & Decision Making, 10(4), 314–331.
perception on Balloon Analog Risk Task: The mediating role of processing speed and
Devlieger, I., Mayer, A., & Rosseel, Y. (2016). Hypothesis testing using factor score
need for cognitive closure. Frontiers in Psychology, 7, 659. https://doi.org/10.3389/
regression: A comparison of four methods. Educational and Psychological
fpsyg.2016.00659
Measurement, 76(5), 741–770. https://doi.org/10.1177/0013164415607618
Lado, M., Otero, I., & Salgado, J. F. (2021). Cognitive reflection, life satisfaction,
Devlieger, I., Talloen, W., & Rosseel, Y. (2019). New developments in factor score
emotional balance, and job performance. Psicothema, 33(1), 118–124. https://doi.
regression: Fit indices and a model comparison test. Educational and Psychological
org/10.7334/psicothema2020.261
Measurement, 79(6), 1017–1037. https://doi.org/10.1177/0013164419844552
Landis, R. S. (2013). Successfully combining meta-analysis and structural equation
Edwards, J. E., & Bagozzi, R. C. (2000). On the nature and direction of relationships
modeling: Recommendations and strategies. Journal of Business and Psychology, 28,
between constructs and measures. Psychological Methods, 5(2), 155–174. https://doi.
251–261. https://doi-org.ezbusc.usc.gal/10.1007/s10869-013-9285-x.
org/10.1037/1082-989X.5.2.155
*Liberali, J. M., Reyna, V. F., Furlan, S., Stein, L. M., & Pardo, S. T. (2012). Individual
Eisend, M., & Küster, F. (2011). The effectiveness of publicity versus advertising: A meta-
differences in numeracy and cognitive reflection, with implications for biases and
analytic investigation of its moderators. Journal of the Academy of Marketing Science,
fallacies in probability judgment. Journal of Behavioral Decision Making, 25(4),
39(6), 906–921. https://doi.org/10.1007/s11747-010-0224-3
361–381. https://doi.org/10.1002/bdm.752
Evans, J. S. B., & Stanovich, K. E. (2013). Dual-process theories of higher cognition:
*Lindeman, M., & Svedholm-Häkkinen, A. M. (2016). Does poor understanding of
Advancing the debate. Perspectives on Psychological Science, 8(3), 223–241. https://
physical world predict religious and paranormal beliefs? Applied Cognitive
doi.org/10.1177/1745691612460685
Psychology, 30(5), 736–742. https://doi.org/10.1002/acp.3248
Evans, J. S. B., & Wason, P. C. (1976). Rationalization in a reasoning task. British Journal
Lipkus, I. M., Samsa, G., & Rimer, B. K. (2001). General performance on a numeracy scale
of Psychology, 67(4), 479–486. https://doi.org/10.1111/j.2044-8295.1976.tb01536.
among highly educated samples. Medical Decision Making, 21(1), 37–44. https://doi.
x
org/10.1177/0272989X0102100105
Fan, X., & Sivo, S. A. (2005). Sensitivity of fit indexes to misspecified structural or
Lord, F. M., & Novick, M. R. (1969). Statistical theories of mental test scores. Addison-
measurement model components: Rationale of two-index strategy revisited.
Wesley Publishers.
Structural Equation Modeling, 12(3), 343–367. https://doi.org/10.1207/
Lorenzo-Seva, U. (1999). Promin: A method for oblique factor rotation. Multivariate
s15328007sem1203_1
Behavioral Research, 34(3), 347–356. https://doi.org/10.1207/s15327906mbr3403_
*Finucane, M. L., & Gullion, C. M. (2010). Developing a tool for measuring the decision-
3
making competence of older adults. Psychology and Aging, 25(2), 271–288. https://
Lorenzo-Seva, U., & Ferrando, J. P. (2021). FACTOR. Unrestricted factor analysis (release
doi.org/10.1037/a0019106
version 11.02.04 x64bits) [computer software]. Rovira i. Tarragona, Spain: Virgili
Fosgaard, T. R., Hansen, L. G., & Piovesan, M. (2013). Separating Will from Grace: An
University. https://psico.fcep.urv.es/utilitats/factor/.
experiment on conformity and awareness in cheating. Journal of Economic Behavior &
*Mækelæ, M. J., Moritz, S., & Pfuhl, G. (2018). Are psychotic experiences related to
Organization, 93, 279–284. https://doi.org/10.1016/j.jebo.2013.03.027
poorer reflective reasoning? Frontiers in Psychology, 9, 122. https://doi.org/10.3389/
*Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic
fpsyg.2018.00122
Perspectives, 19(4), 25–42. https://doi.org/10.1257/089533005775196732
Marsh, H. W., Wen, Z., & Hau, K. T. (2004). Structural equation models of latent
Gervais, W. M., & Norenzayan, A. (2012). Analytic thinking promotes religious disbelief.
interactions: Evaluation of alternative estimation strategies and indicator
Science, 336(6080), 493–496. https://doi.org/10.1126/science.1215647
construction. Psychological Methods, 9(3), 275–300.
Ghazal, S., Cokely, E. T., & Garcia-Retamero, R. (2014). Predicting biases in very highly
Mata, A., Schubert, A. L., & Ferreira, M. B. (2014). The role of language comprehension
educated samples: Numeracy and metacognition. Judgment and Decision Making, 9
in reasoning: How “good-enough” representations induce biases. Cognition, 133(2),
(1), 15–34.
457–463. https://doi.org/10.1016/j.cognition.2014.07.011
McDonald, R. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum.
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The, P. R. I. S. M. A., Group. (2009).
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA
statement. PLoS Medicine, 6, Article e1000097. https://doi.org/10.1371/journal.
* pmed.1000097
Asterisk indicates studies that were included in the meta-analysis.

11
I. Otero et al.
*Morsanyi, K., McCormack, T., & O’Mahony, E. (2017). The link between deductive
reasoning and mathematics. Thinking & Reasoning, 24(2), 234–257. https://doi.org/
10.1080/13546783.2017.1384760
*Otero, I. (2019). Construct and criterion validity of cognitive reflection. Unpublished
doctoral dissertation. University of Santiago de Compostela https://minerva.usc.es/
xmlui/handle/10347/20521.
Otero, I., Salgado, J. F., & Moscoso, S. (2021). Criterion validity of cognitive reflection
for predicting job performance and training proficiency: A meta-analysis. Frontiers in
Psychology, 12, Article 668592. https://doi.org/10.3389/fpsyg.2021.668592
*Patel, N. (2017). The cognitive reflection test: A measure of intuition/reflection, numeracy,
and insight problem solving, and the implications for understanding real-world judgments
and beliefs. Doctoral dissertation. University of Missouri-Columbia https://hdl.han
dle.net/10355/62365.
*Pennycook, G., Cheyne, J. A., Barr, N., Koehler, D. J., & Fugelsang, J. A. (2014b). The
role of analytic thinking in moral judgements and values. Thinking & Reasoning, 20
(2), 188–214. https://doi.org/10.1080/13546783.2013.865000
*Pennycook, G., Cheyne, J. A., Barr, N., Koehler, D. J., & Fugelsang, J. A. (2015). On the
reception and detection of pseudo-profound bullshit. Judgment and Decision Making,
10, 549–563.
*Pennycook, G., Cheyne, J. A., Seli, P., Koehler, D. J., & Fugelsang, J. A. (2012). Analytic
cognitive style predicts religious and paranormal belief. Cognition, 123(3), 335–346.
https://doi.org/10.1016/j.cognition.2012.03.003
Peters, E., & Levin, I. P. (2008). Dissecting the risky-choice framing effect: Numeracy as
an individual-difference factor in weighting risky and riskless options. Judgment and
Decision Making, 6(3), 435–448.
Peters, E., Västfjäll, D., Slovic, P., Mertz, C. K., Mazzocco, K., & Dickert, S. (2006).
Numeracy and decision making. Psychological Science, 17(5), 407–413. https://doi.
org/10.1111/j.1467-9280.2006.01720.x
Pick, D., & Eisend, M. (2014). Buyers’ perceived switching costs and switching: A meta-
analytic assessment of their antecedents. Journal of the Academy of Marketing Science,
42(2), 186–204. https://doi.org/10.1007/s11747-013-0349-2
Podsakoff, N. P., Whiting, S. W., Podsakoff, P. M., & Blume, B. D. (2009). Individual-and
organizational-level consequences of organizational citizenship behaviors: A meta-
analysis. Journal of Applied Psychology, 94(1), 122–141.
*Poore, J. C., Forlines, C. L., Miller, S. M., Regan, J. R., & Irvine, J. M. (2014).
Personality, cognitive style, motivation, and aptitude predict systematic trends in
analytic forecasting behavior. Journal of Cognitive Engineering and Decision Making, 8
(4), 374–393. https://doi.org/10.1177/1555343414554702
*Primi, C., Donati, M. A., Chiesi, F., & Morsanyi, K. (2018). Are there gender differences
in cognitive reflection? Invariance and differences related to mathematics. Thinking
& Reasoning, 24(2), 258–279. https://doi.org/10.1080/13546783.2017.1387606
*Primi, C., Morsanyi, K., Chiesi, F., Donati, M. A., & Hamilton, J. (2015). The
development and testing of a new version of the cognitive reflection test applying
item response theory (IRT). Journal of Behavioral Decision Making, 29(5), 453–469.
https://doi.org/10.1002/bdm.1883
Ratcliff, R. (1993). Methods for dealing with reaction time outliers. Psychological Bulletin,
114(3), 510–532. https://doi.org/10.1037/0033-2909.114.3.510
Reise, S. P. (2012). The rediscovery of bifactor measurement models. Multivariate
Behavioral Research, 47(5), 667–696. https://doi.org/10.1080/
00273171.2012.715555
Revelle, W. (2016). How to: Use the psych package for factor analysis and data reduction.
Northwestern University. Retrieved from https://randilgarcia.github.io/smith-r-wor
kshop/factor.pdf.
Revelle, W. (2021). An introduction to psychometric theory with applications in R: An
overview [classical test theory and the measurement of reliability]. Northwestern
University. Retrieved from https://personality-project.org/r/book/Chapter7.pdf.
Reyna, V. F., & Brainerd, C. J. (2008). Numeracy, ratio bias, and denominator neglect in
judgments of risk and probability. Learning and Individual Differences, 18(1), 89–107.
https://doi.org/10.1016/j.lindif.2007.03.011
Reyna, V. F., Nelson, W. L., Han, P. K., & Dieckmann, N. F. (2009). How numeracy
influences risk comprehension and medical decision making. Psychological Bulletin,
135(6), 943–973. https://doi.org/10.1037/a0017327
Rodriguez, A., Reise, S. P., & Haviland, M. G. (2016). Evaluating bifactor models:
Calculating and interpreting statistical indices. Psychological Methods, 21(2),
137–150. https://doi.org/10.1037/met0000045
Royzman, E. B., Landy, J. F., & Leeman, R. F. (2015). Are thoughtful people more
utilitarian? CRT as a unique predictor of moral minimalism in the dilemmatic
context. Cognitive Science, 39(2), 325–352. https://doi.org/10.1111/cogs.12136
*Ruffle, B. J., & Tobol, Y. (2017). Clever enough to tell the truth. Experimental Economics,
20(1), 130–155. https://doi.org/10.1007/s10683-016-9479-y
Salgado, J. F. (2014). Cognitive reflection test (CRT-10) technical report. Unpublished
manuscript. Department of Work and Organizational Psychology, University of
Santiago de Compostela.
Salgado, J. F. (2016). A theoretical model of psychometric effects of faking on assessment
procedures: Empirical findings and implications for personality at work. International
Journal of Selection and Assessment, 24(3), 209–228. https://doi.org/10.1111/
ijsa.12142
Salgado, J. F. (2017). Ability test in selection. In H. W. Goldstein, E. A. Pulakos,
J. Passmore, & C. Semedo (Eds.), The psychology of recruitment, selection, and employee
retention (pp. 115–150). Wiley Blackwell.
Salgado, J. F., Otero, I., & Moscoso, S. (2019). Cognitive reflection and general mental
ability as predictors of job performance. Sustainability, 11(22), 6498. https://doi.
org/10.3390/su11226498
Schmid, J., & Leiman, J. N. (1957). The development of hierarchical factor solutions.
Psychometrika, 22(1), 53–61. https://doi.org/10.1007/bf02289209
12
Intelligence 90 (2022) 101614
Schmidt, F. L., & Hunter, J. E. (1977). Development of a general solution to the problem
of validity generalization. Journal of Applied Psychology, 62(5), 529–540. https://doi.
org/10.1037/0021-9010.62.5.529
Schmidt, F. L., & Hunter, J. E. (2015). Methods of meta-analysis: Correcting error and bias in
research findings (3rd ed.). Sage.
Schmidt, F. L., & Le, H. (2014). Software for the Hunter-Schmidt meta-analysis methods
(version 2.0) [computer software]. Department of Management and Organizations,
University of Iowa.
Schwartz, L. M., Woloshin, S., Black, W. C., & Welch, H. G. (1997). The role of numeracy
in understanding the benefit of screening mammography. Annals of Internal Medicine,
127(11), 966–972. https://doi.org/10.7326/0003-4819-127-11-199712010-00003
Shtulman, A., & McCallum, K. (2014). Cognitive reflection predicts science
understanding. In P. Bello, M. Guarini, M. McShane, & B. Scassellati (Eds.),
Proceedings of the 36th annual conference of the Cognitive Science Society (pp.
2937–2942). Cognitive Science Society. https://escholarship.org/uc/item/4t79p8pj.
*Sinayev, A., & Peters, E. (2015). Cognitive reflection vs. calculation in decision making.
Frontiers in Psychology, 6, 532. https://doi.org/10.3389/fpsyg.2015.00532
*Sirota, M., Kostovičová, L., Juanchich, M., Dewberry, C., & Marshall, A. C. (2020).
Measuring cognitive reflection without maths: Developing and validating the verbal
cognitive reflection test. Behavioral decision making. Advance online publication.
https://doi.org/10.1002/bdm.2213
*Skagerlund, K., Lind, T., Strömbäck, C., Tinghög, G., & Västfjäll, D. (2018). Financial
literacy and the role of numeracy–how individuals’ attitude and affinity with
numbers influence financial literacy. Journal of Behavioral and Experimental
Economics, 74, 18–25. https://doi.org/10.1016/j.socec.2018.03.004
Stagnaro, M., Pennycook, G., & Rand, D. G. (2018). Performance on the cognitive
reflection test is stable across time. Judgment and Decision Making, 13, 260–267.
*Ståhl, T., & van Prooijen, J. W. (2018). Epistemic rationality: Skepticism toward
unfounded beliefs requires sufficient cognitive ability and motivation to be rational.
Personality and Individual Differences, 122, 155–163. https://doi.org/10.1016/j.
paid.2017.10.026
Stanovich, K. E. (1999). Who is rational? Studies of individual differences in reasoning.
Psychology Press.
Stanovich, K. E. (2009). What intelligence tests miss: The psychology of rational thought. Yale
University Press.
Stanovich, K. E. (2011). Rationality and the reflective mind. Oxford University Press.
Stanovich, K. E., & West, R. F. (1998). Individual differences in rational thought. Journal
of Experimental Psychology: General, 127(2), 161–188. https://doi.org/10.1037/
0096-3445.127.2.161
Stanovich, K. E., & West, R. F. (2000). Individual differences in reasoning: Implications
for the rationality debate? Behavioral and Brain Sciences, 23(5), 645–665. https://doi.
org/10.1017/S0140525X00003435
*Szasci, B., Szollosi, A., Palfi, B., & Aczel, B. (2017). The cognitive reflection test
revisited: Exploring the ways individuals solve the test. Thinking & Reasoning, 23(3),
207–234. https://doi.org/10.1080/13546783.2017.1292954
*Teigen, K. H., Løhre, E., & Hohle, S. M. (2018). The boundary effect: Perceived post hoc
accuracy of prediction intervals. Judgment and Decision Making, 13(4), 309–321. http
://urn.nb.no/URN:NBN:no-74204.
*Teovanović, P., Knežević, G., & Stankov, L. (2015). Individual differences in cognitive
biases: Evidence against one-factor theory of rationality. Intelligence, 50, 75–86.
https://doi.org/10.1016/j.intell.2015.02.008
*Thomson, K. S., & Oppenheimer, D. M. (2016). Investigating an alternate form of the
Cognitive Reflection Test. Judgment and Decision Making, 11(1), 99–113.
*Toplak, M. E., West, R. F., & Stanovich, K. E. (2011). The cognitive reflection test as a
predictor of performance on heuristics-and-biases tasks. Memory & Cognition, 39(7),
1275–1289. https://doi.org/10.3758/s13421-011-0104-1
*Toplak, M. E., West, R. F., & Stanovich, K. E. (2014). Assessing miserly information
processing: An expansion of the cognitive reflection test. Thinking & Reasoning, 20
(2), 147–168. https://doi.org/10.1080/13546783.2013.844729
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases.
Science, 185(4157), 1124–1131. https://doi.org/10.1126/science.185.4157.1124
*Ventis, L. (2015). Thinking fast and slow in the experience of humor. International
Journal of Humor Researh, 28(3), 351–373. https://doi.org/10.1515/humor-2015-
0070
Verbeke, W., Dietz, B., & Verwaal, E. (2011). Drivers of sales performance: A
contemporary meta-analysis. Have salespeople become knowledge brokers? Journal
of the Academy of Marketing Science, 39(3), 407–428. https://doi.org/10.1007/
s11747-010-0211-8
Viswesvaran, C., & Ones, D. S. (1995). Theory testing: Combining psychometric meta-
analysis and structural equations modelling. Personnel Psychology, 48(4), 865–886.
https://doi.org/10.1111/j.1744-6570.1995.tb01784.x
*Weller, J. A., Dieckmann, N. F., Tusler, M., Mertz, C. K., Burns, W. J., & Peters, E.
(2013). Development and testing of an abbreviated numeracy scale: A Rasch analysis
approach. Journal of Behavioral Decision Making, 26(2), 198–212. https://doi.org/
10.1002/bdm.1751
*Welsh, M., Burns, N., & Delfabbro, P. (2013). The Cognitive Reflection Test: How much
more than numerical ability? In M. Knauff, N. Sebanz, M. Pauen, & I. Wachsmuth
(Eds.), Vol. 35. Proceedings of the annual meeting of the Cognitive Science Society (pp.
1587–1592). Psychology Press. http://hdl.handle.net/2440/83719.
Westrick, P. A., Schmidt, F. L., Le, H., Robbins, S. B., & Radunzele, J. M. R. (2021). The
road to retention passes through first year academic performance: A meta-analytic
path analysis of academic performance and persistence. Educational Assessment, 26
(1), 35–51. https://doi.org/10.1080/10627197.2020.1848423
I. Otero et al.
Yılmaz, O., & Sarıbay, S. A. (2016). An attempt to clarify the link between cognitive style
and political ideology: A non-western replication and extension. Judgment and
Decision Making, 11(3), 287–300.
Yılmaz, O., & Sarıbay, S. A. (2017). The relationship between cognitive style and political
orientation depends on the measures used. Judgment and Decision Making, 12(2),
140–147.
*Zonca, J., Coricelli, G., & Polonio, L. (2020). Gaze data reveal individual differences in
relational representation processes. Journal of Experimental Psychology: Learning,
Memory, and Cognition, 46(2), 257–279. https://doi.org/10.1037/xlm0000
*Adam-Troian, J., Caroti, D., Arciszewski, T., & Ståhl, T. (2019). Unfounded beliefs
among teachers: The interactive role of rationality priming and cognitive ability.
Applied Cognitive Psychology, 33(4), 720–727. https://doi.org/10.1002/acp.3547
*Attridge, N., & Inglis, M. (2015). Increasing cognitive inhibition with a difficult prior
task: Implications for mathematical thinking. ZDM Mathematics Education, 47(5),
723–734. https://doi.org/10.1007/s11858-014-0656-1
*Barr, N., Pennycook, G., Stolz, J. A., & Fugelsang, J. A. (2015). Reasoned connections: A
dual-process perspective on creative thought. Thinking & Reasoning, 21(1), 61–75.
https://doi.org/10.1080/13546783.2014.895915
*Bergman, O., Ellingsen, T., Johannesson, M., & Svensson, C. (2010). Anchoring and
cognitive ability. Economics Letters, 107(1), 66–68. https://doi.org/10.1016/j.
econlet.2009.12.028
*Białek, M., Bergelt, M., Majima, Y., & Koehler, D. J. (2019). Cognitive reflection but not
reinforcement sensitivity is consistently associated with delay discounting of gains
and losses. Journal of Neuroscience, Psychology, and Economics, 12(3–4), 169–183.
https://doi.org/10.1037/npe0000111
*Białek, M., & Sawicki, P. (2018). Cognitive reflection effects on time discounting.
Journal of Individual Differences, 39(2), 99–106. https://doi.org/10.1027/1614-
0001/a000254
*Brañas-Garza, P., García-Muñoz, T., & Hernán, R. (2012). Cognitive effort in the beauty
contest game. Journal of Economic Behavior & Organization, 83(2), 254–260. https://
doi.org/10.1016/j.jebo.2012.05.018
*Byrd, N., & Conway, P. (2019). Not all who ponder count costs: Arithmetic reflection
predicts utilitarian tendencies, but logical reflection predicts both deontological and
utilitarian tendencies. Cognition, 192, 103995. https://doi.org/10.1016/j.
cognition.2019.06.007
*Carpenter, J., Graham, M., & Wolf, J. (2013). Cognitive ability and strategic
sophistication. Games and Economic Behavior, 80, 115–130. https://doi.org/
10.1016/j.geb.2013.02.012
*Čavojová, V., Secară, E. C., Jurkovič, M., & Šrol, J. (2019). Reception and willingness to
share pseudo-profound bullshit and their relation to other epistemically suspect
beliefs and cognitive ability in Slovakia and Romania. Applied Cognitive Psychology,
33(2), 299–311. https://doi.org/10.1002/acp.3486
*Choma, B. L., Sumantry, D., & Hanoch, Y. (2019). Right-wing ideology and NS: A
perception of greater ability, but poorer performance. Judgment & Decision Making,
14(4), 412–422.
*Cokely, E. T., & Kelley, C. M. (2009). Cognitive abilities and superior decision making
under risk: A protocol analysis and process model evaluation. Judgment and Decision
Making, 4(1), 20–33.
*Corgnet, B., Desantis, M., & Porter, D. (2018). What makes a good trader? On the role of
intuition and reflection on trader performance. The Journal of Finance, 73(3),
1113–1137. https://doi.org/10.1111/jofi.12619
*Corgnet, B., Espín, A. M., & Hernán-González, R. (2015). Creativity and cognitive skills
among millennials: Thinking too much and creating too little. Frontiers in Psychology,
7, 1626. https://doi.org/10.3389/fpsyg.2016.01626
*Corgnet, B., Espín, A. M., & Hernán-González, R. (2016). The cognitive basis of social
behavior: Cognitive reflection overrides antisocial but not always prosocial motives.
Frontiers in Behavioral Neuroscience, 9, 287. https://doi.org/10.3389/
fnbeh.2015.00287
*Corgnet, B., Hernán-González, R., & Mateo, R. (2015). Cognitive reflection and the
diligent worker: An experimental study of millennials. PLoS ONE, 10, Article
e0141243. https://doi.org/10.1371/journal.pone.0141243
*Drummond, C., & Fischhoff, B. (2017). Development and validation of the scientific
reasoning scale. Journal of Behavioral Decision Making, 30(1), 26–38. https://doi.org/
10.1002/bdm.1906
*Erceg, N., Galić, Z., & Bubić, A. (2019). “Dysrationalia” among university students: The
role of cognitive abilities, different aspects of rational thought and self-control in
explaining epistemically suspect beliefs. Europe’s Journal of Psychology, 15(1),
159–175. https://doi.org/10.5964/ejop.v15i1.1696
*Erceg, N., Galic, Z., & Bubić, A. (2019). Who detects and why? Individual differences in
abilities, knowledge and thinking dispositions among different types of problem
solvers and their implications for the validity of reasoning tasks. PsyArXiv. https://
doi.org/10.31234/osf.io/w5zau
*Erceg, N., Galic, Z., & Ružojčić, M. (2020). A reflection on cognitive reflection–testing
convergent validity of two versions of the cognitive reflection test. PsyArXiv. https://
doi.org/10.31234/osf.io/ewrtq
*Fehr, D., & Huck, S. (2015). Who knows it is a game? On strategic awareness and
cognitive ability. Experimental Economics, 19(4), 713–726. https://doi.org/10.1007/
s10683-015-9461-0
*Fosgaard, T. R., Hansen, L. G., & Wengström, E. (2019). Cooperation, framing, and
political attitudes. Journal of Economic Behavior & Organization, 158, 416–427.
https://doi.org/10.1016/j.jebo.2018.12.010
13
Intelligence 90 (2022) 101614
*Gino, F., & Ariely, D. (2012). The dark side of creativity: Original thinkers can be more
dishonest. Journal of Personality and Social Psychology, 102(3), 445–459. https://doi.
org/10.1037/a0026406
*Goldwater, M. B., Don, H. J., Krusche, M. J., & Livesey, E. J. (2018). Relational
discovery in category learning. Journal of Experimental Psychology: General, 147(1),
1–33. https://doi.org/10.1037/xge0000387
*Gómez-Veiga, I., Vila, J. O., Duque, G., & García-Madruga, J. A. (2018). A new look to a
classic issue: Reasoning and academic achievement at secondary school. Frontiers in
Psychology, 9, 400. https://doi.org/10.3389/fpsyg.2018.00400
*Hanaki, N., Jacquemet, N., Luchini, S., & Zylbersztejn, A. (2016). Fluid intelligence and
cognitive reflection in a strategic environment: Evidence from dominance-solvable
games. Frontiers in Psychology, 7, 1188. https://doi.org/10.3389/fpsyg.2016.01188
*Insler, M., Compton, J., & Schmitt, P. (2015). The investment decisions of young adults
under relaxed borrowing constraints. Journal of Behavioral and Experimental
Economics, 64, 106–121. https://doi.org/10.1016/j.socec.2015.07.004
*Jackson, S. A., Kleitman, S., Howie, P., & Stankov, L. (2016). Cognitive abilities,
monitoring confidence, and control thresholds explain individual differences in
heuristics and biases. Frontiers in Psychology, 7, 1559. https://doi.org/10.3389/
fpsyg.2016.01559
*Mata, A. (2016). Proportion dominance in valuing lives: The role of deliberative
thinking. Judgment and Decision Making, 11(5), 441–448.
*Moritz, B., Siemsen, E., & Kremer, M. (2014). Judgmental forecasting: Cognitive
reflection and decision speed. Production and Operations Management, 23(7),
1146–1160. https://doi.org/10.1111/poms.12105
*Morsanyi, K., Primi, C., Handley, S. J., Chiesi, F., & Galli, S. (2012). Are systemizing and
autistic traits related to talent and interest in mathematics and engineering? Testing
some of the central claims of the empathizing–systemizing theory. British Journal of
Psychology, 103(4), 472–496. https://doi.org/10.1111/j.2044-8295.2011.02089.x
*Muñoz-Murillo, M., Álvarez-Franco, P. B., & Restrepo-Tobón, D. A. (2020). The role of
cognitive abilities on financial literacy: New experimental evidence. Journal of
Behavioral and Experimental Economics, 84, Article 101482. https://doi.org/10.1016/
j.socec.2019.101482
*Narayanan, A., & Moritz, B. B. (2015). Decision making and cognition in multi-echelon
supply chains: An experimental study. Production and Operations Management, 24(8),
1216–1234. https://doi.org/10.1111/poms.12343
*Obrecht, N. A., Chapman, G. B., & Gelman, R. (2009). An encounter frequency account
of how experience affects likelihood estimation. Memory & Cognition, 37(5),
632–643. https://doi.org/10.3758/MC.37.5.632
*Otero, I. (2020). Unpublished raw data on the correlations between the verbal cognitive
reflection test and the numerical cognitive reflection tests and cognitive variables.
Department of Political Science and Sociology, University of Santiago de
Compostela.
*Pennycook, G., Cheyne, J. A., Barr, N., Koehler, D. J., & Fugelsang, J. A. (2014a).
Cognitive style and religiosity: The role of conflict detection. Memory & Cognition, 42
(1), 1–10. https://doi.org/10.3758/s13421-013-0340-7
*Pennycook, G., Ross, R. M., Koehler, D. J., & Fugelsang, J. A. (2016). Atheists and
agnostics are more reflective than religious believers: Four empirical studies and a
meta-analysis. PLoS ONE, 11(4), Article e0153039. https://doi.org/10.1371/journal.
pone.0153039
*Petrova, D., Traczyk, J., & Garcia-Retamero, R. (2019). What shapes the probability
weighting function? Influence of affect, numeric competencies, and information
formats. Journal of Behavioral Decision Making, 32(2), 124–139. https://doi.org/
10.1002/bdm.2100
*Razmyar, S., & Reeve, C. L. (2013). Individual differences in religiosity as a function of
cognitive ability and cognitive style. Intelligence, 41(5), 667–673. https://doi.org/
10.1016/j.intell.2013.09.003
*Sevenants, A., Dieussaert, K., & Schaeken, W. (2011). Truth table tasks: Irrelevance and
cognitive ability. Thinking & Reasoning, 17(3), 213–246. https://doi.org/10.1080/
13546783.2011.569153
*Shenhav, A., Rand, D. G., & Greene, J. (2017). The relationship between intertemporal
choice and following the path of least resistance across choices, preferences, and
beliefs. Judgment and Decision Making, 12(1), 1–18. https://doi.org/10.2139/
ssrn.2724547
*Shenhav, A., Rand, D. G., & Greene, J. D. (2012). Divine intuition: Cognitive style
influences belief in god. Journal of Experimental Psychology: General, 141(3),
423–428. https://doi.org/10.1037/a0025391
*Sirota, M., & Juanchich, M. (2011). Role of NS and cognitive reflection in bayesian
reasoning with natural frequencies. Studia Psychologica, 53(2), 151–161.
**Stupple, E. J. N., Gale, M., & Richmond, C. (2013). Working memory, cognitive
miserliness and logic as predictors of performance on the cognitive reflection test. In
M. Knauff, M. Pauen, N. Sebanz, & I. Wachsmuth (Eds.), Vol. 35. Proceedings of the
35th annual conference of the Cognitive Science Society (pp. 1396–1401). Cognitive
Science Society. No. 35 https://escholarship.org/uc/item/36989187.
*Thompson, V. A., Turner, J. A. P., Pennycook, G., Ball, L. J., Brack, H., Ophir, Y., &
Ackerman, R. (2013). The role of answer fluency and perceptual fluency as
metacognitive cues for initiating analytic thinking. Cognition, 128(2), 237–251.
https://doi.org/10.1016/j.cognition.2012.09.012
*Trippas, D., Pennycook, G., Verde, M. F., & Handley, S. J. (2015). Better but still biased:
Analytic cognitive style and belief bias. Thinking & Reasoning, 21(4), 431–445.
https://doi.org/10.1080/13546783.2015.1016450
*West, R. F., Meserve, R. J., & Stanovich, K. E. (2012). Cognitive sophistication does not
attenuate the bias blind spot. Journal of Personality and Social Psychology, 103(3),
506–519. https://doi.org/10.1037/a0028857