Assessing Computational Thinking: A Systematic Review of Empirical Studies

See discussions, stats, and author profiles for this publication at: https://www.researchgate.
net/publication/338450598
Assessing computational thinking: A systematic review of empirical studies
Article in Computers & Education · April 2020

DOI: 10.1016/j.compedu.2019.103798
CITATIONS READS
24 2,492
5 authors, including:
Xiaodan Tang Yue Yin

University of Illinois at Chicago University of Illinois at Chicago
6 PUBLICATIONS 33 CITATIONS 32 PUBLICATIONS 1,087 CITATIONS
SEE PROFILE SEE PROFILE
Qiao Lin Roxana Hadad

University of Illinois at Chicago University of California, Los Angeles
7 PUBLICATIONS 34 CITATIONS 7 PUBLICATIONS 39 CITATIONS
SEE PROFILE SEE PROFILE
Some of the authors of this publication are also working on these related projects:
Formative Assessment in a Networked Classroom (FANC) View project
DCA110195 View project
All content following this page was uploaded by Xiaoming Zhai on 24 July 2020.
The user has requested enhancement of the downloaded file.

Computers & Education 148 (2020) 103798
Contents lists available at ScienceDirect
Computers & Education

journal homepage: http://www.elsevier.com/locate/compedu
Assessing computational thinking: A systematic review of

empirical studies
Xiaodan Tang a, *, Yue Yin a, Qiao Lin a, Roxana Hadad a, b, c, Xiaoming Zhai d
a
University of Illinois at Chicago, 1200 West Harrison St., Chicago, IL, 60607, USA
b
University of California, Los Angeles, 405 Hilgard Avenue, Los Angeles, CA, 90095, USA
c
Northeastern Illinois University, 5500 North St. Louis Avenue, Chicago, IL, 60625, USA
d
Michigan State University, 426 Auditorium Road, East Lansing, MI, 48824, USA
A R T I C L E I N F O A B S T R A C T
Keywords: With the increasing attention to Computational Thinking (CT) in education, there has been a
Evaluation methodologies concomitant rise of needs and interest in investigating how to assess CT skills. This study sys
Teaching/learning strategies tematically reviewed how CT has been assessed in the literature. We reviewed 96 journal articles
Information literacy
to analyze specific CT assessments from four perspectives: educational context, assessment
Applications in subject areas
construct, assessment type, and reliability and validity evidence. Our review results indicate that
21st century abilities
Computational thinking (a) more CT assessments are needed for high school, college students, and teacher professional
development programs, (b) most CT assessments focus on students’ programming or computing
skills, (c) traditional tests and performance assessments are often used to assess CT skills, and
surveys are used to measure students’ CT dispositions, and (d) more reliability and validity ev
idence needs to be collected and reported in future studies. This review identifies current research
gaps and future directions to conceptualize and assess CT skills, and the findings are expected to
be beneficial for researchers, curriculum designers, and instructors.
1. Introduction
Computational Thinking (CT) has drawn increasing attention in the field of science, technology, engineering, and mathematics
(STEM) education since Wing (2006) promoted it. Cuny, Snyder, and Wing (2010) defines CT as “the thought processes involved in
formulating problems and their solutions so that the solutions are represented in a form that can be effectively carried out by an
information-processing agent (Wing, 2011, p. 1, p. 1)”. Not only is CT grounded on concepts fundamental to computer science (CS), a
field that has dramatically impacted society (Wing, 2006; 2008), but it is integral to modern research and problem-solving work of
STEM (Henderson, Cortina, & Wing, 2007). Thus, CT should be embedded in the educational system as a substantial learning goal to
prepare students with competency in their future life (Grover & Pea, 2013). The International Society for Technology in Education and the
Computer Science Teachers Association (CSTA & ISTE, 2011) developed resources on approaches to bringing CT into K-12 settings.
Meanwhile, the National Research Council (NRC) organized two workshops with scholars in CS and education around the scope and
nature of CT and the pedagogical aspects of CT (National Research Council, 2010; 2011; 2012). These efforts together with others in
* Corresponding author. Department of Educational Psychology, University of Illinois at Chicago, 1040 W. Harrison St. (MC 147), Chicago, IL,
60607, USA.
E-mail addresses: xtang1322@gmail.com (X. Tang), yueyin@uic.edu (Y. Yin), qlin7@uic.edu (Q. Lin), rhadad@ucla.edu (R. Hadad), zhaixia1@
msu.edu (X. Zhai).
https://doi.org/10.1016/j.compedu.2019.103798
Received 28 May 2019; Received in revised form 29 December 2019; Accepted 31 December 2019
Available online 8 January 2020
0360-1315/© 2020 Elsevier Ltd. All rights reserved.
X. Tang et al. Computers & Education 148 (2020) 103798
the education community promoted a milestone in STEM education in 2013—the Next Generation Science Standards (NGSS Lead
States, 2013) listed “using mathematics and computational thinking” as one of the Science and Engineering Practices that integrate
both disciplinary core ideas and crosscutting concepts.
Drawing upon the growing interest in integrating CT in STEM education, the field dedicated many efforts to promote and examine
students’ CT skills. These efforts include, but are not limited to the following ones: (a) developing CT-integrated curriculum (e.g., Rich,
Spaepen, Strickland, & Moran, 2019; Sung, 2019); (b) inventing CT-inspired teaching and learning tools (e.g., Bers, 2010; Grover,
2017a, 2017b; Weintrop et al., 2014); (c) building CT-embedded learning environment (e.g., Mun ~ oz-Repiso & Caballero-Gonz� alez,
2019), and (d) developing assessments focusing on students’ CT skills (e.g., Gonza �lez, 2015; Korkmaz, Çakir, & Ozden,
€ 2017a, 2017b).
These studies generated a body of literature that help us understand the nature of CT, the CT integration in STEM classrooms, and the
features of students’ performance in CT practices.
Some researchers have synthesized the work related to CT. Lockwood and Mooney (2018) summarized CT research in secondary
education and provided information on the subjects used to teach CT, the tools used to teach and assess CT, and benefits and barriers of
incorporating CT in secondary education. Hsu, Chang, and Hung (2018) explored teaching and learning activities and strategies when
promoting CT. However, systematic reflection is lacking on the evaluation tools of student CT skills and performance used in these
studies, making future research directions on promotion and assessment of CT practices unclear. In fact, both Lockwood and Mooney
(2018) and Hsu et al. (2018) rendered it pressing to address the education community’s uncertainty in how to best assess CT skills in
their reviews.
Our study aims to systematically review CT assessment research in more detail than previous reviews regarding CT implementation
contexts, CT and CT-related constructs, CT assessment tools, and their reliability and validity evidence across all educational levels. In
particular, our review focuses specifically on the CT studies that apply assessments for the educational levels from kindergarten to
college education and professional development for teachers. In addition, we reviewed these studies in terms of what educational
contexts and subject domains these studies are conducted at, what types of the CT constructs are measured (we classified the constructs
Fig. 1. Different definitions of computational thinking.
2
in terms of CT and CT-related learning outcomes), what assessment tools have been employed to measure these types of CT constructs,
and finally what the reliability and validity evidence are being reported. To sum, Lockwood and Mooney (2018) provided a big picture
of current CT studies in secondary education, while our review contributes more detailed information regarding the development and
implementation of CT assessments.
1.1. Definition and significance of CT
In 1980, Seymour Papert first used the term “computational thinking” and suggested that computers might enhance thinking and
change patterns of knowledge accessibility (Papert, 1980). He further emphasized that all children should have access to computers as
a way to shape their learning and express their ideas (Papert, 1996). Later, in her influential article, Wing (2006) echoed these ideas by
illuminating the concept of CT and broadcasting its applications in problem solving. She stated that CT should be at the core of K-12
curricula and called for research on effective ways of teaching CT to students. Since then, CT has drawn increasing attention from
educators and educational researchers and has been identified as a critical competence that would equip students with foundational
skills to learn STEM (Weintrop et al., 2014).
Despite of a research history of around 15 years, the field has not reached a consensus on the definition of CT (National Research
Council, 2011). As such, we constructed a diagram to illustrate some of the well-cited definitions (See in Fig. 1). As shown in the figure,
many researchers defined CT in a way of drawing from programming and computing concepts. For example, Brennan and Resnick
(2012) developed a theoretical framework of CT that involves three key dimensions, in which one of them refers to computational
concepts, including programming terms of sequences, loops, parallelism, events, conditionals, operators, and data. The other two
dimensions are computational practices, including the processes of iteration, debugging, and abstraction, and computational per
spectives, including expressing, connecting, and questioning. Another defining framework that originates from computing concepts
were proposed by Weintrop et al. (2016). They classified CT into four major categories with 22 sub-skills: data practices, modeling &
simulation practices, computational problem-solving practices, and systems thinking practices. Based on this framework, they
developed a series of CT enhanced lesson plans for high school STEM classrooms. Denner, Werner, and Ortiz (2012) defined CT as a
united competence, which is composed of three key dimensions of CT: programming, documenting and understanding software, and
designing for usability.
Different from the skills in working with computing or programming activities, some researchers regarded CT as a set of compe
tences requiring students to develop both domain-specific knowledge and problem-solving skills. For example, CSTA & ISTE, 2011
provided a list of vocabularies for CT: algorithms & procedures, automation, simulation, parallelization, algorithms and procedures,
automation, simulation, parallelization. They suggested that those skills can be used to solve problems in everyday life, different
subject domains, and across different grade levels. Selby and Woollard (2013) proposed operational definitions of CT skills including
abstraction, decomposition, algorithmic thinking, evaluation, and generalization. As this operational definition is based on a
meta-analysis of various studies of CT, it is broadly adopted in many studies (e.g., Atmatzidou & Demetriadis, 2016; Leonard et al.,
2018). During an experiment that assessed the impact of CT modules on preservice teachers, Yadav, Mayfield, Zhou, Hambrusch, and
Korb (2014) explained five CT concepts: problem identification and decomposition, abstraction, logical thinking, algorithms, and
debugging with concrete examples from day-to-day life and related these concepts to preservice teachers’ personal experiences.
Despite the controversies surrounding the definition of CT, researchers have agreed that involving CT has an immense potential to
transform how we approach subject domains in classrooms (e.g., Barr, Harrison, & Conery, 2011; Barr & Stephenson, 2011; Jona et al.,
2014; Lee et al., 2011; Repenning, Webb, & Ioannidou, 2010; Wing, 2006). Weintrop et al. (2016) concluded with three main benefits
of embedding CT into STEM classrooms: building a reciprocal connection between math, science, and CT; constructing a more
accessible classroom context for teachers and students; and making math and science classrooms updated with current professional
practices.
In addition to the theoretical discussion, many studies have attempted to integrate CT in classrooms. For example, Malyn-Smith and
Lee (2012) have facilitated the exploration of CT as a foundational skill for STEM professionals and how professionals engaged CT in
routine work and problem solving. Later, Lee, Martin, and Apone (2014) integrated CT into K-8 classrooms through three types of
fine-grained computational activities: digital storytelling, data collection and analysis, and computational science investigations.
Further, researchers have developed CT interventions on various subject domains such as biology and physics (Sengupta, Kinnebrew,
Basu, Biswas, & Clark, 2013), journalism and expository writing (Wolz, Stone, Pearson, Pulimood, & Switzer, 2011; Wolz, Stone,
Pulimood, & Pearson, 2010), science in general (Basu, Biswas, & Kinnebrew, 2017a, 2017b; Basu et al., 2016; Weintrop et al. 2016),
sciences and arts (Sa�ez-Lo�pez, Roma �n-Gonza�lez, & V�azquez-Cano, 2016a, 2016b), and mathematics (Wilkerson-Jerde, 2014).
Taken together, researchers have illustrated the importance of teaching and learning CT and its integration in other subject do
mains. However, as indicated by Kaleliog �lu, Gülbahar, and Kukul (2016) in their review on CT studies, the current CT concept and
definition lacks scientific justifications. Not surprisingly, researchers would hold different perspectives when applying, interpreting,
and assessing the proposed CT concept and definition.
1.2. CT assessment
Assessment plays a critical role when educators introduce CT into K-12 classrooms (Grover & Pea, 2013). Kaleliog �lu et al. (2016)
also advocated to have more discussions on how to assess students’ mastery and application of CT skills in real-life situations. In this
study, we categorized CT assessments according to McMillan’s (2013) paradigms of classroom assessment. Some CT studies employed
selected-response and/or constructed-response tests, e.g., Shell and Soh (2013a, 2013b) developed a paper-pencil test to assess college
3
students’ CS knowledge and CT skills; also, Chen et al. (2017a, 2017b) designed an instrument with 15 multiple-choice questions and
eight open-ended questions to assess students’ application of CT skills to solve daily life problems. Performance or portfolio assessment
is another major assessment tool. Researchers created programming or CT activities for students to complete and then employed a
grading rubric to evaluate their work products, e.g., the Fairy Assessment of Alice programming, an environment which allows users to
build 3D virtual worlds (Werner, Denner, Campe, & Kawamoto, 2012); the analysis of digital portfolios designed by students to
complete e-textiles projects using CT (Fields, Shaw, & Kafai, 2018; Lui et al., 2019), and the evaluation of students’ Scratch projects
based on a visual programming environment (Garneli & Chorianopoulos, 2018). Questionnaires and interviews have also been used.
For instance, Sa�ez-Lo
�pez et al. (2016a, 2016b) used a questionnaire to examine primary school students’ perceptions of computational
concepts after learning visual programming language on the Scratch platform.
Researchers have examined the quality of CT assessment. For example, to examine the reliability and validity of a self-efficacy
perception scale for CT skills, Gülbahar, Kert, and Kaleliog �lu (2018) conducted exploratory and confirmatory factor analysis. Wein
trop et al. (2016) conducted interviews to probe students’ strategy on designing video games using block-based programming
language.
A systematic review of specific CT assessments may yield insights into ways of improving and designing effective assessment tools
so that two goals can be achieved: (a) the scholarly community and school practitioners can remain updated in the availability of CT
assessments and their characteristics. (b) Researchers can continually add to this scholarship by investigating unexamined but
important topics surrounding CT assessment, the most relevant constructs in CT assessment, appropriate assessment formats to
measure CT, and the necessary reliability and validity evidence.
In this study, we conducted a systematic review with the purposes to reflect on prior studies and to identify the gaps by specifically
focusing on one aspect of the literature—the assessment of CT. We analyzed the current state and features of studies exploring CT
assessments and suggested future directions regarding how to assess CT for various purposes. The following research questions (RQ)
formed the basis of this review:
RQ1: What are the educational contexts to which CT assessments have been applied?
RQ2: What CT constructs are measured by the CT assessments?
RQ3: What assessment tools are used to assess CT constructs?
RQ4: What is the reliability and validity evidence of these CT assessments?
2. Method
2.1. Literature search
We searched three widely used and comprehensive digital databases to ensure the search covered all the relevant literature and
journals: Education Resources Information Center (ERIC) (http://www.eric.ed.gov/), PsycINFO (http://www.apa.org/psycinfo/), and
Google Scholar (http://www. scholar.google.com/). We first collected all journal articles with the extract key phrase “computational
thinking” alone in all fields of each article and by doing so, we were able to include articles that contributed to assessment but did not
mention assessment as a key term in their description. Although CT is related to computing, programming, and coding, we did not
search for these words as alternative keywords because we considered CT as a distinct concept from computing/programming/coding
and we only focused on studies in which researchers acknowledged CT and used the term “computational thinking.”
With regard to the reference type, we decided to exclude conference papers in this review for the following reasons. (a) The journal
articles have reported most of the significant scientific results according to the Bradford’s Law (Testa, 2009). In this case, conference
papers would document similar results with those in journal articles. (b) Most conference papers on CT studies lack concrete infor
mation regarding their assessment, which may impede readers from making fair comparison between assessments. This argument is
also bolstered by the literature review of Zhang and Nouri (2019), in which they stated that the lack of vital information in some CT
studies can hinder the replication of studies. (c) The major drawback of not including conference papers though is that we may have
missed the most recent development. To make up for this problem, using the same procedure as before, we conducted an updated
literature search after the initial literature search.
In particular, we used the following inclusion criteria to select articles that are (a) using “computational thinking” in any part of the
paper (such as title, abstract, keywords, or main text); (b) published before August 2019; (c) peer-reviewed journal articles; (d)
available in full-text; (e) empirical studies containing assessment outcomes in terms of CT skills or CT-related skills; and (f) written in
English. After the initial search, we utilized a snowball method using the references of the selected articles, so that we could track
articles which we may have missed in the earlier stage. This initial literature search resulted in 361 journal articles. The first author
then downloaded all the initially selected articles.
After the collection of the articles, the first and third author reviewed titles, abstracts, and methods sections together in order to rule
out articles based on a set of exclusion criteria: (a) The “computational thinking” is not examined by an empirical research. (b) The
article is a theoretical work or a content analysis of current CT materials. (c) No information on assessment is provided. We developed
the list of inclusion and exclusion criteria by modifying established criteria used in earlier reviews (e.g., Haseski, Ilic, & Tugtekin,
2018; Lockwood & Mooney, 2018; Shute, Sun, & Asbell-Clarke, 2017; Zhang & Nouri, 2019). Disagreements between the two authors
were resolved through discussion and further review of the disputed studies. Based on the reduction of articles following the inclusion
and exclusion criteria by a group review, the number of articles was narrowed down to 96 in the final review. Among them, the initial
search produced 77 relevant articles published before December 31, 2018. A follow-up search for articles published before August
4
2019 resulted in another 19 journal articles.
2.2. Analysis
We coded the literature by systematically classifying the texts into categories in three stages according to the procedures of a
content analysis (Fraenkel, Wallen, & Hyun, 2015). First, we developed a coding scheme to systematically exact information from each
selected article in accordance with the four research questions. An Excel spreadsheet was used to store and analyze all data. Second,
using the initial coding scheme, the first three authors reviewed and coded the same 11 articles that were randomly chosen. During the
review, we improved the initial coding scheme by modifying and clarifying categories. Third, after the coding scheme became stable,
the first and third author coded eight articles independently and reached an inter-rater agreement of 91.8% across all the categories.
The discrepancies were resolved by discussing with the second author. Finally, given that an acceptable inter-rater agreement has been
achieved, the first author independently coded the rest of the articles. The basic coding results were shown in Appendix A and the
review process was demonstrated in Fig. 2. Based on the coding results, frequencies and proportions for each category were computed
and reported in the tables and figure. Then, a detailed explanation of the patterns emerged from the reviewed studies to support the
four research questions was given in the Results section, coupled with examples for each category.
3. Results
3.1. Educational contexts of the CT assessments
Educational levels: Researchers conducted studies and implemented CT interventions across various educational levels. As shown in
Table 1, middle and elementary schools were the most researched educational levels that covered almost one third of the reviewed
studies respectively, followed by high schools (15%) and colleges (15%). The rest integrated CT into teacher education programs or
other professional practices (13%). Although some preservice teacher education programs were in college, we classified them into the
category of teacher education to reveal the current state of professional development on CT applications. In general, considerable
research on CT assessment has been devoted to K-8 schools, much more than high schools, colleges, teacher education, or other
professional practices.
Subject matter domains: We examined the subject domains where the reviewed CT assessments applied. As shown in Table 1,
programming and CS were the subject matter most often researched (43%), which was consistent with the findings of Lockwood and
Mooney (2018), followed by topics on robotics and game design (25%). A few studies (21%) implemented CT into non-CS STEM
curriculum or activities. Non-STEM subjects, including poetry, journalism, etc., were the least researched (9%). Generally, most re
searchers restricted the development of CT skills within the CS related subjects, while other researchers extended CT skills to other
non-CS subjects. Although CT by its nature originated from CS in an effort to encourage people to think like a computer scientist,
several researchers argued that the importance of CT lay in its ability to deepen understanding of other subjects (e.g., Weintrop et al.,
2016; Wing, 2008). However, the results revealed that the implementation of CT in the non-CS classrooms still lagged behind.
Fig. 2. The literature review process of this study.
5
Table 1
Educational contexts and assessment tools of the reviewed CT assessments.
Variables Categories Numbers Percent
Educational level K-elementary 31 28.18%

Middle 30 27.27%
High 17 15.45%
College 17 15.45%
Teacher 15 13.64%
Subject matter Programming and CS 40 43.48%
Robotics and game design 23 25.00%
Non-CS STEM 20 21.74%
Non-STEM 9 9.78%
Educational setting Formal 60 67.42%
Informal 27 30.34%
Assessment tool Traditional 34 24.11%
Portfolio 47 33.33%
Interview 22 15.60%
Survey 38 26.95%
Note: Some studies used more than one category so the total number (i.e., the denominator) to calculate the percentage is the total number of
each feature used in these studies.
As the social movement of CT has called for the involvement of CT into other disciplines in K-12 classrooms (diSessa, 2018), we
analyzed the relationship between educational levels and subject domains manifested in the reviewed studies. We found that the
distribution of the four major subject domains (i.e., programming, robotics/game, STEM, non-STEM) for the elementary school level
was similar as those for the middle school level (Fig. 3). However, consistent with the previous finding of less research for high school,
college, and teacher education, there was no intervention or curriculum pertinent to non-STEM subjects for high school and college
students and surprisingly, no research on integrating CT with non-CS STEM subjects for teacher professional development in the
reviewed articles. Although CT is regarded as at the core of all STEM subject domains (Henderson et al., 2007) and the idea to infuse CT
into K-12 classrooms has been proposed for a decade, few studies have focused on the professional training for K-12 teachers on how to
integrate CT with STEM subjects in practice.
Educational settings: As shown in Table 1, about one-third of the studies were conducted in an informal educational context, which
included summer camps, after-school programs, or other out-of-school activities. The remaining studies were conducted in the formal
context (i.e. classrooms), covering more than half of studies. The high proportion of studies conducted in the formal setting appeared to
align with the move of involving CT into subject domain standards.
3.2. Constructs of the CT assessments
As discussed in the introduction, CT has been defined across the literature with both divergence and agreement. Hence, we were not
surprised to find great diverseness in the ways that CT constructs were defined and operationalized in assessment. We constructed a
hierarchical structure to classify the common patterns that emerged among all the constructs being measured based on two classifi
cations: (a) Sullivan and Heffernan (2016)’s categories of first-order and second-order learning outcomes regarding the relationship
between CT and other subject domains and (b) McMillan (2013)’s categories of cognitive constructs and non-cognitive dispositions and
skills.
The first level of this structure denotes a classification of the first-order use of constructs and the second-order use of constructs
(Sullivan & Heffernan, 2016). The first-order usage of knowledge refers to a direct cognitive manifest of the studied domain knowl
edge, which refers to CT skills defined and assessed independently of other subject domains in this review. The second-order
Fig. 3. Relationship between educational levels and subject domains of the reviewed CT assessments (color). (For interpretation of the references to
color in this figure legend, the reader is referred to the Web version of this article.)
6
application denotes a representation of relevant domain knowledge as a derivative of CT integration, such as programming skills,
STEM, and non-STEM knowledge. Based on the first level classification of the first-order and second-order usage, we then divided each
into the second level classification of cognitive and noncognitive constructs based on McMillan’s (2013) identification of constructs.
We further classified the categories of second-order cognitive and non-cognitive constructs according to the relevant subject domains
as the third level of this structure. Finally, we ended up with seven sub-categories and computed their proportions and frequencies
researched by the reviewed articles (shown in Table 2) and we also listed some sample questions in Appendix B.
First-order cognitive CT: More than one-third of the reviewed studies directly assessed CT concepts and skills. They were derived
from CS or programming concepts but were further derived as cognitive thinking skills and can be applicable to all subject domain
problem-solving processes. Hence, teachers without knowledge in CS and students without any prior experience in programming are
able to understand and apply these types of higher order thinking skills in teaching and learning. The first-order use of CT has been
supported by CSTA & ISTE, 2011. They listed examples of applying each component across various subjects including math, science,
history, etc. As an example of implementing the CSTA and ISTE’s (2011) CT framework and Selby and Woollard’s (2013) CT defi
nitions, Bebras International Contest was initiated in Lithuania in 2004 which aimed to promote CT among K-12 students (Cartelli,
Dagiene, & Futschek, 2012, pp. 35–46; Dagiene_ & Futschek, 2008). Different from programming-based CT tests, the Bebras tasks aimed
to elicit students’ CT (Palts, Pedaste, Vene, & Vinikiene, _ 2017) to solve problems for potential real-life scenarios without tapping
specific subject contents and using programming platforms or devices. From another perspective, ISTE (2015) summarized CT as a
common reflection of creativity, algorithmic thinking, critical thinking, problem solving, cooperative thinking, and communication
skills. Durak and Saritepeci (2018) designed a CT questionnaire with these five factors as subscales and found CT thinking skills were
positively correlated with general thinking skills.
First-order non-cognitive: About seven percent of studies assessed participants’ dispositions and attitudes toward CT. For example,
Mesiti, Parkes, Paneto, and Cahill (2019) designed a survey to elicit middle and high school students’ self-efficacy and interests related
to problem decomposition after completing CT tasks mediated by animated films. Considering professional development as an
important role in teachers’ recognition and application of CT in their classrooms, Leonard et al. (2017) employed an existing teachers’
CT belief survey to probe their comfort, interest, and classroom practice of CT skills. However, few studies have focused on the dis
positions that would strengthen CT skills and its components, such as the dispositions of persistence and tolerance for ambiguity as
suggested by CSTA & ISTE, 2011.
Second-order cognitive programming: Programming and computing related concepts were the constructs measured with one-fourth of
the reviewed studies. Many researchers defined CT in terms of programming or computing concepts, so that they designed in
terventions based on programming activities and assessed students’ programming skills as evidence of students’ CT skills. For example,
Scratch, a block-based programming environment, was developed at MIT for users to design interactive media. Later, an assessment
tool of Scratch, Dr. Scratch, was developed and utilized by many researchers to evaluate users’ Scratch projects and assess their
programming and CT skills. Echoing the core ideas of Scratch, Brennan and Resnick (2012) proposed the 3D CT framework in which
the cognitive aspect of CT was computational concepts. The application of this framework and the Scratch platform orchestrated a
collection of studies that defined CT concepts congruently with programming concepts (e.g., Falloon, 2016; Garneli & Chorianopoulos,
2018; Grover, Pea, & Cooper, 2015; Jenkins, 2015; Lye & Koh, 2014; Pugnali, Sullivan, & Bers, 2017; Rom� an-Gonz� alez,
P�erez-Gonza �lez, Moreno-Leo �n, & Robles, 2017a, 2017b; Zhong, Wang, Chen, & Li, 2016a, 2016b). For example, Pugnali et al. (2017)
defined CT using programming concepts, including sequencing, loops, conditionals, and debugging, and analyzed students’ projects
using a graphical coding application on the iPad (Scratch Jr.) and tangible programmable robotics kit (KIBO). It seems natural to define
CT by referring programming concepts because they are sometimes related to each other and sharing similarities. However, highly
relying on programming terms to define the constructs of CT assessment reveals a lack of connection or applicability to non-CS
subjects.
Second-order cognitive STEM: Nine percent of the studies embedded CT into non-CS STEM interventions and measured non-CS STEM
knowledge as the constructs. For example, by using a CT-based learning environment for K-12 science, Computational Thinking in
Simulation and Modeling (CTSiM), Basu et al. (2016) argued that a lack of domain knowledge was one of the challenges students faced
Table 2
A classification structure of CT constructs measured in the reviewed studies.
Sullivan & Hefferman McMillan Subject domains Examples (sample questions in Appendix B.) Frequency Percent
(2018) (2013)
First-order Cognitive CT skills Bebras items (Dolgopolovas, Jevsikova, Dagiene, & 61 41.78%
Savulioniene, 2016)
Non-Cognitive CT propositions CT perceptions (Leonard et al., 2018) 11 7.53%
Second-order Cognitive Programming/ Scratch projects (Grover et al., 2015) 33 22.60%
computing/CS
STEM (exclude CS) Lattice Land (Pei et al., 2018) 14 9.59%
Non-STEM Poetic thinking and CT skills (Jenkins, 2015) 1 0.68%
Non-Cognitive Programming/ Perceptions about programming (Adler and Kim, 2018) 21 14.38%
computing/CS
STEM/non-STEM Comfort, interest, and classroom practice of CT skills ( 5 3.42%
S�
aez-L�
opez et al., 2016)
Note: Some studies assessed more than one construct so the total number (i.e., the denominator) to calculate the percentage is the total number of each
construct type used in these studies.
7
when developing their programming and simulation models. With a focus on physics knowledge, Jaipal-Jamani and Angeli (2017a,
2017b) assessed pre-service teachers’ understanding of gears and relevant physics knowledge after participating in activities with
LEGO WeDo robotics kits. To build a bridge between CT and math proficiency, Pei, Weintrop, and Wilensky (2018) examined how
students’ mathematical habits of mind aligned with CT in terms of four overarching categories: data, modeling and simulation,
computational problem-solving, and systems thinking. These studies showcased the CT infusion into STEM education and assessment.
Second-order cognitive non-STEM: Only one CT assessment study integrated CT into non-STEM subjects. Jenkins (2015) examined
the potential of using a block-based programming poem generator program to evaluate students’ development of CT skills and poetic
thinking in English. The test of poetic thinking and CT skills measured the statutory literacy skills of ‘adapting structures in writing’,
‘using a wide range of sentence structures’ and ‘using knowledge of word roots and families’ with the programming concepts of se
quences, loops and events that were defined in the Brennan and Resnick’s (2012) CT framework.
Second-order non-cognitive: Around fifteen percent of the studies examined participants’ dispositions and attitudes toward CT-
related concepts and activities. For instance, Adler and Kim (2018) created a web-based simulation to help teach Newton’s Second
Law of Motion in the science methods course for preservice teachers. After working with the simulation, preservice teachers were asked
to complete a post-questionnaire relating to their experience with this simulation and Scratch programming. As an example of probing
elementary students’ self-efficacy of non-STEM subjects after receiving CT-integrated arts curriculum, Sa �ez-Lo
�pez et al. (2016a,
2016b) used a survey to examine students’ perceptions of art history.
In summary, the above review suggests that many studies chose to measure programming or computing concepts as a represen
tation of CT skills or defined cognitive CT skills using programming and CS concepts. We consider CT and CS as related, but not the
same. Although it is understandable that CT skills can be defined and operationalized differently, most of the reviewed articles have not
explicitly explained why assessing CT skills can be interchangeable with assessing programming or computational concepts.
3.3. Tools for assessing CT
Despite the variability in assessment design for CT, four categories of assessment types emerged from the literature, including the
traditional test composed of selected- or constructed-response questions, portfolio assessment, interviews, and surveys. Some studies
employed more than one assessment type in order to collect multidimensional evidence of students’ CT skills. The frequency of each
type was revealed in Table 1.
Selected- or constructed-response tests: Some studies (24%) chose to develop an assessment composed of multiple choice and/or open-
ended questions, usually evaluated by correctness and completeness and designed for summative purposes. For instance, Jenson and
Droumeva (2016) designed a test to evaluate students’ existing knowledge of CS concepts such as variables, operations, and functions
as evidence of student CT proficiency. College students’ learning and retention of CT skills was assessed in both Peteranetz, Flanigan,
Shell, and Soh (2017) and Flanigan, Peteranetz, Shell, and Soh (2017a, 2017b) studies via a web-based, 13-item test developed by CS
and engineering faculty. The test addressed common core CS contents including selection, looping, arrays, functions, algorithms,
search, and sorting.
The usage of this traditional assessment type suggests a trend: although CT is considered as a cognitive thinking process, many
researchers regard it as a learning product (e.g., knowledge or skills gained) when assessing it. It further implies that CT skills can be
regarded as quantifiable mastery of knowledge regarding CT components. Practically speaking, teachers and researchers may find a
reliable and valid knowledge assessment, which can be conveniently adapted into their classrooms or interventions and use it to
measure students’ CT knowledge for a summative assessment purpose. From another perspective of promoting an ongoing learning
experience, Fields, Lui, and Kafai (2019) argued that the traditional assessment failed to capture the process of CT learning when
students worked with their hands-on projects.
Portfolio assessment.: Portfolio assessment, a type of performance assessment, refers to a purposeful, systematic process of collecting
and evaluating various types of student products to examine the attainment of learning targets (McMillan, 2013). More than one-third
of the reviewed studies used portfolio assessment to evaluate students’ CT skills through their projects, notes, or other direct obser
vations. Like a performance assessment, researchers used a grading rubric to indicate different levels of achievement for each
dimension of CT performance or a checklist to indicate whether a certain criterion is met (i.e., binary coding). For example, Bers,
Flannery, Kazakoff, and Sullivan (2014) evaluated students’ knowledge of debugging, correspondence, sequencing, and control flow
by assigning the appropriate level of achievement based on a scoring rubric to each student’s robotics project.
Another widely used portfolio assessment comes from the analysis of Scratch projects. Moreno-Leo �n, Robles, and Roma �n-Gonza �lez
(2017a, 2017b) utilized the rubric provided by the abovementioned Dr. Scratch to analyze students’ Scratch projects on seven con
cepts: abstraction and problem decomposition, parallelism, logical thinking, synchronization, algorithmic notions of flow control, user
interactivity, and data representation, and at three mastery levels of each dimension: basic, developing, proficient. As this type of
rubrics are typically graded by human raters, it required a clear distinction between levels of performance for raters to identify the
rating that best represented a student’s CT level.
Another method of portfolio analysis counted the presence of each CT dimension. It is useful for researchers who are interested in
tracking which CT components students used more frequently through verbal communication or analysis of students’ projects. Denner
et al. (2012) coded student-designed games and counted the presence of each coding category to identify how often they used the
features in the Stagecast Creator program, a programming platform that allows users to choose visual characters and before-after rules
instead of writing programming commands.
To summarize, the application of portfolio assessment could capture a holistic view of what skills students have obtained through
projects or work products. For studies involving hands-on collaborative projects, portfolio assessment would help to promote and
8
evaluate students’ communication in CT literacy as indicated by Lui et al. (2019a, 2019b) and can also serve as a formative assessment
tool providing students with feedback potentially beneficial to their future learning. Many researchers chose to use portfolio assess
ment when they employed programming platforms or instructional tools in their interventions to support students’ hands-on learning
activities. From another perspective, Chen et al. (2017a, 2017b) pointed out some challenges of using performance assessment to
measure CT: it may limit the use of a pre-post design and can only be implemented on a particular programming or computing
platform. They called for an assessment convenient for teachers to use in classrooms and applicable across platforms so that students’
CT performance can be comparable across studies using different instructional tools.
Surveys: Surveys are often used for investigating affective/non-cognitive learning outcomes, particularly for motivations and at
titudes toward CT learning. Twenty-six percent of studies developed surveys that used quantitative items (e.g., a Likert scale) and/or
open-ended questions. A majority of surveys were designed to collect student self-report responses, while some were administered to
teachers to collect their perceptions of CT during professional development interventions (Yadav et al., 2014). Although surveys were
often used to probe participants’ non-cognitive outcomes, several studies used surveys to elicit students’ cognitive CT skills. For
example, Bower, Wood, Lai, Howe, Lister, Mason, Highfield, et al. (2017) conducted qualitative thematic analysis of students’
open-ended responses to identify students’ usage of CT concepts and examine their awareness of CT. The frequent use of the survey
approach reveals its advantages of convenient administration, efficient data collection from a large sample size, and quantifiable
results. However, its self-report feature leads to the limitation of tapping students’ spontaneous perceptions or attitudes. In addition,
surveys might not be the best way to understand young children’s interests toward CT as they might not understand the survey
questions. To fill this gap, the usage of surveys could be coupled with other methods (e.g., interviews, focus groups) in order to
investigate students’ emergent in-depth thinking processes.
Interviews: Researchers conducted interviews to probe participants’ understanding of CT skills and coded their behaviors or verbal
communication using pre-developed protocols in fifteen percent of the reviewed studies. This approach is particularly useful for
exploring the uncertainty concerning how participants developed and applied CT. In a study by Cetin (2016), pre-service teachers were
interviewed and encouraged to express their experience of Scratch-based classroom teaching. Cetin (2016) thus analyzed the emergent
patterns from the interview data. Atmatzidou and Demetriadis (2016) used a more diagnosed approach, the think-alouds strategy
(Ericsson & Simon, 1998), by asking students to speak out when solving robot programming tasks. They analyzed the CT concepts
students used and interviewed students about their perceptions of CT concepts, understanding of basic programming concepts, and
their views of development of CT skills. Researchers usually employed interviews to support or elaborate on the results of traditional or
portfolio assessment by specifying students’ thinking processes of using CT skills to solve problems or difficulties they faced when
working on the CT-related hands-on projects. The challenges of using interviews, on the other hand, include its high cost and long time
spent on interviewing and coding the data as well as its small distribution to students, which makes it difficult to be quantified.
As demonstrated before, more than one third of studies (see in Table 3) chose to use more than one assessment tool to measure
various CT constructs in order to examine different aspects of student CT learning experiences. Specifically, around 20 percent studies
employed traditional tests together with other assessment tools in order to hold a more comprehensive view of students’ CT perfor
mance. For example, Grover et al. (2015) employed a traditional knowledge test to assess students’ CT knowledge, used surveys to
probe students’ perceptions of CT and CS, and interviewed low-performing students about their difficulties in applying CT to complete
their final projects. However, in concert with the less frequent single use of interview, a combined use of assessment tools to collect
both quantitative and qualitative data (i.e., interview data) was rarely studied. This lack of combined use highlights a need for a more
in-depth investigation of cognitive process of developing CT skills.
As most studies involved interventions, the pre-post assessment design was used to evaluate the intervention effects for 27 percent
of the studies. Further, when exploring the proximity between instruction and assessment according to the theory of Ruiz-Primo,
Shavelson, Hamilton, and Klein (2002), we found that most assessments were sensitive to their instructional modules and focused on
intervention related knowledge. Although the reviewed studies showed evidence for the improvement of CT by conducting corre
sponding instruction or intervention, it is unclear whether CT could help to improve students’ knowledge of other disciplines or
Table 3
The frequency of each single use and combined use of CT assessment tools.
Assessment tools Frequency Percent
Single use Traditional 15 15.63%

Portfolio 24 25.00%
Interview 8 8.33%
Survey 18 18.75%
Combined use Traditional þ Portfolio 6 6.25%
Traditional þ Survey 4 4.17%
Traditional þ Portfolio þ Interview 1 1.04%
Traditional þ Portfolio þ Survey 4 4.17%
Traditional þ Interview þ Survey 3 3.13%
Traditional þ Portfolio þ Interview þ Survey 1 1.04%
Portfolio þ Interview 5 5.21%
Portfolio þ Survey 4 4.17%
Portfolio þ Survey þ Interview 2 2.08%
Survey þ Interview 1 1.04%
Note: The denominator is the total number of the reviewed studies (N ¼ 96).
9
thinking skills in general. Given that CT can be implemented in teaching different disciplines, it would be interesting to explore if
students still had significant learning improvements using CT assessment tools free from particular programming platforms and
specific interventions and materials.
3.4. Reliability and validity evidence of the CT assessments
Among the 96 studies, fewer than half (45%) reported reliability evidence and 18 percent provided validity evidence. Some
reliability evidence (i.e., inter-rater reliability or Cohen’s kappa) was provided when human coding was undertaken, especially for
interviews, performance assessments, and open-ended responses. Some researchers who employed multiple choice or survey questions
reported internal consistency of tests to determine if students received similar scores on items measuring the same CT constructs.
Of the studies reporting validity, some researchers reported various types of validity evidence. For instance, Rom� an-Gonz�
alez,
P�erez-Gonza �lez, and Jim�enez-Fern� andez (2017a, 2017b) correlated student’s CT score with their problem-solving test scores, an
assessment evaluating speed and flexibility in performing logical operations. In another case, S� aez-L�opez et al. (2016a, 2016b)
emphasized the importance of using multiple methods to conduct data triangulation and validation. Collaborating with nine subject
matter experts, they evaluated the content validity of their survey questionnaire on perceptions of art history and CT and reported an
acceptable level of Aiken V content validity index. They also examined the construct validity of the questionnaire by using exploratory
factor analysis. As another example of examining content validity, Djambong and Freiman (2016) organized a panel of experts to
evaluate if their test contents corresponded to the intended constructs. They also suggested to analyze students’ think-alouds to
examine the validity evidence of Bebras items. Further, to evaluate the validity of Bebras items, Araujo, Andrade, Guerrero, and Melo
(2019) conducted confirmatory factor analysis to determine if there is statistical evidence to support that Bebras tasks assessed all the
claimed CT skills. Korkmaz et al. (2017a, 2017b) developed a five-point CT scale consisting of 29 items to examine students’ CT skill
levels. They studied the construct validity of this scale by using exploratory factor analysis, confirmatory factor analysis, and examined
item distinctiveness analyses through an independent-samples t-test among students of different performance levels. They further
evaluated reliability based on split half correlations of the scale, Spearman Brown reliability coefficient, Guttmann Split-Half value,
and Cronbach Alpha coefficient. Test-retest was also applied to evaluate the measurement consistency. This study showcased a solid
process of reliability and validity analyses of a self-developed CT scale.
Although some reliability and validity information are provided for CT assessments, most of the CT assessments lacked reliability
and validity evidence. Without this evidence, it is difficult for the field to use these assessments with confidence in classrooms to
measure students’ CT learning, especially in high-stake tests.
4. Discussion
CT is a relatively fast-moving field that has been explored by researchers for the last decade. Through this systematic review, we
mapped the current territory of CT assessment implemented across all educational levels, identified what has been explored, and
recognized the research gaps.
First, the majority of studies tended to focus on cultivating CT in the elementary and middle school grade levels. It is worth noting
that although it is challenging to conduct CT assessments developmentally appropriate for younger children due to their limited
reading and understanding skills (Zhang & Nouri, 2019), researchers have tried to implement CT at early stages of students’ cognitive
development. However, as no rationale suggests that elementary and middle schools are the only critical stage for students to learn CT,
it is necessary to further enrich the literature on examining CT assessments appropriate for high school (Flo
�rez et al., 2017) and college
students. By doing so, researchers and practitioners can find resources for a complete developmental trajectory of students’ CT skills.
Second, more CT assessments should be developed for CT interventions or activities applied in informal educational contexts.
Grover and Pea (2013), Kj€ allander, Åkerfeldt, Mannila, and Parnes (2018), and Martin (2017) suggested that informal educational
contexts such as makerspaces and DIY workshops play critical roles in implementing CT education. Hadad, Kachovska, Thomas, and
Yin (2019) observed various forms of formative assessments when facilitating scaffolded instruction of CT with high school students in
a making context. According to our review, CT assessment was lacking in informal contexts, so that it was difficult for the research
community to evaluate the impact of these informal-context-based interventions on improving CT.
Third, CT assessment for teacher education and professional development workshops was insufficiently researched. Kang, Dono
van, and McCarthy (2018) found that teachers had less knowledge and confidence toward teaching CT compared to teaching other
subject contents, and English (2018) claimed that it was problematic to introduce CT in classrooms without teacher professional
development. Thus, the field needs more solid research and practice to help teachers implement CT integration. Accordingly, more
assessments to measure teachers’ CT proficiencies should be developed as well.
Fourth, more assessments should be developed to emphasize the alignment between CT skills and domain knowledge, so that they
can better serve the trending integration of CT into STEM and non-STEM subjects. CT has the potential to deepen students’ under
standing of various subject domains, including that of STEM, non-STEM, and everyday life problem solving (CSTA & ISTE, 2011;
Weintrop et al., 2016; Wing, 2008). Although it is necessary to further verify how CT will help students build thinking skills in general,
Wing (2011, pp. 20–23) emphasized that CT has begun to influence other disciplines as it can go beyond computer science. In addition,
researchers (Jacob, Nguyen, Tofel-Grehl, Richardson, & Warschauer, 2018; Weintrop et al., 2016) stated that bringing CT into both
STEM and non-STEM classrooms would prepare students for the 21st century digital citizenship. From the perspective of learning
sciences, Grover and Pea (2018, pp. 19–37) suggested the mediating role of CT interventions and assessments that could prepare
students for learning STEM and non-STEM subjects. In light of the benefits of applying CT into other subject domains, we call for
10
CT-embedded assessments that evaluate domain knowledge for researchers and educators to adapt in their interventions or class
rooms. Further, compared to the assessments infusing CT with STEM subjects, the integration of CT into non-STEM assessments is
seriously lacking. Assessment in this field is necessary, given that the integration of CT with non-STEM fields are growing (e.g.,
Jenkins’s, 2015, poetry teaching based on a CT approach).
Fifth, interviews were currently under-utilized for measuring CT and deserved more attention. Given that CT functions as a set of
complex mental operations, it is necessary to explicate the components embedded in this cognitive process (Ambrosio, Almeida,
Macedo, & Franco, 2014). Researchers may conduct more interviews or think-alouds to collect in-depth, qualitative data as suggested
by Werner, Denner, and Campe (2014). Especially with the current tools, such as automatic transcription and analysis technology,
interview including think-alouds may play a more important role than before in studying students’ thinking.
Finally, reliability and validity evidence of assessment tools should be reported when applicable. As reliability and validity in
formation serve as indicators of assessment quality, their results would help researchers to make revisions when improving current CT
assessments and developing new CT instruments. Further, the report of acceptable reliability and validity indices would bolster the
broader distribution of the CT instrument as researchers and educators would prefer to use a reliable and valid assessment tool.
Even more fundamentally, the lack of consensus on the definition of CT resulted in a confusion in the constructs of CT assessment.
Researchers (e.g., Hadad & Lawless, 2014; Haseski et al., 2018; Yadav, Good, Voogt, & Fisser, 2017) called for an agreement on the
definition of CT and an alignment between CT application and subject domains. Without a theoretical framework to organize the
consistency among various definitions of CT, it is difficult to reach a comprehensive understanding of what constitutes CT, how CT is
different from other thinking processes, and how to assess students’ CT. Specifically, as noted in this review, the partial overlap of CT
and programming skills lends to the conflation of CT and programming assessments, which is common in most of the current CT
literature. When tracing back to the calls from Papert (1980) and Wing (2006; 2008), CT and programming concepts should be viewed
as separate but compatible cognitive tools that respectively expand the ways of problem-solving. Given the current state of CT
assessment, it is particularly critical for future research to investigate a theoretical framework of learning (e.g., Alexander’s Model for
Domain Learning) and assessment (e.g., Evidence-Centered Design) to disentangle the unique learning development of CT skills and
capture student CT performance.
Another gap is that the current reliance on programming concepts as the constructs of CT assessment requires computing or
programming proficiency for test designers and teachers. It may deter the application of CT skills in other subject domains from people
who are unfamiliar with CS and/or from schools or after-school programs without sufficient access to computers and programming
platforms. The lack of CT assessment in non-STEM fields may be largely due to this confusion between CT and programming skills. As
suggested in the review of Kaleliog �lu et al. (2016), CT can be implemented in broader learning contexts rather than focusing on
computational solutions. Further, Nishida et al. (2009) and Lockwood and Mooney (2018) suggested that the introduction of CT
learning could be done without computers; and Sentance and Csizmadia (2017) found that teachers preferred to use unplugged,
hands-on activities without computers to teach CT. Hennessey, Mueller, Beckett, and Fisher (2017) conducted a content analysis of the
Ontario elementary school curriculum by examining the terms or phrases associated with CT processes. They found that most of the
programming-related terms from Brennan and Resnick’s (2012) framework were rarely mentioned. Hence, more CT assessments free
from computers or programming platforms should be developed in order to broaden the integration of CT skills into curriculum and
programs that lack technology equipment or professional resources and to connect with students who do not see themselves as part of
the dominant computing culture. The “unplugged” CT assessment, which corresponds to Grover and Pea’s (2018) call on teaching and
learning CT skills without a computer or programming, can provide direction on CT skills in non-computing environments. As one
example of such studies, Yin, Hadad, Tang, and Lin (2019) designed a CT achievement assessment that measures CT-integrated STEM
learning and grounded test questions in the context of maker activities regarding physics and engineering without using computer or
maker tools.
Practically speaking, CT can be integrated into all subject domains across all educational levels. Different types of assessment tools
can be used for different educational purposes. Specifically, (a) teachers may use a traditional test with selected- and/or constructed-
response questions to evaluate students’ CT knowledge for a summative purpose, and researchers are able to evaluate the effect of their
CT intervention through carrying out a traditional CT test in the pre-post mode. (b) In order to glean an impression of students’
application of CT when working on hands-on projects, portfolio-driven approach situates CT assessment in a real-world context and
further allows teachers and researchers to provide formative feedback for students to improve their understanding and experience in
CT. (c) Surveys play an important role in probing students’ perceptions and attitudes toward CT, which would also inform teachers and
researchers of the potential improvement they could make to correspond to students’ self-efficacy and motivations in learning CT. (d)
Researchers may use interviews as a qualitative approach to undertake case studies for a thorough investigation of a student’s problem-
solving process based on CT and challenges they encounter, which could supplement the summative results of traditional assessments
or the formative results of performance assessments. In addition, researchers could employ think-alouds to understand how students
come up with the solution to each CT question and to examine whether they applied the CT skills the test claimed to measure. Such
information would allow researchers to improve their CT assessment. (e) Finally, the combination of different assessment tools may be
used as well to triangulate different CT assessments and provide a comprehensive evaluation of students’ CT learning.
5. Limitations
In the wake of the popularity in CT, researchers (e.g., Brown, 2017; Fl�
orez et al., 2017) have indicated concerns over the repre
sentation of gender and minority groups in CT-related education and research. Consistent with these concerns, many studies reported
demographic information when they conducted CT-related interventions. Among the interventions reviewed in this study, 67% of the
11
reviewed studies reported gender information and 34% provided information on race and ethnicity. Most of them balanced the gender
ratio, and many studies involved students of color, but few studies reported any information about the assessment features related to
students’ demographic backgrounds.
We decided not to focus on the demographic features of the participants in our review for the following reasons. Although many
studies conducting CT-related interventions reported demographic information, no study reported how they developed their assess
ment based on specific participants’ demographic features or how the assessment features were associated with student demographic
information, except for their grade levels. This might be because the CT assessment, as most assessments in education do, are
intentionally developed to serve a general population rather than a specific population to avoid stereotyping and bias (McMillan,
2013). However, applying CT assessment in different demographic groups may contribute to extending the current literature. Future
research may study the demographic issues, such as item bias, cultural responsiveness of CT assessments, and differential item
functions of CT assessments.
6. Conclusion and suggestions
This literature review analyzes specific CT assessments and identifies current gaps and future directions to conceptualize and
measure CT skills.
The results showed that CT assessments were developed and applied unequally in educational contexts. They were more frequently
used in elementary and middle schools than higher grades, and in formal educational settings more than informal ones. With a larger
proportion of the studies related to programming or CS, more research needs to study how CT assessment can be applied to the broader
content subjects. Studies measured a variety of CT related constructs, ranging from scientific knowledge, direct CT skills, computing
skills, to attitudinal perceptions toward CT. The most commonly measured constructs were students’ direct CT skills and programming
skills. Around 80 percent of studies assessed cognitive constructs while the rest measured affective outcomes. Four types of CT
assessment emerged from literature: traditional assessment composed of selected- and/or constructed-response questions, portfolio
assessment, survey, and interview. The majority of studies employed traditional paper-pencil tests and portfolio assessments to
evaluate CT and CT-related skills. Finally, only a few of the studies reported the reliability and validity evidence of their assessments.
CT is a fascinating and broad field. Our review intends to map the CT assessment territory which has been explored. The results
show that great work has been done but more is still needed. In particular, we suggest that researchers and CT assessment developers
consider the following takeaways when designing CT assessments: (a) create more CT assessments for high school, college, professional
development, and informal education settings; (b) focus on the assessment constructs aligned with the corresponding CT definitions
and with the subject-matter knowledge to promote the integration between CT and subject domains; (c) consider the concurrent use of
qualitative measures collected by interviews, think-alouds, or focus groups to better understand students’ proficiency of CT; (d) report
reliability and validity evidence to confidently qualify the assessment; (e) ground the definitions of CT with unique features that can be
distinct from programming, computing or other similar concepts; and (f) design common CT assessments that can be applicable across
platforms and devices in order to compare students’ CT performance under varied conditions of intervention.
Finally given that CT assessments have been developed in different fields for different grades, it would be helpful to build a
searchable database where the assessment tools are systematically collected, categorized, and organized, so that both researchers and
practitioners who need CT assessments can readily search for those tools to avoid reinventing the wheel. Instead they can build new
improvement on the existing achievement, e.g., empirically examining the reliability and validity of existing assessments and
developing new assessments to measure CT in a new field or unexamined student levels or teacher professional development.
CT can be applied in different subjects across different grade levels, which brings challenges as well as opportunities. Teachers and
researchers across different disciplines and different educational levels should increase collaboration so that we can assess and pro
mote CT systematically. In this way, assessments can be designed to map the learning progression of CT in each discipline and to
encourage students to apply CT skills into learning other disciplines.
Funding
This work was supported by the National Science Foundation (NSF) under grant with an ID of 1543124.
Declaration of competing interest
None.
CRediT authorship contribution statement
Xiaodan Tang: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing - original draft. Yue Yin:
Conceptualization, Methodology, Investigation, Supervision, Writing - review & editing. Qiao Lin: Formal analysis, Writing - review &
editing. Roxana Hadad: Funding acquisition, Project administration, Resources, Writing - review & editing. Xiaoming Zhai: Writing -
review & editing.
12
Acknowledgement
This material is based upon work supported by the National Science Foundation (NSF) under grant with an ID of 1543124. Any
opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily
reflect the views of the NSF.
Appendix A. The Reviewed Computational Thinking Studies
Authors School level Subject domain Educational Assessment tools

setting
Adler and Kim (2018) Teacher CS/Programming Formal Survey

Altanis, Retalis, and Petropoulou (2018) Middle Robotics/Game and CS/ Formal Portfolio
Programming
Angeli and Valanides (2019) K-elementary Formal Traditional, Portfolio
Asad, Tibi, and Raiyn (2016) Elementary CS/Programming Formal Survey
Atmatzidou and Demetriadis (2016) High Robotics/Game Formal Traditional, Survey,
Interview
Bagley and Rabin (2016) College STEM Formal Interview
Basawapatna (2016) Middle STEM Formal Portfolio
Basogain, Olabe, Olabe, and Rico (2018) Elementary, Middle, CS/Programming Formal Traditional, Portfolio
High
Basu et al. (2016) Middle STEM Formal Traditional, Portfolio
Basu et al. (2017a, 2017b) Middle STEM Formal Traditional, Portfolio
Benakli, Kostadinov, Satyanarayana, and Singh College STEM Formal Survey
(2017)
Berland and Lee (2011) College Robotics/Game Informal Interview
Bers (2010) K-elementary Robotics/Game and STEM Formal Portfolio
Bers et al. (2014) K-elementary Robotics/Game and CS/ Formal Portfolio
Programming
Bower, Wood, Lai, Howe, Lister, Mason, Highfield, Teachers non-STEM Informal Survey
et al. (2017)
Brady et al. (2017) High CS/Programming Formal Survey
Bucher (2017) Teachers non-STEM Informal Interview
Burleson et al. (2018) Elementary CS/Programming Informal Portfolio
Cetin (2016) Teachers CS/Programming Formal Traditional, Survey
Cetin and Andrews-Larson (2016) Teachers CS/Programming Formal Traditional, Survey,
Interview
Chang and Peterson (2018) Teachers non-STEM Formal Interview
Chen et al. (2017a, 2017b) Elementary Robotics/Game Formal Traditional
Choi, Lee, and Lee (2017) Elementary CS/Programming Formal Traditional
Csernoch, Bir�o, M�
ath, and Abari (2015) College STEM Formal Portfolio
Denner et al. (2012) Middle Robotics/Game and CS/ Informal Portfolio
Programming
Denner, Werner, Campe, and Ortiz (2014) Middle CS/Programming Informal Portfolio, Survey
Dolgopolovas et al. (2016) College STEM Formal Traditional
Durak and Saritepeci (2018) Middle, High Traditional, Survey
Falloon (2016) Elementary STEM Formal Portfolio
Flanigan et al. (2017a, 2017b) College CS/Programming Formal Traditional
Gadanidis, Clements, and Yiu (2018) Elementary STEM Formal Portfolio
Gandolfi (2018) College Robotics/Game Informal Survey
Garneli and Chorianopoulos (2018) Middle STEM and CS/Programming Informal Portfolio
Grover et al. (2015) Middle CS/Programming Formal Traditional, Portfolio,
Survey, Interview
Günbatar and Bakırcı (2019) Teachers non-STEM Formal Survey
Hershkovitz et al. (2019) Elementary CS/Programming and Informal Portfolio
Robotics/Game
Hestness, Jass Ketelhut, McGinnis, and Plane Teachers CS/Programming Informal Portfolio, Interview
(2018)
Hsiao et al. (2019) Elementary Robotics/Game and CS/ Formal Traditional
Programming
Izu, Mirolo, Settle, Mannila, and Stupuriene K-elementary,Middle, High Traditional
(2017)
Jaipal-Jamani and Angeli (2017a, 2017b Teachers Robotics/Game Formal Traditional, Portfolio, Survey
Jenkins (2015) Middle non-STEM Formal Traditional
Jenkins (2017) Middle non-STEM Formal Portfolio
Jenson and Droumeva (2016) Elementary Robotics/Game and CS/ Informal Traditional
Programming
Jun, Han, Kim, and Lee (2014) Elementary Formal Traditional
Kale, Akcaoglu, Cullen, and Goh (2018) Teachers Survey
(continued on next page)
13
(continued )
Authors School level Subject domain Educational Assessment tools
setting
Kong, Chiu, and Lai (2018) Elementary Formal Survey

Korkmaz and Bai (2019) High Survey
Korkmaz et al. (2017a, 2017b) College Formal Survey
Korucu, Gencturk, and Gundogdu (2017) Elementary Formal Survey
Kukul and Karatas (2019) Middle, High Survey
Lachney, Babbitt, Bennett, and Eglash (2019) Middle STEM and CS/Programming Informal Portfolio, Survey, Interview
Lai, Chen, Lai, Chang, and Su (2019) College CS/Programming Formal Portfolio, Survey
Lee et al. (2014) Elementary, Middle Robotics/Game Informal Interview
Leonard et al. (2016) Middle Robotics/Game Formal Portfolio
Leonard et al. (2018) Teacher, K-12 Robotics/Game Formal Portfolio, Survey
students
Liu, Zhi, Hicks, and Barnes (2017) Middle Robotics/Game and CS/ Informal Survey
Programming
Lui et al. (2019a, 2019b) High STEM Formal Portfolio
Malyn-Smith and Lee (2012) Middle, High, STEM Informal Portfolio
College
Marcelino, Pessoa, Vieira, Salvador, and Mendes Teachers CS/Programming Formal Portfolio, Interview
(2018)
Mesiti et al. (2019) Middle, High Informal Portfolio, Survey, Interview
Moreno-Le� on et al. (2017a, 2017b) Informal Portfolio
Mouza, Marzocchi, Pan, and Pollock (2016) Middle CS/Programming Informal Traditional, Portfolio, Survey
Mouza, Yang, Pan, Ozden, and Pollock (2017) Teachers non-STEM Formal Survey
Mu~
noz-Repiso and Caballero-Gonz� alez (2019) K-elementary Robotics/Game and CS/ Formal Portfolio
Programming
Pala and MihçiTürker (2019) Teachers CS/Programming Formal Survey
Peel, Sadler, and Friedrichsen (2019a, 2019b) High STEM Formal Portfolio, Interview
Pei et al. (2018) High STEM Formal Portfolio
Pellas and Peroutseas (2016) High CS/Programming Informal Portfolio, Interview
Peteranetz et al. (2017) College CS/Programming Fromal Traditional
Pinkard, Martin, and Erete (2019) Elementary, Middle CS/Programming Informal Traditional, Portfolio,
Interview
Pugnali et al. (2017) K-elementary Robotics/Game and CS/ Informal Portfolio
Programming
Rodríguez-Martínez, Gonz�alez-Calero, and Elementary Formal Traditional
S�aez-L�
opez (2019)
Roman-Gonzalez et al. (2017a, 2017b) Elementary, Middle, Formal Traditional
High
Rom� an-Gonz�alez, P�erez-Gonz�
alez, Moreno-Le�
on, Elementary, Middle, Formal Traditional, Survey
and Robles (2018) High
Rose, Habgood, and Jay (2017) Elementary Formal Portfolio
S�
aez-L�opez et al. (2016) Elementary non-STEM Formal Traditional, Portfolio, Survey
Sengupta et al. (2013) Middle STEM Both Traditional
Shell and Soh (2013a, 2013b) College CS/Programming Formal Traditional
Sherman and Martin (2015) College CS/Programming Formal Portfolio
Sung, Ahn, and Black (2017) K-elementary STEM and CS/Programming Informal Traditional, Portfolio
Taylor and Baek (2019) Elementary Robotics/Game and CS/ Formal Traditional, Portfolio, Survey
Programming
Thomas, Rankin, Minor, and Sun (2017) Middle Robotics/Game Informal Portfolio
Tran (2019) Elementary CS/Programming Formal Traditional, Survey,
Interview
Tsai, Shen, Tsai, and Chen (2017) College CS/Programming Formal Traditional
Turchi, Fogli, and Malizia (2019) High Informal Portfolio
Weintrop et al. (2016) Middle, High, Robotics/Game and STEM Informal Portfolio, Interview
College
Wilkerson-Jerde (2014) Middle STEM Formal Traditional, Portfolio
Wolz et al. (2011) Middle non-STEM Both Survey, Interview
Wong and Cheung (2018) Elementary CS/Programming Formal Interview
Wu (2018) Middle Robotics/Game Informal Portfolio, Survey
Wu, Hu, Ruis, and Wang (2019) College CS/Programming Formal Interview
Yadav et al. (2014) Teachers non-STEM Formal Traditional, Survey
Yagci (2019) High Survey
Yildiz, Yilmaz, and Yilmaz (2017) Middle Robotics/Game Informal Survey
Yuen and Robbins (2014) College CS/Programming Formal Interview
Zhong et al. (2016a, 2016b) Elementary CS/Programming Formal Portfolio
14
Appendix B. Sample Questions for Each Category of Constructs
I. First-order cognitive CT construct
Beaver paddles in his canoe on a river. The river has a number of little lakes (Fig. 1). Beaver likes all lakes of the river and has
thought of an algorithm to make sure that he reaches every lake. He knows that at each lake there is a maximum of two rivers that he
has not yet seen. If beaver arrives at a lake he decides which river to take with the following rules:
� If there are two rivers he has not yet seen, he takes the river on his left hand side.
� If there is one river which beaver has not yet seen, beaver takes this river.
� If beaver has seen all the rivers from a little lake, he paddles his canoe one lake back towards the previous lake.
Fig. 1. Task ‘‘Beaver in his canoe’’.

Beaver stops his day of canoeing if he has seen everything and has come back to the start point. In Fig. 1 you can see the river and
the little lakes where beaver paddles his canoe.
In each little lake beaver sees a different animal. Beaver writes down the animal name when he sees an animal for the first time.
In which order will beaver write down the animals?
Answer (the correct answer is written in bold).
a Fish, frog, crocodile, turtle, stork, snake, otter, duck.

b Fish, crocodile, snake, stork, duck, otter, frog, turtle.
c Fish, frog, turtle, crocodile, stork, otter, duck, snake.
d Fish, frog, turtle.
Source: Dolgopolovas et al. (2016).
II. First-order non-cognitive CT construct
1. Computational thinking is understanding how computers work.

2. Computational thinking involves thinking logically to solve problems.
3. Computational thinking involves using computers to solve problems.
4. Computational thinking involves abstracting general principles and applying them to other solutions.
5. I do not think it is possible to apply computing knowledge to solve other problems.
6. I am not comfortable with learning computing concepts.
7. I can achieve good grades (C or better) in computer courses.
8. I can learn to understand computing concepts.
9. I use computing skills in my daily life.
10. I doubt that I have the skills to solve problems by using computer applications.
11. I think computer science is boring.
12. The challenge of solving problems using computer science appeals to me.
13. I think computer science is interesting.
14. I will voluntarily take computing courses if I were given the opportunity.
15. Computational thinking can be incorporated in the classroom by using computers in the lesson plan.
16. Computational thinking can be incorporated in the classroom by allowing students to problem solve.
15
17. Knowledge of computing will allow me to improve my performance in my career.

18. My career does not require that I learn computing skills.
19. I expect that learning computing skills will help me to achieve my career goals.
20. I hope that as my career continues it will require the use of computing concepts.
21. Having background knowledge and understanding of computer science is valuable in and of itself.
Note: It’s a 4-point Likert-type scale that ranged from 1 ¼ strongly disagree, 2 ¼ disagree, 3 ¼ agree, and 4 ¼ strongly agree.
Source: Leonard et al. (2018).
III. Second-order programming/computing/CS construct
Source: Grover et al. (2015).
IV. Second-order non-CS STEM construct
Find at least one example of every possible triangle area in the given 4 � 4 lattice (there are 16 possible areas). It asks students to
explore shapes and size of lattice triangles and discover (or re-discover) some unexpected results.
Sources: Pei et al. (2018).
V. Second-order non-STEM construct
16
Source: Jenkins (2015).
VI. Second-order non-cognitive programming/CS/computing construct
1) You created a mini science project using the scratch today. Now, how do you perceive Programming as an educational tool?
2) What challenges do you foresee in incorporating programming into your teaching?
3) Do you think programming can improve students’ critical thinking or modify any misconceptions?
4) Do you have any future project ideas using Scratch programming for your teaching?
5) On a scale from 1 to 5, how likely are you to incorporate programming when teaching:
6) Did you have any misconception about the Earth’s rotation around the sun before this activity? What about now?
Source: Adler and Kim (2018a, 2018b.
17
VII. Second-order non-cognitive non-CS construct
1. Understood artistic elements in paintings.

2. Learned biographical and historical contents of Spanish painters.
3. Increased cultural and artistic competence to understand paintings.
4. Improved the ability to understand artistic expressions from different eras.
5. Analyzed historical and artistic content in paintings.
Note: It’s a 5-point Likert-type scale with 1 as strongly disagree and 5 as strongly agree.
Source: S� aez-Lo
�pez et al. (2016)
References
diSessa, A. A. (2018). Computational literacy and “the big picture” concerning computers in mathematics education. Mathematical Thinking and Learning, 20(1), 3–31.
Adler, R. F., & Kim, H. (2018a). Enhancing future K-8 teachers’ computational thinking skills through modeling and simulations. Education and Information
Technologies, 23(4), 1501–1514.
Ambrosio, A. P., Almeida, L. S., Macedo, J., & Franco, A. H. R. (2014). Exploring core cognitive skills of Computational Thinking. In Paper presented at the psychology of
programming interest group Annual conference 2014 (PPIG 2014).
Araujo, A. L. S. O., Andrade, W. L., Guerrero, D. D. S., & Melo, M. R. A. (2019). How many abilities can we measure in computational thinking?: A study on Bebras
challenge. In Paper presented at the proceedings of the 50th ACM technical symposium on computer science education.
Atmatzidou, S., & Demetriadis, S. (2016). Advancing students’ computational thinking skills through educational robotics: A study on age and gender relevant
differences. Robotics and Autonomous Systems, 75(Part B), 661–670.
Barr, D., Harrison, J., & Conery, L. (2011). Computational thinking: A digital age skill for everyone. Learning and Leading with Technology, 38(6), 20–23.
Barr, V., & Stephenson, C. (2011). Bringing computational thinking to K-12: What is involved and what is the role of the computer science education community? Acm
Inroads, 2(1), 48–54.
Basu, S., Biswas, G., & Kinnebrew, J. S. (2017). Learner modeling for adaptive scaffolding in a Computational Thinking-based science learning environment. User
Modeling and User-Adapted Interaction, 27(1), 5–53.
Basu, S., Biswas, G., Sengupta, P., Dickes, A., Kinnebrew, J. S., & Clark, D. (2016). Identifying middle school students’ challenges in computational thinking-based
science learning. Research and Practice in Technology Enhanced Learning, 11(1), 13.
Bers, M. U., Flannery, L., Kazakoff, E. R., & Sullivan, A. (2014). Computational thinking and tinkering: Exploration of an early childhood robotics curriculum.
Computers & Education, 72, 145–157.
Bower, M., Wood, L. N., Lai, J. W. M., Howe, C., Lister, R., Mason, R., et al. (2017). Improving the computational thinking pedagogical capabilities of school teachers.
Australian Journal of Teacher Education, 42(3), 53–72.
Brennan, K., & Resnick, M. (2012, April). New frameworks for studying and assessing the development of computational thinking. In , Vol. 1. Proceedings of the 2012
annual meeting of the American educational research association, vancouver, Canada (p. 25).
Brown, J. C. (2017). A metasynthesis of the complementarity of culturally responsive and inquiry-based science education in k-12 settings: Implications for advancing
equitable science teaching and learning. Journal of Research in Science Teaching, 59(9).
Cartelli, A., Dagiene, V., & Futschek, G. (2012). Bebras contest and digital competence assessment: Analysis of frameworks. Current trends and future practices for digital
literacy and competence. Hershey, PA: IGI Global.
Cetin, I. (2016). Preservice teachers’ introduction to computing: Exploring utilization of scratch. Journal of Educational Computing Research, 54(7), 997–1021.
Chen, G., Shen, J., Barth-Cohen, L., Jiang, S., Huang, X., & Eltoukhy, M. (2017). Assessing elementary students’ computational thinking in everyday reasoning and
robotics programming. Computers & Education, 109, 162–175.
CSTA, & ISTE. (2011). Operational definition of computational thinking for K–12 education. Retrieved from http://csta.acm.org/Curriculum/sub/CurrFiles/
CompThinkingFlyer.pdf.
Cuny, J., Snyder, L., & Wing, J. M. (2010). Demystifying computational thinking for non-computer scientists. Unpublished manuscript in progress, referenced in http://
www.cs.cmu.edu/CompThink/resources/TheLinkWing.pdf.
Dagiene,_ V., & Futschek, G. (2008). Bebras international contest on informatics and computer literacy: Criteria for good tasks. In Paper presented at the 3rd international
conference on informatics in secondary schools - evolution and perspectives: Informatics education - supporting computational thinking torun, Poland.
Denner, J., Werner, L., & Ortiz, E. (2012). Computer games created by middle school girls: Can they be used to measure understanding of computer science concepts?
Computers & Education, 58(1), 240–249.
Djambong, T., & Freiman, V. (2016). Task-based assessment of students’ computational thinking skills developed through visual programming or tangible coding environments.
International Association for Development of the Information Society.
Durak, H. Y., & Saritepeci, M. (2018a). Analysis of the relation between computational thinking skills and various variables with the structural equation model.
Computers & Education, 116, 191–202.
English, L. (2018). On MTL’s second milestone: Exploring computational thinking and mathematics learning. Mathematical Thinking and Learning, 20(1), 1–2.
Ericsson, K. A., & Simon, H. A. (1998). How to study thinking in everyday life: Contrasting think-aloud protocols with descriptions and explanations of thinking. Mind,
Culture and Activity, 5(3), 178–186.
Falloon, G. (2016). An analysis of young students’ thinking when completing basic coding tasks using Scratch Jnr. On the iPad. Journal of Computer Assisted Learning,
32(6), 576–593.
Fields, D. A., Lui, D., & Kafai, Y. B. (2019). Teaching computational thinking with electronic textiles: Modeling iterative practices and supporting personal projects in
exploring computer science. In Computational thinking education (pp. 279–294). Singapore: Springer.
Fields, D. A., Shaw, M. S., & Kafai, Y. B. (2018). Personal learning journeys: Reflective portfolios as “objects-to-learn-with” in an e-textiles high school class. In
V. Dagiene, & E. Jastu_e (Eds.), Constructionism 2018: Constructionism, computational thinking and educational innovation: Conference proceedings (pp. 213–223).
Vilnius, Lithuania, http://www.constructionism2018.fsf.vu.lt/proceedings.
Flanigan, A. E., Peteranetz, M. S., Shell, D. F., & Soh, L.-K. (2017a). Implicit intelligence beliefs of computer science students: Exploring change across the semester.
Contemporary Educational Psychology, 48, 179–196.
Fl�
orez, F., Casallas, R., Hern�andez, M., Reyes, A., Restrepo, S., & Danies, G. (2017). Changing a generation’s way of thinking: Teaching computational thinking
through programming. Review of Educational Research, 87(4), 834–860.
Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2015). How to design and evaluate research in education (9th ed.). New York, NY: McGraw-Hill Education.
Garneli, V., & Chorianopoulos, K. (2018). Programming video games and simulations in science education: Exploring computational thinking through code analysis.
Interactive Learning Environments, 26(3), 386–401.
Gonz�alez, M. R. (2015). Computational thinking test: Design guidelines and content validation. In Paper presented at the proceedings of EDULEARN15 conference.
18
Grover, S. (2017a). Assessing algorithmic and computational thinking in K-12: Lessons from a middle school classroom. In Emerging research, practice, and policy on
computational thinking (pp. 269–288). Springer.
Grover, S. (2017b). Assessing algorithmic and computational thinking in K-12: Lessons from a middle school classroom. Emerging research, practice, and policy on
computational thinking (pp. 269–288). Springer.
Grover, S., & Pea, R. (2013). Computational thinking in K–12: A review of the state of the field. Educational Researcher, 42(1), 38–43.
Grover, S., & Pea, R. (2018). Computational thinking: A competency whose time has come. Computer science education: Perspectives on teaching and learning in school.
London: Bloomsbury Academic.
Grover, S., Pea, R., & Cooper, S. (2015). Designing for deeper learning in a blended computer science course for middle school students. Computer Science Education, 25
(2), 199–237.
Gülbahar, Y., Kert, S., & Kalelio� glu, F. (2018). The self-efficacy perception scale for computational thinking skill: Validity and reliability study. Turkish Journal of
Computer and Mathematics Education, 10(1), 1–29.
Hadad, R., Kachovska, M., Thomas, K., & Yin, Y. (2019). Practicing formative assessment for computational thinking in making environments. Journal of Science
Education and Technology, 1–12.
Hadad, R., & Lawless, K. (2014). Assessing computational thinking. In M. Khosrow-Pour (Ed.), Encyclopedia of information science and technology (pp. 1568–1578).
Hershey, PA: IGI Global.
Haseski, H. I., Ilic, U., & Tugtekin, U. (2018). Defining a new 21st century skill-computational thinking: Concepts and trends. International Education Studies, 11(4),
29–42.
Henderson, P. B., Cortina, T. J., & Wing, J. M. (2007). Computational thinking. ACM SIGCSE Bulletin, 39(1), 195–196.
Hennessey, E. J. V., Mueller, J., Beckett, D., & Fisher, P. A. (2017). Hiding in plain sight: Identifying computational thinking in the Ontario elementary school
curriculum. Journal of Curriculum and Teaching, 6(1), 79–96.
Hsu, T. C., Chang, S. C., & Hung, Y. T. (2018). How to learn and how to teach computational thinking: Suggestions based on a review of the literature. Computers &
Education, 126, 296–310.
ISTE. (2015). CT leadership toolkit.
Jacob, S., Nguyen, H., Tofel-Grehl, C., Richardson, D., & Warschauer, M. (2018). Teaching computational thinking to English learners. NYS TESOL journal, 5(2).
Jaipal-Jamani, K., & Angeli, C. (2017a). Effect of robotics on elementary preservice teachers’ self-efficacy, science learning, and computational thinking. Journal of
Science Education and Technology, 26(2), 175–192.
Jenkins, C. (2015). Poem generator: A comparative quantitative evaluation of a microworlds-based learning approach for teaching English. International Journal of
Education and Development using ICT, 11(2).
Jenson, J., & Droumeva, M. (2016). Exploring media literacy and computational thinking: A game maker curriculum study. Electronic Journal of E-Learning, 14(2),
111–121.
Jona, K., Wilensky, U., Trouille, L., Horn, M., Orton, K., Weintrop, D., et al. (2014). Embedding computational thinking in science, technology, engineering, and math
(CT-STEM). In Paper presented at the future directions in computer science education summit meeting, Orlando, FL.
Kalelio�glu, F., Gülbahar, Y., & Kukul, V. (2016). A framework for computational thinking based on a systematic research review. Baltic Journal of Modern Computing, 4
(3), 583.
Kang, E. J. S., Donovan, C., & McCarthy, M. J. (2018). Exploring elementary teachers’ pedagogical content knowledge and confidence in implementing the NGSS
science and engineering practices. Journal of Science Teacher Education, 29(1), 9–29.
Kj€
allander, S., Åkerfeldt, A., Mannila, L., & Parnes, P. (2018). Makerspaces across settings: Didactic design for programming in formal and informal teacher education
in the nordic countries. Journal of Digital Learning in Teacher Education, 34(1), 18–30.
Korkmaz, O.,€ Çakir, R., & Ozden,
€ M. Y. (2017a). A validity and reliability study of the computational thinking scales (CTS). Computers in Human Behavior, 72, 558–569.
Lee, I., Martin, F., & Apone, K. (2014). Integrating computational thinking across the K–8 curriculum. Acm Inroads, 5(4), 64–71.
Lee, I., Martin, F., Denner, J., Coulter, B., Allan, W., Erickson, J., … Werner, L. (2011). Computational thinking for youth in practice. Acm Inroads, 2(1), 32–37.
Leonard, J., Barnes-Johnson, J., Mitchell, M., Unertl, A., Stubbe, C. R., & Ingraham, L. (2017). Developing teachers’ computational thinking beliefs and engineering practices
through game design and robotics. North American Chapter of the International Group for the Psychology of Mathematics Education.
Leonard, J., Mitchell, M., Barnes-Johnson, J., Unertl, A., Outka-Hill, J., Robinson, R., et al. (2018). Preparing teachers to engage rural students in computational
thinking through robotics, game design, and culturally responsive teaching. Journal of Teacher Education, 69(4), 386–407.
Lockwood, J., & Mooney, A. (2018). Computational thinking in education: Where does it fit? A systematic literary review. International Journal of Computer Sciences
and Engineering Systems, 2(1), 41–60.
Lui, D., Walker, J. T., Hanna, S., Kafai, Y. B., Fields, D., & Jayathirtha, G. (2019). Communicating computational concepts and practices within high school students’
portfolios of making electronic textiles. Interactive Learning Environments, 1–18.
Lye, S. Y., & Koh, J. H. L. (2014). Review on teaching and learning of computational thinking through programming: What is next for K-12? Computers in Human
Behavior, 41, 51–61.
Malyn-Smith, J., & Lee, I. (2012). Application of the occupational analysis of computational thinking-enabled STEM professionals as a program assessment tool. The
Journal of Computational Science Education, 3(1), 2–10.
Martin, C. (2017). Libraries as facilitators of coding for all. Knowledge Quest, 45(3), 46–53.
McMillan, J. H. (2013). Classroom assessment: Principles and practice for effective instruction (6th ed.). Boston: Pearson/Allyn and Bacon.
Mesiti, L. A., Parkes, A., Paneto, S. C., & Cahill, C. (2019). Building capacity for computational thinking in youth through informal education. Journal of Museum
Education, 44(1), 108–121.
Moreno-Le� on, J., Robles, G., & Rom�an-Gonz� alez, M. (2017). Towards data-driven learning paths to develop computational thinking with scratch. IEEE Transactions on
Emerging Topics in Computing, (99). pp. 1-1.
Mu~ noz-Repiso, A. G. V., & Caballero-Gonz� alez, Y. A. (2019). Robotics to develop computational thinking in early childhood education. Comunicar: Media Education
Research Journal, 27(59), 63–72.
National Research Council. (2010). Committee for the Workshops on Computational Thinking: Report of a workshop on the scope and nature of computational thinking.
Washington, DC: National Academies Press.
National Research Council. (2011). Committee for the Workshops on Computational Thinking: Report of a workshop of pedagogical aspects of computational thinking.
Washington, DC: National Academies Press.
National Research Council. (2012). A framework for K–12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Academies Press.
NGSS Lead States. (2013). Next generation science standards: For states, by states. Washington, DC: The National Academies Press.
Nishida, T., Kanemune, S., Idosaka, Y., Namiki, M., Bell, T., & Kuno, Y. (2009). A CS unplugged design pattern. ACM SIGCSE Bulletin, 41(1), 231–235.
Palts, T., Pedaste, M., Vene, V., & Vinikiene, _ L. (2017). Tasks for assessing skills of computational thinking. In Paper presented at the proceedings of the 2017 ACM
conference on innovation and technology in computer science education.
Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. Basic Books, Inc.
Papert, S. (1996). The connected family: Bridging the digital generation gap (Vol. 1). Taylor Trade Publishing.
Peel, A., Sadler, T. D., & Friedrichsen, P. (2019a). Learning natural selection through computational thinking: Unplugged design of algorithmic explanations. Journal
of Research in Science Teaching.
Pei, C., Weintrop, D., & Wilensky, U. (2018). Cultivating computational thinking practices and mathematical habits of mind in Lattice Land. Mathematical Thinking and
Learning, 20(1), 75–89.
Peteranetz, M. S., Flanigan, A. E., Shell, D. F., & Soh, L.-K. (2017). Computational creativity exercises: An avenue for promoting learning in computer science. IEEE
Transactions on Education, 60(4), 305–313.
19
Pugnali, A., Sullivan, A., & Bers, M. U. (2017). The impact of user Interface on young children’s computational thinking. Journal of Information Technology Education:
Innovations in Practice, 16, 171–193.
Repenning, A., Webb, D., & Ioannidou, A. (2010). March). Scalable game design and the development of a checklist for getting computational thinking into public
schools. In Proceedings of the 41st ACM technical symposium on Computer science education (pp. 265–269). ACM.
Rich, K. M., Spaepen, E., Strickland, C., & Moran, C. (2019). Synergies and differences in mathematical and computational thinking: Implications for integrated instruction.
Interactive Learning Environments.
Rom� an-Gonz� alez, M., P�
erez-Gonz�alez, J.-C., & Jim�
enez-Fern�andez, C. (2017a). Which cognitive abilities underlie computational thinking? Criterion validity of the
computational thinking test. Computers in Human Behavior, 72, 678–691.
Ruiz-Primo, M. A., Shavelson, R. J., Hamilton, L., & Klein, S. (2002). On the evaluation of systemic science education reform: Searching for instructional sensitivity.
Journal of Research in Science Teaching: The Official Journal of the National Association for Research in Science Teaching, 39(5), 369–393.
S�
aez-L�opez, J.-M., Rom� an-Gonz�alez, M., & V�azquez-Cano, E. (2016). Visual programming languages integrated across the curriculum in elementary school: A two year
case study using “scratch” in five schools. Computers & Education, 97, 129–141.
Selby, C. C., & Woollard, J. (2013). Computational thinking: The developing definition. In Paper presented at the 18th annual conference on innovation and technology in
computer science education, Canterbury.
Sengupta, P., Kinnebrew, J. S., Basu, S., Biswas, G., & Clark, D. (2013). Integrating computational thinking with K-12 science education using agent-based
computation: A theoretical framework. Education and Information Technologies, 18(2), 351–380.
Sentance, S., & Csizmadia, A. (2017). Computing in the curriculum: Challenges and strategies from a teacher’s perspective. Education and Information Technologies, 22
(2), 469–495.
Shell, D. F., & Soh, L.-K. (2013a). Profiles of motivated self-regulation in college computer science courses: Differences in major versus required non-major courses.
Journal of Science Education and Technology, 22(6), 899–913.
Shute, V. J., Sun, C., & Asbell-Clarke, J. (2017). Demystifying computational thinking. Educational Research Review, 22, 142–158.
Sullivan, F. R., & Heffernan, J. (2016). Robotic construction kits as computational manipulatives for learning in the STEM disciplines. Journal of Research on
Technology in Education, 48(2), 105–128.
Sung, E. (2019). Fostering computational thinking in technology and engineering education: An unplugged hands-on engineering design approach. Technology and
Engineering Teacher, 78(5), 8–13.
Testa, J. (2009). The Thomson Reuters journal selection process. Transnational Corporations Review, 1(4), 59–66.
Turchi, T., Fogli, D., & Malizia, A. (2019). Fostering computational thinking through collaborative game-based learning. Multimedia Tools and Applications, 78(10),
13649–13673.
Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., et al. (2016). Defining computational thinking for mathematics and science classrooms. Journal of
Science Education and Technology, 25(1), 127–147.
Weintrop, D., Beheshti, E., Horn, M. S., Orton, K., Trouille, L., Jona, K., et al. (2014). Interactive assessment tools for computational thinking in High School STEM
classrooms. In Paper presented at the international conference on intelligent Technologies for interactive entertainment.
Werner, L., Denner, J., & Campe, S. (2014). Using computer game programming to teach computational thinking skills. In Paper presented at the Learning, education and
games.
Werner, L., Denner, J., Campe, S., & Kawamoto, D. C. (2012). February). The fairy performance assessment: Measuring computational thinking in middle school. In
Proceedings of the 43rd ACM technical symposium on computer science education (pp. 215–220). ACM.
Wilkerson-Jerde, M. H. (2014). Construction, categorization, and consensus: Student generated computational artifacts as a context for disciplinary reflection.
Educational Technology Research & Development, 62(1), 99–121.
Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35.
Wing, J. M. (2008). Computational thinking and thinking about computing. Philosophical Transactions of the Royal Society of London - A: Mathematical, Physical and
Engineering Sciences, 366(1881), 3717–3725.
Wing, J. (2011). Research notebook: Computational thinking—what and why. The Link Magazine.
Wolz, U., Stone, M., Pearson, K., Pulimood, S. M., & Switzer, M. (2011). Computational thinking and expository writing in the middle school. ACM Transactions on
Computing Education, 11(2), 9.
Wolz, U., Stone, M., Pulimood, S. M., & Pearson, K. (2010). March). Computational thinking via interactive journalism in middle school. In Proceedings of the 41st ACM
technical symposium on Computer science education (pp. 239–243). ACM.
Yadav, A., Good, J., Voogt, J., & Fisser, P. (2017). Computational thinking as an emerging competence domain. In Competence-based vocational and professional
education (pp. 1051–1067). Springer.
Yadav, A., Mayfield, C., Zhou, N., Hambrusch, S., & Korb, J. T. (2014). Computational thinking in elementary and secondary teacher education. ACM Transactions on
Computing Education, 14(1), 5.
Yin, Y., Hadad, R., Tang, X., & Lin, Q. (2019). Improving and assessing computational thinking in maker activities: The integration with physics and engineering learning
Journal of science Education and technology (pp. 1–26).
Zhang, L., & Nouri, J. (2019). A systematic review of learning computational thinking through Scratch in K-9. Computers & Education, 141, 103607.
Zhong, B., Wang, Q., Chen, J., & Li, Y. (2016). An exploration of Three-Dimensional Integrated Assessment for computational thinking. Journal of Educational
Computing Research, 53(4), 562–590.
References for Appendix A. The reviewed CT studies
Adler, R. F., & Kim, H. (2018b). Enhancing future K-8 teachers’ computational thinking skills through modeling and simulations. Education and Information
Technologies, 23(4), 1501–1514. https://doi.org/10.1007/s10639-017-9675-1.
Altanis, I., Retalis, S., & Petropoulou, O. (2018). Systematic design and rapid development of motion-based touchless games for enhancing students’ thinking skills.
Education Sciences, 8, 15.
Angeli, C., & Valanides, N. (2019). Developing young children’s computational thinking with educational robotics: An interaction effect between gender and
scaffolding strategy. Computers in Human Behavior. https://doi.org/10.1016/j.chb.2019.03.018.
Asad, K., Tibi, M., & Raiyn, J. (2016). Primary school pupils’ attitudes toward learning programming through visual interactive environments. World Journal of
Education, 6(5), 20–26.
Bagley, S., & Rabin, J. M. (2016). Students’ use of computational thinking in linear algebra. International Journal of Research in Undergraduate Mathematics Education, 2
(1), 83–104. https://doi.org/10.1007/s40753-015-0022-x.
Basawapatna, A. (2016). Alexander meets michotte: A simulation tool based on pattern programming and phenomenology. Educational Technology & Society, 19(1),
277–291.
Basogain, X., Olabe, M.A., � Olabe, J. C., & Rico, M. J. (2018). Computational thinking in pre-university blended learning classrooms. Computers in Human Behavior, 80,
412–419. https://doi.org/10.1016/j.chb.2017.04.058.
Basu, S., Biswas, G., & Kinnebrew, J. S. (2017). Learner modeling for adaptive scaffolding in a Computational Thinking-based science learning environment. User
Modeling and User-Adapted Interaction, 27(1), 5–53. https://doi.org/10.1007/s11257-017-9187-0.
Benakli, N., Kostadinov, B., Satyanarayana, A., & Singh, S. (2017). Introducing computational thinking through hands-on projects using R with applications to
calculus, probability and data analysis. International Journal of Mathematical Education in Science & Technology, 48(3), 393–427. https://doi.org/10.1080/
0020739X.2016.1254296.
Berland, M., & Lee, V. R. (2011). Collaborative strategic board games as a site for distributed computational thinking. International Journal of Game-Based Learning, 1
(2), 65–81. https://doi.org/10.4018/ijgbl.2011040105.
20
Bers, M. U. (2010). The TangibleK robotics program: Applied computational thinking for young children. Early Childhood Research & Practice, 12(2), 1–20.
Bower, M., Wood, L. N., Lai, J. W. M., Howe, C., Lister, R., Mason, R., et al. (2017). Improving the computational thinking pedagogical capabilities of school teachers.
Australian Journal of Teacher Education, 42(3), 53–72.
Brady, C., Orton, K., Weintrop, D., Anton, G., Rodriguez, S., & Wilensky, U. (2017). All roads lead to computing: Making, participatory simulations, and social
computing as pathways to computer science. IEEE Transactions on Education, 60(1), 59–66.
Bucher, T. (2017). ‘Machines don’t have instincts’: Articulating the computational in journalism. New Media & Society, 19(6), 918–933. https://doi.org/10.1177/
1461444815624182.
Burleson, W. S., Harlow, D. B., Nilsen, K. J., Perlin, K., Freed, N., Jensen, C. N., … Muldner, K. (2018). Active learning environments with robotic tangibles: Children’s
physical and virtual spatial programming experiences. IEEE Transactions on Learning Technologies, 11(1), 96–106.
Cetin, I., & Andrews-Larson, C. (2016). Learning sorting algorithms through visualization construction. Computer Science Education, 26(1), 27–43. https://doi.org/
10.1080/08993408.2016.1160664.
Chang, Y.-H., & Peterson, L. (2018). Pre-service teachers’ perceptions of computational thinking. Journal of Technology and Teacher Education, 26(3), 353–374.
Chen, G., Shen, J., Barth-Cohen, L., Jiang, S., Huang, X., & Eltoukhy, M. (2017). Assessing elementary students’ computational thinking in everyday reasoning and
robotics programming. Computers & Education, 109, 162–175. https://doi.org/10.1016/j.compedu.2017.03.001.
Choi, J., Lee, Y., & Lee, E. (2017). Puzzle based algorithm learning for cultivating computational thinking. Wireless Personal Communications, 93(1), 131–145.
Csernoch, M., Bir� o, P., M�
ath, J., & Abari, K. (2015). Testing algorithmic skills in traditional and non-traditional programming environments. Informatics in Education,
14(2), 175.
Denner, J., Werner, L., Campe, S., & Ortiz, E. (2014). Pair programming: Under what conditions is it advantageous for middle school students? Journal of Research on
Technology in Education, 46(3), 277–296. https://doi.org/10.1080/15391523.2014.888272.
Dolgopolovas, V., Jevsikova, T., Dagiene, V., & Savulioniene, L. (2016). Exploration of computational thinking of software engineering novice students based on
solving computer science tasks. International Journal of Engineering Education, 32(3), 1–10.
Durak, H. Y., & Saritepeci, M. (2018b). Analysis of the relation between computational thinking skills and various variables with the structural equation model.
Computers & Education, 116, 191–202. https://doi.org/10.1016/j.compedu.2017.09.004.
Flanigan, A. E., Peteranetz, M. S., Shell, D. F., & Soh, L.-K. (2017b). Implicit intelligence beliefs of computer science students: Exploring change across the semester.
Contemporary Educational Psychology, 48, 179–196. https://doi.org/10.1016/j.cedpsych.2016.10.003.
Gadanidis, G., Clements, E., & Yiu, C. (2018). Group theory, computational thinking, and young mathematicians. Mathematical Thinking and Learning: International
Journal, 20(1), 32–53. https://doi.org/10.1080/10986065.2018.1403542.
Gandolfi, E. (2018). You have got a (different) friend in me: Asymmetrical roles in gaming as potential ambassadors of computational and cooperative thinking. E-
Learning and Digital Media, 15(3), 128–145. https://doi.org/10.1177/2042753018757757.
Günbatar, M. S., & Bakirci, H. (2019). STEM teaching intention and computational thinking skills of pre-service teachers. Education and Information Technologies, 24
(2), 1615–1629.
Hershkovitz, A., Sitman, R., Israel-Fishelson, R., Eguíluz, A., Garaizar, P., & Guenaga, M. (2019). Creativity in the acquisition of computational thinking. Interactive
Learning Environments, 27(5–6), 628–644. https://doi.org/10.1080/10494820.2019.1610451.
Hestness, E., Jass Ketelhut, D., McGinnis, J. R., & Plane, J. (2018). Professional knowledge building within an elementary teacher professional development
experience on computational thinking in science education. Journal of Technology and Teacher Education, 26(3), 411–435.
Hsiao, H.-S., Lin, Y.-W., Lin, K.-Y., Lin, C.-Y., Chen, J.-H., & Chen, J.-C. (2019). Using robot-based practices to develop an activity that incorporated the 6e model to
improve elementary school students’ learning performances. Interactive Learning Environments. https://doi.org/10.1080/10494820.2019.1636090.
Izu, C., Mirolo, C., Settle, A., Mannila, L., & Stupuriene, G. (2017). Exploring Bebras tasks content and performance: A multinational study. Informatics in Education, 16
(1), 39–59.
Jaipal-Jamani, K., & Angeli, C. (2017b). Effect of robotics on elementary preservice teachers’ self-efficacy, science learning, and computational thinking. Journal of
Science Education and Technology, 26(2), 175–192. https://doi.org/10.1007/s10956-016-9663-z.
Jenkins, C. W. (2017). Classroom talk and computational thinking. International Journal of Computer Sciences and Engineering Systems, 1(4), n4.
Jun, S., Han, S., Kim, H., & Lee, W. (2014). Assessing the computational literacy of elementary students on a national level in Korea. Educational Assessment, Evaluation
and Accountability, 26(4), 319–332. https://doi.org/10.1007/s11092-013-9185-7.
Kale, U., Akcaoglu, M., Cullen, T., & Goh, D. (2018). Contextual factors influencing access to teaching computational thinking. Computers in the Schools, 35(2), 69–87.
https://doi.org/10.1080/07380569.2018.1462630.
Kong, S.-C., Chiu, M. M., & Lai, M. (2018). A study of primary school students’ interest, collaboration attitude, and programming empowerment in computational
thinking education. Computers & Education, 127, 178–189. https://doi.org/10.1016/j.compedu.2018.08.026.
Korkmaz, O.,€ & Bai, X. (2019). Adapting computational thinking scale (CTS) for Chinese high school students and their thinking scale skills level. Online Submission, 6,
6.
Korkmaz, O.,€ Çakir, R., & Ozden,
€ M. Y. (2017b). A validity and reliability study of the computational thinking scales (CTS). Computers in Human Behavior, 72, 558–569.
https://doi.org/10.1016/j.chb.2017.01.005.
Korucu, A. T., Gencturk, A. T., & Gundogdu, M. M. (2017). Examination of the computational thinking skills of students. Journal of Learning and Teaching in Digital Age,
2(1), 11–19.
Kukul, V., & Karatas, S. (2019). Computational thinking self-efficacy scale: Development, validity and reliability. Informatics in Education, 18(1), 151–164.
Lachney, M., Babbitt, W., Bennett, A., & Eglash, R. (2019). Generative computing: African-american cosmetology as a link between computing education and
community wealth. Interactive Learning Environments. https://doi.org/10.1080/10494820.2019.1636087.
Lai, Y.-H., Chen, S.-Y., Lai, C.-F., Chang, Y.-C., & Su, Y.-S. (2019). Study on enhancing aiot computational thinking skills by plot image-based vr. Interactive Learning
Environments. https://doi.org/10.1080/10494820.2019.1580750.
Leonard, J., Buss, A., Gamboa, R., Mitchell, M., Fashola, O. S., Hubert, T., et al. (2017). Using robotics and game design to enhance children’s self-efficacy, STEM
attitudes, and computational thinking skills. Journal of Science Education and Technology, 25(6), 860–876. https://doi.org/10.1007/s10956-016-9628-2.
Leonard, J., Mitchell, M., Barnes-Johnson, J., Unertl, A., Outka-Hill, J., Robinson, R., et al. (2018). Preparing teachers to engage rural students in computational
thinking through robotics, game design, and culturally responsive teaching. Journal of Teacher Education, 69(4), 386–407. https://doi.org/10.1177/
0022487117732317.
Liu, Z., Zhi, R., Hicks, A., & Barnes, T. (2017). Understanding problem solving behavior of 6-8 graders in a debugging game. Computer Science Education, 27(1), 1–29.
Lui, D., Walker, J. T., Hanna, S., Kafai, Y. B., Fields, D., & Jayathirtha, G. (2019). Communicating computational concepts and practices within high school students’
portfolios of making electronic textiles. Interactive Learning Environments. https://doi.org/10.1080/10494820.2019.1612446.
Marcelino, M. J., Pessoa, T., Vieira, C., Salvador, T., & Mendes, A. J. (2018). Learning computational thinking and scratch at distance. Computers in Human Behavior,
80, 470–477. https://doi.org/10.1016/j.chb.2017.09.025.
Moreno-Le� on, J., Robles, G., & Rom� an-Gonz�alez, M. (2017). Towards data-driven learning paths to develop computational thinking with scratch. IEEE Transactions on
Emerging Topics in Computing.
Mouza, C., Marzocchi, A., Pan, Y.-C., & Pollock, L. (2016). Development, implementation, and outcomes of an equitable computer science after-school program:
Findings from middle-school students. Journal of Research on Technology in Education, 48(2), 84–104. https://doi.org/10.1080/15391523.2016.1146561.
Mouza, C., Yang, H., Pan, Y.-C., Ozden, S. Y., & Pollock, L. (2017). Resetting educational technology coursework for pre-service teachers: A computational thinking
approach to the development of technological pedagogical content knowledge (TPACK). Australasian Journal of Educational Technology, 33(3), 61–76.
Pala, F. K., & Mıhçı Türker, P. (2019). The effects of different programming trainings on the computational thinking skills. Interactive Learning Environments. https://
doi.org/10.1080/10494820.2019.1635495.
Peel, A., Sadler, T. D., & Friedrichsen, P. (2019b). Learning natural selection through computational thinking: Unplugged design of algorithmic explanations. Journal
of Research in Science Teaching. https://doi.org/10.1002/tea.21545.
21
Pellas, N., & Peroutseas, E. (2016). Gaming in second life via Scratch4SL: Engaging high school students in programming courses. Journal of Educational Computing
Research, 54(1), 108–143. https://doi.org/10.1177/0735633115612785.
Pinkard, N., Martin, C. K., & Erete, S. (2019). Equitable approaches: Opportunities for computational thinking with emphasis on creative production and connections to
community. Interactive Learning Environments. https://doi.org/10.1080/10494820.2019.1636070.
Rodríguez-Martínez, J. A., Gonz� alez-Calero, J. A., & S�
aez-L�
opez, J. M. (2019). Computational thinking and mathematics using scratch: An experiment with sixth-grade
students. Interactive Learning Environments. https://doi.org/10.1080/10494820.2019.1612448.
Rom� an-Gonz� alez, M., P�erez-Gonz�alez, J.-C., & Jim�enez-Fern�andez, C. (2017b). Which cognitive abilities underlie computational thinking? Criterion validity of the
computational thinking test. Computers in Human Behavior, 72, 678–691. https://doi.org/10.1016/j.chb.2016.08.047.
Rom� an-Gonz� alez, M., P�erez-Gonz�alez, J.-C., Moreno-Le�on, J., & Robles, G. (2018). Extending the nomological network of computational thinking with non-cognitive
factors. Computers in Human Behavior, 80, 441–459. https://doi.org/10.1016/j.chb.2017.09.030.
Rose, S. P., Habgood, M. P. J., & Jay, T. (2017). An exploration of the role of visual programming tools in the development of young children’s computational
thinking. Electronic Journal of E-Learning, 15(4), 297–309.
S�
aez-L�opez, J.-M., Rom� an-Gonz�alez, M., & V� azquez-Cano, E. (2016). Visual programming languages integrated across the curriculum in elementary school: A two year
case study using “scratch” in five schools. Computers & Education, 97, 129–141. https://doi.org/10.1016/j.compedu.2016.03.003.
Shell, D. F., & Soh, L.-K. (2013b). Profiles of motivated self-regulation in college computer science courses: Differences in major versus required non-major courses.
Journal of Science Education and Technology, 22(6), 899–913. https://doi.org/10.1007/s10956-013-9437-9.
Sherman, M., & Martin, F. (2015). The assessment of mobile computational thinking. Journal of Computing Sciences in Colleges, 30(6), 53–59.
Sung, W., Ahn, J., & Black, J. B. (2017). Introducing computational thinking to young learners: Practicing computational perspectives through embodiment in
mathematics education. Technology, Knowledge and Learning, 22(3), 443–463.
Taylor, K., & Baek, Y. (2019). Grouping matters in computational robotic activities. Computers in Human Behavior, 93, 99–105. https://doi.org/10.1016/j.
chb.2018.12.010.
Thomas, J. O., Rankin, Y., Minor, R., & Sun, L. (2017). Exploring the difficulties African-American middle school girls face enacting computational algorithmic
thinking over three years while designing games for social change. Computer Supported Cooperative Work, 26(4–6), 389–421. https://doi.org/10.1007/s10606-
017-9292-y.
Tran, Y. (2019). Computational thinking equity in elementary classrooms: What third-grade students know and can do. Journal of Educational Computing Research, 57
(1), 3–31.
Tsai, C.-W., Shen, P.-D., Tsai, M.-C., & Chen, W.-Y. (2017). Exploring the effects of web-mediated computational thinking on developing students’ computing skills in a
ubiquitous learning environment. Interactive Learning Environments, 25(6), 762–777. https://doi.org/10.1080/10494820.2016.1181093.
Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., et al. (2016). Defining computational thinking for mathematics and science classrooms. Journal of
Science Education and Technology, 25(1), 127–147. https://doi.org/10.1007/s10956-015-9581-5.
Wilkerson-Jerde. (2014). Construction, categorization, and consensus: Student generated computational artifacts as a context for disciplinary reflection. Educational
Technology Research & Development, 62(1), 99–121.
Wong, G. K.-W., & Cheung, H.-Y. (2018). Exploring children’s perceptions of developing twenty-first century skills through computational thinking and programming.
Interactive Learning Environments. https://doi.org/10.1080/10494820.2018.1534245.
Wu, B., Hu, Y., Ruis, A. R., & Wang, M. (2019). Analysing computational thinking in collaborative programming: A quantitative ethnography approach. Journal of
Computer Assisted Learning. https://doi.org/10.1111/jcal.12348.
Wu, M. L. (2018). Educational game design as gateway for operationalizing computational thinking skills among middle school students. International Education
Studies, 11(4), 15.
Yagci, M. (2019). A valid and reliable tool for examining computational thinking skills. Education and Information Technologies, 24(1), 929–951.
Yildiz, H. D., Yilmaz, F. G. K., & Yilmaz, R. (2017). Examining the relationship between digital game preferences and computational thinking skills. Contemporary
Educational Technology, 8(3), 359–369.
Yuen, T. T., & Robbins, K. A. (2014). A qualitative study of students’ computational thinking skills in a data-driven computing class. ACM Transactions on Computing
Education, 14(4), 1–19. https://doi.org/10.1145/2676660.
Zhong, B., Wang, Q., Chen, J., & Li, Y. (2016). An exploration of Three-Dimensional Integrated Assessment for computational thinking. Journal of Educational
Computing Research, 53(4), 562–590. https://doi.org/10.1177/0735633115608444.
22
View publication stats

Assessing Computational Thinking: A Systematic Review of Empirical Studies

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Assessing Computational Thinking: A Systematic Review of Empirical Studies

Uploaded by

Copyright:

Available Formats

See discussions, stats, and author profiles for this publication at: https://www.researchgate.

Assessing computational thinking: A systematic review of empirical studies

Article in Computers & Education · April 2020

Xiaodan Tang Yue Yin

SEE PROFILE SEE PROFILE

Qiao Lin Roxana Hadad

SEE PROFILE SEE PROFILE

Formative Assessment in a Networked Classroom (FANC) View project

DCA110195 View project

The user has requested enhancement of the downloaded file.

Contents lists available at ScienceDirect

Computers & Education

Assessing computational thinking: A systematic review of

Fig. 1. Different definitions of computational thinking.

1.1. Definition and significance of CT

2.1. Literature search

2019 resulted in another 19 journal articles.

3.1. Educational contexts of the CT assessments

Fig. 2. The literature review process of this study.

Educational level K-elementary 31 28.18%

3.2. Constructs of the CT assessments

3.3. Tools for assessing CT

Single use Traditional 15 15.63%

3.4. Reliability and validity evidence of the CT assessments

6. Conclusion and suggestions

Declaration of competing interest

CRediT authorship contribution statement

Appendix A. The Reviewed Computational Thinking Studies

Authors School level Subject domain Educational Assessment tools

Adler and Kim (2018) Teacher CS/Programming Formal Survey

Kong, Chiu, and Lai (2018) Elementary Formal Survey

Appendix B. Sample Questions for Each Category of Constructs

I. First-order cognitive CT construct

Fig. 1. Task ‘‘Beaver in his canoe’’.

a Fish, frog, crocodile, turtle, stork, snake, otter, duck.

Source: Dolgopolovas et al. (2016).

II. First-order non-cognitive CT construct

1. Computational thinking is understanding how computers work.

17. Knowledge of computing will allow me to improve my performance in my career.

III. Second-order programming/computing/CS construct

Source: Grover et al. (2015).

IV. Second-order non-CS STEM construct

Sources: Pei et al. (2018).

V. Second-order non-STEM construct

Source: Jenkins (2015).

VI. Second-order non-cognitive programming/CS/computing construct

Source: Adler and Kim (2018a, 2018b.

VII. Second-order non-cognitive non-CS construct

1. Understood artistic elements in paintings.

References for Appendix A. The reviewed CT studies

View publication stats

You might also like