See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/359208907

Evaluation and developmental suggestions on undergraduates' computational

thinking: a theoretical framework guided by Marzano's new taxonomy

Article · March 2022

DOI: 10.1080/10494820.2022.2042311

CITATIONS READS

9 613

4 authors:

Lihui Sun Linlin Hu

Minzu University of China Beijing Normal University
14 PUBLICATIONS 301 CITATIONS 9 PUBLICATIONS 263 CITATIONS

SEE PROFILE SEE PROFILE

Danhua Zhou Weipeng Yang

30 PUBLICATIONS 386 CITATIONS

The Education University of Hong Kong
117 PUBLICATIONS 1,665 CITATIONS
SEE PROFILE
SEE PROFILE

All content following this page was uploaded by Lihui Sun on 14 March 2022.

The user has requested enhancement of the downloaded file.

 Interactive Learning Environments

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/nile20

Evaluation and developmental suggestions

on undergraduates’ computational thinking: a
theoretical framework guided by Marzano’s new
taxonomy

Lihui Sun, Linlin Hu, Danhua Zhou & Weipeng Yang

To cite this article: Lihui Sun, Linlin Hu, Danhua Zhou & Weipeng Yang (2022): Evaluation
and developmental suggestions on undergraduates’ computational thinking: a theoretical
framework guided by Marzano’s new taxonomy, Interactive Learning Environments, DOI:
10.1080/10494820.2022.2042311

To link to this article: https://doi.org/10.1080/10494820.2022.2042311

Published online: 13 Mar 2022.

Submit your article to this journal

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=nile20
INTERACTIVE LEARNING ENVIRONMENTS
https://doi.org/10.1080/10494820.2022.2042311

Evaluation and developmental suggestions on undergraduates’

computational thinking: a theoretical framework guided by
Marzano’s new taxonomy
a b c d
Lihui Sun #, Linlin Hu #, Danhua Zhou # and Weipeng Yang #
a b
School of Education, Minzu University of China, Beijing, People’s Republic of China; School of Education, Tianjin
University, Tianjin, People’s Republic of China; cDepartment of Education, Beijing Normal University, Beijing,
People’s Republic of China; dDepartment of Early Childhood Education, Education University of Hong Kong, Hong
Kong, People’s Republic of China

ABSTRACT ARTICLE HISTORY

As a core competence in 21st century, computational thinking (CT) is of Received 8 July 2021
great significance for undergraduates. However, there are few Accepted 7 February 2022
researches on the CT evaluation about undergraduates, and nor do a
KEYWORDS
theoretical framework. Additionally, it is not clear what factors may Computational thinking;
affect their CT development. Therefore, according to Marzano’s new undergraduates; theoretical
taxonomy of educational objectives, this study constructed a theoretical framework; scale validation;
framework for the CT evaluation, which included the field of the CT personal factors; contextual
knowledge and the CT system. Then a Computational thinking scale factors
(CTS) among 737 undergraduates was verified. CTS was adopted to
evaluate undergraduates’ CT skills, and the factors that may affect their
CT skills are discussed according to the framework. The results showed
that: (1) the CTS had sufficient reliability and validity among the sample
undergraduates; (2) the CT skills of the sample undergraduates were in
the middle and upper level (M = 15.71, SD = 2.18); (3) personal factors
(gender and major) and contextual factors (educational environment
and school support) are related to undergraduates’ CT skills. Several
suggestions and countermeasures were put forward for the
undergraduates’ CT cultivation on the framework, which will provide a
reference for CT in undergraduate education.

1. Introduction
Because of the rapid development of information and intelligent technologies and the goal of
twenty-first-century literacy education, computational thinking (CT) has already become an unne-
gligible part of talent training, which has also been incorporated into the educational policies of
many countries over the past decade (Denning & Tedre, 2019; Nouri et al., 2020). As the future
backbone of social constructions, the contemporary undergraduates are attaching great signifi-
cance to the cultivation of their CT skills. However, the current researches on CT are mainly
focused on K-12, while there is a lack of studies on the undergraduates’ CT development
under the context of higher education (Lyon & Magana, 2020). The terminology CT originated
from Papert (1980) and were adopted to describe the thought processes of children interacting
with computers through programming. But since then, unlike Papert, researchers have defined CT
with an emphasis on concepts that are usually involved in programming or part of computer
science (Tikva & Tambouris, 2020). In 2006, Wing (2006) described CT’s a thought process of

CONTACT Lihui Sun slhphd@yeah.net

#
The four authors had equal contributions as principal authors of the manuscript.
© 2022 Informa UK Limited, trading as Taylor & Francis Group
2 L. SUN ET AL.

‘problem solving, designing systems, and understanding human behavior’ for all people. This
definition of Wing pushed CT education to the stage of K-12.
The appropriate methods of enabling K-12 students to better transition to computer science
majors in universities and increasing their willingness to undertake computing-related careers in
the future are the main reasons for the introduction of CT into K-12 schools in some countries
(NRC, 2011; Xing, 2021). But most people have virtually ignored the educational influences of
CT as a thinking skill of undergraduates. Most attentions to computing may be focused on the
undergraduates majoring in computing and engineering, but not so much was paid to consider-
ing the CT skills as a kind of universal thinking. As Wing (2008) said, this field of computing was
in a unique situation of not only being computational concepts to teach but also being a tool to
teach. Therefore, CT should be a thinking skill for all undergraduates in all professional fields, be
embedded into the classes for students of all majors, and become a thinking ability to guide their
daily lives and works.

1.1. The definition of CT

The definition of CT has always been the focus of attention and discussion. Wing (2008) believed that
CT was the thinking process of discovering problems and their solutions. CT absorbs both the math-
ematical and engineering thinking and overlaps with the systematic thinking (Wing, 2006; 2008;
2010). Deriving from this view, Curzon (2015) defined CT as the basic skill of problem-solving,
which can transform a real situational problem into a problem representation. Simultaneously,
Barr and Stephenson (2011) contended that CT involves the problem-solving skills and specific per-
sonalities shown in the solving process, such as confidence and perseverance in face of specific pro-
blems. Similarly, Nouri et al. (2020) demonstrated the general skills related to twenty-first century
including the CT skills and connected the cognitive skills and attitudes, language skills, creative
problem-solving skills, cooperativity skills all to CT.
Other researchers divided CT into different dimensional frameworks to facilitate people better
understanding and practicing CT. For instance, the three-dimensional framework of CT was pro-
posed by Brennan and Resnick (2012) including three dimensions: computational concept
(sequences, loops, events, parallelism, conditionals, operators, and datasets), computational prac-
tices (being incremental and iterative, testing and debugging, reusing, and remixing, abstract-
ing, and modularizing), and computational perspectives (expressing, connecting, and
questioning). Moreover, these definitions usually provided different concepts and skills to
describe CT as a problem-solving process. Anderson (2016) explained the five CT components:
decomposition, pattern recognition, algorithm design of the abstract, and evaluation of the sol-
ution. Summarizing the topics found from each study, the most common elements include
abstraction, algorithms, data, problem decomposition, parallelism, debugging and testing, and
control structures (Rose et al., 2018). To better incorporate CT into the classroom, the Inter-
national Society for Technology in Education (ISTE) provided the operational definition of CT
as a common reflection of creativity, algorithmic thinking, critical thinking, problem-solving,
cooperativity, and communication skills (ISTE, 2015). Literacy of the twenty-first century required
learners to maintain the abilities of critical thinking and problem-solving, communication and
cooperativity, creative innovation, sense of responsibility, and information literacy, which can
be achieved through novel practical CT skills (Grover & Pea, 2013). To sum up, CT was recog-
nized as a thinking ability at comprehensive high-level, which was reflected in its problem-
solving ability and extensions.

1.2. The evaluation of CT skills

It is difficult for CT to be successfully incorporated into the K-12 education system if its measurement
and evaluation are not emphasized (Grover & Pea, 2013). Researchers have developed a variety of CT
 INTERACTIVE LEARNING ENVIRONMENTS 3

evaluation methods, which were mainly reflected in different CT evaluation constructs and types of
evaluation tools (Tang et al., 2020). It has been generally believed that CT can be developed into pro-
gramming activities, thus CT evaluation is also based on a platform or software, through the evalu-
ation of students’ works to measure their CT skills respectively. For example, Sáez-López et al. (2016)
have developed a visual block creative calculation test to separately evaluate the CT skills of primary
school students after the Scratch training. In Alice’s 3D virtual world, students adopted program-
ming to create fairy tales and then were evaluated for their CT by analyzing their digital collections
(Fields et al., 2018). In addition, CT evaluation has also been applied to the process of solving prac-
tical problems. Román-González et al. (2017) constructed a Computational Thinking Test (CTT) to
individually evaluate 12–13-year-old students’ CT skills. Bebras Computing Challenge developed
specific test questions for students at different ages and grades (Bebras-Ireland, 2019). It has also
been an important method to divide CT into different dimensions for evaluating CT by the form
of scale. Tsai et al. (2021) regarded CT as a problem-solving process rather than a result before devel-
oping and verifying a scale to evaluate students’ CT processes under specific problem situations
from five dimensions respectively: abstraction, decomposition, algorithmic thinking, evaluation,
and generalization.

1.3. Evaluation of undergraduate’ CT skills

From reviewing the existing researches on the evaluation of students’ CT skills, the number of
effective tools for evaluating the undergraduates’ CT skills was relatively lacking. At present, the
evaluation of undergraduates’ CT skills has mostly adopted the method of qualitative analysis. For
example, Butler and Leahy (2021) conducted qualitative interviews with 51 pre-service teachers in
higher education to investigate their understanding of CT and how professional design facilitated
to develop their understanding. Pan et al. (2016) adopted the method based on mind mapping
to evaluate students’ CT skills in Photoshop classes at colleges and universities. Only after
Korkmaz et al. (2017) combined the CT definition of ISTE, a computational thinking scale (CTS)
was developed and comprised of creativity, algorithmic thinking, critical thinking, problem
solving, and cooperativity to evaluate the undergraduates’ CT skills, this is one of the main tools
from the perspective of quantitative data analysis. In the framework of CTS for the classification of
CT, creativity drives students to find solutions; algorithmic thinking is an ability necessary for every-
one in the information society; critical thinking is one of the higher skills to understand and express
the ideas of oneself or others. Students’ problem-solving ability is a system composed of many
elements, such as understanding, analysis, reasoning, practice, expression, and so on. Cooperativity
can be defined as the educational use of a team so that students can work together to maximize the
learnings of their own and the others. The purpose of CT education is not to directly help students do
well in the computer field, but to ensure that CT skills can be applied to their courses and daily lives.
During the process of applying CT to solve problems, people have already developed a variety of
high-level thinking skills, which can better explain why CT should be embedded into the classrooms
and learnings of undergraduates in various professional fields as a kind of thinking skill. In this study,
Korkmaz’s CTS scale was adopted to evaluate the CT skills of Chinese undergraduates.

1.4. Predictors of CT skills

Through the summary of the existing literature, it is found that various factors may affect CT in
different ways. The influential factors of CT skills can be discussed separately from the perspectives
of personal factors and contextual factors. First of all, gender is one of the most important factors.
Traditionally, the fields of computer and technology were generally male-dominated, while the inno-
vation of CT enables females to access technology in an interesting and meaningful way, which may
balance the dominance of boys and girls in the field of science and technology (Espino & González,
2016). In this regard, Atmatzidou and Demetriadis (2016) found that females needed more time to
4 L. SUN ET AL.

acquire CT skills at similar levels than males. Some studies have otherwise proved that girls pre-
sented more potential to develop CT skills in an environment irrespective of technical support
(Sun et al., 2020, 2021a). Additionally, most studies believed that it was easier to develop students’
CT skills in computers, mathematics, and science majors, but the other researchers have introduced
CT into English, history, and structured authoring (Evia et al., 2015). Besides, Román-González et al.
(2017) conducted a study to examine the relationship between grade level and CT skill level, and
found that there was a positive correlation between these two. On the contrary, Durak and Saritepeci
(2018) utilized the education level to represent the students’ grades. Through their modeling, it was
found that education level made a negative prediction of CT skills. They believed that this negative
correlation might mean that the CT skills of senior high school students would be negatively affected
by the improvement of their education levels.
The second category is the contextual factors related to the external environments of the stu-
dents. It has been well known that the infrastructure environments of urban schools are generally
better than those of rural schools. Leonard et al. (2018) explored how to train teachers for
helping the rural students participate in CT through robotics, game design, and culturally responsive
teaching, which contributed to the promotion of educational equity and regional cultural differences
on CT. When students received school education, the school support would involve external factors
which could be classified as the environmental factors affecting the students’ development. Based
on the theory of psychological autonomy proposed by Sternberg and Grigorenko (1997), individual
behaviors and activities needed to be organized according to internal and external demands. The
support provided by the external environment might also affect the development of students’ CT
skills to a certain extent. For example, Kale et al. (2018) asserted that school support determined
the teachers’ access to CT skills and CT technology, which in turn affected the development of stu-
dents’ CT skills. Similarly, Lee and Lee (2021) maintained in an informatics course that teacher
support exerted a significant impact on the deep learning methods and could affect the students’
CT skills substantially. To sum up, it can be seen that most of the previous studies were mainly
focused on exploring the influential factors to CT skills of K-12 students, while the influential
factors to undergraduates’ CT skills were not clarified. Drawing lessons from the experiences of
CT teachings in K-12 education and combining with the characteristics of undergraduate education,
this study summarized specific kinds of CT factors that can be divided into two categories: personal
factors (i.e. gender, major, and grade) and contextual factors (i.e. basic educational environment and
school support).

1.5. Theoretical framework of CT evaluation

The purpose of the evaluation is to play a reference and guiding role in future teaching. Marzano’s
new taxonomy of educational objectives was adopted as the theoretical basis for CT evaluation,
which corresponded to the five dimensions of the CTS. Marzano asserted that the human learning
process included three main systems: the self-system, the metacognitive system, and the cognitive
system, plus the factor of knowledge. Based on this, Marzano proposed a two-dimensional evalu-
ation system. The first dimension was knowledge, which contained six types in three different
domains: information (facts, organizational ideas), mental procedures (intellectual skills, intellectual
processes), and physical procedures (psychological skills, psychological processes). The second
dimension was process operation, which included six of three systems. The first to the fourth
ones were the knowledge retrieval, comprehension, analysis, and knowledge utilization, which
belonged to the cognitive system; the fifth one was the metacognitive system; and the sixth one
was the self-system (Marzano, 2001). The self-system was applied to generate the value judgment
for the learning task, the metacognitive system was adopted to generate the problem-solving strat-
egies and methods, and the basic cognitive skills of the cognitive system were utilized to complete
the task. The operation and cooperation of these three systems were inseparable from the support-
ing basis of the knowledge field.
 INTERACTIVE LEARNING ENVIRONMENTS 5

Compared with Bloom’s classification of educational goals, Marzano’s theory is more suitable for
the teaching and evaluation of high-level thinking. First of all, the Bloom framework indicated that
knowledge was a platform without change and knowledge accumulation rose to the thinking level
by a certain extent, so that knowledge contents and thinking operations were actually separated in
isolation. However, CT skills are embodied in the processes of students operating the technical tools
or solving the practical problems during their lives and studies, therefore its knowledge contents and
thinking operations could not be separated from each other. Secondly, Bloom divided the thinking
process into Knowledge, Comprehension, Application, Analysis, Synthesis, and Evaluation with a
single progressively linear relationship, while Marzano explained the relationship between knowl-
edge and thinking system in the two-dimensional space. Thus, there would be no emphasis on
the advanced relationship between thinking systems from low-level to high-level. Considering CT
as a thinking ability at comprehensive high-level, there will be no obviously advanced relationship
among the evaluation dimensions of CT within this study. Finally, the Bloom framework was
described more empirically, and it was difficult for teachers to clearly define it in the actual teachings,
while the teaching feasibility of Marzano theory was much stronger, which made it more convenient
for this study to put forward more targeted guidance based on the results of the CT evaluation
survey (Marzano, 2001; Sheng, 2008).
Each level of Marzano’s new taxonomy of educational objectives identifies a specific type of
thinking, which can teach students the information and skills related to knowledge retrieval, com-
prehension, analysis, knowledge utilization, metacognition, self-systems, etc., for effectively improv-
ing their abilities. Accordingly, a key use of Marzano taxonomy can be constructing a framework for
thinking skills courses, which serves as a foundation for explaining the thinking skills and processes.
Applying CT as high-level thinking, the learning process of learners can be well mapped to the cor-
responding level of the new taxonomy. Based on the classification framework of Marzano’s edu-
cational goals, this study integrated the ISTE’s definition of CT and constructed a theoretical
framework for its evaluation, to provide targeted suggestions and countermeasures for the
follow-up improvements of undergraduates’ CT skills. As shown in Figure 1.
Knowledge domain – computational concepts, computational practices, and computational per-
spectives. This study adopted Brennan and Resnick’s (2012) three-dimensional CT framework (com-
putational concepts, computational practices, and computational perspectives) to effectively fit the
three aspects of the knowledge domain in the new taxonomy of educational objectives (information,
mental procedures, physical procedures), for building an effective framework of its own for under-
standing the CT from the knowledge domain. The field of knowledge is intertwined with the thinking
system, which permeates the processes of solving problems by CT. Among them, the information

Figure 1. Theoretical framework of CT evaluation.

6 L. SUN ET AL.

field tends to focus on declarative knowledge about “what is”, which matches the definition of the
CT concepts from the three-dimensional framework. Students need to integrate the fundamental CT
concepts, such as sequences, loops, conditions, etc., so that they can extract them at any time from
the subsequent psychological operations. The mental procedures are mainly comprised of pro-
cedural knowledge about “how to do”, which are consistent with the definition of CT practice
and representing the one that students develop in programming. For example, they can be abstract-
ing a real problem from real life into another one that can be solved programmatically, before con-
stantly debugging and iterating until the problem is finally solved. The psychomotor procedures
involve one of physical actions, which also points to the definition of CT perspectives, i.e. it is man-
ifested by the integration of knowledge concepts and mental action processes. Understanding the
interaction between skills and strategies through CT concepts and practices during the process of
problem-solving will facilitate to continuously develop a perspective of solving the real problems
within oneself and the world (Marzano, 2001; Resnick, 1987).
Meanwhile, the processing level is divided by the new taxonomy of educational objectives, and
the cognitive metacognitive, and selfsystems constitute the three systems of thinking, which interact
with the three fields of knowledge in different ways. Moreover, CTS divided CT into five sub-skills:
problem-solving, algorithmic thinking, critical thinking, creativity, and cooperation (Korkmaz et al.,
2017). These sub-skills can also correspond to the three systems of thinking in the new taxonomy:
Cognitive system – problem-solving and algorithmic thinking. The cognitive system consists of
four levels including the extraction, understanding, analysis, and application of the information in
turn. The basis of CT is reflected during the process of programmed problem-solving. Based on
the description of problem-solving within the operational definition of ISTE (2015), its process was
also described as the following four levels: Organization and Analysis, Abstraction and Represen-
tation, Programming and Iteration, Migration and Transformation. The CT cognitive system points
to the “conceptual dimension” of the basic CT cognition and skill application. Algorithmic thinking
is related to the concepts of creating and processing algorithms, while the creation of algorithms is
mainly a human activity (Katai, 2014). Algorithmic thinking drives students to identify, integrate, and
examine the CT knowledge, before producing a new understanding of CT and then applying the
problem-solving ability to the specific problem situations. When facing a newly uncertain
problem, the students can concentrate it into a concrete and solvable form, and create a reasonable
way to solve it based on the existing facts. CT is an extension of the individual problem-solving ability
(Yagci, 2019). Therefore, this study corresponds to the cognitive system in two dimensions: problem-
solving and algorithmic thinking.
Metacognitive System – Critical Thinking. The main function of the metacognitive system is to
determine the goal and to monitor both the accuracy and clarity of the thinking process (Marzano &
Kendall, 2007). It refers to the formulation, execution, and self-judgment of goals accomplished by
specific types of knowledge (Marzano, 1998). The CT metacognitive system mainly focuses on the
“process dimension” of problem solutions and strategies generated by learners from CT. Through
critical thinking, students can complete the evaluation, self-reflection, and judgment respectively
of a specific field of CT, and eventually improve the entire process (Dam & Volman, 2004). Critical
thinking in CT is embodied into the tolerance of ambiguity and the judgment of various ways for
solving problems. Therefore, this study correspond with critical thinking to the metacognitive
system.
Self-system— – Creativity and Cooperativity. Self-system contains beliefs about the impor-
tance of knowledge, the efficacy, and the emotions related to knowledge. The self-system includes
the arrangements of attitudes, beliefs, and emotions respectively. The CT self-system and the internal
driver of applying CT to address the real-life problems and the “ideological dimension” related to the
construction of thinking. The two dimensions of creativity and cooperativity in CT are reflected by
the confidence in dealing with complex problems, the perseverance in solving difficult ones, and
the communication and cooperativity with others to achieve common goals and solutions. Creativity
is understood as the capacity of an individual to form a new idea, insight, invention, or artistic object
 INTERACTIVE LEARNING ENVIRONMENTS 7

of social, spiritual, esthetic, scientific, or technological values, which is a characteristic feature of each
individual at different levels, which affects the students’ understanding of CT knowledge (Plucker
et al., 2004). Furthermore, the cooperativity can improve the students’ understandings or abilities
of cooperative problem-solving, while the individuals cooperating with different skills for a deliber-
ate purpose of solving complex problems are inevitable in the twenty-first century (Korkmaz et al.,
2017). Therefore, this study corresponds with the two dimensions of creativity and cooperativity to
the self-system.

1.6. Research objectives

This study takes Mazano’s new taxonomy of educational objectives as the theoretical framework, in
order to provide corresponding countermeasures and suggestions for the evaluations and develop-
ments of undergraduates’ CT skills. However, within the current CT evaluation researches, there has
been a considerable lack of effective criteria for undergraduates’ CT evaluations, and the factors of
undergraduates related to their CT skills have not been fully clarified, which are the questions that
this study is committed to solving. The framework of research design is shown in Figure 2. The theor-
etical contribution of this study is to provide a framework for CT evaluation. The practical significance
lies in the practical applications of CTS based on this provided evaluation framework.
This study proposed the following hypotheses:
Hypothesis 1: The Chinese version of CTS owns sufficient reliability and validity in the samples of
undergraduates.

Hypothesis 2: Personal factors (i.e. gender, major, and grade) are related to the undergraduates’ CT skills.

Hypothesis 3: Contextual factors (i.e. basic educational environment and school support) are related to the
undergraduates’ CT skills.

Figure 2. Research and design framework.

8 L. SUN ET AL.

2. Methods
2.1. Sample
As a universal thinking skill, CT exists in all aspects of study and life, which is reflected during the
thinking process of programmed problem-solving. Everyone owns and has applied CT to solve pro-
blems, even sometimes unconsciously. Similarly, the purpose of CT evaluation is not to require the
students devoting themselves directly to the study of computer science or programming, but to
implement CT into the students’ courses and daily activities to guide the solutions of real-life pro-
blems, so that promote the developments of CT in undergraduate education. The undergraduates
in the survey sample all have computer learning experience in K-12 education and undergraduate
education, while CT is also reflected by the study of basic courses at schools (such as mathematics,
science). In this study, the survey purpose is to understand the general level of undergraduates’ CT
skills and their influential factors, rather than to test whether undergraduates’ CT skills level has been
developed through the teaching interventions.
This study initially adopted the method of sampling survey, randomly distributed the question-
naire online, and carefully calculated the CT scale to collect data. A total of 760 questionnaires
were collected. Among them, 23 were recognized as invalid due to void values and 737 were recog-
nized as valid, indicating an effective rate of 96.97%, while the samples were from 133 universities in
26 provinces and cities in China. The survey period ranged from May 2020 to October 2020. More-
over, 187 (25.4%) were filled by male participants, and 550 (74.6%) were filled by females. Out of all
the respondents, 161 (21.85%) were freshmen, 187 (25.37%) were sophomores, 227 (30.8%) were
juniors, 136 (18.45%) were seniors, and 26 (3.53%) were the 5th-grade students (such as the
medical students) respectively. When verifying the CTS among the Chinese undergraduates, the
sample (N = 737) was randomly divided into two different groups of Exploratory Factor Analysis
(EFA) and Confirmatory Factor Analysis (CFA) respectively. Details about the respondents were
given in Table 1.

2.2. Measures
The Chinese Version of Computational Thinking Scales. The CTS of Korkmaz et al. (2017) were
translated from English into Chinese after obtaining permission from the developer, which consisted
of 29 items based on its five factors (i.e. creativity, algorithmic thinking, cooperativity, problem-
solving, and critical thinking). Items were rated by a 5-point Likert scale (from 1 = strongly disagree
to 5 = strongly agree). To ensure the accuracy of the translation, this study followed the back-trans-
lation method. It requires the translated target language to be re-translated into the original text
which would be compared with the back-translation text to determine whether there is a
problem within the translated text (Yu & Richardson, 2015). Two experts in educational technology
proficient in both English and Chinese were arranged to provide the translation. First, one of the

Table 1. The distribution of the universities, grades, regions, and genders of the sample.
Gender Region University
County First-class First-class discipline General
Grade Male Female City seat Town Rural university university university
Freshman 53 108 60 40 13 48 72 44 45
Sophomore 52 135 45 38 12 92 23 45 119
Junior 36 191 57 29 18 123 21 11 195
Senior 35 101 38 22 13 63 3 24 109
5th-grade 11 15 5 3 0 18 6 7 13
Total 187 550 205 132 56 344 125 131 481
Proportion 25.4 74.6 27.8 17.9 7.6 46.7 16.9 17.8 65.3
%
 INTERACTIVE LEARNING ENVIRONMENTS 9

experts was asked to translate the English version of CTS into Chinese. Next, the other expert was
asked to translate the Chinese version of CTS back into English. Finally, this study compared the
English version of back-translated CTS with the original English version of CTS and found that
their linguistic descriptions were almost the same.
To improve the scale being more consistent with the CT evaluation framework constructed after
the translation, this study appropriately modified the item according to the CT knowledge field in the
framework without changing the connotations of the original scale, in order to match the specific CT
requirements in the framework of those three thinking systems. For example, one of the items in the
critical thinking dimension of the CT metacognitive system as “I am proud of being able to think with
a great precision” was adapted into “I am proud of being able to accurately design and consider the
planning and implementation of each step”. This study transformed the phrase of “think with a great
precision” into a more detailed one of “accurately design and consider the planning and implemen-
tation of each step”, in which the design, planning, and implementation refer to the specific CT prac-
tices in the field of CT knowledge. Given the differences under the cultural contexts and participants’
backgrounds, it remained necessary to verify the psychometric properties in the Chinese context and
make essential amendments.
Demographic Information Survey. In this study, the background information of the students
was collected also through a questionnaire survey. Studying the influence of these variables on
CT skills is of great significance for countries with imbalanced development of educational
environments. First of all, personal factors such as gender, major, and grade can reflect the
inherent characteristics of undergraduates. The students’ gender (boy = 1, girl = 2) and grade
(freshman = 1, sophomore = 2, junior = 3, senior = 4, 5th-grade = 5) were selected by the way of
specific questions. Students’ majors were selected according to the discipline categories in
which they specialize, following the revised catalog of academic degrees and talent training
issued by the Ministry of Education of the People’s Republic of China (2011). The catalog is
divided into 14 disciplines of philosophy, economics, law, pedagogy, literature, history, science,
engineering, agronomy, medicine, military science, management, art, and the interdisciplinary cur-
ricula, numbered 1–14, respectively. In addition, this study chose the basic education environment
and school support as the background factors to explore their relationship with CT skills. The basic
educational environment represented the students’ different environments at the basic education
stage under the Chinese education context. It can be divided into the rural environment = 1, the
township environment = 2, the county environment = 3, and the urban environment = 4, respect-
ively. Students can choose any of these according to their actual situations. School support indi-
cates the quality level of education received by students at different school levels. Universities
at different levels presented differences in the student quality, faculty resources, employment pro-
spects, and else. In this regard, they were divided into the first-class universities = 1, the first-class
disciplinary universities = 2, and the general universities = 3. Students can choose according to
their university levels.

2.3. Data analyses

Firstly, the suitability of this data analysis was tested by the assumption of normality and the deter-
mination of missing data. Secondly, Exploratory Factor Analysis (EFA) and CFA were conducted cor-
respondingly with two separate sets of data. According to the rule that the ratio of the sample size to
variables should be over 5:1 (Pompili et al., 2007), the sample (N = 737) was randomly divided into
two groups: one group (N = 362) for EFA which determined the construct validity of the CTS, and
the other group (N = 375) for CFA to test the model fitness in the current sample. Cronbach’s
alpha reliability coefficient and Construct Reliability (CR) were also computed to assess the reliability
of the whole scale and its factors. Thirdly, this study conducted the preliminary analyses, including
tests for mean differences through the independent sample t-test and one-way Analysis of Variance
(ANOVA). This study also assessed the correlations between the five variables (i.e. gender, majors,
10 L. SUN ET AL.

grade, basic educational environment, and school support) and CT as well as its own five factors.
Because the independent and dependent variables involve the continuous and discontinuous
ones, Spearman’s rank correlation coefficient was selected for assessing the correlation between
the independent and dependent variables, while Pearson product-moment correlation coefficient
was selected for analyzing and calculating the correlation between CT and its factors. Lastly, this
study performed a hierarchical regression analysis to examine whether the five influential factors
could predict CT skills. Gender, majors, grade, basic educational environment, and school support
were entered respectively as the predictor variables. IBM SPSS 26.0 and AMOS 26.0 were both
adopted in these data analyses.

3. Results
3.1. Validity of the CTS
To determine the validity of the CTS scale, EFA and CFA were analyzed, respectively.
Exploratory factor analysis. The EFA was conducted on the first sample (N = 362). The condition
of separating the scale into the factors was determined by the basic component analysis, and the
KMO value of the scale was found to be 0.856 (>0.80), whereas Bartlett’s value was χ 2 (362) =
2648.348, DF = 406, p < 0.000, indicating the scale was suitable for EFA. Five factors with the eigen-
values over 1.50 were found as the result via the Varimax vertical rotation technique. This was con-
sistent with the CTS of Korkmaz et al. (2017), in which the five factors corresponded to creativity (C),
algorithmic thinking (A), cooperativity (O), problem-solving (P), and critical thinking (T), respectively.
In this study, the factor loading cutoff was 0.5 for the choice of each item. Based on the results of the
factor loading calculations for five items, two of them were with factor loadings below 0.5, and three
ones with factor loadings divided into the two factors were removed from the 29-item scale. As
shown in Figure 3, the factor loadings of the items remained in the analysis were all clustered
under the five factors. They were found to range between 0.518 and 0.840 for each item. The five
factors jointly explained the variance of 57.549%. A scale consisting of 24 items and 5 factors was
made available as a result.
Confirmatory factor analysis. The CFA was conducted on the other sub-sample of 375 under-
graduates to validate the five-factor construct of the scale. As shown in Figure 4, the factor
loading varies within a range of 0.33–0.73 for the creativity factor, 0.61–0.83 for the algorithmic think-
ing factor; 0.76–0.91 for the cooperativity factor, 0.57–0.68 for the problem-solving factor, and 0.73–
0.80 for the critical thinking factor, respectively, indicating that all factors in the Chinese version of
CTS are both convergent and valid (Edwards, 2010).
The values of the goodness of fit were also examined separately: χ 2/df = 2.242, RMSEA = 0.050,
GFI = 0.927, IFI = 0.948, CFI = 0.947, TLI = 0.938. According to the acceptance criteria of fit index pro-
posed by Kline (2005), 0 < χ 2/df < 3 RMSEA < 0.05 belonged to the good range of acceptance, and
0.90 < GFI < 0.95 IFI < 0.95 IFI < 0.90 < CFI < 0.97 0.90 < TLI < 0.95 was also acceptable. It can be
seen from Table 2 that the values of goodness of fit obtained by CTA are within the acceptable
range, indicating that the Chinese version of CTS has achieved the good effectiveness of
convergence.

Table 2. The fitness index values obtained from the CFA.

Examined Fitness Index Perfect Fitness Acceptable Fitness CFA Fitness Index
χ 2/df 0 < χ 2/df <3 3 < χ 2/df < 5 2.242
RMSEA 0 < RMSEA <0.05 0.06 < RMSEA <0.08 0.050
GFI 0.95 < GFI<1 0.90 < GFI < 0.95 0.927
IFI 0.95 < IFI<1 0.90 < IFI < 0.95 0.948
CFI 0.97 < CFI<1 0.90 < CFI < 0.97 0.947
TLI 0.95 < TLI<1 0.90 < TLI < 0.95 0.938
 INTERACTIVE LEARNING ENVIRONMENTS 11

Figure 3. Factor analysis results of the scale as per factors.

3.2. Reliability of the CTS

Internal consistency level. The results showed that the Cronbach’s alpha reliability coefficient of the
scale was 0.852 (greater than 0.80), which indicated that the Chinese version of CTS realized a high
internal consistency. The coefficients of each factor were: creativity (0.846), algorithmic thinking
(0.821), cooperativity (0.892), problem-solving (0.820), and critical thinking (0.725), respectively.
According to DeVellis (1991), when the Cronbach’s alpha reliability coefficient reached 0.7–0.8, it
would indicate that the scale owned a considerable reliability; when it reached 0.8–0.9, it would indi-
cate that the reliability of the scale was significantly good. The results revealed that the Chinese
version of CTS verified in this study possessed a high internal consistency.
12 L. SUN ET AL.

Figure 4. Confirmatory factor analysis diagram of the scale.

Combination reliability. The combination reliability (CR) of each factor was calculated by
Formula 1. Results showed that the CR of the creativity factor was 0.594, the algorithmic thinking
factor’s CR was 0.848, the cooperativity factor’s CR was 0.885, the problem-solving factor’s CR
was 0.652, and the CR of the critical thinking factor was 0.839, respectively. Except for the low CR
of the creativity factor, those of other factors were all higher than 0.6, indicating that most factors
maintained a high constancy level.
 INTERACTIVE LEARNING ENVIRONMENTS 13

Formula 1. CR calculation formula

(Sl)2
CR =
[(Sl)2 + Su]
λ = Factor loading Θ = measurement error
In summary, the revised Chinese version of CTS based on the five-factor CTS of Korkmaz et al.
(2017) presented a clear factor structure to be verified by the EFA, which included 5 factors and
24 items. The values showed that the revised CTS also maintained high levels of reliability and val-
idity, which could be adopted to evaluate the CT skills of Chinese undergraduates effectively. Hereby,
Hypothesis 1 was supported.

3.3. Descriptive analyses

This study conducted an independent sample t-test and a one-way Analysis of Variance (ANOVA) to
analyze the mean differences in five factors of CT skills among 737 Chinese undergraduates. The
results were shown in Table 3.
The minimum score of undergraduates’ CT skills (out of 25) was 6.9, the highest score was 24.14,
and the average was 15.71, respectively. It could be concluded that the overall CT skills of Chinese
undergraduates were concentrated in the upper-middle range. For each factor of CT (out of 5), the
students’ scores were: problem-solving (3.27), cooperativity (3.25), creativity (3.20), critical thinking
(3.19), and algorithmic thinking (2.78).
As shown in Table 4, there was a significant difference in CT skills between the female and male
students (t = 2.755, p = 0.005), while males’ CT skills (M = 16.08, SD = 2.38) were stronger than that of
females (M = 15.58, SD = 2.06). There were also significant differences in CT skills among the basic
educational environment (F = 2.573, p = 0.018), majors (F = 2.927, p = 0.001), and university supports
(F = 4.270, p = 0.014). However, the difference between the grades was not significant (F = 0.184, p =
0.947). These results indicated that the CT skills of Chinese undergraduates differ with the four influ-
ential variables.

3.4. Correlational analyses

The results of the correlational analysis are listed in Table 4. They indicated significant correlations
between gender, major, and CT skills. And the relationships between the basic educational environ-
ment, school support, and CT skills were also positively significant. However, the correlation between
the grade and CT skills was found to be insignificant. Based on the criteria of Gignac and Szodorai
(2016) in evaluating the magnitude of correlational coefficients, the levels of the correlations
found between the five prediction factors and CT factors indexes ranged from the small (r =
0.015) to the medium (r = 0.132). These effect sizes indicated that the gender, major, basic edu-
cational environment, and school support all presented small to medium extent of associations
with various factors of CT, respectively.

Table 3. Descriptive statistics of the variables (N = 737).

Gender Grade Major Basic education environment School support
Factors Max Min M SD V t F
C 1.57 4.43 3.20 0.45 0.20 1.366 0.914 1.986* 3.372* 2.210
A 1.00 5.00 2.78 0.68 0.46 3.283** 0.187 3.076** 0.723 7.246**
O 1.00 5.00 3.25 0.82 0.68 0.524 0.363 3.433** 0.850 0.157
T 1.00 5.00 3.19 0.74 0.54 2.346** 0.752 2.590** 0.695 4.808**
P 1.00 5.00 3.27 0.69 0.47 1.331 0.828 1.438 3.308* 0.846
CT 6.90 24.14 15.71 2.18 4.78 2.755** 0.184 2.927** 2.573* 4.270*
Note: * = p < 0.05, ** = p < 0.01, N = number of samples, M = mean, SD = Standard deviation, V = variance.
 14
L. SUN ET AL.
Table 4. Correlation coefficients among the variables.
Factors Gender Grade Major Basic education environment School support C A O T P CT
Gender – .054 .043 −.131** .274** −.067 −.107** −.036 −.103** −.021 −.095**
Grade – .138** .062 .367** 0.21 0.030 .016 .026 .046 .015
Major -– .076* .220** −.046* −.033* −.038* −.030* −.036 −.055*
Basic education environment – −.003 .093* .041 .049 .037 .087* .085*
School support – .066 .132** .013 .095** .018 .089*
C – .465** .473** .642** .196** .719**
A – .322** .638** .138** .728**
O – .533** .082* .756**
T – .081* .864**
P – .112**
CT –
Note: * = p < 0.05, ** = p < 0.01.
 INTERACTIVE LEARNING ENVIRONMENTS 15

Table 5. The relationship between study grade and CT in different groups.

Gender School support Basic education environment
First-class First-class discipline General County
Male Female university university university City seat Town Rural
Study grade
C −.118 −.011 −.108 .006 −.007 .004 −.131 −.180 −.013
A −.043 .030 −.123 .072 .098 .063 −.132 .139 .035
O −.068 .005 −.221* .071 .003 .027 .005 −.169 −.018
T −.094 −.017 −.152 .037 .020 −.013 −.017 −.026 .011
P .099 .038 .222* .048 .022 .029 −.026 .080 .120
CT −.071 .013 −.173* .076 .044 .033 −.145 −.040 .034
Note: * = p < 0.05.

In the general sample, there was no significant correlation between the students’ grades and CT.
To further explore this, the specific samples of different groups were selected according to their
characteristics (such as gender, basic educational environment, and school support) to further
analyze the relationship between the grades and CT skills of undergraduates in China. As shown
in Table 5, there was no significant correlation between the grades and CT skills of undergraduates
of different genders or from different basic educational environments. However, when considering
different types of school supports, there was a significant correlation between the grades and CT
skills of students with first-class university support, especially in terms of cooperativity and
problem-solving ability, while the correlations between these two variables of the student groups
with other types of school supports were also found to be insignificant.

3.5. Hierarchical regression analyses

To further explore the prediction function between various predictors and CT skills, the entry method
was adopted for the regression analysis. As shown in Table 6, the results partially confirmed Hypoth-
esis 2, while gender (β = -−0.493, t = −2.787, p < 0.01) and major (β = −0.098, t = −2.011, p < 0.05) can
both significantly predict the undergraduates’ CT skills; however, there was no significant relation-
ship between CT skills and the grade either. Hypothesis 3 was supported in this research as basic
educational environment (β = 0.710, t = 2.314, p < 0.05) and school support (β = 0.242, t = 2.409, p
< 0.05) could both predict CT skills positively.
Moreover, the Step 2 Regression Analysis was performed according to the correlation between
the five predictors found in correlation analysis and the multiple factors of CT, the results from
the hypothesized connection of the five predictor variables with the five factors of CT skills were

Table 6. Regression analyses of five influencing factors as predictors of CT skills.

Dependent variable Predictor variables Β T R2 ΔR 2
Step 1 CT Gender −.493 −2.787** 0.010 0.009
Grade .017 0.227 − −
Major −.098 −2.011* 0.016 0.013
Basic education environment .071 2.314* 0.007 0.006
School support .242 2.409* 0.008 0.006
Step 2 C Major −0.120 −1.903* 0.013 0.012
Basic education environment 0.108 1. 710* 0.007 0.006
A Gender −0.188 −3.283** 0.014 0.013
Major −0.024 −1.241 0.002 0.001
School support 0.117 3.623** 0.018 0.016
O Major −0.124 −1.846* 0.011 0.010
T Gender −0.155 −2.665** 0.010 0.008
Major −0.020 −1.044 0.001 0.000
School support 0.085 2.591** 0.009 0.008
P Basic education environment 0.106 1.813* 0.010 0.008
Note: * = p < 0.05, ** = p < 0.01.
16 L. SUN ET AL.

Figure 5. Predictive effects between influencing factors and CT.

described in Table 6. It could be seen from Table 6 that gender could predict both the algorithmic
thinking and critical thinking of CT, and so could school support. Majors mainly predicted the crea-
tivity and cooperativity, while the basic educational environment could positively predict the crea-
tivity and problem-solving ability. A more clarified and intuitive representation could also be seen in
Figure 5.

4. Discussion
The primary objective of this study is to evaluate the undergraduates’ CT skills and their influence
mechanisms based on the constructed CT evaluation framework, and to provide guiding strategies
for their developments. According to the framework, this study localized the CTS and tested the
reliability and effectiveness of the Chinese version of CTS respectively. The results showed that
the Chinese version of CTS presented high reliability and validity and could be applied to evaluate
the undergraduates’ CT skills effectively. Furthermore, different predictors, such as gender, major,
basic educational environment, school support, exerted different predictive effects on the
different factors of Chinese undergraduates’ CT skills. Then this study adopted this tool to re-evalu-
ate the students’ CT skills and explore the predictive factors that affected them. And finally, sugges-
tions and countermeasures were proposed for the future development of undergraduates’ CT skills,
according to the theoretical framework of CT evaluation.
Based on Marzano’s new educational objectives taxonomy, this study constructed a theoretical
framework for CT evaluation. This study advocated that it was very necessary to establish the
relationship between the thinking levels of CT and its knowledge fields when evaluating CT skills.
Under the guidance of the taxonomy, the CT evaluation framework constructed in this study realized
the connection from the knowledge fields to the thinking system (Marzano, 2001). According to this
framework, the development of CT evaluation tools can be implemented in a two-dimensional way:
on the one hand, the dimensions of the knowledge fields may be combined with the teaching objec-
tives of CT to determine which knowledge of CT to be evaluated (CT concepts, CT practices, and CT
perspectives); on the other hand, the dimension of the thinking system can take the CT knowledge as
the basis for connecting the three CT systems (CT cognitive system, CT metacognitive system, and CT
self-system) to the fields of knowledge. During the process of evaluation, this study can evaluate the
interactions between students through the three CT systems and the three fields of CT knowledge, to
develop a comprehensive and systematic evaluation of CT skills from knowledge to thinking.
In this study, gender could also predict the undergraduates’ algorithmic thinking and critical
thinking. The general CT skills of males were significantly higher than that of females. These
 INTERACTIVE LEARNING ENVIRONMENTS 17

findings were consistent with the results of previous studies on gender differences in education
(Román-González et al., 2017). As Atmatzidou and Demetriadis (2016) found, females needed
more time than males to acquire similar CT skills. This gender gap could be attributed to the differ-
ences in the cognitive development structures of students of different genders. Neuroscience studies
have shown that girls developed faster in vocabulary, reading, and writing skills, while boys knew
better about logical numbers and spatial relationships (Sousa & Tomlinson, 2011). Therefore, it
would be necessary to adopt different teaching methods for students of different genders. A
report by Angeli and Valanides (2020) concluded that there was a significant interaction between
gender and scaffolding strategies with boys benefiting more from the manipulation-based activities
and girls benefiting more from the collaborative writing activities. Likewise, gender differences may
also be caused by females’ misunderstandings and lack of confidence or interest in computer
science. Through qualitative research, Dasuki and Quaye (2017) proposed that students’ lack of
intrinsic motivation or expectation for the future was the main reason for the failures of computing
education in developing countries. According to the theoretical framework of CT evaluation, the
advancement of girls’ CT skills can focus on developing their self-system and mobilizing their interest
and autonomy in computing activities. To achieve these, teachers may need to provide correspond-
ing programs for teaching and training students to transform the problems from life and study into a
procedural perspective to solve. Educators also need to stimulate girls’ interests from the early stage
of compulsory education and encourage them to actively participate in CT activities, which will be
essential to narrow the gender gap in CT and enhance the status of women in the computer field.
The results also demonstrated that students’ majors were related to their CT skills, especially to
the domains of creativity and cooperativity, which was consistent with the current findings within
the fields of CT and natural sciences. Evia et al. (2015) argued that it was easier to develop CT
skills of students in the majors of computer, mathematics, and science. Luo et al. (2020) contended
that the teaching and researching activities of CT for students in science majors may be relatively
easy to launch. According to Moreno-León et al. (2016), additionally, the cooperation between the
majors of Math and Scratch would benefit the students’ CT skills. However, CT as a comprehensive
skill can be introduced and developed in all disciplines, not just in science and technology. Growing
attention has been paid to the applications of CT in art, linguistics, and other humanities and social
sciences. In a meta-analysis, Sun et al. (2021b) proved that there were both positive and significant
effects of implementing CT education in different disciplines. Some scholars focused on the
implementations and evaluations of CT in the teachings of humanities, arts, and social sciences
(Guzdial, 2008; Sáez-López et al., 2016). Anderson (2016) urged for introducing CT into the teachings
of psychological and behavioral sciences to solve the open problems through a variety of potential
solutions and take the procedural steps to develop the research paradigms. Therefore, when com-
bining CT with humanities, this study should pay attention to the developments of students’ CT cog-
nitive system and CT metacognitive system, increase the infiltration of computational knowledge
and activities into the fields of humanities and social sciences, and cultivate the students’ abilities
to solve problems step by step.
The hypothesis that the basic educational environment could predict students’ CT skills were also
verified by the findings that the basic educational environment positively predicted the creativity
and problem-solving. This study also found that students from an urban basic educational environ-
ment would present a higher CT skill than those from other types of basic educational environment
within these study samples. The differences in the basic educational environment may be reflected
in the teaching resources and the diversity of teaching methods, while the differences in these
resources and methods may be explained as a reason for the uneven development of students’
CT skills in different basic educational environments. For example, Birney and McNamara (2019)
maintained that the interdisciplinary studys’ teaching methods with richer contents and practices
in the urban could improve CT skills more effectively. In the research of Kale et al. (2018), it was
found that teachers’ understanding of CT education and their acquisition of CT technology varied
between the rural and urban areas, and the level of CT skills reported by teachers in rural areas
18 L. SUN ET AL.

was significantly lower than that in urban areas. To solve this problem, this study needs to pay more
attention to the regulation of students’ CT metacognitive systems, especially to students in under-
developed regions such as the rural areas. Metacognitive systems are adopted to establish relevant
goals for an activity to increase the understanding and application of the specific information pre-
sented. Therefore, educational practitioners need to enrich the classroom teachings and activities,
and to increase the ways for students to capture and develop their CT. Effective teaching design
can promote the students’ learning and optimize the teaching effects (Gagne & Briggs, 1974). For
example, Jun et al. (2017) asserted that a design-based learning approach could improve CT skills.
Educational robots could also be applied to implement the activities aimed at developing students’
CT skills in the classroom. Chevalier et al. (2020) constructed an operational framework to support
the teachers in designing, implementing, and evaluating the educational robot activities aimed at
developing the CT skills, and also provided the preliminary evidence for activities that require
specific teaching interventions.
School support would positively predict the algorithmic thinking and critical thinking. The
general CT skills of undergraduates from first-class universities would outperform the CT skills of
those from other levels of universities in China. There were significant differences between
various universities in China, with higher-level ones being able to provide students with more
learning resources of high-quality. According to the theory of psychological autonomy proposed
by Sternberg and Grigorenko (1997), the external environment that students were exposed to
would affect their behaviors. It could be further inferred that school support, as an important
part of undergraduates’ external environments, would also affect the development of their CT
skills significantly. As proposed in the research by Kale et al. (2018), different school supports deter-
mined the teachers’ different accesses to CT skills and CT technologies, and rich school support
could promote the teachers’ CT teaching effects, which in turn would affect the development
of students’ CT skills (Master et al., 2016). Given the differences at the school levels, it is particularly
important to effectively enhance the enthusiasms and adjustment abilities of undergraduates to
actively participate in computing courses. Schools can provide students with professional or infor-
mal environments and opportunities to participate in the cultivation of their CT skills. With the
advancement of reform, universities at different levels continued to develop the majors such as
artificial intelligence and intelligent education to provide diversified assistances for students’ CT
educations, gradually narrowing the CT education gap caused by different levels of school
supports.
Interestingly, students’ grades were found as not related to any CT factors. With the growth of the
grades, there was no significant difference in the CT skills of undergraduates in China. This showed
that the current CT educations for undergraduates had not produced a significant effect on the aca-
demic performances. Basic computer courses are one of the primary means to improve the under-
graduates’ CT skills in China. In recent years, the Department of Higher Education has proposed a
new idea of teaching reform on basic computer courses based on CT development, which has pro-
moted the related researches and teaching improvements on the cultivation of CT skills in certain
colleges and universities in China (Ministry of Education of the People’s Republic of China, 2018).
During the basic computer courses, teachers should cultivate students’ CT skills and improve their
information literacy under the guidance of the CT teaching concepts. However, the results of this
study disclosed that the current achievement of CT education for undergraduates was insignificant,
and there remained many difficulties and challenges during the reform process (He et al., 2014).
Therefore, deepening the CT reform in undergraduate education may be the focus that deserves
more attention to in the future, and the curricula of CT education could reflect more systematization,
long-term natures, and continuities. As found in the relevant studies classified by school support,
there was a significant correlation between the grades and CT skills of students with first-class uni-
versity supports. In other words, improving the level of school support may be a starting point to
enhance the academic effectiveness of basic computer teaching, and how to effectively connect
the courses with school support is another topic worthy of further discussion. This discovery
 INTERACTIVE LEARNING ENVIRONMENTS 19

would be ground-breaking and providing great value for implementing continuous long-term teach-
ings of CT at different levels of universities.

4.1. Implications for policy and practice

Firstly, when evaluating the CT skills, educators should establish the valid relationships between the
knowledge fields of CT and the thinking systems, and comprehensively assess the development of
students’ CT skills from the two-dimensional perspective of thinking and knowledge. Secondly,
societies, families, and schools should all pay more attention to the females’ CT educations from
an early age, and encourage them to engage in STEM courses and related activities. In the class-
rooms, teachers can organize specific activities for girls to stimulate their interests in learning CT
and help them to build confidence, which is essential for the sustainable development of girls’ CT
skills. Thirdly, it is crucial to strengthen the comprehensive education of CT for different undergradu-
ate majors. The cross-integration of professional and general courses related to CT could be provided
for students of different majors. For example, the combination of linguistics and artificial intelligence
with the cross-disciplinary approach could be applied to promoting the CT developments of stu-
dents from different majors. Fourthly, starting from the basic educational environment may be
one of the most effective ways to improve the students’ CT. To narrow the gap of CT in the basic
educational environment, CT education could be gradually infiltrated into the teachers’ ideologies
and teaching practices, especially for those in rural China. Fifthly, the first-class universities could
be more inclusive and open, and provide corresponding services and supports to other universities,
especially with courses related to CT. It will also be quite helpful to strengthen the cooperation and
communication among universities at different levels, and narrow their differences in the develop-
ments of students’ CT with different school support. Finally, this study showed that the relationship
between grade and CT was insignificant. Therefore, it is suggested that undergraduates could invest
more energy and time on participating into the CT-related curricula, to develop their CT skills
substantially.

4.2. Limitations and future research directions

Notably, there have been some limitations to this study. First of all, it took Chinese undergraduates as
the sample to verify the evaluation scale of their CT, and put forward the corresponding guidance for
the teachings of CT in undergraduates education. As a consequence, any generalization under other
social and cultural backgrounds or other educational levels should be very cautious. Future studies
might have to examine the students from different countries, at different educational stages, such as
high schools, middle schools, and primary schools. Secondly, as this study adopted a cross-sectional
research design to investigate the connection between influential factors and CT skills, it might be
difficult to draw causal inferences from the present findings. The future researches can focus on col-
lecting the longitudinal data about the development of undergraduates’ CT skills, exploring the CT
levels and evaluation methods of different grades at universities, and providing more systematic and
coherent teaching guidance with corresponding evaluation criteria for the development of under-
graduates’ CT skills. Thirdly, this study took self-reported measures for evaluating the students’
CT skills, which were susceptible to the common method variance and social desirability bias.
Future researches could adopt other strategies (e.g. peer-reported measures, teacher-reported
measures, and parent-reported measures) to assess the students’ CT skills and address this methodo-
logical shortcoming. Lastly, contextual variables such as basic educational environment and school
support were only reported in this particular study, which limited the testing of their moderation
effects. Future studies could collect more data from other contextual variables like information
about teachers, families, and schools, all of which might influence the CT skills of students
significantly.
20 L. SUN ET AL.

5. Conclusion
Based on Marzano’s new taxonomy of educational objectives, this study constructed a two-dimen-
sional theoretical framework for CT evaluation from the fields of CT knowledge to CT systems, which
would provide theoretical guidance for the follow-up development of CT evaluation criteria. Accord-
ing to this framework, this study has developed a CT evaluation scale for undergraduates based on a
Chinese sample, and has verified both its reliability and validity. It adopted this tool to evaluate the
CT skills of undergraduates and found them to be at the upper-middle level. This study further
explored the personal and environmental factors affecting the undergraduates’ CT skills, and pro-
posed suggestions and countermeasures for their developments, according to the theoretical frame-
work of CT evaluations. Future researches can focus on investigating and reforming the learning
environments and courses to improve the undergraduates’ CT skills, which will be conducive to
their lifelong successes.

Acknowledgements
This work was supported by the National Social Science Foundation Youth Project in Pedagogy of China [grant numbers
CCA190261].

Disclosure statement
No potential conflict of interest was reported by the author(s).

Funding
This work was supported by the National Social Science Foundation Youth Project in Pedagogy of China [grant number
CCA190261].

Data availability statement

The data of this study is not yet available online in any institutional database. However, after making a request to the
author and justifying the use, we will send the entire data package upon request. The request should contact Dr Lihui
Sun: slhphd@yeah.net.

ORCID
Lihui Sun http://orcid.org/0000-0002-9188-9022
Linlin Hu http://orcid.org/0000-0001-7833-6522
Danhua Zhou http://orcid.org/0000-0001-9217-9527
Weipeng Yang http://orcid.org/0000-0002-8057-2863

References
Anderson, N. D. (2016). A call for computational thinking in undergraduate psychology. Psychology Learning & Teaching,
15(3), 226–234. https://doi.org/10.1177/1475725716659252
Angeli, C., & Valanides, N. (2020). Developing young children’s computational thinking with educational robotics: An
interaction effect between gender and scaffolding strategy. Computers in Human Behavior, 105, 105954. https://
doi.org/10.1016/j.chb.2019.03.018
Atmatzidou, S., & Demetriadis, S. (2016). Advancing students’ computational thinking skills through educational
robotics: A study on age and gender relevant differences. Robotics and Autonomous Systems, 75, 661–670. https://
doi.org/10.1016/j.robot.2015.10.008
Barr, V., & Stephenson, C. (2011). Bringing computational thinking to K-12: What is involved and what is the role of the
computer science education community? ACM Inroads, 2(1), 48–54. https://doi.org/10.1145/1929887.1929905
Bebras−Ireland. (2019). The Bebras computational thinking challenge. https://bebras.techweek.ie/
 INTERACTIVE LEARNING ENVIRONMENTS 21

Birney, L., & McNamara, D. (2019). The curriculum and community enterprise for restoration science S.T.E.M. + C pro-
fessional learning model: Expansion and enhancement. Journal of Curriculum and Teaching, 8(3), 122–131. https://
doi.org/10.5430/jct.v8n3p122
Brennan, K., & Resnick, M. (2012, April 13–17). New frameworks for studying and assessing the development of compu-
tational thinking. Annual Meeting of the American Educational Research Association (pp. 1–25), Vancouver, BC,
Canada.
Butler, D., & Leahy, M. (2021). Developing preservice teachers’ understanding of computational thinking: A construc-
tionist approach. British Journal of Educational Technology, 52(3), 1060–1077. https://doi.org/10.1111/bjet.13090
Chevalier, M., Giang, C., Piatti, A., & Mondada, F. (2020). Fostering computational thinking through educational robotics:
A model for creative computational problem solving. International Journal of STEM Education, 7(1), 39. https://doi.
org/10.1186/s40594-020-00238-z
Curzon, P. (2015). Computational thinking: Searching to speak. http://teachinglondoncomputing.org/free-workshops/
computational-thinking/searching-to-speak/
Dam, G. T., & Volman, M. L. (2004). Critical thinking as a citizenship competence: Teaching strategies. Learning and
Instruction, 14(4), 359–379. https://doi.org/10.1016/j.learninstruc.2004.01.005
Dasuki, S., & Quaye, A. (2017). Undergraduate students’ failure in programming courses in institutions of higher edu-
cation in developing countries: A Nigerian perspective. The Electronic Journal of Information Systems in Developing
Countries, 76(1), 1–18. https://doi.org/10.1002/j.1681-4835.2016.tb00559.x
Denning, P. J., & Tedre, M. (2019). Computational thinking. MIT Press.
DeVellis, R. F. (1991). Scale development: Theory and application. SAGE.
Durak, H. Y., & Saritepeci, M. (2018). Analysis of the relation between computational thinking skills and various variables
with the structural equation model. Computers & Education, 116, 191–202. https://doi.org/10.1016/j.compedu.2017.
09.004
Edwards, B. D. (2010). Confirmatory factor analysis for applied research. Organizational Research Methods, 13(1), 214–
217. https://doi.org/10.1177/1094428108323758
Espino, E., & González, G. (2016, September 13–16). Gender and computational thinking: Review of the literature and appli-
cations. The XVII International Conference on Human Computer Interaction (pp. 1–2), New York: ACM.
Evia, C., Sharp, M. R., & Pérez-Quiñones, M. A. (2015). Teaching structured authoring and DITA through rhetorical and
computational thinking. IEEE Transactions on Professional Communication, 58(3), 328–343. http://works.bepress.
com/matthew_sharp/3/ https://doi.org/10.1109/TPC.2016.2516639
Fields, D. A., Shaw, M. S., & Kafai, Y. B. (2018, August 21–25). Personal learning journeys: Reflective portfolios as “objects-to-
learn-with” in an e-textiles high school class. V. Dagiene, & E. Jastu_e (Eds.), Constructionism 2018: Constructionism,
computational thinking and educational innovation: Conference proceedings (pp. 213–223), Vilnius, Lithuani. http://
www.constructionism2018.fsf.vu.lt/proceedings
Gagne, R. M., & Briggs, L. J. (1974). Principles of instructional design. Holt, Rinehart & Winston.
Gignac, G. E., & Szodorai, E. T. (2016). Effect size guidelines for individual differences researchers. Personality and
Individual Differences, 102, 74–78. https://doi.org/10.1016/j.paid.2016.06.069
Grover, S., & Pea, R. (2013). Computational thinking in K-12: A review of the state of the field. Educational Researcher, 42
(1), 38–43. https://doi.org/10.3102/0013189X12463051
Guzdial, M. (2008). Education paving the way for computational thinking. Communications of the ACM, 51(8), 25–27.
https://doi.org/10.1145/1378704.1378713
He, M., Chen, W., Chen, X., Ye, X., Yang, F., & Wang, K. (2014). Analysis on the reform method of university computer basic
course based on computational thinking. Computer Engineering and Science, 36(S1), 96–99. (In Chinese). https://doi.
org/10.13966/j.cnki.kfjyy j.2008.02.001
ISTE. (2015). CT leadership toolkit. http://www.iste.org/docs/ctdocuments/ct-leadershipt-toolkit.pdf?sfvrsn¼4
Jun, S., Han, S., & Kim, S. (2017). Effect of design-based learning on improving computational thinking. Behaviour &
Information Technology, 36(1), 43–53. https://doi.org/10.1080/0144929X.2016.1188415
Kale, U., Akcaoglu, M., Cullen, T., & Goh, D. (2018). Contextual factors influencing access to teaching computational
thinking. Computers in the Schools, 35(2), 69–87. https://doi.org/10.1080/07380569.2018.1462630
Katai, Z. (2014). The challenge of promoting algorithmic thinking of both sciences- and humanities-oriented learners.
Journal of Computer Assisted Learning, 31(4), 287–299. https://doi.org/10.1111/jcal.12070
Kline, R. B. (2005). Principles and practice of structural equation modeling (2nd ed.). The Guilford Press.
Korkmaz, Ö, Çakir, R., & Özden, M. Y. (2017). A validity and reliability study of the computational thinking scales (CTS).
Computers in Human Behavior, 72, 558–569. https://doi.org/10.1016/j.chb.2017.01.005
Lee, M., & Lee, J. (2021). Enhancing computational thinking skills in informatics in secondary education: The case of
South Korea. Educational Technology Research and Development, 2021, 1–25. https://doi.org/10.1007/s11423-021-
10035-2
Leonard, J., Mitchell, M., Barnes-Johnson, J., Unertl, A., Outka-Hill, J., Robinson, R., & Hester-Croff, C. (2018). Preparing
teachers to engage rural students in computational thinking through robotics, game design, and culturally respon-
sive teaching. Journal of Teacher Education, 69(4), 386–407. https://doi.org/10.1177/0022487117732317
22 L. SUN ET AL.

Luo, F., Antonenko, P. D., & Davis, E. C. (2020). Exploring the evolution of two girls’ conceptions and practices in com-
putational thinking in science. Computers & Education, 146, 103759. https://doi.org/10.1016/j.compedu.2019.103759
Lyon, J. A., & Magana, A. J. (2020). Computational thinking in higher education: A review of the literature. Computer
Applications in Engineering Education, 28(5), 1174–1189. https://doi.org/10.1002/cae.22295
Marzano, R. J. (1998). A theory-based meta-analysis of research on instruction (technical report). Mid-continent Regional
Educational Laboratory.
Marzano, R. J. (2001). Designing a new taxonomy of educational objectives. Corwin Press.
Marzano, R. J., & Kendall, J. S. (2007). The new taxonomy of educational objectives. Corwin Press.
Master, A., Cheryan, S., & Meltzoff, A. N. (2016). Computing whether she belongs: Stereotypes undermine girls’ interest
and sense of belonging in computer science. Journal of Educational Psychology, 108(3), 424–437. https://doi.org/10.
1037/edu0000061
Ministry of Education of the People’s Republic of China. (2011). Notice on the issuance of the catalogue of disciplines for
degree awarding and Talent training (2011). http://www.moe.gov.cn/srcsite/A22/moe_833/201103/t20110308_
116439.html
Ministry of Education of the People’s Republic of China. (2018). Notice on the publication of the list of computer curri-
culum reform projects in universities. http://edu.sh.gov.cn/xxgk_jyyw_gdjy_5/20200514/0015−gw_418212019
002.html
Moreno−León, J., Robles, G., & Román−González, M. (2016). Code to learn: Where does it belong in the K−12 curricu-
lum? Journal of Information Technology Education: Research, 15, 283–303. https://doi.org/10.28945/3521
Nouri, J., Zhang, L., Mannila, L., & Noren, E. (2020). Development of computational thinking, digital competence and 21st
century skills when learning programming in K− 9. Education Inquiry, 11(1), 1–17. https://doi.org/10.1080/20004508.
2019.1627844
NRC. (2011). Report of a workshop on the pedagogical aspects of computational thinking. Nat. Acad. Press. https://doi.org/
10.17226/13170
Pan, T. T., Zhan, G. H., & Li, Z. H. (2016). Exploration on the cultivation of computational thinking ability in the teaching of
PhotoShop. L. Huang, J. Cao & J. Gao (Eds.), Proceedings of the first international conference on information technol-
ogy in education and learning (pp. 32–35). 1st International Conference on Information Technologies in Education
and Learning (ICITEL), December 19–20.
Papert, S. (1980). Mindstorms: Children, computers and powerful ideas. Basic Books, Inc.
Plucker, J., Beghetto, R., & Dow, G. (2004). Why isn’t creativity more important to educational psychologists? Potentials,
pitfalls, and future directions in creativity research. Educational Psychologist, 39, 83–96. https://doi.org/10.1207/
s15326985ep3902_1
Pompili, M., Tatarelli, R., Rogers, J. R., & Lester, D. (2007). The hopelessness scale: A factor analysis. Psychological Reports,
100(2), 375–378. https://doi.org/10.2466/pr0.100.2.375-378
Resnick, L. (1987). Education and learning to think. National Academy Press.
Román-González, M., Pérez-González, J., & Jiménez-Fernández, C. (2017). Which cognitive abilities underlie compu-
tational thinking? Criterion validity of the Computational Thinking Test. Computers in Human Behavior, 72, 678–
691. https://doi.org/10.1016/j.chb.2016.08.047
Rose, S., Habgood, J., & Jay, T. (2018, October 4–5). Pirate plunder: Game-based computational thinking using Scratch
blocks. 12th European Conference on Game Based Learning (pp. 556–564), Sophia Antipolis, France. SKEMA
Business School
Sáez-López, J., Román-González, M., & Vázquez-Cano, E. (2016). Visual programming languages integrated across the
curriculum in elementary school: A two year case study using “scratch” in five schools. Computers & Education,
97, 129–141. https://doi.org/10.1016/j.compedu.2016.03.003
Sheng, Q. L. (2008). Foster high-level ability of problem-solving: The systemic explanation of Marzano’ s taxonomy of
cognitive objectives. Open Education Research, 14(2), 10–21. (In Chinese). https://doi.org/10.13966/j.cnki .kfjyy j.2008.
02.001
Sousa, D. A., & Tomlinson, C. A. (2011). Differentiation and the brain: How neuroscience supports the learner-friendly class-
room. Bloomington.
Sternberg, R. J., & Grigorenko, E. L. (1997). Are cognitive styles still in style? American Psychologist, 52(7), 700–712. https://
doi.org/10.1037/0003-066X.52.7.700
Sun, L., Hu, L., Yang, W., Zhou, D., & Wang, X. (2020). STEM learning attitude predicts computational thinking skills
among primary school students. Journal of Computer Assisted Learning, 37(2), 346–358.
Sun, L., Hu, L., & Zhou, D. (2021a). Which way of design programming activities is more effective to promote K-12 stu-
dents’ computational thinking skills? A meta-analysis. Journal of Computer Assisted Learning, 37(4), 1048–1062.
Sun, L., Hu, L., & Zhou, D. (2021b). Improving 7th-graders’ computational thinking skills through unplugged program-
ming activities: A study on the influence of multiple factors. Thinking Skills and Creativity, 42, 1000926.
Tang, X. D., Yin, Y., Lin, Q., Hadad, R., & Zhai, X,M. (2020). Assessing computational thinking: A systematic review of
empirical studies. Computers & Education, 148, 103798. https://doi.org/10.1016/j.compedu.2019.103798
 INTERACTIVE LEARNING ENVIRONMENTS 23

Tikva, C., & Tambouris, E. (2020). Mapping computational thinking through programming in K-12 education: A concep-
tual model based on a systematic literature review. Computers & Education, 162, 104083. https://doi.org/10.1016/j.
compedu.2020.104083
Tsai, M. J., Liang, C. J., Lee, S. W., & Hsu, C. Y. (2021). Structural validation for the developmental model of computational
thinking. Journal of Educational Computing Research, 60, 1–18. https://doi.org/10.1177/07356331211017794
Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35. https://doi.org/10.1145/1118178.
1118215
Wing, J. M. (2008). Computational thinking and thinking about computing. Philosophical Transactions of the Royal
Society a Mathematical Physical and Engineering Sciences, 366(1881), 3717–3725. https://doi.org/10.1098/rsta.2008.
0118
Wing, J. M. (2010). Computational thinking: What and why? http://www.cs.cmu.edu/~CompThink/resources/
TheLinkWing.pdf,2010-11-17.\
Xing, W. (2021). Large-scale path modeling of remixing to computational thinking. Interactive Learning Environments, 29
(3), 414–427. https://doi.org/10.1080/10494820.2019.1573199
Yagci, M. (2019). A valid and reliable tool for examining computational thinking skills. Education and Information
Technologies, 24(1), 929–951. https://doi.org/10.1007/s10639-018-9801-8
Yu, T., & Richardson, J. (2015). Examining reliability and validity of a Korean version of the community of inquiry instru-
ment using exploratory and confirmatory factor analysis. The Internet and Higher Education, 25, 45–52. https://doi.
org/10.1016/j.iheduc.2014.12.004

View publication stats