The Journal of Educational Research

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/vjer20

Evaluating scientific thinking among Shanghai’s

students of high and low performing schools

Irfan Ahmed Rind & Bo Ning

To cite this article: Irfan Ahmed Rind & Bo Ning (2020): Evaluating scientific thinking among
Shanghai’s students of high and low performing schools, The Journal of Educational Research,
DOI: 10.1080/00220671.2020.1832430

To link to this article: https://doi.org/10.1080/00220671.2020.1832430

Published online: 20 Oct 2020.

Submit your article to this journal

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=vjer20
THE JOURNAL OF EDUCATIONAL RESEARCH
https://doi.org/10.1080/00220671.2020.1832430

Evaluating scientific thinking among Shanghai’s students of high

and low performing schools
Irfan Ahmed Rinda,b and Bo Ningc
a
Department of Education, Sukkur IBA University, Sukkur, Pakistan; bTeachers’ College, Columbia University, New York City, New York, USA;
c
Research Institute for International and Comparative Education, Shanghai Normal University, Shanghai, China

ABSTRACT ARTICLE HISTORY

Shanghai is dubbed as a role model for science and mathematics education as its fifteen-year-olds Received 22 April 2020
have been outperforming all in the Program for International Student Assessment (PISA) since Revised 29 August 2020
2009. Shanghai’s achievements are attributed to its interest in adopting innovative international Accepted 1 October 2020
trends in education and equally effectively implementing these at its high and low-performing
KEYWORDS
schools. One such trend in science education is based on the univariable and Control of Variable Multivariable thinking;
(CoV) strategies. This model is also used in constructing higher-order thinking items in science control-of-variables; scien-
and mathematics assessments in PISA. Our first objective was to understand if students of tific thinking;
Shanghai mastered the CoV strategies. Beyond CoV models, the emerging trend in science educa- science education
tion promotes multivariable thinking among young adolescents. Our second objective was to
understand if Shanghai has adopted this emerging trend and prepared its students on multivari-
able thinking. Using specially designed and previously validated assessments, we measured and
compared the CoV and multivariable thinking skills of fifteen-year-olds representing one high and
one low-performing school. Our results highlighted the equally exceptional performance of both
schools in the CoV tasks and comparatively poor performance in the multivariable causal reason-
ing and prediction tasks. These findings may offer one aspect to understand Shanghai’s perform-
ance in the PISA, at the same time highlight the weaknesses in its contemporary
science education.

Introduction producing ‘test-taker experts’ who perform exceptionally

The 2009 results of the Program for International Student
Assessment (PISA) caught the attention of the world when
Shanghai outperformed all the major countries in science
and mathematics. Considering that PISA assesses higher-
order thinking skills, including critical thinking and prob-
lem-solving abilities of fifteen-year-old school students in
science and mathematics (OECD, 2015), it was surprising
for both national and international critics to comprehend
Shanghai’s performance (for a detailed review, see Zhang &
Akbik, 2012). Nevertheless, Shanghai has maintained its top
position in science and mathematics in the PISA of 2011,
2015, and 2018. This continuous educational development of
Shanghai attracted different countries, which looked toward
Shanghai for teaching styles of science and mathematics
(Coughlan, 2014; Gibb, 2015; OECD 2010, 2011).
Since China has historically cherished the traditional
standardized examinations that promote drills and rote
memorization (Tan & Ng, 2018), how students of Shanghai
excelled in the higher-order thinking skills became a topic
of debate in academic circles. Some criticized the education
system of Shanghai as ‘elitist,’ one that rewards the high per-
formers and abandoned the low performing students, thus
well in the examinations of any sort, including PISA, with-
out having mastery over higher-order thinking skills
(Hongbing, 2010; Ringmar, 2013). Others criticized the high
scores of students which are achieved at the cost of a heavy
workload, low social skills, and insufficient development of
creativity (Xie, 2010; Zhang & Akbik, 2012; Zhao, 2011).
Most highlighted criticism, which is also relevant to this
study, was related to the representativeness of the sample of
PISA, questioning whether Shanghai chose the high-per-
forming schools or students for the test and that the low-
performing schools of suburban areas would struggle in
such international tests (Dillon, 2010).
Liang et al. (2016) have presented a comprehensive
response to these criticisms and highlighted different reform
initiatives behind Shanghai’s success. They stressed on the
leading position Shanghai has taken in China’s socioeco-
nomic and educational reforms since 1978, which made it
sensitive to global trends. Shanghai has adopted an open-
door policy and opened itself up for innovative international
trends in education, most of which are mastered with trial
and error (Zhang & Akbik, 2012). One such contemporary
model of science education is based on the univariable

CONTACT Bo Ning nibo2003@163.com Research Institute for International and Comparative Education, Shanghai Normal University, Guilin Road 100,
Building Jiaoyuanlou, Room 1106, Shanghai 200234, China.
ß 2020 Taylor & Francis Group, LLC
2 I. A. RIND AND B. NING
models, particularly the Control of Variable (CoV) — a
strategy to hold the influence of other variables constant so
that the effect of a focal variable can be identified (Kuhn
et al., 2008). Such CoV strategies are embedded in the
national curricula of science education for the development
of scientific reasoning among students (Australian
Curriculum, 2015; National Curriculum of England, 2013;
Next Generation Science Standards, 2013). China also
adopted a similar approach to science education (Guan &
Meng, 2007; Hongbing, 2010; Huang et al., 2017; Jiaoshou,
2004). Interestingly, these univariable and CoV models are
also used in constructing many higher-order thinking items
in the assessments of science in PISA (OECD, 2015; Wasis,
2014). Have students of Shanghai mastered the CoV strat-
egies and therefore performed well in the PISA? This is an
interesting question that has never been raised in evaluating
Shanghai’s performance in PISA.
Although CoV is an important strategy, in the real world
of science and scientific thinking, multiple variables con-
tinue to be a major presence. This limitation of CoV models
raised the importance of multivariable thinking— the coord-
ination of the effects of multiple variables on an outcome
(Kuhn et al., 2008). This emerging trend in science educa-
tion is getting its place in the West. Has Shanghai already
adopted this emerging trend? Most importantly, are these
scientific thinking skills, i.e., CoV and multivariable think-
ing, equally promoted among the students of high and low-
achieving schools around the greater Shanghai region? All
these unanswered questions reflect a gap in the growing
body of literature around Shanghai’s educational reforms
and its performance in science. This study attempts to
bridge this gap by measuring fifteen-year-old students’ CoV
and multivariable thinking skills using specially designed
and previously validated assessments. Focusing on a high
and a low-performing school located in the greater Shanghai
region, this study offers a new perspective on students’ sci-
entific thinking through their CoV and multivariable think-
ing skills.
Literature review
Much of scientific thinking and scientific theory building
pertains to the development of causal models between varia-
bles of interest. The most common approach used in science
education for the causal models is referred to as Control of
Variable (CoV), which, as mentioned earlier, is embedded in
the national curricula of science education of many devel-
oped countries.
CoV
The concept of a variable and its primary role in doing sci-
ence is usually introduced in early grades. This concept is
usually introduced through a simple experiment: ‘An out-
come O occurs when a positive level (e.g. a flame) of an
antecedent variable A, such as heat, is introduced, but O
fails to occur when this positive level of A is absent (e.g. no
flame or other heating agent is present)’ (Kuhn, 2016, p.
393). This experiment is based on single variable logic,
where all other variables are removed or kept constant
except the focal one to assess its influence on an outcome
(Zimmerman, 2000, 2007). This strategy is usually referred
to as CoV and was first introduced by Inhelder and
Piaget (1958).
These univariable designed experiments help students in
constructing data that influence their conceptual representa-
tions of a single cause and effect (Koerber et al., 2005; Rind
et al., 2019; Rind & Kadiwal, 2016).
However, this model has its limitations. For example, it
has been argued that construal of mechanism and processing
of evidence very often function together to support causal
inference (Keil, 2006; Klahr, 2002; Koslowski, 1996). With
this assumption, student’s data-influenced conceptual repre-
sentations of causal effects should be facilitated by generat-
ing plausible mechanisms that might underlie such effects
(Kuhn, 2016). However, if a student’s explanation and avail-
able evidence conflict, it is most probable that the student’s
explanation will overpower the evidence (Williams et al.,
2013). Therefore, students must not only learn to construct
data-based explanations but also learn to evaluate whether
their explanations are correct. These limitations of the uni-
variable model have generated concerns and criticisms
among educators and researchers who encourage science as
inquiry and that scientific practice should be enriched by
contextualization through the use of a more authentic multi-
variable model (Duschl, 2008; Kuhn, 2016, 2018; Williams
et al., 2013).
Multivariable model
The advocates of the multivariable model argue that the
purpose of science education is not only to prepare students
to deal with the problems presented in the controlled envir-
onment of laboratories but to prepare them for real-life
problems. In real-world situations, however, single variables
rarely contribute to an outcome. A combination of multiple
variables jointly influences, additively or interactivity, to an
outcome, a concept contrary to the CoV approach. The abil-
ity to realize the power of multivariable on an outcome
enhances students’ deep analytical and predictive skills
(Kuhn, 2016; Kuhn et al., 2015). Moreover, students need to
comprehend that ‘covariation between two variables need
not be perfect in order for a relationship to exist between
them because effects of other factors, as well as measure-
ment error, are likely to play a role’ (Kuhn et al., 2017, p.
234). This understanding of multivariable causality is a
move toward authentic scientific thinking (Bo et al., 2020;
Rind, in press).
Method
This study is mainly guided by the following
research questions:
1. To what extent are the CoV skills developed among the
students of High Performing School (HPS) and Low
 THE JOURNAL OF EDUCATIONAL RESEARCH 3

Figure 1. An example of a question asked in the CoV assesment.

Performing School (LPS), and is there any significant Participants

difference in the performance of both schools?
Participants were 26 fifteen-year-old 9th-grade students (with
2. To what extent are the multivariable causal reasoning
an equal representation of males and females) drawn from
skills developed among the students of HPS and LPS,
one HPS located in downtown Shanghai and 29 students of
and is there any significant difference in the perform-
the same age and grade (female 58%) from one LPS located
ance of both schools?
in the outskirts of the Jinshan district of Shanghai. Schools
3. To what extent are the multivariable analysis and pre-
in Shanghai are usually tagged as high or low-performing by
diction skills developed among the students of HPS and
the school district administrators using a broader evaluation
LPS, and is there any significant difference in the per-
and accountability practices (see Jensen & Farmer, 2013 for
formance of both schools?
a detailed review). Since we were interested in students’
4 I. A. RIND AND B. NING
performance, we collected data on students’ scores of the
last three grades. On average, the students of the HPS
secured 86% scores in earlier grades, whereas, the students
of LPS secured 67% scores. All the students of former school
received privately paid at-home tutoring in the last three
years, compared to only 32% of the students in the latter
school (see Zhang & Bray, 2018 for a detailed review on the
impact of private tutoring on students’ performance).
Household income of parents may explain the discrimin-
ation of access to private tutoring. According to the data
collected from schools and students, it is estimated that
roughly 85% of the students of the former school belonged
to upper- and upper-middle-class families, thus majority of
them have had the access to expensive private tutoring. In
contrast, most of the students in the latter school belonged
to lower-middle-class families, and therefore about one-third
could afford private tutoring.
For data collection, we approached the management of
both schools via email followed by a face-to-face meeting.
After ensuring that the identities of the schools and students
will be kept anonymous, the management of both schools
agreed to grant us access to the students. On our request,
one science teacher from each school volunteered to facili-
tate us in data collection. After explaining them in great
detail on the nature of the tasks, we requested the facilitat-
ing teachers to present these tasks as graded activities among
students to get their maximum input. After reviewing the
assessments and consultation with school management, the
facilitating teachers agreed. However, to align our tasks with
the pre-scheduled class activities, the facilitating teachers
proposed to conduct all the three assessments (see 3.2 for
details) on different days. The facilitating teacher of the LPS
also agreed to let one of us conduct the same assessments
on selected students of grade 8 for pilot testing (see 3.2.4
for details).
Assessments
Three assessments used in this study were prepared by
Kuhn and colleagues for the students of the same age group.
The first task assessed students’ CoV skills. The next two
tasks assessed students’ multivariable thinking, particularly
their multivariable causal reasoning skills, and multivariable
analysis and prediction skills.
Assessment of CoV skills
This assessment was adopted from Kuhn et al. (2017), which
reads as follows:
The city of New York wants to get new trains for the subway.
They can make trains with short or long Car Size and they can
make trains with four or six Number of Wheels
This description was followed by three pairs of multiple-
choice items. Each pair presents a case for students to evalu-
ate regarding different car designs and offers different justi-
fications to students for their choice. The first item of each
pair had four options, each option having two images of car
designs with different sizes and several wheels. The second
item of the pair had four options, each option presenting
different justifications related to the first item (see Figure 1
for an example). Students were required to evaluate and
identify the correct design and justification for each pair.
The aim was to assess how many students from each group
consistently constructed a control comparison across three
cases as well as provided appropriate justifications.
Assessment of multiple causal reasoning skills
The second test was adopted from Kuhn et al. (2015) to
assess students’ multiple causal reasoning skills. It was based
on a single, unelaborate scenario to generate initial
responses from the students. The responses were assessed to
determine if students made a causal claim and proposed a
means of evaluating it. The scenario reads as follows:
The Public Health department of Portland Ohio has noticed
that the percentage of residents diagnosed with cancer is much
higher in the inner city than in the outlying neighborhoods. The
department is undertaking a study to find out why. You have
been assigned the job. Describe how you would conduct
this study.
The responses generated from this task were analyzed
and coded to pinpoint if students identified the potential
cause(s), and how they related different causes to one
another, as well as to examine if students compared two
kinds of contexts (i.e. inner city and outlying neighbor-
hoods). Using the analytical model of Kuhn et al. (2015, p.
99), students’ responses were coded into one of the follow-
ing three levels and sub-levels:
1. Responses that didn’t find any causal variables.
1.1. No-variable without group comparison
1.2. No-variable with group comparison
2. Responses that identified a single variable
2.1. Univariable without group comparison (i.e. only
A is the cause of an outcome)
2.2. Univariable with group comparison
3. Responses that identified two or more variables and the
way the relationship between these variables were
conceptualized
3.1. Multivariable-alternative with group comparison
(i.e. either A or B is the cause of an outcome)
3.2. Multivariable-alternative without group comparison
3.3. Multivariable-additive with group comparison
(i.e. both A and B and possibly C are the causes
of an outcome)
3.4. Multivariable-additive without group comparison
3.5. Multivariable-interactive with group comparison
(i.e. A and C effect B, and B is the cause of
an outcome)
3.6. Multivariable-interactive without group comparison
The responses of each student were taken as a whole and
were coded into one of the above levels. One of the authors
and a native Chines data analyst coded all responses inde-
pendently, yielding a percentage agreement of 89%. The dif-
ferences were resolved by discussion.
 THE JOURNAL OF EDUCATIONAL RESEARCH 5

Figure 2. Life expectancy scenario followed by items to assess the multivariable analyis and prediction.

Assessment of multivariable analysis and prediction skills
This, a slightly more sophisticated, test was used to assess
students’ multivariable analysis and prediction skills.
Developed by Kuhn et al. (2015) and further validated by
other studies (Arvidsson, 2018; Kuhn et al., 2017), this task
presented authentic data from the World Bank 2010 on two
highly effective contributing factors (Employment and
Family Size), two moderately effective contributing factors
(Education and Home Climate), and one noncontributing
factor (Country Size) on the average Life Expectancy (LE)
across different countries. The data were simplified and pre-
sented in the graphs (see Figure 2). The task started with a
description (see Figure 2). Following an introduction,
6 I. A. RIND AND B. NING
Table 1. Type of students’ responses in percentage to identify the cause(s) of an outcome with and without group.
SINGLE-VARIABLE
No-variable Univariable
GROUPS
Without Group With Group Without Group With Group Without Group With Group Without Group With Group Without Group With Group
HPS (N ¼ 26) – 8 46 19
LPS (N ¼ 29) 14 7 52 17
Note: HPS and LPS refer to high-performing school and low-performing school respectively.
students were asked to predict the LE of nine countries,
based on their status on the five identified variables (i.e.
Employability, Family Size, Education, Climate, and Country
Size) .
The first question asked students to highlight the fac-
tor(s) (i.e. Employability, Family Size, Education, Climate,
and Country Size) that they would need to know to predict
the LE of a country. The aim was to assess if students were
able to recognize the need to take into account all the four
contributing factors, but not the noncontributing factor, in
predicting the LE of the country.
The second question presented the levels of the five fac-
tors (for example, Employability¼ high; Family Size¼ small,
Education¼ higher, Climate¼ Hot, and Country Size¼
small) and asked the students to highlight which factor(s)
they do not need to predict the LE of a country. Here, the
aim was to assess if students were able to identify the non-
contributing effect of “Country Size” on the LE of
the country.
The final question asked students to predict the LE (i.e.
High, Medium, Low, or Very Low) of the country based on
the information provided. The aim was to assess if students
considered all relevant factors and appropriately weighed
moderately and highly effective factors to construct a valid
prediction. As seen in Figure 2, the weight of two moder-
ately effective contributing factors (Education & Home
Climate) contributes less to the outcome than do the two
highly effective contributing factors (Employment & Family
Size). The same sequence was repeated for all nine countries.
The responses on nine countries were analyzed to find out
the number of correct predictions, attributions to contribu-
ting and noncontributing factors in the prediction, and the
consistency of the attribution across nine countries.
In order to ensure that students understood the task, the
facilitating teachers of each school read the description
aloud and explained each graph to the class. The teachers
then asked one random student from each group to attempt
the first set of the questions aloud, just to ensure that the
students understood the activity.
Validation and pilot testing
All three assessments we translated into Mandarin and pilot
tested on a separate group of students. To check the validity
of the translated assessments, we followed different proce-
dures Kuhn and colleagues used in their studies. First, we
selected 10 students from the LPS. One of the researchers,
in the presence of facilitating teacher, instructed students on
the first task and allowed them sufficient time to complete
it. In the end, the facilitating teacher asked students to point
MULTIVARIABLE
Multivariable-alternative Multivariable-additive Multivariable-interactive
8 12 4 – 4 –
10 – – – – –
out any word or phase in the task or in the oral instruction
of the task that they couldn’t understand. Meanwhile, the
researcher evaluated the scores of randomly selected three
students. The researcher, then, conducted an oral examin-
ation of each of the three students separately. During the
oral examination, students were asked 3 to 4 questions con-
structed on the CoV principle. Their responses were scored,
which were then compared with their scores on the CoV
task. The same process was repeated for task 2 and task 3.
The comparative analysis of scores of tasks and oral exami-
nations suggested that all three tasks assessed what they
were intended to assess.
Based on students’ feedback on the tasks, some words
and phrases were modified for better comprehension.
Students viewed the researcher as an outsider, thus experi-
enced fear and anxiety. Therefore, it was decided that facili-
tating teachers from each school would conduct the real
tests for the ease and comfort of the students.
Analysis and results
Findings on COV skills
This task attempted to answer the first research question,
i.e., how students of HPS and LPS performed on the CoV
assessment and if there is any significant difference in the
performance of both schools. It is done by analyzing stu-
dents’ skills to consistently construct a control comparison
and provide appropriate justification across all the three
cases presented in the task. The results show that 24 of 26
students (92%) of HPS, and 26 of 29 students (90%) of LPS
consistently constructed a controlled comparison across all
three cases, with no significant difference in the perform-
ance of both groups, X2(1) ¼ .525, p ¼ .469.
Similarly, 22 students (84%) of HPS and 25 students
(86%) of LPS consistently constructed a controlled compari-
son and also provided appropriate justifications across all
three cases, with no significant difference in the perform-
ance of both groups, X2(1) ¼ .035, p ¼ .853.
These findings suggest that students of both schools have
a good command of the problems based on CoV principles
and that there is no significant difference in their perform-
ance on the CoV task.
Findings on multiple causal reasoning skills
This task attempted to answer the second research question,
i.e., how students of HPS and LPS perform on the multivari-
able causal reasoning assessment and if there is any signifi-
cant difference in the performance of both schools. To

Table 2. Correct predictions across nine countries.
Predictions HPS (N ¼ 26) # (%) LPS (N ¼ 29) # (%)
Nine countries 6 (23) 5 (17)
Eight countries 4 (15) 4 (14)
Seven countries 8 (31) 10 (34)
Six countries 4 (15) 5 (17)
Five countries 3 (12) 1 (4)
Four countries 0 4 (13)
Three countries 1 (4) 0 Contributing Factor-4 (Climate)
Note: HPS and LPS refer to high-performing school and low-performing school
respectively.
answer this question, the given task assessed two skills of
the students. First, it assessed if students were able to iden-
tify the potential cause(s) of ‘cancer in the inner city and
outlying neighborhoods’, and ways in which students related
different causes to one another. Second, if students com-
pared two kinds of contexts (i.e. inner city and outlying
neighborhoods). Table 1 summarizes the results showing
that most of the students of both schools either identified
no variable or single-variable causes for the out-
come (cancer).
In the multivariable categories, most of the responses fell
in the multivariable-alternative with no response of the LPS
students and only two responses of HPS students falling
into the multivariable-additive and interactive categories.
Collectively, 7 responses (27%) of HPS and 5 responses
(17%) of LPS fell into the multivariable category, with no
significant difference in the performance of both groups (p
¼ .517, Fisher’s exact test).
Similarly, 10 of 26 (38%) students of HPS and 7 of 29
students (24%) of LPS realized the need to compare the con-
texts (i.e. the number of cancer cases in the inner city vs.
outlying neighborhoods), which also shows no significant
difference in the performance of both groups, (p ¼ .381,
Fisher’s exact test).
These results show that most of the students of both HPS
and LPS tend to consider single variables in determining the
cause of an outcome. A very few students of both schools
were multivariable thinkers and felt the need to identify
multiple factors to determine an outcome, and the need to
compare two contexts to draw their conclusions.
Findings on the assessment of multivariable analysis
and prediction skills
This task attempted to answer the third research question,
i.e., how students of HPS and LPS perform on the multivari-
able analysis and prediction assessment and if there is any
significant difference in the performance of both schools. To
answer this question, the task assessed if (a) students’ cor-
rectly predicted the LF across nine countries, (b) students
were able to identify the contributing and noncontributing
factors in their prediction, and (c) the consistency in identi-
fying the four contributing factors in their predictions across
the nine countries.
a. Students’ predictions were compared to a model of cor-
rect prediction (weighting the two moderate factors as
contributing half as much to the outcome as the two
THE JOURNAL OF EDUCATIONAL RESEARCH 7
Table 3. Average scores of contributing and noncontributing factors influenc-
ing students’ prediction across nine countries.
Average (SD)
Factors HPS LPS t p
Contributing Factor-1 (Employment) 6.3 (1.22) 5.9 (1.83) 2.47 .173
Contributing Factor-2 (Family Size) 6.6 (1.33) 6.1 (1.00) 1.104 .275
Contributing Factor-3 (Education) 4.2 (.94) 4.8 (.75) 1.129 .258
3.6 (1.36) 3.5 (1.30) .355 .735
Noncontributing Factors (Country Size) 2.7 (.80) 3 (1.30) -.958 .354
Note: HPS and LPS refer to high-performing school and low-performing school
respectively.
strong factors) for each of the nine countries (Kuhn
et al., 2017). Each correct prediction was given 1 point.
As measured against this model, the students of HPS
scored an averaged of 7.8 (SD ¼ 1.51; Shapiro-Wilk Test
> .05) points, whereas the students of LPS scored an
average of 7.2 (SD ¼ 1.78; Shapiro-Wilk Test > .05)
points. There is no significant difference in the perform-
ance of both groups in this task, t ¼ .322, p ¼ .714.
Table 2 shows the consistency of correct predictions
across different countries. Only, 6 of 26 students (23%) of
HPS correctly predicted the LF of all the nine countries as
compared to 5 out of 29 students (17%) of LPS, showing no
significant difference, (p ¼ .739, Fisher’s exact test).
b. For each country, students had four contributing factors
to choose from, each contributing factor carries one
point. It was expected that multivariable thinking stu-
dents would choose all four contributing factors thus
score 4 points for each country, and 36 points for all
the nine countries. The average scores of all the contri-
buting factors for the students of HPS and LPS are 21.7
(SD ¼ 5.46; Shapiro-Wilk Test > .05) and 22.5
(SD ¼ 5.78; Shapiro-Wilk Test > .05), respectively, with
no significant difference, t ¼ -.539, p -.592. Table 3
shows the average scores on each contributing factor
against nine countries for the students of HPS and LPS.
It shows that there is no significant difference in the
average scores of both schools’ students for the two
highly effective and two moderately effective contribu-
ting factors.
In this task, it was also expected that students would not
consider the noncontributing factor (i.e. Country Size) to
predict the LF of the countries due to its neutralizing effects
(see Figure 2). For each country, students were given one
point if they did not select the noncontributing factor.
There were a total of nine points. Table 3 shows that the
students of HPS scored an average of 2.7 (SD ¼ .80;
Shapiro-Wilk Test > .05) on noncontributing factor,
whereas, students of LPS scored an average of 3 (SD ¼ 1.30;
Shapiro-Wilk Test > .05), with no significant difference, t ¼
.958, p ¼ .354.
Moreover, Table 3 also shows a pattern in the average
scores of students from high to low in the spectrum of
highly effective contributing factors to a noncontributing
factor. This suggests that students were tempted to focus
8 I. A. RIND AND B. NING
Table 4. Consistency of students’ attributions.
Attributions
Chose four contributing factors consistently across 9 countries
Chose multiple consistent (but not all four contributing) factors across 9 countries
Chose multiple but inconsistent factors across 9 countries
Chose only one consistent causal factor across 9 countries
Chose only one but inconsistent factor across 9 countries
more on the highly effective contributing factors as influenc-
ing their predictions.
c. Table 4 shows students’ consistency in identifying the
four effective contributing factors in their prediction
across the nine countries. Only 7 of 26 students (27%)
of HPS chose all the four effective contributing factors
consistently across all nine predictions, as compared to
5 of 29 students (17%) of LPS, with no significant differ-
ence, (p ¼ .517, Fisher’s exact test). These results show
that only a few students realized the contribution of
multiple factors affecting an outcome consistently. The
majority showed inconsistency in realizing the need to
consider multiple factors in their predictions, which
reflect their weak multivariable thinking skills.
Discussion and conclusion
The present study offers a new perspective on scientific
thinking skills among the Shanghai students of high and
low-performing schools. The findings show that the students
of both schools performed well in the CoV task and that
there is no significant difference in the performance of both
schools. As mentioned earlier that the CoV task was adopted
from Kuhn and colleagues who used it to measure the CoV
skills among the USA students of the same age. They found
exceptional results on this task by those students who were
specially trained on these CoV skills for weeks (Jewett &
Kuhn, 2016; Kuhn et al., 2017). In this study, however,
Shanghai students of both high and low-performing schools
scored exceptionally well on the same task without any
training. This suggests that Shanghai schools, either high or
low-performing, have been effectively teaching CoV strat-
egies through the regular science curriculum. These findings
may also provide a new perspective to explain Shanghai’s
good performance in the PISA, considering that most of the
higher-order items in the science and mathematics section
of PISA are based on the CoV principle (OECD 2015).
The good performance of the students on the CoV task
may offer a glimpse of the impact of recent reforms in sci-
ence and mathematics education in Shanghai that have spe-
cifically focused on the development of scientific thinking
through inquiry curriculum and teaching (Liang et al.,
2016). The review of the science and mathematics curricu-
lum suggests that many activities are mainly based on the
CoV principle (Ma, 2016). Teachers have also trained them-
selves accordingly. The multi-layered training offered to the
teachers at different stages of their careers reinforces the use
of CoV strategies in their teaching (Yao & Guo, 2018).
Besides, each school is allowed to develop contextually rele-
vant, inquiry-based courses and introduce at least one of
HPS (N ¼ 26) # (%) LPS (N ¼ 29) # (%)
7 (27) 5 (17)
7 (27) 12 (41)
11 (42) 7 (24)
1 (4) 5 (17)
0 0
such courses from grades 1 to 5, and two courses from
grades 6 to 9. Such freedom in course design and implemen-
tation may allow teachers to be more creative in their teach-
ing (Liang et al., 2016).
At the same time, the equally good performance of LPS
on the CoV task in this study may offer a glimpse of educa-
tional reforms Shanghai introduced to deal with the regional
disparity in teaching quality. Using different financing, man-
agement, and twinning strategies, Shanghai has improved
the low-performing schools in many ways. For example, the
municipal government transfers half of education surplus
taxes to economically weak districts, and mandates that top
secondary schools reserve some spaces for middle school
graduates from poor districts (Liang et al., 2016). The most
important initiative has been the introduction of an
‘entrusted school’ management system (Jensen & Farmer,
2013), which allows additional financing, management, and
professional support from HPSs to LPSs. This marriage
between HPSs and LPSs generates mutual professional
development training, teacher and student exchanges, and
joint goals (Liu, 2016).
Despite the remarkable performance on the CoV task,
students of both schools performed poorly on the multivari-
able causal reasoning and prediction tasks. These findings
highlight that although the students reflected developed sci-
entific thinking skills based on the CoV principle, there are
opportunities to achieve what Kuhn et al. (2017, p. 247) call
‘authentic scientific practice’ that incorporates multivariable
thinking among students. Kuhn et al. (2017) argue that the
execution of CoV-based experiments presents a ‘narrow slice
of authentic scientific inquiry and arguably needs to be
enriched by contextualization in a more authentic multivari-
able model’ (p. 233). Authentic scientific practice prepares
students for real-life situations. Everyone makes inferences
about causality as a part of thinking (Bo et al., 2020; Rind,
2014, 2015, 2016; Rind et al., 2016, 2019). However, single
cause thinking only prepares students for the controlled
environment of laboratories. Although contemporary
research on developing scientific thinking skills involves the
management of multiple variables, the focus is on neutraliz-
ing the effects of additional variables to ascertain the effect
of a single variable on an outcome. This trend in science
education has attracted Shanghai, which adopted this model
and mastered it, as reflected in the PISA scores as well as
from the findings of the current study.
The findings of this study have wider implications
beyond Shanghai, as this study attempted to highlight one
of the core principles, i.e., CoV, of scientific thinking used
in the national curricula of science all around the world,
and showed its limitations. The emerging research in scien-
tific thinking argues that mastery over the use of CoV

strategies does not guarantee mastery over the multivariable
causal reasoning or prediction skills (Kuhn, 2007; Kuhn
et al., 2009, 2015; Kuhn & Pease, 2008). The students
trained on the CoV skills trend to adopt a univariable
approach in determining the causal relationship, which is
reflective in the performance of the students of both schools
on the multivariable causal reasoning task. Most of the stu-
dents either pointed out to single or no variables in deter-
mining the outcome. Similarly, on the multivariable analysis
and prediction task, students made the faulty predictions
and were inconsistent in considering contributing and non-
contributing factors on their predictions. Kuhn and Dean
(2004) found similar results among adolescents and some
adults while assessing their multivariable analysis and
prediction skills. They found that apart from making faulty
predictions, their research participants often made inconsist-
ent causal attributions across consecutive predictions.
Moreover, they often failed to implicate as many variables as
influencing their predictions as they had earlier identified as
causal when asked to make explicit judgments of the causal
roles of each variable (in a multivariable context in
which a CoV method must be used to identify casual and
non-casual effects).
Kuhn et al. (2008) explained these faulty errors of the
individuals by conceptualizing the relationship between CoV
and multivariable analysis. They argued that both CoV and
multivariable analysis and prediction can be regarded in an
analysis of variance (ANOVA) framework. The first assump-
tion of ANOVA is that causes have consistent effects under
the same considerations, and second that multiple effects
may operate jointly on an outcome, in either additive or
interactive manner. However, the faulty errors made by
Kuhn and Dean’s (2004) participants as well the students in
this study, show a pattern, which violated the analysis of
variance assumptions of consistency and additivity of effects.
Kuhn and Dean (2004) characterized these patterns as
reflecting an immature mental model of multivariable caus-
ality. This study as well as earlier research (Dean Jr & Kuhn,
2007; Kuhn & Dean Jr, 2004; Rind, in press; Zohar, 1995)
makes it clear that children and even many adults can not
readily conceptualize additive main effects, much less the
interaction effects. This calls to upgrade the current
approach that conceptualizes scientific thinking using uni-
variable models. There is a need to move to multivariable
models to achieve a more authentic scientific thinking.
We conclude with an acknowledgment that the conclu-
sions drawn from this study are not representative of all of
Shanghai. We used a very small sample of students from
only two schools to make our case. However, what we have
tried to show here is that Shanghai has great potential to
bring highly effective change to its education system. It
adopted the univariable and CoV models in its science edu-
cation and has surprised the world with its extraordinary
performance in PISA since 2009. However, the contempor-
ary discourse in science education is moving slightly away
from univariable models toward multivariable models. This
may be something that Shanghai and the rest of the world
need to consider for the future educational reforms.
THE JOURNAL OF EDUCATIONAL RESEARCH 9
Funding
This work was part of the study entitled an empirical study of princi-
pals’ living condition in primary and secondary schools, sponsored by
Chinese National Social Sciences Fund Education Youth Project
(No. CHA180269).
ORCID
Irfan Ahmed Rind http://orcid.org/0000-0002-2686-6166
Bo Ning https://orcid.org/0000-0003-3741-2693
References
Arvidsson, T. S. (2018). Individualized scaffolding of scientific reasoning
development-complementing teachers with an auto-agent. Columbia
University.
Australian Curriculum. (2015). Australian senior secondary curriculum
for science. http://www.australiancurriculum.edu.au/science/curricu-
lum/f-10?layout=1
Bo, N., Rind, I. A., & Asad, M. (2020). Influence of teacher educators on
the development of prospective teachers’ personal epistemology and tol-
erance. Sage Open, 10(1), 215824402091463–215824402091414. https://
doi.org/10.1177/2158244020914639
Coughlan, S. (2014). Shanghai teachers flown in for maths. BBC News.
https://www.bbc.com/news/education-26533428
Dean, D., Jr., & Kuhn, D. (2007). Direct instruction vs. discovery: The
long view. Science Education, 91(3), 384–397.
Dillon, S. (2010). Top test scores from Shanghai stun educators. The
New York Times. https://www.nytimes.com/2010/12/07/education/
07education.html
Duschl, R. (2008). Science education in three-part harmony: Balancing
conceptual, epistemic, and social learning goals. Review of Research in
Education, 32(1), 268–291. https://doi.org/10.3102/0091732X07309371
Gibb, N. (2015). The maths teachers of Shanghai have the perfect for-
mula for learning. The Guardian. https://www.theguardian.com/
commentisfree/2015/nov/26/maths-teachers-shanghai-china-uk
Guan, Q., & Meng, W. (2007). China’s new national curriculum
reform: Innovation, challenges and strategies. Frontiers of Education
in China, 2(4), 579–604. https://doi.org/10.1007/s11516-007-0043-6
Hongbing, J. (2010). Using International Standards to Examine
Chinese Education. 国际标准“体检”中国教育. Section 12. People’s
Daily 人民日报.
Huang, X., Ding, L., & Hu, B. (2017). Science curriculum and imple-
mentation in senior secondary school. In Liang, Ling L., Liu,
Xiufeng, Fulmer, Gavin W. (Eds) Chinese science education in the
21st century: Policy, practice, and research (pp. 101–132). Springer.
Inhelder, B., & Piaget, J. (1958). The growth of logical thinking from
childhood to adolescence: An essay on the construction of formal oper-
ational structures (Vol. 22). Psychology Press.
Jensen, B., & Farmer, J. (2013). School turnaround in Shanghai: The
empowered-management program approach to improving school per-
formance. Center for American Progress.
Jewett, E., & Kuhn, D. (2016). Social science as a tool in developing
scientific thinking skills in underserved, low-achieving urban stu-
dents. Journal of Experimental Child Psychology, 143, 154–161.
https://doi.org/10.1016/j.jecp.2015.10.019
Jiaoshou, G. L. B. (2004). China’s science curriculum reform. Asia-
Pacific Forum on Science Education, 5(2). https://www.eduhk.hk/
apfslt/v5_issue2/foreword/cindex.htm
Keil, F. C. (2006). Explanation and understanding. Annual Review of
Psychology, 57, 227–254. https://doi.org/10.1146/annurev.psych.57.
102904.190100
Klahr, D. (2002). Exploring science: The cognition and development of
discovery processes. MIT press.
Koerber, S., Sodian, B., Thoermer, C., & Nett, U. (2005). Scientific rea-
soning in young children: Preschoolers’ ability to evaluate
10 I. A. RIND AND B. NING
covariation evidence. Swiss Journal of Psychology, 64(3), 141–152.
https://doi.org/10.1024/1421-0185.64.3.141
Koslowski, B. (1996). Theory and evidence: The development of scientific
reasoning. Mit Press.
Kuhn, D. (2007). Reasoning about multiple variables: Control of varia-
bles is not the only challenge. Science Education, 91(5), 710–726.
https://doi.org/10.1002/sce.20214
Kuhn, D. (2016). What do young science students need to learn about
variables? Science Education, 100(2), 392–403. https://doi.org/10.
1002/sce.21207
Kuhn, D. (2018). A role for reasoning in a dialogic approach to critical
thinking. Topoi, 37(1), 121–128. https://doi.org/10.1007/s11245-016-
9373-4
Kuhn, D., Arvidsson, T. S., Lesperance, R., & Corprew, R. (2017). Can
engaging in science practices promote deep understanding of them?
Science Education, 101(2), 232–250. https://doi.org/10.1002/sce.21263
Kuhn, D., & Dean Jr, D. (2004). Connecting scientific reasoning and
causal inference. Journal of Cognition and Development, 5(2),
261–288. https://doi.org/10.1207/s15327647jcd0502_5
Kuhn, D., Iordanou, K., Pease, M., & Wirkala, C. (2008). Beyond con-
trol of variables: What needs to develop to achieve skilled scientific
thinking? Cognitive Development, 23(4), 435–451. https://doi.org/10.
1016/j.cogdev.2008.09.006
Kuhn, D., & Pease, M. (2008). What needs to develop in the develop-
ment of inquiry skills?. Cognition and Instruction, 26(4), 512–559.
https://doi.org/10.1080/07370000802391745
Kuhn, D., Pease, M., & Wirkala, C. (2009). Coordinating the effects of
multiple variables: A skill fundamental to scientific thinking. Journal
of Experimental Child Psychology, 103(3), 268–284. https://doi.org/
10.1016/j.jecp.2009.01.009
Kuhn, D., Ramsey, S., & Arvidsson, T. S. (2015). Developing multivari-
able thinkers. Cognitive Development, 35, 92–110. https://doi.org/10.
1016/j.cogdev.2014.11.003
Liang, X., Kidwai, H., & Zhang, M. (2016). How Shanghai does it:
Insights and lessons from the highest-ranking education system in the
world. The World Bank.
Liu, P. (2016). Transforming turnaround schools in China: Strategies,
achievements, and challenges. Frontiers of Education in China, 11(3),
374–414. https://doi.org/10.1007/BF03397128
Lu, J., & Xiaohu, L. (2011). How to view 2009 Shanghai PISA result?
The review of the reflection on Shanghai middle school students’ first
participation in the international test. 如何看待上海2009 年PISA 测
评 结 果 —中 国 上 海 中 学 生 首 次 参 加 国 际 测 评 结 果 反 响 述 评 .
Shanghai Research on Education 上海教育科研, 11–19.
Ma, M. (2016). The development and characteristics of elementary sci-
ence curriculum in China. US-China Education Review, 6(11),
642–649.
National Curriculum of England. (2013). National curriculum of
England: Science programmes of study. https://www.gov.uk/govern-
ment/publications/national-curriculum-in-england-science-pro-
grammes-of-study
Next Generation Science Standards. (2013). Next generation science
standards. https://www.nextgenscience.org/
OECD. (2010). PISA & TALIS 2008 technical report. OECD Pub.
OECD. (2011). Successful reformers in education: Lessons from PISA for
the United States. OECD.
OECD. (2015). Summary description of the seven levels of proficiency in
science in PISA 2015. Organisation for Economic Co-Operation and
Development. https://www.oecd.org/pisa/test/summary-description-
seven-levels-of-proficiency-science-pisa-2015.htm
Rind, I. A. (2014). Investigating students’ experiences in an ESL pro-
gramme at the University of Sindh, Jamshoro, Pakistan. Compare: A
Journal of Comparative and International Education, 44(6),
1002–1004. https://doi.org/10.1080/03057925.2013.822244
Rind, I. A. (2015). Gender identities and female students’ learning
experiences in studying English as Second Language at a Pakistani
University. Cogent Education, 2(1), 1115574. https://doi.org/10.1080/
2331186X.2015.1115574
Rind, I. A. (2016). Conceptualizing students’ learning experiences in
English as Second Language in higher education from structure and
agency. Cogent Social Sciences, 2(1), 1191978. https://doi.org/10.
1080/23311886.2016.1191978
Rind, I. A. (In press). Developing adult students’ multivariable thinking
capabilities. International Journal of Science and Mathematics
Education, 19.
Rind, I. A., & Kadiwal, L. (2016). Analysing institutional influences on
teaching– learning practices of English as second language pro-
gramme in a Pakistani university. Cogent Education, 3(1), 1160606.
https://doi.org/10.1080/2331186X.2016.1160606
Rind, I. A., Mari, M. A., & Heidari-Shahreza, M. A. (2019). Analysing
the impact of external examination on teaching and learning of
English at the secondary level education. Cogent Education, 6(1).
https://doi.org/10.1080/2331186X.2019.1574947
Rind, I. A., Shahriar, A., & Fatima, S. (2016). Rural-ethnic identities &
students’ learning experiences in english as second language pro-
gramme in a public sector university of pakistan. The Sindh
University Journal of Education, 45(1), 1–20.
Ringmar, S. (2013). Here’s the truth about Shanghai schools: They’re
terrible. The Guardian. https://www.theguardian.com/commentis-
free/2013/dec/28/shanghai-china-schools-terrible-not-ideal
Tan, C., & Ng, C. S. L. (2018). Assessment reform in Shanghai: Issues
and challenges. International Journal of Educational Reform, 27(3),
291–309. https://doi.org/10.1177/105678791802700304
Wasis. (2014). Analyzing physics items of UN, TIMSS, and PISA based
on higher-order thinking and scientific literacy. In International
Conference on Research, Implementation and Education of
Mathematics and Sciences (pp. 147––154). https://www.academia.
edu/download/54606793/17-Wasis.pdf
Williams, J. J., Lombrozo, T., & Rehder, B. (2013). The hazards of
explanation: Overgeneralization in the face of exceptions. Journal of
Experimental Psychology: General, 142(4), 1006–1014. https://doi.org/
10.1037/a0030996
Xie, X. (2010). What does Shanghai PISA test tell us? The Balanced
development of education is not impossible. PISA 测试上海夺冠回答
了什么—教育均衡并非可望不可. 中国青年报. China Youth.
Yao, J.-X., & Guo, Y.-Y. (2018). Core competences and scientific liter-
acy: The recent reform of the school science curriculum in China.
International Journal of Science Education, 40(15), 1913–1933.
https://doi.org/10.1080/09500693.2018.1514544
Zhang, W., & Bray, M. (2018). Equalising schooling, unequalising pri-
vate supplementary tutoring: Access and tracking through shadow
education in China. Oxford Review of Education, 44(2), 221–238.
https://doi.org/10.1080/03054985.2017.1389710
Zhang, C., & Akbik, A. (2012). PISA as a legitimacy tool during China’s
education reform: Case study of Shanghai. TranState Working Papers
No. 166, Universit€at Bremen.
Zhao, X. (2011). Development under stress: The culture of academic
competition and adolescent friendship participation in China’s second-
ary school. Harvard Graduate School of Education.
Zimmerman, C. (2000). The development of scientific reasoning skills.
Developmental Review, 20(1), 99–149. https://doi.org/10.1006/drev.
1999.0497
Zimmerman, C. (2007). The development of scientific thinking skills in
elementary and middle school. Developmental Review, 27(2),
172–223. https://doi.org/10.1016/j.dr.2006.12.001
Zohar, A. (1995). Reasoning about interactions between variables.
Journal of Research in Science Teaching, 32(10), 1039–1063. https://
doi.org/10.1002/tea.3660321005