Performance analysis of Israeli students in PISA math tests

by

Zbigniew Marciniak, Agnieszka Sułowska, Antonina Sułowska

May, 2022
1
Contents

Introduction…………………………………………………………………………………………………………………………….. 4
What can we learn about the performance of the Israeli students from
the international PISA Assessment Reports?................................................................................ 5
Performance of Israeli students from the perspective of test items…………….…………………….…… 10
Observations concerning the results of Israeli students in different content domains ………..… 15
Conclusions and recommendations………………………………………..……………………………………………… 22

2
Introduction

The rapidly changing world creates new challenges for education. It is a truism that fast
development of technology creates increasing demand for good mathematical education, not
only in industry but also in many other domains of social life demanding more and more complex
decision making, also based on understanding of big data. Moreover, many traditional math tasks
are taken over by machines (like, e.g., the understanding and use of train timetables). At the
same time, we observe the rapid growth of the number of new tasks which cannot be resolved
by a direct application of a math tool traditionally practiced at school, but rather require a
creative approach. In consequence, contemporary mathematical education should prepare
students to be ready to tackle problems with math tools in ways not practiced earlier in
classrooms.
The OECD PISA assessment has put this issue as the corner stone of its construction and created
the concept of mathematical literacy. The PISA Framework for PISA 2012 described it as follows:
An individuals’ capacity to formulate, employ, and interpret mathematics in a variety of contexts. It
includes reasoning mathematically and using mathematical concepts, procedures, facts and tools to
describe, explain and predict phenomena. It assists individuals in recognizing the role that mathematics
plays in the world and to make the well-founded judgements and decisions needed by constructive,
engaged and reflective citizens.1

Moreover, over the last decade a growing demand for more complex math skills, including the
mathematical reasoning, has been observed. The Board of Participating Countries has approved
the following modification of the above definition for the 2021 and consecutive cycles of PISA:
Mathematical literacy is an individual’s capacity to reason mathematically and to formulate, employ, and
interpret mathematics to solve problems in a variety of real-world contexts. It includes concepts,
procedures, facts and tools to describe, explain and predict phenomena. It assists individuals to know the
role that mathematics plays in the world and to make the well-founded judgments and decisions needed
by constructive, engaged and reflective 21st century citizens.2

Because of the unique perspective taken by PISA, directly relating education to the demands of
the real world, measured in the domains of mathematics, reading and science, the PISA test is
now used as an important assessment tool in many regions around the world. It was
implemented in 43 countries and economies in the first assessment (32 in 2000 and 11 more in
2002), 41 in the second assessment (2003), 57 in the third assessment (2006) and 75 in the fourth
assessment (65 in 2009 and 10 in 2010). So far, 65 countries and economies have participated in
PISA 2012. In 2018, the last fully completed PISA cycle, 79 countries and economies decided to
participate.

1
PISA Framework 2012
2
PISA 2021 Mathematics Framework, OECD 2018

3
One can also observe growing political interest in PISA results. On the publication day, the results
of a new PISA cycle become headlights in all main global media. However, the public interest is
usually restricted to the lead tables and to its changes with respect to the previous assessment.
The PISA Assessment Report published by the OECD after every cycle provides many detailed
analyses of the outcomes. What is most important, OECD makes all assessment data available on
its website to anybody interested. The analysis of this set of data can provide useful hints how
education could be improved.
The authors of this report have used this opportunity in 2004 in the case of mathematical
education in Poland, after the release of the PISA 2003 results. Our country’s result was then
listed below the OECD average in mathematics. The subscales from the international report
revealed no particular reason for that. Thus, we then decided to conduct a deepened study of
the results of Polish students on the item-by-item basis. This has led us to interesting conclusions
which have become the base of the national curriculum reform in 2008 and the consequent
revision of the national exams. We believe that the substantial improvement of the Polish PISA
results since 2012 can also be attributed to those changes.
The present study contains the results of a similar study of the PISA results of Israeli students.
We took the 2012 PISA Assessment results as the key reference point, as it was the last PISA test
with mathematics as the main domain. On such occasions the set of test items is richer than
usual, which is important, as possible conclusions can only stem from statistical observations. In
the two consecutive studies – in 2015 and 2018 – a subset of those items has been tested and,
obviously, we verified our observations against those data sources as well.
We hope that our report will prove useful in the Israeli discussion on the mathematical education.

4
What did we learn about the performance of Israeli students from the
international PISA Assessment Reports?

We start with the most important conclusions of the OECD PISA Assessment Reports concerning
mathematics in Israel.
In the general comments sections, the reports mention Israel several times, as follows:

• Between the 2000 and 2012 PISA assessments, Albania, Israel and Poland increased the
share of top performers and simultaneously reduced the share of low performers in
reading.
• Between 2006 and 2012, Italy, Poland and Qatar, and between 2009 and 2012, Estonia,
Israel and Singapore increased the share of top performers and simultaneously reduced
the share of low performers in science.
• Between 2003 and 2012 Italy, Poland and Portugal reduced the proportion of low
performers and increased the proportion of high performers. This was also observed in
Israel, Qatar and Romania between 2006 and 2012, and in Ireland, Malaysia and the
Russian Federation between 2009 and 2012.
• In 24 countries and economies out of the 63 that took part in both PISA 2015 and 2018
(Austria, Belgium, Brazil, the Czech Republic, Chile, Costa Rica, Croatia, Estonia, France,
Germany, Greece, Hong Kong [China], Hungary, Ireland, Israel, Korea, Lebanon, Lithuania,
Mexico, Moldova, New Zealand, Sweden, the United Arab Emirates and the United
States), no significant change in performance was observed, between 2015 and 2018, in
any of the three core subjects that PISA assessed.
• The ten PISA participants with the widest spread in scores (score-point difference
between the top and bottom 10% of students) are Israel, Belgium, the Slovak Republic,
New Zealand, France and Korea as well as the partner countries and economies Chinese
Taipei, Singapore, Shanghai-China and Qatar.
• In Israel, Austria, Italy, New Zealand and Luxembourg, which are situated in the middle of
the performance distribution, the share of boys who attain at the highest proficiency
levels is considerably larger than the share of girls who do, by a difference of 7.7 to 5.8
percentage points.

5
Israel did not participate in the 2003 edition of the PISA Assessment, where mathematics has
been the main domain for the first time. However, we have the average results of Israeli students
in all subsequent editions of the PISA test:
2003: -
2006: 442
2009: 447
2012: 466
2015: 470
2018: 463
Recall that the PISA scale is calibrated so that the OECD countries’ average is equal to 500 points
and the standard deviation is equal to 100 points. It means that Israel’s score in mathematics
remains all the time below the OECD average.
To compute the values assigned to countries, PISA applies the probabilistic Rasch model, which
locates each tested student, as well as each item on the above scale. It is arranged so that a
student located at the same point of the scale as an item has a 50% chance to solve this item.
Moreover, the PISA scale is divided into intervals, defining six levels of mathematical proficiency.
Level 6 (669 points of higher) represents the highest level of skills; level 1 (between 358 points
and 420) is the lowest.
The mathematical competences related to the levels are described in the OECD Reports, e.g.3,
page 61, Fig. I.2.21.
The description of mathematical competences at level 1 is as follows:
At Level 1, students can answer questions involving familiar contexts where all relevant information is
present and the questions are clearly defined. They are able to identify information and to carry out routine
procedures according to direct instructions in explicit situations. They can perform actions that are almost
always obvious and follow immediately from the given stimuli.

It means that students at this level of mathematical proficiency are able only to reproduce the
basic skills acquired at school. One should also notice that in each country some students fall
below level 1, which means that they are missing even those very basic skills.

3
PISA 2012 Results: What Students Know and Can Do, vol. 1, OECD 2014

6
Here is the distribution of Israeli students among the proficiency levels in mathematics starting
from the year 2012:

Level <1 1 2 3 4 5 6
2012 15.9 17.6 21.6 21.0 14.6 7.2 2.2
2015 15.0 17.1 21.1 21.7 16.1 7.1 1.9
2018 17.7 16.4 20.7 21.0 15.4 7.0 1.8
Source: PISA 2015 Results, Table 1.4.2a; PISA 2018 Results, Table I.B1.2
The graphic representation of this distribution in 2012 is available in the PISA Report quoted in
footnote 3), on page 52, Fig. I.2.22.
International studies of PISA results used to analyze in detail the fractions of students at both
ends of the scale: strictly below level 2 and strictly above level 4. For Israel and the OECD
countries average it is as follows:

Israel OECD Israel OECD

year level 1 or level 1 or levels levels
below below 5 and 6 5 and 6
2006 42.0 22.5 6.1 12.5
2009 39.5 22.0 5.9 12.5
2012 33.5 22.2 9.4 12.9
2015 32.1 22.6 8.9 12.6
2018 34.1 23.9 8.8 10.9
Source: PISA 2015 Results, Table 1.4.2a; PISA 2018 Results, Table I.B1.2
We see that over the 12-year period, the fraction of the weakest Israeli students was decreasing.
A little disturbing is the increase of this fraction in the 2018 cycle. From the PISA data, as well
from the experience of many participating countries, it follows that it is much easier to improve
the results of the weakest students than to increase the fraction of the best students. The latter
was achieved over the last 20 years by very few countries only.
When mathematics is the main domain of the PISA study (so far: 2003, 2012 and just now, in
2022), the international PISA report also presents the distribution of results with respect to
various subscales. Let us see what this additional information from the year 2012 can add to the
general picture of the Israeli mathematical education.

7
The PISA Framework describes the modeling cycle, which describes the way in which we apply
mathematics to solve real life problems. It consists of the consecutive application of three
processes:
• formulating situations mathematically,
• employing mathematical concepts, facts, procedures and reasoning,
• interpreting, applying and evaluating mathematical outcomes,
referred to in abbreviated form as “formulate, employ and interpret”.

The Israeli students’ results on those subscales in 2012 were as follows:

Formulate:
Average result of the Israeli students: 465. It gives position no. 40 on the country list.
Similar countries: United States, Hungary, Croatia.
The average result of boys is larger than the average result of girls by 15 points.
Employ:
Average result of the Israeli students: 469. It gives position no. 41 on the country list.
Similar countries: United States, Croatia, Sweden.
The average result of boys is larger than the average result of girls by 9 points.
Interpret:
Average result of the Israeli students: 462. It gives position no. 42 on the country list.
Similar countries: Slovak Republic, Russian Federation, Lithuania, Greece.
The average result of boys is larger than the average result of girls by 18 points.
The differences of Israeli students on the subscales related to those processes are not large. They
might indicate that Israeli schools devote most attention to the middle stage of the modelling
cycle, i.e., to purely formal activities. Also, in the scope of the process Employ the difference
between the results of boys and girls is the smallest. It need not mean that the applications of
mathematics are not present in classes – it may very well be that it is simply indistinguishable for
the students from formal mathematics. Mastering ‘formulating’ and ‘interpreting’ requires free
discussions and making many errors on the way to the final solution; when these processes are
strictly guided by a teacher, they just become procedures to be remembered for future use.

8
The above conjecture is supported by the Israeli students’ replies to some questions in the
Students’ Questionnaire. When asked to evaluate the exposure to applications of mathematics,
their answers locate Israel at position 53 (out of 65). For example, when asked if they have ever
encountered any problem about tiling a floor, they mostly answer ‘no’ (Israel at position 57). On
the other hand, they declare good familiarity with abstract mathematical concepts, such as linear
equations (Israel at position 19), the quadratic function (Israel at position 33) or even the complex
numbers (Israel at position 16).
***
Another set of subscales concerns the content domains. To avoid confusion related to the
differences between various traditions of understanding the standard names of subdomains of
mathematics (algebra, geometry, etc.), PISA has assigned four original names to content
domains:
• Quantity

• Space and shape

• Change and relationships
• Uncertainty and data
We recall the definitions of those domains in the next section. In fact, this will be the main part
of our analysis of the Israeli students’ performance in mathematics.

9
Performance of the Israeli students from the perspective of test items

Although the proficiency level descriptors give some idea of the staggered scale of difficulty of
the PISA test, it provides only very general picture of the domain of mathematics in PISA. It is not
easy to interpret results of PISA from the perspective of those descriptions and recommend
practical actions taken by schools or teachers, aimed at the improvement of students’
mathematical skills.
Therefore, we decided to analyze the performance of the Israeli students within the content
domains together with a detailed analysis of their performance when solving similar groups of
items.
We have gathered the results of our detailed item-by-item analysis in Appendix A. We encourage
the Readers to browse through this vast material to make their own observations and draw
further conclusions. Here is a guide to the structure of an item description.

Each PISA item has a dedicated page in the enclosed spreadsheet – Appendix A. In the upper left
corner of that page you will find the basic metrics of the item:
Item id – that is the item code, as it appears in the data sets in the OECD databases;
Item name – that is the official name of the item, usually closed with term like ‘Q01’ indicating
that it was question 1 in a unit consisting of one or more items related to the same context
situation. The name placed on a red background means that the item text is still confidential. The
item name placed on a green background means that the item text has been released. If that
happened, the item text is included, for the convenience of the Readers, in Appendix B;

10
Item format – one of the few formats of response expected from the students.
The response types distinguish between selected response items and constructed response
items. Selected response items include simple multiple choice, complex multiple choice, in which
students must select correct answers to a series of multiple-choice items, and, for computer-
based items, “selected response variations”, such as selecting from options in a drop-down box.
Constructed response items include those that can be scored routinely (such as a single number
or simple phrase, or, for computer-based items, those for which the response can be captured
and processed automatically), and others that need expert scoring (e.g., responses that include
an explanation or a long calculation)4;
Content – one of the four content domains: Quantity, Space and shape, Change and relation-
ships, Uncertainty and data. We quote below the official definitions of these domains from the
PISA Framework.
The content category Quantity incorporates the quantification of attributes of objects,
relationships, situations, and entities in the world, which requires an understanding of various
representations of those quantifications, and judging interpretations and arguments based on
quantity. It involves understanding measurements, counts, magnitudes, units, indicators, relative
size, and numerical trends and patterns, and employing number sense, multiple representations
of numbers, mental calculation, estimation, and assessment of reasonableness of results.
The content category Uncertainty and data covers two closely related sets of issues: how to
identify and summarize the messages that are embedded in sets of data presented in different
ways, and how to appreciate the likely impact of the variability that is inherent in many real
processes. Uncertainty is part of scientific predictions, poll results, weather forecasts and
economic models; variation occurs in manufacturing processes, test scores and survey findings;
and chance is part of many recreational activities that individuals enjoy. Probability and statistics,
taught as part of mathematics, address these issues.
The content category Change and relationships focuses on the multitude of temporary and
permanent relationships among objects and circumstances, where changes occur within systems
of interrelated objects or in circumstances where the elements influence one another. Some of
these changes occur over time; some are related to changes in other objects or quantities. Being
more literate in this content category involves understanding fundamental types of change and
recognizing when change occurs so that suitable mathematical models can be employed to
describe and predict change.
The content category Space and shape encompasses a wide range of phenomena that are
encountered everywhere: patterns, properties of objects, positions and orientations,
representations of objects, decoding and encoding of visual information, navigation, and
dynamic interaction with real shapes and their representations. Geometry is essential to space

4
A quote from the PISA 2012 Report, page 40.

11
and shape, but the category extends beyond traditional geometry in content, meaning and
method, drawing on elements of other mathematical areas, such as spatial visualization,
measurement and algebra. Mathematical literacy in space and shape involves understanding
perspective, creating and reading maps, transforming shapes with and without technology,
interpreting views of three-dimensional scenes from various perspectives, and constructing
representations of shapes.5
Context - the four context categories identify the broad areas of life in which the problems may
arise:

• personal, which is related to individuals’ and families’ daily life;

• societal, which is related to the community – local, national or global – in which an
individual life;
• occupational, which is related to the world of work; or
• scientific, which is related to the use of mathematics in science and technology.
According to the framework, these four categories are represented in the test by equal numbers
of items.
Process – Formulate, Employ or Interpret, as described in the previous section;
World corr 2012 – the fraction of correct answers of students across the world;
Israel corr 2012, 2015, 2018 – the fraction of correct Israeli students’ answers in PISA 2012, 2015,
2018. If a number is missing, that means the item was not used in later cycles of PISA;
Israel’s position 13/69 – this means that the fraction of correct answers of Israeli students places
them at position 13 out of 69 countries whose students were solving this item;
PISA scale – the value of the item on the PISA scale;
PISA level – the mathematical competency level to which the value of the item belongs.
On top of the page, to the right from the item metrics, we have described the main mathematical
demand of the item formulated, without revealing the text of the confidential items.
Beneath the description, to the left, we have placed the list of countries whose students were
solving this particular item, in the decreasing order of the fraction od students who have
succeeded to answer correctly (the corr-all column). The next two columns give the fractions of
correct responses by boys (corr-boys) and by girls (corr-girls) respectively. The column tagged
missing gives the fraction of students, who did not even attempt to solve the item.
Among the listed countries you can also find the positions AVG and TOT. The first number is the
average of the means of the results of the countries which are the OECD members. The second

5
A quote from the PISA 2012 Report, page 38

12
number is the average result of all students who live in the OECD countries – as if they all lived in
one country.
Finally, below to the right you will find a graph showing the decile distribution of results of Israeli
students versus the results of the students of the world. These graphs were created as follows.
All Israeli students have been ordered by their total score on the PISA test. Then this list has been
divided into 10 groups. In particular, group number 1 represents 10% of the Israeli students with
the lowest score on the PISA test; group number 10 represents 10% of the Israeli students with
the highest score on the PISA test. For each of those groups the fraction of students from this
group who did solve the problem is marked – this gives the blue line on the diagram. The orange
line is created in the same way, but with respect to the population of all students of the world
who were solving this particular item on the PISA 2012 test.
Comparing such distributions was a key tool in our past study of the Polish results – we apply
here the same methodology.
A crucial decision had to be made which background data to choose to compare with the
performance of Israeli students. We decided to take as background the whole world. Another
possibility would be to choose the average of the OECD countries or one specified country. We
had to resolve the same dilemma in 2003 and decided then to compare our students with the
entire world, for the simple reason that only this choice will give a chance to identify some
advantages of our students.
To illustrate this dilemma, let us look at the following diagram, which represents the results for
16 items, of diversified difficulty, which – in our opinion – did require a substantial input of
mathematical reasoning.

13
 90

80

70

60

50

40

30

20

10

World (AVG) Israel 2012 Poland 2012

The blue bars represent the world average fraction of correct responses; the orange dots show
the Israeli students’ results and the gray dots – results of Polish students. Due to the selection of
the world as the reference point, we see that the Israeli students in some items get results better
than the world average. We have included the Polish results to stress that our students’ results
in 2003 were falling below the Israeli dots from 2012.
Over the years, when our students have progressed, we started doing such analyses relative to
the average result of the OECD countries. We strongly believe that soon the time will come when
the Israeli students will be compared to the OECD as well and their results will be at the gray
spots or higher.

14
Observations concerning the results of Israeli students in different content
domains

From the review of all items, it clearly follows that there occur noticeable differences in the
results of Israeli students in the four content domains. Also, the domain averages, quoted in the
International Report confirm this observation.
The Israeli students achieved the best average result, equal to 480, solving problems from the
domain Quantity. The domains Change and relationships and Uncertainty and data were for them
of almost equal difficulty, but brought significantly lower results, 462 and 456 respectively. The
domain Space and shape caused most trouble – the average result was here only 449 points.
These data were reported in 2012; in the next two PISA cycles mathematics was not the main
domain and the content domains subscales were not reported.
Let us have a closer look at these domains. By design, there are equally represented in the test –
each was represented by 21 items. The following table reports how many items in each domain
was, harder, easier or of comparable difficulty for Israeli students compared to an average
student of the world:

Quantity Change and rel. Uncertainty Space and shape

Israel < World 7 12 14 18
Israel ≈ World 2 3 2 1
Israel > World 12 6 5 2

From this perspective, the problem with solving items representing Space and shape is even more
transparent.
Another confirmation of this observation comes from the statistic of omissions. Let us recall that
an item is treated as missing only if it was skipped by the student while solving the test, without
leaving even the slightest trace in the booklet which might be interpreted as an attempt to solve
this item.

15
In general, Israeli students omit items on average more often than students of the world. The
situation that an item is omitted by a smaller fraction of Israeli students than in the world has
happened really seldom. From among 84 items, that happened only 6 times. Two of those items
belonged to the Quantity domain, 3 in the Change and relationships domain and 1 in the Space
and shape domain. Here is the list of those items:
PM828Q01 (Carbon Dioxide), Change and relationships, PISA level 5
PM943Q01 (Arches), Change and relationships, PISA level 3
PM486Q02 (Cash withdrawal), Change and relationships, PISA level 2
PM192Q01T (Containers), Quantity, PISA level 4
PM800Q01 (Computer game), Quantity, PISA level below 1
PM033Q01 (View of the room), Space and shape, PISA level 2
The following table compares the distribution of the average omission fractions calculated for
each content domain in Israel and worldwide, and also their mutual proportion.

%% missing Israel %% missing World Israel/World

Quantity 8.4 6.8 1.23
Uncertainty and data 10.0 7.5 1.33
Change and relationships 21.8 17.3 1.26
Space and shape 19.4 14.1 1.58

Again, the domain Space and shape appears to be the biggest problem. However, we should not
forget that the item difficulty need not be the only reason for omissions. We have observed in
Poland instances of easy items which were omitted by students because they were bored with it
due to intensive exploitation of a similar type of problem at school.
We should also note that the four content domains had a different average difficulty of items.
The Quantity items were the easiest, with average difficulty of 487. Next was the Uncertainty and
data, with average difficulty of 522. The domain Change and relationships was substantially more
difficult, with average difficulty of 580, but the Space and shape was the most important domain,
with average difficulty of 599.
The next table shows the distribution of items among the PISA competency levels.

< Level 1 Level 1 Level 2 Level 3 Level 4 Level 5 Level 6

Space and shape 0 1 3 4 4 3 6
Change and relationships 0 0 5 3 4 6 3
Uncertainty a data 2 2 3 5 5 3 1
Quantity 2 2 3 8 5 1 0
Σ 4 5 14 20 18 13 10

16
Of course, these differences between domains were not intentional by design. They show that
geometry seems to be (on average) difficult globally. Nevertheless, we must stress that in many
countries, including Poland, no essential differences between the results belonging to different
content domains occured.
The observed differences should be also useful in looking for ways to improve the results of Israeli
students at the top levels: 5 and 6, what – as we understand ─ is one of the priorities of TFF. From
this perspective, Space and shape problems seem to be a good investment.
The general conclusion following from the above data is that Israeli students are relatively weaker
in solving geometry problems. It is worth noticing, that although the average difficulties of items
belonging to Space and shape and Change and relationships are not very different (19 points
difference on the PISA scale), the items belonging to the geometry domain created more
problems for Israeli students.
On the other hand, the average difficulties of items from the domains Change and relationships
and Uncertainty and data differ substantially (by 42 points on the PISA scale), yet Israeli students
were able to solve items belonging to these two domains with similar success. It follows that the
item difficulty alone does not necessarily provide the complete explanation of the difficulties
related to the content domains.
It seems that successful solvability of items belonging to the domain Change and relationships,
despite their relative higher difficulty, can be explained by their similarity to formal school
mathematics.
As the Students’ Questionnaire shows, Israeli students perceive school mathematics as formal
knowledge, which they learn by practicing various formal procedures. Therefore, even though
their knowledge could be easily applied in real life, they still claim that very little applied
mathematics can be encountered in classrooms. It is quite probable that they were offered tasks
concerning the application of mathematical tools to the real world. However, students did not
perceive them as ‘true’ mathematics, as they were not accompanied by procedures to be
followed in the future.
If the above claims reflect the truth, the problem lies in pedagogy which concentrates on the
introduction to students a sequence of mathematical concepts and on training the fluency of
performing with them well defined set of math operations, instead of treating them as flexible
tools which can be creatively applied when solving problems, also those which were not practiced
in class.
Of course, the same task can be for one student a completely routine exercise, while another
student needs to employ deep reasoning to reach the solution. However, when we look at this
issue from the statistical perspective, we can discover certain regularities reflecting the most
common teaching practices. For example, when an item, or a group of items has a large fraction
of omissions, this can be interpreted as a situation when students face mathematical demands

17
going beyond the practiced school routines. At the same time, high omission rates indicate that
only very low fraction of students is ready to compensate the lack of a ready-to-use procedure
by applying their autonomous reasoning, or – at least – is ready to enact an attempt to do so.
We have identified in Poland in 2003 the same problem. We decided to introduce into the school
practice problems which require mathematical reasoning and strategic thinking. As we aimed at
the improvement of mathematical skills of the whole population of students, we strived to
introduce reasoning items of diversified difficulty – from very easy, to quite advanced. Of course,
it requires a substantial change in the teaching practices. We have achieved that by implementing
the new requirements in the national exams, which forced a shift in pedagogy. It was effective,
as teachers were strongly motivated by the need to prepare students to successful exams.
We do not claim that such a move is possible in every country, or – even when implemented –
will bring a positive change, like in Poland. However, we are completely convinced that it is
possible to offer reasoning problems easy enough to encourage also weaker students to engage
their creative thinking in the classes of mathematics. Examples of such problems you could see
in our middle school exams booklets during the study visit of TFF in Poland.
The main tool remains the right teaching which lets students work on their own to overcome
difficulties and find the solution. It is not supported by teachers who practice an authoritarian
way of teaching and who measure their professional success by the level of students’ efficiency
to reproduce typical procedures.
We have identified some types of items at which a significant fraction of Israeli students were
ready to perform an analysis which goes beyond school routines. Those items ask students to
analyze the functionality of a well-defined algorithm and discuss its possible modifications, like
in PM411Q01-Q02 (Diving).
On the other hand, we have identified 7 items that caused the most problems to Israeli students,
when compared with all tested students of the world.

World World
PISA Israel AVG of Israel AVG of
Item Name Content Scale Level corr-all correct missing missing
PM943Q02 Arches Q2 SS 785 6 2,2 6,9 36,7 26,1
PM00GQ01 Advertising Column Q1 SS 760 6 3,8 7,7 15,7 10,8
PM00KQ02 Wheelchair Basketball Q2 SS 700 6 4,9 14,0 17,9 10,1
PM406Q02 Running Tracks Q2 SS 696 6 8,2 14,2 58,6 42,5
PM464Q01T The Fence Q1 SS 652 5 13,9 21,0 23,7 18,3
PM406Q01 Running Tracks Q1 SS 645 5 12,0 22,0 34,8 24,3
PM423Q01 Tossing Coins Q1 U 401 1 64,5 74,5 2,7 1,8

18
Notice that six of those items belong to the Space and shape domain and all belong to the 5th or
6th mathematical proficiency PISA level. These items have the following common features:
- They involve a complex geometric figure or a non-routine situation like movement or
the need to interact between the plane and spatial geometry,
- The solution requires a design of process of solving, consisting of several steps,
- There is no formula to which one can plug in the data from the item,
- Students must compare properties of more than one figure.
All that requires that the student has to make autonomous, non-routine decisions while solving
the problem.
At the other end we have two items belonging to the Space and shape domain, where the
Israeli students got results better than or similar to an average world student:

World World
PISA Israel AVG of Israel AVG of
Item Name Content Scale Level corr-all correct missing missing
PM949Q01T Roof Truss Design Q1 SS 467 2 61,5 62,4 3,0 2,0
PM949Q02T Roof Truss Design Q2 SS 594 4 41,2 32,1 2,3 1,8

Both items require only the routine school competences – one step operations relying on making
comparisons of segments or angles in a typical school geometrical configuration. These examples
seem to support our conjecture as to the sources of geometrical difficulties of Israeli students.
Geometry is the most vulnerable domain, as it is not so easy to turn the teaching of geometry
into a set of procedures to remember. Still, in the above pair of items it was possible.
The last troublesome item deals with a probabilistic situation – it tests understanding of the
concept of events independence. However, this single item does not give sufficient base for a
general conclusion.
It is also worth noticing that for items which resemble typical school geometry, the omission rate
does not depart significantly from the world omission rate. However, in items less similar to
typical school exercises the Israeli students’ omission rate was substantially higher.
Additional information supporting the above conclusions comes from the decile distributions for
each item.
For PM00GQ01 (Advertising Column), even the best Israeli students stayed behind average world
students [Israel 19%, World 34%]. The same phenomenon occurs for PM00KQ02 (Wheelchair
Basketball) [Israel 21%, World 49] and PM943Q02 (Arches) [Israel 16%, World 31%]. For
PM406Q01 (Running Tracks) the Israeli students from the three top deciles fell behind the
average world students. A similar situation took place for PM464Q01T (The Fence).

19
When analyzing the results obtained for the remaining items belonging to the domain Space and
shape, the unit Spacers, consisting of three items PM992Q01-Q03 calls for attention. These items
require that a student employs mathematical reasoning in order to count objects in a less typical,
yet practical geometric situation. In this type of items Israeli students were able to achieve results
close to the world average.
A careful review of the units Bricks, Pipelines, Map, View of the room, Tile arrangement confirms
the claim that Israeli students obtain results below the world average almost always when the
item departs from the standard school geometry and the task requires taking an autonomous
action which goes outside the typical problem-solving schemes practiced in classrooms. It is clear
that such items always give students more trouble, but we want to stress the fact that an average
world student deals with such items more successfully.
***
Turning to the content domain Uncertainty and data we come across an interesting situation.
The result of Israeli students for PM423Q01 (Tossing Coins) belonged to the weakest. On the
other hand, PM408Q01T (Lotteries), Israeli students obtain a result better the world average
[Israel 38%, World 33%]. Both items were testing the understanding of the concept of
independent events. This pair of items shows that the result does not depend solely on the
mathematical content – the context in which the content is situated also plays an important role.
However, we had too few items of this type to formulate a responsible conclusion.
Israeli students were as good as average students of the world when it came to dealing with data
provided in tabular form, also when they had to confront it with another source of information.
It is confirmed by the items PM803Q01T (Labels) and PM955Q02 (Migration). We want to stress
the fact observed on the decile graphs that the relatively good results in those items was worked
out not only by the top Israeli students, but also by the good and average ones.
***
The content domain Quantity seems to be the easiest for Israeli students. Results for the items
from this domain prove that they can successfully deal with tasks which require an analysis of a
given procedure or algorithm. A good example is the pair PM411Q01-Q02 (Diving), where the
Israeli students starting from deciles 4 to 8 were better than their world counterparts. Israeli
students can also successfully deal with an analysis of an algorithm, even when it is quite
complex, like in PM442Q02 (Braille) or PM603Q01T (Number check). Comparable results were
obtained in items which required a verification of a given criterion like in PM906Q01-02 (Crazy
ants) or PM905Q01-02 (Tennis balls
Some Quantity items resemble typical school exercises. It is not difficult to identify a simple math
tool to perform the necessary calculations. However, part of these items require a mathematical
reasoning, for example to plan the sequence of calculations. This observation is confirmed by
items PM909Q01-Q03 (Speeding Fines) and PM559Q01 (Telephone rates). Such planning was

20
necessary for example in PM564Q01-02 (Chair lift) where Israeli students were better than the
world students across the whole decile scale.
***
Items belonging to the content domain Change and relationships that were easy for Israeli
students required good understanding of a procedure described in the item text and then
applying it to some data. It concerned items PM446Q01 (Thermometer Cricket), PM828Q01-03
(Carbon Dioxide) and PM99801-04 (Bike Rental). This corresponds well with the observation
about the ability to analyze and to apply algorithms in the content domain Quantity.
Much weaker results concern items which require mathematical reasoning when analyzing a
situation or a procedure, like in PM192Q01 (Containers), PM571Q01 (Stop the Car) a more
complex data analysis, like in PM155Q01-04 (Population Pyramids), PM982Q01-04 (Employment
data) or PM953Q02-04 (Flue Test).
Israeli students found also more difficult than the world students tasks which asked for an
interpretation of given formulas, like in PM954Q01-04 (Medicine Doses), PM571Q01 (Stop the
Car) or PM915Q01-02 (Carbon Tax).

21
Conclusions and recommendations

The PISA test is a statistical tool, with all consequences of this fact. First of all, the PISA estimate
is based on a measurement of a very carefully drawn random sample of students. This means
that it reflects the effects of the common teaching practices. Because of that it makes no justice
to the efforts of the best teachers who may achieve splendid teaching results, yet their number
is too low to be caught by the sampling sieve. This circumstance should be taken into account
also when reading our conclusions.
The general picture of the mathematical education in Israel at the lower secondary level does not
fit the expectations of the PISA mathematical framework, reflected by the set of PISA items. It
seems that Israeli students have sufficient formal mathematical knowledge to solve pure
mathematical problems proper for their age. However, we got the impression that majority of
students have not enough courage to attack problems which depart from the standard types of
problems on which they have worked at school. ‘To attack a problem’ in many instances means
‘to start working on it’. This can be caused by the style of teaching. Many teachers of mathematics
(around the world) attempt to make the subject easy by equipping their students with ‘ready to
use’ procedures. Once students practice using them long enough, both teachers and students
can get the impression that the pupils are fluent in mathematics.
PISA rightly calls to the attention the simple fact, that real life very rarely poses problems which
perfectly fit to the trained procedures. With the development of technology, the very basic
mathematical procedures are more and more often performed automatically by machines. At
the same time, we encounter the growing number of problems which come as a surprise and yet
can be easily solved with elementary mathematical tools, if we can ‘think mathematically’. PISA
calls this thinking ‘mathematical reasoning’.
It came as a surprise to Polish teachers that mathematical reasoning can be performed not only
by the top students, who are participating in Mathematical Olympiads. There exist many nice
mathematical problems which are very easy from the formal point of view and yet to solve it a
student has to discover an association which is not transparent at the first glance. Overcoming
this kind of difficulty builds student’s confidence and encourages him/her to try again.
However, the only way to enhance the skill of mathematical reasoning is to do it on your own. It
takes time to deal with unsuccessful attempts. It also requires a lot of patience on the side of the
teacher, because results come only with time. The temptation to show students how to do it is
very strong, mostly because of economy of time reasons.
In conclusion, building in students the attitude that ‘I can solve anything, whatever they give me’
is, in our opinion, the main goal of teaching mathematics. Thus, our first recommendation would
be to teach mathematical reasoning, by offering students problems at diversified level of
difficulty and asking them to find the way to solve them on their own.

22
The general perception of PISA is that it deals exclusively with applied mathematics. It is true that
most PISA items are presented in a realistic context. In fact, when constructing PISA items, a lot
of effort is devoted to the authenticity of the context. Nevertheless, PISA has the ambition to
measure the basic mathematical skills through the lens of the use of mathematics in real life.
Therefore, it is not a good idea to teach applications of mathematics just by showing to students
efficient and ready to use procedures of dealing with practical problems. A better strategy, in our
opinion, is the concentration on the fundamental skills and asking students to discover their
applications in a wide variety of contexts. This requires creative, independent attempts by the
students to overcome the difficulties. Teachers have the important role to offer to an individual
student tasks which will allow him/her to encounter a success in those attempts. When this
freedom of search is missing and the process of solving problems is strictly guided by a teacher,
the solutions just become procedures to be remembered for future use. Then, in the PISA
context, students try to fit the item to one of the trained procedures and drop the item if this
cannot be easily done. This is manifested by increased fractions of ‘missing’ items.
Finally, we strongly encourage the Israeli teachers to introduce more geometry problems in
classrooms. They are by nature less algorithmic, so mathematical reasoning has to be employed.
It can be supported by visualizations, both by spontaneous drawings as well as by a dedicated
software. As PISA items dealing with the geometric context are usually at the upper end of the
PISA mathematical competency scale, it is also the right way to increase the fraction of Israeli
students at levels 5 and 6.
Good luck!

23