Developing Non-routine Problems Related to Graphs for

Measuring Students’ Mathematical Abilities

Dedek Faradilla1, C M Zubainur1, Z Abidin2

1
Mathematics Education Department, Syiah Kuala University, Banda Aceh, 23111,
Indonesia
2
Mathematics Education Department, Ar-Raniry National Islamic University, Banda
Aceh, Indonesia

E-mail: cut.morina@fkip.unsyiah.ac.id

Abstract. Non-routine problems have been perceived as a media to challenge students’

mathematical abilities since they require more holistic or broader thinking in solving their problems
due to their unclear procedures. Unfortunately, many students’ handbooks nowadays do not serve
sufficient non-routine problems, thus it is necessary to develop such problems basically can train
and sharpen students’ critical thinking to solve mathematical issues. This study aims at developing
the non-routine problems related to graphs to measure mathematical abilities of the students in
junior high schools. The research design used is developmental research which is based on the
Tessmer’s development model (1993). The instruments used to collect data for this study were
validation scoring sheets, questionnaires of teacher and students, and non-routine problem sheets.
The results showed that 17 non-routine problems related to graphs had met the valid criteria based
on the validation scroring sheet results. Then, based on the analysis of the questionnaire the use of
graphs in solving non-routine problems shows satisfactory effect on developing students’
mathematical abilities. Therefore, it can be concluded that the non-routine questions that have been
developed within this study are valid, practical, and effective so that they can be used to measure
students' mathematical abilities in classroom learning.

1. Introduction
Mathematics learning that is carried out by teachers in the classroom is expected to foster mathematical
abilities among students [1]. Learning strategies that can foster students' mathematical abilities are by
training students to solve problems which can lead the students to think more deeply and critically [2].
The problems are called non-routine problems, kinds of problems which have unclear procedures because
they do not use steps, procedures, and algorithms that are usually obtained in class [3,4,5]. It is important
to give non-routine problems to students when learning mathematics because it can encourage students to
use their various mathematical abilities so that they will be familiar with the form of the questions
[6,7,8,9].

1
In mathematics learning in junior high school, the material taught is classified into four aspects, namely
numbers, algebra, geometry and measurement, as well as probability and statistics. One of the materials
that can be found in these four aspects is material related to graphs, namely function material, straight line
equations, quadratic equations, comparisons, and other materials. Graphs is known as a visual aid in
mathematics. The ability to make and interpret graphic forms is noted as a basic skill that is important for
students in learning mathematics [10,11]. Competencies that exist in the scope of mathematics material
stipulated by Permedikbud no. 21 of 2016 is a graphic, because graphs are widely used to illustrate
various mathematical problems. Therefore, graphs are important to learn in mathematics learning.

The results of the analysis of the seventh and eighth grade students' books for material related to graphs
showed that out of 173 questions, only 3 were included in the non-routine category. Problems for material
related to the dominant graph are in the form of routine questions which only require reading, making and
interpreting the graph in simple form. This is in line with the results of the analysis which shows that the
number of non-routine questions related to graphs in junior high school mathematics textbooks is still
inadequate [12]. Most of the content in junior high school mathematics textbooks contains instructions on
analyzing data on charts. However, the focus is on reading and interpreting charts, and how to make
graphs in simple form rather than in the form of annealing and interpreting further [13].

Responding to problems that arise in Indonesian mathematics education, an effort should be done so that
students can improve their mathematical abilities in solving non-routine questions related to graphs. One
way that students get used to solving non-routine questions related to graphs and can help teachers in non-
routine question-based learning is to provide a question bank and its solutions. The questions made were
in the form of non-routine questions related to graphs on mathematics material in Junior High Schools.
Therefore, it is necessary to develop non-routine questions related to graphs. This is expected to help
teachers provide the questions needed to train students in an effort to foster mathematical abilities.

2. Methods
The development model in this study is the formative evaluation design proposed by Tessmer (1993). This
design constitute mathematical description of indicators, mathematical problem-solving, valid, and
practical. In addition, it consists of two stages, namely preliminary stage of preparation and formative
evaluation; self-evalution, prototyping, and field test [14]. At the preliminary stage, curriculum analysis is
carried out, analysis of questions in student textbooks on material related to graphics, and designing non-
routine questions related to graphics. The formative evaluation stage consists of self-evaluation, expert
review, one-to-one, small group and field test. The product of the research is non-routine questions related
to graphics that have valid, practical, and effective classifications to measure students' mathematical
abilities. Non routine questions related to graphics are declared valid based on the results of the validation
sheet score given at the expert review stage. Non-routine questions related to graphs are declared practical
based on the results of the practicality test sheet score at the small group stage. Non-routine questions
related to graphics are declared effective if more than 50% of students give positive responses, teachers
give positive responses, and student test results fall into the good category at the field test stage.

3. Result and Discussion

The results of this study are described based on Tessmer's [14] stages which consist of preliminary and
formative evaluation stages consisting of self-evaluation, expert review, one-to-one, small group, and field
test.

3.1. Preliminary Stage.

At this stage, preparation and design are carried out. The results of the preparation stage are the amount of
math material related to graphs and the percentage of the number of non-routine questions related to
graphs in student books. The results obtained at the design stage were draft and rubric non-routine
questions related to graphs. The total number of draft non-routine questions related to the resulting graph
is 20 questions.

3.2. Self-evaluation Stage.

At this stage the draft questions at the final stage are assessed by the researchers themselves and
colleagues. The results of this stage are non-routine questions related to the revised graph which is named
Prototype I.

3.3. Expert Reviews Stage

At this stage Prototype I is validated based on content, construct, and language by five experts consisting
of three lecturers and two teachers. Each expert is given a prototype I in order to provide comments and
suggestions that are useful for revising prototype I. At this stage the experts are also given a validation
sheet for non-routine questions related to graphs. Based on the expert's judgment, it is generally stated that
the questions can be used with some components needing revision. After the questions are revised based
on comments and suggestions from experts, then the experts will fill out the validation sheet. The results
of the validation sheet analysis obtained an average of 4.21. These results indicate that the validity is in
the valid criteria.

3.4. One-to-One Stage

At this stage, the researcher gave questions to year seven and year eight students consisting of six students
divided into three year seven students and three year eight students. Students are selected with high,
medium, and low abilities based on the teacher's recommendations for mathematics. At this stage, each
student face to face with the researcher is asked to work on each item. To find out the difficulties that
students might face, researchers interacted directly with students to observe the responses given and the
obstacles students might face when working on the questions. Based on the results of expert reviews and
one-to-one results, prototype II is obtained which is the revised result of prototype I which has been valid
based on validator comments and suggestions. Prototype II consists of 17 questions divided into 9
questions for year seven and eight questions for year eight which will then be tested in the small group
stage.

3.5. Small Group Stage

At this stage prototype II was given to 12 students, each of which consisted of six year seven students and
six year eight students. The ability of these students was divided into three high ability students and three
medium ability students for each class. Students are asked to do nine questions for year seven and eight
questions for year eight. Students are also asked to write comments on the problem being worked on. At
the small group stage, the practicality of the questions was also assessed so that the questions could be
applied by the teacher and easy to use by students. To test the practicality of the questions, the teacher
needs to fill out the practicality test questionnaire. The practicality of non-routine questions related to the
developed graphs was obtained from the responses of five expert practitioners (teachers). Based on the
answers and comments of students in solving the revised prototype II questions, it produced a practical
prototype III based on the results of the teacher's assessment.

3.6. Field Test Stage

At this stage the prototype III was tested on the test subjects, namely 32 students of year seven, 30
students of year eight. This stage aims to assess the effectiveness of non-routine questions related to
graphs to measure students' mathematical abilities. The criteria for the effectiveness of non-routine
questions related to graphs, namely (1) more than 50% of students gave positive responses; (2) the teacher
gave a positive response; (3) the score of the results of the non-routine questions related to graphs after
averaging is included in the good category for students' mathematical abilities in this study, namely
problem-solving abilities, reasoning skills, communication skills, connection skills, and mathematical
representation abilities.

Student responses were obtained through a response questionnaire given to 62 students. The results of
students' positive responses to non-routine questions related to graphs obtained an average percentage of
61.6%. Based on the effectiveness criterion, more than 50% of students gave a positive response.

The teacher's response can be seen based on the results of the practicality test questionnaire. Based on the
questionnaire, on average, the teacher chose to strongly agree and agree on the non-routine question
statement related to the developed graphs which can help the teacher to improve the quality of learning
and non-routine questions related to the developed graphs that can help the teacher to find out things that
students have not understood and things -that the student has understood. Based on the effectiveness
criteria, namely the teacher has fulfilled a positive response.

The third effectiveness criterion was obtained from the scoring results based on the students' answers to
prototype III. It aims to assess the category of mathematical ability. These abilities include problem
solving skills, reasoning skills, communication skills, connection skills and mathematical representation
skills. The results of the field test were analyzed based on indicators of mathematical ability and scoring
rubrics.

The students' mathematical problem solving abilities for year seven were in the good category, namely 16
students, and 16 other students were in the sufficient category. Whereas in year eight students fall into
three categories of problem solving abilities, namely very good, good, and sufficient. Students who are
included in the very good category are one student, the good category is 13 students, while the other 16
students are included in the enough category.

The students' mathematical reasoning abilities for year seven were included in the very good category,
namely two students, 18 students were included in the good category, and the remaining 20 students were
included in the sufficient category. Whereas for year eight, students who got the very good category were
two students, the good category was 13 students, and the remaining 15 students were included in the
sufficient category.

The communication skills for year seven are divided into two, namely good and sufficient. Students who
are included in the good category are 17 students, and the rest, which are 15 students, are included in the
sufficient category. Whereas for year eight, students who got the very good category were one student,
both 13 students and 16 students enough.
The connection ability for year seven is divided into three categories, namely very good, good, and
sufficient. Students who are included in the very good category, namely two students, 11 students are
included in the good category, and the remaining 19 students are included in the sufficient category.
Whereas for year eight, students who got the very good category were three students, the good category
was seven students, 13 students were in the enough category, and seven students were in the poor
category.

The representation ability for year seven is divided into two categories, namely good and sufficient.
Students who are included in the good category are 17 students, and the remaining 13 students are
included in the sufficient category. Meanwhile, year eight received three categories, namely very good,
good, and sufficient. Students who got the very good category were two students, the good category was
12 students, and the remaining 16 students were included in the enough category.

Based on the results of field trials (field tests), besides obtaining the quality of the questions developed,
students 'learning outcomes were also obtained, namely students' mathematical abilities. Based on the
results of the analysis of students 'mathematical abilities, in general, students' abilities are categorized into
four levels, namely very good, good, adequate, and lacking. This shows that the questions that have been
developed can measure the various mathematical abilities of junior high school students. Especially for
VII and VIII grades. The results of the analysis also show that the average high-level mathematical ability
of students is in the good and sufficient category.

3.7. Discussion
The criteria for the questions developed were valid, practical, and effective. Problems that have been
developed can be declared valid, namely through the validation process by five validators (experts) who
provide comments and suggestions for improving the questions in terms of content, construct, and
language. This is a reference in revising non-routine questions related to the graphs that have been
developed. This is in line with Stahnke, Schueler, and Roesken-Winter [15] who stated that the validity of
an evaluation tool can be made based on logical or theoretical considerations. In order for the results of
these theoretical considerations to be adequate, they should be carried out by experts who have experience
with the evaluation tool. This is in accordance with the opinion of Ghazali [16] that validation activities
are carried out by several experienced experts or experts to assess the newly designed product. The non-
routine question set related to the graphic developed by the researcher was recommended by the validator
as a set of questions that were feasible to be applied in classroom learning.

The second criterion that makes the questions developed suitable for use in class is the practitioner states
that the questions developed are applicable and the level of product implementation is in a good category.
Saygili's [17] research results also show that learning using non-routine problems can improve students'
mathematical abilities. Students' mathematical abilities can develop when given non-routine problem
solving activities [18]. Due to the importance of mathematical abilities, it is important for teachers to
develop these abilities by training students with questions that can develop these mathematical abilities
[19]. Non-routine questions can help students to prepare for the demands of the 21st century. Students not
only need to know basic facts and skills, but also have to master how to make decisions, prioritize things,
develop strategies, and solve problems.

The third criterion that becomes a reference in question development is effective. In this study, the
measurement of the effectiveness of the questions that has been developed is based on investigations
through student and teacher responses, as well as student learning outcomes. Problems that are said to be
effective can be obtained through teacher responses, if the teacher gives a positive response to the
resulting questions, the effectiveness of the questions is declared good [20]. For example, the teacher's
response expressing interest in non-routine questions related to graphs and these questions can train
students' high-level mathematical abilities. This is in line with Akker [21] which states two things that
must be fulfilled to see the effectiveness of the instrument, namely, 1) an expert/practitioner based on his
experience states that these questions are effective. in this case the teacher as an experienced practitioner
has stated that the resulting questions have been effective, 2) operationally it gives the expected results,
namely the elaboration of the process to produce questions understood by the teacher or reader so that the
teacher reader can learn/follow the stages in the development of non-routine questions related to graphs.

4. Conclusion
This research produces a product in the form of non-routine questions related to graphs to measure
students' mathematical abilities, namely problem solving, reasoning, communication, connection, and
mathematical representation abilities. Based on the research results, it can be concluded that the
development of non-routine question products related to graphs to measure students' mathematical
abilities is carried out through the preliminary and formative evaluation stages to obtain valid, practical
and effective question products. A total of 17 non-routine questions related to the graph were declared
valid based on the results of the validation sheet score by five validators, practical based on the results of
the practicality test score, and effective based on the results of positive student responses, namely 61.6%,
positive responses by the teacher, and related non-routine test results. Based on the results, it can be
concluded that non-routine questions are used to measure students' mathematical abilities in classroom
learning.

References
[1] Hoth J, Kaiser G, Busse A, Doehrmann M, Koenig J, and Blömeke S 2017 Professional
Competences of Teachers for Fostering Creativity and Supporting High-Achieving Students. 49 pp
107-120
[2] Darwani, Zubainur C M, and Saminan 2020 IOP J. Phys.: Conf. Ser. 1460 012019
[3] Kilpatrick J Swafford and Findell 2001 Adding It Up Helping Children Learn Mathematics
(Washington DC: National Academy Press)
[4] Pantziara M, Gagatsis A and Elia I 2009 Ed. Studies in Math. 72 39
[5] Usta, N 2020 Evaluation of Preservice Teachers' Skills in Solving Non-Routine Mathematical
Problems through Various Strategies. 6 pp 362-383
[6] Kolovou. A., van den Heuvel-Panhuizen,. M., & Bakker. A. 2009 Non-Routine Problem Solving
Tasks in Primary School Mathematics Textbooks – A Needle in A Haystack. 8 pp 31-68.
[7] Woodward J, Beckmann S, Driscoll M, Franke M, Herzig P, Jitendra A, Koedinger K R, and
Ogbuehi, P 2012  Improving Mathematical Problem Solving in Grades 4 Through 8: A Practice
Guide (NCEE 2012-4055). (Washington DC: National Center for Education Evaluation and
Regional Assistance, Institute of Education Sciences, U.S. Department of Education)
[8] Memnun, D. S., and Coban, M 2015 Mathematical Problem Solving: Variables that Affect Problem
Solving Success. 3 pp 110-120
[9] Yazgan, Y 2016 Fourth Graders and Non-Routine Problems: Are Strategies Decisive For Success?.
2 pp 100-120
[10] Alkan, F. and Erdem 2011 A Study on Developing Candidate Teachers’ Spatial Visualization and
Graphing Abilities. 15 pp 3446–3450
[11] Kilic, D., Sezen, N., and Sari, M 2012 A Study Of Pre-Service Science Teachers’ Graphing Skill. 46
pp 2937 – 2941.
[12] Rufiana, I. S. 2015 Level Kognitif Soal Pada Buku Teks Matematika Kurikulum 2013 Kelas VII
Untuk Pendidikan Menengah. 3 pp 47-55
[13] Alacaci, C., Lewis, S., O'Brien, G. E., and Jiang, Z 2011 Pre-service elementary teachers’
understandings of graphs.7 pp 3-14.
[14] Tessmer M 1993 Planning and Conducting Formative Evaluations: Improving The Quality of
Education dnd Training. (Psychology Press)
[15] Stahnke, R., Schueler, S., & Roesken-Winter, B 2016 Teachers’ Perception, Interpretation, dnd
Decision-Making: A Systematic Review of Empirical Mathematics Education Research. 48 pp 1-
27.
[16] Ghazali, N. H. M 2016 A Reliability and Validity of an Instrument to Evaluate the School-Based
Assessment System: A Pilot Study. 5(2) pp 148-157.
[17] Saygılı, S 2017 Examining The Problem Solving Skills And The Strategies Used By High School
Students in Solving Non-Routine Problems. 8(2) pp 91-114.
[18] Chong, M. S. F., Shahrill, M., Putri, R. I. I., & Zulkardi 2018 Teaching Problem Solving Using
Non-Routine Tasks. In AIP Conference Proceedings. 1952(1) p. 020020
[19] Jäder, J., Sidenvall, J., & Sumpter, L 2017 Students’ Mathematical Reasoning and Beliefs in Non-
Routine Task Solving. 15(4) pp 759-776.
[20] Tambunan, H 2019 The Effectiveness of the Problem Solving Strategy and the Scientific Approach
to Students' Mathematical Capabilities in High Order Thinking Skills. 14(2) pp 293-302.
[21] Akker, J. V 1999 Design Approaches and Tools in Education and Training. (Dordrecht: Kluwer
Academic Publishers)