Fostering a student’s skill for analyzing test

items through an authentic task

Cite as: AIP Conference Proceedings 1868, 080002 (2017); https://doi.org/10.1063/1.4995186
Published Online: 04 August 2017

Beni Setiawan, and Wahyu Budi Sabtiawan

ARTICLES YOU MAY BE INTERESTED IN

The effectivenes of science domain-based science learning integrated with local potency
AIP Conference Proceedings 1868, 080001 (2017); https://doi.org/10.1063/1.4995185

Developing evaluation instrument based on CIPP models on the implementation of portfolio

assessment
AIP Conference Proceedings 1868, 080003 (2017); https://doi.org/10.1063/1.4995187

Development of the evaluation instrument use CIPP on the implementation of project

assessment topic optik
AIP Conference Proceedings 1868, 080006 (2017); https://doi.org/10.1063/1.4995190

AIP Conference Proceedings 1868, 080002 (2017); https://doi.org/10.1063/1.4995186 1868, 080002

© 2017 Author(s).
 Fostering A Student’s Skill For Analyzing Test Items
Through An Authentic Task
Beni Setiawana), Wahyu Budi Sabtiawan

Science Department, Faculty of Mathematics and Natural Science, Universitas Negeri Surabaya, Unesa
Jl. Ketintang, Surabaya, East Java, Indonesia
a)
Corresponding author: benisetiawan@unesa.ac.id

Abstract. Analyzing test items is a skill that must be mastered by prospective teachers, in order to determine the quality
of test questions which have been written. The main aim of this research was to describe the effectiveness of authentic
task to foster the student’s skill for analyzing test items involving validity, reliability, item discrimination index, level of
difficulty, and distractor functioning through the authentic task. The participant of the research is students of science
education study program, science and mathematics faculty, Universitas Negeri Surabaya, enrolled for assessment course.
The research design was a one-group posttest design. The treatment in this study is that the students were provided an
authentic task facilitating the students to develop test items, then they analyze the items like a professional assessor using
Microsoft Excel and Anates Software. The data of research obtained were analyzed descriptively, such as the analysis
was presented by displaying the data of students’ skill, then they were associated with theories or previous empirical
studies. The research showed the task facilitated the students to have the skills. Thirty-one students got a perfect score for
the analyzing, five students achieved 97% mastery, two students had 92% mastery, and another two students got 89% and
79% of mastery. The implication of the finding was the students who get authentic tasks forcing them to perform like a
professional, the possibility of the students for achieving the professional skills will be higher at the end of learning.

INTRODUCTION
Analysing test items is one of perspective and teacher’s skill, that has to be enhanced, to improve the quality of
questions for assessing student’s achievement. The skill involves the ability to collect, summarizing, and applying
information from the students’ answers to make a judgment in each assessment [1]. The purpose of the analysis is to
examine and evaluate every test item in order to obtain a quality test before it is tested. In addition, the activity is
also to help increase testing through revision or dispose the items that are not effective and to investigate the
diagnostic information on students whether they have or have not been taught to understand the learning material
[2]. The quality test items are able to provide rigorous information in accordance with its purpose, such as which can
determine which learners who have or have not mastered the material taught by the teacher.
The main purpose of test items analysis is to identify weaknesses in the test or and learning [3]. Based on the
aim, test items analysis has many benefits, that are: (1) it can help test users for evaluating the test, (2) it is highly
relevant for the preparation of informal and local tests, such as tests that prepared the teachers for the students in the
class, (3) it supports the development of effective test items, (4) it can improve the test in class, (5) it improves the
validity and reliability of test [3]. In addition, other benefits are: (1) it determines whether a function of test items
works as expected, (2) it provides feedback to the students about their capabilities and is as a basis for discussion in
class, (3) it provides feedback to the teachers about the students’ difficulties, (4) it gives input on certain aspects of
developing curriculum, (5) it is able to give input for revising the material assessed or measured, and (6) it can
increase the skills of test maker [1].
Linn and Gronlund [4] also added the requirements usually fulfilled during analysing test items. They involve:
(1) Is the function of test items appropriate? (2) Is the level of difficulty appropriate? (3) are there irrelevance
contents in the test items? (4) Is the answer choices effective?. Moreover, Linn and Gronlund [4] declared that the

The 4th International Conference on Research, Implementation, and Education of Mathematics and Science (4th ICRIEMS)
AIP Conf. Proc. 1868, 080002-1–080002-7; doi: 10.1063/1.4995186
Published by AIP Publishing. 978-0-7354-1548-5/$30.00

080002-1
purpose of test items analysis is not only limited to enhancing the quality of items, but also there are other
advantages, such as (1) as a basis for a class discussion efficient about the results of the test, (2) as a basis to work
remedial, (3) for increasing a test construction skill.
Various perspectives above show that the analysis are: (1) to determine the test items that defective or do not
work ; (2) to increase the quality of items based on three components, namely, item discrimination index, level of
difficulty, and distractor functioning, as well as to improve learning through test items ambiguity and specific skills
that lead to learners difficult. In addition, items that have been analysed to provide information for the students and
the teachers.
In performing the test items analysis, the test makers could analyze qualitatively in terms of content and form
and also quantitatively in terms of their characteristics of statistics [3] or procedures for increasing through judgment
and empirical [5]. Qualitative analysis includes consideration of content and construct validity, whereas quantitative
analysis includes measurements of item discrimination index, the level of difficulty, validity, and reliability.
There are two ways that can be used to analysis the test items, that is, qualitative and quantitative analysis. Both
of these techniques have advantages and disadvantages. The qualitative way can be conducted using computer
programs involving Excel, Iteman, SPSS, Rascal, Ascal, Bilog, Facet, Bigstep, Quest, and Anates. In this research, it
is focused on the quantitative analysis utilizing Excel and Anates program. The Excel program is good enough to
use in analyzing test items but it is slightly better compared to a calculator. It is because its operation system is also
still manual and the analysts need to input a lot of logic formulas, then its capability to analysis is also limited [6].
On the other hand, the Anates can accommodate all aspects of quantitative analysis, except students' mastery
learning.
In order to assess learning outcomes, the test is expected to be able to represent the sample and produce an
objective and accurate result, thus the teachers have to evaluate the test in many aspects.
Parameters of test quality consist of reliability, item discrimination index, level of difficulty, and distractor
functioning, superior and inferior group, correlation of each item score with total score, and summarizing of analysis
result provided by Anates [7]. This study concerns in five parameters, that are, validity, reliability, item
discrimination index, level of difficulty, and distractor functioning.
A solution to force student’s skill in terms of test item analysis is providing authentic tasks. The authentic tasks
require students to integrate the knowledge, skills, and attitudes as a professional need in the world of work [8].
Thus, students will get a simulation in the real workplace. It is important because according to Boud as cited in
Gulikers et al. [8] explained that there were many graduates who have difficulty when it comes to the situation in the
real workplace. That indicated there was a gap between what the teachers assessed and what the students need in the
real workplace.
Based on all perspectives above, this study provides a description how the authentic task is able to force
student’s skill for analysing test items in the five parameters involving validity, reliability, item discrimination
index, level of difficulty, and distractor functioning.

METHOD
The research design was one-group posttest design, with a descriptive study. The participant was
the students enrolled in assessment class of Science Department, Faculty of Mathematics and Natural Science,
Universitas Negeri Surabaya. They were given a task that was authentic, such as they were required to develop test
items, then analyzing the items. The nature of the task authenticity adapted from Herrington et al. [9] detailed in
Table 1. Table 1 shows how the characteristics of authentic task was implemented technically in the learning.
Hence, the task has been eligible in terms of the authenticity requirements.
In order to fulfil the task completion, the students are required to do analysis of test items consists of 10 multiple
choice and 5 essays questions for each selected basic competence. Moreover, they had to consult to the experts for
reviewing and validating the test. Furthermore, they revised the test items based on the suggestions from the experts.
The next step was that the students analysis the quality of the test using Excel and Anates Software. Each item was
analysed involving validity, reliability, item discrimination index, level of difficulty, and distractor functioning. The
last stage was displaying the results of test analysis in the form of project reports.

080002-2
 TABLE 1. Manifestation of the task authenticity in the learning
Authentic Task Characteristics
Authentic tasks have real-world relevance.
Authentic tasks are ill-defined.
Authentic tasks need over a sustained period of time.
Authentic tasks provide the opportunity for students to
examine the task from different perspectives.
Authentic tasks provide the opportunity to collaborate.
Authentic tasks provide the opportunity to reflect.
Authentic tasks can be integrated and applied across
different subject areas and lead beyond domain-specific
outcomes.
Authentic tasks are integrated with the assessment.
Authentic tasks create a holistic product.
Authentic tasks allow competing solutions and diversity
of outcome.
Manifestations in Learning
The task required the students to do the work of a teacher,
assessor, and researcher in education, that were to create
questions, then analysis each item. It was conducted to
evaluate the theoretical validity of the constructed test.
They were provided only limited information without
given detail instructions, such as student’s worksheet.
Therefore, they had an opportunity to determine how to
resolve or design solutions for their tasks independently.
The task had to be completed within two weeks (not in
the class immediately) because it was quite complex.
They were given the opportunity to explore learning
resources from books, journal articles, and websites.
Through the demand, the students gained diverse
perspectives in solving or designing a solution for the
task.
The students were required to go to school for
collaborating with the teachers and doing the analysis of
test items together.
The students were required to show the performances as a
teacher, assessor, and researcher in educational area. In
addition, they each reflected one another in terms of what
they have acquired during the process of finding a
solution for the task.
The task encouraged the students to integrate the
understanding of assessment, writing skills, and
communication skills.
The task will be assessed using a rubric for the project
report.
The student’s task to be completed included the analysis
of test items and the creation of project report.
The students were given the opportunity to explore
learning resources from books, journal articles, and
websites. In addition, they were allowed to find out
several solutions for the completion of their duties.

The results obtained were analyzed descriptively. The analysis of the data was presented by displaying the
student’s skill in terms of analysing test items, associating with theories or studies from experts, and comparing with
previous relevant research.

RESULTS AND DISCUSSION

Based on the research that has been conducted on the assessment class of Science Department-Unesa, it was
obtained the following data presented in Figure 1.

080002-3
 Analysis Using
Excel

Analysis Using
Anates

FIGURE 1. Graph of Assessment toward Data Processing Using Excel dan Anates Software

Based on the Figure 1, it can be seen that both of assessment item score of data processing using Excel and
Anates Software were in a maximum score of 12 and 18 respectively. In other words, the students achieved the best
score for analysing test items using the two software. It indicated that the students can easily process the raw data in
the form of test results correctly to determine validity, reliability, item discrimination index, the level of difficulty,
and distractor functioning.
Other skills such as making the description of the results and conclusion can be seen in Figure 2 below.

FIGURE 2. Graph of Assessment Toward Making Description and Conclusion

The amount of 36 students got the maximum score of 18 and 4 students got a score of 12 for describing the
results skill as can be seen in the Figure 2. On the other hand, the student's ability to conclude the analysis results
was categorized in high, middle, and low level. There were 35 students getting a maximum score of 10, 6 students
getting a middle score of 8, and 1 student was in a score of 0 as the lowest mark. Based on the result, generally, the
students are able to describe the results and make a conclusion from data processing well. The score of both skills
was obtained from the students who were presenting the data of validity, reliability, item discrimination index, the
level of difficulty, and distractor functioning.

080002-4
 Based on some skills of students in the analysis of the items tested, it was obtained the data of student’s score
representing the skills that can be seen in Figure 3.

Number of Students

Score

FIGURE 3. Graph of Assessment toward Analysis Skill of Student

It can be seen that the thirty-one students got a perfect score for the analyzing, five students achieved 97%
mastery, two students had 92% mastery, and another two students got 89% and 79% of mastery. Assessing the
student’s skill was determined using six assessment parameters, namely the tested sample, type and the number of
question, data processing using Ms. Excel, data processing using Anates Software, description of the result, and
conclusion.
The results, presented in the Figure 1 until 3, indicated that the implementation of the authentic task can
accommodate the student’s skill of analysing test items as prospective science teachers. Factors supporting the
positive results will be discussed below.

Factors Affecting Student’s Skill For Analysing Test Items

The authentic task given to the students was described in the subsection of Research Design and Procedures. The
characteristics of authenticity were reflected in the manifestations listed in the Table 1. The manifestations
predictable plays an important role in the positive results demonstrated in this study. At least there were three
supporting factors as the result of the manifestations.
The first factor is the task of particular relevance to the real world or in accordance with what a professional do
in the world of work. Task-based teaching methods were widely applied in the world [10, 11, 12], one of which was
an authentic task. The task of particular relevance to the professional task in the real world, so that it was able to
stimulate the students to develop their competencies [13, 14, 15, 16]. Application of authentic tasks provided
experience to students and to motivate them [17, 18]. This was in line with Zohooriana [19] and Kozhevnikova [20]
in which the authentic tasks can increase motivation level. Keller [21] has conducted research related to the
environmental influence on behavior and individual learners. The study revealed that there were four factors that
affect one's motivation, namely, interest, relevance, expectations, and outcomes. Therefore, the authentic tasks
having relevance to the professional world is very likely as a contributing factor increased motivation, and indirectly
affect their efforts in developing their skills in completing the task the analysis item.
The second factor is social interaction. One of the authentic task characters is to give students an opportunity to
discuss or interact with others. Arends [22] found that the discussion (language expressed) provided information on
what students already know and to form new knowledge so that it was able to encourage cognitive development.
Wentzel as cited in McInerney and McInerney [23]; Joyce and Weil [24] argued the social interactions improved the
achievement of learners through three different ways, that were improving attention, mutual help, and responsibility.

080002-5
 The third factor is that the authentic task was ill-defined. In other words, the students were required to construct
a solution design for the task completion by themselves rather than following the steps like a recipe. Kolb [25] has
made it clear that the transfer of knowledge was very difficult to do by teachers. A possible way to do by the
teachers was to provide an opportunity to the learner to apply their knowledge and skills in a multidisciplinary
context. It means that it can be realized if the teachers ask the student an authentic assignment. Giving the task
unstructured or requiring students to create their own designs are able to stimulate the improvement of
metacognitive skills. Mastery of metacognitive skills has a positive impact directly on the process and outcomes of
the student’s learning [26, 27, 28]. It is because the students having high metacognitive skills are more aware of the
most effective learning strategy to achievegoals.

CONCLUSION

There were three points of conclusion in this study: 1) the students were able to analyse test items well using
Microsoft Excel and Anates Software represented by the maximum score obtained the students, 2) also the skills of
students to describe and conclude the results of the analysis involving validity, reliability, item discrimination
index, level of difficulty, and distractor functioning were in the maximum score, and 3) the students' scores on the
whole skills were that thirty-one students got a perfect score for the analyzing, five students achieved 97% mastery,
two students had 92% mastery, and another two students got 89% and 79% of mastery.

ACKNOWLEDGMENT

The authors thank the Assessment Tim of Science Department, Faculty of Mathematics and Science, Universitas
Negeri Surabaya (Unesa), due to the cooperation during the implementation of the research.

REFERENCES
1. A. J. Nitko, Educational Assessment of Students: Second Edition, (Merrill an imprint of Prentice Hall Englewood
Cliffs: Ohio, 1996), p. 308-309.
2. L. R. Aiken, Psychological Testing and Assessment: Eight Edition, (Allyn and Bacon: Boston,1994), p. 63.
3. A. Anastasi and S. Urbina, Psicoholological Testing: Seventh Edition, (Prentice-Hall: New Jersey, 1997), p. 172
& 184.
4. R. L. Linn and N. E. Gronlund, Measurement and Assessment in Teaching: Seventh Edition, (Prentice-Hall:
Ohio, 1995), p. 315-318.
5. W. J. Popham, Classroom Assesment: What Teachers Need to Know, (Allyn and Bacon: Boston, 1995), p. 195.
6. http://tulisanterkini.com/artikel/pendidikan/998-pengembangan-baru-asesmen-analisis-hasil.html.
http://adolfr.blogspot.co.id/2009/11/analisis-butir-soal-dengan-iteman.html.
7. J. T. M. Gulikers, T. J. Bastiaens, P. A. Kirschener, and L. Kester, Studies in Educational Evaluation, 32, 381–
400, (2006).
8. J. Herrington, T. C. Reeves, and R. Oliver, A guide to authentic e-learning, (Routledge: UK, 2010).
9. M. R. Hashemi, A. Lamir, and F. Namjoo, English Language and Literature Studies, 1, 2, 14-24, (2011).
10. N. Ruso, EFL Journal, 18, 1–23, (2007).
11. A. Sanchez, International Journal of English Studies, 4, 1, 39–71, (2004).
12. T. M. Duffy, and D. H. Jonassen, Constructivism and the technology of instruction: a conversation, (Lawrence
Erlbaum associates: New Jersey, 1992).
13. J. T. M. Gulikers, T. J. Bastiaens, and R. L. Martens, Computers in Human Behavior, 21, 3, 509–521, (2005).
14. J. L. Shrum and E. W. Glisan, Teacher’s handbook: contextualized language instruction. (Heinle&Heinle:
Boston, 2000).
15. Y. Woo, J. Herrington, Sh. Agostino, and T. C. Reeves, Educause Quarterly, 3, 36- 43, (2007).
16. S. A. Parsons and A. E. Ward, The Reading Teacher, 64, 6, 462-465. (2011).
17. J. E. Brophy, Motivating students to learn, (Routledge: New York, 2010).
18. Z. Zohooriana, Procedia-Social and Behavioral Sciences, 192, 15 – 25, (2015).
19. E. Kozhevnikova, Procedia-Social and Behavioral Sciences, 116, 4462 – 4466, (2014).
20. J. M. Keller, Motivational design of instruction. In C. M. Reigeluth (Ed.), Instructional-design theories and
models: An overview of their current status, (Lawrence Erlbaum Associates: NJ, 1983), p. 383 – 434.

080002-6
21. R. I. Arends, Learning to Teach, (McGraw-Hill: New York, 2012).
22. D. M. McInerney and V. McInerney, Educational Psychology: Constructing Learning: 5th Edition, (Pearson:
New South Wales, 2010).
23. B. Joyce and M. Weil, Models of Teaching: 4th Edition, (Allyn and Bacon: USA, 1992).
24. D. A. Kolb and R. Fry, Toward an applied theory of experiential learning. In: Cooper C,
editor. Theories of group process, (John Wiley: London, 1975), p. 33–58.
25. M. Boekaerts, P. Pintrich, and M. Zeidner, Handbook of self regulation, (Academic Press: San Diego, 2000).
26. B. Eilam, and I. Aharon, Contemporary Educational Psychology, 28, 304–334, (2003).
27. K. Mokhtari and C. Reichard, Journal of Educational Psychology, 94, 249–259, (2002).

080002-7