Validity, Reliability, Practicality, Authentic and Washback

In Assessment
Annisa Ulhasanah
17178046

This article focuses to make an assessment to be valid, reliable, practice, authentic, and
washback. What are the statistical tools to make an assessment valid, reliable, practice,
authentic, and washback. Explaining the uses of language test, five major principles of
language assessment: validity, reliability, practicality, authenticity, and washback. Then
discussion the principle

Keywords: Validity, Reliability, Practicality, Authentic, Washback

Introduction forms of assessment, and a chance to adapt the

assessment with the students’ need and
background (Reynisdottir, 2016; Case,
The real-life language in testing language
2013Wang; 2008 in Reynisdottir, 2016)
due to a new approach of teaching methodology
Generally speaking, valid test itself can
namely communicative language teaching (CLT)
be meant as a test that test what is being taught
since 1970s (Lewkowicz, 2000; Richard, 2006).
and how it is being taught along the learning and
This approach concerns on developing
teaching process. If a test is measured what
communicative competence in using language.
needs to be measured from the students, then it is
Communicative competence includes four
a valid test (Brown & Abeywickrama ,2010).
components of competence; grammatical
Investigating validity can be done by
competence, discourse competence,
analyzing content, construct and face validity.
sociolinguistic competence, and strategic
The content validity of a test is examined
competence as explained by Shumin (2002).
whether the test is a representative sample of the
However, these competences cannot be assessed
content which the test was designed to measure
merely by using a traditional assessment which
is considered to be content validity (Bachman,
usually comes in a form of open-ended, short
1990; Brown, 2010) . The construct validity as
answer, true-false, multiple choice and ,matching
Bachman (1990) mentioned can be concluded
tests (Nasab, 2015; Caliskan & Kasikci, 2010).
exists in a test if it is demonstrated that it
Therefore, the use of an alternative assessment is
measures just the ability which it is supposed to
needed to be able to assess these communicative
measure. Meanwhile, face validity refers to the
competences. Any assessment that can comes in
degree to which a test looks right, and appears to
multiple forms and can reflect students learning,
measure the knowledge or abilities it claims to
achievement, motivation, attitudes on
measure, based on the subjective judgment of the
instructionally-relevance classroom activities is
examinees who take it, the administrative
called as an authentic assessment ( O’Malley &
personnels who decide on its use, and other
Valdes-Peirce , 1996). The authentic assessment
psychometrically unsophisticated observers
is able to provide an illustration what might be
(Mousavi, 2009 as cited in Brown &
found or encountered in the real world by
Abeywickrama , 2010).
students, a meaningful way of assessing, many
 The reliability of a test mostly concerns
with the consistency of a test and a test would be
valid if it is reliable. Best and James (2003) also
explains that a test is reliable to the extent that it
measures accurately and consistently from one
time to another time For instance, when people
perform differently in different time and the
smaller the difference between two sets of
scores, the more reliable the test is (Hughes ,
2002). In other words, having a reliable test
would minimize error in order to avoid
unidentical score for the test taker after being re-
tested. Rater judgement is one of methods in
determining the test reliability (Kumar, 1996 in
Sak, 2008) .
The consistency in rater judgments
should be determined by relying on inter-rater
and intra-rater reliability (Brown, 2005:185).The
degree to which scores from two or more
markers agree is considered to be an inter-rater
reliability (Nunan, 1992:14-15; Weir & Roberts,
1994:172). In other words, inter-rater reliability
refers to the consistency between the marks
given by different teachers. Doubts upon inter-
scorer reliability could arise when the same
quality of answers is given different scores by
different teachers. On the other hand, the intra-
rater measured by having an assessor measure
balance and then repeat the measurement of the
same person after a specifid time lapses (Downs
et all, 2013). That is to say, intra-scorer
reliability refers to marks given by the same
teacher on different occasions. An example of
intra-scorer reliability at stake is when a teacher
gets tired of marking and starts to give lower
marks as time goes on. Consistent grading is
essential in order to ensure the reliability of test
scores. Scorer reliability can be improved by a
marking scheme or a scoring rubric that is
prepared in advance and used to assist.
DISCUSSION
The content validity represented by
learning objective and basic competence
indicators in both products is categorized as
valid. A test can be defined as a valid in its
content if it represents sample of the language
skills, structures or the subject matter (Bachman,
1990; Brown, 2010). This finding is also in line
with the finding found by Suarimbawa et all and
Ing et all in the same year of 2017. Suarimba et
all found that the authentic assessment used in
one of junior high schools of Singaraja had been
based on the curriculum 13. In other other
words, from its content, the assessment was
designed under the learning objective of
curriculum 13. It had assessed attitudes,
knowledge and skill of the students. The content
validity also can be fulfilled by the teacher –
made assessment of Chinese Elementary Schools
in Johor ( Ing, 2017: 193).
Nevertheless, it was found in the
interview that some units had a problem with the
content of the learning which is not suitable with
the learning objective and students’ ability.
Inappropriate context and task to the learning
object given in the oral test also was found by
Salabary in 2000. He found that lack of features
of conversational interaction, limited ramge of
interactional context and lack of specification of
content areas to be addressed (Salbary, 2000:
31). It is suggested that the content of the task
and topic should be more authentic.
Moreover, the teachers also suggested
that the instruction should also be clearer and
more specific to be undertood either by teachers
or students. The tasks and topics in the product
should also be related to students’ life to
encourage and motive them to perform in oral
test in front of the class. Validity, particularly
content validity , in the authentic assessment
should ensure that the tasks given are alike with
what they encounter in their real life as Gosh et
all( 2017) explains that content validity should
ascertain if the authentic tasks resemble real-
world scenarios, encompassing wide but
required content and assessing only intended
outcomes.
The construct validity also was
categorized as valid by all teachers. The
construct validity is fulfilled if the speaking test
measures the components of fluency and
accuracy. Similarly, Nakamura in 1997
investigated the construct validity of an English
speaking test. It examined whether the nine traits
of pronounciation, grammar, discourse, fluency,
content, vocabulary, comprehensibility were
relevant and separable parts of speaking ability.
It was found that the nine traits were functioning
as factor construct elements in oral test.
However, it was found in the interview
that some criteria in the scoring rubric need to be
adjusted, added and omitted so that the category
of the construct was not very valid category.
Rating criteria can greatly affect the construct
validity of a test (Brown & Taylor, 2006;
Esquinca, Yaden, & Rueda, 2005; Hubbard,
Gilbert, Pidcock, 2006 as cited in Pishgadam &
Shams, 2012:72). Grammar suggested in both
grades not to be assessed since junior high
school students were still in very low level in
English ability. They had not got any English
material in their previous level of education.
Therefore, some teachers suggested grammar
should not be assessed in the oral performance
test. Nevertheless this argument could not be
taken as a note to improve the product since it
would treat the validity, particularly the
construct validity. The construct validity ensures
that the assessment content is assessed the
learning objective in the curriculum (Aplhine,
2002: 11). The learning objective of curriculum
13 explains that students need to be able to
express their idea orally in appropriate grammar.
In addition, assessing speaking must include
assessment of students’ fluency and accuracy.
One of the indicators in accuracy is the use of
grammar (Brown, 2004:157).
The criteria of assessing ideas,
improvisation, mimic is suggested to be added to
enhance the construct validity of the product.
Having a brilliant idea, making improvisation
and do the right mimic is related to the cognitive
skill of the students as mentioned earlier, the
cognitive also become the target learning of
curriculum 13. It is in line with the research done
Pishgadam & Shams in 2012. They stated that to
avoid misconceptions about the speaking
construct, the nonlinguistic factors involved in
the process should be overlooked and one of the
major nonlinguistic factors can be the cognitive
ones, among which intelligence is of great
importance.
Finally, the face validity also can be
considered as valid one. Types of speaking
activities and tasks and form of authentic
speaking assessment are the two indicators of
this validity type. According to Brown &
Abeywickrama (2010:35) increasing
examinees’perception of fair by concerning to a
well-constructed and familiar tasks, an allotted
time limit, clear instruction, the subject material
in the class, and difficulty level. It was
concluded that both products was valid since it
had fulfilled the criteria of face validiy of
providing constructed and familiar tasks and
clear instructon. The evaluation of face validity
in IELTS speaking test also can fulfill the criteria
of having face validity as the unfamiliarity of
format and lack of authenticity in task can not be
found in the test item (Karim & Haq, 2014:156).
The inter-rater reliability found had
mostly high and moderate correlation
coefficients. Similarly to the findings of Halleck
(1996) investigating the inter-rater reliability of
trained raters on Oral Proficiency Interviews
(OPI), as a result of the correlations computed,
statistically significant results were obtained
since most score were satisfactory in each unit of
both products. It is important to have high rates
reliability between raters, so we can be sure that
students are being scored accurately and fairly
(WIDA, 2017:2).
On the other hand, a study Jafarpur
(1988) reported lower correlation coefficients.
An FSI-type oral interview was used in his study
and it was conducted at Shiraz University. The
performances of 58 students were scored by 3
raters and inter-rater reliability was reported as
between 0.58 and 0.65. The researcher indicated
that since the raters were not language teachers
who received some training, low correlations
may have emerged. In the light of these, it can be
said that the inter-rater reliability of the exam is
not as satisfactory as is expected since the
correlation of the scores of the third pair is fairly
low.
There are many factors that can lower
the inter-rater reliability. A research done by
Attali in 2015 found that assessing the subjective
assessment was dramatically depending on the
rater’s experience, training and bias. Other
researchers( Schaefer, 2008; Kim, H-J, 2015,
Barkaoui, 2010) found that the use of novice and
experienced rater may affect the inter-rater
reliability as the score of experienced rater was
more reliable. Kim also found that giving
training to the novice rater can improve the
consisted and objective score given by the
novice raters sa well as the experienced ones.
These findings suggest that training in scoring
and immediate feedback are as baluable as years
of teaching anad grading experience. In addition,
the published evidence on inter-rater reliability
suggests that high correlation coefficients are
generally achieved when multiple trained raters
are used to score performances (Fulcher, 2003:
142). However, Armes (2016:12) suggested that
the use of at least 2 raters in any speaking test
for practical consideration.
Interpretation of the rubric by the raters
also found to affect the reliability of the test. A
research done by Chong and Romney in 2016
indicated the fair agreement of the inter-rater
reliability was due to the absence of formal
training in using the rubric.
The raters themselves are not the only
source of affecting the reliability of a test.
Gebril in 2009 found that type and number of
tasks students are asked to do can have a
measurable effect on performance and in turn
affect the reliability of the score. Test procedure
also becomes the source of the unreliability as
Cassady found in 2005 that procedures for test
administration affect the outcome of test.
Conclusion
The Products are valid in term of its content,
construct and face validity to be use by teachers
in helping them to assess speaking ability under
the principle of Curriculum 13. Although some
arguments from the teachers as some of the
material is not interested, related to the
curriculum, the ambiguous instruction and the
unspecified and inapproriated criteria in scoring
rubric, overall it had fulfilled the content,
construct and face validity criteria as both
products included most of the learning objectives
demanded to be achieved in the curriculum 13.
In sum, both products can be implemented in
junior high schools with some improvement to
be made. It can ease teachers’ burdern in finding
the appropriate authentic assessment in assessing
speaking as it has variation in types of speaking
assessment and it represents the curriculum 13
demand.
References
Abdollah. 2016. An Analysis of Authentic
Assessment in 2013 Curriculum Used by
the English Teacher at SMA Negeri 4
Malang.Malang. University of
Muhammadiyah Malang.
Attali, Y. 2016. A Comparison of Newly-Trained
and Experienced RATERS ON a
standardized Writing Assessment.
Doi:10.1177/02655332215582283.
Armes, J.W. 2016. Quantifying the Qualitative:
Increasing the Reliability of Subjective
Language AssessmentS. The University of
San Fransisco.
Bachman, Lyle F. 1990. Fundamental
Consideration in Language Testing.
Oxford: Oxford University press.
Bachman, Lyly F & Palmer, A.S. 2009.
Language Testing in Practice: Designing
and Devloping useful Language tests.
New York. Oxford University Press.
Bernhardt, E.B et all. 2004. The Practicality and
Efficiency of Web-Based Placement
 Testing for College-Level Language
Program. Foreign language
Annal.vol.37.No.3
Braskamp, L.A & Engberg M.E. 2014.
Guidelines for Judging the Effectiveness
of Assessing Student Learning. Chicago.
Loyola University.
Barkaoui, K. 2010. Variability in ESL Esssay
Rating Processes: The role of the Rating
Scales and Rater Experience. Language
Assessment Quarterly 7, no.1, 54-
74.Doi:10.1080/1543300903464418.
Brown, H.Douglas and Priyanda
Abeywickrama.2010. Language
Assessment: Principles and Classroom
Practices. New York: Pearson Education.
Buzzeto-More, Nicole. 2010. Assessing the
Efficacy and Effectiveness of an E-
Portfolio Used for Summative Assessment.
Interdiciplinary Journal of E-Learning and
Learning Objects. Volume 6.
Brown, H.Douglas, 1994. Teaching Action by
Principles.An Integrated Approach to
Language Pedagogy (3rd Ed).New Jersey:
Prentice Hall Regents.
Case. R. 2013. Four Principles of Authentic
Assessment. Blogs.ubc.ca/file. Retrieved
February 13th 2018.
Cambridge English. 2016. Principles of Good
Practice. Cambridge University.
Chong, Alan & Romkey, Lisa. 2016. Testing
Inter-Rater Reliability in Rubrics for
Large Scale Undergraduate Independed
Projects. Pro. 2016 Candadiam
EngineeringEducation Association
(CEEA160 Conf.
Gay, l. R. and P. Airasian. 2009. Educational
Research: Competencies for Analysis and
Applications. New Jersey: Pearson
Education.Inc.
Gebril, A. 2009. Score Generalizability of
Academic Writing Tasks: does one test
method fit it all?. Language Testing 26,
no.4,507-31.doi
10.1177/0265532209340188.
Genesee, Fred and Upsur, John A.2002.
Classroom-based Evaluation in Second
Language Education. Cambridge:
Cambridge University Press.
Gosh , Samrat et all. 2017. Improving the
Validity and Reliability of Authentic
Assessment in Seafarer Education and
Training: a conceptual and practical.
WMU Journal of Maritime Affairs.
Gronlund. E.N & Waugh. K.C (2009).
Assessment of Student Achievement Upper
Saddle River, New Jersey. Pearson
Education, Inc.
Ing. L.M et all. 2015. Validity of Teacher –Made
Assessment: A table of Specification.
Asian Journal Science, vol.11. Canada.
Canadian Center of Science and
Education.w .
Harris, David, p. Testing English as a Second
Language. New York: mc.Graw Hill Book
Company.
Hidayanti. N. 2016. The Authenticity of English
Language Assessment for the Twelfth
Graders of SMK Negeri 4 Surakarta.
Premise Journal Volume 5 no.1
Hughes, Arthur.2003. Testing for Language
Teachers. Cambridge: Cambridge
University Press.
Ing et all. 2015. Validity of Teacher –Made
Assessment; A table of Specification
 Approach. Asian Social Science: Vol11,
no.5:2015. ISSN 1911-2017
Johnson, Nat. 2007. A Consideration of
Assessment Validity in Relation to
Classroom Practice. Cambridge
University.
Karim, S. & Haq.N.2014. An Assessment of
IELTS Speaking Test. International Journal of
Evaluation and Research in Education
(IJERE).Vol.3, no. 3, September 2014, pp.152-
157.
Kimberlin, C.I & Winterstein, A.G. 2008.
Validity and Reliability of Measurement
Instruments Used in Research. Research
Fundamental -Vol 65 Dec 1. 2008.
Kim, Hyun Jung. 2015. A Qualitative Analysis of
Rateer Behavior on an L2 Speaking Assessment.
Language Assessment Quarterly 12, no.3.
Lewkowicz, Jo.A. 2000. Authenticity in
Language Testing: Some Outstanding
Questions. Language Testing 2000 17
(1)43-64.
Luoma. S. 2004. Assessing Speaking.
Cambridge: Cambridge University Press.
Olfos. R & Zulantary. H. 2007. Reliability and
Validity of Authentic Assessment in a Web
Based Course. Educational Technology &
Society, 10 (4).156-173.
Ojung. J & Allida . D. 2017. A Survey of
Authentic Assessment Used to Evaluate
English Lnaguage Learning in Nandi
Central Sub-County Secondary Schools ,
Kenya. Baraton Interdicipline Research
Journal, 7 (special issue), pp 1-11.
O’ Malley, J.Michael & Pierce, Lorraine
Valdez.1996. Aunthentic Assessment For
English Langauge Leaners: Practical
Approaches For Teachers. Addison-
Wesley Publishing Company.
McAlphine, Mhairi. 2002. Principle of
Assessment. University of Glasgow.
Marhaeni. A.A.I.N et all. 2014. Toward
Authentic Language Assessment: A Case
in Indonesia EFL Classroom. The
European Conference on Language
Leaning.
Nasab, Fatemeh Ghanavati. 2015. Alternative
versus Traditional Assessement. Journal of
Applied Linguistic and Language
Research. Volume 2, Issue 6, 2015,
pp.165-178.ISSN:2376-760 X
Nation, I.S.P & Newton,J. 2009. Teaching
ESL/EFL Listening and Speaking. New
York &London. Routledge.
Natsir, Yuliana et all. 2018. The Rise and Fall of
Curriculum 2013: Insights on the Attitude
Assessemnt from Practicing Teachers.
SHS Web of Conferences 42.00010.
Nunan, David. 1988. Syllabus Design
(Language Teaching: A Scheme for
Teacher Education). UK: OUP Oxford.
P Pishgadam, R & Shams, M.A. 2013. A New
Look into the construct Validity of the
IELTS Speaking Module. The journal of
Teaching Language skills (JTLS) 5(1),
Spring 2012, ser 70/4.ISSN:2008-88191.
—71-90
Quaid, E.D. 2018. Reviewing the IELTS
Speaking Test in East Asia: Theoritical
and Practice-based Insights. Language
Testing in Asia (2018)8:2 DOI
10.1186/s40468-018-0056-5.
Reynisdottir. B.B. 2016. The Efficacy of
Authentic Assessment. University of
Iceland.
Richards, Jack C.2006. Communicative
Language Teaching Today. New York.
Cambridge University Press.
Richards, J & Renandya.2002 . Methodology in
Language Teaching. New York.
Cambridge University Press.
Riduwan. 2004. Metode dan Teknik Menyusun
Tesis. Bandung. Alfabeta.
Riemer, M. J.2007. Communication Skills for the
21st Century Engineer. Australia: Global
Journal of Engineering Education.
Rudner, Lawrence M. 1994. Question to Ask
When Evaluating Tests. ERIC Digest.
Sak, Gonca. 2008. An investigation of Validity
and Reliability of the Speaking Exam
at TurkishUniversity. Middle East
Technical University.
Schaefer, Edward. 2008. Rater Bias Patterns in
an EFL Writing Assessment.
Language Testing 25, no.4 465-93.
Srikaew, D et all. 2016. Development of an
English Speaking Skill Assessment
Model for Grade 6 Students by Using
Portfolio. Procedia-Sosioal and
Behavioral Sciences 191 (2015)764-
768.
Suarimbawa. K.A et all. 2017. An Analysis of
Authentic Assessment Implementation
Based on Curriculum 2013 in SMP Ngeri
4 Singaraja. Journal of Education
Research and Evaluation. Vol.1(i)pp.38-
45.
Sudjana. 2008. Metoda Statistika. Bandung.
Parsito.
Swanson. 2002. Constructing Written Test
Questions for the Basic and Clinical
Sciences. Third Edition (Revised) 2002.
National Board of Medical Examiners.
Available at
http://www.abme.org/PDF/Item Writing
2003/2003IWGwhole.pdf.
Trisanti. N. 2014. English Teacher’s Perspective
on Authentic Assessment
Implemnetation of Curriculum 2013.
The 61st Teflin International
Conference.UNS Solo.
Wang. Y. A. 2008. Authenticity in Language
Testing. Hawai Pacific Uneversity. Teson
Work Paper Series 6(2).
WIDA Resources for Educators. 2017.
Maintaining Rater Reliability in Scoring.
Yuji, Nakamura. 2008. Theoritical and Practical
Issues of Assessing Speaking in University
Entrance Examinations.Keio University.