Which should be the priority concern for test developers - validity or reliability?

Why?

Validity and reliability refer to the technical quality and accuracy of data instruments
and usefulness of the test. They are important for defining, measuring bias and
distortion. These are the two most important features of a test. As teachers, we should
examine these features when evaluating the suitability of the test for your use.But
validity is more a priority concern for test developers than reliability. Validity is the
accuracy of an assessment. The validity of a test is therefore concerned with the
extent to which it actually measures what it is supposed to measure. In assessment, a
test’s validity is a measure of it’s relevance to the content. The aim in evaluating a test
in terms of its validity is to determine the extent to which the test scores will predict
something about the student’s performance.The question of validity is the form of the
test, the purpose of the test and the population for whom it is intended. Therefore, we
cannot ask the general question “Is this a valid test?”. The question to ask is “how
valid is this test for the decision that I need to make?” or “how valid is the
interpretation I propose for the test?” Validity is the most important issue for test
developers. Validity tells you if the characteristic being measured by a test is related
to the content taught in class.Validity gives meaning to the test scores and the
evidence indicates that there is linkage between test performance and class
performance. Validity also describes the degree to which you can make specific
conclusions or predictions about people based on their test scores. In other words, it
indicates the usefulness of the test.

On the other hand, Reliability is the degree to which an assessment tool produces
stable and consistent results. If a teacher created a quiz to measure students’ ability to
solve quadratic equations, then the teacher should be able to assume that if a student
gets an item correct, he or she will also get other, similar items correct. Reliability is
not at all concerned with intent, it is instead asking whether the test used to collect
data produces accurate results. If you are focusing on the reliability of a test, as a
teacher all you need to ask is—are the results of the test consistent? If the student
take the test today, a week from now and a month from now, will my results be the
same? Teachers may have a concern that the way a student’s performance is being
evaluated today will approximate the way it might evaluated later, assuming it
remains unchanged.

So in this case, Validity is a priority concern for test developers.

Which is harder to establish - validity or reliability?

Validity is harder to establish because it is a bit more complex and it is more difficult
to assess than reliability. As teachers, we have to assess that the test or assessment
being measured is aligned with content and to what extent does the test items
adequately and representatively illustrates the content area to be measured. We also
have to assess to what extent does the test covers all aspects of the concept being
measured. Finally we have to assess if the scores of the test reflect the items being
measured.
Should we always check each test we make whether it is reliable and valid?

No, As teachers we cannot always check each test we make whether it is reliable and
valid. A test cannot be valid unless it’s reliable. An assessment can provide you with
consistent results, making it reliable, but unless it measures what it’s supposed to, it is
not valid. However, a test can be reliable without being valid. An assessment that has
very low reliability will also have low validity; clearly a measurement with very poor
accuracy or consistency is unlikely to be fit for its purpose. But, by the same token,
the things required to achieve a very high degree of reliability can impact negatively
on validity. For example, consistency in assessment conditions leads to greater
reliability because it reduces 'noise' (variability) in the results. On the other hand, one
of the things that can improve validity is flexibility in assessment tasks and
conditions. Such flexibility allows assessment to be set appropriate to the learning
context and to be made relevant to particular groups of students. Insisting on highly
consistent assessment conditions to attain high reliability will result in little flexibility,
and might therefore limit validity.

Can we not trust our gut feelings especially if you are an experienced teacher?

If you find yourself entrusted with the education of someone else, you have probably
demonstrated a solid knowledge of a subject. It is also likely that the experience and
communication skills are evident when interacting with others. Teaching is not
something that all people can do effectively, especially not something as intensive and
specific as aviation.

Despite the mystique promoted by many institutions of higher learning, “book

knowledge” can only go so far. It is very difficult for texts and diagrams to fully
convey all that is necessary to understand a subject truly. They certainly cannot
duplicate the real-world experience. A driver’s manual can describe everything about
a city’s vehicle laws, but it can’t teach you how to manage the stress of operating a
truck on a ruthlessly busy interstate. A parenting class can show you how to breathe
during delivery, but it cannot accurately describe the level of fear and pain
experienced during labor.

Gaps between knowledge and experience are bridged by excellent teachers. After
studying all of the facts, taking tests, and learning theories on a subject, students still
have questions. These remaining curiosities are where a teacher’s “gut feelings” and
instincts become very valuable.

References

Middleton, F. (2019, July 3). Reliability vs validity: What’s the difference? Retrieved
May 15, 2021, from https://www.scribbr.com/methodology/reliability-vs-
validity/#:~:text=Reliability%20and%20validity%20are%20concepts,the
%20accuracy%20of%20a%20measure.
Validity and Reliability in Testing – What do They Mean? (2020, October 19).
Retrieved May 16, 2021, from https://wonderlic.com/blog/assessments/validity-
and-reliability/

Reliability and Validity. Retrieved May 16, 2021, from

https://assessment.tki.org.nz/Using-evidence-for-learning/Working-with-data/
Concepts/Reliability-and-validity

Why You Should Trust Your Gut When Teaching. (2018, December 07). Retrieved
May 16, 2021, from https://cfiacademy.com/why-you-should-trust-your-gut-
when-teaching/

Clive said,

Which should be the priority concern for test developers -

validity or reliability? Why?
Validity refers to how accurately a method measures what it is
intended to measure.  Reliability refers to how consistently a
method measures something. Even though both validity and
reliability are important in developing tests, validity is generally
more important than reliability and should therefore be given more
priority. Validity is in fact the most important characteristic of a test
or assessment technique. To obtain useful results, the methods you
use to collect your data must be valid. Tests and other assessment
methods are used by teachers and administrators to gather valuable
data about students’ understand of important topics, how effective
the teacher is and possible areas for improvement. It is therefore of
fundamental importance that when tests are designed and
implemented they are actually measuring what they are supposed
to be measuring. Otherwise data obtained from the testing
processing can be very misleading and could lead to incorrect and
potentially harmful decisions being taken.
Which is harder to establish - validity or reliability?
Validity is generally harder to assess than reliability. It can be
estimated by comparing the results obtained to other relevant data
or theory. In evaluating the validity of certain types of relationships
it might be necessary to consider the generalizability of the results.
If scores are used to measure variations in levels of ability for
example, it is important that your results reflect the real variations
as accurately as possible. The teacher has to ensure that the
assessment chosen is of high quality and targeted to measure
exactly what he or she wants to know. It should be thoroughly
researched and based on existing knowledge. This adds to the level
of difficulty in effort involved in establishing validity.
Should we always check each test we make to determine
whether it is reliable and valid?
Validity and reliability are important qualities of assessment and can
be easily compromised
(Shermis and Di Vesta, 2011).  It is important to always check
every test for both reliability and validity. If the test is not valid, all
the other important features of the test will be adversely affected.
To be successful at continuous assessment it is important that all
assessments, whether formative or summative be both valid and
reliable. This is vital to ensuring that the various objectives of
education are achieved so that effective feedback is provided to
teacher, administration, students and parents. Therefore
assessment should be careful planned so that they fit into the
overall plan for the curriculum and that they will be effective at
measuring what aspects of the curriculum they are supposed to be
measuring. It is also important that each assessment will achieve
roughly the same results when the same methods are used under
the same circumstances. Knowledge of test score reliability and
validity is important for making instructional and evaluation
decisions about students
Can we not trust our gut feelings especially if you are an
experienced teacher?
I don’t believe that we should rely entirely on trusting our instinct
about whether a test is valid or reliable. Validity and reliability need
to be established for every assessment item to ensure that
objectives are achieved and that the bad decisions are avoided.
Teachers can draw on the skills and experience of their colleagues
as well in pilot testing assessment instruments before actually
administering the test. Any modifications can therefore be made to
improve the test. It is best to use a more objective, rational
approach rather than a more subjective approach based on opinions
when designing and administering these tests
Reference
Shermis, M. D. & Di Vesta, F. J. (2011). Classsroom Assessment in
Action. Rowman & Littlefield Publishers, Inc.