Evaluation of

technology and language

learning

Latifa Essaidi
Ibtisam Serda
Ilham Mouzoun
01 Definition

02 Challenges in evaluating technology in language learning

03 Evaluation arguments
04 Improving evaluation

05 Conclusion
 Technology will never replace
great teachers, but in the hands
of great teachers,
it’s transformational.
George Couros
 The evaluation of technology for language learning
involves investigating and judging the appropriateness of
computer-assisted language learning (CALL) software for
a given language-learning setting. This process also
includes identifying effective ways to implement the
software in the setting, assessing its success, and
making decisions about its continued use or adjustments
for future use. Evaluation in this context is distinguished
from research, as it is driven by pragmatic decision-
making questions about the effectiveness of technology
in a particular context

Levy and Stockwell (2006) and Chapelle

In 2006, Hubbard defined evaluation of computer-
assisted language learning (CALL) as the process of
investigating a piece of CALL software to judge its
appropriateness for a given language-learning setting,
identifying ways it may be effectively implemented in
that setting, and assessing its degree of success and pragmatic decision-making questions about
determining whether to continue use or to make
adjustments in implementation for future use.
(2001) defined CALL evaluation involves the
systematic assessment of a CALL program or
environment to determine its effectiveness,
usability, appropriateness for specific learners
and learning contexts, and its potential for
facilitating language acquisition. It is driven by
whether or not something worked in a
particular context, while drawing upon
empirical results of research as one source of
support for an evaluation argument.
Challenges in evaluation technology in
language learning:
Experts in the fields of technology and language learning, including Tomlinson (2003),
Alderson and Beretta (1992), Norris et al. (2009), Garrison and Anderson (2003), and Lynch
(1996), recognize the complex difficulties associated with evaluating materials, activities,
courses, and programs incorporating new technologies. They emphasize that technology
adds intricacy because of the unique aspects being assessed. Unlike conventional
evaluation approaches used for materials or program evaluation, those dealing with
technology must appraise the quality of innovation rather than merely comparing it to the
existing methods. Following Alderson’s (1992) suggestions, the principles of evaluation
play a vital role, taking into account factors like purpose, audience, evaluator(s), content,
method, and timing.
What kind of arguments do
professionals provide to support
claims about the value of language
learning and technology ?
 01 02 03 04 05
Comparative Authenticity Corpus Pedagogy-
Theory-based
argument argument linguistic based
argument
argument argument
 COMPARATIVE
ARGUMENT
Comparative arguments in support of technology for language learning are constructed
through quantitative research, comparing students' outcomes in computer-assisted
language learning to traditional classroom instruction. This involves investigating two
groups of students undergoing the same instructional process, with the experimental
group receiving technology-assisted intervention. Statistical results quantify differences
in achievements across multiple learning outcomes like grammar, writing, speaking,
reading, and listening. Meta-analysis studies, such as Zhao's, provide substantial
support, indicating that overall, technology-using groups tend to outperform those in
the no-technology condition. Despite this, professionals in computer-assisted language
learning express concerns about methodological issues and the need for more detailed
insights to improve instruction effectively.
 AUTHENTICITY
ARGUMENT
The authenticity argument emphasizes that language tasks become valuable when they
align with the technologies students use daily. It's not about tech tasks being better than
no-tech; it's about making language learning authentic to how students communicate
and access information outside class. For example, using mobile devices in language
learning is supported because students already use them. This idea aligns with preparing
students for real-world language use. In technology-based language learning, it's
important to consider students' needs for both language skills and digital abilities,
ensuring effective use of technology through proper training.
CORPUS LINGUISTIC
ARGUMENT
The idea behind the corpus linguistic argument is that technology can help students
access the language data they need for learning. This argument is rooted in the concept
of data-driven learning, suggesting that, similar to linguists studying language, learners
can benefit from directly examining language samples. By searching a corpus of texts,
learners can find words, phrases, and grammatical constructions, solving problems
independently, raising linguistic awareness, finding models, and testing hypotheses. The
support for these claims often comes from empirical studies, varying in methodologies
from short experimental studies to more extended ecological ones. For instance, studies
have explored specific language aspects or general progress in gaining independence,
often comparing different variations of data-driven learning pedagogy.
THEORY-BASED
ARGUMENT
The idea behind the corpus linguistic argument is that technology can help students
access the language data they need for learning. This argument is rooted in the concept
of data-driven learning, suggesting that, similar to linguists studying language, learners
can benefit from directly examining language samples. By searching a corpus of texts,
learners can find words, phrases, and grammatical constructions, solving problems
independently, raising linguistic awareness, finding models, and testing hypotheses. The
support for these claims often comes from empirical studies, varying in methodologies
from short experimental studies to more extended ecological ones. For instance, studies
have explored specific language aspects or general progress in gaining independence,
often comparing different variations of data-driven learning pedagogy.
 PEDAGOGY-BASED
ARGUMENT
Teachers and developers often use pedagogical arguments to support technology in
learning, relying on established teaching principles and innovations. Unlike theoretical
approaches, these principles come from past research, practical insights, and a desire to
improve teaching with technology. Pedagogical arguments play a crucial role in
evaluating specific benefits beyond language learning theories, addressing aspects like
data-driven learning, vocabulary acquisition, or intercultural communicative
competence. These arguments guide the evaluation process, helping educators shape,
conduct, and interpret research to enhance teaching practices.
Figure 1 shows how the pedagogical and tech-nological sides are interconnected by the CoI, all aiming at the development
of the oral skill.Language learning pedagogy, specifically the communicative approach, seems like a feasible way to reach
the goals for verbal and written communication, since its focus is on language use instead of on learning language and
grammatical rules (Whong, 2011). thus, learners are encouraged to speak as much as possible in order to communicate,
emphasizing the process rather than the final product. Mistakes are treated secondarily with the teacher being careful not
to cause the learners distractions while communicating and interacting.
Improving evaluation:
How does an evaluator
decide upon appropriate
claims for a particular
evaluation?
 Improving evaluation involves understanding the types of
arguments that appear in evaluations, formulating specific
claims, selecting appropriate methods, and ensuring clear
communication of the intended claims and results of the
evaluation. Evaluators should consider the audience for the
results of the evaluation and its context, and the feasibility of
investigating the claims. They should also learn from existing
research and collaborate with stakeholders. The concepts of
argument and claim are useful for evaluators as they
communicate what their studies are and are not intended to do.
The growing number of syntheses of research on technology and
language learning need to identify the intended claims made in
the studies. The evaluation plan is an essential tool for monitoring
progress and making evidence-based decisions. Here are some
steps to improve evaluation practices:
 1.Clarify program
objectives and goals:
Understand the purpose of
the evaluation and the desired
outcomes. This will help in
Here are some developing evaluation
questions and selecting
steps to improve appropriate methods.
evaluation practices:
3.Select appropriate
methods:
Choose the most suitable
methods to collect, analyze,
and report data based on the
evaluation objectives,
research questions, and
available resources.
2.Develop evaluation
questions
Formulate specific questions
that the evaluation aims to
answer. These questions
should be aligned with the
program objectives and goals.
4.Ensure clear
communication:
Clearly communicate the
intended claims and results of
the evaluation to stakeholders.
This will facilitate the synthesis
of results and help in making
evidence-based decisions.
 5.Plan a timeline:
Develop a realistic timeline
for the evaluation process,
including data collection,
analysis, and reporting.
7.Learn from
existing research:
Review existing research and
syntheses in the field to inform
the evaluation plan and
improve the quality of the
evaluation.
6.Ensure feasibility:
Assess the feasibility of
investigating the evaluation
claims and ensure that the
methods used are appropriate
for the purpose of the
evaluation
8.Collaborate:
Engage stakeholders, including
program developers, researchers,
and other interested parties, in the
development of the evaluation plan
to ensure a collaborative and
inclusive process.
 Conclusion
EVALUATIONS TYPICALLY HAPPEN IN SPECIFIC
CONTEXTS WITH TEACHERS AND LEARNERS, BUT IN
THE FIELD OF COMPUTER-ASSISTED LANGUAGE
LEARNING (CALL), PROFESSIONALS ENGAGE IN
EVALUATIONS WHERE TEACHERS AND LEARNERS MAY
NOT REPRESENT THE COMPLETE AUDIENCE. A
COMPREHENSIVE VIEW OF EVALUATION SHOULD GO
BEYOND DECISIONS IN A SPECIFIC CONTEXT, FOCUSING
ON CLAIMS AND ARGUMENTS SUPPORTING STUDIES.
AUTHORS ARE ENCOURAGED TO BE CLEAR ABOUT
THEIR CLAIMS AND ARGUMENTS IN PUBLISHED
EVALUATION STUDIES, WITH GUIDANCE AVAILABLE IN
ARGUMENT-BASED VALIDATION IN LANGUAGE TESTING.