Running head: DIAGNOSTIC ASSESSMENT FAMILIARIZATION 1

CALIFORNIA STATE UNIVERSITY

MONTEREY BAY

Diagnostic Assessment Familiarization

CAPSTONE Report

Submitted in partial satisfaction of requirements of the degree of

MASTER OF SCIENCE in

Instructional Science and Technology

Luay Askar

April 24, 2018

Capstone Approvals: (At least one advisor and capstone instructor should approve)

Dr. Donald Fischer ___________________________ _____________

Advisor Name Signature Date

Dr. Miguel Lara ___________________________ _____________

Capstone Instructor Name Signature Date
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 2

Table of Contents
Executive Summary ............................................................................................................ 4

Introduction and Background ............................................................................................. 4

Problem Description ....................................................................................................... 6

Target Audience .............................................................................................................. 6

Context Analysis ............................................................................................................. 7

Content Analysis ............................................................................................................. 7

Literature Review............................................................................................................ 9

Solution Description ......................................................................................................... 10

Goals of the project ....................................................................................................... 10

Learning Objectives ...................................................................................................... 11

Learning Theories and Instructional Strategies ............................................................ 11

Analysis of tasks/contents ............................................................................................. 16

Media and Delivery System Decision........................................................................... 16

Methods & Procedure ....................................................................................................... 17

List of Major Deliverables ............................................................................................ 17

Process and Steps .......................................................................................................... 18

Resources .......................................................................................................................... 19

Timeline/progress report ................................................................................................... 20

DIAGNOSTIC ASSESSMENT FAMILIARIZATION 3

Development plan ......................................................................................................... 20

Implementation plan/usability ...................................................................................... 20

Evaluation &Testing Plan ................................................................................................. 21

Formative Evaluation .................................................................................................... 21

Summative Evaluation .................................................................................................. 21

References ......................................................................................................................... 22

Appendices ........................................................................................................................ 24

Appendix A ................................................................................................................... 24

Appendix B ................................................................................................................... 25

Appendix C ................................................................................................................... 25
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 4

Executive Summary
The learning process is a collaborative work of both the instructor and the learner.
Implementing the student-centered approach in the classroom has changed the role of the teacher
from “Sage on Stage to Guide on the Side” (King, 1993). The teacher needs to facilitate, to
prepare materials, and to assess the students. There are different types of assessments using
different tools, all aiming at gathering data, and the teachers require being familiar with the
diagnostic tool they will be using for the assessment. “Assessment is the process of gathering
data. More specifically, assessment is the ways instructors gather data about their teaching and
their students’ learning” (Hanna & Dettmer, 2004).
Diagnostic Assessment is a tool that allows learners to focus on the very difficult tasks of
reading, listening and speaking at the proficiency levels of four-year and six-year university
language majors in course lengths ranging from 36 weeks to 64 weeks depending on the
difficulty of the language.
The teachers should be familiar with the different tools needed to conduct an assessment
of their learners in order to guide and support them in the learning process. A training module
will be designed to familiarize teachers with the basic principles of Diagnostic Assessment (DA).
Different learning theories and instructional strategies will be used in designing and developing
this training module. Adobe Captivate, this smart authoring platform, will be used to create a
fully responsive training module. The implication of this training will enhance the teaching-
learning process through its impact on both the teachers and the learners. Teachers will have an
idea about their teaching strategies and plans that will allow them to modify it. Learners will be
able to identify their areas of strength and weakness and receive appropriate recommendations.

Introduction and Background

The instructor requires knowledge of the diagnostic tools and thus the DA in order to
implement diagnostic teaching and differentiated instruction in the classroom and outside the
classroom for both formal and informal assessment.
Diagnostic Assessment (DA) is an educational tool used to identify learners’ areas of
strengths and weaknesses for each of the language proficiency skills (Reading, Listening and
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 5

Speaking). Alderson (2004) came up with a list of features that can help to clarify the benefits of
DA, of these:
 Identify strengths and weaknesses in a learner’s knowledge or use of language.
 Have a focus on weaknesses that lead to remediation in further instruction.
 Enable detailed analysis and report of responses to items or tasks.
 Provide feedback that can be acted upon.
The proposed training module is designed to introduce instructors at the Defense
Language Institute (DLI) to the principles of Diagnostic Assessment (DA). The targeted audience
is newly joined and seasoned instructors, aiming at familiarizing the former to the basic
principles and the process of conducting DA, and as a refresher training for the latter. This
familiarization training is not a replacement for the DA specialist certification training, but it will
help the teachers to conduct informal DA for their students in order to support them during the
learning process. According to current regulations, only certified DA specialists write formal DA
reports.
The mission of the DLI is to provide culturally based foreign language education,
training, evaluation and sustainment to enhance the security of the nation. At the DLI, the time
allocated to each language program differs depending on the category of the language being
taught. Recently, the DLI set up a new standard for students to pass the final exams with the
goal of achieving an Interagency Language Roundtable (ILR) level of 2+/2+/2, in reading,
listening and speaking skill modalities, respectively. This goal is a challenge for both the
students and the teachers. For the students, acquiring a new language is not a personal goal but
also a job-related goal, thus requiring more effort and dedication. As for the teachers, their
standards surpassed the traditional teaching standards to include and not limited to;
implementing more advanced technology in the classroom as well as implementing diagnostic
teaching and differentiated instruction in their instructional model. Traditional teaching
following the rule “one size fits all” is no longer acceptable, if we are aiming to achieve the DLI
goals of a proficiency level of 2+/2+/2 in the three skills (listening, reading, and speaking).
Therefore, this familiarization training is to introduce the teachers to the basics of DA,
what and how to prepare material for DA, how to construct DA questions, when to conduct it,
which student requires DA and how to conduct it.
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 6

Problem Description
The DLI is aiming towards teachers adhering to a student-centered teaching methodology
with the teachers being facilitators that guide the students through their learning process. Also,
the teachers are required to prepare teaching materials and activities to enhance learning in
addition to preparing assessment material to diagnose areas that need improvement for each
learner. This should be followed by an interventional instructional plan to overcome the
difficulty. Being a trainer for the DA specialist certification course, I noticed that most of the
participants lack the basic knowledge about DA, and the conducted needs analysis confirmed
that. This problem has a direct negative influence on the outcome of the teaching-learning
process, and thus on achieving the organizational goal.

Target Audience
Data collected from the learners’ analysis was provided from four sources; from the
direct supervisors through their class observations and from follow-up checks of teachers’
performance. The other source is direct interviews (sensing sessions) with teachers conducted by
the academic specialist and the dean. The third source is the questionnaire survey given to the
teachers by the end of the DA certification course (retrospective data collection). The last source
is a questionnaire I conducted recently through Survey Monkey (see Appendix A for the
questionnaire). The responses collected from the survey include suggestions from the
participants that, if they had been enrolled in familiarization training, it would have made their
certification course more productive. Another issue is the diversity of knowledge about the basic
principles of DA among the participants attending the certification course. This varies from a
novice with lack of knowledge to experts requiring honing of their experience. The results of the
survey conducted using Survey Monkey revealed that around 30% of the responders have little
information about DA, 39% less often conduct DA and another 39% conduct it with moderate
frequency. Also, around 50% of the respondents have less often been exposed to a DA report.
These figures confirm that the audience has knowledge about DA ranging from a novice with
lack of knowledge to experts that have had limited experience with DA or have practiced it in a
limited way. Collectively, all these analyses support the proposal of conducting familiarization
training that will cover the issues of DA, as well as set a solid ground for the DA certification
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 7

course, which is a long training process. In addition, such training will be considered as a
norming session for the certified DA specialists.

Context Analysis
This training is faced with a logistical problem of time constraints. The time that will be
allotted to the training will affect the teaching hours of the teacher. The training for diagnostic
teaching is already being conducted at the DLI by faculty development specialists over four
hours, usually one morning or afternoon. The proposed DA familiarization training is not a
substitute for the DA specialist certification training course. The DA certification training
requires 64 hours of face-to-face instructions in addition to assignments, practicum and the
submission of a final portfolio. This will increase the out-of-classroom time for the instructors
and thus their teaching hours which is part of the teachers’ standards.
During the proposed familiarization training, the trainees will not conduct DA for a
student as in the DA certification training. Therefore, it will start as blended with an instructor-
led session for the first iteration so as to pinpoint areas that require modification or adjustment.
Then the following iteration will be self-instructional, asynchronous training delivered to the
participants on whatever platform is available at DLI. Also, instruction on how to proceed in the
training will be given as a Portable Document Format (PDF). The time intended for the blended
course is 1-2 days and for the training around 4-6 hours, but the latter will depend on the
instructors’ time as they can do the training at their pace.
This training will save time for the organization and for the instructor and will not require
a classroom or a conference room to conduct the training. The teachers can undertake the
training at their workstation and they can pause the training in order to attend a class or to
perform any other task. Also, the availability of MacBook and iPad for each teacher in addition
to the high-speed internet will facilitate the use of the self-instruction and asynchronous training.

Content Analysis
The DA certification course consists of five Modules (M); M1(personal profile), M2(text
selection & constructing questions), M3(dynamic assessment, recall protocol & writing
assessment), M4(DA interviews), and M5(differentiated instructions & instructional plan). This
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 8

proposed familiarization training is intended to use the principles of the two main modules
(M2&M4) that contain the practical information required for the DA interview process. The
training will consist of four modules:
 Module One (ILR familiarization &passage rating) will contain information
related to selecting reading passages to be used during the DA interview. The
level of this material is according to the ILR descriptors, these descriptors will be
provided to the audience as an attachment of soft copy or can download it from
the link http://govtilr.org/Skills/ILRscale1.htm. The audience will be provided
with reading sample passages at different levels. The audience will assign a level
of the passages and check their selection with justification according to the ILR
descriptors.
 Module Two (ILR familiarization &passage rating) will contain information
related to selecting listening passages to be used during the DA interview. The
level of this material is according to the ILR descriptors, these descriptors will be
provided to the audience as an attachment of soft copy or can download it from
the link http://govtilr.org/Skills/ILRscale1.htm. The audience will be provided
with listening sample passages at different levels. The audience will assign a level
of the passages and check their selection with justification according to the ILR
descriptors.
 Module Three (DA familiarization) will familiarize the learners with the DA
interview protocol for the three skills (speaking, reading, and listening), where the
audience will be familiar with the concept of DA interview for the three skills.
 Module Four (DA familiarization) will familiarize the learners with the DA
interview reports for the three skills (speaking, reading, and listening). The
audience will be provided with the electronic forms of speaking, reading, listening
skill and for the recommendation. Filled forms will be provided for reference and
to have an idea how to complete the report and recommendation forms.
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 9

Literature Review
Many of the studies conducted about assessment approach the topic from the learners’
perspective, while few articles approach the topic from the teachers’ perspective. In addition,
studies are mainly directed towards the different types of assessments and specifically towards
Diagnostic Assessment (DA). The review of the literature was done to support the proposed DA
familiarization training.
Two articles, one by Blood (2011) and another by Dunn & Mulvenon (2009), mentioned
that diagnostic assessment is a dynamic process used to assess, evaluate and give feedback to
learners. Diagnosis, from the perspective of Blood (2011), is a term familiar in the medical field
but in second language acquisition, broadly speaking, diagnosis is any form of assessment
whether formative or summative that yields feedback. According to Dunn & Mulvenon (2009),
“Although an assessment may be designed and packaged as a formative or summative
assessment, it is the actual methodology, data analysis, and use of the results that determine
whether an assessment is formative or summative” (p.2). Blood also emphasized that post-
assessment feedback be “ample, detailed and innovative”.
The two authors emphasized the importance of the content of the assessment in revealing
more accurate data; Blood questioned whether the tasks used by the teacher to construct
assessment material are appropriate for a formative or a summative assessment, while Dunn &
Mulvenon questioned if the assessment will still be called formative if the teachers use it for
grading.
In conclusion, differentiation between the two forms of assessment (formative and
summative) should not be according to their nomenclature. The main points of differentiation
one should keep in mind are; where the assessment is used, when it is used and how it is used.
The most important question teachers should keep in their mind is; what they are going to do
with the collected data? Answering this question will give the appropriate description or name of
the assessment.
In a study conducted in Colombia by Mendoza (2009), there was a difference between
teachers’ perception and the use of assessment in the classroom and “there is a tendency to use
traditional assessment instead of alternative assessment”(p.55). These findings are confirmed by
the study of Tsagari (2016) about the assessment orientation of teachers in two Mediterranean
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 10

countries, where teachers concentrate on the traditional use of assessment with contents limited
to assessing vocabulary and grammar and which lack creativity and higher order thinking. In
both articles, the assessment used by the teachers tends to be more summative than formative. In
addition, both articles recommend that teachers need more training in language assessment.
Mertler (2003), conducted a study comparing the in-service teachers with the pre-service
teachers (recent graduates) in regards to “assessment literacy”. This article confirms what
Mendoza (2009) and Tsagari (2016) recommended in their studies; that teachers need more
training in regard to assessment. In addition, Mertler mentioned, “Previous studies have reported
that teachers feel and actually are unprepared to adequately assess their students”(p.22). Also, the
teachers believed that they did not have sufficient training to build-up the skills in assessing the
learners (p.22). He compared in this study pre-service with in-service teachers with respect to
many standards including learners’assessment and found that in-service teachers scored higher
than pre-service. Mertler proposed a question for teacher preparation faculties as to whether on-
the-job (in the classroom) assessment training is more of a preferred approach than training
during undergraduate study. This study highlights the importance of understanding the concept
of assessment, training and experience that will be gained from classroom application. This study
emphasizes on implementing more in-classroom training for the pre-service teachers during their
undergraduate study.

Solution Description

Goals of the project

The proposed DA familiarization training is linked in concept to diagnostic teaching.
Diagnostic teaching is based on using diagnostic tools during the teaching-learning process in the
classroom, while the DA is mainly conducted outside the classroom. The benefit of the former is
the ability to identify the learner’s areas that require improvement within the teaching hour and
that could be related to that specific hour’s contents or contexts. Also, it gives the teachers an
idea of their lesson plan, activities, teaching strategies, and what sort of modification can he
implement. As for the DA, the desired outcome is broader and gives an idea of what major issue
is taking place in the learning process of that particular learner. This DA is neither related to a
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 11

specific teaching hour, nor to a certain topic. It gives an overall reflection of the learner’s
performance from the beginning of the course to the current time.
The goals of this DA familiarization training include and are not limited to:
 Increase awareness of the instructors about the importance of DA in revealing the
areas of strength and areas that need improvement at the current time for that
specific learner/student.
 Explain to the teachers the importance of ILR levels and their implementation in
DA through a selection of level-appropriate texts.
 Demonstrate to the instructors how to conduct DA interview for the three skills;
reading, listening and speaking.
 Illustrate to the teachers how to fill and complete DA report forms for the three
skills and that for the recommendations.

Learning Objectives
The instructor at the end of this DA familiarization training will be able to:
 Identify and select level-appropriate reading and listening material following the
ILR descriptors as a guideline with 85% accuracy.
 Demonstrate knowledge of the DA protocol from memory, and the steps to be
taken into consideration before, during, and after the interview with 100%
accuracy.
 Analyze the DA report written by the DA specialist, following the protocol as a
guideline with 85% accuracy.

Learning Theories and Instructional Strategies

Implementation of learning theories requires in-depth analysis of the principles of each
theory and how it justifies the learning process. The learning theories emerged chronologically as
separate theories, but in real-world it is difficult to separate the main theories or to draw a sharp
line separating one theory from another. The instructional material integrates a variety of
learning theories and instructional strategies to develop memorable and effective instruction
(Ertmer and Newby, 1993).
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 12

Following Pavlov’s model, Skinner in 1930’s pointed out that the outcome produced by a
response was the important event in changing behavior (Skinner, 1953, as cited in Gredler,
2009). Taking Behaviorists’ point of view, learners are provided with a stimulus (a text in this
capstone) with a targeted response (assigning the appropriate ILR level) through the application
of instruction provided by the instructor (ILR descriptors). The most important factor influencing
learning is the sequence of introducing the texts after reviewing the ILR descriptors
(environmental conditions). The introduction of the texts will be in a sequential manner from
lower to higher levels along with Behaviorists’ concepts. Memory is not typically addressed by
behaviorists and the use of periodic practice or review serves to maintain a learner’s readiness to
respond, Schunk, 1991(cited in Ertmer, 1993). Transfer of learning is a result of generalization of
learned material or skill, in this case applying the appropriate ILR descriptors to assign an
appropriate level to a text. Behaviorist strategies that are incorporated in this capstone
strengthen the stimulus-response (text-ILR level) association and facilitate learning by recalling,
association and generalization. According to Ertmer (1993), the trainer’s job is to provide cues to
elicit the desired response, provide practice and arrange environmental conditions for the learner
to make the correct response.
The “Black Box” of the behaviorism theory constitutes the cognitive part of the learning
process as theorized by the early cognitivists; Lewin (1951), Bloom (1956) and Jean Piaget
(1960s). In cognitive theory learning is concerned not so much with what learners do but with
what they know and how they come to acquire it, Jonassen, 1991b (cited in Ertmer, 1993).
Participants in this capstone require higher-order thinking, therefore, they should participate
actively in the learning process. Because assigning an ILR level to a text cannot be practiced or
performed following an equation and is not a straight-forward application of descriptors, the
participant is required to analyze the text, i.e. use higher-order thinking to correlate the
descriptors to the text and assign it a level. Following a cognitivist approach, learning is
influenced by the learners’ thoughts, beliefs and values and memory is the storing of the
knowledge and the ability to retrieve it. In addition, cognitivism requires the learners’ readiness,
motivation and awareness in order to function or perform the task, that is applying the ILR
descriptors to assign a level for a target language text. The strategies used include simplification
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 13

through chunking the training material, which is in this case, giving each ILR level individually
while covering one skill at a time (reading, listening and speaking).
The cognitivism theory is one of the important learning theories that an instructional
designer should understand. According to (Ertmer & Newby, 1993) “cognitive theories focus on
the conceptualization of students’ learning processes and address the issues of how information
is received, organized, stored and retrieved by the mind.” In addition, when designing
instruction, the designer takes into consideration the transfer of learned information and this is
based on many factors like the way the instruction is delivered, learners’ prior knowledge and the
context of the learning process. Ertmer & Newby (1993), mentioned the difference between
behaviorists and cognitivists focus on knowledge transfer, where the former focuses on the
environment and the latter stresses on the processing strategy.
The participants in this training have different background knowledge about DA and
about the ILR descriptors and the correlation and application of the two. Implementing
cognitivism theory is through providing the participants with the ILR descriptors and the
authentic text and asking them to apply the new information and their knowledge to assign an
appropriate level for each text. Through this instructional strategy, participants will perform the
steps of receiving, organizing, storing and later retrieve it in order to transfer it during the
application that is text selection and rating.
The theory of social cognitivism of Albert Bandura plays a role in this training through
providing models of different authentic texts rated according to the ILR descriptors. This will
evolve self-efficacy in the participants to perform rating and prepare level appropriate material
for the use in the classroom and for DA interviews.
Instructional theories are design-oriented, “they describe methods of instruction and the
situations in which those methods should be used, the methods can be broken into simpler
components methods, and the methods are probabilistic (Reigeluth, 1999, p.7). Gagne’s nine
events of instruction (Gagne, 1985) and Merrill’s first principles of instruction (Merrill, 2002)
are the two instructional design theories adopted to design this training. Some of the steps of
these theories are not applicable for this training, but as a follow-up for on-job application, i.e.
after training implementation. The application of Merrill’s phases for effective instruction (figure
1) is implemented in this training design.
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 14

Each module is built on a problem. For example, module one involves selecting
appropriate ILR level passages., Therefore, the training will start with activation of the
participants’ previous knowledge followed by demonstration while the participants observe
(providing ILR descriptors and rated texts samples). Then learners will apply this knowledge to
rating texts provided. The final step of integration requires a follow up when the learners
integrate the new knowledge into their everyday situation.

INTEGRATION ACTIVATION

PROBLEM

APPLICATION DEMONSTRATION

Figure 1. Merrill’s Phases for Effective Instruction

The other instructional theory that is also applicable is the Nine Events of Gagné (figure
2). The on-screen agent will attract the attention of the learners to the subject. This is followed
by providing the objectives through defining the ILR scales and their use followed by the
activation of prior knowledge through presenting the descriptors (activation of schemata). Then,
examples will be provided to enhance learning and provide guidance, after which the learners
will practice and elicit performance. The learners will receive feedback when submitting an
incorrect answer leading them to the appropriate ILR descriptors. The performance will be
assessed the results of the MCQ. The ninth event of Gagné (enhance retention and transfer) is
enhanced through performing the exercises.
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 15

Figure 2. The Nine Events of Gagné with the steps for the ILR Familiarization training.
This training module follows the basics of the ADDIE model, the analysis is performed
for both the organization and learners(teachers). The design of the training module consists of
different sections; one section presents the ILR descriptors and activities related to them like
matching, assigning level for passages. The other section of the module includes an introduction
to DA interview protocol, the interview report forms, and step by step instruction on how to
complete these forms. Some of the material used to develop this training is adapted from
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 16

previous ILR training and from training designed for DA certification. The adapted material is
modified for online training purposes.

Analysis of tasks/contents
During the training, two types of test items will be used; declarative and procedural.
 From memory, the audience will be given a matching quiz. They will match the
ILR descriptive with the level with 100% accuracy.
 Given the ILR descriptors, the audience will read a passage and will assign a level
to that passage with 85% accuracy.
 Given the ILR descriptors, the audience will listen to a passage and will assign a
level to that passage with 85% accuracy.
 From memory and from the DA protocol handout, the audience will identify the
right steps of a DA interview with 100% accuracy.
Participants will receive feedback in different ways; the matching activity will give an
immediate feedback upon failure to match correctly. Feedback for assigning an incorrect level
for passages (reading and listening) will be given to the participant by providing the correct level
and leading him to the appropriate ILR descriptors.

Media and Delivery System Decision

Following the analysis, a suitable delivery mode for the participants is a blend of
instructor-led and self-instruction. Thus, the former will be synchronous and the latter will be
asynchronous. The instructor-led session may be required at the beginning as a delivery mode.
Later on, the delivery mode for the training will be self-instruction. This mode will enhance
knowledge acquisition, knowledge retention and problem-solving required for this training. In
addition, the self-instruction mode will reduce the out-of-class hours, is more appropriate
because the contents are relatively stable, and the training will be repeated often.
Participants will receive instructions on the same platform where the training will be
delivered. There will be step-by-step instructions on the opening page in addition to a PDF file
attached to download and print if required. All the material will be available for the participants
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 17

to access through their computers as each participant possesses a MacBook that contains the Mac
side and virtual Windows side, in case the participants are not familiar with Mac.
I will consider using Adobe Captivate in designing this training and the participants will
be able to access the published module either on a portable CD or it can be uploaded to the share
drive of the DLI.
Challenges that may be anticipated include those related to the design and development
of the module those will be discussed with the academic advisor and capstone instructor. One of
the challenges was conducting the survey for audience analysis and that required taking
permission and performing an online training before using the survey monkey. Challenges that
appeared after developing the activities using google forms and google quiz is the inability to use
these at the DLI so I am intending to reconstruct the activities using Captivate. I am expecting to
face some administrative implication challenges that I have to deal with accordingly.

Methods & Procedure

List of Major Deliverables

The main list of deliverables in this proposed training includes and is not limited to the
following:
 The Survey Monkey questionnaire was created and delivered to the faculty after
taking all the official permission. The link to the questionnaire sent to faculty via
DLI email. https://www.surveymonkey.com/r/CBB9SR2
 Participants will be provided with a link to access detailed ILR descriptors.
http://govtilr.org/Skills/ILRscale1.htm
 Knowledge check activities created in Adobe Captivate.
 Reading texts with different ILR levels will be selected and delivered to
participants in Adobe Captivate for the text rating activity.
 Listening passages with different ILR levels will be selected and delivered to
participants in Adobe Captivate for the text rating activity.
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 18

 Using Google forms to deliver MCQ quiz about reading ILR descriptors
https://docs.google.com/forms/d/e/1FAIpQLSccj1gAgd4Seavhyi6nS2vmFjht77IpLn
ZPnIuzcibATgX_7g/viewform?usp=sf_link
 Using Google forms to deliver MCQ quiz about listening ILR descriptors
https://docs.google.com/forms/d/e/1FAIpQLSeQA2FzSjBLQhag1GI00SgiBGganQ
XbjSeLsd1j8s7BoIqBBA/viewform?usp=sf_link
 Using Google forms to deliver quiz about DA interview protocol.
https://docs.google.com/forms/d/e/1FAIpQLSfVeGw7_ti1t_fqdHTgvoOsaYVrLmm
BlHSfXewLHEtT7lMjRQ/viewform?usp=sf_link
 Upload DA report electronic forms so that the trainees will be able to download
and use it in the future.
 Feedback will be delivered to participants during each step when the participant
gives a wrong answer.

Process and Steps

The use of self-instruction online training will overcome the time constraints and the

scheduling issues for the teaching faculty members. In addition, such training sessions will

ensure that all the teachers will have equal opportunity to receive essential training. The module

will be developed in Adobe Captivate 2017. The steps in developing this project after conducting

needs analysis and deciding on the design will include:

 The first module will present ILR descriptors for reading skill followed by a

knowledge check, then passage rating activity to check comprehension. Feedback

will be provided when submitting a wrong answer.

 The second module will present ILR descriptors for listening skill followed by a

knowledge check, then passage rating activity to check comprehension. Feedback

will be provided when submitting a wrong answer.

DIAGNOSTIC ASSESSMENT FAMILIARIZATION 19

 The third module will present the DA interview protocol, where the participant

will review the items of the protocol. This is followed by knowledge check then a

comprehension check quiz.

 The fourth module will be an overview of the DA reports and recommendation

forms followed by questions regarding the analysis of the contents.

Resources
The training material will include material previously developed in addition to newly
developed or created ones. The resources that are needed to complete this project within the
assigned timeline include:
Adobe Captivate 2017 released $ 350
Adobe Audition Free
Camtasia 9 Free through CSUMB
Norton antivirus $ 80
Laptop computer HP $ 650
Ink Jet printer $ 35
Printer cartridge $ 54
Designer fee $ 35/hr., 8hr/week, total around $1000
Internet connection $ 61/month, $ 366 for six months
CD-ROM/DVD Free
The total amount $ 2,535
The shared folder used to upload the training modules will be on the DLI server;
therefore, requiring no eligible fees. The assistant of the school Information Technologist may be
required. In addition to the technical skill acquired during the MIST classes, the designer will
seek assistance from CSUMB teaching faculty especially those related to video production and
Adobe Captivate 2017.
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 20

Timeline/progress report

Development plan

This capstone is a requirement for the master's degree in the MIST program which is a 16
months extended program started on August 22, 2017. The expected graduation date is January
2019.
 The capstone as a requirement for the IST 522 course started in August 2017 and
the first draft was submitted on December 10, 2017.
 The module one (ILR familiarization/reading) is designed as a requirement for the
IST 526 course started on March 26, 2018 and will be submitted on May 18,
2018.
 The module two (ILR familiarization/listening) will be designed during the
months of June, July and August 2018.
 The module three (DA interview protocol) will be designed during the month of
September 2018.
 The module four (DA interview forms) will be designed during October 2018.
 During the months of November and Mid-December 2018, I will compile the four
modules and adjust accordingly.

Implementation plan/usability

This training is planned to be completed online as self-instructed training. To conduct a

“Beta Test”, the facilitators and the SMEs will take the training in order to test the contents and
how they are connected to each other. In addition, they will respond to a usability questionnaire
and can give a recommendation on applicability, timing and any technical issue (see Appendix B
for the questionnaire). This is followed by a “Pilot Test” where a number of targeted audience
will take the training in a classroom or conference room under the supervision of the
trainer/facilitator. Also, the participants will respond to a usability questionnaire (see Appendix
B for the questionnaire).
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 21

Evaluation &Testing Plan

Formative Evaluation
During training, the trainees will have different forms of activities that they require to
submit in order to get feedback. These activities are constructed on Google forms, and thus when
submitting the form, results will appear to the trainer or facilitator and will give an idea about the
completeness of the training and the accuracy of the trainee’s performance. In order to have the
best results out of this, the designer will share Google forms with the trainees’ facilitators and
supervisors.
The design of the training is to be used by different trainees and on different occasions,
so to evaluate the usability of the training module, the trainee will answer an evaluation
questionnaire (see Appendix B for the questionnaire). Also, looking at the training records will
give an idea about compliance, that is whether the training material has been implemented as
designed or not.

Summative Evaluation
The trainee will answer a Kirkpatrick Level 1 (reaction ) evaluation questionnaire (see
Appendix C for the questionnaire). This evaluation will give the opportunity to have feedback on
certain areas that require revision in the future. Level 2 (learning) evaluation will include a test
instrument like matching ILR activities, assigning a rate to a reading or listening passage, and
filling in a DA report after watching a mock DA interview. The facilitator will follow up on the
learners’ performance and whether the knowledge has been retained by the learner and
transformed to in-job practice. In Level 3 (application) evaluation, the trainees will apply the
information they gained in the classroom and in their job. This evaluation can be recognized by
their direct supervisors through reflection at their counseling reports and on students’
performance. As for evaluation level 4 and 5, they are beyond the scope of this project.
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 22

References
Alderson, J. C. (2004). Diagnosing Foreign Language Proficiency. pp.11-12 London:
Continuum.
Blood, I. (2011). Diagnostic second language assessment in the classroom. Columbia
University academic commons. Working Papers in TESOL & Applied Linguistics, 11(1),
57-58. Retrieved from https://doi.org/10.7916/D8ZG6RTP.
Defense Language Institute [Website]. Retrieved from
http://www.dliflc.edu/about/mission-vision/
Dunn, K. and Mulvenon, S. (2009). A critical review of research on formative assessment: the
limited scientific evidence of the impact of formative assessment in education. Practical
Assessment Research & Evaluation,14(7). Retrieved from
http://pareonline.net/getvn.asp?v=14&n=7
Ertmer, P. A. & Newby, T. J. (1993). Behaviorism, cognitivism, constructivism: Comparing
critical features from an instructional design perspective. Performance Improvement
Quarterly, 65 (4), 50-92.
Gagné, R. (1985). The conditions of learning and the theory of instruction (4th ed.). New York:
Holt, Rinehart, and Winston.
Gredler, M. (2009). Learning and instruction: theory into practice. Upper Saddle River, NJ:
Pearson Educational Ltd.
Hanna, G. S., & Dettmer, P. A. (2004). Assessment for effective teaching: Using context-
adaptive planning. Boston, MA: Pearson A&B.
King, A. (1993). From Sage on the Stage to Guide on the Side. College Teaching, 41(1),
30-35.
Mendoza, A. & Arandia, R. (2009). Language testing in Colombia: a call for more teacher
education and teacher training in language assessment. Profile Issues in Teachers`
Professional Development, 11(2), 55-70. Retrieved from
http://www.scielo.org.co/scielo.php?script=sci_arttext&pid=S1657-
07902009000200005&lng=en&tlng=en.
Merrill, D. (2002). First principles of instruction. Educational Technology Research and
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 23

Development, 50(3), 43-59. Retrieved on 3/17/2018 from

https://www.indiana.edu/~tedfrick/aect2002/firstprinciplesbymerrill.pdf
Mertler, C. (2003). Preservice versus inservice teachers' assessment literacy: Does classroom
experience make a difference?. ERIC. Retrieved from https://eric.ed.gov/?id=ED482277
Piskurich, G., M. (2015). Rapid instructional design: Learning ID fast and right, Hoboken, NJ:
John Wiley & Sons, Inc.
Reigeluth, C. (1999). Instructional design theories and models: a new paradigm of instructional
theory. Mahawah, NJ, Lawrence Erlbaum Associate, Publishers.
Tsagari, D. (2016). Assessment orientations of state primary EFL teachers in two
Mediterranean Countries. Center for Educational Policy Studies Journal / CEPS Journal,
6(1), 9-30. Retrieved from https://ojs.cepsj.si/index.php/cepsj/article/view/102
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 24

Appendices

Appendix A
Survey Monkey Questionnaire

1 little 3 moderate 5 a lot

1. How much information do you know about DA? 1 3 5

2. How often do you conduct DA? 1 3 5

3. How often do you read a DA report? 1 3 5

4. How familiar are you with ILR levels? 1 3 5

5. How confident are you in selecting level appropriate reading

1 3 5
passages?

6. How confident are you in selecting level appropriate listening

1 3 5
passages?

7. How confident are you in constructing level-appropriate questions

1 3 5
for the selected passages?

8. How familiar are you with the principles and structure of OPI (oral
1 3 5
proficiency interview)?

9. How confident are you in using computers? 1 3 5

10. How familiar are you with using different software to record and edit
1 3 5
audios?

https://www.surveymonkey.com/r/CBB9SR2
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 25

Appendix B

Appendix C
Level 1 Evaluation
DIAGNOSTIC ASSESSMENT FAMILIARIZATION 26