EVALUATION PLAN ONLINE LEARNING OBJECT 1

Evaluation Plan for 2-Way Prepositions in German Online Learning Object

Kayla Weiglein
CECH, Instructional Design & Technology, University of Cincinnati
IDT 8130: Master’s Project
Dr. Seo
January 23, 2022
EVALUATION PLAN ONLINE LEARNING OBJECT 2

Table of Contents
Abstract……………………………………………………………………………………………3
Introduction………………………………………………………………………………...……...4
Background………………………………………………………………………………….….…4
Learning Object Overview……………………...…………………………………………………5
Target Audience…………………………………………………………………………………...6
Needs………………………………………………………………………………………………7
Evaluation Methodology…………………………………………………………………………8
Evaluation Instruments………………………………………………..…………………………..9
Sampling Methodology……………………………………………………………………..……10
Analysis Procedures……………………………………………………………………...………11
Timeline………………………………………………………………………………………….12
Appendix A………………………………………………………………………………………15
Appendix B…………………………………………………………………………………..…..16
References………………………………………………………………………………..………22
EVALUATION PLAN ONLINE LEARNING OBJECT 3

Abstract
The evaluation plan for this online learning object for beginner and intermediate level German

on 2-Way prepositions in German will be assessed by a subject matter expert and adult learners

at the University of Cincinnati. The categories of focus for this evaluation are (1) Technology

Adoption including effort expectancy, attitude, facilitating conditions, and previous experiences,

and (2) Usability which includes readability and ease of use. These areas will be evaluated by

subject matter experts and target learners. Subject matter experts will navigate through the online

learning object and evaluation questionnaire before the target learners. Once the subject matter

experts provide feedback, revisions will be made to the online learning object before distributing

the questionnaire to the learners. The target learners will fully engage with the learning object

including the completion of the formative and summative assessments. The questionnaire results

will be analyzed along with the assessments in order to conduct the necessary revisions to the

learning object to increase usability and accessibility and reduce technological barriers for the

content of the learning object to be the central focus for future use in the classroom.
EVALUATION PLAN ONLINE LEARNING OBJECT 4

Evaluation Plan for 2-Way Prepositions in German Learning Object

The overarching goal of this evaluation plan is to research, gather, and interpret data of

an online learning object (Appendix A) that discusses the topic of Wechselpräpostitionen or two-

way prepositions in the German language. The online learning object encompasses a mixture of

lesson slides, formative, and summative assessments. The purpose of the learning object is to

provide students independent and self-navigational additional practice to a grammatical topic

that is taught in the second semester German classroom. Prior to implementing this learning

object in my current second semester German classroom, I hope to receive feedback and revise

my plan in the following areas: usability and technological adoption of the learning object to

ensure that it is barrier free, content and structure that is level appropriate for my audience, and

that the learning objects can be successfully and feasibly met.

Background
Second semester beginner German or German 1002 is crucial semester for adult learners

since the level of difficulty in the target language drastically increases. In German 1001, learners

are introduced to the language slowly and basic concepts such as the alphabet, numbers, and

basic conjugation are introduced. By the end of the first semester of German, students should

ideally be at an A1 language level. These levels are based on instruments used within the

Common European Framework of References for Languages: Learning, Teaching, Assessment

(CEFR). This framework was created to “provide a transparent, coherent and comprehensive

basis for the elaboration of language syllabuses and curriculum guidelines, the design of teaching

and learning materials, and the assessment of foreign language proficiency” (Council of Europe,
EVALUATION PLAN ONLINE LEARNING OBJECT 5

2022). These levels are rooted in the communicative approach which is a teaching approach used

in second language learning that places the primary focus on enhancing of communicative

proficiency which is done through the application of language rules in order to create

grammatical structures (Irmawati, 2012, p. 90). Essentially, learners focus on patterns and

memorizing phrases rather than individual grammatical concepts.

To bridge the gap between A1 and A2 is significantly easier than bridging the gap

between A2-B1. This is due to the increase in more complex grammatical structures and

introduction to new cases. One of these new concepts is 2-Way prepositions that govern both the

dative and accusative cases in German. Two-way prepositions are not only challenging, but they

are also the most frequently used. For example, the prepositions in (in), an (on), and auf (on or

on top of) are all two-way prepositions that are need early on in learning German. Additionally,

some of them are similar to English and therefore, are cause for confusion for native English

speakers.

I created this learning object to provide students with additional lessons and practice

activities that can be conducted at the pace of each individual student. Even though the initial

target audience is second semester German learners, the learning object can also be used in other

language classroom levels as part of the course for a review. The learning object was not meant

to be used as an official assessment or to replace a regular course lesson, but rather a low stakes

practice exercise to serve as a review and additional aid for struggling students or students who

tested into 1002 and have not had German since high school which could have been years prior

to entering my classroom.

Learning Object Overview

EVALUATION PLAN ONLINE LEARNING OBJECT 6

The Wechselpräpositionen learning object was created in the format of digital storytelling

where students are narrated through a series of lesson slides and activities that address the

various uses of the two-way prepositions. According to Robin (2008) digital storytelling is “the

technology application that is well-positioned to take advantage of user-controlled content and to

help teachers overcome some of the obstacles to productively using technology in their

classrooms” (p.222). This fusion of technology and traditional storytelling allows the designer

creative freedom in choosing topics that tailor to their classrooms or training and create a self-

guided narrative.

When launching the learning object, learners are introduced to a narrator and a menu that

the students can view and skip through to reach the desired content. The narration continues

throughout the entire experience for the students to receive additional guidance throughout the

entire learning process of the object. If the student chooses to go through the entire learning

object in order, they will go through a slide of learning objectives and then a menu where they

can choose a button to decide where to start. The reason for this flexibility is for students to find

the resources they want to. Since attention spans vary, students have the opportunity to narrow

down their choices that best suit their individual needs. Once the students navigate through

lesson slides, they are then provided with examples which ultimately lead to a practice exercise.

Immediate feedback is given when the learner submits their answers after completing each

exercise. The formative assessments are more explicit grammar exercises, which allow learners

to immediately practice what they learned on previous slides. According to Meyer et al. (2014),

formative assessments are defined “as a process used by teachers and students during instruction

that provides feedback to adjust ongoing teaching and learning to improve students’ achievement

of intended instructional outcomes." The summative assessments are towards the end of the
EVALUATION PLAN ONLINE LEARNING OBJECT 7

learning object and place the prepositions in more real-word contexts to show the students

contextualized instances where the prepositions are needed and why the differences in their

various definitions are vital to convey meaning in the target language. According to Meyer et al.

(2014), summative assessments “are assessments that measure what students know and do not

know at a particular point in time”. For the online learning object, the summative assessment is

in the form of a writing activity where students have to apply newly acquired knowledge into

forming a letter to a friend in the target language that requires them to utilize 2-way prepositions.

Target Audience

According to Dick et al. (2014) the target population is/are “the ones [leaners] you want

to “hit” with the appropriate instruction” (p. 96). The target audience or demographic for this

learning object would be adult learners in their second semester of German in a university

setting. Alternatively, the learners test at an A1 level in German prior to beginning the course in

which the learning object has been introduced or have two or more years of high school German.

Either way, the learners need to have previous German language knowledge at a beginner A1

level. The students are typically a mix of men, women, or non-binary learners. The students

come from various knowledge and socio-economic backgrounds which means that their German

language knowledge needs to be assessed early on in order to tailor the target language

instruction to meet the needs of all the students. At the beginner level at UC, the majority of

students are either interested in the target language and want to pursue a major or minor, or they

need to fulfill language requirements before graduating, participating in a co-op, or to be eligible

to participate in a study abroad program.

Needs
EVALUATION PLAN ONLINE LEARNING OBJECT 8

         The needs of the students are individually based. There are students who need

accommodations such as extra time on assignments, but there are overall needs in the classroom

that go beyond a form provided at the beginning of the semester. Since all of the learners come

in with unique backgrounds and abilities, it is vital to learn what works best for them as a

teaching style. Are they more visually oriented? Are some students more advanced than others?

Do some students require more explanation and guidance? Do they work well independently?

These questions are imperative to find out towards the beginning of the semester to ensure that

lessons are feasible, goal oriented, and that there is time to accommodate various needs. For

example, students in previous German 1002 courses have expressed a preference for chunking

information into small pieces rather than tacking larger topics all at once with only summative

assessments.

Evaluation Methodology

In order to evaluate my learning object, I plan on using a usability questionnaire. I

believe that this is the best fit for my artifact since it is digital and will be implemented in an

online class setting. This also allows the participate more control over their overall experience

without the pressure of time or someone questioning them in person. The overall aim is to

ascertain the usability and technology adoption of the learning object which includes the effort

expectancy, attitude towards technology usage, facilitating conditions, ease of use, and

readability. These aims, however are subject to change as the evaluation process continues and

unique needs arise after revisions are made to my initial evaluation. Since class time cannot be

taken to administer the questionnaire or have a focus group, the questionnaire would need to be

completed outside of class without any time constraints. To ensure that as many target learners as
EVALUATION PLAN ONLINE LEARNING OBJECT 9

possible will complete the online learning object and the questionnaire, extra credit will be

offered as an incentive.

I want the technology and usability components to be barrier free in order for students

spend more time engaging with the content rather than focusing on the digital environment.

According to Tcha-Tokey et al. (2016), technology adoption is “the actions and decisions taken

by the user for a future use or intention to use the virtual environment” (p. 3). This measures the

success the participants or in this case my teammates and hopefully students will have with this

type of new a perhaps unfamiliar technological lesson delivery. Tcha-Tokey et al. (2016)

describes usability as the “ease of learning (learnability and memorizing) and the ease of using

(efficiency, effectiveness and satisfaction) the virtual environment” (p. 3). Usability gauges the

usefulness of the learning object. I thought the usability and technology adoption would aid me

in determining what changes I could make to the environment itself in order to make it more

engaging and barrier free.

Technology Adoption

Technology adoption is one of the two components in my questionnaire I want assessed.

In order to do that, I am utilizing the Questionnaire for User Satisfaction (QUIS) which is

measurement tool created to “assess a computer user's subjective satisfaction with the human-computer

interface” (Questionnaire for User Interface Satisfaction | Digital Healthcare Research, 2022). It

is important to me that the content is the focus of the learning object, and the technology component is

barrier free and easily accessible. This questionnaire aids in questions regarding the screen visibility,

terminology, system information, learning factors, and system capabilities. An example of a QUIS

question is “The online learning object was simple and easy to use” (see Appendix B). This data will
EVALUATION PLAN ONLINE LEARNING OBJECT 10

allow me to make the necessary revisions to my learning object in order to simplify or break down the

material for the students as to not be overwhelming.

Usability

For then usability portion of my questionnaire, I utilized the USE questionnaire which

stands for Usefulness, Satisfaction, and Ease of use. Each of these components is what

determines user satisfaction and regularity of use that demonstrates the participants sense of what

is and what is not usable and apply this among various domains (Lund, 2001). An example of a

usability question is “the navigation of this online learning object is manageable” (see Appendix

B). This will determine how user-friendly the learning object is and will aid in future revisions of

the learning object to ensure that components such as navigation are not barriers.

Sampling Methodology

The participants included in this evaluation are a subject matter expert and a group of 7-9

university of Cincinnati undergraduate students. This sample size is conducive to the number of

target learners that I have in my current undergraduate course. The amount of target learners

varies from course to course meaning that there is no consistency in enrollments per semester.

The subject matter expert would be provided access to the learning object and the

questionnaire prior to the distribution to the students. They will navigate through the learning

object and the questionnaire in order to provide crucial feedback that I can use to revise both the

learning object and questionnaire before administering it to the student group. The feedback will

not only be helpful regarding the technical aspect of the learning object, but also the content.

Since the learning object relies heavily on previous knowledge required by the learns, it is
EVALUATION PLAN ONLINE LEARNING OBJECT 11

imperative that the content not be too challenging and that a learner from any background can

participate and succeed.

Analysis Procedures 

I plan on primary using the Likert Scale to collect statistical data along with open ended

questions to provide the participant with variety in formatting. The Likert Scale will have a five

point range from Strongly Agree to Strongly Disagree. The open-ended questions will be read by

me for commonalities in order to proceed with post evaluation revisions. Open-ended comments

will also provide an opportunity for the target learners to express their individual ideas and

thoughts.

The questionnaire I plan on creating will be conducted through a Microsoft Form that

consists of between twenty to twenty-five questions. I chose Microsoft Forms because it has a

built in Likert Scale function and it is supported by University of Cincinnati, meaning that I can

create a link for anyone with the UC organization. UC also supports Google Forms....Fifteen to

twenty questions will be close ended using a Likert scale, five will be a mixture of open-ended

questions and multiple-choice questions. I want more than just close-ended feedback because I

wanted to afford the participants an opportunity to express, in their own words, their engagement

with the artifact rather than having the answers formulated for them. The questionnaire will be

posted as a link within a module on Canvas along with the online learning object that the

participants can access at any time. The estimated time for the questionnaire is between six to

eight minutes.

Most of the assessment in the evaluation plan encompass quantitative data, therefore, a

statistical analysis will be implemented. The quantitative data in the questionnaire will be
EVALUATION PLAN ONLINE LEARNING OBJECT 12

conducted using a Likert scale. According to Nemoto (2014), a Likert scale “is a psychometric

scale that has multiple categories from which respondents choose to indicate their opinions,

attitudes, or feelings about a particular issue” (p.2). A Likert Scale questionnaire is a frequently

used instrument for measuring effectiveness such as motivation and self-efficacy. It aids in

collecting statistical data from a large body of participants in an easy manner. Since the students

will not receive a grade for the completion of the learning object and questionnaire, whether or

not they passed the activities will not negatively impact their grades and will allow them to focus

on the technology and usability rather than having the pressure of having to get all the answers

correct.

To analyze the results of the quantitative data from the questionnaire, the mean, average,

and standard deviation will be collected from both the subject matter expert and the target

learners. The questionnaire also includes open-ended questions which will need to be analyzed as

qualitative data. The questions will need to be coded in order to evaluate trends. The open-ended

questions are also categorized to fit with the quantitative data. I will thoroughly read the

responses. These are open to change based on the findings.

Timeline and Conclusion

Table 1. Proposed Timeline

Task Date

Evaluation Plan Draft January 23rd

Evaluation Plan Revision February 9th

Send Questionnaire and Learning Object to February 9th

Subject Matter Expert

EVALUATION PLAN ONLINE LEARNING OBJECT 13

Questionnaire and Learning Object due from February 11th

Subject Matter Expert

Send Announcement regarding Evaluation to February 14th

participants

Participants complete and return February 14th- February 17th

questionnaire

Evaluation Analysis February 20th

Evaluation Report Draft February 27th

Evaluation Report Revised March 13th

Artifact Revision March 27th

Portfolio Completion April 3rd

Presentation and Defense April 7th

Revisions to the evaluation plan and the creation of the evaluation tools will be

completed by February 9th, 2022, based upon feedback from both my partners and professor. The

online learning object and questionnaire is anticipated to be sent on Wednesday, February 9,

2022, and feedback will be received by or on Friday, February 11, 2022. Prior to sending out the

learning object to the target participants, revisions will be made based on the feedback received

by the subject matter expert. My target audience or in this case my 1002 beginner German

language course learners will be notified about the evaluation using email and Canvas by

Monday, February 14, 2022.

After receiving the questionnaire and learning object, learners will be given until

Thursday February 17, 2022, to utilize and navigate through the learning object and filling out
EVALUATION PLAN ONLINE LEARNING OBJECT 14

the questionnaire. Since the learning object and questionnaire are to be conducted outside of the

classroom, all the participants will have ample to time and flexibility to complete the tasks.

All the data collected from the participants will be evaluated for usability and technology

adoption by February 20, 2022. Based off the feedback received, the required revision will be

made to the online learning object by March 27, 2022. Finally, my revised and updated portfolio

that includes a round of peer feedback will be prepared by April 3, 2022, and the portfolio

defense will follow on April 7, 2022.

EVALUATION PLAN ONLINE LEARNING OBJECT 15

Appendix A

Online learning Object

EVALUATION PLAN ONLINE LEARNING OBJECT 16

Appendix B

User Experience Questionnaire

(Adapted from Lund, 2001, USE and Chin et al., 1988 QUIS Questionnaires)
EVALUATION PLAN ONLINE LEARNING OBJECT 17
EVALUATION PLAN ONLINE LEARNING OBJECT 18
EVALUATION PLAN ONLINE LEARNING OBJECT 19
EVALUATION PLAN ONLINE LEARNING OBJECT 20
EVALUATION PLAN ONLINE LEARNING OBJECT 21
EVALUATION PLAN ONLINE LEARNING OBJECT 22

References

Boone, H. N., & Boone, D. A. (2012). Analyzing Likert data. Journal of extension, 50(2), 1-5.

Chin, J., Diehl, V., & Norman, K. (1988). Development of an instrument measuring user

satisfaction of the human-computer interface. Paper presented at the, 130202 213-218.

https://doi.org/10.1145/57167.57203

Council of Europe. (2022). The CEFR Levels. Common European Framework of Reference for

Languages (CEFR). Retrieved February 6, 2022, from

https://www.coe.int/en/web/common-european-framework-reference-languages/level-

descriptions

Dick, W., Carey, L., & Carey, J. O. (2015). The systematic design of instruction (8th ed.).

Hoboken, NJ: Pearson. \

Irmawati, N. D. (2012). Communicative approach: An alternative method used in improving

students' academic reading achievement. English Language Teaching (Toronto), 5(7), 90-

101. https://doi.org/10.5539/elt.v5n7p90

Lund, A. (2001). Measuringn Usability with USE Questionnaire. ResearchGate. Retrieved

February 6, 2022, from

https://www.researchgate.net/publication/230786746_Measuring_Usability_with_the_US

E_Questionnaire

Meyer, A., Rose, D. H., & Gordon, D. T. (2014). Universal design for learning: Theory and

practice. Wakefield, MA: CAST Professional Publishing, an imprint of CAST.

Nemoto, T., & Beglar, D. (2014). Developing Likert-scale questionnaires. In N. Sonda & A.

Krause (Eds.), JALT2013 Conference Proceedings. Tokyo: JALT.

EVALUATION PLAN ONLINE LEARNING OBJECT 23

Questionnaire for User Interface Satisfaction | Digital Healthcare Research. (2022). AHRQ.

Retrieved February 6, 2022, from https://digital.ahrq.gov/health-it-tools-and-

resources/evaluation-resources/workflow-assessment-health-it-toolkit/all-workflow-

tools/questionnaire

Robin, B. R. (2008). Digital Storytelling: A Powerful Technology Tool for the 21st Century

Classroom. Theory Into Practice, 47(3), 220–228.

https://doi.org/10.1080/00405840802153916

Tcha-Tokey, K., Loup-Escande, E., Christmann, O., & Richir, S. (2016, March). A questionnaire

to measure the user experience in immersive virtual environments. In Proceedings of the

2016 virtual reality international conference (pp. 1-5).