Evaluation Report

Content
Overview....1
Subject-matter expert
review.....1
One-on-one
evaluations....3
Small group evaluation...6
Data Collection
Instruments..13
Results.....22
Recommendations/Conclusions.....29

Overview
The purpose of this evaluation was to determine the effectiveness of the self-paced online
instructional unit that has been created and developed for the University of Memphis. The unit
will teach learners about the history and importance of engaged scholarship at the university.

The unit was evaluated by the Subject Matter Expert (SME) Jennifer Barker, professors at the
University of Memphis, and former and current students of the university. The results of the
evaluation has helped in developing and improving the content of the unit and to determine the
time requirements needed to complete the unit. The objective of the unit is to inform the
students and community of the University of Memphis of how engagement is beneficial to both
parties.

Subject-Matter Expert Review

Purpose
The purpose of a subject-matter expert review is to help the designer develop and validate
information in the online instructional unit. They are needed to make changes in the design and
information of the unit so that it will properly serve the purpose of the needed clientele. The
SMEs opinions are needed because she is able to help make sure the information is correct
and relevant. For this unit, the SME was given an electronic version of the unit, the Change
Request Form, and a SME Evaluation Form. The SME reviewed the contents and objectives to
determine:

How well the unit addressed the objectives given

What content could be added to improve the success of meeting the objectives

What content could be deleted

Data Collection Methods and Procedures

The SME was given the SME Evaluation Form and the Change Request Form. She was
instructed to go through each of the five (5) modules with our Project Manager via the phone.
The SME was to write down any changes that she felt were needed and anything that she
wanted to see. She was then to write down these changes on the request form from which the
designers would go through and determine if those changes would be honored and the
reasoning behind the decisions.

Data Analysis/Results/Recommendations
The results of the SME Evaluation were listed on the Change Request Form. The wanted,
accepted, and denied changes are as follows:

Description of Change
Wanted

Approved/Rejected

Description of Change
Made

Content: Edit to Michigan

State Credit, Add comma

Approved

Made changes listed

Content: Correct all

suggested changes to
wording and grammar

Approved

Made changes listed

Content-Branding: I believe
the new branding removes
the spaces in the
abbreviation of the
University's name; change all
U of M to UofM

Approved

Made changes listed

Content: Edit Welcome

page as described below

Approved

Made changes listed

Content: Eliminate Tools of

Engagement as descriptor
for online modules; call them
learning modules, or just
modules.

Approved

Made changes listed

Graphics: Alter supportive

data to match UofM themed
colors, described below

Approved

Made changes listed

Graphics: Adjust the size of

the navigation arrows to
match height of progress bar,
continue to keep rightalignment

Rejected

Reason: Size change effects

ease of learner navigation
and time constraints

One-on-One Evaluations
Purpose
The purpose of a one-on-one review is to help the designer determine the effectiveness of the
instructional unit on a real participant. They are needed to allow the designer to watch a user go
through the unit and be able to answer questions/comments in real time. Overall, the designer
was able to gain feedback that will allow for valuable changes to be made.

Data Collection Methods and Procedures

The three participants were given the website of the online instructional unit that included the
online pre/post tests. There were five (5) pre/post tests that contained 5-7 multiple choice,
true/false, and open-ended questions corresponding to the objectives. The Observation Notes
Form contained sections for the participant to ask questions and note comments pertaining to a
specific module and page. The Interview Protocol Form contained eight (8) sections for the
participant to comment both positives and negatives, as well as, four (4) open ended questions.
The attitude survey contained 10 multiple choice and short answer questions that demonstrate
how the user felt while taking the self-paced unit.

Data Analysis and Results

The results of the Attitude Survey and the pre and post tests were combined with the small
group evaluations so that individual data will not be shown, but the combined data can be found
below in the Results section of the Evaluation Report.
The three (3) participants gave the following questions and comments via the Observation
Notes Form and the Interview Protocol Form:

Module 3 (Assessment):

The final question does not make sense

Content:

1st question says nothing about community stakeholders in the answer choices,
but correct answer does. Didnt seem like there was a good answer choice

Very informative

Example stories were very helpful in making sure/reinforcing what I need

Practice:

Pre-tests should be embedded in the modules

All practices were beneficial for comprehension

People will skip if possible, so maybe make typing in box a requirement

Feedback:

Positive and simple which is good

Gained more knowledge of engage community and capacity building

Readability:

Good

There are several questions in the pre/post tests that have typos or wording that
simply doesnt make sense

User or reader friendly

Very easy to read grouping and bullets were helpful

Layout:

Easy to follow through modules

Switching from module 1 to module 2 was tricky, but then used back arrow

Good

Good

Graphics:

Nice choice- didnt distract me from my lesson

Good

Good

Colors:

Nice and represent UofM colors

Fine

Ease of Use:

Easy to use

Need to imbed pretests

Overall, how do you feel about this unit?

Would be helpful if you were considering volunteering. May scare some off if they
are just looking into a little volunteer work

Good

Did you feel that this unit would be effective for audiences needing this information?

Yes, breaks down attitude, expectations and what you are getting into

Yes

What part of this unit did you find most appealing?

Explanations of differences and how to deal with them

The people you are helping are not lower than you and can be a valuable asset

Asset-based vs. Deficit based approach

Capacity building and good partnership

Would you recommend taking this unit to a friend?

Yes if they were going to get into real volunteering

Yes, but without the constant assessment (pre/post tests in particular) These felt
like they were there to benefit the university or whoever developed the module,
not to sid in the learners understanding of the content

Yes

Recommendations
It is recommended by the design team that:

The assessment questions would not be changed due to them being approved by the
SME in the previous evaluations. She felt that the questions were fine as is, so they
were not changed.

The comments made and the observations made showed that there may be confusion in
moving on to the next module. This was decided by the design team that action on that
would be tabled until the small group evaluations to determine if this was a consistent
issue.

Typos were corrected in the assessments.

Small Group Evaluations

Purpose
The purpose of a small group evaluation is to help the designer determine the effectiveness of
the instructional unit on a real participant. They are needed to allow the designer to watch a
user go through the unit and be able to answer questions/comments in real time. Overall, the
designer was able to gain feedback that will allow for valuable changes to be made.

Data Collection Methods and Procedures

The seven participants were given the website of the online instructional unit that included the
online pre/post tests. There were five (5) pre/post tests which contained 5-7 multiple choice,
true/false, and open-ended questions corresponding to the objectives. The Observation Notes
Form contained sections for the participant to ask questions and note comments pertaining to a
specific module and page. The Interview Protocol Form contained eight (8) sections for the
participant to comment both positives and negatives, as well as, four (4) open ended questions.
The attitude survey contained 10 multiple choice and short answer questions that demonstrate
how the user felt while taking the self-paced unit.

Data Analysis and Results

The results of the Attitude Survey and the pre and post tests were combined with the one-to-one
group evaluations so that individual data will not be shown, but the combined data can be found
below in the Results section of the Evaluation Report.
The seven (7) participants gave the following questions and comments via the Observation
Notes Form and the Interview Protocol Form:

Module 1:

Need back button or instructions to return to module if a link is clicked

Module 2:

Why the need for the personality test?

Is the personality tests optional?

Module 4:

The last question needs to be reworded

Content:

Modules a little long. Hard to remember when taking assessments

Very informative

Example stories were very helpful

Practice:

Pre-tests should be imbedded in the modules

All practices were beneficial for comprehension

People will skip if possible, so maybe make typing in box a requirement

Feedback:

Positive

Great way to break up dense info- gives participant time to pause and reflect

It was engaging

Good

Fine

Readability:

Good

It was very easy to read, but there was a lot of content

Simple

Easy to read

User or reader friendly

Very easy to read grouping and bullets were helpful

Layout:

Easy to follow through modules

Need a way to go back or forward at the end of a module

Very nice looking

Professional looking site. Blends into the UofM site well

Good

Good

Graphics:

Nice choice- didnt distract me from my lesson

Good

Good

Helped break up the monotony

Wish there were more

Appropriate and nice

Colors:

Nice and represent UofM colors

Fine

UofM colors

The muted colors are nice- less likely to be distracting

Good tones

Ease of Use:

Easy to use

After an assessment, I wasnt sure how to access the next module. I had to scroll
around and figure it out myself

Not sure how to get to the next module after test

Liked the forward and back arrows

All links worked

Some modules lacked continuity- some you clicked a hyperlink for more info,
others a button

Overall, how do you feel about this unit?:

I enjoyed reading the info- very informative and practical. I did read through the
1st and 2nd module entirely, but the last three I just skimmed through. Hard to
stay focused for all at one time.

Great info, very long

I really enjoyed the information presented and think it is great for the university.

Nice

Good

Did you feel that this unit would be effective for audiences needing this information?:

Yes, breaks down attitude, expectations and what you are getting into

Yes

Yes- definitely recommend reading modules over time- not in one seating

Yes

Yes, although it is a lot of material and may need to be broken up over multiple
seatings

What part of this unit did you find most appealing?:

I really liked the graphics

It is comprehensive and addresses many aspects of community engagement. I

was pleasantly surprised to see content on power and privilege, identity, etc.
these are very important and Im glad you included them.

10

As faculty- I enjoyed the amount of information provided and the links to research
externally. As a student, I might feel overwhelmed.

Information and practical examples

Would you recommend taking this unit to a friend?:

Yes if they were going to get into real volunteering

Yes, but without the constant assessment (pre/post tests in particular). These felt
like they were there to benefit the university or whoever developed the module,
not to sid in the learners understanding of the content.

Yes

Recommendations
It is recommended by the design team that:

The assessment questions would not be changed due to them being approved by the
SME in the previous evaluations. She felt that the questions were fine as is, so they
were not changed.

The comments made and the observations made showed that there may be confusion in
moving on to the next module. This was decided by the design team that action on that
would be tabled until the small group evaluations to determine if this was a consistent
issue.

11

Data Collection Instruments

SME Notes Form

12

13

Observation Notes Form

Interview Protocol Form

14

15

16

Attitude Survey

17

18

19

Pre and Post Tests

Module 1
https://docs.google.com/forms/d/18BBnpEsiuf6Oo0wxyGn-8lq8vYs3diLcAbbKpKXJl1s/viewform

Module 2
https://docs.google.com/forms/d/1rYC3qUm_InDFmotnYVAivvADY9Xj0Aoq7I6R8XZsBpI/viewform?usp
=send_form

Module 3
https://docs.google.com/forms/d/16P3gWxUYl5W0Ggzgx1inLv5MQEmLa6Xgskrv461LDA/viewform?usp=send_form

Module 4
https://docs.google.com/forms/d/10SxHeLdgrAQzhs7fL7NsciKW6MNrCOxLckLq0eKHKc/viewform?usp=send_form

Module 5
https://docs.google.com/forms/d/1OhTRupM61cqeJCHNkN1dZ065wliH4KO5aflBbA2mSY/viewform?usp=send_form

20

Results
Assessment Items by Objectives
Module 1
Objectives:
1. What it means for the U of M to be an engaged institution with an urban-serving
research history
2. The responsibility that the urban-serving research mission places on faculty and
students to engage with community partners
3. How engagement enhances the undergraduate academic experience
Question 1
Objective
1
Objective
2
Objective
3

Question 2

Question 3

Question 4

Question 5

Question 6

Question 7

21

Module 2
Objectives:
1. How people differ in their levels of power and privileges
2. How being sensitive to these differences allows people to
communicate and work more effectively with each other
3. How culture shapes identity and relationships
4. How to be aware of devaluing or privileging one perspective over
another
Question 1
Objective
1

Question 2

Question 3

Question 4

Question 5

Objective
4

Question 7

Objective
2
Objective
3

Question 6

X
X

X
X

22

Module 3
Objectives:
1. Understand the practices associated with partnership building.
2. Understand the skills you need to be a good partner.
Question
1

Question
2

Question
3

Question
4

Question
5

Question
6

Question
7

Question
8

Objective
1
Objective
2

23

Module 4
Objectives:
1.

Understand how asset based community engagement is a particular way of framing

community work.

2.

Compare and contrast asset based and deficit based approaches to community work.

3.

Identify seven types of community assets associated with people and place.

4.

Consider why word choice and language is important in framing community work.

5.

Understand the benefits and challenges associated with asset based community
engagement.

6.

Recognize the value of asset based community engagement in creating community

empowerment and sustained change.

Question
1
Objective
1

Question
2

Question
4

Question
5

Question
6

Question
7

Question
8

Objective
2

Objective
3

Objective
4

Objective
5
Objective
6

Question
3

24

Module 5
Objectives:
1. Learn that capacity building represents a significant mindset shift in community work
2. Define capacity, capacity building, and capacity building mindset
3. Appreciate the importance of culture and context in capacity building
4. Consider what you might do to build capacity in your community work
5. Understand the challenges and benefits of capacity building
6. Recognize why capacity building is essential in creating sustained change in
communities

Objective 1
Objective 2
Objective 3
Objective 4

Question 1

Question 2

Question 3

Question 4

Question 5

Question 7

X
X
X

Objective 5
Objective 6

Question 6

X
X

25

Pre and Post Tests

Here are the results of the pre and post tests broken up by module and question. The line in
blue shows the number of participants that got that question correct on the pre-test, the orange
line shows the number of participants that got the same question correct on the post-test, and
the grey line shows the total number of people taking that given assessment. The results are
shown in graph form below. It is noted that not all module assessments have the same number
of participants taking it. This is due to some participants choosing not to take a given
assessment.

26

27

28

Attitude Survey Results

These are the results of the Attitude Survey that the one-to-one and small group participants
completed at the end of the entire unit. As the graphs show, the majority of the responses to
how the participants felt about each part of the unit is shown in yellow and dark blue. This
showed that they agreed or agreed strongly which in turn was a positive for the instructional
unit. The overall feeling towards the unit as a whole was very positive.

29

Recommendations/Conclusions
It is the recommendation of the design team to have the final product be considered as
complete. All changes have been made and the unit has been reviewed by the design team.
The evaluations show that the overall feedback was positive and that the content and design
were pleasing to the overwhelming majority. The results of the pre and post tests show that
each module taught the desired objectives and upon review of the modules, the participants
showed growth in their knowledge. The design team will hand off the product to the client for
them to send to Michigan State university for review.

30

Client/Sponsor Sign Off

___________________________________________________

Jennifer Barker, Subject Matter Expert

Department of Architecture
University of Memphis Engaged Scholarship Committee

___________________________________________________

Dr. Carmen Weaver, Project Sponsor

The University of Memphis

___________________________________________________

Kaylah Holland, Project Manager

Visible Innovation

_________________

Date

_________________

Date

04/28/2016
_________________

Date

31