Running head: SLO MOD 4 1

Student Learning Outcomes Training Module 4: Assessment

Lynnette Mann

California State University, Monterey Bay

July 20, 2018

IST622 Assessment and Evaluation

Dr. Bade Su
SLO MOD 4 2

Table of Contents
Background ......................................................................................................................... 4

Purpose of Instruction ..................................................................................................... 5

Design of Prototype ............................................................................................................ 6

Learning Objectives ........................................................................................................ 7

Learner Analysis ............................................................................................................. 8

Prototype Assessment ......................................................................................................... 8

Test Sample..................................................................................................................... 8

Test Instruments .............................................................................................................. 9

Learning Effectiveness (Gains)....................................................................................... 9

Usability Assessment .................................................................................................... 10

Summary ........................................................................................................................... 10

References ......................................................................................................................... 11

Appendices ........................................................................................................................ 12

Appendix A: Results of Training Opening Survey ....................................................... 12

Appendix B: Results of Training Closing Survey ........................................................ 12

Appendix C: Assessment of Training ........................................................................... 13

Appendix D: Usability Test Results ............................................................................. 15

Appendix E: Notice to Test Participants ....................................................................... 16

SLO MOD 4 3
SLO MOD 4 4

Student Learning Outcome Training

The Accrediting Commission for Community and Junior Colleges (ACCJC) is the

accrediting authority for the 114 community colleges of California. Colleges do not take the

process to accreditation lightly and invest significant time and resources to comply with the

process. The consequences of losing accreditation are dire: the institution would suffer the

forfeiture of federal funding as well as federal financial aid to its students, the institution could

not award degrees, and athletics programs would have to be closed down.

The guidelines for the accrediting process are laid out in a published set of standards:

ACCJC Accreditation Standards 2014. One significant component of the Standards, ACCJC

Standards I.B.1 and 3, II.A.1-3, 11 addresses student learning outcomes, and the remainder of

this report will focus on ways to satisfy the criteria laid out there (ACCJC Accreditation

Standards, 2014).

Background

In March 2017, as part of the accreditation process, the External Evaluation Team

representing the ACCJC performed an onsite evaluation of West Hills College Coalinga

(WHCC). Several compliance issues were noted and relayed to the college and governing

commission via the team’s Evaluation Report (Snow-Flamer, 2017) as well as a letter to the

college president (Winn & Rodriguez, 2017). As of July 2017 West Hills College is on

“warning” by the ACCJC and reaffirmation of accreditation is delayed. Although the college

remains accredited, we have eighteen months to rectify the compliance issues that were noted,

otherwise we may face additional sanctions and possible loss of accreditation.

SLO MOD 4 5

This report will address recommendations 9: 2nd bulleted item and 10, which fall within

the scope of ACCJC Standard II.A (Student Learning Programs and Support Services:

Instructional Programs).

Purpose of Instruction

The compliance issues arise from the college’s lack of progress in the development of

quantifiable Student Learning Outcomes along with regular assessment of students at the course

level using SLOs with defined rubrics. These SLOs are clearly defined statements used to

measure a student’s competency. The recommendations to the college reference noncompliance

with ACCJC Standards:

• Recommendation 9 (2nd bullet) aligns with Standards I.B.1, II.A.1-3,11 and 16

fully and meaningfully assess all courses, certificate and degree programs using

student learning outcomes assessment to improve student learning and ensure

that faculty and staff fully engage in the student learning outcomes assessment

process (Snow-Flamer, 2017)

• Recommendation 10 aligns with Standard II.A.3

In order to meet Standards, the team recommends that the College ensure that all

course syllabi include the approved student learning outcomes and that the

officially approved course outlines contain student learning outcomes. (Snow-

Flamer, 2017)

This training module will provide West Hills College faculty and staff an orientation and

training in the development and use of effective and measurable Student Learning Outcomes

(SLOs). The e-learning module will consist of lessons, practice drills, and quizzes on the

rationale for, and the techniques for building SLOs. These lessons will include advice, direction,
SLO MOD 4 6

and other forms of feedback within the content to guide the learner to form critical thinking and

decision-making skills regarding the creation and assessment of effective and measurable SLOs.

Design of Prototype

The prototype designed and developed is the model for one of the four planned modules

which comprise the capstone project: Module 4: Assessment. The objective of the content of this

prototype module is to focus the learner’s attention on using appropriate assessment mechanisms

aligning with the language of well-written SLOs. In the series of modules for the capstone, this is

prototype will be the fourth of four modules.

The module was built using Adobe Captivate software to be deployed to learners using

the college’s Canvas learning management system. All Canvas dashboards will have the SLO

training module added for easy access and availability.

The features of Captivate made it the logical solution. The application allows for the

distribution of content with audio including captioning. As the module progresses, the learner is

guided through the materials interspersed with activities to support active learning and promote

practice. The module opens with addressing the objective and introduction of common

assessment tools used in the academic setting. Then the emphasis is directed to language used in

SLOs, cognitive domain verbs.

There are six areas of cognitive domain verbs: Remembering, Understanding, Applying,

Analyzing, Evaluating, and Creating. After each set of two areas of cognitive domain the learner

is provided verbal (and captioned) instructions for an activity. This practice complements the

context of aligning commonly used cognitive domain verb in SLOs and classifying the verbs as

measurable or not. The learner has the opportunity to review all six areas before moving to

another practice activity. This second type of interactive practice concentrates on classifying
SLO MOD 4 7

selected verbs to either lower or higher cognitive domains. Instructions for interacting with the

practice are provided verbally (and captioned). After the learner submits their attempt, the

answers are revealed. As the module continues the content addresses aligning assessment and

SLO language and how the language aligns with the order of cognitive domain. The learner then

has another opportunity to exercise what was taught. This activity provides the learner practice in

categorizing assessment with cognitive order (higher/lower/both). The module ends with a

summary.

Learning Objectives

Terminal objective for the capstone project

• Given a scenario, the learner will compose a student learning outcome meeting all

criteria of a well-written student learning outcome.

• Given a scenario, the learner will evaluate student learning outcome with

corresponding assessment for alignment of cognitive domain.

Terminal objectives for the prototype module

(Enabling objectives for the capstone project)

• Given examples, the learner will recognize types of assessment tools with 100%

accuracy.

• Given a set of assessment tools, the learner will classify the assessment tool with

the context of the cognitive domain, with 90% accuracy.

• Given a list of cognitive domain verbs, the learner will distinguish between low-

and high-order verbs with 90% accuracy.

• Given a scenario, the learner will select the appropriate assessment for an SLO

with 90% accuracy.

SLO MOD 4 8

Learner Analysis

There are two target groups of learners: WHCC faculty and staff. Assessment of college

courses is within the purview of the faculty. It is the charge of the faculty to develop curriculum,

build SLOs for that curriculum, and to devise and implement corresponding assessment tools,

and assess SLOs on a regular basis. Therefore, all faculty are invited to complete the proposed

training module as well as those selected staff (including administrators) who work directly with

the SLO and curriculum processes.

SLOs have been a hot topic of discussion for the past several months due to the

increasing attention given to the accrediting report and the current issues with WHCC being out

of compliance with ACCJC’s standards. The training will address some common misconceptions

or preconceived notions, give the learner foundational information about SLOs (including

rationale), and provide the basis for developing effective, measurable SLOs. Learners will have a

range of previous knowledge and therefore the training is designed in such a way that all learners

will gain some new and/or corrective knowledge of well-written SLO.

Prototype Assessment

Test Sample

To test the prototype for the intended outcome (learning) and usability a convenience

sample was use consisting of fellow instructors (colleagues), students, and family. Colleagues

have similar background and prior knowledge as the anticipated faculty target learners. The

students and family, generally, had little knowledge of the subject before completing the training

which would be similar to the intended college staff target learners. Twenty-two participants

were in the sample test group.

SLO MOD 4 9

Test Instruments

Test instruments will differ between the prototype and the final capstone. For the

capstone project the learners test for content competency will be built into the modules and

scores housed in Canvas. Since the sample test group did not have access to Canvas a secondary

means of testing had to be instituted. For the prototype, Google surveys were implemented for

pre, post, and usability teasing. Links to the surveys were provided to the sample group via email

and Canvas notifications (Appendix E) with instructions for order of completion: pretest,

training, then posttest.

The pretest (Training Opening Survey) consisted of ten content-based questions. The

participants’ mean score was 5.86 with a standard deviation of 1.88. The posttest (Training

Closing Survey) consisted of the same ten content-based question. The participants’ mean score

was 8.18 with a standard deviation of 1.70. The graph of the result score for the pretest is fairly

normally distributed (Appendix A) whereas the post test is left skewed (Appendix B) which is

the intended outcome.

Learning Effectiveness (Gains)

After actively participating in this training, faculty and staff will increase their knowledge

to be able to: recognize types of assessment tools; classify assessment tools with agreement of

the cognitive domain: distinguish between low-and high-ordered cognitive domain verbs; and

select appropriate assessment for an SLO. A pretest was given to the participants and scores were

recorded. The participants actively worked through the training. A posttest was then given to the

participants. Posttest scores were recorded. Using a dependent statistical t-test with a confidence

of 95% there is enough evidence to support the claim that learning has taken place (that is

knowledge has increased) (Appendix: C).

SLO MOD 4 10

Usability Assessment

In addition to the posttest the participants were asked to evaluate the training within the

context of usability. The questions focused on the organization of the content, ease of use, and

quality of the language used. Participants provided feedback as to which areas could be

improved ( Appendix: D). One issue arose with the last activity; the pacing did not permit

completion of the task. Additionally, instructions for activities will be analyzed for clarity or

need for rewording. These suggestions will be incorporated into the project prior to submission

of the final capstone project.

Summary

The prototype was developed with the target audience, college faculty and staff, and

designed with ease of use and interactive learning in mind. The conclusion of the results of the

tests phase is learning is taking place. Some participants encountered issues with parts of the

module and shared their experience in the feedback of the usability test. This feedback will guide

modifications to the prototype prior to releasing the final version of the project. Overall the

project is heading towards success.

SLO MOD 4 11

References

ACCJC Accreditation Standards. (2014). Retrieved from The Accrediting Commission for

Community and Junior Colleges: https://accjc.org/eligibility-requirements-standards-

policies/#accreditation-standards

Educational Technologies. (2016, December 6). Blackboard to Canvas Migration. Retrieved

from University of Missouri: http://canvas-

migration.missouri.edu/support/solutions/articles/11000007641-adaptive-release-in-

canvas

Office for Civil Rights. (1990). Americans with Disabilities Act (ADA). Retrieved December 9,

2017, from U.S. Department of Education:

https://www2.ed.gov/about/offices/list/ocr/docs/hq9805.html

Snow-Flamer, K. (2017). Evaluation Team Report. Novato: ACCJC.

Winn, R., & Rodriguez, R. (2017, June 23). West Hills College Coalinga Accreditation.

Retrieved from West Hills College Coalinga ACCJC Letter:

http://www.westhillscollege.com/coalinga/about/accreditation/documents/20170623-

whc-accjc-letter.pdf
SLO MOD 4 12

Appendices

Appendix A: Results of Training Opening Survey

Appendix B: Results of Training Closing Survey

SLO MOD 4 13

Appendix C: Assessment of Training

Participant Before After

1-BB 6 8
2-CM 8 8
3-DM 9 10
4-DR 6 6
5-GL 6 9
6-GP 3 8
7-IY 6 7
8-JY 6 9
9-JR 5 10
10-KR 6 10
11-LG 4 7
12-LJ 6 7
13-MD 5 10
14-ML 4 7
15-MAL 4 7
16-MAR 3 9
17-OF 7 10
18-RB 9 10
19-SG 9 8
20-SA 3 3
21-SC 7 9
22-YV 7 8

t-Test: Paired Two Sample for Means

Before After
Mean 5.863636 8.181818
Variance 3.551948 2.917749
Observations 22 22
Pearson Correlation 0.437033
Hypothesized Mean Difference 0
df 21
t Stat -5.68676
P(T<=t) one-tail 6.04E-06
t Critical one-tail 1.720743
P(T<=t) two-tail 1.21E-05
t Critical two-tail 2.079614
SLO MOD 4 14

t stat (abs) 5.686761

critical value 2.079614
p-value 1.21E-05
critical value < t stat yes
p-value < alpha yes
reject null hypothesis yes
statistically significant difference yes

Effect size (mean1 - mean2)/larger standard

dev -0.13521
larger than 0.2 no
larger than 0.5 no
larger than 0.8 no
practically significant no

H1: μ1 ≠
H0: μ1 = μ2 μ2
Reject
Decision: H0
There is enough evidence to support the claim that the mean scores (for
before the training and after the training) are not equal.
SLO MOD 4 15

Appendix D: Usability Test Results

Question 1: The module is easy to navigate.

Question 1: If answered No: What would make the navigation more straight forward?

Question 1: Results

Yes 91%
No 9%

Question 2: The content is well organized.

Question 2: If answered No: What recommendations do you have to improve the organization?

Question 2: Results

Yes 100%
No 0%

Question 3: The directions for the activities are easy to follow.

Question 3: If answered No: Which activity were the directions unclear?

Question 3: Results

Yes 91%
No 9%

Question 4: Did you notice any spelling or grammar errors?

Question 4: If answered Yes: Describe the location in the module you saw any spelling/grammar

errors. If you know the slide number, please include that in your response.

Question 4: Results

Yes 23%
No 77%
SLO MOD 4 16

Question 5: Did you experience any issues with the pacing of the module? (Too fast/slow)

Question 5: If answered Yes: Please describe the pacing issue you ran into.

Question 5: Results

Yes 23%
No 77%

Appendix E: Notice to Test Participants

MATH 025 – Introduction to Statistics - Extra Credit Opportunity - For Lynn's Studies
Lynnette Mann
All Sections
Jul 09 at 1:54am
.
EXTRA CREDIT OPPORTUNITY
For my own studies I need to gather data regarding some training I've created. If you have the
time AND it doesn't interfere with your own studies please participate as my beta testers for part
of my Capstone project.
If you do choose to participate - it shouldn't take more than 30 minutes to complete all three
parts. The length of time that it might take you are just estimates. Your time is valuable so, if you
do all three (3) parts I will give you 10 points extra credit.
PART 1: (5 minutes)
Please complete the "before" quiz (10 multiple-choice questions). I'm not expecting you to know
any of the answers and it's OK if you don't. Just do the best you can do.
Here's the link you will need to click on: Before Quiz (Links to an external site.)Links to an
external site.
PART 2: (15-20 minutes)
Now run through the actual training. This training is intended for teachers to go through so the
topic probably won't be very interesting to you - sorry about that :-) The basic idea of this
training is to help teachers match the right type of test with what we want students to learn.
It's kinda like this... you wouldn't give a student a multiple-choice test if you wanted them to
paint a picture of fruit. It's just not the right type of test for the action.
Here's the link for the training: Module 4: Assessment (Links to an external site.)Links to an
external site.
PART 3: (5-8 minutes)
SLO MOD 4 17

Time for me to gather more data and it's your turn to take the "after" quiz (which are the same 10
questions as the "before" quiz). I need to gather data to see if "learning" took place. Again,
please do your best on the quiz.
At the end of the quiz there are a few extra questions about the design of the training. Pretty
much these questions are to see if I need to fix anything so the training works better. Please be
honest - you won't hurt my feelings. If something is broke, I need to fix it.
Here's the link you will need to click on: After Quiz (Links to an external site.)Links to an
external site.
The deadline for completing this is: 11:59pm, Sunday, 16-July.
Thanks!!
See you Tuesday,
Lynn