Effective SLO Assessment in the Online Environment

Jim Julius, MiraCosta College Kevin Kelly, Wiley Learning Institute

Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute

Effective SLO Assessment in the Online Environment
Before we get started…

Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute

What is your role at your institution?
 Faculty
 Professional Development/Instructional Design  Academic Technology/eLearning  Student support staff  Administration – Academic Affairs

 Administration - Student Affairs
 Administration - Other
Effective SLO Assessment in the Online Environment - Module 1

3

Are you experienced?
HOW LONG HAVE YOU BEEN TEACHING HOW LONG HAVE YOU BEEN USING ONLINE TOOLS FOR TEACHING

 0-1 years  2-5 years  6-10 years  11+ years

 0-1 years  2-5 years  6-10 years  11+ years

Effective SLO Assessment in the Online Environment - Module 1

4

What are your SLO responsibilities?
 SLO and/or assessment committee member
 Department coordinator for SLO/assessment  Course author/creator  Instructor – conduct assessment of SLOs  Staff – support the systems where SLO data lives

 SLO lead for department
 Assessment officer
Effective SLO Assessment in the Online Environment - Module 1

5

Overall agenda

Choose appropriate assessment strategies Choose appropriate assessment tools Managing consequences of online SLO assessment Analyze, report, and act upon online SLO data
Effective SLO Assessment in the Online Environment - Module 1

6

Effective SLO Assessment in the Online Environment
Module 1: Choose appropriate assessment strategies
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute

Module 1 agenda
 (Re)imagine assessing your learning outcomes
Consider students’ level of thinking Consider students’ use of online environments

 Select appropriate assessment strategies
Consider reasons to assess in different ways Consider Universal Design for Learning principles

Effective SLO Assessment in the Online Environment - Module 1

8

(Re)imagine Assessing Learning Outcomes

(Re)imagine Assessing Learning Outcomes
 Key factors to keep in mind:
 Consider the level(s) of learning you want students to achieve or demonstrate

 Make the objectives measurable
 Avoid common misconceptions
TRUE or FALSE: “online” means all learning or assessment must take place behind a computer screen. TRUE or FALSE: “distance” education means students are located far away

Effective SLO Assessment in the Online Environment - Module 1

10

(Re)imagine Assessing Learning Outcomes Consider Level(s) of Learning
Cognitive (knowledge) - Bloom’s Taxonomy (revised)
Remembering: Recall/recognize specific information Understanding: Lowest level of understanding

Applying: Implement (use) information
Analyzing: Break information into parts and describe the relationship Evaluating: Make a judgment about materials or methods Creating: Generate new ideas, products and ways of viewing things
Image source: http://www.iftf.org

See http://itc.utk.edu/~bobannon/classifications.html#cognitive_domain
Effective SLO Assessment in the Online Environment - Module 1

11

(Re)imagine Assessing Learning Outcomes Consider Level(s) of Learning
Psychomotor (skills)
Observing: Active mental attending of a physical event

Imitating: Attempt to copy a physical behavior
Practicing: Try a specific physical activity over and over Adapting: Fine tuning, make minor adjustments in a physical activity
Image source: http://www.sharjah.ac.ae

See http://ets.tlt.psu.edu/learningdesign/objectives/psychomotor
Effective SLO Assessment in the Online Environment - Module 1

12

(Re)imagine Assessing Learning Outcomes Consider Level(s) of Learning
Affective (attitudes or feelings)
Receiving: Shows awareness. Responding: Shows attention and motivation.

Valuing: Demonstrates commitment.
Organizing: Integrates and prioritizes new values.
Image source: Karin Kirk, SERC http://serc.carleton.edu

Characterizing by value: Behavior reflects new values.

http://www.uwsp.edu/education/lwilson/curric/affectiv.htm
Effective SLO Assessment in the Online Environment - Module 1

13

Thought prompt
 To what extent is it possible for students to achieve higher level outcomes in online/distance environments?
 Harder
 Roughly the same  Easier
Effective SLO Assessment in the Online Environment - Module 1

Creative Commons Attribution: billsoPHOTO

14

(Re)imagine Assessing Learning Outcomes Consider Environment
 Tech-enabled F2F classes
 Assessing outcomes might reflect enhanced capabilities with tech

 Flipped classes
 Assessing outcomes might reflect action or performance focus

 Online classes
 Assessing outcomes might include student interaction with the world
Effective SLO Assessment in the Online Environment - Module 1

15

Activity
 Consider example outcomes. What types of assessment work via online/distance learning?  Rethink assessing your outcomes in an online/ distance context.

Effective SLO Assessment in the Online Environment - Module 1

16

Select Appropriate Assessment Strategies

Thought prompt
 Which activities are supported in online/ distance learning?  Which activities are hindered in online/ distance learning?

Effective SLO Assessment in the Online Environment - Module 1

18

Assessing performance
Different strategies used for class assessments
• Written: tests, quizzes, essays, written assignments, case studies

• Oral: recitations, student presentations, oral exams
• Manual: lab practicum, individual and team projects, skills observation What other types of assessments do you use?
Effective SLO Assessment in the Online Environment - Module 1

19

Reasons to assess learning differently
Traditional F2F Methods • Standardized testing (multiple choice, etc.) • Written work (essays, research reports, etc.) • Student presentations • Lab work • Team projects Possible F2F Issues • Does not support all learners • Hard to require student feedback in classroom • Not enough time in class; usually not recorded • Hard to make up if you miss the lab exam • Hard to determine who did what, and when
20

Effective SLO Assessment in the Online Environment - Module 1

Thought prompt
 Does one assessment provide enough information to show that a student has learned something?

Effective SLO Assessment in the Online Environment - Module 1

21

Apply Universal Design for Learning principles

 Universal Design for Learning (UDL)
 The core goal is to “Teach every student”  Principles guide instructors to provide multiple pathways for students to succeed
Representation: Share content in more than one format Engagement: Use multiple ways to make learning meaningful Expression: Provide more than one way for students to show what they know

See full presentation about UDL and online assessment
Effective SLO Assessment in the Online Environment - Module 1

22

Providing assessment choices and levels
 Provide alternative questions
 Provide different assignments

 Let students submit in different formats
 Use two or three of these strategies together

Effective SLO Assessment in the Online Environment - Module 1

23

Activity
Select assessment strategies that work for example outcomes or for your own outcomes

Effective SLO Assessment in the Online Environment - Module 1

24

Effective SLO Assessment in the Online Environment
Module 2: Choose appropriate assessment tools
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute

Module 2 agenda
 Select tools for online class activities
Consider best tool for students to achieve outcomes Consider tools for different parts of the assessment process Consider level of support for using the tools

 Put it all together with ePortfolios

Effective SLO Assessment in the Online Environment - Module 2

26

Select tools for online activities

Using technology for assessment
 Low-stakes quizzes/High-stakes exams
 Quizzes through a Learning Mgmt System (e.g., Moodle)

 Written assignments - Iterative/peer-review process
 Calibrated Peer Review  Online paper submission tools (e.g., Turnitin)  Electronic portfolios

 Written assignments - Grading/feedback process
 Screencast (e.g., Screenr, Echo360)  Tablet Apps (iAnnotate)

Effective SLO Assessment in the Online Environment - Module 2

28

Using technology for assessment
 Individual or small group presentations
 Online presentation tools (e.g., VoiceThread)

 Labs
 Commercial lab kits sent to students’ homes  “Kitchen science” lab  Virtual labs/simulations

 Intelligent tutors (e.g., ALEKS)

Effective SLO Assessment in the Online Environment - Module 2

29

Using technology for assessment
 Individual or small group projects
 Google docs or wiki pages for team organization/coordination/workflow  Blogs or ePortfolios to share and assess final products

 Performance-based assessment (science, creative arts)
 Upload observation logs by trusted expert(s)  Audio/video recording

 Other?

Effective SLO Assessment in the Online Environment - Module 2

30

Choosing technology for assessment
x: where y: when z: who physical

synchronous
collaborative

digital

individual

asynchronous

Effective SLO Assessment in the Online Environment - Module 2

31

Choosing appropriate technology
Questions to consider
• How will using this technology help students learn? • Do students have access to this technology? • Do students have access to training and support to use this technology? • Is this technology accessible to students with disabilities?

Effective SLO Assessment in the Online Environment - Module 2

32

Discussion
• Which technologies have you used for your classes?
• To assess performance

• Which technologies are you considering for assessment?
• Web 2.0 tools and other online environments • Mobile technologies • ePortfolios

Effective SLO Assessment in the Online Environment - Module 2

33

Choosing appropriate technology
 Resources to investigate
• Virtual resource site - teaching with technology
 http://www.umuc.edu/virtualteaching/

• Discussion-based online teaching
 http://cgi.stanford.edu/~dept-ctl/tomprof/postings.php (#561)

• How technology influences student success • MERLOT & Pachyderm (CSU tools)
 http://www.merlot.org; http://www.pachyderm.org

Effective SLO Assessment in the Online Environment - Module 2

34

Examples: Integrating Technology
 English - ENG 214 – Kory Ching
 Writing in a Digital Age

 Instructional Tech - ITEC 299 – Kevin Kelly  How 2 Lrn w ur iPod

Effective SLO Assessment in the Online Environment - Module 2

35

Activity
1. Make a teaching-with-technology plan
Objectives Resources Activities Assessments

2. Share elements of your plan with the group

Effective SLO Assessment in the Online Environment - Module 2

36

The role of rubrics
 Tie rubric criteria to specific outcomes
 Repeatedly assess across multiple assignments

Effective SLO Assessment in the Online Environment - Module 1

37

Consider support for assessment tools
 Campus apps (e.g., LMS features—quizzes or tests, assignment submissions)
 3rd party apps (e.g., Turnitin, Google Picasa)  Publisher tools (e.g., WileyPLUS)  Digital feedback tools (e.g., digital rubric within ePortfolio, audio/video/screencast feedback)

Effective SLO Assessment in the Online Environment - Module 1

38

Put it together with ePortfolios

Gathering evidence over time
 Electronic portfolios allow different stakeholders to view progress of

 ePortfolio resource site: http://eportfolio.sfsu.edu

Effective SLO Assessment in the Online Environment - Module 1

40

Align with Outcomes at Multiple Levels

Effective SLO Assessment in the Online Environment - Module 1

41

ePortfolios enable…

Effective SLO Assessment in the Online Environment - Module 1

42

Assessment via ePortfolios
 ePortfolios to document:
 Self-assessment  Peer review  Instructor feedback (rubric)

Effective SLO Assessment in the Online Environment - Module 1

43

Assessment examples  Rubric comments can be shared in addition to automated systems

Effective SLO Assessment in the Online Environment - Module 2

44

Synthesis

foster learning improvement elevate self evaluation cultivate critical thinking

provide guidelines & grades

require attention to purpose

provide feedback & rubrics

reward peer review

encourage use game & share make exemplars collaboration & UDL elements a priority

Effective SLO Assessment in the Online Environment - Module 2

45

Added benefits of ePortfolios
 Demonstration of student achievement
 Accreditation  Program Assessment  Student Bridge to Transfer/Workforce  Student Recruitment

Effective SLO Assessment in the Online Environment - Module 1

46

Effective SLO Assessment in the Online Environment
Module 3: Managing consequences of online SLO assessment
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute

Potential Concerns
 Institutions may be out of compliance with regulations re: DE student authentication
 Faculty may not be employing online course design practices designed to encourage academic integrity

 Institutions may need to update policies to reflect regulations and/or good practice
 Faculty workload issues?
Effective SLO Assessment in the Online Environment - Module 1

48

Potential Benefits
 Students may develop “showcase” resources beneficial to their next steps
 Thinking about improving online SLO assessment may result in improvements to on-ground courses

 Other?

Effective SLO Assessment in the Online Environment - Module 1

49

Discuss: Authentication & Academic Integrity
 See handouts: WCET best practices & statements from regulators & institutions
 Do your institutions have policy that “enforces” these ideas?  What practices do your institutions follow (or not) that might help implement these ideas, regardless of policy?
Effective SLO Assessment in the Online Environment - Module 1

50

Discuss: Authentication & Academic Integrity
 Do you (should you) require use of an institutional course management system for online assessment?  What do you do about third party systems?  Under what circumstances should proctoring be encouraged? Required? And what should “proctoring” include?
Effective SLO Assessment in the Online Environment - Module 1

51

Effective SLO Assessment in the Online Environment
Module 4: Analyze, report, and act upon online SLO data
Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute

Potential Concerns
 Institutions may not be collecting/ aggregating assessment data in useful ways related to outcomes  Institutions may not be disaggregating assessment data into DE and non-DE  Institutions may not be seriously looking at DE vs. non-DE results and following through with improvement plans
Effective SLO Assessment in the Online Environment - Module 1

53

Activity
 In your table, look at the data packet provided. Assume you are all part of an academic discipline/department.  What does the data tell you? What does it not tell you? What questions does it cause you to ask?
 What is required to get buy-in for this kind of review process?
Effective SLO Assessment in the Online Environment - Module 1

54

Effective SLO Assessment in the Online Environment - Module 1

57

Effective SLO Assessment in the Online Environment - Module 1

58

Points to Ponder
 Do tools and processes support meaningful collection & analysis of course SLO data by course modality? Is there faculty buy-in for this?  Are DE courses defined, tracked, and reported in a consistent manner?
 Is it possible/reasonable to make comparisons across departmental, college-wide and statewide DE data? What are the limitations?
Effective SLO Assessment in the Online Environment - Module 1

59

Points to Ponder
 Do you have a way of tracking DE program outcomes? (Do you have a way of tracking DE programs?)  What will the ultimate outcome be of SLO mania combined with ratcheting up DE standards?

Effective SLO Assessment in the Online Environment - Module 1

60

Session Resources
● Universal Design for Learning (UDL) and Online Assessment (Prezi)
● Language on DE Authentication (Google doc) ● WCET Best Practice Strategies for Academic Integrity in Online Ed (PDF) ● Making Online Assessment More Secure (1-pager) (.doc)

Contacts: ● Jim Julius : jjulius@miracosta.edu ● Kevin Kelly: kkelly@wiley.com

Sign up to vote on this title
UsefulNot useful