Using Mixed Methods to Assess the Efficacy of a First-Year Experience Course and Program

Kevin Coughlin, Dean, Institutional Research, Planning, and Effectiveness Eileen DeLuca, Dean, College and Career Readiness

Edison State College

Open access, baccalaureate-granting state college. Five-county service district comprises three counties along Floridas southwestern Gulf coast, and two inland counties.

Annual Enrollment Data

Table 5.2 Five-Year Unduplicated Headcount by Ethnicity, District 2007-08 African American 1,810 2008-09 2,122 2009-10 2,686 2010-11 2,839 2011-12* 2,757

Asian/Pacific Islander
Caucasian Hispanic

387
11,881 2,959

427
13,226 3,565

468
14,598 4,163

502
14,519 4,112

484
13,906 3,906

Native American
Two or More Races Not Reported Total Year-to-Year % Change Five-Year % Change

42
--449 17,528

54
--354 19,748 12.7%

73
303 999 23,290 17.9%

79
1,018 1,541 24,610 5.7%

63
1,494 1,425 24,035 -2.3% 37.1%

*2011-12 data include end of term summer/fall data and beginning of term spring semester data .

Achieving the Dream Risk Factor Data

Five risk factors from the July 2009 Achieving the Dream report (July/August 2009, Data Notes) were applied to Edison State College FTIC students.

Source: Edison State College Institutional Research, Planning and Effectiveness (2012). Focal Point.

Achieving the Dream Risk Factor Data

Source: Edison State College Institutional Research, Planning and Effectiveness (2012). Focal Point.

Development of Quality Enhancement Plan (QEP)

August 2008-College initiated engagement in internal and external research to identify the QEP topic that would have the greatest impact on student learning. Fall 2009-College embarked upon a nine-month, externally guided self-study using the Foundations of Excellence in the First College Year (FOE) assessment model. April 2009-QEP focus emerged as a unique version of a firstyear experience (FYE) course with a focus on developing critical thinking and success strategies among students.

Research Base for QEP

Various models of first-year seminars have been shown to have a significant impact on students in terms of retention, persistence, student satisfaction, and academic performance.

Retention (Potts & Schultz, 2008; Miller, Janz & Chen, 2007; Ryan & Glenn, 2004; Derby & Smith, 2004; Jamelske, 2008) Persistence (Lang, 2007; Porter & Swing, 2006; Stovall, 2000) Student satisfaction and engagement (Summerlee & Murray, 2010; OGara, Karp, & Hughes, 2009; Engberg & Mayhew, 2007) Academic performance for both academically underprepared and well-prepared students (Potts & Schultz, 2008; Jamelske, 2008; Lang, 2007, Stovall, 2000)

QEP Goal and Program Outcomes

Primary Goal Enable first-time-in-college students to become self-reliant learners imbued with critical thinking skills. Student Outcomes Facilitate an increase in student retention rates, rates of persistence, and graduation rates. Foster increased rates of student satisfaction and student engagement. Faculty and Staff Outcomes Apply newly obtained knowledge to their practices to promote critical thinking and enhance the likelihood of success for first-year students. .

Student Learning Outcomes based on Frameworks

Critical Thinking

Apply the intellectual traits, standards, and elements of reasoning in the context of their personal and academic lives Demonstrate intellectual rigor and problemsolving skills by analyzing and evaluating information, generating ideas, and resolving issues Explore how background experiences impact their values and assumptions and explain how they influence personal relationships

Student Learning Outcomes based on Frameworks

Applied Learning

Enhance their awareness of the larger diverse community both inside and outside Edison State College
Apply one or more skills learned in the FYE course to other academic endeavors

Student Learning Outcomes based on Frameworks

Relevancy

Construct a plan for a successful path into and through completion of a degree or certificate Evaluate student-to-student and student-tofaculty interactions, and reflect on their relationship to academic, career, and social development Reflect on the General Education competencies at Edison State College and articulate their application to academic and career goals

Student Learning Outcomes based on Frameworks Success Strategies

Develop strategies for effective written and verbal communication, use of technology, listening, reading, critical thinking and reasoning Demonstrate independence and self-efficacy through effective personal management, use of college resources and the development of positive relationships with peers, staff and faculty

Cornerstone Experience Requirement Timeline

Initial All FTIC degree-seeking students testing into 2 or more Implementation developmental courses Beginning Spring 2012

YEAR 1 2012 - 2013

YEAR 2 2013 2014 YEAR 3 2014 - 2015 YEAR 4 2015 2016 YEAR 5 2016 - 2017

All FTIC degree-seeking students testing into 2 or more developmental courses

All FTIC degree-seeking students testing into any number of developmental courses All FTIC degree-seeking students testing into any number of developmental courses All FTIC degree-seeking students

All FTIC degree-seeking students

Direct Measures
Department/ Unit Outcome Critical Thinking Student Learning Outcomes Measurement Method/Assessment Tool Outcome-Specific Goal (Performance Expectation)

Results of the Critical Thinking Journal By the end of the Spring 2012 semester, 70% assessment scored with the Critical of students who complete the course will Thinking Rubric achieve a 3 (accomplished) or higher on all relevant aspects of the rubric).
Final Essay Assignment scored with Critical Thinking Rubric By the end of the Spring 2012 semester, 70% of students who complete the course will achieve a 3 (accomplished) or higher on all relevant aspects of the rubric. After completing the Cornerstone Experience course, students will have significant improvement in the following Critical Thinking Dispositions: Truth Seeking, Open Mindedness, Analyticity, Systematicity, Inquisitiveness, Confidence in Judgment, Maturity in Judgment.

Scores on the California Critical Thinking Disposition Inventory

Direct Measures
Department/ Measurement Unit Method/Assessment Tool Outcome Success Skills Scores on the Smarter Student Measure Learning Readiness Learning Indicator life factors, Outcomes personal attribute, technology knowledge, and technical competency items. Success Strategies Group Presentation rubric Outcome-Specific Goal (Performance Expectation) After completing the Cornerstone Experience course, students will have significant improvement in the following indicators: Personal Attributes, Life Factors, Technology Knowledge and Technology Competency.

Qualitative data from Final Essay assignment used to Develop a Success Strategies Rubric

By the end of the Spring 2012 semester, 70% of students that complete the course will achieve a 3 (accomplished) or higher on all relevant aspects of the rubric. An analysis of randomly selected essays will yield categories and concepts that can be used to identify success strategies that are most salient among students. This qualitative assessment will inform the development of a survey to be employed in subsequent terms.

Indirect Measures
Department/ Unit Outcome Student retention, persistence, and graduation rates. Measurement Method/Assessment Tool Within course completion rate (derived from course grade distributions) Term-to-term retention reports (derived from the Banner Student Information System) Outcome-Specific Goal (Performance Expectation) Once fully implemented, students will successfully complete the Cornerstone Experience at a rate of 85% with a C or better. Using AY 2011-12 baseline data, term-to-term retention will increase by 5% each year.

Year-to-year retention reports (derived from the Banner Student Information System)

Using AY 2011-12 baseline data, year-to-year retention will increase by 3% each year.

Indirect Measures
Department/ Measurement Unit Method/Assessment Outcome Tool
Student retention, persistence, and graduation rates. Cohort graduation reports derived through the Banner Student Information System

Outcome-Specific Goal (Performance Expectation)

This analysis will use the cohort graduation rate associated with students that entered ESC as FTIC during AY 10-11 (baseline). Cohorts from AY 11-12 and AY 12-13 who graduate within 150% of the expected time required will increase by 10% Cohorts from AY 13-14 and AY 14-15 who graduate within 150% of the expected time required will increase by 10% Cohort from AY 15-16 who graduate within 150% of the expected time required will increase by 10%

Course Outcome items Beginning AY 2012-13, faculty results for these items will from SIR II: 29, 30, 31, meet or exceed the comparative mean for four-year 32, 33 and Student institutions. Effort and Involvement items: 34, 35 and 36

Indirect Measures
Department/Unit Outcome Student satisfaction and student engagement. Measurement Method/Assessment Tool Outcome-Specific Goal (Performance Expectation)

Engaged Learning items from the Beginning AY 2012-13, there will be a 5% SENSE: 19a, 19b, 19e, 19g, 19h, 19i, increase in the Engaged Learning 19j, 19k, 19l, 19m, 19n, 19o, 19q, 20d2, benchmark over the previous years results. 20f2, and 20h2 Student-Faculty interactions items Beginning AY 2012-13, there will be a 5% from CCSSE: 4k, 4l, 4m, 4n, 4o, and 4q increase in the Student-Faculty interactions benchmark over the previous years results. Faculty/Student Interaction items from SIR II and a subset of Active and Collaborative Learning items from CCSSE (4f, 4g, 4h, and 4r) Qualitative data from focus group responses Beginning AY 2012-13, faculty results for these items will meet or exceed the comparative mean for four-year institutions. QEP Assessment committee will analyze levels of students satisfaction and engagement through a series of focus group discussions. Concepts and categories derived through this analysis will inform program and curricular enhancements.

Indirect Measures
Department/Unit Outcome Faculty application of professional development to promote critical thinking and enhance the likelihood of success for firstyear students. Measurement Method/Assessment Tool Academic Challenge items from CCSSE: 4p, 5b, 5c, 5d, 5e, 5f, 6a, 6c, 7, 9a (Fall 2012) Outcome-Specific Goal (Performance Expectation) Beginning AY 2012-13, there will be a 5% increase in the Academic Challenge benchmark over the previous years results. Following completion of the professional development modules, 80% of trained faculty will report using critical thinking and first-year student success strategies as measured on Likert scale items. Beginning AY 2012-13, faculty results for these items will meet or exceed the comparative mean for four-year institutions.

Professional Development Surveys

SIR II Communication items: 6, 7, 8, 9 and 10

Indirect Measures
Department/Unit Measurement Outcome Method/Assessment Tool Staff application of professional development to promote critical thinking and enhance the likelihood of success for firstyear students. Professional Development Surveys Outcome-Specific Goal (Performance Expectation) Following completion of the professional development modules, 80% of trained staff and administrators applying critical thinking and first-year student success strategies as measured on Likert scale items. Beginning AY 2012-13, there will be a 5% increase in A Plan and Pathway to Success benchmark over the previous years results.

SENSE items from A Plan and a Pathway to Success category: 18d, 18g, 18e, 18f, and 18h

Highlights of Quantitative Measures-Direct

Rubric Design and Data Collection Critical Thinking Journal Studies California Critical Thinking Disposition Inventory (CCTDI) Pre-Post SmarterMeasure Learning Readiness Indicator Pre-Post

Rubric Design and Data Collection

Common Course Assignments

Critical Thinking Journal Group Project: Success Strategies Final Essay Edison GPS (Go Picture Scribe)

Rubric Design and Data Collection

Critical Thinking Journal

Actual Results The students achievement of each dimension (Clarity, Accuracy, Relevance, Significance, and Logic) of the rubric was measured on a 4-point scale. Overall means for each dimension: Clarity: 2.73 (64.59% received 3 or higher) Accuracy: 2.94 (80.73% received 3 or higher) Relevance: 3.03 (85.37% received 3 or higher) Significance: 2.92 (75.79% received 3 or higher) Use of the Lee Campus Academic Success and College Prep Center labs has become more fluid. Students with writing needs receive assistance in either lab. Use of Results Students will receive rubric scores and feedback for each entry, rather then at mid-point and final. Faculty provide increased feedback on clarity and use of Standard English. Faculty encourage usage of Writing Centers.

Logic: 3.00 (82.70% received 3 or higher)

The goal for Accuracy, Relevance, Significance, and Logic were met. The number of students receiving a 3 or better for Clarity fell short of the stated goal (-.5.41%) with Clarity being the dimension with the lowest of the overall means.

Rubric Standardization
1. Please comment on how you believe the Critical Thinking Rubric worked for you in scoring essays today. 2. Looking at the levels or performance on the rubric, are any too similar? (e.g., Is it difficult to distinguish between a score of 4 and a score of 3?) Explain. 3. Examine the five criteria listed. Is there any redundancy? Do you believe you believe you may be scoring students more than once on the same criterion? 4. In what ways would you change the Rubric for ease of use? Use the attached form to be specific. 5. Do you have any other comments or suggestions about the Critical Thinking Rubric as a tool for scoring journal entries?

Inter-Rater Correlations

CCTDI Pre-test/Post-test

SmarterMeasure Pre-test/Post-test

Highlights of Quantitative Measures-Indirect

Term-to-Term Retention

Highlights of Quantitative Measures-Indirect

Table 1 Term to Term Retention by Base Fall Term Not Retained Following Term Fall 2011 Frequency 194

Retained Following Term 535

Totals

729 53.45

Percent Overall
Row Percent Column Percent

14.22
26.61 54.04

39.22
73.39 53.23

Fall 2012

Frequency Percent Overall Row Percent Column Percent

165 12.10 25.98 45.96

470 34.46 74.02 46.77

635 46.55

Total

Frequency Percent

359 26.32

1005 73.68

1364 100.00

X2 (1, N = 1364) = 0.069, p < 0.793

Highlights of Quantitative Measures-Indirect

Table 2 Fall 2012 Term to Term Retention by Participation in SLS 1515 Not Retained Retained Following Term Following Term Not in Frequency 57 104

Totals

161 25.35

SLS 1515

Percent Overall
Row Percent Column Percent

8.98
35.40 34.55

16.38
65.60 22.13

Enrolled in SLS 1515

Frequency Percent Overall Row Percent Column Percent

108 17.01 22.78 65.45

366 57.64 77.22 77.87

474 74.65

Total

Frequency Percent

165 25.98

470 74.02

635 100.00

X2 (1, N = 635) = 9.95, p < 0.002

Highlights of Qualitative Measures-Direct

Final Essay Assignment Grounded Theory Survey Development

Thematic Coding

Final Essay Prompt: Reflect upon the main themes that emerged from all the class topics and discussions and extract those ideas that you found most influential or important. Then create a new plan for achieving success in college and for establishing positive relationships with peers, staff and faculty. Random sample of 33% of sections.

Three-raters used a modified version of Grounded Theory to engage in open coding of essays followed by selective coding.
Essays were reviewed until saturation occurred. Codes were grouped into Concepts and Categories. Categories served as framework for development of a Success Strategies Survey.

Grounded Theory Method adapted from Charmaz, K. (2008) in Hess-Biber, S.N. and Leavy, P. (Eds.) Handbook of Emergent Methods. The Guilford Press.

Concepts-Listing, Describing, Applying Communication/Listening Skills

Diversity Knowledge Small Group Communication Skills Relationship Building Personality Inventories Campus Engagement Goal-Setting

Categories

Communication Strategies

Time Management Skills

Organizational Skills Persistence Critical Thinking Skills Course Success Strategies Learning Styles Use of College Resources

Goal Attainment Strategies

Cognitive Strategies

Please indicate the level of improvement you have made in the following areas as a result of completing the SLS 1515 Cornerstone Experience course. I didn't do I didn't do I didn't do this this much this much I have I completing the Please indicate the level of improvementbefore takingin the following areas as a result ofhave always you have made much before before always done well at SLS 1515 Cornerstone Experience course. taking this this class taking this done well at Response Answer Options this, but I have class and and now I class and this and Count made some now I still have now I have haven't improvement I didn't do this don't improved a Iimproved a changed didn't do this I didn't do this much before I always have always much before great dealhave well at Idone well at much before little this taking done
Answer Options

Arriving to class on time. Attending class.

Arriving to class on time.

2.5% (1) 2.5% (1)

taking this class taking this class and now and now I have class and now I have improved a I still don't improved a great deal little

Reviewing the course 2.5% schedule.Attending class. Using a calendar and/or (0) 7.7% (3) 10.3% (4) 33.3% (13) 48.7% 0% 2.5% 17.5% 25.0% lists to make sure course schedule. 7.5% Reviewing the (3) (1) (7) (10) assignments are completed on calendar and/or lists to make 7.5% (3) 2.5% (1) 17.5% (7) 25.0% (10) 47.6% Using a time. sure assignments Working on large are completed on time. Working on large projects 7.5% 22.5% projects incrementally incrementally 7.5% 10.0% (4) 22.5% (9) 10.0% 10.0% (4) 60.0% 10.0% (little by little, not waiting until the last (3) (3) (4) (9) (4) (little by little, not waiting minute). until the last minute).

0% (0)

2.5%
(1)

5.0% (2) 2.5% (1)

5.0%

7.7% (1) 2.5% (3)

(2)

10.3% (18) 33.3% (1) 10% (4)(4) 40.0% (16) 46.0% (13)

5.0%

5.0% (2) 10% (4)

this and haven't changed

(2)

46.0% (18) 40.0% (16) 46.0%

this, but I have Response Count made some improvement

42.5% (17)

42.5% (17) 46.0% (18)

40

40 40 39

48.7% (18) (19) 40

(19)

47.6% (19)
40

39

40

(19)

(20)

60.0% 40 (20)

40

Please indicate the level of improvement you have made in the following areas as a result of completing the SLS 1515 Cornerstone Experience course. I didn't do I didn't do this this much much before before taking taking this this class and class and now I have now I still improved a don't little I didn't do this much I have before always taking this done well at class and this and now I have haven't improved a changed great deal

Answer Options

I have always done well at Response this, but I have Count made some improvement

Using small group communication skills. Participating and asking questions when appropriate. Forming a relationship with other students. Meeting with the professor outside of class for help. Thinking critically about texts and lectures.

2.5%
(1)

10.0%
(4)

35.0%
(14)

17.5%
(7)

36.0%
(14)

40 40 40

7.5%
(3)

12.5%
(5)

22.5%
(9)

25.0%.
(10)

32.6%
(13)

2.5%
(1)

17.5%
(7)

20.0%
(8)

27.5%
(11)

32.6%
(13)

23.1%
(9)

20.5%
(8)

10.3%
(4)

15.4%
(6)

30.8%
(12)

39

5.1%
(2)

5.1%
(2)

36.9%
(14)

23.1%
(9)

30.8%
(12)

39

Highlights of Qualitative Measures-Indirect

Focus Groups

Highlights of Qualitative Measures-Indirect

Focus Group: Student Satisfaction and Engagement Focus group leader #1 will welcome the students, ask each to tell their name, describe the purpose of the session, and pose the questions to the group. At the end of the session, s/he will briefly summarize the main points and ask if anyone has anything they would like to add or amend.

1. What attracted you to Edison State College? (Why did you choose Edison State College?) 2.Tell us about your experience in SLS 1515 (Cornerstone Experience) course. 3. Which aspects of the SLS 1515 (Cornerstone Experience) course are you most satisfied with? 4. In what ways did the SLS 1515 (Cornerstone Experience) course help you achieve your academic or career goals? 5.Tell us about the Peer Architect (peer mentor) assigned to your class. (Follow-up: How did the Peer Architect help you achieve your academic or career goals?)

6. Which aspects of the SLS 1515(Cornerstone Experience) course are you least satisfied with? 7. Describe the campus-based activities you participated in as a result of taking the SLS 1515 course.

8. Outside of attending classes, tell us about any college activities that you participated in this term. 9. Outside of attending classes, what type of campus-based activities would you be interested in participating in? 10. What is the most effective way for the college to encourage your participation in campus activities?

Focus Group Themes

Engaging through the Passport Assignment Requesting more sports and/or intramural activities

Receiving communication about and timing of campus events

Not seeing the value of the Critical Thinking Exam Interacting with professor and other students

Requesting more technology training

QUESTIONS?

Contact
Kevin Coughlin Dean, Institutional Research, Planning and Effectiveness

kcoughlin@edison.edu
Eileen DeLuca Dean, College and Career Readiness and QEP Director

ecdeluca@edison.edu

http://www.edison.edu/cornerstone

References

Derby, D. & Smith, T. (2004). An orientation course and community college retention. Community College Journal of Research and Practice. 28, pp. 763-773. Engberg, M.E. & Mayhew, M.J. (2007). The influence of first-year success courses on student learning and democratic outcomes. Journal of College Student Development, 48(3), 241-1280010081.

Jamelske, E. (2008). Measuring the impact of a university first-year experience program on student GPA and retention. Higher Education, 57, 373-391. DOI 10.1007/s10734-008-9161-1.
Lang, D. (2007). The Impact of a first-year experience course on the academic performance, persistence, and graduation rates of first-semester college students at a public research university. Journal of the First-Year Experience & Students in Transition, 2007, vol. 19, no. 1 pp. 9-25. Miller, J.W., Janz, J.C. & Chen, C. (2007). The retention impact of a first-year seminar on students with varying pre-college academic performance. Journal of the FirstYear Experience and Students in Transition 19(1). 47-62.

References continued

OGara, L. Karp, M.M. & Hughes, K. (2009). Student success courses in the community college: An exploratory study of student perspectives. Community College Review. vol. 36, n. 3. pp.195-218. Porter, S. R. & Swing, R.L. (2006). Understanding how first-year seminars affect persistence. Research in Higher Education. vol 47, no. 1.pp. 89-109. Potts, G. & Schultz, B. (2008). The Freshman seminar and academic success of at-risk students. College Student Journal. pp. 647-658. Ryan, M.P & Glenn, P.A. (2004). What do first-year students need most: Strategies instruction or academic socialization? Journal of College Reading and Learning, 34 (2), Spring 2004. pp. 428. Stovall, M. (2000). Using success courses for promoting persistence and completion. New Directions for Community Colleges. No. 112, Winter 2000. pp.45-54. Summerlee, A. & Murray, J. (2010). The impact of enquiry-based learning on academic performance and student engagement. Canadian Journal of Higher Education. vol. 40, no. 2 pp. 78-94.